Skip to content
Nov 11 / Barry

Short thoughts: robots.txt and paywalls

Murdoch’s reluctance to simply use the robots.txt protocol to block google from indexing News Corp properties is most likely not because of technical ignorance. He may be angling for

  1. an exclusive deal with a search engine
  2. a special indexing setup so that they show up in search results, but searchers are directed to a login page before accessing the content.

Edit: Ben Barren notes that Calcanis has been positing this very idea.

Filed under analysis
  • It's all a move to get another search engine (ahem... Bing) to bid on the rights to list them exclusively.
  • That's a very good thought. Not all search engines respect robots.txt restrictions, but I've never heard that Google doesn't abide by them.
blog comments powered by Disqus