Short thoughts: robots.txt and paywalls
Murdoch’s reluctance to simply use the robots.txt protocol to block google from indexing News Corp properties is most likely not because of technical ignorance. He may be angling for
- an exclusive deal with a search engine
- a special indexing setup so that they show up in search results, but searchers are directed to a login page before accessing the content.
Edit: Ben Barren notes that Calcanis has been positing this very idea.
That’s a very good thought. Not all search engines respect robots.txt restrictions, but I’ve never heard that Google doesn’t abide by them.