Murdoch’s reluctance to simply use the robots.txt protocol to block google from indexing News Corp properties is most likely not because of technical ignorance. He may be angling for
- an exclusive deal with a search engine
- a special indexing setup so that they show up in search results, but searchers are directed to a login page before accessing the content.
Edit: Ben Barren notes that Calcanis has been positing this very idea.
That’s a very good thought. Not all search engines respect robots.txt restrictions, but I’ve never heard that Google doesn’t abide by them.
It’s all a move to get another search engine (ahem… Bing) to bid on the rights to list them exclusively.