Yahoo!, Google, Microsoft Clarify robots.txt Support

Yahoo!, Google, Microsoft Clarify robots.txt Support

Today, Google, Yahoo!, and Microsoft have come together to post details of how each of them support robots.txt and the robots meta tag. While their posts use terms like "collaboration" and "working together", they haven't joined together to implement a new standard (as they did with sitemaps.org). Rather, they are simply making a joint stand in messaging that robots.txt is the standard way of blocking search engine robot access to web sites. They have identified a core set of robots.txt and robots meta tag directives that all three engines support:

For robots.txt, they all suport:


* Disallow

* Allow

* use of wildcards

* Sitemap location

For robots meta tags, they all support:


* noindex

* nofollow

* noarchive

* nosnippet
* noodpt

Google and Yahoo! already supported and documented each of these and Microsoft supported most of them before this announcement. In their posts, they also list the directives they support that may not be supported by the other engines.