Comments on: Google Does Not Obey Robots.txt

By: Andy Beard

Andy Beard — Thu, 20 Sep 2007 21:35:23 +0000

Hmm, it seems it has stopped being indexed now, that is interesting. It might actually be working as intended.

By: Josh

Josh — Thu, 20 Sep 2007 18:03:39 +0000

Hi Andy, thanks for your comment.

Feedburner is saying that Googlebot is spidering your feed? Have you seen it indexed? (e.g., cache:example.com/page/feed/)

On at least one of the sites that I’ve noticed this problem on, Google’s robots.txt tool says that the pages are blocked. But Google still spiders and indexes them…

By: Andy Beard

Andy Beard — Thu, 20 Sep 2007 17:50:25 +0000

What I have seen is something slightly different.

Feedburner provide a way to block search engines from crawling your feed. It is code in the RSS itself.
Within my Feedburner stats it is still saying that Google is visiting (no not just the bot picking up feeds for subscribers)