Comments on: MSN Live Search Only Has Partial Support for Wildcards in Robots.txt http://pocketseo.com/msn/177 Practical SEO blog Thu, 24 Jun 2010 20:09:07 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: 6 Reasons Why Clean URLs Matter - Pocket SEO http://pocketseo.com/msn/177/comment-page-1#comment-769 6 Reasons Why Clean URLs Matter - Pocket SEO Sat, 22 Mar 2008 16:30:19 +0000 http://pocketseo.com/msn/177#comment-769 [...] doesn’t yet officially support wildcards, though Google, Yahoo, and MSN all do. [UPDATE: MSN doesn’t fully support wildcards in robots.txt] For example, Google could block the duplicate URLs above on YouTube with the following robots.txt [...] [...] doesn’t yet officially support wildcards, though Google, Yahoo, and MSN all do. [UPDATE: MSN doesn’t fully support wildcards in robots.txt] For example, Google could block the duplicate URLs above on YouTube with the following robots.txt [...]

]]> By: Michael VanDeMar http://pocketseo.com/msn/177/comment-page-1#comment-498 Michael VanDeMar Tue, 27 Nov 2007 15:31:49 +0000 http://pocketseo.com/msn/177#comment-498 Well, aside from the fact that MSNdude hasn't been the most credible source lately imo (ie. the reason he gives for the Live bot referrer spamming our sites), you also need to take into account that at times MSN completely ignores robots.txt. For instance, that same spamming bot is downloading AdSense, which as you can see is <i>clearly</i> blocked by Google: http://pagead2.googlesyndication.com/robots.txt Well, aside from the fact that MSNdude hasn’t been the most credible source lately imo (ie. the reason he gives for the Live bot referrer spamming our sites), you also need to take into account that at times MSN completely ignores robots.txt. For instance, that same spamming bot is downloading AdSense, which as you can see is clearly blocked by Google:

http://pagead2.googlesyndication.com/robots.txt

]]> By: Google’s Vastly Improved Webmaster Guidelines - Pocket SEO http://pocketseo.com/msn/177/comment-page-1#comment-487 Google’s Vastly Improved Webmaster Guidelines - Pocket SEO Mon, 26 Nov 2007 22:09:16 +0000 http://pocketseo.com/msn/177#comment-487 [...] That page also provides a great argument for updating the robots.txt standard to include wildcards (*) and end of line characters ($). It is not possible to block complex dynamic URLs without wildcards and end of line characters. MSN Live, for example, does not fully support wildcards in robots.txt. [...] [...] That page also provides a great argument for updating the robots.txt standard to include wildcards (*) and end of line characters ($). It is not possible to block complex dynamic URLs without wildcards and end of line characters. MSN Live, for example, does not fully support wildcards in robots.txt. [...]

]]>