Pocket SEO » MSN

MSN Live Search Only Has Partial Support for Wildcards in Robots.txt

Josh — Sun, 18 Nov 2007 22:54:16 +0000

Google and Yahoo both support wildcards (*) and end of string ($) characters in robots.txt files.

MSN’s Live Search is a little more confusing because they only have very limited support for wildcards in robots.txt files. Based on their docs, it looks like wildcards are supported. These are valid robots.txt rules for MSN’s Live.com:

User-agent: msnbot
Disallow: /*.PDF$
Disallow: /*.jpeg$ 
Disallow: /*.exe$

However, “MSNdude” recently stated on WebmasterWorld that “Live Search does not support wildcards in robots.txt today; we are thinking about it.”

An asterisk that substitutes for another set of characters is a wildcard, so this statement is confusing.

I think that wildcards should be added to the robots.txt standard. Wildcards in robots.txt files are essential for the ability to block certain kinds of dynamic URLs. The original robots.txt standard should be updated and MSN should fully jump on board.

MSN Keyword Referrer Spam Continues, Distorting Analytics Everywhere

Josh — Fri, 16 Nov 2007 10:15:31 +0000

As mentioned in an earlier post, MSN Live Search is sending referrer spam. If you don’t know about it yet, read my previous article on it. The referrer spam is those highly competitive, one-word keywords that Live.com is supposedly referring to your Web site.

The referrer strings look like this:
http://search.live.com/results.aspx?q=travel&mrt=en-us&FORM=LIVSOP

Notice that the keywords are generally one word and highly competitive and some of the keywords are porn-related.

I wish that A-list bloggers would pick this story up so that people would start to pay attention and get Microsoft to stop. (This blog doesn’t get enough traffic to spread the word by myself.) This referrer spam is distorting everyone’s logs, and making it look like Live.com gets more traffic than it really does.

I mentioned why I think it is referrer spam in this WebmasterWorld thread.

“MSNdude” said that the fake referrers were part of a “quality check”. What kind of quality check would require sending referrers to webmasters? There is no reason that a webmaster would have to see a keyword in the referrers for Microsoft to do a “quality check”.

I believe that they send referrer strings that match the content of the site to encourage webmaster to click through to the Live.com search pages in the hope that it will drive up Live.com’s traffic. I am guessing that they probably think it will cause more people to start using Live.com regularly.

Sending a fake referrer = referrer spam.

MSN Live Search Indexing URL Fragments

Josh — Tue, 23 Oct 2007 19:04:58 +0000

I noticed that MSN was indexing URL fragments. A URL fragment is the part of a URL that comes after a pound sign like this:

http://example.com/page#section

Here is a screenshot.

(Sorry, I had to block out part of the URL.)

Google removes the fragments from URLs.

What will the effect be of indexing anchors as separate URLs? Single pages with multiple URLs? Is it just a temporary MSN SERPs bug like this one?

MSN Live.com Sending Referrer Spam?

Josh — Thu, 18 Oct 2007 19:32:48 +0000

Lately I’ve been noticing a lot of strange one-word referrers from MSN Live Search in my logs. Today I saw a post by BitWorm that reveals this as referrer spam-like behavior from Microsoft.

The hits come from IP addresses beginning with 65.55.165.* and the referrer strings have this format:
http://search.live.com/results.aspx?q=firefox&mrt=en-us&FORM=LIVSOP

The keyword (highlighted above) is always one word, and usually appears on the Web site somewhere, with the exception of the porn related keywords.

MSNdude posted a comment on the Webmaster World thread about this issue:

Thanks for all the feedback on this thread.

First, we appreciate the concerns and issues that have been raised and apologize for any incovenience this might have caused.

Second, we want to explain what this is all about. The traffic you are seeing is part of a quality check we run on selected pages. While we work on
addressing your conerns, we would request that you do not actively block the IP addreses used by this quality check; blocking these IP addresses could prevent your site from being included in the Live Search index.

Please keep the feedback and thoughts coming as we will use this to help improve this process and make sure that it impacts your sites as little as possible.

thanks
- msndude (msd)

How can MSN call this a “quality check”? They are sometime using explicit porn keywords that do not appear anywhere on my sites.

I recommend reading through the entire Webmaster World thread about this MSN referrer issue.

I wonder if MSN is doing this for old-fashioned referrer spam reasons. The keywords that they use in the referrer string are related to the site’s content. So Microsoft sends keywords that they think the webmaster might be interested in so that the webmaster will click through to MSN Live Search.

The reason that I suspect that it might be traditional referrer spam is because MSN is trying very hard to get people to use MSN Live Search, even to the point of buying Google AdWords. Every time you click through your logs to MSN Live Search, it boosts Live.com’s traffic stats.

Also check out the post about it on andrewu.co.uk, Odd Referrer Spam-Like Behaviour From Microsoft.

If you notice this behavior from Microsoft’s bots on your sites blog about it and let people know about the problem.

I’m going to call this referrer spam until MSN speaks out about the issue and stops pinging my logs with porn keywords. They seem to have been silent except for that one odd comment on Webmaster World by “msndude”.

Update: MSN continues assault on logfiles

I also recommend reading this excellent smackdown which gives details about how this Microsoft bot is even downloading Adsense JavaScript from Web sites.

MSN Search Buys Google Adwords

Josh — Mon, 11 Jun 2007 23:21:57 +0000

While searching for the keyword blog in Google today I noticed that MSN’s Live.com was buying Adwords.

Here is a closeup of the ad:

I was curious about what MSN might be advertising so I clicked on the ad. It took me to the MSN Search results for the query what is a blog:

Here is a closeup:

It seems like an act of desperation for MSN to buy Google Adwords for MSN’s own SERPs. The ad didn’t even take me to Live.com, but to the URL http://search.msn.com/results.aspx?q=what+is+a+blog&FORM=SEMTC3&vendor=google&pkw=what+is+a+blog

Serious Bug in MSN SERPs

Josh — Mon, 11 Jun 2007 04:04:53 +0000

MSN Live Search is not displaying URLs correctly. In the screenshot below, it shows URLs from PocketSEO.com without a trailing slash. Yet the link above it goes to the correct URL (with the trailing slash).

A slash is the symbol for a directory. In *nix, a directory is treated as a file. A URL without a trailing slash is a different URL than one with a trailing slash. MSN Live Search is displaying the wrong URLs in the SERPs.