I noticed that Google was indexing one of my blog’s feeds that I thought I had blocked with robots.txt:
Continue reading →

Graywolf has an interesting post about using sitemaps for competitive intelligence:
Continue reading →

I’ve seen this problem a surprising number of times: client accidentally blocks entire Web site with robots.txt file.

It often happens when files are copied from the dev server to the live server, and the dev server has a robots.txt file that blocks all robots.

Here is what it looks like:
Continue reading →

I often hear SEOs and Web developers make broad statements about large Web sites after just a few minutes looking at them. The statement often goes something like this:

“This site looks well optimized and gets millions of visitors per month. Your title tags are optimized and you have meta tags. Just keep building more content for your users and search engines will love it. There is nothing more do to for the site’s SEO.”

There is a lot more to SEO than just optimizing title elements (not title tags) and meta tags, or randomly “adding more content”. There is almost always more SEO that can be done to a Web site, no matter how much traffic it already gets. This post lists some ways that one can start analyzing a large site and find issues that could be addressed to increase search engine referrals.
Continue reading →

I’ve been reading SEO forums lately and have read some comments that there is absolutely no difference between using static or dynamic URLs.

A site can get indexed and ranked well if it has dynamic URLs, but that does not mean that dynamic URLs are as good as clean URLs.

Note: if you have dynamic URLs, don’t just change them to clean URLs after reading this without knowing what you are doing. I’ll cover the risks involved with doing that in another post. Changing URL structure from dynamic to static would not be recommended in all cases. This post just contains general advice on why you should start with clean URLs.
Continue reading →

Forbes.com has an article called The Saboteurs of Search which contains a slideshow of “7 Ways your Site Can be Sabotaged”.
Continue reading →

On large, complex sites you will often find many "hidden" subdomains that duplicate the content on the main domain. You might even find that the dev server has been indexed (e.g., http://dev.example.com/). Checking for subdomains should be part of every site audit.

A simple Google query will show you a list of subdomains on a site by excluding domains that begin with www:
Continue reading →