Pocket SEO » Site Analysis

Robots.txt – Watching the Minor Details

Josh — Sat, 22 Mar 2008 19:04:34 +0000

I noticed that Google was indexing one of my blog’s feeds that I thought I had blocked with robots.txt:

The indexed URL is pocketseo.com/domains/7/feed. Notice that it doesn’t have a trailing slash.

My robots.txt rule is:

Disallow: /*/feed/

WordPress URLs often have trailing slashes. I don’t want to remove that trailing slash from robots.txt otherwise it would block any URL that contains the word “feed”.

It’s not a major problem, but here is a quick solution for that one indexed URL:

Disallow: /*/feed/
Disallow: /domains/7/feed

It’s not important on a site like pocketseo.com, but I think that robots.txt rules are important on large sites and they can sometimes be tough to get right.

XML Sitemaps and Competitive Intelligence

Josh — Wed, 26 Dec 2007 13:04:46 +0000

Graywolf has an interesting post about using sitemaps for competitive intelligence:

…people who are using automated solutions like popularity contest are telling you their most highly trafficked pages. Other people who are generating Sitemap XML files in a more manual fashion, are telling you the pages they want to rank. Chances are good the pages they want to rank for are the â€œmoney pagesâ€.

It’s an interesting idea. If you are worried about someone doing this to your sites, you could try to hide your sitemaps by giving them unpredictable names and only using the ping technique to tell search engines where the sitemap index is located.

If you want to find your competitors’ sitemaps you could use search engine queries like this:

If you want to make sure that search engines can’t index your sitemap files you might be able to block them with an x-robots-tag HTTP header telling Google and Yahoo not to index them.

EDIT: I removed the idea of blocking the sitemap with a robots.txt noindex directive, because it probably won’t work in this situation.

How to Spot the Ultimate Robots.txt Mistake

Josh — Wed, 24 Oct 2007 23:40:36 +0000

I’ve seen this problem a surprising number of times: client accidentally blocks entire Web site with robots.txt file.

It often happens when files are copied from the dev server to the live server, and the dev server has a robots.txt file that blocks all robots.

Here is what it looks like:

The Google SERPs will not show a text snippet, and the title often becomes the words from the domain name (or the query) in lowercase letters:

Here is a live example from msplinks.com—they block the entire domain with robots.txt:

So if you see that kind of result in Google’s SERPs for your site, check the robots.txt file.

This mistake won’t knock your site out the SERPs right away, but it will probably reduce traffic very quickly.

One way to prevent it is not to block robots on the dev server with the robots.txt file. Use password protection instead.

Introduction to Advanced SEO Site Analysis for Large Web Sites

Josh — Thu, 18 Oct 2007 17:02:18 +0000

I often hear SEOs and Web developers make broad statements about large Web sites after just a few minutes looking at them. The statement often goes something like this:

“This site looks well optimized and gets millions of visitors per month. Your title tags are optimized and you have meta tags. Just keep building more content for your users and search engines will love it. There is nothing more do to for the site’s SEO.”

There is a lot more to SEO than just optimizing title elements (not title tags) and meta tags, or randomly “adding more content”. There is almost always more SEO that can be done to a Web site, no matter how much traffic it already gets. This post lists some ways that one can start analyzing a large site and find issues that could be addressed to increase search engine referrals.

The site: Query

One of the first things to look at on a large Web site is the site: query in Google, Yahoo, and MSN. Just go to each of the three search engines and enter site:example.com and the search engines will return a list of indexed pages.

You can use Google’s URL parameters to switch to 100 results per page which allows to you scan the URLs that have been indexed. Look for titles and text snippets (usually the meta description) that are identical across many pages. Also scan the indexed URLs for things like:

URLs that should not be indexed
URLs that were indexed over HTTPS
URLs that have many parameters
URLs that were indexed on the www subdomain like http://www.example.com/ as well as without a subdomain like http://example.com/.

Google will only show 1000 results. To show more than 1000 results you have to get more specific by querying directories. For example, to find all of the pages in a directory called products use this Google query: site:http://example.com/products/. You can leave the http:// part off, but be aware that a URL that starts with http:// is different than one that starts with https://. If you include the www Google will only show results from the www subdomain. If you leave the www off, you will get indexed pages from all subdomains.

Also run the site: query in Yahoo and MSN. Yahoo will often reveal strange indexed URLs like the ones created by ASP.NET Session Service. Those ASP.NET URLs look like this, with the highlighted part being a kind of session ID that changes:
http://example.com/(lit3py55t21z5v55vlm25s55)/Application/SessionState.aspx

It’s also useful to check how MSN Live Search has indexed your site because it can provide additional data about how your site is being crawled.

Find Subdomains

You should check for subdomains on the site to see if there is duplicate content or other unwanted sections of the site being indexed. Large Web sites often have multiple subdomains for load balancing or other reasons. One common situation is: www.example.com, www2.example.com, www3.example.com—all with the same content.

Here is an example from CNN.com:

I’ve marked the subdomains that contain duplicate content.

For example, the subdomain www-cgi.cnn.com has over 17,000 pages indexed and they look like duplicates of the pages on the main domain.

Notice that beta.cnn.com has also been indexed. It looks like someone tried to block that from robots with a robots.txt file that contains the rule Disallow: /beta (hint: that won’t work).

Off-Domain Duplicate Content

In addition to duplicate content on subdomains, you may also find the site duplicated on other domains. Often people will say things like, “I want my site to be accessible at example.com, example_1.com, and example_2.com”. They end up with their entire site duplicated across other domains.

You can search for these duplicate domains by copying a sentence of text (or a unique string of words) from the Web site and searching Google with it. Put the sentence in quotes to return only exact matches. If the content appears on other sites, those sites will probably turn up in a Google search.

You can also often find clients’ other domain names by using the ip: query in MSN Live Search. Use the Show IP Firefox Extension to find the IP address of the client’s Web site. Then go to MSN Live Search and type in:
ip:nnn.nnn.nnn.nnn (replace the letters “n” with the IP address)

MSN will return a list of other sites at the same IP address.

Also, ask the client, “Do you have any other domain names that point to the same server/content?”

Duplicate Content From Referrers

URLs with referrer information (e.g., affiliate program tracking) can create multiple URLs for a single page of content.

Here is another example from CNN.com showing that the home page is indexed with many different URLs:

You can find these in Google with a query like:
intitle:”some text from the page’s title element”

Site Spider

It is useful to locate URLs that return 302, 403, 404, or other undesired headers. It’s also good to locate URLs that don’t exist, but that send 200 OK headers.

I used to write short custom scripts to check response headers on sites, but now I usually use the free AuditMyPC.com Sitemap Generator which can generate a useful spreadsheet of HTTP response headers.

If you are using the AuditMyPC.com tool, be sure to respect robots.txt and rel=nofollow as shown below:

You can also set a crawl delay so that it doesn’t put too much load on their servers.

Analyzing the Raw Logs

Grepping your raw log files can turn up a lot of useful information about how spiders are perceiving your site. Basically, you want to extract all of the hits that were made by Google’s, Yahoo’s, and MSN’s spiders, and then see what kind of response codes were sent.

This is useful for finding URLs that search engines are requesting but not indexing, or large numbers of internal 302 redirects that you might have missed.

Site Architecture

This is a large topic, so this is just a quick overview of where you might start:

Check each page that is linked to from the home page. Are they category pages? Do the category pages link to the pages that will bring maximum SEO benefit? Is the site keeping the important content within a few clicks of the home page?

Piece by Piece

Attack the site’s SEO issues in smaller chunks. Optimize smaller sections of the site at a time and observe how the search engines respond. Then move on to the next SEO issue.

Conclusion: SEO for Large Web Sites

The techniques above are just a few places to start when analyzing a large Web site. Even those few analysis methods should give you lots of SEO material to work on that go beyond the obvious basics like page titles, meta tags, and alt attributes.

6 Reasons Why Clean URLs Matter

Josh — Mon, 15 Oct 2007 19:36:22 +0000

I’ve been reading SEO forums lately and have read some comments that there is absolutely no difference between using static or dynamic URLs.

A site can get indexed and ranked well if it has dynamic URLs, but that does not mean that dynamic URLs are as good as clean URLs.

Note: if you have dynamic URLs, don’t just change them to clean URLs after reading this without knowing what you are doing. I’ll cover the risks involved with doing that in another post. Changing URL structure from dynamic to static would not be recommended in all cases. This post just contains general advice on why you should start with clean URLs.

1. The search engines recommend clean URLs

MSN says it very plainly, “Keep your URLs simple and static.”

Google — “If you decide to use dynamic pages (i.e., the URL contains a “?” character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.”

Update: (26 Nov 07) Google has updated their Webmaster Guidelines to specifically talk about problems with dynamic URLs:

Google indexes dynamically generated webpages, including .asp pages, .php pages, and pages with question marks in their URLs. However, these pages can cause problems for our crawler and may be ignored. If youâ€™re concerned that your dynamically generated pages are being ignored, you may want to consider creating static copies of these pages for our crawler. If you do this, please be sure to include a robots.txt file that disallows the dynamic pages in order to ensure that these pages arenâ€™t seen as having duplicate content.

Yahoo’s answer is more complex because they have a dynamic URL tool in SiteExplorer. Yahoo clearly states some of the problems that dynamic URLs can create for spiders:

You might use URL parameters in your site to perform various functions apart from modifying the content of the page, such as,

session ids – for tracking user sessions

source trackers â€“ for tracking the sources which are sending referrals to your pages and site

format modifiers â€“ for print formats etc

In these cases, our crawler will come across different versions of your site’s URLs that are substantially, if not exactly, similar in content. This causes numerous problems:

Such URLs look like different documents to our crawler and create excessive crawling on your site.

We do our best to detect duplicates among these pages, and the detected duplicates are prevented from ranking well. However, if we are unable to detect the duplicate, this results in duplicate results from your site competing for positions in search results

When people link to your site with different versions of these URLs, it fragments the link referrals to your site across multiple different URLs even though it’s the same page.

Ask.com doesn’t have much information on dynamic URLs, except for a quick mention on their UK site:

We include a select number of dynamic URLs in our index. However, they are screened to detect likely duplicates before downloading.

Note that Ask.com carefully uses the phrase, “a select number”.

Search engines are not saying that they cannot spider and index dynamic URLs, but they are giving very clear hints that it is easier for them to index static URLs.

2. Dynamic URLs probably do not pass link value in the same way

Matt Cutts talked a little bit about this in one of his videos. Here is a transcription, with relevant parts highlighted:

Does Google treat Dynamic Pages differently than static pages…?

Good question. To a first approximation, we do treat static and dynamic pages in similar ways in ranking. So let me explain that in a little more detail.

Pagerank flows to dynamic URL’s in the same way they flow to static URL’s. And so, if you’ve got New York Times linking to a dynamic URL you’ll still get the Pagerank benefit, and it will still flow the Pagerank Benefit. There are other Search Engines who in the past have said, ‘OK, we’ll go one-level deep from static URLs, so we’re not going to crawl from a dynamic URL, but we’re willing to go into the dynamic URL space from a static URL’. So, the short answer is Pagerank still flows just the same between static and dynamic.

Let’s go into the more detailed answer.

The example you gave actually has 5 parameters, and one of them is like a Product ID with like 2725. You definitely can use too many parameters. I would absolutely opt for 2 or 3 at the most, if you have any choice whatsoever. And try to avoid long numbers, because we can think that those are session IDs. Any extra parameters that you can get rid of are always a good idea.

And remember that Google is not the only Search Engine out there, so if you have the ability to basically say I’m going to use a little bit of mod_rewrite and I’m going to make it look like a static URL, that can often be a very good way to tackle the problem. So, PageRank still flows but, experiment! If you don’t see any URLs that have the same structure, or the same number of parameters as you’re thinking about doing; it’s probably better if you can either cut back on the number of parameters or shorten them somehow or try to use mod_rewrite.

So Matt Cutts is saying that, “Pagerank flows to dynamic URL’s in the same way they flow to static URL’s.” Does that mean that PageRank does not flow from dynamic pages the same way as it does from static URLs? For example, if a dynamic URL links to another dynamic URL, is it the same as if a static URL links to a page?

Yahoo used to link to a good powerpoint presentation from this page called Search Friendly Design (hopefully they will put it back or update it).

Here is a screenshot from the presentation:

Database-Driven Sites

What gets crawled

Static URLs

Dynamic Pages with in-links from static pages

Links between dynamic pages are problematic for crawlers (some get crawled, some don’t)

Limit URL depth when using dynamic-to-static

The presentation was from 2004, and it’s possible that Yahoo has upgraded their indexer to the point where dynamic URLs don’t matter at all anymore, but I doubt that Yahoo handles dynamic URLs in the exact same way as static URLs.

Here is another slide from the presentation where Yahoo (Tim Mayer) says that Yahoo “…won’t crawl… spider ‘traps’ (dynamic content)”.

Not all dynamic content is a “spider trap”, but the concept of spider trap should be kept in mind when looking at dynamic URLs. Consider the perspective of a search engine architect: dynamic Web sites can generate an unlimited number of pages. The spider has to be able to automatically detect when it’s getting caught in an endless loop of dynamic content (“spider trap”). This is probably why Google strongly recommends not to exceed a a couple of URL parameters.

The search engines representatives are clearly hinting that you should watch out for dynamic URLs.

3. Dynamic URLs often work even when the parameters are reversed, added, or removed

For example, the URL http://example.com/index.php?name=abc&num=123 might also load at http://example.com/index.php?num=123&name=abc. If search engines find both links you may end up with multiple indexed URLs for the same page of content.

Here is an example of dynamic URLs causing problems on YouTube—a site that would benefit from some SEO if Google weren’t automatically inserting the site in the SERPs with Universal Search:

Both URLs load the same page of content and both are indexed by Google.

4. Dynamic URLs are harder to block with robots.txt

It is more difficult to block pages’ dynamic URLs with robots.txt files. The robots.txt standard (which needs updating) doesn’t yet officially support wildcards, though Google, Yahoo, ~~and MSN~~ all do. [UPDATE: MSN doesn't fully support wildcards in robots.txt] For example, Google could block the duplicate URLs above on YouTube with the following robots.txt rules, but it’s an extension of the standard and may not work with all robots:

Disallow: /*mode= Disallow: /*search=

If you only use static URLs for your site, you also have the advantage of being able to easily block weird query strings that might appear on your site.

For example, if all of your main URLs are clean you can block weird URLs that you might not have anticipated like http://example.com/widgets/5/?ajax_callback=true.

Update: 27 Nov 2007 Please disregard the following advice for the moment and see this post about Google and robots.txt before implementing it:

~~A single robots.txt rule should at least block those kinds of unexpected URLs from at least Google and Yahoo:~~

# this blocks all dynamic URLs from Google and Yahoo Disallow: /*?

Warning: Be very careful before implementing that last robots.txt rule though because there may be essential pages on the site that have dynamic URLs. There are many factors to consider before doing that, such as “are there linked-to, indexed dynamic URLs on the site that are 301 redirecting to new URLs?” Basically, it should only be done on new sites where you are sure that you have constructed the site so that your essential site structure does not contain any dynamic URLs.

5. Clean URLs are more memorable

It’s better to send people the URL http://example.com/pagename than http://example.com/index.php?page=pagename.

If you have an ecommerce site, it’s easier for your customers to remember http://example.com/widgets than http://example.com/index.php?category_id=widgets.

6. Tim Berners-Lee says so (basically)

Tim Berners-Lee, the inventor of the WWW, has a great page of information about URL structure. He doesn’t specifically mention dynamic URLs, but reading the article will give a sense of how to create good URLs. File name extensions and software mechanisms should not appear in URLs.

Conclusion: why static URLs are better than dynamic

I’ve heard even some well-known SEOs say things like “there is no difference between static and dynamic URLs—just look at the dynamic URLs on my well-indexed site as an example”.

I’m not saying that dynamic URLs won’t get indexed or won’t rank highly. Search engines can spider and index dynamic URLs. But anyone who says, “I already rank highly for my main keywords; I don’t need SEO” does not understand the potential of SEO. Traffic can almost always be increased on a Web site, even on sites that rank #1 for highly competitive keywords.

Negative SEO

Josh — Thu, 16 Aug 2007 19:00:45 +0000

Forbes.com has an article called The Saboteurs of Search which contains a slideshow of “7 Ways your Site Can be Sabotaged”.

They quoted Matt Cutts:

Matt Cutts, a senior software engineer for Google, says that piling links onto a competitor’s site to reduce its search rank isn’t impossible, but it’s extremely difficult. “We try to be mindful of when a technique can be abused and make our algorithm robust against it,” he says. “I won’t go out on a limb and say it’s impossible. But Google bowling is much more inviting as an idea than it is in practice.”

Defense against “negative SEO” involves knowing how it is performed. I’ve categorized this post under Site Analysis because it involves things you might look for when analyzing a site.

This is Forbes.com’s list of the 7 ways to sabotage a Web site:

Google Bowling — Creating many low-quality inbound links to a site. Easy to discover.
Tattling — Reporting paid links to Google through a spam report. If you don’t buy links, it probably isn’t a problem. People who would use the dark side of SEO probably buy links also.
Google Insulation — Creating content that ranks above the competitor’s site.
Copyright Takedown Notices — Reporting another Web site for copyright infringement can have it taken down for 10 days. This is probably illegal and opens the perpetrator to lawsuits.
Copied Content — Creating duplicate content. More difficult to trace, but you can defend against it by monitoring for duplicate content. If they are using proxies to duplicate your content you can send alternate content to the proxies by IP address.
Denial of Service — Crash the Web site with a DOS attack. Definitely illegal.
Click Fraud Simulate click fraud on competitor’s Web site.

Has anyone been affected by these techniques, or know of any others?

How to Discover Subdomains on Client Sites

Josh — Thu, 24 May 2007 03:27:47 +0000

On large, complex sites you will often find many "hidden" subdomains that duplicate the content on the main domain. You might even find that the dev server has been indexed (e.g., http://dev.example.com/). Checking for subdomains should be part of every site audit.

A simple Google query will show you a list of subdomains on a site by excluding domains that begin with www:

site:example.com -www.example.com

Change the view to 100 per page and visually scan the SERPs for subdomains. You can quickly convert the view to 100 results per page by appending the text &num=100 to the Google SERPs URL.

Below is an example query that shows the subdomains on Google.com. Note the &num=100 on the end of the URL that tells Google to display 100 results:

http://www.google.com/search?q=site%3Agoogle.com+-www.google.com&num=100