Here is a simple technique for rapid indexing by Google:
(Clarification: this technique is not just for blogs. It works great and is much easier to implement than XML sitemaps which are difficult to get right on large sites.)
Google uses it Blogsearch feature to determine when your site was updated based on your RSS/ATOM feeds. You can often find your content in the Google SERPs very soon after it appears in Google Blogsearch.
This technique does not provide results every time, but works in many cases. It may depend on things like whether your site has already been recently crawled or whether it is worth crawling every time you ping Blogsearch.
RSS & ATOM feeds have data in them that tell search engines when a specific page was created and modified. In RSS it looks like this:
The technique is simple:
- Make sure that your RSS/ATOM feed is not blocked to search engines, specifically Google. Don’t block your main feed with robots.txt or with Feedburner’s noindex feature.
- Make sure all new content that is added to your site has an XML feed, either RSS or ATOM (I think ATOM is better in general, but it doesn’t matter for this technique)
- You may have to also have categories enabled in the feed as well as auto-discovery in the HTML head on the home page. I haven’t tested those aspects in depth.
- Ping Google Blogsearch
You will often then appear in both Google Blogsearch and the regular Google SERPs very quickly sometimes in just a few minutes. RSS/ATOM feeds are not just for blogs…
In the case of this blog post, it took 2 minutes to appear in Google Blogsearch:
Shortly after, the text snippet of the home page in Google’s regular SERPs reflected the new content: