I’ve seen this problem a surprising number of times: client accidentally blocks entire Web site with robots.txt file.
It often happens when files are copied from the dev server to the live server, and the dev server has a robots.txt file that blocks all robots.
Here is what it looks like:
The Google SERPs will not show a text snippet, and the title often becomes the words from the domain name (or the query) in lowercase letters:
Here is a live example from msplinks.com—they block the entire domain with robots.txt:
So if you see that kind of result in Google’s SERPs for your site, check the robots.txt file.
This mistake won’t knock your site out the SERPs right away, but it will probably reduce traffic very quickly.
One way to prevent it is not to block robots on the dev server with the robots.txt file. Use password protection instead.
