Hide your dev and stage environments with robots.txt

Have you ever had a client call you up saying their new website, which has not launched yet, is showing up in Google?

No matter how much you think, "we won't get crawled", those pesky search engines always seem to find your development site. Thankfully there is an agreed-upon standard for removing a website (or just certain pages) from search engines called the robots.txt file.

Drupal actually ships with a robots.txt that tells search engines to ignore certain kinds of pages on every Drupal site - you may also wish to add other rules, depending on your use case. If you want to tell search engines to get off your porch, try this:

# tell search bots to ignore this site
User-agent: *
Disallow: /

For more info about how to make robots do your bidding, check out http://www.robotstxt.org or the Drupal Handbook page on robots.txt