The Googlebot
For those of you who don't know what a googlebot is, it crawls your site and indexes the content for google searches.
As I develop I want to make sure that my half-finished product does not appear in any search results, so it's important that googlebot knows to ignore my site. Here are some sources explaining how the googlebot works, and how to have it ignore your site.
Googlebot,
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=182072
rel=“nofollow”,
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=96569
telling the googlebot no,
http://www.mattwaite.com/posts/2009/apr/12/telling-google-bot-no/

0 Comments:
Post a Comment
Subscribe to Post Comments [Atom]
<< Home