Wednesday, November 24, 2010
Do you know how Google's crawler, Googlebot, handles conflicting rules in your robots.txt
file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite
song? The answers to these questions (except for the last one :)), along with lots of other
information about controlling the crawling and indexing of your site, are now available on
code.google.com
:
Controlling crawling and indexing
![Googlebot doing its happy dance](https://developers.google.cn/static/search/blog/images/import/5484548329262f897fbe7f16ca9fe4f9.png?authuser=2)
Now site owners have a comprehensive resource where they can learn about robots.txt files,
robots meta
tags, and X-Robots-Tag
HTTP header rules. Please share your
comments, and if you have questions you can post them in our
Webmaster Help Forum.