Monday, March 12, 2012
Crawl errors is one of the most popular features in Webmaster Tools, and today we're rolling out some very significant enhancements that will make it even more useful.
We now detect and report many new types of errors. To help make sense of the new data, we've split the errors into two parts: site errors and URL errors.
Site Errors
Site errors are errors that aren't specific to a particular URL—they affect your entire site. These include DNS resolution failures, connectivity issues with your web server, and problems fetching your robots.txt file. We used to report these errors by URL, but that didn't make a lot of sense because they aren't specific to individual URLs—in fact, they prevent Googlebot from even requesting a URL! Instead, we now keep track of the failure rates for each type of site-wide error. We'll also try to send you alerts when these errors become frequent enough that they warrant attention.
Furthermore, if you don't have (and haven't recently had) any problems in these areas, as is the case for many sites, we won't bother you with this section. Instead, we'll just show you some friendly check marks to let you know everything is hunky-dory.
URL errors
URL errors are errors that are specific to a particular page. This means that when Googlebot tried to crawl the URL, it was able to resolve your DNS, connect to your server, fetch and read your robots.txt file, and then request this URL, but something went wrong after that. We break the URL errors down into various categories based on what caused the error. If your site serves up Google News or mobile (CHTML/XHTML) data, we'll show separate categories for those errors.
Less is more
We used to