List of Google's special-case crawlers
The special-case crawlers are used by specific Google products where there's an agreement between
the crawled site and the product about the crawl process. For example, AdsBot
ignores
the global robots.txt user agent (*
) with the ad publisher's permission. The general
technical properties
of Google's crawlers also apply to the special-case crawlers.
The
special-case crawlers may ignore robots.txt rules and so they operate from a different IP range
than the common crawlers. The IP ranges are published in the
special-crawlers.json object. The
special-case crawlers' reverse DNS mask matches
rate-limited-proxy-***-***-***-***.google.com
.
The following list shows the special-case crawlers, their user agent strings as they appear in
the HTTP requests, their user agent tokens for the User-agent:
line in robots.txt,
and the products that are affected by crawl preferences for the crawler. The list is not
exhaustive, it only covers the requestors that are more likely to show up in log files and that
we've received questions about.
APIs-Google
-
User-Agent in HTTP requests APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
robots.txt User-agent token in robots.txt APIs-Google
The global user agent (*
) is ignored.Example robots.txt group user-agent: APIs-Google allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the APIs-Google
user agent affect the delivery of push notification messages by Google APIs. AdsBot Mobile Web
-
User-Agent in HTTP requests Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
robots.txt User-agent token in robots.txt AdsBot-Google-Mobile
The global user agent (*
) is ignored.Example robots.txt group user-agent: AdsBot-Google-Mobile allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the AdsBot-Google-Mobile
user agent affect Google Ads' ability to check web page ad quality. AdsBot
-
User-Agent in HTTP requests AdsBot-Google (+http://www.google.com/adsbot.html)
robots.txt User-agent token in robots.txt AdsBot-Google
The global user agent (*
) is ignored.Example robots.txt group user-agent: AdsBot-Google allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the AdsBot-Google
user agent affect Google Ads' ability to check web page ad quality. AdSense
-
User-Agent in HTTP requests Desktop agent Mediapartners-Google
Mobile agent (Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
robots.txt User-agent token in robots.txt Mediapartners-Google
The global user agent (*
) is ignored.Example robots.txt group user-agent: Mediapartners-Google allow: /archive/1Q84 disallow: /archive/
Affected products Crawling preferences addressed to the Mediapartners-Google
user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads. Google-Safety
-
User-Agent in HTTP requests Google-Safety
robots.txt The Google-Safety user agent ignores robots.txt rules. Affected products The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences.
Retired special-case crawlers
The following special-case crawlers are no longer in use, and are only noted here for historical reference.
Retired Google crawlers | |||||
---|---|---|---|---|---|
Duplex on the web |
Supported the Duplex on the web service.
|
||||
Web Light |
Checked for the presence of the
|
||||
AdsBot Mobile Web |
Checks iPhone
web page ad quality.
Ignores the global user agent (
|
||||
Mobile Apps Android |
Checks Android app page
ad quality.
Obeys
|
||||
Google Favicon |
|