List of Google's special-case crawlers

The special-case crawlers are used by specific Google products where there's an agreement between the crawled site and the product about the crawl process. For example, AdsBot ignores the global robots.txt user agent (*) with the ad publisher's permission. The general technical properties of Google's crawlers also apply to the special-case crawlers.

The special-case crawlers may ignore robots.txt rules and so they operate from a different IP range than the common crawlers. The IP ranges are published in the special-crawlers.json object. The special-case crawlers' reverse DNS mask matches rate-limited-proxy-***-***-***-***.google.com.

The following list shows the special-case crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.

User-Agent in HTTP requests 
APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)
robots.txt
User-agent token in robots.txt

APIs-Google

The global user agent (*) is ignored.
Example robots.txt group 
user-agent: APIs-Google
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the APIs-Google user agent affect the delivery of push notification messages by Google APIs.
User-Agent in HTTP requests 
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
robots.txt
User-agent token in robots.txt

AdsBot-Google-Mobile

The global user agent (*) is ignored.
Example robots.txt group 
user-agent: AdsBot-Google-Mobile
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the AdsBot-Google-Mobile user agent affect Google Ads' ability to check web page ad quality.
User-Agent in HTTP requests 
AdsBot-Google (+http://www.google.com/adsbot.html)
robots.txt
User-agent token in robots.txt

AdsBot-Google

The global user agent (*) is ignored.
Example robots.txt group 
user-agent: AdsBot-Google
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the AdsBot-Google user agent affect Google Ads' ability to check web page ad quality.
User-Agent in HTTP requests
Desktop agent 
Mediapartners-Google
Mobile agent 
(Various mobile device types) (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
robots.txt
User-agent token in robots.txt Mediapartners-Google

The global user agent (*) is ignored.
Example robots.txt group 
user-agent: Mediapartners-Google
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the Mediapartners-Google user agent affect Google AdSense. The AdSense crawler visits participating sites in order to provide them with relevant ads.
User-Agent in HTTP requests 
Google-Safety
robots.txt The Google-Safety user agent ignores robots.txt rules.
Affected products The Google-Safety user agent handles abuse-specific crawling, such as malware discovery for publicly posted links on Google properties. As such it's unaffected by crawling preferences.

Retired special-case crawlers

The following special-case crawlers are no longer in use, and are only noted here for historical reference.

User-Agent in HTTP requests 
Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
robots.txt
User-agent token in robots.txt AdsBot-Google-Mobile

The global user agent (*) is ignored.
Affected products Crawling preferences addressed to the AdsBot-Google-Mobile user agent affected Google Ads' ability to check iPhone web page ad quality.
User-Agent in HTTP requests 
Mozilla/5.0 (Linux; Android 11; Pixel 2; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Mobile Safari/537.36
robots.txt
User-agent token in robots.txt DuplexWeb-Google

Duplex on the web may ignore the * wildcard.
Affected products Supported the Duplex on the web service.
User-Agent in HTTP requests 
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon
robots.txt
User-agent token in robots.txt Googlebot-Image
Googlebot
User-Agent in HTTP requests 
AdsBot-Google-Mobile-Apps
robots.txt
User-agent token in robots.txt AdsBot-Google-Mobile-Apps

The AdsBot-Google-Mobile-Apps user agent obeyed AdsBot-Google robots rules, but ignored the global user agent (*).
Affected products Crawling preferences addressed to the AdsBot-Google-Mobile-Apps user agent affect Google Ads' ability to check Android app page ad quality.
User-Agent in HTTP requests 
Mozilla/5.0 (Linux; Android 4.2.1; en-us; Nexus 5 Build/JOP40D) AppleWebKit/535.19 (KHTML, like Gecko; googleweblight) Chrome/38.0.1025.166 Mobile Safari/535.19
robots.txt
User-agent token in robots.txt googleweblight

The googleweblight user agent was used only for explicit browse requests of a human visitor, and so it ignored robots.txt rules, which are used to block automated crawling requests.
Affected products The Web Light user agent checked for the presence of the no-transform header whenever a user clicked your page in Search under appropriate conditions.