List of Google's common crawlers

Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically. The general technical properties of Google's crawlers also apply to the common crawlers.

The common crawlers generally crawl from the IP ranges published in the googlebot.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.

The following list shows the common crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.

User-Agent in HTTP requests
Googlebot Smartphone 
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop 
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Rarely:

  • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • Googlebot/2.1 (+http://www.google.com/bot.html)
robots.txt
User-agent token in robots.txt Googlebot
Example robots.txt group 
user-agent: Googlebot
allow: /archive/1Q84
disallow: /archive
Affected products Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover.
User-Agent in HTTP requests 
Googlebot-Image/1.0
robots.txt
User-agent token in robots.txt Googlebot-Image
Googlebot
Example robots.txt group 
user-agent: Googlebot-Image
allow: /archive/1Q84
disallow: /archive/moons.jpg
Affected products Crawling preferences addressed to the Googlebot-Image user agent affect Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented.
User-Agent in HTTP requests 
Googlebot-Video/1.0
robots.txt
User-agent token in robots.txt Googlebot-Video
Googlebot
Example robots.txt group 
user-agent: Googlebot-Video
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the Googlebot-Video user agent affect video-related Google Search features and other products dependent on videos.
User-Agent in HTTP requests Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings.
robots.txt
User-agent token in robots.txt Googlebot-News
Googlebot
Example robots.txt group 
user-agent: Googlebot-News
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the Googlebot-News user agent affect all surfaces of Google News (for example, the News tab in Google Search and the Google News app).
User-Agent in HTTP requests
Desktop agent 
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
Mobile agent 
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
robots.txt
User-agent token in robots.txt Storebot-Google
Example robots.txt group 
user-agent: Storebot-Google
allow: /archive/1Q84
disallow: /archive/konbini
Affected products Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping).
User-Agent in HTTP requests
Desktop agent 
Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)
Mobile agent 
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)
robots.txt
User-agent token in robots.txt Google-InspectionTool
Googlebot
Example robots.txt group 
user-agent: Google-InspectionTool
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products.
User-Agent in HTTP requests 
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; GoogleOther)
 
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36
robots.txt
User-agent token in robots.txt GoogleOther
Example robots.txt group 
user-agent: GoogleOther
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. It has no effect on Google Search or other products.
User-Agent in HTTP requests 
GoogleOther-Image/1.0
robots.txt
User-agent token in robots.txt GoogleOther-Image
GoogleOther
Example robots.txt group 
user-agent: GoogleOther-Image
allow: /archive/1Q84
disallow: /archive/moon.jpg
Affected products Crawling preferences addressed to the GoogleOther-Image user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs.
User-Agent in HTTP requests 
GoogleOther-Video/1.0
robots.txt
User-agent token in robots.txt GoogleOther-Video
GoogleOther
Example robots.txt group 
user-agent: GoogleOther-Video
allow: /archive/1Q84
disallow: /archive
Affected products Crawling preferences addressed to the GoogleOther-Video user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs.
User-Agent substring in HTTP requests 
Google-CloudVertexBot
robots.txt
User-agent token in robots.txt Google-CloudVertexBot
Googlebot
Example robots.txt group 
user-agent: Google-CloudVertexBot
allow: /archive/1Q84
disallow: /archive/
Affected products Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products.
User-Agent in HTTP requests Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
robots.txt
User-agent token in robots.txt Google-Extended
Example robots.txt group 
user-agent: Google-Extended
allow: /archive/1Q84
disallow: /archive/
Affected products Google-Extended is a standalone product token that web publishers can use to manage whether their sites help improve Gemini Apps and Vertex AI generative APIs, including future generations of models that power those products. Grounding with Google Search on Vertex AI does not use web pages for grounding that have disallowed Google-Extended. Google-Extended does not impact a site's inclusion or ranking in Google Search.

A note about Chrome/W.X.Y.Z in user agents

The string Chrome/W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 41.0.2272.96. This version number increases over time to match the latest Chromium release version used by Googlebot.

If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.