Tuesday, June 28, 2022
Over the last few days we've received a great deal of questions about a recent update to our documentation about Googlebot. Namely, we've documented that Googlebot only ever "sees" the first 15 megabytes (MB) when fetching certain file types. This threshold is not new; it's been around for many years. We just added it to our documentation because it might be helpful for some folks when debugging, and because it rarely ever changes.
This limit only applies to the bytes (content) received for the initial request Googlebot makes, not the referenced resources within the page.
For example, when you open https://example.com/puppies.html, your browser will
initially download the bytes of the HTML file, and based on those bytes it might make further
requests for external JavaScript, images, or whatever else is referenced with a URL in the HTML.
Googlebot does the same thing.
What does this 15 MB limit mean to me?
Most likely nothing. There are
very few pages
on the internet that are bigger in size. You, dear reader, are unlikely to be the owner of one,
since the
median size of a HTML file is about 500 times smaller:
30 kilobytes (kB).
However, if you are the owner of an HTML page that's over 15 MB, perhaps you could at least move
some inline scripts and CSS dust to external files, pretty please.
What happens to the content after 15 MB?
The content after the first 15 MB is dropped by Googlebot, and only the first 15 MB gets forwarded
to indexing.
What content types does the 15 MB limit apply to?
The 15 MB limit applies to fetches made by Googlebot (Googlebot Smartphone and Googlebot Desktop)
when fetching
file types supported by Google Search.