# Google Crawling and Indexing | Google Search Central

# Overview of crawling and indexing topics

The topics in this section describe how you can control Google's ability to find and parse
your content in order to show it in Search and other Google properties, as well as how to
prevent Google from crawling specific content on your site.

Here's a brief description of each page. To get an overview of crawling and indexing, read
our [How Search works](https://developers.google.com/search/docs/fundamentals/how-search-works) guide.

| Topics ||
|---|---|
| [File types indexable by Google](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types) | Google can index the content of most types of pages and files. Explore a list of the most common file types that Google Search can index. |
| [URL structure](https://developers.google.com/search/docs/crawling-indexing/url-structure) | Consider organizing your content so that URLs are constructed logically and in a manner that is most intelligible to humans. |
| [Sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) | Tell Google about pages on your site that are new or updated. |
| Crawler management | - [Ask Google to recrawl your URLs](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) - [Managing crawling of faceted navigation URLs](https://developers.google.com/search/docs/crawling-indexing/crawling-managing-faceted-navigation) - [Large site owner's guide to managing your crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) - [How HTTP status codes, and network and DNS errors affect Google Search](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) - [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers) |
| [robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro) | A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. |
| [Canonicalization](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) | Learn what URL canonicalization is and how to tell Google about any duplicate pages on your site in order to avoid excessive crawling. Learn how Google auto-detects duplicate content, how it treats duplicate content, and how it assigns a *canonical URL* to any duplicate page groups found. |
| [Mobile sites](https://developers.google.com/search/docs/crawling-indexing/mobile) | Learn how you can optimize your site for mobile devices and ensure that it's crawled and indexed properly. |
| [AMP](https://developers.google.com/search/docs/crawling-indexing/amp) | If you have AMP pages, learn how AMP works in Google Search. |
| [JavaScript](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) | There are some differences and limitations that you need to account for when designing your pages and applications to accommodate how crawlers access and render your content. |
| [Page and content metadata](https://developers.google.com/search/docs/crawling-indexing/special-tags) | - [Use valid HTML to specify page metadata](https://developers.google.com/search/docs/crawling-indexing/valid-page-metadata) - [All `meta` tags that Google understands](https://developers.google.com/search/docs/crawling-indexing/special-tags) - [Robots `meta` tag, `data-nosnippet`, and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) - [Block indexing with the `noindex` `meta` tag](https://developers.google.com/search/docs/crawling-indexing/block-indexing) - [Make your links crawlable](https://developers.google.com/search/docs/crawling-indexing/links-crawlable) - [Qualify your outbound links to Google with `rel` attributes](https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links) |
| Removals | - [Control what you share with Google](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) - [Remove a page hosted on your site from Google](https://developers.google.com/search/docs/crawling-indexing/remove-information) - [Remove images hosted on your page from search results](https://developers.google.com/search/docs/crawling-indexing/prevent-images-on-your-page) - [Keep redacted information out of Google Search](https://developers.google.com/search/docs/crawling-indexing/keep-redacted-information-out) |
| Site moves and changes | - [Redirects and Google Search](https://developers.google.com/search/docs/crawling-indexing/301-redirects) - [Site moves](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes) - [Minimize A/B testing impact in Google Search](https://developers.google.com/search/docs/crawling-indexing/website-testing) - [Temporarily pause or disable a website](https://developers.google.com/search/docs/crawling-indexing/pause-online-business) |