# FAQ: Google Crawl Budget

Source: https://docs.arcxp.com/en/products/general/faq--google-crawl-budget.html

## Does Google have a crawl budget?

Yes, Google [explicitly says](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1) that they have a crawl budget: 

The web is a nearly infinite space, exceeding Google’s ability to explore and index every available URL. As a result, there are limits to how much time Googlebot can spend crawling any single site. The amount of time and resources that Google devotes to crawling a site is commonly called the site’s _crawl budget_.

## What affects Google’s crawl budget?

Google [says](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#general%5Ftheory): 

Crawl budget is determined by two main elements: _crawl capacity limit_ and _crawl demand_.

…

Taking crawl capacity and crawl demand together, Google defines a site’s crawl budget as the set of URLs that Googlebot can and wants to crawl. Even if the crawl capacity limit isn’t reached, if crawl demand is low, Googlebot will crawl your site less.

See also: 

- [Crawl capacity limit](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#crawl-capacity-limit): Google limits how often it crawls a site to ensure it doesn’t overwhelm the site’s servers, and because Google doesn’t have infinite computing resources.
- [Crawl demand](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#crawl-demand): Google determines how much computing resources it wants to spend crawling a site based on signals like how popular the site is and how often it is updated.

## What should I do to make the most of my site’s crawl budget?

See Google’s list of [best practices here](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#best%5Fpractices), which contains three main strategies: 

1. **Manage your URL inventory**: You can [hide URLs](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#hide%5Furls) that you don’t want to appear in search results by [updating your robots.txt file](https://dev.arcxp.com/arc-io/developer-docs/managing-txt-files-with-outbound-feeds), so that Googlebot doesn’t spend time crawling these pages.
2. **Make your pages efficient to load**: Making your site load quickly means that Google [may be able](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#improve%5Fcrawl%5Fefficiency) to crawl more of your site within its limited crawl budget. Of course, a faster site is also better for users, and is part of the [page experience signal](https://developers.google.com/search/docs/appearance/page-experience) that Google incorporates into ranking. See below for further tips for optimizing your site performance:

  - [How to optimize your PageBuilder Engine bundle size for better page load and render performance](https://dev.arcxp.com/pagebuilder-engine/how-to-guides/optimization-and-deployment/how-to-optimize-your-pagebuilder-engine-bundle-size-for-better-page-load-and-render-performance/)
  - [Measuring and monitoring your page performance](https://docs.arcxp.com/en/products/pagebuilder/pagebuilder-engine/measuring-and-monitoring-your-page-performance.html)
  - [Self-service Performance Debugging](https://dev.arcxp.com/pagebuilder-engine/how-to-guides/optimization-and-deployment/self-service-performance-debugging/)
  - [Profiling Code Performance and Arc XP](https://dev.arcxp.com/pagebuilder-engine/how-to-guides/optimization-and-deployment/profiling-code-performance-and-arc-xp/)
  - [Reviewing PageBuilder performance](https://docs.arcxp.com/en/products/delivery/reviewing-pagebuilder-performance.html)
3. **Monitor your site crawling**: You can view the [Crawl Stats report](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#monitor%5Fcrawling%5Findexing) in [Google Search Console](https://support.google.com/webmasters/answer/9128668?hl=en&ref%5Ftopic=9128571&sjid=15047215805625868498-NA) to learn if Google is having any issues crawling your site.

## How do I know if Google is having issues crawling my site?

Google makes information available in the [Crawl Stats report](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget?visit%5Fid=638242529410408664-3667990112&rd=1#availability%5Fissues):

Improving your site availability won’t necessarily increase your crawl budget; Google determines the best crawl rate based on the crawl demand, as described previously. However, availability issues do prevent Google from crawling your site as much as it might want to.

## Diagnosing

Use the [Crawl Stats report](https://support.google.com/webmasters/answer/9679690) to see Googlebot’s crawling history for your site. The report shows when Google encountered availability issues on your site.

If you review Search Console information and you have a need to do deep technical analysis of how Googlebot is interacting with your site, you can create a support ticket to request Edge log forwarding. See [CDN log delivery with Arc XP](https://docs.arcxp.com/en/products/delivery/arc-xp-cdn/cdn-log-delivery-with-arc-xp.html).

### What else might affect my site’s traffic from Google services?

Sites may experience referral traffic changes even if there are no reported issues from crawling. Sometimes Google makes changes to its products or experiences outages which can affect publishers’ site traffic, even if there was no change to the publisher website itself. 

We recommend checking the [Google Search Status Dashboard](https://status.search.google.com/summary) to see if any recent changes or incidents have been identified that could affect referrals from Google products. These incidents typically affect a [large number](https://developers.google.com/search/help/status-dashboard#detection) of sites and users. 

![gen_google_search_status_dashboard.png](https://docs.arcxp.com/images/learning-center/image/img-e5a67417daf2ecaf8558ebf40dae1395.png)
