FAQ: Google Crawl Budget
Does Google have a crawl budget?
Yes, Google explicitly says that they have a crawl budget:
The web is a nearly infinite space, exceeding Google's ability to explore and index every available URL. As a result, there are limits to how much time Googlebot can spend crawling any single site. The amount of time and resources that Google devotes to crawling a site is commonly called the site's crawl budget.
What affects Google’s crawl budget?
Google says:
Crawl budget is determined by two main elements: crawl capacity limit and crawl demand.
…
Taking crawl capacity and crawl demand together, Google defines a site's crawl budget as the set of URLs that Googlebot can and wants to crawl. Even if the crawl capacity limit isn't reached, if crawl demand is low, Googlebot will crawl your site less.
See also:
Crawl capacity limit: Google limits how often it crawls a site to ensure it doesn’t overwhelm the site’s servers, and because Google doesn’t have infinite computing resources.
Crawl demand: Google determines how much computing resources it wants to spend crawling a site based on signals like how popular the site is and how often it is updated.
What should I do to make the most of my site’s crawl budget?
See Google’s list of best practices here, which contains three main strategies:
Manage your URL inventory: You can hide URLs that you don’t want to appear in search results by updating your robots.txt file, so that Googlebot doesn’t spend time crawling these pages.
Make your pages efficient to load: Making your site load quickly means that Google may be able to crawl more of your site within its limited crawl budget. Of course, a faster site is also better for users, and is part of the page experience signal that Google incorporates into ranking. See below for further tips for optimizing your site performance:
Monitor your site crawling: You can view the Crawl Stats report in Google Search Console to learn if Google is having any issues crawling your site.
How do I know if Google is having issues crawling my site?
Google makes information available in the Crawl Stats report:
Improving your site availability won't necessarily increase your crawl budget; Google determines the best crawl rate based on the crawl demand, as described previously. However, availability issues do prevent Google from crawling your site as much as it might want to.
Diagnosing
Use the Crawl Stats report to see Googlebot's crawling history for your site. The report shows when Google encountered availability issues on your site.
If you review Search Console information and you have a need to do deep technical analysis of how Googlebot is interacting with your site, you can create a support ticket to request Edge log forwarding. See CDN log delivery with Arc XP.
What else might affect my site’s traffic from Google services?
Sites may experience referral traffic changes even if there are no reported issues from crawling. Sometimes Google makes changes to its products or experiences outages which can affect publishers' site traffic, even if there was no change to the publisher website itself.
We recommend checking the Google Search Status Dashboard to see if any recent changes or incidents have been identified that could affect referrals from Google products. These incidents typically affect a large number of sites and users.
 |