# What is Crawl Budget?

What is Crawl Budget? The number of pages Google allocates to crawling your site in a given period. For small sites, not an issue.

## How it works in practice

Crawl budget matters for stores with a large catalog. If Google's budget is 500 pages/day and your site has 5,000 pages, 4,500 pages won't get crawled today. If your best product pages aren't crawled, they won't get indexed — and won't rank.

## Related metrics

- Robots.txt — guides the crawl budget
- Sitemap — helps Google crawl correctly
- Technical SEO — crawl budget is a technical component
- Index Coverage — a result of crawl budget

## Common misconception

'Crawl budget matters to everyone.' For a store with 100 products, crawl budget isn't a problem. For a store with 50,000 pages (including filters, variants, tags), it's critical. If Googlebot wastes crawl budget on low-value faceted-navigation URLs, it never reaches your product pages.

## How to fix it

Crawl budget being wasted? (1) GSC → Settings → Crawl Stats — check which URLs get crawled, (2) Robots.txt: block filters, tags, and search-result pages with no SEO value, (3) Canonical URLs for duplicate pages, (4) XML sitemap — include only valuable pages, not every URL.

## Real-world example

Apparel store: 2,000 products + 40,000 URLs from faceted navigation (color=red&size=M, etc.). Google mostly crawled the faceted URLs. New products went unindexed for weeks. Robots.txt block on faceted URLs → crawl on products tripled. New-product indexing: 3 days (down from 3 weeks).

## Where to start

A crawl-budget audit is relevant from 5,000+ pages. GSC Crawl Stats → what is Google crawling? If most of the crawl goes to low-value URLs, robots.txt optimization is a quick win.

---

Category: SEO · From the [Jimmy OS KPI Glossary](https://jimmy-os.com/glossary/kpi.md) · Source page: https://jimmy-os.com/glossary/kpi/crawl-budget