Crawl Budget
The number of pages Google allocates to crawling your site in a given period. For small sites, not an issue. For stores with 10,000+ pages, it's critical that Google 'spends' its crawl on the pages that matter.
In practice
Crawl budget matters for stores with a large catalog. If Google's budget is 500 pages/day and your site has 5,000 pages, 4,500 pages won't get crawled today. If your best product pages aren't crawled, they won't get indexed — and won't rank.
Common misconception
'Crawl budget matters to everyone.' For a store with 100 products, crawl budget isn't a problem. For a store with 50,000 pages (including filters, variants, tags), it's critical. If Googlebot wastes crawl budget on low-value faceted-navigation URLs, it never reaches your product pages.
How to fix it
Crawl budget being wasted? (1) GSC → Settings → Crawl Stats — check which URLs get crawled, (2) Robots.txt: block filters, tags, and search-result pages with no SEO value, (3) Canonical URLs for duplicate pages, (4) XML sitemap — include only valuable pages, not every URL.
Example
Apparel store: 2,000 products + 40,000 URLs from faceted navigation (color=red&size=M, etc.). Google mostly crawled the faceted URLs. New products went unindexed for weeks. Robots.txt block on faceted URLs → crawl on products tripled. New-product indexing: 3 days (down from 3 weeks).
Priority
A crawl-budget audit is relevant from 5,000+ pages. GSC Crawl Stats → what is Google crawling? If most of the crawl goes to low-value URLs, robots.txt optimization is a quick win.
Related terms
Keep learning. Metrics that touch the same area:
- Robots.txtSEOA text file at yourdomain.com/robots.txt that tells bots which pages to crawl and which to skip. Allow/Disallow by path. One of the most basic tools in technical SEO.
- SEO – Search Engine OptimizationSEOThe process of optimizing for high rankings in organic search results. Includes: technical SEO (speed, structure), on-page (content, keywords), off-page (inbound links).
- Google Search Console (GSC)SEOA free Google tool: keywords, clicks, impressions, CTR, ranking, crawl errors, and indexing issues.
- Keyword ResearchSEOThe process of identifying what your target audience searches for: search volume, competition, and search intent (informational/transactional/navigational). The foundation of all SEO and paid advertising.
- On-Page SEOSEOImprovements made directly on a page: H1-H3 headings, meta descriptions, URLs, content, alt text, internal links, schema markup. Fully within your control.
- BacklinksSEOLinks from external sites — a 'vote of confidence' in Google's eyes. Quality links (from authoritative sites) impact ranking. Quantity matters less than quality.
Jimmy tracks your Crawl Budget, automatically
WhatsApp alerts when something important changes. No dashboards, no hassle.