# What is Robots.txt?

What is Robots.txt? A text file at yourdomain.com/robots.txt that tells bots which pages to crawl and which to skip. Allow/Disallow by path.

## How it works in practice

Robots.txt is the site's 'gatekeeper.' You tell Google: 'yes to crawling products and the blog, no to crawling admin and faceted navigation.' On large stores, robots.txt saves valuable crawl budget and directs it to content worth ranking.

## Related metrics

- Crawl Budget — robots.txt protects crawl budget
- Sitemap — robots.txt points to the sitemap's location
- Technical SEO — robots.txt is a basic technical tool
- Index Coverage — robots.txt affects what gets indexed

## Common misconception

'Disallow means the page won't get indexed.' Wrong. Disallow prevents crawling, not indexing. A page disallowed in robots.txt can still get indexed if other sites link to it. To block indexing, use a noindex meta tag, not robots.txt.

## How to fix it

Robots.txt errors? (1) GSC → Robots Testing Tool — check the file is valid, (2) Check you haven't accidentally blocked important pages (some stores have blocked their entire site by mistake), (3) Verify: check /robots.txt in a browser — is it readable and correct? (4) Googlebot can ignore robots.txt — noindex is more reliable for blocking.

## Real-world example

Electronics store: robots.txt accidentally blocked /product/ (meant to block only /product/filter/). Google stopped crawling product pages. Organic traffic fell 60% in a month. GSC → Index Coverage revealed the issue. Fix + re-crawl request: recovered over 8 weeks.

## Where to start

Check robots.txt immediately after launch and whenever you change URL structure. Rule: don't block what isn't necessary. On large stores, block faceted navigation + admin + internal search results.

---

Category: SEO · From the [Jimmy OS KPI Glossary](https://jimmy-os.com/glossary/kpi.md) · Source page: https://jimmy-os.com/glossary/kpi/robotstxt