Robots.txt
A text file at yourdomain.com/robots.txt that tells bots which pages to crawl and which to skip. Allow/Disallow by path. One of the most basic tools in technical SEO.
In practice
Robots.txt is the site's 'gatekeeper.' You tell Google: 'yes to crawling products and the blog, no to crawling admin and faceted navigation.' On large stores, robots.txt saves valuable crawl budget and directs it to content worth ranking.
Common misconception
'Disallow means the page won't get indexed.' Wrong. Disallow prevents crawling, not indexing. A page disallowed in robots.txt can still get indexed if other sites link to it. To block indexing, use a noindex meta tag, not robots.txt.
How to fix it
Robots.txt errors? (1) GSC → Robots Testing Tool — check the file is valid, (2) Check you haven't accidentally blocked important pages (some stores have blocked their entire site by mistake), (3) Verify: check /robots.txt in a browser — is it readable and correct? (4) Googlebot can ignore robots.txt — noindex is more reliable for blocking.
Example
Electronics store: robots.txt accidentally blocked /product/ (meant to block only /product/filter/). Google stopped crawling product pages. Organic traffic fell 60% in a month. GSC → Index Coverage revealed the issue. Fix + re-crawl request: recovered over 8 weeks.
Priority
Check robots.txt immediately after launch and whenever you change URL structure. Rule: don't block what isn't necessary. On large stores, block faceted navigation + admin + internal search results.
Related terms
Keep learning. Metrics that touch the same area:
- Crawl BudgetSEOThe number of pages Google allocates to crawling your site in a given period. For small sites, not an issue. For stores with 10,000+ pages, it's critical that Google 'spends' its crawl on the pages that matter.
- SEO – Search Engine OptimizationSEOThe process of optimizing for high rankings in organic search results. Includes: technical SEO (speed, structure), on-page (content, keywords), off-page (inbound links).
- Google Search Console (GSC)SEOA free Google tool: keywords, clicks, impressions, CTR, ranking, crawl errors, and indexing issues.
- Keyword ResearchSEOThe process of identifying what your target audience searches for: search volume, competition, and search intent (informational/transactional/navigational). The foundation of all SEO and paid advertising.
- On-Page SEOSEOImprovements made directly on a page: H1-H3 headings, meta descriptions, URLs, content, alt text, internal links, schema markup. Fully within your control.
- BacklinksSEOLinks from external sites — a 'vote of confidence' in Google's eyes. Quality links (from authoritative sites) impact ranking. Quantity matters less than quality.
Jimmy tracks your Robots.txt, automatically
WhatsApp alerts when something important changes. No dashboards, no hassle.