What is robots.txt?
robots.txt is a text file at the root of your website (yourdomain.com/robots.txt) that tells web crawlers which pages they can and cannot access. It's part of the Robots Exclusion Protocol, a standard followed by all major search engines.
The rise of AI crawlers in 2025
In 2025, a new type of crawler has emerged: AI training and search crawlers. These include:
| Crawler | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data + ChatGPT search |
| ClaudeBot | Anthropic | Training data + Claude search |
| PerplexityBot | Perplexity AI | AI search results |
| Googlebot-Extended | Google AI features | |
| CCBot | Common Crawl | Open dataset for AI training |
Should you allow or block AI crawlers?
Allow AI crawlers if:
- You want your site to appear in AI search results (ChatGPT, Perplexity, Claude)
- You want your tools or content recommended by AI assistants
- You're a tool website, blog, or informational site
Block AI crawlers if:
- You have proprietary content you don't want used for AI training
- You run a paywalled site and don't want free content scraped
- You have legal concerns about your content being used in training data
robots.txt syntax
Allow all crawlers (default)
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Block specific paths
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Allow specific AI crawlers
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
Block AI crawlers from training data
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
How to generate robots.txt for free
- Go to Robots.txt Generator
- Toggle which crawlers to allow or block
- Add paths you want to disallow
- Configure AI crawler settings
- Copy or download the generated robots.txt
- Place the file at your domain root:
yoursite.com/robots.txt
Crawl-blocking is not index-blocking
This is the single most misunderstood point about robots.txt, and it causes real damage.
Disallow tells a crawler not to fetch a URL. It does not tell a search engine not to index it. If another site links to a disallowed page, Google can still list that URL in results — showing the bare address with "No information is available for this page", because it was forbidden from fetching the content that would describe it.
Worse, a noindex meta tag on a disallowed page is never seen, because the crawler is not allowed to fetch the page containing it. The two directives cancel each other out.
| Goal | Correct mechanism |
|---|---|
| Keep a page out of search results | noindex meta tag or X-Robots-Tag header — and allow crawling |
| Save crawl budget on worthless URLs | Disallow in robots.txt |
| Protect private data | Authentication. Never robots.txt |
| Hide a plain-text file from search | X-Robots-Tag: noindex HTTP header |
That last row matters for files with no <head> — .txt, .json, .csv. There is nowhere to put a meta tag, so the header is the only option.
robots.txt is public, and it is a map
Anyone can read yoursite.com/robots.txt, including people looking for things you would rather they did not find. Listing Disallow: /admin-backup/ publishes the existence and location of that directory to every visitor.
Never treat robots.txt as a security boundary. If a path must stay private, put it behind authentication. If you want it out of the crawl but not advertised, block it by a broader pattern or handle it at the server.
Groups do not inherit
A User-agent block is matched, not merged. A crawler that matches a named group follows only that group and ignores the wildcard entirely:
User-agent: *
Disallow: /api/
User-agent: GPTBot
Allow: /
Here GPTBot may crawl /api/, because its own group never mentions it. The wildcard block applies only to crawlers with no named group of their own.
The fix is to repeat the directive in every named group that needs it:
User-agent: GPTBot
Allow: /
Disallow: /api/
This is an easy mistake to ship, because the file looks like /api/ is blocked for everyone.
Edge providers can override the file
If your site sits behind Cloudflare, Fastly, or a similar edge layer, that layer may have its own bot controls — Cloudflare's "Block AI Scrapers and Crawlers" toggle is the common example. Those rules act before your robots.txt is ever served, so a permissive file can coexist with bots being blocked at the network edge.
Verify with a spoofed user agent rather than trusting the file:
curl -A 'GPTBot' -I https://yoursite.com/
curl -A 'Googlebot' -I https://yoursite.com/
A 200 means the request got through. A 403 means something upstream is blocking it, whatever robots.txt says.
Frequently asked questions
Do all crawlers obey robots.txt?
It is voluntary. Major search engines and the main AI crawlers honour it. Scrapers and malicious bots ignore it entirely, which is another reason it is not a security tool.
Where must the file live?
At the domain root: https://example.com/robots.txt. It is not read from a subdirectory, and each subdomain needs its own.
Does blocking AI crawlers hurt my search rankings?
Blocking GPTBot or ClaudeBot does not affect Google Search rankings — those are separate crawlers. But Google-Extended controls Gemini and AI Overviews, so blocking it can remove you from AI-generated answers while leaving classic search results unaffected.
How long until changes take effect?
Crawlers cache robots.txt, typically for up to 24 hours. Google Search Console can force a refresh.
Should I add a Sitemap line?
Yes. Sitemap: https://example.com/sitemap.xml is independent of any user-agent group and helps every crawler discover your pages.
Verify your robots.txt
After uploading, test in Google Search Console:
Settings → robots.txt → Test
Then confirm real-world behaviour with the curl commands above — Search Console validates syntax, but only a live request proves what your edge layer actually does.