How do you check a robots.txt file?
Open https://yoursite.com/robots.txt, copy the whole file into the checker above and enter the path you care about. The tool reads the file as RFC 9309 describes and shows, for 22 search and AI crawlers, whether that path is allowed and which rule on which line decides.
It works on pasted text because a web page cannot read another site's files. For Google's own view, Search Console's robots.txt report shows the files Google fetched for your top 20 hosts, any parsing errors, and lets you request a recrawl.
How are robots.txt rules matched?
A crawler finds the group addressed to it, then the longest matching rule in that group. The details, from RFC 9309 and Google's specification:
- Groups. One or more User-agent lines followed by rules. A blank line does not end a group; a User-agent line after a rule does.
- Choosing a group. Tokens match case-insensitively, and every group naming the crawler is merged. With none, it follows
*; with no*either, nothing is blocked. - A named group replaces
*. GPTBot with a group of its own ignores every*rule. - Choosing a rule. The longest matching path wins and Allow wins a tie.
*matches any characters and$marks the end, so/*.pdf$covers PDFs. Paths are case-sensitive and include the query string. - Encoding. Non-ASCII characters are compared percent-encoded, and escaped letters such as
%62are decoded first. - Edges. Rules before the first User-agent apply to no one, and
/robots.txtis always allowed. Crawlers must read at least 500 KiB; Google ignores anything beyond it. A 4xx response lets crawlers fetch anything; a 5xx makes them assume everything is blocked.
Which AI crawlers read robots.txt?
These are the tokens the checker tests, each verified against the operator's documentation on 17 October 2026. Google-Extended and Applebot-Extended are control tokens, not crawlers: they govern how data already fetched by Googlebot or Applebot may be used. We left out ByteDance's Bytespider because we found no documentation from ByteDance.
| Token | Operator | Purpose |
|---|---|---|
| Googlebot | Search, including AI Overviews and AI Mode | |
| Google-Extended | Gemini training and grounding (control token) | |
| Bingbot | Microsoft | Bing search |
| Applebot | Apple | Spotlight, Siri and Safari search; follows Googlebot rules if not named |
| Applebot-Extended | Apple | Apple model training (control token) |
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ChatGPT-User | OpenAI | User-initiated; robots.txt "may not apply" |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Claude search |
| Claude-User | Anthropic | User-initiated |
| PerplexityBot | Perplexity | Perplexity search; not training |
| Perplexity-User | Perplexity | User-initiated; "generally ignores robots.txt" |
| Meta-ExternalAgent | Meta | Model training and indexing |
| Meta-WebIndexer | Meta | Meta AI search |
| Meta-ExternalFetcher | Meta | User-initiated; "may bypass robots.txt" |
| Amazonbot | Amazon | Amazon products; may train Amazon AI models |
| Amzn-SearchBot | Amazon | Search such as Alexa; not training |
| Amzn-User | Amazon | User-initiated; "may not follow all" rules |
| CCBot | Common Crawl | Open web archive, reused for AI training |
| DuckAssistBot | DuckDuckGo | AI-assisted answers; not training |
| MistralAI-User | Mistral AI | User-initiated |
How do you block AI training but stay in search?
Choose Block AI training in the generator. It blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot and CCBot, and leaves search and AI search crawlers such as OAI-SearchBot and PerplexityBot on your * rules. Three things to know:
- Blocking Google-Extended does not remove you from AI Overviews. Google says AI features in Search are controlled through Googlebot and snippet controls.
- Blocking GPTBot does not hide you from ChatGPT search, which uses OAI-SearchBot. OpenAI says changes take about 24 hours.
- A crawler with its own group needs the
*paths repeated; the generator does it for you.
What do crawl-delay and noindex do in robots.txt?
Neither is part of RFC 9309. Google ignores Crawl-delay and stopped honouring noindex in robots.txt on 1 September 2019; use a noindex meta tag or X-Robots-Tag header instead. Bing honours Crawl-delay as a window of 1 to 30 seconds per fetch, Anthropic and Common Crawl document support for it, and Apple and Amazon say they do not follow it.
Common robots.txt mistakes
- A named group that forgets the
*rules. In the example file, Bingbot's own group means it may crawl/admin/and/search; the checker flags this. - A leftover
Disallow: /from a staging site, blocking every crawler without its own group. - Hiding private pages. The file is public and lists exactly those paths. Use a login.
- Disallowing a page to remove it from Google. A blocked URL can still be indexed from links, and Google must crawl a page to see its noindex.
- Blocking CSS and JavaScript that crawlers need to render pages.
- One file for every host. Each protocol and host needs its own robots.txt at the root.
How Olano helps
Our robots.txt files for olano.ai and olano.sg open "All crawlers welcome": a * group that allows everything, then 20 named groups that allow everything too, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended, plus a Sitemap line. Paste one above and every crawler reads Allowed. We want answer engines to read and cite our pages, and we publish an llms.txt as well.
Our agents sit on the other side of the request: they read the web for the businesses they work for, through 450+ connector tools, MCP servers and, when an administrator turns it on, a browser of their own. That browser treats every page as untrusted, and when a site refuses it, the agent stops and leaves the site alone for 15 minutes. The platform and security pages show how agents, approvals and access controls fit together.