AI crawler robots.txt generator
AI SearchChoose which AI crawlers can access your site, keeping AI search and citation bots while blocking model training if you want to. Free, no signup, and it runs entirely in your browser.
| Crawler | Purpose | Allow |
|---|---|---|
| OAI-SearchBot OpenAI. Powers ChatGPT search results and citations. | AI search | |
| ChatGPT-User OpenAI. Fetches pages when a ChatGPT user asks it to. | AI search | |
| GPTBot OpenAI. Collects data to train OpenAI models. | Training | |
| Claude-SearchBot Anthropic. Indexes pages for search results in Claude. | AI search | |
| Claude-User Anthropic. Fetches pages when a Claude user asks it to. | AI search | |
| ClaudeBot Anthropic. Collects data to train Anthropic models. | Training | |
| PerplexityBot Perplexity. Indexes pages for Perplexity answers. | AI search | |
| Google-Extended Google. Controls use for Gemini training; does not affect Google Search or AI Overviews. | Training | |
| Applebot-Extended Apple. Controls use of content to train Apple models. | Training | |
| CCBot Common Crawl. Open web archive widely used for model training. | Training | |
| Bytespider ByteDance. Crawler associated with ByteDance AI products. | Training | |
| Meta-ExternalAgent Meta. Collects data for Meta AI models. | Training |
Blocking a search crawler can remove you from that assistant's answers. Robots.txt is a request that reputable crawlers honor, not an access control.
How to use it
- Start from a preset such as Allow AI search, block training.
- Adjust the Allow checkbox for any individual crawler in the table.
- Copy the rules and add them to your existing robots.txt file.
Two kinds of AI crawler
AI companies run different crawlers for different jobs, and the distinction matters for your visibility. Training crawlers, such as GPTBot, ClaudeBot, CCBot and Meta-ExternalAgent, collect content to build or improve AI models. Search and retrieval crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, fetch pages so an assistant can answer a question and cite sources. User-triggered fetchers, such as ChatGPT-User and Claude-User, load a page when a person asks the assistant to look at it.
Blocking a training crawler limits how your content is used to train future models. Blocking a search or retrieval crawler can remove your pages from that assistant's answers and citations. Many businesses are happy to be cited and recommended but less comfortable with training, which is why the default preset allows AI search and blocks training.
Google-Extended and Applebot-Extended
Google-Extended is not a separate crawler that visits your site. It is a control token that tells Google whether content it crawls may be used for Gemini model training and grounding. Blocking it does not affect Google Search rankings, indexing or AI Overviews, which are handled by Googlebot. If you block Googlebot, you leave Google Search entirely, so this tool never touches it.
Applebot-Extended works in a similar way for Apple: it controls whether content crawled by Applebot can be used to train Apple models, without removing you from Apple search features.
Choosing a setting
Allow all suits most small businesses that want maximum exposure and are not concerned about training use. Allow AI search, block training is a balanced default for publishers and brands that want citations without contributing to model training. Block all AI is a deliberate trade-off: it reduces your chance of appearing in AI answers, which is a growing source of discovery. If AI visibility is part of your plan, see our AI search optimization service.
Limits to keep in mind
Robots.txt is a request, not access control. Reputable AI companies state that their crawlers honor it, but the file cannot stop a bot that ignores it; that needs firewall or bot management rules. Blocking also does not remove content already collected. Crawler names change over time, so review the list occasionally against each company's documentation.
Paste the output into your existing robots.txt rather than replacing it, then check a few URLs against each bot with the robots.txt tester. Remember that a crawler with its own group ignores the general User-agent: * rules, so add any paths you want blocked for everyone to those groups too.
Questions, answered
Will blocking GPTBot remove my site from ChatGPT answers?
Blocking GPTBot opts you out of OpenAI model training. ChatGPT search results and citations rely on OAI-SearchBot, and user-requested page visits use ChatGPT-User. If you want to stay visible in ChatGPT answers, keep OAI-SearchBot allowed. Blocking all three reduces your chance of being cited by ChatGPT.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content may be used for Gemini training and grounding. Google Search indexing, rankings and AI Overviews are handled by Googlebot, which is unaffected. Never block Googlebot unless you want to leave Google Search, because that removes your pages from regular results too.
Do AI crawlers actually obey robots.txt?
The major AI companies, including OpenAI, Anthropic, Google, Apple and Perplexity, publish their crawler names and state that they honor robots.txt. Still, robots.txt is a voluntary standard. Bots that ignore it or disguise their user-agent need to be handled with server, firewall or CDN-level bot rules instead.
Where do these rules go?
Add them to your existing robots.txt file at the root of your domain, keeping your current rules and Sitemap line. Do not replace the whole file. Because each named crawler follows only its own group, repeat any general Disallow paths inside the AI crawler groups if you want those paths blocked for them as well.
Find out what is holding your rankings back
Send us your site. Within 2 business days you get a written audit, an AI visibility check and a short fix list ranked by impact. Free, with no contract attached.