What Should My robots.txt Look Like for AI Search?
Your robots.txt needs to allow AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. Exact configuration here.
Block AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). Allow AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-session crawlers (ChatGPT-User, Claude-User). This protects your content from free training while letting AI search engines cite you.
The recommended robots.txt configuration
There are three types of AI crawler. Block training bots (they take your content to train models). Allow search bots (these power the AI search results that cite you). Allow user bots (triggered when someone asks an AI about you).
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
# TRAINING CRAWLERS (block)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# SEARCH CRAWLERS (allow - these power AI search results)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Applebot
Allow: /
User-agent: GoogleOther
Allow: /
# USER-SESSION CRAWLERS (allow - when users ask AI about you)
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Perplexity-User
Allow: /The three types of AI crawler
| Type | Crawler | Platform | What it does | Allow? |
|---|---|---|---|---|
| Training | GPTBot | OpenAI | Scrapes content to train AI models | Block |
| Training | ClaudeBot | Anthropic | Scrapes content to train Claude | Block |
| Training | Google-Extended | Trains Gemini and AI Overviews | Block | |
| Training | Applebot-Extended | Apple | Trains Apple Intelligence / Siri | Block |
| Training | CCBot | Common Crawl | Open dataset used by many AI models | Block |
| Search | OAI-SearchBot | OpenAI | Indexes for ChatGPT search results | Allow |
| Search | Claude-SearchBot | Anthropic | Indexes for Claude search results | Allow |
| Search | PerplexityBot | Perplexity | Indexes for Perplexity answers | Allow |
| Search | Googlebot | Indexes for Google Search + AI Overviews | Allow | |
| Search | Bingbot | Microsoft | Indexes for Bing + Copilot | Allow |
| User | ChatGPT-User | OpenAI | When a user asks ChatGPT to visit your site | Allow |
| User | Claude-User | Anthropic | When a user asks Claude to visit your site | Allow |
| User | Perplexity-User | Perplexity | When Perplexity fetches your page for a user | Allow |
How to check your current robots.txt
Go to yoursite.com/robots.txt in a browser. If you see Disallow: / for any of the crawlers above, that platform is blocked from accessing your content.
Common mistakes
- WordPress default blocks - some WordPress themes and security plugins add AI crawler blocks by default
- Cloudflare bot protection - Cloudflare's bot management can block AI crawlers without you knowing. Check Security > Bots
- Blocking GPTBot but allowing OAI-SearchBot - you need both for full ChatGPT visibility
- Not having a robots.txt at all - this is fine (all bots are allowed by default) but explicit rules are better practice
Related questions
Oliver Mackman
AI Search Analyst, SEOCompare
Oliver leads SEOCompare's editorial and comparison research. With over a decade in digital marketing, he oversees agency evaluation, tool testing, and AI search data analysis.
Last reviewed: 7 April 2026
Need help with AI search visibility?
Get a free AI visibility audit to see how your business appears across ChatGPT, Gemini, Perplexity, and AI Overviews.
Request your free audit