23% of UK Business Websites Block AI Crawlers

7 out of 30 UK SME websites (23%) block at least one AI crawler in their robots.txt. GPTBot is blocked most often at 23.3%. ClaudeBot follows at 16.7%. PerplexityBot at 3.3%. Most of these blocks look unintentional. The same sites had active SEO signals showing they want to be found online.

Blocking rates by crawler

Why blocks are usually unintentional

Nearly all sites blocking AI crawlers also had active SEO signals. They had optimised titles, meta descriptions, Google Analytics, and sitemaps. These businesses want to be found online. They block AI crawlers without knowing it.

The most common causes:

• WordPress security plugins - Wordfence, Sucuri, and iThemes Security sometimes add blanket bot blocks. These include AI crawlers. The site owner installs the plugin for security and never checks which bots get blocked
• Cloudflare bot protection - Bot Fight Mode and Super Bot Fight Mode can block AI crawlers. This setting is often on by default
• SEO plugins with AI opt-outs - Yoast and Rank Math added AI crawler blocking options. Some are on by default
• Copied robots.txt files - Sites built from templates sometimes inherit rules that block crawlers the original developer added
• Hosting provider defaults - Some managed WordPress hosts add AI crawler blocks at the server level

The GPTBot distinction matters

Some businesses block GPTBot (training data) on purpose while allowing OAI-SearchBot (search indexing). They do not want to train AI models but do want to appear in ChatGPT search results. This is a valid choice.

But our data shows several sites that block GPTBot without knowing OAI-SearchBot exists. They blocked the well-known crawler and left the lesser-known one unaddressed. This gap suggests the blocks were not planned.

Training vs search crawlers: the new distinction

Note: our research came before AI platforms split their crawlers into training and search bots. The smart approach now is to block training bots (GPTBot, ClaudeBot, Google-Extended) while allowing search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking all AI crawlers still hurts you. But blocking training bots while allowing search bots protects your content and keeps you visible.

Business impact

Block OAI-SearchBot and ChatGPT search cannot find you. Block Claude-SearchBot and Claude cannot browse your pages. Block PerplexityBot and you vanish from Perplexity results. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) is a valid choice. It does not affect AI search visibility. It only controls whether your content trains future models.

ChatGPT holds 78% of AI search market share. AI referral traffic grows 1,300% year on year. Blocking search crawlers means missing a fast-growing channel.

How to check and fix

1. Check your robots.txt - visit yoursite.com/robots.txt and search for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended
2. Check Cloudflare - if you use Cloudflare, go to Security > Bots and review your bot management settings
3. Check WordPress plugins - review security plugin settings for any bot blocking rules
4. Explicitly allow - add User-agent: GPTBot / Allow: / lines for each AI crawler you want to allow

See our complete robots.txt configuration guide for the exact code to copy into your file.

See all AI search statistics · Get a free crawler access audit