Which AI Bots Do WordPress Sites Block? 1,000 robots.txt Files Checked

Should you block AI bots in robots.txt, and which ones? To see what site owners actually do, we checked the robots.txt files of 1,000 popular WordPress sites. 19.3% block at least one AI bot. Most of them block the bots that collect training data and leave the bots that put sites into AI answers alone. Here is what we found, what each bot really does according to its operator, and a robots.txt policy that fits most sites.

Key findings

• 19.3% of the 1,000 WordPress sites block at least one AI bot with a full-site Disallow: /.
• Training crawlers are blocked most: CCBot 16.6%, GPTBot 16.4%, Bytespider 16.1%, ClaudeBot 15.2%, Google-Extended and Applebot-Extended 14.3% each.
• AI search and user bots are blocked far less: PerplexityBot 7.4%, ChatGPT-User 7.3%, OAI-SearchBot 4.9%, Claude-User 4.8% and Claude-SearchBot 4.1%.
• Of the sites that block GPTBot, 70.1% still allow OAI-SearchBot, so they can appear in ChatGPT search.
• Only 2.2% block all 13 AI user agents we checked.
• 12.3% publish an llms.txt file.

Training, search and user bots: what each one does

AI companies run different bots for different jobs, and blocking each one has a different effect. Mixing them up is how sites end up invisible in AI answers by accident. The purposes below come from each operator's own documentation at the time of writing.

OpenAI

• GPTBot crawls content that may be used to train OpenAI's generative AI foundation models.
• OAI-SearchBot surfaces websites in ChatGPT's search features. Sites that block it are not shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User visits a page when a ChatGPT user or a custom GPT asks for it. OpenAI says it is not used to crawl the web automatically and, because these visits are user-initiated, "robots.txt rules may not apply."

OpenAI's bot documentation says each setting is independent, and that robots.txt changes can take about 24 hours to reach its search systems.

Anthropic

• ClaudeBot collects public web content that could contribute to model training. Blocking it keeps your future content out of training data.
• Claude-SearchBot indexes content to improve Claude's search results. Blocking it may reduce your visibility in Claude's answers.
• Claude-User fetches pages when a Claude user's question needs them.

Anthropic's crawler help page says its bots honor robots.txt. The older anthropic-ai token, which 14.2% of the sites we checked still block, is not among the bots Anthropic lists today.

Perplexity

• PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
• Perplexity-User visits pages to answer a user's question. Perplexity's bot documentation says that since a user requested the fetch, it "generally ignores robots.txt rules."

Google-Extended and Applebot-Extended

These are robots.txt tokens, not separate crawlers. They control how content already crawled by Googlebot or Applebot may be used for AI.

• Google-Extended controls whether your content can be used to train future Gemini models and for grounding in Gemini apps and Vertex AI. Google's crawler overview says it does not affect inclusion or ranking in Google Search. It also does not control AI Overviews, which are part of Search.
• Applebot-Extended opts your content out of training Apple's foundation models. Apple's Applebot page says such pages can still appear in search results in Spotlight, Siri and Safari.

Common Crawl, Meta and ByteDance

• CCBot is the crawler of Common Crawl, a non-profit that publishes an open web archive "accessible and analyzable by anyone." AI developers are among the users of that archive, which is why many sites that opt out of training block it. Common Crawl's CCBot page shows the Disallow: / rule to block it.
• meta-externalagent is used, according to Meta's crawler documentation, for training AI models and for improving products by indexing content directly.
• Bytespider is ByteDance's crawler. We could not find current official documentation of its purpose, so we don't describe it further. In our data, 145 of the 161 sites that block it also block GPTBot.

What 1,000 WordPress sites block in robots.txt

Here is the share of all 1,000 sites that block each AI user agent with a full-site rule:

1. CCBot: 16.6%
2. GPTBot: 16.4%
3. Bytespider: 16.1%
4. ClaudeBot: 15.2%
5. Applebot-Extended: 14.3%
6. Google-Extended: 14.3%
7. anthropic-ai: 14.2%
8. meta-externalagent: 14.2%
9. PerplexityBot: 7.4%
10. ChatGPT-User: 7.3%
11. OAI-SearchBot: 4.9%
12. Claude-User: 4.8%
13. Claude-SearchBot: 4.1%

The split is sharp. Every training-type bot sits between 14% and 17%. Every search or user bot sits below 8%. Site owners who block AI bots mostly want to stay out of model training, not out of AI answers.

Most sites that block AI bots keep AI search open

Of the 193 sites that block any AI bot, 107 (55.4%) block only training-type bots: GPTBot, ClaudeBot, anthropic-ai, CCBot, Bytespider, Google-Extended, Applebot-Extended and meta-externalagent. The single most common setup, on 64 sites, blocks exactly those eight and nothing else.

The vendor-level numbers tell the same story:

• 164 sites block GPTBot. 115 of them (70.1%) leave OAI-SearchBot allowed.
• 152 sites block ClaudeBot. 112 of them (73.7%) leave Claude-SearchBot allowed.
• All 49 sites that block OAI-SearchBot also block GPTBot. Nobody blocked ChatGPT search while allowing OpenAI training.

The other 86 blocking sites (8.6% of the 1,000) block at least one search or user bot. Only 33 sites (3.3%) block all five of the search and user bots we checked, and 22 (2.2%) block all 13 AI user agents.

PerplexityBot: blocked like a training bot, documented as a search bot

One result stands out. 7.4% of sites block PerplexityBot, more than any other search or user bot. Yet Perplexity documents PerplexityBot as the crawler that surfaces and links sites in its search results, and says it is not used to train foundation models.

So a site that blocks PerplexityBot to avoid training gets little out of it, and gives up citations in Perplexity answers. If your goal is to opt out of training, check that PerplexityBot is not on your list by accident, for example from a copied blocklist.

What blocking each kind of AI bot means for visibility

• Blocking training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) keeps your future content out of those training sets. Per the vendors' docs, it does not remove you from Google Search, ChatGPT search, Claude search or Apple search features.
• Blocking AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes or reduces your pages in those AI search answers. Fewer citations means fewer clicks from AI tools.
• Blocking user bots (ChatGPT-User, Claude-User, Perplexity-User) stops assistants from reading your page when someone asks about it or pastes your link, at least for the operators that honor robots.txt for these fetches.

Remember that robots.txt is a request, not a lock. The major AI companies say they follow it, but some user-triggered fetchers may not, and a bot that ignores robots.txt isn't stopped by it.

A recommended robots.txt policy for most sites

For most blogs, businesses and shops, being cited in AI answers is worth more than the training opt-out costs. A reasonable default:

1. Always allow AI search and user bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
2. Decide on training bots based on your content. If your writing is the product (courses, paid research, a publisher's archive), block them. If you mainly want to be found, allowing them costs little.
3. Never block Googlebot or Bingbot, and never put Disallow: / under User-agent: * on a live site.
4. Keep a Sitemap line in robots.txt so every crawler can find your pages.

If you choose to block training but keep AI search, the rules look like this:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

Sitemap: https://example.com/wp-sitemap.xml

A crawler follows the most specific group that names it, so these groups don't change anything for Googlebot. Google's robots.txt introduction explains how groups and rules work. For a deeper, OpenAI-focused walkthrough, see our guide on how to block GPTBot in WordPress.

How to block AI bots in robots.txt on WordPress

WordPress serves a virtual robots.txt, built on the fly. You can change it in three ways:

1. With an SEO plugin that edits the virtual robots.txt. This is the easiest and needs no file access.
2. With a physical file. Upload a robots.txt file to your site's root folder. It replaces the virtual one completely, so include the default WordPress rules and your Sitemap line.
3. At your CDN or firewall. Some can block or challenge bots before they reach WordPress. Check these settings too, since they can block bots you meant to allow.

After any change, open yoursite.com/robots.txt to confirm the result, and purge any cache for that URL.

In RankFlect SEO

Go to RankFlect SEO → Settings → Robots & AI crawlers. Each of 15 AI crawlers has an Allowed / Blocked switch, with a short label for what it does, such as "OpenAI training" or "ChatGPT search". All are allowed by default, and RankFlect SEO adds a Disallow: / group for each one you block. See Edit robots.txt for the full list and the file editor if your site has a physical robots.txt file.

The free site audit warns you when "AI search crawlers are blocked", so a copied blocklist can't quietly remove you from ChatGPT, Claude or Perplexity answers. See every site audit check.

To see which AI bots actually visit, open the free AI crawlers report under AI Visibility. It shows visits per bot with their purpose (training, AI search or user request), the pages they read and how many visits were fetched live to answer someone.

llms.txt: only 12.3% publish one

12.3% of the sites we checked serve an llms.txt file, a Markdown index of key pages for AI tools. It was more common on sites that don't block AI bots (14.1%) than on sites that block at least one (4.7%), which fits: sites that welcome AI tools also make their content easier for them to read.

llms.txt is not a Google ranking factor, and Google has said so. It is a convenience for AI agents and assistants. RankFlect SEO publishes it for free; see llms.txt and Markdown pages and our guide to llms.txt on WordPress.

How we ran this study

On November 5, 2026, we took the Tranco top-sites list (list ID JZPGY, generated November 4, 2026, from tranco-list.eu). We went down the list from the top, about 12,000 domains, and requested each homepage once over HTTPS. A site counted as WordPress when its homepage HTML had a WordPress generator tag or /wp-content/ or /wp-includes/ paths. We kept the first 1,000 WordPress sites that returned HTTP 200.

For each site we fetched the homepage, /robots.txt and /llms.txt, with a crawler that identified itself as RankFlectSEO-Research/1.0. 3.2% of sites had no robots.txt we could read. Percentages are of all 1,000 sites unless stated otherwise.

Limits to keep in mind:

• A site counts as blocking a bot only if its robots.txt has a full-site Disallow: / for that user agent. Partial blocks, CDN or firewall rules and server-level blocks are not counted, so real blocking is likely higher.
• These are popular sites from the top of the Tranco list, not a sample of all WordPress sites.
• We grouped bots as training, search or user bots based on each vendor's documentation. Bytespider and the legacy anthropic-ai token are in the training group because sites blocked them alongside the training crawlers.

If you want to set your AI crawler policy with switches instead of hand-written rules, download RankFlect SEO for free.

Frequently asked questions

How many websites block AI bots in robots.txt?

In our study of 1,000 popular WordPress sites, 19.3% blocked at least one AI bot with a full-site rule in robots.txt. The most blocked were CCBot (16.6%), GPTBot (16.4%) and Bytespider (16.1%).

Does blocking AI bots hurt my Google rankings?

No. Googlebot is separate from Google-Extended, and Google says Google-Extended does not affect inclusion or ranking in Google Search. Blocking AI search bots such as OAI-SearchBot or PerplexityBot does reduce your chances of being cited in those AI answers.

Which AI bots should I block in robots.txt?

If you want to stay out of model training, block training bots such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. Keep OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user bots allowed if you want to appear in AI answers. Most sites in our study that block AI bots follow this pattern.

Do AI bots respect robots.txt?

The major AI companies say their crawlers follow robots.txt. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. To block a bot that ignores robots.txt, you need server or CDN rules.