16 of 17 OpenAI partner sites allow ChatGPT's search bot. Three allow Claude's search bot.

On October 19, Anything Engine Optimization read the crawler rules on 52 sites, then asked ChatGPT and Claude for the same coverage from 12 of them.

What changed

Anything Engine Optimization read a fixed list of 55 news, trade and reference sites, set on September 11, and could read 52 of the files. It says the list is not a sample of the whole web. Each result is for the homepage path. Several sites block only some sections. The count records the written rule. It does not show whether the server enforces it.

Those rules live in robots.txt.

On the 52 files, OpenAI’s search bot, OAI-SearchBot, was blocked on 16 sites. Anthropic’s search bot, Claude-SearchBot, was blocked on 28. PerplexityBot was blocked on 30. Googlebot and Bingbot were blocked on one site, Reddit, whose file blocks every crawler.

Seventeen of the 52 have a public OpenAI deal. Reuters is on OpenAI’s partner list without announced terms, and the count includes it in the 17. Of those 17, 16 let OAI-SearchBot in and 3 let Claude-SearchBot in: Axios, the Financial Times and The Washington Post. Reddit is the partner that blocks OAI-SearchBot. Thirteen sites let ChatGPT’s search bot in and block Claude’s. All 13 are on OpenAI’s partner lists, and 11 name Claude-SearchBot explicitly. The BBC has no OpenAI deal in this count. It blocks OAI-SearchBot and lets Claude-SearchBot in.

The 35 sites without an OpenAI deal treat the three search bots alike. Fifteen block OAI-SearchBot, 14 block Claude-SearchBot, and 15 block PerplexityBot.

Training bots split on the partner sites. Three of the 17 block GPTBot. Sixteen block ClaudeBot. Google-Extended, Google’s AI-training control, is blocked on 12 partner sites and on 12 sites with no OpenAI deal. Anything Engine Optimization says none of these publishers has an announced licensing deal with Anthropic.

The same day, Anything Engine Optimization opened one new chat in the free ChatGPT app and one in claude.ai, on Sonnet 5.5, with web search on. It asked each app to find that week’s AI coverage from each of 12 sites and to link the articles. It says this was one run each, and that answers vary from run to run.

ChatGPT linked articles on the site itself for 9 of the 12. It did not do that for The New York Times, the BBC or CNN. Of those 12, those three block OAI-SearchBot.

claude.ai linked articles on the site itself for 4: CNN, The Washington Post, Axios and TechCrunch. CNN’s file blocks Claude-SearchBot, and claude.ai still linked CNN.

Asked who broke two stories, ChatGPT linked The Verge’s original and Reuters’ original. claude.ai named both outlets but did not link the originals. It gave secondary write-ups, and a Reuters wire copy on TradingView.

What a block does

Anything Engine Optimization quotes OpenAI on the search bot. Sites that disallow OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.”

The other names do other jobs. In this count, GPTBot and ClaudeBot are the training bots. ChatGPT-User fetches a page when a person asks. Anything Engine Optimization says OpenAI’s rule is that robots.txt rules may not apply to ChatGPT-User. It says Perplexity says Perplexity-User generally ignores robots.txt rules. It says Anthropic says all three of its bots honor the file.

Anything Engine Optimization calls a robots.txt line a request, not a lock. It says Anthropic’s and Perplexity’s search tools may also use other companies’ search indexes. It says a block on their own bot stops that bot from fetching the page, and does not close every route to the reporting. It says Google’s position is that blocking Google-Extended does not change a site’s inclusion in Google Search, which is where AI Overviews and AI Mode appear.

Yelp’s homepage path blocks OAI-SearchBot, Claude-SearchBot and PerplexityBot, leaves Googlebot allowed, and blocks Google-Extended. Reddit licenses its content to Google and OpenAI while its file blocks every crawler. The count shows where the rules and the deals line up. It does not show why the rules were written.

Check the lines on your site

Open the robots.txt file on your own site. Search for each name. Mark it Disallow, Allow, or not listed.

1. OAI-SearchBot, for ChatGPT search.
2. GPTBot, for OpenAI training.
3. ChatGPT-User, for a page a person asked ChatGPT to open.
4. Claude-SearchBot, for Claude search.
5. ClaudeBot, for Anthropic training.
6. PerplexityBot, for Perplexity search.
7. Perplexity-User, for a page a person asked Perplexity to open.
8. Googlebot, for Google Search.
9. Google-Extended, for Google’s training control.

Read the nine marks as separate choices. A search bot, a training bot, and a bot sent because a person asked are different lines.

Then ask one question, once. Write a question a customer asks before they hire or buy, in the customer’s words. Ask it in free ChatGPT and in Claude, in a new chat each time. Save both answers with the date. Mark each link: your page, your rival’s page, another site, or no link. Treat the pair as one run. Anything Engine Optimization said answers vary from run to run.

What to do this week

• Open your robots.txt and mark the nine bot names. Keep the search lines and the training lines separate.
• Ask one customer question in free ChatGPT and in Claude, and save both answers with the date.
• If a site that covers your trade is one of the 52, read its row before you assume both apps can fetch it.
• Ask the same question again next month, and set the two pairs side by side.

To have a dated read of where AI names you, your rival, or nobody, for the questions your customers ask, book a war briefing with New Reward and bring your rival’s name. A real person walks you through the report on a call each month.