
When Cloudflare reviewed the 10,000 largest domains on the web in June 2025, roughly 14 percent of those with a robots.txt file had rules aimed specifically at AI bots. The rules are often written to stop AI models from training on the content. But the same file also decides whether ChatGPT, Claude, Copilot, Gemini and Google AI Overviews can fetch, read and cite your content when a customer asks a question.
What happened. The major AI services use different crawlers for different purposes. OpenAI explicitly separates the bot that collects training data from the bot that fetches sources for ChatGPT’s search answers, and Google controls AI Overviews and Gemini with different signals.
Why it matters. A block meant to stop training can also shut you out of the answers where customers compare suppliers. The risk lies both in robots.txt and in the security settings of network services such as Cloudflare.
What you can do. Review which bots you let in, make a deliberate decision for each service, and then track whether you are actually mentioned and cited.
Training and search are handled by different bots
The key distinction is between letting an AI service train on your content and letting it use that content as a source in an answer. These are two different decisions, and the services handle them in different ways.
ChatGPT. OpenAI’s OAI-SearchBot is used to show websites in ChatGPT’s search features. According to OpenAI, sites that block it do not appear in search answers, although they can still show up as plain navigation links. GPTBot, on the other hand, collects content that may be used to train models, and blocking it does not affect visibility in search.
Claude. Anthropic’s Claude-SearchBot indexes content to improve Claude’s search results, and blocking it may reduce your visibility there. ClaudeBot collects content that may be used for model training, and Claude-User fetches pages when a user asks Claude a question. According to Anthropic, its bots respect robots.txt.
Google AI Overviews and AI Mode. Google says there are no additional technical requirements. The page must be indexed and eligible to be shown with a snippet in Google Search, and access is governed by the rules for Googlebot.
Gemini. Google-Extended affects neither whether a site appears in Google Search nor how it ranks. It does, however, control whether the content may be used to train Gemini models and for so-called grounding in the Gemini apps, that is, when Gemini pulls content from the search index to answer a question.
Copilot. Microsoft reports citations in Copilot and in Bing’s AI summaries in Bing Webmaster Tools and states that Bing respects the preferences expressed in robots.txt.
The common trap: blocking everything labeled AI
GPTBot was the most frequently blocked bot in Cloudflare’s review. 312 domains shut it out fully or partially. That is a reasonable choice for anyone who doesn’t want to contribute to model training. The problem arises when the block is broader than intended, for example when a ready-made list of AI bots also includes the bots that fetch sources for answers.
Network and security services have made the choice easier, but also easier to get wrong. Cloudflare has announced that, from September 15, 2026, new domains will by default block training bots and agents on pages that show ads, while search bots are let in. Multi-purpose bots such as Googlebot, Bingbot and Applebot are assessed according to the most restrictive rule. According to Cloudflare, they are therefore blocked for customers who have chosen to block training.
For a B2B company, the consequences can be significant. Googlebot controls not only AI Overviews but your entire presence in Google Search, and Microsoft reports Copilot citations through Bing Webmaster Tools.
Access is a requirement, not a guarantee
Letting the right bots in makes you possible to cite. It doesn’t make you cited. Google itself points out that neither indexing nor display is guaranteed, even when the technical requirements are met.
That’s why the technical setup needs to be followed up with measurement. Since February 2026, Bing Webmaster Tools shows how often a website is cited in Copilot and Bing’s AI summaries, which pages are cited and which search phrases the AI used when it fetched the content. For ChatGPT, Claude, Gemini and Google AI Overviews you need to measure in other ways: ask the questions your customers ask and see whether you are mentioned, cited or missing, and do it more than once.
What it means for you
Start by reviewing the settings that control who is allowed to read your website.
Read your robots.txt. Check which bots are named and whether search bots such as OAI-SearchBot, Claude-SearchBot, Googlebot and Bingbot are blocked.
Check your CDN and firewall. Review the AI and bot settings at your network provider, especially if the site sits behind Cloudflare.
Make two separate decisions. Decide whether you want to contribute to model training and whether you want to appear as a source in AI answers. The answers don’t have to be the same.
Weigh Google-Extended carefully. It doesn’t affect Google Search, but it can affect whether your content is used when the Gemini apps build their answers.
Measure the result. Track how you are mentioned and cited over time, both in Bing Webmaster Tools and through recurring questions in each service.
Technical access is the foundation. After that, the work is about making sure your own content, and what others write about you, gives the AI services something to cite, so that more potential customers find their way to you. Want to know how your company appears in ChatGPT, Claude, Copilot, Gemini and Google AI Overviews today? Get in touch with us and we’ll go through it together.
Sources
Overview of OpenAI Crawlers – OpenAI
Anthropic’s crawlers – Anthropic
AI features and your website – Google Search Central
Google’s common crawlers – Google for Developers
Introducing AI Performance in Bing Webmaster Tools Public Preview – Microsoft Bing, February 2026
Your site, your rules: new AI traffic options for all customers – Cloudflare, July 2026
From Googlebot to GPTBot: who’s crawling your site in 2025 – Cloudflare, July 2025