Should robots.txt block AI crawlers? A 26-publisher snapshot and policy guide

How robots.txt matching works

A site’s robots.txt is a text file at the top-level path, such as https://example.com/robots.txt. User-agent identifies the crawler group; Disallow and Allow describe paths. Under RFC 9309, a matching dedicated group is used; if no group matches, the wildcard group can apply. Matching paths are evaluated by specificity.

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

In this example, GPTBot is refused site-wide, while other bots without a matching dedicated group encounter the wildcard restriction on /admin/. A bot being absent by name does not mean it is unblocked: a wildcard Disallow: / could still refuse it.

Robots rules are not a password, firewall or guarantee of compliance. They also differ from search-indexing and preview controls. Sensitive content needs real authentication or other appropriate access control; do not treat a robots file as a security boundary.

In the tables, Blocked means the applicable rules prohibit the root path; Explicitly allowed means a dedicated allowance; Allowed, path limits means the root is permitted while some paths remain restricted. Not named means no dedicated group and no root block under the snapshot’s wildcard rules. It never means that every path or network request is allowed.

Know what each bot setting controls

Google-Extended is a product-control token, not a separate HTTP request user-agent. Grounding means providing search-index content at answer time; it is not the same process as model training. The token does not affect Google Search inclusion or ranking. A policy described as “block training, retain search” therefore needs an explicit decision about Google grounding too.

After reading a page, an agent may still need to log in, obtain authorization and verify the result. These later steps belong to Agent Experience (AX); robots.txt alone cannot establish whether they are possible.

For AI Overviews and AI Mode, Google documents Googlebot access and ordinary indexing and snippet controls, including nosnippet, data-nosnippet, max-snippet and noindex. Those choices can also affect ordinary search presentation. Since 2026 there is also a separate Search Console opt-out; the next section sets out what each control reaches.

The OAI-SearchBot navigation-link exception is equally important: refusing that bot does not establish that the URL can never appear anywhere in ChatGPT. These provider descriptions explain documented behaviour, not measurements of compliance by every bot.

Controls × scope: what each Google, OpenAI and Cloudflare setting affects

The previous table says which bot is which. This one says what changes when a control is used. Between late 2025 and 2026 all three providers changed their rules: Google added a Search Console opt-out for AI search features, OpenAI revised its crawler documentation, and Cloudflare announced purpose-based defaults.

On 7 November 2026 LS-Labs asked Gemini 3.8 Flash and GPT-6 Luna the same set of Traditional Chinese questions, once without and once with web search. All four answers knew that blocking Google-Extended does not remove a site from AI Overviews. None named the Search Console opt-out, and none mentioned Cloudflare’s 2026 defaults. The table follows each provider’s current official documentation, cell by cell.

Google: the Search Console switch and Google-Extended each cover half

Google announced the new site-level control in a 3 June 2026 blog post, beginning with a test among some UK site owners and noting its engagement with regulators such as the UK’s Competition and Markets Authority. A later note says that from 31 August 2026 the control reached all websites worldwide. Taiwan is not named, but it falls within that worldwide rollout. The setting sits under Settings in Search Console as “Search generative AI”.

According to the Search Console help page, an opted-out site’s content does not appear in AI Overviews, AI Mode or generative AI features in Discover, and is not used to generate those responses, so the site receives no traffic or impressions from them. Google says the control is not a ranking or inclusion signal for other parts of Search. Google’s AI optimization guide also states that a site must stay included in this setting to be eligible for generative AI features in Search.

The switch does not govern training. The help page says it does not affect AI training and points to Google-Extended for that, and Gemini Apps is not in its listed scope. Google-Extended works the other way round: it covers Gemini model training and Gemini Apps / Vertex AI grounding, not AI Overviews. The two are complementary. A site that only wants to leave AI search features changes Search Console; a site that wants neither use sets both.

Training controls are also changing. In a 30 October 2026 Search Central post, Google announced plans to support a shared AI training signal in robots.txt with other AI providers, aiming for 2027; the crawler documentation says early 2027 and gives the form user-agent: __usage_ai-training. If a site opts out of training with that shared token and does not mention Google-Extended, Google will treat it as a Google-Extended disallow. Google-Extended itself continues to work.

OpenAI: the December 2025 revision made older guides obsolete

OpenAI’s crawler documentation was revised around 9 December 2025, as PPC Land and others reported, yet many guides and AI answers still follow the old version. The current page differs in four places.

ChatGPT-User is initiated by users, so robots.txt rules “may not apply”, and it is not used to decide whether content may appear in search. When a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both search and training; the OAI-SearchBot entry no longer says it is excluded from training, so the bot to refuse for training is GPTBot. Sites that refuse OAI-SearchBot are not shown in ChatGPT search answers but can still appear as navigational links, and changes take about 24 hours. The page also adds OAI-AdsBot, which visits only pages submitted as ChatGPT ads to check them against ad policies; the data it collects is not used to train foundation models.

So refusing GPTBot, OAI-SearchBot and ChatGPT-User does not cut a site off from OpenAI entirely, although that is exactly what Gemini’s offline answer recommended in the LS-Labs test. robots.txt states a preference to automated crawlers. Stopping user-initiated fetches needs server or CDN rules, and it also means a ChatGPT user who pastes the URL will not be able to read it.

Cloudflare: from default blocking to purpose categories

All four AI answers stopped at 1 July 2025, when new domains began blocking AI crawlers by default and Pay per crawl launched in private beta, returning HTTP 402 to crawlers that have not agreed to pay. Gemini’s offline answer even placed both in 2024; what launched in July 2024 was the opt-in one-click block. None of the answers mentioned the two changes that followed.

On 24 September 2025 Cloudflare introduced the Content Signals Policy, adding three purpose signals to robots.txt, search, ai-input and ai-train; its managed robots.txt defaults to search=yes, ai-train=no. Cloudflare states that these signals express preferences and are not technical countermeasures, and that some companies may simply ignore them.

On 1 July 2026 Cloudflare announced purpose categories including Search, Agent and Training, and said that from 15 September 2026 new domains would block Training and Agent by default on pages that display ads, while Search stays allowed. The same post says defaults follow the most restrictive applicable rule, so multi-purpose crawlers such as Googlebot, Applebot and Bingbot will be blocked for customers who choose to block Training. For site owners this is the change with the widest effect: a rule meant only for training can also stop crawling for Google Search and AI Overviews.

The documentation and the announcement do not fully agree. The AI Crawler and Agent access policies page, updated 2 November 2026, lists Allow as the new-domain default for all three categories, and says Training actions set to Block also apply to mixed-purpose crawlers. The managed robots.txt page says that with Bot Preference Sync on and Training set to Disallow, mixed-use crawlers are excluded from the edge block. Block and Disallow therefore behave differently; after any change, confirm that Googlebot requests still get through. Pay per crawl remained in closed beta in November 2026.

lslabs.tw can serve as a reference point. Since 4 November 2026 its Cloudflare AI crawler and agent policies allow all three categories and Bot Preference Sync is off, so the robots.txt stays as the site wrote it, with no managed block prepended; only a few individual crawlers are blocked.

International publishers: 12 historical snapshots

* denotes an allowlist: bots not admitted by a dedicated allowance fall under the root block. ** denotes cooperation through a parent company: The Wall Street Journal through News Corp and The Verge through Vox Media. “Not checked” is not evidence of no agreement. The partnership column records only the original review’s checked announcements, not a complete contract register.

Eleven of the 12 snapshots blocked ClaudeBot and eleven blocked PerplexityBot. New York Times, BBC, CNN and Bloomberg also blocked OAI-SearchBot. The four publishers with cooperation checked in the source review—The Wall Street Journal, Financial Times, The Verge and Axios—did not prohibit GPTBot at the root in these files. Coexistence does not establish that a contract caused a robots rule.

The Wall Street Journal and Reuters used allowlists. The former listed three OpenAI bots; Reuters admitted OAI-SearchBot and ChatGPT-User but not GPTBot. Partial-path restrictions still matter beyond the root-path classification shown here.

Taiwan publishers: 14 readable snapshots

CommonWealth Magazine returned HTTP 403 and Business Weekly HTTP 404; neither was counted as an observed allow or block. The usable media total is 12 international plus 14 Taiwan files, not all 28 attempted files.

Six Taiwan publishers blocked GPTBot and ClaudeBot. For OAI-SearchBot, UDN, Liberty Times, Central News Agency and The News Lens had no dedicated group and no wildcard root block; The Reporter explicitly allowed it; China Times blocked it. ETtoday and SET News explicitly allowed the four OpenAI, Anthropic and Perplexity bots shown. Six other publishers did not name any of the five table entries and did not prohibit their root access through wildcard rules.

No Taiwan snapshot explicitly named Claude-SearchBot. Two blocked PerplexityBot. Absence could reflect a deliberate choice or an unmaintained list; the file alone cannot tell which.

These files do not establish the publishers’ motives. Pew’s 2025 analysis found AI-summary-source clicks in 1% of visits to Google pages with an AI summary, using browsing data from 900 US adults in March 2025. That is relevant traffic context, not evidence that these 26 publishers adopted rules for that reason. The discussion of revenue and licensing trade-offs is LS-Labs’ interpretation.

Five policy patterns and what a robots file cannot answer

Cloudflare’s July 2025 announcement discussed AI-crawler choices for new domains and the Pay Per Crawl beta, which was still a closed beta in November 2026. Such network policies are not inferable solely from a public robots file, unless the site enables Cloudflare’s managed robots.txt, which prepends Content-Signal and Disallow lines. Cloudflare’s 2026 categories and defaults are covered in the controls table above. Immediate news, paid investigations and practical information can also warrant different content-level decisions rather than one uniform site-wide preference.

How the decision differs outside publishing

Commerce platforms and independent stores

The same-day Amazon snapshot named 99 crawlers and refused them site-wide, including training, search and user-triggered agents. Walmart’s snapshot restricted functional paths such as accounts, APIs, site search and orders without AI-specific groups. These are historical file observations, not live tests or evidence of every platform’s current access policy.

Separately, Shopify explains Catalog and crawling as different discovery routes. Restricting a public crawler does not necessarily remove products supplied through an eligible Catalog integration or another feed. Check the actual channel settings as well as robots rules. See agentic commerce readiness for data and checkout boundaries.

Communities and licensed access

The Reddit snapshot refused all crawlers through a wildcard rule and referenced its public-content policy. Yet OpenAI’s May 2024 partnership announcement described access through Reddit’s Data API. Public crawling and licensed data access are different doors. Dcard’s snapshot restricted an account-related path without AI-specific rules; no licensing conclusion follows from that absence.

Brand, B2B and SaaS websites

For a site whose main purpose is accurate product information or enquiries rather than pageview revenue, being read accurately can be valuable. Check stale prices, specifications trapped in images and conflicting documentation versions. Whether to permit training remains a separate licensing decision. Member areas, administration and test environments need access control.

Local businesses

Maps, reviews and community content can describe a restaurant or clinic even when its official site is unavailable. The six-restaurant case study observed businesses without confirmed independent domains appearing in answers. An official robots file cannot govern all those third-party sources; maintain the platform information you control too.

A starting framework for your own policy

The following is a decision framework, not a measured optimum or blanket recommendation for each sector.

A common starting example permits OAI-SearchBot, Claude-SearchBot and PerplexityBot while refusing GPTBot and ClaudeBot. Google-Extended is included below only if the owner intends to refuse both its training and specified grounding uses. Other bots still follow existing dedicated or wildcard groups.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

This is an illustration, not a replacement file to paste without checking existing groups and protected paths. Merge and test the intended effect before applying it.

Four checks before changing a rule

1. Compliance differs by use. OpenAI and Perplexity document exceptions for user-triggered retrieval. Protect non-public data with authentication.
2. The network layer may disagree. CDN and firewall rules can block a request permitted by robots.txt. Since 2026, blocking Cloudflare’s Training category can also stop mixed-use crawlers such as Googlebot. Test the authorised path rather than inferring access from the text file.
3. Subdomains are separate. A news subdomain and a video subdomain need their own applicable files; Anthropic explicitly reminds owners to set preferences for each subdomain.
4. Names and purposes change. Check the wildcard rule when a bot is not named, and record the date and reason for policy changes. A missing name does not reveal the owner’s intent.

A prompt to prepare a review, not delegate the decision blindly

Give an assistant the business model, content types, desired search visibility, restricted uses and current complete robots file. A vague question about whether to block AI cannot account for information it has not received.

Review this site's AI crawler policy.
Site and business model: [URL, type, revenue model]
Content categories: [public products, paid articles, member area...]
Want discoverable through AI search: [content]
Do not want used for training: [content]
Must require authorised access: [content]
Current complete robots.txt: [paste file]

1. Explain root-path rules for GPTBot, OAI-SearchBot,
ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User,
PerplexityBot, Perplexity-User and Google-Extended.
Apply wildcard rules where no dedicated group exists.
Separate partial-path restrictions and unknown effects.
2. Recommend preferences by content and use: training,
search and user-triggered retrieval. For Google-Extended,
state its combined training and grounding implications.
3. Draft a merged candidate file, preserving unrelated rules.
4. Identify work robots.txt cannot do: authentication,
indexing/snippet controls, CDN policy and licensed feeds.
5. Identify uncertainties and the official documents to check.

Check bot names against provider documentation, manually inspect several matching examples and treat the recommendation as a starting point. The assistant cannot infer undisclosed licensing terms, lost revenue or the actual effect of a firewall.

Common questions

Does blocking GPTBot remove a site from ChatGPT search?

Not necessarily. GPTBot and OAI-SearchBot have separate purposes. An OAI-SearchBot opt-out affects search answers but still permits the documented navigational-link exception.

Does Google-Extended block AI Overviews?

No. It governs Gemini training and specified Gemini Apps / Vertex AI grounding uses. AI Overviews and AI Mode follow Google Search controls; to leave them alone, use the Search Console “Search generative AI” setting, available to sites worldwide since 31 August 2026.

How can a site stay out of AI Overviews but keep ordinary search?

Opt out of “Search generative AI” under Settings in Search Console. Google announced the control on 3 June 2026 and extended it to sites worldwide on 31 August 2026. An opted-out site does not appear in AI Overviews, AI Mode or Discover’s AI features, and Google says the setting is not a ranking signal for other search results. It does not affect AI training or cover Gemini Apps; those still need Google-Extended.

Does blocking AI training crawlers in Cloudflare affect Google Search?

It can. Cloudflare’s 1 July 2026 announcement says customers who block Training will also block multi-purpose crawlers such as Googlebot, Applebot and Bingbot. Its documentation adds that with Bot Preference Sync on and Training set to Disallow, mixed-use crawlers are excluded from the edge block. After changing the setting, confirm that Googlebot requests are not blocked.

Which Anthropic bots can be configured separately?

ClaudeBot, Claude-SearchBot and Claude-User correspond to model-development collection, search and user-requested retrieval. Check the provider’s current description before editing.

Can robots.txt protect private information?

No. It communicates crawling preferences. Use appropriate authentication and authorisation for information that must not be publicly accessible.

Methods, evidence and sources

The original 23 September 2026 collection used public HTTP GET from a Taiwan network location for 16 Taiwan and 12 international publishers plus Amazon, Walmart, Reddit and Dcard. The two failed publisher files were excluded, leaving 26 media snapshots. The publisher list was purposive, not random. The workspace retains original files, timestamps and SHA-256 hashes; this article is not a public raw-data download.

Classification applies RFC 9309 to root-path access. It does not measure every path, geographic variant, CDN policy or actual request outcome. The media tables describe only that collection date. Cooperation was checked in the original review only for News Corp, Financial Times, Vox Media and Axios. LS-Labs reported no commercial relationship with the named publishers, platforms or AI providers in the source article.

Every cell in the controls table was rechecked against Google, OpenAI and Cloudflare pages on 7 November 2026; where a page is undated, the check date is given. The OpenAI revision date comes from third-party reporting, as the OpenAI page carries no revision date. Cloudflare’s 1 July 2026 announcement and its documentation updated on 2 November 2026 describe new-domain defaults differently; both are reported rather than reconciled. The AI-answer observations come from a single LS-Labs run on 7 November 2026: 15 Traditional Chinese questions without a system prompt, sent through the official APIs to Gemini 3.8 Flash (requested as gemini-3.7-flash; the response reported 3.8) and GPT-6 Luna, each once without and once with web search. A single run does not show stable model behaviour.

• RFC 9309 and Google’s robots.txt introduction.
• OpenAI crawler documentation; Anthropic bot documentation; Perplexity crawler documentation.
• Google-Extended and Google AI search controls.
• Google’s 3 June 2026 announcement (updated 31 August 2026), the Search generative AI control help page, the AI optimization guide and the 30 October 2026 training-control post.
• PPC Land on the December 2025 OpenAI crawler revision.
• Cloudflare’s July 2024 one-click block, Content Signals Policy (24 September 2025), 1 July 2026 announcement, AI Crawler and Agent access policies, managed robots.txt and Pay per crawl documentation.
• News Corp, Financial Times, Vox Media, Axios and Reddit partnership announcements.
• Pew’s Google click study, 2025; Cloudflare’s July 2025 policy announcement.
• Shopify robots guidance and Catalog discovery guidance.

To evaluate what follows permitted discovery, use the separate GA4 referral guide; its sessions do not establish training use or all AI exposure.