Cloudflare's Disallow AI Training Setting: What It Changes for AEO

Cloudflare changed a previously awkward crawler decision on 15 September 2026. A site owner can now select Disallow AI Training while continuing to allow accountable mixed-use crawlers to index the site for search.

That separation is useful for AEO because blocking a mixed-use crawler at the network edge can also erase the search access a site meant to preserve. It is not a new ranking tactic, and it is not a universal opt-out from AI-generated answers.

The operational question is narrower: can we refuse model training without accidentally refusing discovery? Cloudflare's new control makes the answer closer to yes, but the details differ by operator.

What Cloudflare changed

Cloudflare now exposes Search, Training and Agent traffic as separate domain-level decisions. For Training, site owners can choose Allow, Disallow AI Training, Block on pages with ads or Block.

The distinction between Disallow and Block carries most of the consequence.

With Disallow AI Training, Bot Preference Sync publishes applicable no-training directives in robots.txt. Cloudflare then allows mixed-use crawlers that qualify for its Accountable designation to continue search crawling. Training-only crawlers, and crawlers that do not meet Cloudflare's criteria, are blocked at the edge.

With Block, Cloudflare now blocks every crawler classified as performing training, including mixed-use Applebot, Bingbot and Googlebot. That can affect ordinary search discovery. A team that wants search visibility but not training should therefore not treat the two settings as interchangeable.

Cloudflare says the new controls are available on all plans. Existing configurations are migrated, but the practical advice is still to inspect the resulting settings and the live robots.txt rather than assume a legacy toggle means what it used to mean.

The operator support is uneven

Cloudflare groups Apple, Google and Microsoft under one Accountable designation, but the current mechanisms are not identical.

Google

Google's documented control is Google-Extended. It is a product token in robots.txt, not a separate crawler making its own requests. Google says the token controls whether content already crawled by Google may be used for future Gemini model training and for grounding in Gemini Apps and Vertex AI.

Google also states that Google-Extended does not affect inclusion or ranking in Google Search. That makes it a real training-and-grounding control without turning it into an SEO lever. Its documented affected products do not include AI Overviews or AI Mode, so it should not be described as an opt-out from those Google Search surfaces.

Apple

Apple documents a similar separation through Applebot-Extended. Apple says the token does not crawl pages itself. It governs whether data collected by Applebot may be used to train Apple's general-purpose foundation models, while pages can remain discoverable in Apple search features.

Apple separately supports nosnippet for web answers and descriptions. That is an important example of why training and answer presentation need different policy decisions. Refusing model training does not automatically refuse a generated summary.

Microsoft

Microsoft is the current caveat. Cloudflare says Bing's domain-level robots.txt no-training mechanism is targeted for early 2027. Until then, selecting Disallow AI Training does not automatically communicate that preference to Bing through robots.txt.

Bing's existing controls are page-level meta directives. Its documentation says NOCACHE can keep a URL, title and snippet eligible for Bing Chat while limiting training use to that URL, title and snippet. NOARCHIVE is stricter: the page remains eligible for ordinary Bing search, but is not included or linked in Bing Chat answers and is not used for model training. Those are materially different AEO outcomes, so the choice should be made page by page rather than copied from a generic crawler list.

Training control is not answer control

The most likely implementation mistake is to collapse three questions into one:

1. May this operator index the page for search?
2. May it use the page to train a model?
3. May it quote, summarize or ground an answer with the page at response time?

Cloudflare's new setting addresses the second question while trying to preserve the first. It does not settle the third across engines. Cloudflare says summary controls are the next area of work, and current answer controls remain operator-specific.

That separation also changes how the decision should be measured. Crawler logs can show access. Search Console and Bing Webmaster Tools can show parts of search visibility. Prompt sampling can show whether a page is still cited or summarized. None of those observations proves whether a model provider used historical content for training.

A safe rollout workflow

Treat the setting as a production access change, not a dashboard preference.

1. Write the policy before changing the toggle

Decide separately whether the business wants Search, Training and Agent traffic. An ad-supported publisher may reach a different answer from a product catalogue that values broad discovery. Record any page-level exceptions, especially paywalled or licensed material.

2. Capture the current state

Save the existing Cloudflare settings and the live robots.txt. Check for custom rules aimed at Googlebot, Bingbot, Applebot, OAI-SearchBot and other discovery crawlers. A group-level setting can be correct while an older custom rule still blocks the route that matters.

3. Change one layer at a time

If the policy is "search yes, training no," keep Search on Allow and set Training to Disallow AI Training. Decide Agent access independently. Do not use Training Block unless losing mixed-use search crawlers is intentional.

For Bing, document the gap until the planned robots.txt support arrives. If a page needs a stricter rule today, evaluate NOCACHE and NOARCHIVE against the page's desired presence in Bing Chat before deploying either.

4. Verify access and visibility

Fetch the public robots.txt after the change and confirm the generated groups match the policy. Then monitor verified crawler requests, search impressions and a fixed answer sample for several crawl cycles.

For small samples, manual checks are enough. For a repeated cross-engine set, CiteCue can monitor whether named pages and brands continue to appear, keep the same prompts in the comparison and route supported regressions into a remediation queue. Disclosure: AEO Landscape and CiteCue have common ownership; see our full disclosures. CiteCue observations are not crawler logs, model-training records or proof that an operator honored a preference. Their value here is catching visible changes after an infrastructure decision.

5. Keep a dated change log

Record the setting, affected domain, implementation date, live directives and validation results. Revisit the Microsoft path when its planned domain-level support ships, and review Cloudflare's classification changes rather than assuming today's list is permanent.

For the broader crawl audit, use our technical AEO checklist. If visibility changes after rollout, work through the citation troubleshooting sequence before blaming the training preference.

What not to claim

There is no published evidence that refusing training improves rankings, citations or traffic. Apple and Google explicitly separate their extended training tokens from search ranking, while Microsoft says its current meta controls preserve ordinary search eligibility.

There is also no defensible basis for saying one Cloudflare switch prevents every possible use of content. The system combines published preferences, crawler classification and edge enforcement. Its effectiveness depends on the operator, the crawler identity and the specific use being controlled.

The credible AEO position is therefore simple: preserve the discovery you want, refuse the reuse you do not want, and measure each layer with the tool that can actually observe it.

Sources and verification

• Cloudflare: Have it both ways — stay discoverable in search while disallowing AI training
• Cloudflare: Say it once — introducing Bot Preference Sync
• Google Crawling Infrastructure: Google-Extended
• Apple Support: About Applebot
• Bing Webmaster Blog: Options for controlling content use in Bing Chat

Sources checked 16 September 2026. This article was researched, drafted and published through AEO Landscape's automated editorial workflow. Product behavior and commitments are attributed to the primary sources above; the rollout workflow and AEO interpretation are ours.