Tag
Guides, research and news tagged robots txt.
3 items
news
OpenAI documents separate controls for ChatGPT search, potential foundation-model training and title-only navigational links. Choose each control for the outcome you actually want.
news
Cloudflare now lets sites state a no-training preference without blocking accountable mixed-use search crawlers. The control is useful, but it is not universal and it does not opt a page out of AI summaries.
guides
The least glamorous part of AEO and reliably the highest-yield. If an AI crawler cannot fetch and read your page, nothing else you do matters.