AI Training Preferences robots.txt, Set and Honor Them

OpenAI, Google, Microsoft, and Anthropic have lined up behind one machine-readable line in robots.txt, and both sides of the crawl are already misreading it. Publishers are treating the AI training preferences robots.txt signal as a switch that stops AI from touching their content. Agent teams are treating it as someone else's standards theater. Neither reading survives contact with the actual mechanics.

The signal is a coordination layer, not a control layer. Four companies promise to honor a stated training preference. Every crawler that never signed the list ignores it for free, and anyone running a fetcher just inherited an etiquette bar that enterprise buyers will audit. This guide ships both sides, the robots.txt policies publishers can deploy today, and the compliance path for crawlers that want to be trusted.

What the AI Training Preferences robots.txt Signal Is, and Who Honors It

Which AI companies honor the shared robots.txt signal? The named list so far is OpenAI, which announced the convention, plus Google, Microsoft, and Anthropic, as The Verge reported. The mechanics look simple on the surface: a site states preferences in robots.txt at the level of use rather than access, covering things like whether its content may be used for model training, and participating providers map those preferences onto their own crawlers.

The layer it operates at is the interesting part. robots.txt has only ever expressed access, meaning you may fetch this path, you may not. The shared signal stretches the same file into expressing use, meaning you may fetch, and here is what you may do with what you fetch. That stretch is why a preference line and a Disallow rule are different tools aimed at different readers, and why per-agent blocks did not become obsolete overnight.

Honoring the signal is voluntary. It is a coordination convention with reputational enforcement, not a legal obligation, and it binds no crawler whose operator has not committed to it.

Voluntariness cuts in both directions. A provider honoring the signal today can change policy tomorrow, and a scraper that never joined the list was never covered to begin with. Treat the participant list as a fact you re-check quarterly, not a property of the web.

Publisher Implementation, Three robots.txt Policies

If you came here asking how to opt out of AI training in robots.txt, the next three blocks are yours. Each relies on per-agent Disallow rules the major providers already document, so it keeps working regardless of how the new shared preference fields are named or rolled out. Add the shared-signal preference line once your target providers publish the field names they actually read, but treat the blocks below as the durable layer.

Policy 1, Block AI Training Outright

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Two details matter. Google-Extended is a separate product token from Googlebot, so blocking it affects Google's AI training without touching Google Search. And the agent list shifts as providers add crawlers, so confirm current names against each provider's crawler documentation before you ship.

What operators tend to miss: training blocks rot silently. Providers keep adding product tokens, the Google-Extended pattern on repeat, and every token you never listed never sees your Disallow. Fold the agent-list audit into the quarterly participant-list re-check you already owe, one calendar entry covering both.

Policy 2, Keep Search Surfacing, Block Training

robots.txt is allow-by-default, so this policy is Policy 1 plus explicit reassurance for agents that surface your content in search products rather than train on it:

# Training and indexing agents for AI products
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Search-side agents, explicitly allowed
User-agent: OAI-SearchBot
Disallow:

User-agent: Claude-SearchBot
Disallow:

An empty Disallow means allow everything in that group. This is also the policy shape the shared signal compresses into a couple of preference fields (no training, search surfacing fine), which is why it is the most common choice for publishers who want traffic from AI answers without contributing training data.

Policy 3, Per-Provider Granularity

Licensing deals, partnerships, or simple distrust of one provider call for a mixed policy:

User-agent: GPTBot
Disallow:            # empty means allowed, per your agreement

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Verify Before You Trust It

Your GPTBot robots.txt entry is only as real as the file the bot actually fetches.

• Run curl -A "GPTBot" https://example.com/robots.txt from a clean connection. You want a 200 with your file, served as text, not a CDN error page.
• Watch access logs for the agent names above. A 200 on /robots.txt from each crawler is the ground truth that it saw your policy.
• Purge your CDN after deploying. Edge-cached robots.txt plus each crawler's own cache means a preference change propagates over days, not minutes.

What the Signal Covers, and What It Leaves Open

An AI training preferences robots.txt opt-out buys exactly one row in the ledger below. Everything else publishers worry about is governed by a different mechanism, or by nothing at all.

Publisher concern	Covered?	What governs it instead
Training by OpenAI, Google, Microsoft, Anthropic	Yes, as a voluntary commitment	Per-agent Disallow rules as a backstop
Training by every other crawler	No	Per-agent blocks where names exist, bot management at the edge, legal tools
Being crawled for search	No, and not intended to be	Standard search engine controls
Being quoted or cited in an answer	No	Provider product policy and separate agents
Being summarized or surfaced in answers	Not via the training opt-out	The shared signal's other preference fields, where a provider implements them
Legal enforceability	None on its own	TDM-Rep in the EU, contracts, licensing

Does blocking GPTBot stop ChatGPT from citing my site? No. GPTBot fetches pages for training data; OpenAI documents separate agents for search (OAI-SearchBot) and user-initiated fetches (ChatGPT-User), and blocking one does not block the others (OpenAI's bot documentation). If your objection is being surfaced rather than being trained on, the training opt-out is the wrong lever.

The long tail is the bigger exposure. Cloudflare's 2025 crawler measurements reported that a large share of AI crawler traffic comes from bots that disregard robots.txt. The shared signal disciplines the majors, not the swarm; for the swarm you need rate limiting, bot management, or lawyers.

Propagation is also slow, and enforcement is reputational. A crawler that cached your robots.txt this morning may keep fetching under that view all day, and a non-participant may keep fetching forever. Neither case is a bug; each is the system doing exactly what a voluntary convention can do.

Prior Signals, From GPTBot to TDM-Rep

The robots.txt vs noai meta tag vs TDM-Rep question is a layering question, not a fight with one winner. Each layer covers a different failure mode.

Signal	Layer	Who reads it	Enforcement
Shared robots.txt preference	Host, file-level	Participating providers	Voluntary, reputational
Provider user agents (GPTBot, ClaudeBot, Google-Extended, CCBot)	Host, per-agent	Each provider's own crawlers	Voluntary, provider-documented
noai / noimageai meta tags	Page-level	Implementers who opt in	Voluntary
TDM Reservation Protocol	Site, machine-readable	Text and data mining actors	Statutory hook in the EU
ai.txt well-known file	Site, licensing terms	Implementers who opt in	Voluntary
C2PA provenance metadata	Content-level	Downstream tooling	Voluntary, aimed at attribution more than exclusion

Provider-specific AI crawler user agents predate the shared signal (GPTBot arrived in 2023, and Common Crawl's CCBot has fed training corpora far longer), so they remain the practical way to address crawlers individually when your policy differs by provider. Policy 3 above leaned on exactly that.

The noai meta tag, introduced by DeviantArt in 2022, is page-level and blunt:

<meta name="robots" content="noai, noimageai">

The TDM Reservation Protocol is the one with teeth. The W3C spec lets a site express a mining reservation in a place TDM tools are expected to check, via a /.well-known/tdmrep.json file or an HTTP response header:

{ "tdm_reservation": 1 }

In the EU this carries weight the robots.txt convention cannot match on its own. The DSM Directive's text and data mining exception permits mining unless rights holders reserve against it, and the directive conditions that reservation on being machine-readable. TDM-Rep is the spec built to satisfy the condition; a voluntary robots.txt convention alone does not guarantee it. If you have EU exposure, layer it alongside your robots policy rather than choosing between them.

The ai.txt draft at the IETF pushes further toward machine-readable licensing, a well-known file stating which uses are permitted, closer to a contract than a crawler directive. None of these layers replaces the row above. Publishers with real exposure run several and keep them consistent.

Scraper Implementation, Honoring Preferences in Your Crawler

If you are wondering how to make a custom crawler respect robots.txt, the mechanics fit in an afternoon. The policy decisions around them are the hard part. The emerging minimum bar for agent teams has four parts: a named and contactable user agent, preference checks at fetch time, robots state cached, with deliberate 404 and unreachable behavior per RFC 9309, and the ability to purge previously fetched content when a preference changes.

Step 1, Identify Your Crawler Honestly

Every request should carry a user agent a publisher can act on:

PastAgent/1.2 (+https://example.com/bot; contact@example.com)

That is how to identify an AI agent crawler user agent from the other side of the wire: product token, version, policy URL, contact. Anonymity is the tell of a crawler that plans to misbehave.

Step 2, Fetch, Cache, and Honor robots.txt

RFC 9309's failure semantics are where homemade fetchers go wrong. A 404 robots.txt means allow everything. A robots.txt that stays unreachable can be treated as disallow everything, so a publisher's robots outage can silence well-behaved crawlers, and your tolerance for that case is a decision you should make deliberately. Cache what you fetch, revalidate at least daily following common crawler practice, and honor Cache-Control when the server sends it.

import time
from urllib import robotparser

UA = "PastAgent/1.2 (+https://example.com/bot)"

class RobotsGate:
    def __init__(self, ttl=24 * 3600):
        self.ttl, self.cache = ttl, {}

    def allowed(self, url):
        origin = "/".join(url.split("/")[:3])
        ts, rp = self.cache.get(origin, (0, None))
        if time.time() - ts > self.ttl:        # daily revalidation
            rp = robotparser.RobotFileParser()
            rp.set_url(origin + "/robots.txt")
            rp.read()
            self.cache[origin] = (time.time(), rp)
        return self.cache[origin][1].can_fetch(UA, url)

Two caveats worth knowing. Python's robotparser maps network errors to disallow-all, which is stricter than the RFC's "may," so confirm that matches your intent. And the library parses access rules only; the shared signal's preference fields need a small parser of their own, keyed to whatever each provider documents.

Step 3, Purge on Preference Change

Keep an ingest ledger: every document stored alongside the robots.txt digest in effect when it was fetched. When a refetch shows a changed digest, re-evaluate retained documents against the new policy and purge or flag what is no longer permitted. Without this, your compliance story ends at fetch time, which is exactly where an auditor will look.

A Compliance Checklist for Agent Teams

AI crawler compliance is turning into procurement language, so make the bar explicit before someone else does.

• Named, contactable user agent. Product token, version, policy URL, and an inbox a human reads.
• Preference checks at fetch time. Not at index time, not at crawl-plan time. The check is cheap; retrofitting trust is not.
• Robots state cached, with deliberate 404 and unreachable behavior. Revalidate daily and honor Cache-Control, following common crawler practice.
• Purge-on-preference-change. An ingest ledger with robots digests, plus a tested path to delete what a policy change withdraws.
• Provenance logging. For any document, be able to say under which policy version it was collected.
• A public policy page. What you fetch, what you keep, how long, and how to ask you to stop.

Ship these six and you are ahead of most of the market, including some very large participants.

Failure Modes and Edge Cases

Most broken AI training preferences robots.txt deployments trace back to a short list of mistakes.

• The comma-listed user agent.User-agent: *, GPTBot on one line does nothing useful. Each user agent gets its own line, and when several groups match your agent, only the most specific one applies, so a specific agent never inherits the * group's rules.
• Orphan directives. A Disallow placed before any User-agent line belongs to no group and is ignored. Directives follow their User-agent lines, always.
• CDN-cached robots.txt. Your policy change is invisible until the edge cache expires or is purged. Stack that on each crawler's own cache and a preference change takes days to reach every compliant bot. Purge on deploy, every time.
• Mid-crawl preference flips. A crawler that validated robots at 09:00 can legitimately keep fetching under that view for hours. If you need an effect measured in minutes, enforce it at the edge with user-agent firewall rules; robots.txt is the wrong instrument.
• License text contradicting machine signals. Pages that say "all rights reserved" while robots.txt allows everything send contradictory instructions, and machine readers follow the machine-readable layer. Make them agree, or accept that the robot layer wins by default.
• Your own robots.txt downtime. Per the RFC's failure semantics, a 404 flips every compliant crawler to allow-all, while sustained unreachability can silence them entirely. Monitor /robots.txt like a production endpoint, because it is one.

What to Ship This Week

Publishers

1. Pick one of the three policies above and deploy it.
2. Purge the CDN, then verify per agent with curl and access logs.
3. Add the shared-signal preference fields once your target providers document the names they read.
4. Decide whether TDM-Rep belongs in your stack. EU exposure answers that for you.
5. Calendar a quarterly re-check of the participant list, because it will move.

Agent teams

1. Name your crawler and publish its policy page.
2. Put a robots gate behind every fetch, with deliberate 404 and unreachable behavior.
3. Set your cache TTL and honor server Cache-Control.
4. Stand up the ingest ledger and the purge path, then test both.
5. Write the one-page policy document enterprise buyers will eventually ask for.

The shared signal gave publishers one line that four companies promise to honor, and gave agent operators an etiquette bar that is fast becoming the price of enterprise trust. Neither side gets control. Both sides get the first entry in a policy stack, and this week is early enough to ship yours.