Making it easier for websites to express AI training preferences
OpenAI has announced that it will honor a shared robots.txt signal designed to let website operators communicate their preferences about AI training to participating providers. The announcement names no launch date, no list of other participating providers and no technical specification beyond the shared-signal concept.
The mechanism builds on robots.txt, the long-standing plain-text file that sites publish at their root to tell crawlers which paths they may visit. The new use extends that convention from access control toward preference signaling: rather than only saying where a crawler may go, a site can express how its content may be used for training. In practice, a signal like this is only as strong as the number of crawlers that read and respect it, which is why the phrase "participating providers" matters.
OpenAI's commitment covers the signal itself; the announcement does not describe enforcement, verification or what happens to content already collected. This suggests the practical effect depends on adoption across both publishers and AI developers, and on whether sites can express nuance such as opting out of training while still allowing search indexing.
Why it matters: publishers gain a standardized place to state training preferences, and developers building crawlers or content pipelines may need to parse and honor the signal. For product teams, whether it becomes a dependable compliance checkpoint rather than a courtesy depends on how widely providers adopt it, which is an inference rather than a stated fact.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
OpenAI will honor a shared robots.txt signal to help website operators communicate their AI training preferences to participating providers.