Cloudflare and OpenAI: Making AI discovery better for everyone

Cloudflare published early results from a joint research pilot with OpenAI that began in July 2026, testing whether signals gathered from Cloudflare's network can help OpenAI's search crawler find pages it would otherwise miss. The pilot ran across a participating cohort representing roughly 1% of Cloudflare's total network. According to Cloudflare, that slice was enough for OpenAI to discover and index hundreds of millions of new web pages, which now serve as additional sources ChatGPT can draw on when answering questions.
The division of labor matters. Cloudflare supplies discovery signals — clues about which URLs exist and are worth fetching — while OpenAI independently evaluates and ranks how those pages appear, judging them on relevance and helpfulness. Cloudflare describes the arrangement as a feedback loop intended to improve both how much of the web gets discovered and the quality of what people ultimately find.
The pages in question tend to be the long tail rather than the head. Cloudflare's framing is that the web is full of specifics: the manual for an exact appliance model, current passport requirements and fees, a restaurant's latest menu and hours. Those pages can be valuable even when few other sites link to them, which is precisely the situation traditional link-based discovery handles poorly. Cloudflare contrasts this with the earlier web, where discoverability meant ranking highly on a search results page and a human decided what to do next. In an AI-mediated setting, a system has to locate the source, judge whether it is useful, and then connect the person to it through an answer, a citation, or a visit to the site itself. ChatGPT Search links back to relevant sources, giving users a path to the original content and the publisher behind it.
Cloudflare says the pilot also produced lessons about what makes a page worth crawling at all. Automated traffic can make a nonexistent page look interesting, for example: bots repeatedly request files that are not there. Some sites redirect nearly any address to their homepage, and other addresses resolve to copies of the same content. Detecting these patterns lets the crawler skip work before fetching a page, which reduces wasted effort and, Cloudflare argues, benefits site owners by cutting unnecessary load.
A second line of testing concerns safety signals that flag potentially harmful pages. Cloudflare's example is a legitimate site with a single compromised page, such as a fake login form planted by an attacker. Flagging that individual page rather than the whole domain is meant to protect users without penalizing the site owner. Cloudflare says early results suggest these signals can identify pages that should be kept out of an index, though it characterizes this as testing rather than a shipped capability.
On efficiency, Cloudflare points out that keeping current by refetching every page more often would be expensive for everyone. A crawler that repeatedly downloads an unchanged page adds load for the site and yields nothing. Filtering keeps crawlers away from pages that do not exist or that redirect elsewhere, while change signals indicate when a page is worth revisiting and when it is not. Cloudflare acknowledges a trade-off: discovering new pages initially increases crawling while those pages are evaluated and indexed. The stated goal is to keep discovering and stay current while reducing the total work required. For Cloudflare customers that means less unnecessary load; for OpenAI it means search resources go toward information that helps people.
Cloudflare also situates the pilot alongside OpenAI's broader product direction, noting GPT-6 and an "Intelligent UI" effort aimed at more visual, interactive ChatGPT experiences for shopping, learning and answers. The discovery work is presented as complementary to those advances rather than part of them.
The next phase, per Cloudflare, will focus on improving page selection, testing faster and more reliable change signals, and measuring what happens after content is discovered. Cloudflare says it will continue working with OpenAI to understand whether better site discovery translates into real business outcomes.
Two governance points are stated explicitly. A signal about a page does not grant permission to access it — domain owners' choices remain in force. And this research uses metadata about the web for search and discovery rather than sharing page content for model training. Cloudflare points site owners to its AI Crawl Control tool for understanding and managing how AI systems access a site.
Why it matters: Publishers and site operators now have a clearer picture of how AI search discovery is being wired up — network-level metadata feeding a crawler that OpenAI still ranks on its own terms, with indexing gains concentrated in pages that few sites link to. For developers and product teams, the practical levers named here are crawler management (AI Crawl Control) and the distinction between discovery metadata and training data, which is where permission and access decisions get made. Whether indexing translates into traffic or revenue is explicitly still an open question that Cloudflare says the next phase will measure.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
How Cloudflare and OpenAI are working together to help AI find useful content, keep up with changes, and make fewer wasted requests — making the experience better for users and website owners alike.