Cloudflare said on October 15, 2026, that signals from a participating cohort representing about 1% of its total network had helped OpenAI discover and index hundreds of millions of previously unknown web pages. The company described the figure as an early result from its research pilot, which it announced with OpenAI in July.
What the pilot is designed to do
Cloudflare’s July 8 announcement named content freshness, traffic quality, and actual page changes as examples of network signals the pilot would explore. The idea is to help a search crawler find relevant pages and spot when they may merit another visit.
The companies have separate roles: Cloudflare supplies signals that can help with discovery; OpenAI’s crawler finds pages, and OpenAI independently evaluates and ranks them for relevance and helpfulness, according to Cloudflare’s October update. Cloudflare also says its filtering can screen out nonexistent URLs, redirects, and duplicate copies before a crawler fetches a page. ChatGPT Search can link users to relevant original sources, the company says.
Cloudflare said the next phase would focus on better page selection, faster change signals, and measuring outcomes after discovery. Newly found pages can initially increase crawling while OpenAI evaluates and indexes them, so discovery alone does not mean that crawl activity has already fallen.
Separate controls for website owners
The research pilot is distinct from Cloudflare’s AI Crawl Control product. Its official documentation describes tools for viewing AI crawler activity, setting allow or block rules for individual crawlers, and monitoring compliance with robots.txt, the standard file websites use to communicate crawler rules. Cloudflare says AI Crawl Control is available on all plans; pay-per-crawl is in private beta.
Cloudflare also says the pilot uses web metadata for search and discovery rather than sharing page content for model training, and that a signal about a page does not grant permission to access it. The company is testing page-level signals for potentially harmful pages, such as a compromised page on an otherwise legitimate site.
