Cloudflare and OpenAI Expand Web Discovery to Help ChatGPT Find More Useful Content

A joint research pilot has helped OpenAI discover and index hundreds of millions of previously unknown webpages, while testing ways to improve search relevance, crawler efficiency, and website control.

Cloudflare and OpenAI are reporting positive early results from a research collaboration focused on helping AI-powered search discover useful information across the internet. The initiative aims to make relevant website content easier for ChatGPT to find, while giving website owners greater control over how AI systems access their pages.

The companies began the research pilot in July 2026 to examine whether signals from Cloudflare’s global network could help OpenAI’s search crawler identify additional web content. According to the early findings, the initiative has helped bring previously undiscovered pages into OpenAI’s search index, expanding the information available to ChatGPT users.

For publishers and website operators, the work addresses an increasingly important challenge: ensuring that useful content can be discovered by AI systems, represented accurately, and linked back to its original source.

Improving Access to Valuable Web Content

The internet contains a wide range of practical information, from product-specific installation instructions and equipment manuals to updated government requirements, service details, and local business information. Such pages can be highly useful even when they receive few links from other websites.

As people increasingly use AI systems to find information and complete tasks, website discovery is becoming more complex than achieving a prominent position in traditional search results. AI systems must locate relevant sources, assess their usefulness, and connect users with the information they need through answers, citations, or links to original websites.

Cloudflare provides network-based signals that can help OpenAI identify additional pages for its search crawler to examine. OpenAI separately assesses and ranks the discovered pages according to their relevance and usefulness.

The participating group represented approximately 1% of Cloudflare’s overall network. Within this pilot, OpenAI has discovered and indexed hundreds of millions of new webpages, according to the companies.

By including links to relevant sources, ChatGPT Search can also help users visit original websites and discover the publishers and businesses responsible for the information.

The initiative complements OpenAI’s broader work on ChatGPT, including GPT-6 and Intelligent UI, which are designed to support more visual and interactive experiences for shopping, learning, and finding answers. Improved web discovery helps these experiences draw on a wider range of relevant and current online information.

Moving Beyond URL Discovery to Relevant Results

Discovering a webpage is only the first step. Search systems must also determine whether the page exists, whether it can be accessed, and whether it provides distinct information that may be useful to a visitor.

The research has identified several patterns that can reduce crawling efficiency. Automated systems may repeatedly request files that do not exist, while some websites redirect nearly every requested address to their homepage. Other URLs may lead to duplicate copies of existing content.

Identifying these patterns before fetching a page can reduce unnecessary crawler activity and help search systems concentrate on more useful destinations.

The companies are also testing signals designed to identify potentially harmful webpages. A legitimate website, for example, may contain an individual compromised page hosting a fraudulent login form. Identifying the specific page rather than treating the entire domain as harmful could help protect users without unnecessarily penalizing the website owner.

Early findings indicate that these signals can help identify pages that should be excluded from a search index.

Improving Crawler Efficiency for Website Owners

As AI-driven discovery grows, websites have an opportunity to reach audiences who are actively looking for their information. However, repeatedly crawling every page can increase server load without providing additional value, particularly when content has not changed.

Cloudflare and OpenAI are exploring ways to make crawling more selective. Filtering can help prevent crawlers from repeatedly visiting nonexistent pages, redirects, and other unproductive URLs. Signals about content changes can also help determine when a page needs to be revisited and when another fetch is unnecessary.

The process requires a balance between discovering new material and avoiding excessive requests. Newly identified pages may initially require additional crawling while they are assessed and indexed. Over time, better signals could help maintain discovery and freshness while reducing unnecessary activity.

For Cloudflare customers, this could mean less avoidable traffic from crawlers. For OpenAI, it could allow more search resources to be directed towards information that helps answer users’ questions.

Next Steps for the Research Collaboration

The next stage of the pilot will focus on improving page selection, testing faster and more reliable signals for detecting content changes, and measuring the outcomes after pages are discovered.

Cloudflare and OpenAI also plan to continue examining how improved website discovery can translate into meaningful business results for content owners and publishers.

Website owners’ access preferences remain important throughout this process. A signal identifying a webpage does not itself provide authorization to access that page. According to the companies, the research uses web metadata to support search and discovery rather than sharing page content for model training.

Website operators can use Cloudflare AI Crawl Control to understand and manage how AI systems access their websites.

The findings point to an emerging area of collaboration between network infrastructure providers and AI search platforms: improving the discovery of useful online information while making crawling more efficient and giving website owners greater visibility into how automated systems interact with their content.