The Internet has a second audience

Cloudflare reported that for the first time, more than half of the traffic on its network was not human, with AI agent requests growing by more than 1,700% over the past year. The company said it handled an average of 115 million HTTP requests per second as of its latest figures, up from 63 million at the end of 2024, with peaks above 150 million. It said agent traffic is now the fastest-growing segment of automated traffic, follows human routines such as a weekly rhythm and a summer holiday dip, and grows with how many questions people ask rather than how much a site publishes.

The company described declining human traffic in heavily crawled categories including Retail, Computer Software, IT & Services, and Financial Services, where it said human traffic fell as much as 40% in less than a year. It also said crawler requests for AI training rose from 22% of crawler requests in Spring 2025 to 52% by June 2026, based on crawlers' stated purpose.

On controls, Cloudflare said it replaced a single block AI bots switch in July with separate Search, Agent, and Training toggles available on every plan including Free, and that fewer than 1% of its sites block search crawlers while 17% block training. On September 15 it shipped Disallow AI Training, which keeps sites indexed for search while instructing operators not to use content for training; Apple, Google, and Microsoft have committed to honor it. Web Bot Auth, which lets operators including OpenAI, Google, and AWS cryptographically sign agent requests, sees more than 500 billion verified bot requests each week.

Cloudflare announced two products in beta. Pay Per Use pays publishers when content is actually used rather than crawled, with verified crawler buyers reporting each use that Cloudflare checks before billing the buyer and paying the publisher, and publishers able to opt in and out. Monetization Gateway, a closed beta for eligible U.S. Cloudflare customers, sets prices using the Rules language and returns HTTP 402 Payment Required via the open x402 protocol so agents pay sellers directly, pricing per request, query, or token at fixed or capped prices. Cloudflare said its own AI Gateway uses Monetization Gateway to let agents pay for inference. It also said Markdown for Agents and WebMCP are aimed at making sites easier for agents to read and act on, and cited a joint research project with OpenAI announced in July, plus reporting in August 2026 on an Agentic Internet it described as readable, discoverable, callable, and payable.

Why it matters: Sites that previously blocked all automation may now need to distinguish search crawlers, training crawlers, and agents, since the granular controls and paid access paths cover every plan, including Free, while the two monetization products remain in beta with unsolved pricing and discovery.