AI crawlers renegotiate the web's bargain (DataDome Q2)

AI crawlers are rewriting a deal the web has relied on for twenty years.

Every request imposes a cost on the website receiving it: bandwidth, compute and capacity. Traditional search crawlers justified that cost by sending people back. The crawler read your pages, indexed them and helped someone find you.

AI agents do not all make the same exchange, and DataDome's latest figures show the gap widening quickly.

Its network processed 17.7 billion AI-agent requests last quarter, up from 12.2 billion in Q1. But the volume is not the interesting part. Crawl activity and referral value are now moving in opposite directions.

Meta makes the point sharply. Meta-ExternalAgent, which collects content for purposes including model training, grew 74%. Meta-WebIndexer, which indexes the live web for current answers, grew 163%. Together they generated more than half the AI traffic DataDome recorded, while sending almost no visitors back so far.

ChatGPT shows the reverse pattern. Requests from ChatGPT-User fell 6%, yet referrals from ChatGPT grew 17% and accounted for between 80 and 88% of all AI-driven referrals each month.

The lighter footprint delivered most of the measurable return.

So the old assumption that being read leads to being found no longer holds by default. Some agents consume infrastructure and return visitors. Others consume infrastructure and return almost nothing.

Site owners can distinguish between them, but only if they are looking. That means identifying the agent, verifying that identity and deciding whether the access it wants serves the business.

Most websites still treat crawling as a binary choice: allow bots or block them.

It is no longer a binary choice. It is a commercial one.

The crawlers have already rewritten the deal. The only question is whether you accept it by accident or set your own terms.