Agent-readiness: cheap basics over vendor theatre

Cloudflare scanned the 200,000 busiest websites to check if they are ready for AI agents.

The answer: barely. Which is no surprise.

Only ~4% had declared AI usage preferences in robots.txt. Markdown content negotiation (which helps agents "see" the site) was at a similar level, and newer agent standards were barely visible.

For thirty years, your website has had two audiences: people and the search engines that send people to you. Now, agentic AI is arriving on the scene. The AI agent does not browse your site the way a person does, or crawl it quite the way Google does. It wants clean, machine-readable content, and it can increasingly act, compare, and buy on someone's behalf.

While the pressure to get "agent-ready" is building, it is exactly where I would slow down.

Most of the urgency is coming from companies that sell the fix, and many of the new standards are not actually read by much yet.

Google has said plainly that you do not need special files, AI-specific markup or Markdown versions of your site to appear in its generative AI search features. A fair amount of "agent readiness" is putting up signposts that nothing is following.

And indicating a capability you do not have is worse than none. The agent tries the door, finds nothing behind it, and moves on.

So do the cheap, durable things and leave the rest. Let agents reach your content. Keep it clean, structured and current. Maintain your robots.txt and sitemap. Set sensible rules for the bots you do and do not want. None of that is wasted, whatever the standards settle on.

Any elaborate protocols, DNS changes and such can wait until something actually reads them, or until your platform really has something to offer an AI agent.

The agents are coming. But the cheapest way to be ready for them is to simply do the part that nobody is trying to sell you.