Cloudflare's latest AI rankings expose the web's biggest free rider

At the start of every quarter, I check in with Cloudflare to see how the biggest AI companies are treating the web. The latest data tells a familiar story: Strip-mining continues apace.

For the week of July 1 through 7, the latest "crawl-to-refer" data shows Anthropic remains the biggest outlier. The AI company's bots crawled webpages about 2,800 times for every one referral sent back.

OpenAI was far behind, followed by Perplexity in third. Then, there's another big drop to Microsoft, followed by Google.

DuckDuckGo is one of the only players that comes close to offering a fair balance between taking from the web and giving back, with three scrapes for every one referral, according to Cloudflare data.

For Anthropic, there was an improvement from roughly 8,800-to-1 in early April. A quick check of other recent seven-day periods, though, shows Anthropic is still scraping like crazy.

During the first seven days of May, for instance, the company's bots scraped sites 24,700 times for every one referral. (Maybe some of its bots went on summer break in early July?)

Cloudflare's metric measures how often AI company bots ask to crawl webpages compared with how often their services send users back to those sites. The figures are an important proxy for whether AI giants are sustaining, or undermining, the economic bargain that historically powered the web, where sites allowed free crawling in exchange for traffic.

In the new generative AI world, this deal is breaking down. Now, AI answer engines and chatbots give users direct answers, making people less likely to visit the websites that created the content in the first place. That reduces the financial incentive to put high quality content on the web.

Anthropic promotes itself as the most ethical AI company, but this data is a reminder that what's ethical or not is sometimes in the eye of the beholder. I'll leave you to decide.

In recent months, Anthropic has criticized rivals for using its AI model outputs to develop and improve their own models. It calls these techniques "distillation attacks" and says this is against its terms of service.

Step back a bit, and this begins to look a lot like what Anthropic has been doing to websites: Collecting their content, often against content owners' wishes, and using it to develop or improve its own products.

Welcome to the modern web, Anthropic. You may have to get used to your outputs being used in ways you don't like.

Note: Anthropic has previously disputed Cloudflare's methodology, saying it could not verify the company's calculations and arguing that new search features are increasing referrals.

Sign up for BI's Tech Memo newsletter here. Reach out to me via email at [email protected].