Matthew Prince, the co-founder and CEO of Cloudflare was interviewed by Amol Rajan, a BBC journalist here. (This podcast may not be available outside of the UK).
It was a very interesting discussion with someone who is really thinking hard about the future possible business models of the internet with the existence of mass AI.
Matthew carries some clout as Cloudflare acts a significant gatekeeper in between internet users and content hosted on the internet. They respond to 81m internet requests per second and manage 20-25% of internet traffic and see 6 billion people on the internet (ie the global population). They see themselves as one of the guardian of the internet.
Existing Search is like a treasure map
Existing search is treasure map - it's not precise, a user still had to search around on the destination page to discover the answer to their question. They might click on multiple links and cross reference the information elsewhere, but each user views multiple pages, which is the lifeblood of the internet.
AI is an answer engine
Now AI creates an answer engine - there's no intermediary and there is much fewer page views. There's no digging and no critical analysis. (And as a side note, there is less critique of the AI's response - it is taken as fact, with no thinking!)
Crawls per visitor
The historical 'deal' is that search engines could copy the content on any page so that the content creator's page appears on the treasure map. And in return, the search engine would return traffic back to the site from the search results.
- 10 years ago, the ratio of crawls to page view was two crawls for every visitor received.
- Today, that ratio has shifted to 20 crawls to 1 visitor, due to AI overviews, now embedded at the top of the results from the big search engines.
- For OpenAI, that ratio is 1,500 crawls to 1 visitor
- For Anthropic, it is 40k crawls per visitor!
So yesterday's business model now longer works. And content creators need a new deal.
The future scenarios
Scenario 1: Content creators wither and die
But Matthew thinks that's unlikely: answer engines need new and constant evolving content - this is the fuel that drives the internet
Scenario 2: Black Mirror scenario
There is a cabal of (say) 5 major powerful AI engines + their family of content creator groups. The revenue from the AI engines can afford to pay for content creation for their AI engines.
Matthew shudders at this dystopian outlook, as it over concentrates power in these behemoths.
Scenario 3: Traffic isn't the metric that matters.
A new value exchange will arise using a new metrics. That hasn't been discovered yet.
Internet Content is like Swiss Cheese
Knowledge on the internet is kinda like Swiss cheese - mainly solid, but with some holes in it. And people will get compensated for filling in the holes.
And yes, there precedent for this: there is a guy that makes €40m from picking up all the Spotify search results which didn't return much content. This guy then creates music for these unfulfilled requests eg music with disco beat about dancing with your cat.
Matthew's solution
The existing search deal continues, but the Search Engines are forced to crawl a site twice: once for search and once for AI. Their crawlers are clearly labelled, so that content creators still benefit from search visitors, but they can block AI crawlers.
Presumably both side await a new compensation arrangement between AI company and the content provider - a model that hasn't been found yet.
