AI changes Search - lifeblood of today's internet

Matthew Prince, the co-founder and CEO of Cloudflare was interviewed by Amol Rajan, a BBC journalist here. (This podcast may not be available outside of the UK).

It was a very interesting discussion with someone who is really thinking hard about the future possible business models of the internet with the existence of mass AI.

Matthew carries some clout as Cloudflare acts a significant gatekeeper in between internet users and content hosted on the internet. They respond to 81m internet requests per second and manage 20-25% of internet traffic and see 6 billion people on the internet (ie the global population). They see themselves as one of the guardian of the internet.

Existing Search is like a treasure map 

Existing search is treasure map - it's not precise, a user still had to search around on the destination page to discover the answer to their question. They might click on multiple links and cross reference the information elsewhere, but each user views multiple pages, which is the lifeblood of the internet.

AI is an answer engine

Now AI creates an answer engine - there's no intermediary and there is much fewer page views. There's no digging and no critical analysis. (And as a side note, there is less critique of the AI's response - it is taken as fact, with no thinking!)

Crawls per visitor

The historical 'deal' is that search engines could copy the content on any page so that the content creator's page appears on the treasure map. And in return, the search engine would return traffic back to the site from the search results.

• 10 years ago, the ratio of crawls to page view was two crawls for every visitor received. 
• Today, that ratio has shifted to 20 crawls to 1 visitor, due to AI overviews, now embedded at the top of the results from the big search engines.
• For OpenAI, that ratio is 1,500 crawls to 1 visitor
• For Anthropic, it is 40k crawls per visitor!

So yesterday's business model now longer works. And content creators need a new deal.

The future scenarios

Scenario 1: Content creators wither and die

But Matthew thinks that's unlikely: answer engines need new and constant evolving content - this is the fuel that drives the internet

Scenario 2: Black Mirror scenario 

There is a cabal of (say) 5 major powerful AI engines + their family of content creator groups. The revenue from the AI engines can afford to pay for content creation for their AI engines. 

Matthew shudders at this dystopian outlook, as it over concentrates power in these behemoths. 

Scenario 3: Traffic isn't the metric that matters. 

A new value exchange will arise using a new metrics. That hasn't been discovered yet.

Internet Content is like Swiss Cheese

Knowledge on the internet is kinda like Swiss cheese - mainly solid, but with some holes in it. And people will get compensated for filling in the holes. 

And yes, there precedent for this: there is a guy that makes €40m from picking up all the Spotify search results which didn't return much content. This guy then creates music for these unfulfilled requests eg music with disco beat about dancing with your cat.

Matthew's solution

The existing search deal continues, but the Search Engines are forced to crawl a site twice: once for search and once for AI. Their crawlers are clearly labelled, so that content creators still benefit from search visitors, but they can block AI crawlers. 

Presumably both side await a new compensation arrangement between AI company and the content provider - a model that hasn't been found yet.

Advertisers spend £1 in every £4 on the internet

The Internet Advertising Bureau in the UK (in conjunction with PriceWaterhouseCoopers) publishes its bi-annual advertising spend study.

Here are some headline figures that made me think:

• Online advertising’s share hits 25%, as advertisers spend £1 in every £4 on the internet. That represents £4 billion!
• Social media boosts online display by 27.5% on a like-for-like basis to £945 million. 
• There was nearly 200% surge in display advertising in a social media environment (on a like-for-like basis) and 91% year-on-year (absolute growth) in video formats. 
• Paid-search continues to perform strongly with growth of 8% year-on-year on a like-for-like basis to £2,346 million, representing 57% of total online spend (61% in 2009).
• Mobile advertising has experienced a 'staggering' 116% year on year growth (on a like for like basis), up from 32% in 2009. (Arthur: don't get excited, it is only worth £83 million, so 'staggering' is over egging it!)

Other stats

• By December 2010, the UK’s active online user base had grown to 40.3 million. 
• 47% of at home UK internet users access the internet with a connection speed of between 2Mb and 8Mb. 14% access online with a connection speed of over 8Mb. (Source: UKOM APS/Netview December 2010).
• Social networks now account for 25% of the time spent online in the UK. 

Google does click to call for free

Google Maps has added Click to Call to their results. Briefly, as you browse Google Maps, you see a store that you want to call, you click on the 'Call' link and a dialog pops up. Enter your telephone number and press Call. Google calls your number first, when you answer, it dials the store, connecting both of you. More details here.

Here's an example of hardware stores in Seattle. I don't know how international this is.

The fascination is that Google is providing this service for free. It does, of course, give all stores / anything on Google maps a freephone number.

The opportunity for pranks is unending - you could besiege your next door neighbor anonymously. As for getting the local pizza company’s phone to ring off the hook with puzzled customers on the other end of the line ... well, you can see the possibilities for a malicious script....