AI changes Search - lifeblood of today's internet

Matthew Prince, the co-founder and CEO of Cloudflare was interviewed by Amol Rajan, a BBC journalist here. (This podcast may not be available outside of the UK).

It was a very interesting discussion with someone who is really thinking hard about the future possible business models of the internet with the existence of mass AI.

Matthew carries some clout as Cloudflare acts a significant gatekeeper in between internet users and content hosted on the internet. They respond to 81m internet requests per second and manage 20-25% of internet traffic and see 6 billion people on the internet (ie the global population). They see themselves as one of the guardian of the internet.

Existing Search is like a treasure map 

Existing search is treasure map - it's not precise, a user still had to search around on the destination page to discover the answer to their question. They might click on multiple links and cross reference the information elsewhere, but each user views multiple pages, which is the lifeblood of the internet.

AI is an answer engine

Now AI creates an answer engine - there's no intermediary and there is much fewer page views. There's no digging and no critical analysis. (And as a side note, there is less critique of the AI's response - it is taken as fact, with no thinking!)

Crawls per visitor

The historical 'deal' is that search engines could copy the content on any page so that the content creator's page appears on the treasure map. And in return, the search engine would return traffic back to the site from the search results.

• 10 years ago, the ratio of crawls to page view was two crawls for every visitor received. 
• Today, that ratio has shifted to 20 crawls to 1 visitor, due to AI overviews, now embedded at the top of the results from the big search engines.
• For OpenAI, that ratio is 1,500 crawls to 1 visitor
• For Anthropic, it is 40k crawls per visitor!

So yesterday's business model now longer works. And content creators need a new deal.

The future scenarios

Scenario 1: Content creators wither and die

But Matthew thinks that's unlikely: answer engines need new and constant evolving content - this is the fuel that drives the internet

Scenario 2: Black Mirror scenario 

There is a cabal of (say) 5 major powerful AI engines + their family of content creator groups. The revenue from the AI engines can afford to pay for content creation for their AI engines. 

Matthew shudders at this dystopian outlook, as it over concentrates power in these behemoths. 

Scenario 3: Traffic isn't the metric that matters. 

A new value exchange will arise using a new metrics. That hasn't been discovered yet.

Internet Content is like Swiss Cheese

Knowledge on the internet is kinda like Swiss cheese - mainly solid, but with some holes in it. And people will get compensated for filling in the holes. 

And yes, there precedent for this: there is a guy that makes €40m from picking up all the Spotify search results which didn't return much content. This guy then creates music for these unfulfilled requests eg music with disco beat about dancing with your cat.

Matthew's solution

The existing search deal continues, but the Search Engines are forced to crawl a site twice: once for search and once for AI. Their crawlers are clearly labelled, so that content creators still benefit from search visitors, but they can block AI crawlers. 

Presumably both side await a new compensation arrangement between AI company and the content provider - a model that hasn't been found yet.

Prompt engineering - how to do it better

In my previous post, AI Code Generation to upend Product Management, I discussed the importance of prompt engineering. 

Thanks to a recent edition of Lenny's Podcast, here are some recommendations about how to most effectively deploy prompt engineering

Tip 1. Examples!

As well as instructing the AI for your desired outcome, provide examples as well. 

Tip 2. Decomposition

Decomposing a project into smaller tasks first is much more effective net-net. ie 

Step 1. 'Tell me the significant steps required to build XYZ'

Step 2. Review those steps

Step 3. 'Now use those steps to build XYZ'

is more effective than simply 'Build XYZ'.

Tip 3. Self Criticism 

Asking the AI to critique the results that you have previously requested (self-criticism) can lead to smarter, more accurate outputs ie between Step 1 and Step 2 above, add this step:

Step 1b: 'Please analyse and critique the steps that you have just outlined.'

AI code generation to upend Product Management

In the last week, I had a fiddle with Lovable and Bolt – two very exciting AI Code generation engines – and, well, I loved ‘em!

The astonishing growth of AI Code generation engines

Loveable	$10m ARR in 60 days with 15 people (listen to Lenny’s podcast with the founder, Anton Osika)
Bolt	$40m ARR in 5 months with less than 20 people (listen to Lenny's podcast with the founder, Eric Simons)

My definition of an AI code generation engine

They use generative AI to take some requirements and to squirt out fully functional code – right into your lap. People have been using ChatGPT to help them correct their code or to hook up APIs. Lovable and Bolt punch through to a new ceiling, IMHO.

lovable - What I did

In 20 minutes, I typed in a specification for a wizard consisting of 6 webpages to achieve an objective. It was about three quarter of a page long with lots of headings and bullet points. I cut and paste this into Loveable’s super simple prompt.

The spec was reasonably clear to someone with my industry, but it definitely needed context. 

What happened next

There was some processing. And then, a stream of code appeared: first the JavaScript, then the style sheets and finally the HTML of each of the six pages.

When the dust had settled, there was a preview button. I can’t say this button worked first time. I had asked for the results to be delivered as a zip file (not knowing that there was preview functionality). It tried to render the zip file in a browser and failed. My bad.

After a bit of tinkering with the spec, I bashed in my v2 instructions. And BANG – a functional webpage, nicely styled, with sensible prompts, populated with sample data.

Whoa! Ready to publish? Do you want to purchase a domain? Are you ready to launch your new business??

Totally lovable – hence the name.

Bolt.app

Bolt was even more impressive. Here's a better review than I can do: Bolt AI: Build, Fix, and Deploy Full Stack Apps Faster Than Ever!

I had my specifications in PowerPoint because I had previously built some sample layouts to explain my wants to a developer. Consuming PowerPoint was a fail. So I converted a table in PowerPoint with sample data into text that said, “The table should have the following fields: …..”. (Another learning point.)

You can specify the programming language and / or tech stack that you’d like to generate. I created a supabase account – an online database – and hooked it up to my Bolt app. I was also obliged to purchase 20M tokens per month for $20. 

What happened next

Database instructions appeared. Would I like to put these into supabase? Yes, please. Boom – tables substantiated. 

Code generated next. Preview. BOOM – an application appears. When I interacted with the app, I could data appearing in supabase in another browser window. Corks - this is for real! I could edit data in the tables, create new rows, all of which was immediately visible in the app. Wonderous!

All for $10, possibly even less if I hadn't spent so much time messing around with PowerPoint fails!

My conclusions so far and some crystal balling

This tech is perfect for green field projects: super rapid prototyping of concepts and ideas. Next up will be enhancing existing tech stacks with improvements. This will require deep integration with github and the implementation of a corporate knowledge library (design considerations, technical mandates (use this tech or we have a license for this technology) or tech standards and red lines) or industry-speak or corporate euphemisms. 

What WILL happen next to Product Management, Engineering and Testing

Well, very interesting indeed. Developers are still the all-powerful supremos in any tech company – they were the highest-paid and the rarest resource. Their pedestal has been shortened somewhat. They are still essential: it is plainly obvious that the output of these tools won’t work perfectly: prior integrations exist already. Security & maintainability are critical. 

BUT profound understanding of the business requirements and deliverables have always been important. This code-gen technology makes that even more important if, in 5 minutes, you can build your own deliverable. Product Management will change – for me it already has.

New Role: Product AI Gen Engineer

Cast forward, I think there will be a new role generated in the product development organisation: I haven’t got a good name yet, but Product AI Gen Engineer, perhaps?

These people are a combination of Product Manager, Business Analyst, Prompt Engineer and Tester. Let’s break down the job description.

Business Analyst

An excellent understanding of the requirements of the functionality and what the delivered results should look like. Clear unambiguous specifications have never been in greater demand. From my own experience with Lovable, then the ambiguity or a critical typo (even in my case, formatting of bullet points!) really made a difference to the net result. 

Prompt Engineer

This is a new skill that dovetails the business requirements with experience of using a particular AI tool: knowing how to surgically correct instructions without messing up everything else. It does come back to unambiguity. 

Do you remember when Internet Search became available: some people became experts at entering search terms to find the right information: it was a balance of specifying the topic of interest with distinctive keywords which would return the result set in which the answer to your question could be found. It will be the same with these AI code generating skills. 

Acceptance Tester

There will be a very tight cycle: requirements, delivery, review, respecify. Functional Acceptance testing will be a skill that needs to be glued onto to the end of Product Manager skillset. AI generated functionality, from I can see, delivers lots of common sense: there’s field level checking, but it is the functional flow through multiple steps where the AI is likely to brittle – and often as a result of ambiguity in the specifications. 

Conformity / Continuity Validations: Given the sensitivity of the delivered results to the specifications, there will be a renewed emphasis on making sure that, towards the end of the development cycle when minor tweaks are being asked for, that it is only the tweaks that are changed and not unanticipated knock-on effects. Code modification analysis between each build becomes really important. Bolt has this, but this, in itself, requires AI to interpret. Regression testing is critical. 

AI can be used for these two tasks (code generation and testing), but I assume that you will want to use two different AIs to do these two tasks, so that you don’t get over-fitting between developer and tester.

CONCLUSION: Very exciting