For 30 years, the world wide web has run on a surprisingly profound social contract: most sites are free for search engines to access, but if you use their content, you give credit by linking to the source.
Recently, that social contract has begun to collapse. Artificial intelligence (AI) tools are crawling sites not to link to them, but to train models and generate answers (which may or may not be accurate).
When you search for something, ChatGPT’s response or Google’s AI Overviews may still include links to sources, but they’re a kind of optional extra to the main answer.
This has triggered a bad dynamic for website owners, the public, and even AI companies themselves: as websites lose traffic (and revenue), many are beginning to block AI scraping tools, meaning AI results depend more on low-quality websites (many of which are also generated by AI). As a result, good information can be harder than ever to find.
How we got here
In the early days of the world wide web, search engines and content creators came to an agreement about crawling (the practice of technologically examining a site to index it, so it can be served up in search results). Content creators would provide access to their sites for free, and even allow search engines to reproduce small snippets of text.
In return, search engines provided links to the sites owned by content creators, who benefited from that web traffic. If content creators didn’t like the deal, they could prevent search engines from crawling their site with instructions in a file called robots.txt.
But if AI tools no longer provide web traffic, it cuts content creators out of the economic loop. There are also other costs associated with each visit to a website, so AI crawling can cost website providers money while not giving them any of the ad or other revenue that would come from human traffic. AI crawlers also crawl more deeply and more intensely than traditional web crawlers, magnifying that cost.
This change in traffic patterns isn’t a small or hypothetical problem. Cloudflare, a web hosting and service company that manages 30% or more of the top 10,000 sites on the internet, estimates over half of all web traffic is now AI bots.
Some of this will be AI agents supervised directly by people, but the majority will be crawlers. Site owners can use robots.txt to ask AI crawlers to stay off their sites – but some AI companies may ignore this polite request.
If the AI companies do honour the request, that can create a different problem. Sites containing misinformation are far less likely to ban AI crawlers, so the AI answers won’t be informed by high-quality sources.
What’s happening in the short term
On the horizon is an event dubbed “Google Zero” – the day when through-traffic from Google drops to nothing. While some grey-haired diehards (like one of the authors of this piece) might still click through to verify AI answers, this traffic is rapidly dwindling, as a direct result of AI summaries.
A study of Wikipedia confirms this, showing that traffic in the English-language version of the site dropped off quickly with the launch of AI summaries on Google in English, and that the same pattern occurred in other languages as AI summaries were rolled out. Never having to click through to get an answer might seem great for information seekers, but the reality is more complex.
Many sites are now blocking AI crawlers altogether. Site owners who decide to block AI crawlers are less likely to be linked in AI Overviews answers, even when the AI tool can still access the content to ground its answers (using a technique called retrieval-augmented generation).
Alternative “pay to crawl” models have been suggested as a way to compensate content creators, but haven’t gained traction.
Come September 15, Cloudflare sites will block AI crawlers by default on pages that contain advertising (and therefore make money for content creators).
This means up to 30% of the world’s top sites will no longer appear in Google AI Overviews summaries. It also means that much of what AI is being trained on will itself be AI-generated text.
What it means for you
So what does this mean when you’re looking for information? The quality of AI summaries is likely to go down, at least in the short term, while the new economics of the web get sorted out.
This will happen for two reasons. The first is that high-quality content is less likely to go into those AI summaries – one recent study found that already, around 1 in 6 sources used by AI search tools is itself an AI-generated website.
The second reason is that, as AI models are trained on more AI text, their output may degrade (a phenomenon known as model collapse).
As a result, search engines that depend less on AI may become more reliable. The challenge is finding one that doesn’t use an AI-based crawler. They do exist: ZDNet recommends Mojeek, PCMag recommends Brave, and Ban the Bots lists several, including one specifically for “small producer” content such as blogs.
For now, whatever search engine you’re using, the best thing you can do is to scroll down and click on some actual search results. This benefits content creators and is also more likely to give you more accurate information.
- is an Associate Dean, Interaction, Technology and Information, RMIT University
- is a Senior Lecturer, School of Computing Technologies, RMIT University
- This article first appeared in The Conversation
