Google announced the launch of AI Overviews, its generative artificial intelligence-fuelled search feature, in May 2024. Initially named Search Generative Experience, AI Overviews offers Google users AI-generated answers by sourcing and summarizing information from different websites.
These AI responses are positioned at the top of the page for immediate visibility. The aim is to improve user experience by providing an alternative and more straightforward way to access information while enhancing the relevance of search results.
This feature has slowly been offered to the public, having initially been made available exclusively in the United States. AI Overviews is available worldwide and has been rolled out to more than 100 additional countries, including Canada.
AI Overviews represents a key effort by Google to capitalize on the rapid emergence of generative AI technology amid fierce competition between AI-enabled search engines in the market. It’s a direct response to Open AI’s SearchGPT and Microsoft Bing’s Deep Search, which is powered by OpenAI’s ChatGPT.
At the same time, conversational AI chatbots like ChatGPT, Microsoft’s Copilot and Google’s Gemini continue to resonate with users worldwide. Google’s investment in AI is critical to its ability to stay in the AI race among the other tech giants.
Concerns about AI-assisted search
The implementation of AI Overviews, however, has raised concerns among businesses, website managers and online advertisers.
Critics worry this feature could decrease traffic to their websites if users were to rely too heavily on AI Overviews and ignore the links to websites displayed in the search results.
Paid advertisements and sponsored content play a pivotal role in the revenue streams of companies and website operators. If traffic to websites diminishes, the incentive for these companies to invest in these advertising formats could decline, potentially disrupting the multi-billion-dollar online advertising industry.
To better understand this, we conducted a study at HEC Montréal’s Tech3Lab to investigate the potential impact of search generative AI features like AI Overviews on user perceptions and behaviours compared to those associated with regular online search queries.
User search behaviour and perceptions
We developed a set of four Google search scenarios, either AI-assisted or non-AI-assisted. The two AI-assisted scenarios included an AI-generated overview at the top of the search results, while the two non-AI-assisted scenarios consisted of a regular Google search experience.
For each scenario pair, participants performed a search for informational purposes and another search related to a product purchase. During each search task, users’ click behaviour (the number of clicks), cognitive load (the mental effort required to process information) and visual attention were measured.
We used pupillometry — measuring pupil size and reactivity — and analyzed screen recordings to track these metrics. After completing each task, participants shared their perceptions through questionnaires.
Through this experimental approach, we were able to achieve two goals. First, we determined whether AI-generated overviews in search results significantly impact user perceptions of the relevance, usefulness and satisfaction with search results.
Second, we observed whether such generative AI summaries significantly impacted user behaviour in terms of the number of clicks on links appearing in the search results. This provided insight into the potential impact generative AI summaries might have on organic search traffic.
What did we find?
The results of our study suggest that the presence of AI-generated overviews has no significant impact on user perceptions of relevance, usefulness and satisfaction with search results. There was also no significant impact on the clickthrough rate — the ratio of clicks on a link — on links in the search results.
The presence of AI Overviews did not significantly reduce the users’ interaction with Google’s classic list of suggested pages. While this finding indicates AI Overviews might not lead to an immediate or significant shift in website traffic, it’s important to note that user interactions could evolve over time as people become more familiar with this new feature.
As such, the competition among the market leaders operating in the online search space will likely continue to intensify. For example, earlier this month, Google rolled out a new feature called AI mode that delivers additional AI-generated results for some users.
Tech giants like Google and Microsoft will continue to vie for dominance in the market, aiming to create more engaging, higher-value search experiences for their online users.
- is a Professor of Marketing and RBC Financial Group Chair of E-Commerce, HEC Montréal
- is a Full Professor / Professeur Titulaire, Department of Information Technologies, HEC Montréal
- is an NSERC-Prompt Industrial Research Chair in User Experience and Full Professor of IT, HEC Montréal
- is an Adjunct Professor, Department of Marketing, HEC Montréal
- This article first appeared in The Conversation
Co-researchers Alexander J. Karran, Thadde Rolon-Merette, Eugene Yuzan Guo, Fabien Poivré and Mehdi Benbousta co-authored this article.
