Everyone’s holding their breath and waiting for the AI bubble to pop. It will. A lot of what we’re seeing right now isn’t real innovation; it’s a frantic, FOMO-driven reaction to accessible tech and lowered barriers to entry. It feels a lot like 1999 all over again.
Think back to when the dot-com bubble burst. At the time I was working at Jupiter Media Metrix, measuring the meteoric rise of the internet high fliers such as Excite, Kozmo, Webvan, and others. These companies were delivering on the promise of the internet, and in March 2000, they saw their futures crumble. As a measurement firm with a long list of newly insolvent entities as clients, Jupiter Media Metrix also became part of internet history.
That doesn’t mean that the companies that disappeared in the internet's heyday had bad ideas; Kozmo is a later-day DoorDash, Webvan looked a lot like Peapod does today. Rabid innovation is fueled by great ideas, but when FOMO and speed to market become the most important thing, financial stability gets left in the dust. Conversely, other players not only survived the dot-com bubble bursting but have since thrived (Amazon, Microsoft, Oracle, etc.). In retrospect, you might argue that the dot-com survivors saw the internet as an enabler of a bigger idea.
So, it’s no surprise that many of the management teams at today’s big insights firms stand flat-footed when looking at the current AI-driven innovation happening in research. With substantial financial expectations (Kantar looking for an exit, IPSOS focused on growth, Nielsen refocusing on ratings), the motivation is to focus on using AI to solve core operational challenges first. Taking needless work out of the system is more important than adding new complexity. However, this practical approach introduces a fresh problem; it creates an AI innovation gap.
What do I mean by the AI innovation gap? Artificial intelligence is a technology that has implications across multiple parts of the insights pipeline. Yet, few if any vendors can implement those innovations in parallel across their key processes. This inability to execute on innovation across multiple pathways results in gaps in what an insights company needs to do and what its customers are expecting them to do.
Let’s break it down.
Automation & Optimization:
Modern insights firms lead with operations. Servicing the largest brands in the world comes from being able not only to coordinate multiple market studies amongst different cultures but also from the ability to pull that data together into a cohesive single-source solution. I’ve been at big insights firms when RPA (robotic process automation) failed. The challenge to automating research turned out to lie in the variability of inputs and outputs. The number of exceptions to a standard process that required human oversight exceeded the value of the technology. We resolved that the only way to substantially deal with automation would be in the future through an AI lens. Well, the future is now.
Beyond just automation, the same efforts lead to optimization. As an example, one of the AI projects I worked on redefined the price curve for studies on a client by client basis given the historical performance of accounts. With many clients providing specifications for studies based on creative guesswork, we knew full well that some studies tracked closer to the priced parameters and some didn’t. Deploying a simple machine learning model brought estimated prices much closer to reality, improving the forecasts and stabilizing profitability.
Consumer Experience:
How many of you out there are members of consumer panels? How many of you take surveys or scan receipts on a weekly basis? I’d venture few to none. The reality of the consumer experience in the insights field is that the value proposition sucks. It’s downright horrible, torturous, and soul-sucking. No one wakes up wanting to take a 10-minute survey on low-fat yogurt, let alone a 20-minute survey. Before I had the job of running a consumer panel, the experience was out of sight and out of mind. Corporate rewards and promotions link to client performance and not to how we treat the consumers who create the data we use (see my previous article on consumer panels).
Designers create surveys around the client’s data needs, not for consumer enjoyment. The insights world somehow missed the entire UX revolution of the last decade. We were too busy debating the merits of a 4-point versus a 5-point scale to notice. Surveys today are still among the poorest designed experiences you’ll find out there. A fact on display to every consumer panel respondent who’s completing surveys across multiple research vendors.
We are now seeing an AI-driven evolution of the consumer experience. Companies focused on consumers are few, not to mention the client's motivation is in retaining their Likert scales, but the evolution is well underway.
Data & Insights Synthesis
I’ve spoken about synthetic data in the past, and it is a polarizing topic. But without a doubt, using advanced data methods to take partial data and convert it into meaningful insights is something that holds promise. What makes tools like ChatGPT (i.e. LLMs) work is the ability to make smart predictions on how to complete a sentence. This prediction ability has gone from mediocre in 2023 to mind-blowing in 2025, with many of the AI company leaders claiming the tech to be on track to AGI intelligence by 2030.
Companies exist today that use public data or limited survey data to synthesize insights, consumers, and reports. AI researchers looking for a leg up on the competition are putting a renewed focus on building new models that mimic the human brain and think like we do. It’s a matter of time before this becomes a meaningful affordable technology to help answer more questions for the brands that have limited research budgets.
Storytelling & Discovery
Finally, we have the storytelling and discovery-driven AI innovation. If you haven’t yet listened to an AI-generated podcast (audio overview) from NotebookLM, pause now and go take a listen to the future. The reality of the insights industry is that its primary output is a syndicated data dashboard that three people look at, or a PowerPoint presentation that gets filed away and never seen again. Valuable insights and findings are often siloed and under-leveraged across the organization. Couple that with the large share of research projects that end up being checkbox exercises, research that’s completed to validate performance but rarely used in the organization for future value creation.
Innovation in storytelling and discovery is what we see daily with the current crop of large language models on the market. Google is already releasing training courses on how to leverage NotebookLM for marketing research. Both Anthropic’s Claude and ChatGPT are getting better and taking in data from documents and running a full analysis. Google rolled Gemini into Chrome, and Microsoft has been working on improving CoPilot for the office suite. Using AI to augment insight generation skills is becoming commonplace among practitioners in the insights industry.
While these technologies are massive improvements for the industry, we’re still in the early days. LLMs of today are fantastic yet blunt. We’re not yet at the stage that we trust AI with our credit cards to book travel but love the itineraries it creates. Similarly with insights, we’re augmenting with AI, not using it to do the end to end storytelling and discovery. This technology will get better, and more specific tools tuned to the needs of the insights industry will emerge.
The Gap
Some will consider this heresy, but in the world of insights, AI is a distraction.
Most insights firms don’t have the time, resources or ability to innovate with AI in these areas simultaneously. In fact, many of the smartest people I know prefer innovating in methods and approaches rather than chasing a one-off AI feature. Methods and approaches are what clients pay for and what drives the P&L. Everything around that is fluff. Sure, some fluff helps with the sale: faster turnaround, better quality respondents, more data, fancier dashboard, etc., but the value the client is buying is in the expertise and underlying methodologies.
This is where it falls apart. Everyone is pulling in a different direction.
The insights agencies and panel companies? They’re staring at the huge operations numbers in their P&Ls and thinking, “How will we use AI to do more with less?” Their focus is squarely on Automation & Optimization. It’s not sexy, but it’s where they will save millions.
Meanwhile, the clients, also known as the ones writing the checks, are getting dazzled by demos of AI creating podcasts, beautiful narratives and uncovering hidden gems. They want better outputs and smarter Storytelling & Discovery. They don’t care about the vendor’s operational costs; they care about getting answers that make them look smart in the next board meeting.
Mid-tier insights agencies are attempting to keep up with the larger firms, and for them, Data & Insights Synthesis is a cheat code to put them into the big leagues.
And the poor consumer stuck in the middle? They’re praying that AI will make the data collection experience less of a nightmare. They’re desperate for a better Consumer Experience.
Hopefully, you see the gap? The people paying for the work and the people doing the work want different things from AI. Vendors want AI to fix the biggest limiter on their valuation, while clients are expecting it to reinvent the insight space.
The Bottom Line
The legacy insights firms are making a calculated bet: use AI to cut costs now, get the P&L healthy, and then buy their way into cool, client-facing innovation later. It’s the classic “survive, then thrive” strategy. A strategy proven to be successful in today’s most successful companies.
It’s also a risky gamble. It leaves the door wide open for a new generation of AI-native startups to swoop in and steal clients by focusing on what they want. A classic innovator’s dilemma.
The question isn’t whether the legacy firms will save money with AI. It’s whether they’ll still be the biggest names in the industry by the time they’re done optimizing.
What do you think? Is the industry playing it smart, or are they breeding a faster horse?