When Good Management Leads to Failure
Lessons from The Innovator’s Dilemma
I’ve been thinking a lot about disruption lately, it’s hard not to. Popular media today is filled with stories about how AI is disrupting the world. But the most interesting thing about disruption is its ability to take down industry giants while they’re still posting record profits. The AI wave of disruption doesn’t appear to be slowing down and it appears that Insights is in the crosshairs.
Given all the talk about disruption I went back and reread Clayton Christensen’s The Innovator’s Dilemma. It’s one of my favorite business books, and if you haven’t picked it up in a while, or if you’ve only heard about it secondhand, there’s no better time to revisit it. The lessons the book provides are invaluable for any industry, but for a group close to home, its lessons are worth re-learning because the insights industry is living through its own disruptive moment. A fundamental question any established player in the space will have to answer is, are they walking the same path as the disrupted companies Christensen described?
The Paradox of Good Management
The unsettling part of Christensen’s work is that the companies that failed didn’t have idiots in charge. They were led by some of the best managers in the world. They listened to their customers. They invested in innovation. They focused on their most profitable segments. They did everything the business schools told them to do. And they still got steamrolled by innovation.
This happens because “good management” in a stable market becomes a liability when disruption hits. The practices that make for big company success: staying close to customers, chasing higher margins, focusing on substantial markets, are precisely what blind them to disruptive threats.
Lesson #1: Good managers do what makes sense, and what makes sense is primarily shaped by what creates value today. To ensure consistency in value delivery, managers build organizations that are meant not to change, or if they must change, to change through tightly controlled procedures. This means the very systems that make companies successful are hardwired to resist change.
The Disruptive Playbook
In his book, Christensen lays out a pattern that repeats across industries. Often, disruptive technologies don’t start by being better but are considered worse. They underperform on the metrics that mainstream customers care about. But they’re cheaper, simpler, smaller, or more convenient. And they appeal to a fringe market that the big players don’t care about.
Sound familiar? That’s exactly what’s happening in insights right now. Synthetic respondents, AI-generated personas, and AI-powered storytelling are all technologies easy for the industry veterans to dismiss. But they’re faster, cheaper, good enough and scale massively, something no traditional research does well.
A former colleague of mine, an incredibly smart researcher, told me that a customer who was using a synthetic model for an internal tool is , “Not what our customers want.” The irony is that he was right. The current customers don’t. But that’s the trap.
Lesson #2: Your most profitable customers generally don’t want, and indeed can’t use, products based on disruptive technologies. They offer a different package of attributes valued only in emerging markets remote from, and unimportant to, the mainstream, giving disruptors the opportunity to learn and gradually move up market.
Why Customers Mislead You
Here’s one of the most counterintuitive insights from the book: sometimes it’s right not to listen to your customers. Times when you should invest in lower-performance products that promise lower margins. Times when you should aggressively pursue small markets instead of substantial ones.
Every customer I have had while at an established business wants sustaining innovations, a better and cheaper version of what they already buy. I’ve seen customer advisory boards scratch their heads with wonder at why we’d pursue something they don’t want. They won’t lead you toward disruptive technologies. They can’t. They don’t know they need them yet.
This is the innovator’s dilemma in its purest form. Do what your customers want, or do what will keep you alive in five years? You can’t do both.
Lesson #3: Being a good steward to your core customer can fail you when disruption hits. Focusing on sustaining innovation is tempting for managers who fear cannibalizing sales of their existing products..
The Margin and Timing Trap
Here’s where it gets really uncomfortable for established firms: the math just doesn’t work. Disruptive products are cheaper and offer razor-thin margins. For a company enjoying a cushy 40% on premium insights, a 10% margin business looks like a step backward.
I’ve been in big company quarterly reforecast calls with investors, owners and management who sigh meaningfully when they look at anemic revenue and margin forecasts. With this kind of pressure its easy to see why established firms play the waiting game, deciding to jump in only when the market is “large enough to be interesting.” It sounds prudent, but it’s a fatal mistake.
While they wait, a startup with nothing to lose is thrilled with those 10% margins. They build a business on it, get better, and move upmarket. By the time the market is finally “interesting” to the incumbent, the startup already owns it. I’ve seen this movie before: big firms dismissed DIY tools as toys until SurveyMonkey and Qualtrics became billion-dollar giants. Now, they’re doing it again with AI and synthetic data, creating the very vacuum a new disruptor is about to fill.
Lesson #4: Disruptive products are a trap of bad economics and poor timing. They look too small and unprofitable to matter, until they’re too big to fight.
What Actually Matters: Processes and Values
Here’s where the book gets interesting. Christensen argues that an organization’s capabilities don’t just reside in its people or resources. They reside in its processes and values.
Processes are how you get work done consistently. Values are the criteria you use to make prioritization decisions. And both of these are designed for stability, not change.
Think about a traditional research firm. Their processes are built around fielding surveys, collecting data, managing panels, and delivering reports. Their values prioritize high-margin work, large clients, and proven methodologies.
They believe these are the core, the only things worth improving are the efficiency with which the tasks are completed. They’re what made the company successful.
But when disruption hits, those same processes and values become anchors. You can’t just assign smart people to the problem and expect them to succeed. If the organization’s processes and values don’t fit the new opportunity, it won’t work. Even with the advent of AI, we see the established firms looking at how to deploy AI to make established processes better not disrupt the market (a point I made in a previous article).
Lesson #5: People, technology, and skills have less to do with succeeding in disruptive innovation than do your company processes and what is considered valuable. If your company has ossified processes and value margin stability, disruption will be tough, consider a spin out.
Lessons Learned, Now What?
So what’s the answer? Christensen offers a few principles:
Don’t wait. Small markets don’t become large markets overnight, but they do become large markets. Get in early when you can still learn and adapt.
Use discovery-driven planning. Identify the assumptions your plans are based on. Test them. Adjust. Don’t pretend you know more than you do about emerging markets.
Match the organization to the opportunity. If your core business can’t pursue the disruptive opportunity because of its processes and values, create a separate unit that can.
Accept lower margins initially. Disruptive innovations start with worse economics. That’s okay. The question isn’t whether they’re as profitable as your core business today. It’s whether they’ll be the foundation of your business tomorrow.
The Bottom Line
The innovator’s dilemma doesn’t tell the story of managers being stupid or lazy. It tells the opposite story, a story of ridiculously smart people being trapped by their own success. The things that made you great in one era can become the things that kill you in the next.
For the insights industry, that moment is now. The disruptors aren’t on the horizon; they’re having conversations with your clients. The firms that survive won’t be the ones with the best current products or the most loyal customers. They’ll be the ones willing to cannibalize themselves before someone else does it for them.
Christensen’s book is almost 30 years old, but it reads like it was written yesterday. The patterns he identified keep repeating because the underlying dynamics don’t change. Good management in stable times becomes bad management in disruptive times.
Are you willing to do what feels wrong to do what’s right?