The Tyranny Of Obvious Insights
What's the cost of an insight that really isn't one?
In my past life, there were several moments as a consultant when I wrangled data trying to eke out insights. Sometimes, they were easy. The data was waiting to be mined, and patterns emerged naturally. Other times, it was like looking for a needle in a haystack.
There was one specific assignment I clearly remember. After spending a week trying to figure out how to save costs in sourcing for a major retailer, when presented with the insight on savings, a few heads nodded (good news), and then someone asked the dreadful question, “Got anything else?”
Much like life, when data, people, tools, and incentives all collide, it’s inevitable that some (or many) of these insights tend to be bad. But what’s a ‘bad insight’?
The many flavors of bad insights
There are four broad types of bad insights.
Here’s a more visual mapping of the impact of a bad insight:
An average company will likely struggle with nearly all of these issues
When an insight isn’t one
The most common way that obvious ‘insights’ sneak in is when a narrative of a data pattern is presented as an insight. "Sales decreased by 10% in Q3," is not really an insight, but things like this are packaged as insights in hundreds of corporate reports and slide decks.
But it’s not a problem restricted to corporate PPTs.
This redditor complaint about Oura ring insights is something we’ve all felt at different points:
“It tells me obvious things I already know.
Oura: you slept badly! (I know!)
Oura: you slept well! (I know!)
Oura: try going to bed earlier for longer rest! ( know!)”
Ideally, the sleep ring would remain silent for months and then tell you one piece of insight that will completely change how well you sleep.
However, not everyone is buying it for that kind of insight. Many want information, if only to simply metric-fy their life. Perhaps only a few want deeper insights.
Least common multiple ‘insights’
The Oura ring problem is common with a number of business intelligence and analytics tools.
Many of these tools and platforms are trying to be “all in one” places for users ranging from executives to managers to business analysts to data engineers. Their default operating mode is therefore to go to the common multiple of analyses resulting in a lot of data and obvious conclusions packaged as ‘insights’.
There is also the issue of earning trust. From Hackernews:
“When it comes to business stakeholders, the biggest obstacle is trust. If you're not a data person, or a developer, decision logic being gated behind code is a scary black box. "If I can't control it, I can't trust it".
This, in turn, creates the incentive to “show the work”.
Validation and context gaps
At an organizational level, this is also a common issue across specialized teams (e.g., salespeople or customer service professionals). Employees really love getting their intuitions validated with data, so whenever they’re asked to present their insights, they come back with self-explanatory ones supported by reams of analysis
“Looking at the data, customers loved the 10% promotion we ran; we should do more of it.”
Duh!
Sometimes, even the smartest, well-meaning analyst could make this mistake. After weeks of churning data, you could find a nugget that someone who has spent years in the industry always knew.
The problem here is the context space. A seasoned executive who has spent 10+ years doing something knows things (that are somehow not in the data) that the analyst or consultant doesn’t, and so what looks like a mind-blowing new insight may be an obvious one, which leads to a “got anything else?” type question.
Cost of an obvious insight
It’s hard to do cost attribution here, but there are broadly two kinds of costs here.
The cost of producing an obvious insight: The entire layer of data science, analysts, and the time they spend churning the data can cost several hundred thousand dollars monthly. Every obvious insight, therefore, comes with a price tag that could range from thousands to hundreds of thousands of dollars.
The opportunity cost: Perhaps more expensive is the missed opportunity. The time could have been spent uncovering insights that unlock growth or profitability. Equally, the time lost will further cost the company in terms of lost market share, missed opportunities, and denting of future potential.
As we discussed in the post on the insights stack in organizations, insight generation is still obtuse and complex, and organizations operate in ignorance of the cost of lost insights. You never know what you didn’t find.
There are other, more subtle costs here. As you litter the mindspace with obvious dashboards, slide decks that make you do ‘duh’, and reports that don’t tell anything new, a fatigue for insights sets in, and the ability to absorb a beneficial insight disappears. Real insight is lost in the din of the obvious ones.
What is the latest obvious insight you’ve been rolling your eyes at?
~Babbage Insight
P.S. We’ve only really touched upon the issue of obvious insights here. The impact of the other types of bad insights tends to be more negative (in some examples, it tends to be catastrophic). Fodder for future posts.