The Labour of Business Insight

Or, why the insight stack at your company feels like a Rube Goldberg machine

Much like ‘Strategy’ no one can clearly define what ‘Insight’ is, but everyone wants it. Every business-intelligence-as-a-service platform in the market promises the elusive insight because it's no longer enough if you’re merely creating an informative dashboard.

What the hell is an insight, though?

A lawyer gets a spark that solves the case, or a detective has a revelation about an anomaly in the clues. Christian Bale finds an anomaly in sheets of numbers, shouts Aha! and shorts the entire housing market. It’s all so dramatic in the movies, but rarely ever this dramatic in a business.

Spinning up the insight machine

An exec, squirting at a chart in slide 12 during a monthly leadership meeting, says,

“Why are we losing revenue in…uh…sub-segment X?”

In response, an engine whirs to life:

1. A flurry of notebook scribbles and Slack messages later, a slightly anxious manager asks an analyst to bring him an analysis.

2. The analyst breaks down the reasons for the drop and ‘runs’ the numbers.

3. Data engineering gets some pings because the log data for February has some missing fields.

4. The manager looks at the analysis, tut-tuts, and creates bullet point narratives to repackage it for executive consumption.

5. When the story is presented at the next monthly meeting, it loses its bite.

“Things are fine(ish). Our team is on it. It may also be partly cyclical.”

The executives didn’t get any insights, but just reassurance that a question was asked, an engine spun up in response, a report emerged, and action was taken.

Organization as an insights stack

The above example may seem a bit caricaturish, but it does capture the wobbly layers through which business insights travel today. One lens to view an organizational structure is as a layered insight stack.

A simplistic version of it:

It’s a version of the corporate telephone game where insights are sanded and smoothed over for the layer above to consume.

It’s slow and expensive.

Lost in the layers

Even the most data-oriented, analytically driven companies suffer from a bloated, dysfunctional Rube Goldberg insight stack.

There are multiple reasons for this complexity:

• In the absence of trustworthy, high-value insights, executives would prefer coverage and narratives since it keeps them in the loop (to hopefully spawn insights)

• Goodhart’s Law means that the minute a metric gets attention, it becomes the thing you game across the layers, especially at the managerial level. This distorts every insight that has to pass through these layers.

• Analysts and tools suffer from the Locksmith Paradox.

  Locksmith Paradox: You pay the locksmith more for an hour of fumbling than five minutes of competent work, because competence looks suspiciously like doing nothing. 

• This translates into analysts creating complex charts and detailed analyses. If a business intelligence tool ever said, “No new insights today,” the analyst wouldn’t know what to do, and companies would probably stop buying them.

• We haven’t even discussed the lowest level of complexity in the data stack, specifically with respect to ingestion, integrating various data sources, and bringing order to generate quick and valuable insights.

An ideal insights engine

An ideal insights stack should solve for both use cases:

• Pull: An executive asks a question and gets meaningful answers and follow-up analysis near instantaneously.

• Push: The insights engine notices a weird blip in the data and performs its analysis and presents new insights without being asked, as if it’s a news report for the day.

When insights can travel across the stack nearly instantly, they not only become relevant but also more democratized.

Now, a customer service agent could see the same actionable nugget (relevant to them) as the VP in the company. The whole company gets that much smarter, in real time, without all the interpretive dance and song.

Are we now truly beginning the insights era?

The good news is that we may be entering an era where insights don’t need to be so painfully hunted down. Beyond vector databases, semantic layers, and AI copilots that can write mediocre SQL that humans don’t have to, there are a lot of interesting changes happening in each layer of the stack, nudging it towards a kind of fusion.

Ideally, the question goes into the machine, and a targeted, clear, insightful answer is spit out rather than it being a switch to initiate a corporate song and dance. Maybe then each layer in the organizational stack (in whatever form it exists) will share the same reality.

Until then, let’s enjoy the monthly decks or the 25-page business review report.

~Babbage Insight