Should your KPI come from AI?
Maybe business leaders aren't meant to spend their time policing three-letter abbreviations
If you ask a business leader to express their objectives in plain English, you’ll often get something clear:
We want to improve the quality of our app.
We want to grow our market share in China by 10 bps.
We need to reduce the cost of servicing customers by 20%.
But put that same leader into the organization’s tracking and review process, and suddenly the conversation is filled with new terminology and a phalanx of three-letter abbreviations: KPIs, OKRs, KRAs, etc.
The messy path from business goals to KPIs
Going from a simple business goal to a KPI is where the butter churning begins: how exactly do you measure “brand love”? You don’t, really. Instead, you pick some proxy number, like Net Promoter Score, repeat purchase rate, or Instagram mentions, that you can put in a spreadsheet. Depending on how analytical the company is, people perform all kinds of contortions to find a measurable, trackable proxy for the objective.
No doubt, KPIs are useful. Organizations are machines, and machines need programming. KPIs align people and track progress. But they come with issues:
KPIs aren’t actually the thing you care about. They’re proxies, but once you put a number on a slide or a spreadsheet, the number becomes the goal.
They become susceptible to gaming (Goodhart’s Law: “Once a measure becomes a target, it stops being a good measure”).
They’re often static even when business conditions are dynamic.
There’s no measure of KPI quality. A great KPI can drive millions in value, while an average one might merely keep people busy. What’s the KPI on KPI quality?
They must be mathematically simple to be understood, even when reality is messy, multivariate, and non-linear.
KPIs were invented to make business measurable. Somewhere along the way, they became the business. Some companies have become finely tuned KPI-defining machines: find the right metrics, hire smart people, and let them optimize, sometimes to the extent of the actual product or service being relegated to only these metrics. Yet surveys show dissatisfaction is common. MIT Sloan reports that 60% of managers believe their KPIs need improvement.
KPIs don’t always accurately reflect what the business wants to achieve. Organizations tend to start measuring things that are ‘easy’ to track (rather than what needs to be tracked), and they become tactical targets to game the system.
KPI in the age of AI
The natural question, then, in this age of AI: Can AI craft and track the perfect KPI?
Firms are already beginning to use AI to define smarter KPIs. Unfortunately, this means we get a new acronym: KPAI (KPIs enabled by AIs), with a big chunk of such organizations seeing their KPIs improve once they began using AI to help refine/change their KPIs. What does it mean for a KPI to “improve”? Usually, it means the AI came up with a proxy that aligns more with the actual goal.
You could imagine, in theory, just handing the whole KPI mess over to the machines.
But could organizations ideally let AI define and manage their KPIs? Once you have unconstrained the KPIs from needing to be static, simple, and readable, the ideal version could look different:
A portfolio of metrics, weighted dynamically by the machine. They needn’t be limited to easy/simple definitions that are easy for people to parse and track.
Weights shift with business changes in near real-time, so the KPI could always be the best representation of the business goal in the changing context.
To keep humans in the loop, you just get a nice output in business-goal language. E.g., “We’re delighting customers at 82% efficiency this quarter,” with the mechanics auditable in case anyone asks.
In other words, AI wouldn’t just track KPIs; it could define them on the fly, forever. That’s the ideal end state, anyway. But getting there is not simple.
KPI is part of the business universe
Translating an objective into a KPI sounds like math, but it’s as much being Socrates as Diophantus. By which we mean that it requires mapping the goal into the “business philosophy”—a set of definitions, assumptions, and causal relationships that look perfectly rational internally (perhaps slightly arbitrary from the outside).
For example, consider this simple objective: “We want to drive 20% higher lifetime value per customer.”
Great. Clear. Except, what’s “lifetime value”? No two companies will agree on the exact definition. Company A might use only transactional revenue. Company B might add word-of-mouth impact, network effects, or referrals. Even the definition of “customer” can vary. Someone who bought once? Someone who installed the app? Someone who clicked but abandoned checkout?
Every business runs on its internal philosophical precepts. Only after an objective passes through this filter does the analytical problem of defining a KPI begin.
And this is precisely where AI struggles, unless it can learn the philosophy of the business first.
Semantic layers as starting points
Semantic layers are meant to be friendly middlemen between raw data and business-friendly metrics. They are essentially abstractions that allow data analysts to focus on generating insights. Naturally, this also makes it easy for an AI sitting on top to spit out insights.
However, these layers aren’t meant to hold the business philosophy. They tell you what the metric is, not why it matters or how it connects to other outcomes. They get you 90% of the way: the data is neatly labeled, the definitions are consistent, and the dashboards look tidy. But the messy 10% on how this metric fits into the business narrative, why it matters strategically, and how its relevance shifts when conditions change has to come from somewhere else.
For example, a semantic layer might define “Active Customer” as someone who purchased in the last 90 days. But it won’t encode that active customers drive better loyalty. These are adaptive rules tied to business strategy and change with context.
That is why you probably need the layer on top: the so-called “system of intelligence” layer, where insight engines operate to drive goals and KPIs. Like a GPS guidance that uses the maps created by the semantic layers.
Can Insight engines suggest KPIs?
Absolutely. It’s already beginning to happen. One study found that one-third of enterprises already use AI to generate KPIs, and 90% see measurable improvements: more collaboration, better predictive accuracy, and stronger stakeholder buy-in.
We’re beginning to move beyond static or even reactive KPIs to truly adaptive ones. Here, AI wouldn’t just suggest actions on existing KPIs but rather help define what KPIs to measure in the first place.
For example, an e-commerce team might begin the year tracking “cart abandonment rate” as a core KPI. Mid-year, AI might detect that supply chain delays, not checkout design, are driving lost conversions and suggest a shift toward “delivery promise adherence” as the more relevant success measure. Now, once this is identified, it gets codified in the “semantic abstraction layer” so that it’s easy to keep generating new insights based off this KPI or metric.
The human role would shift to where it matters most: clarifying the business objectives and deciding whether the organization is moving toward them. Leadership would stop policing stale dashboards and start ensuring that these adaptive KPIs are meaningful maps that still lead toward the true north.
~Babbage Insight