Day-one measurement proves AI ROI
Most engineering leaders make one of two critical mistakes when evaluating AI investments. The first is treating usage as proof of value, assuming that high token spend or strong adoption dashboards mean the organization is getting its money's worth. The second is ignoring measurement entirely, letting AI tools proliferate without any framework to assess their impact on engineering outcomes.
Nik Sudan, engineering operations lead at Kraken, rejects both approaches. "One word, four letters, data," he says. "If you're not measuring AI effectiveness, adoption cost, and relating to that as well, the quantitative engineering data like code throughput, DORA, then it's all worthless, 100% worthless. And this has to happen on day one."
Sudan frames AI ROI as a balanced view across three dimensions: throughput, quality, and stability. For throughput, he looks at merge output, frequency, and the maturity and size of contributions. Quality assessment includes rework rates, bug escape rates, and the depth of code reviews. Stability tracking focuses on production health, whether shipped code holds up without incidents or regressions.
The most practical operating measure Sudan recommends is cost per contribution: AI spend divided by merged work. This metric reveals whether engineers are using tokens effectively or burning through budget on low-impact changes. "You take two engineers who have the same output and the same impact," he explains. "One spends thousands of dollars and the other hundreds. The most effective engineer depends on the value created."
The answer depends on whether increased token consumption correlates with genuine value. If an engineer burns thousands of tokens but produces poor-quality, low-impact work, the output becomes worthless regardless of volume. Effective AI use means understanding token economics, selecting the right model for the task, implementing caching strategies, and distinguishing between deterministic workflows and agentic reasoning.
Sudan emphasizes that all of this must be contextualized by project impact. Raw contribution counts reward whoever ships the most trivial changes. Instead, teams need to weigh contributions by business relevance and technical complexity so that measurement reflects actual effectiveness rather than gaming the system.
The timing of this measurement infrastructure matters as much as the metrics themselves. "This has to happen on day one," Sudan insists. Waiting until AI habits harden across the organization makes it exponentially more expensive to retrofit measurement systems and correct course on ineffective practices.
Context makes engineering metrics actionable
Engineering metrics become meaningless when stripped of context. A single company-wide baseline for cycle time or review velocity ignores the reality that different teams work on different services with different dependencies, and some are legitimately slower for entirely valid reasons.
Sudan recommends benchmarking at the right level of granularity, by team, by frontend versus backend, by mobile versus web, rather than flattening the organization into a single aggregate number. At Kraken, they isolate development groups in LinearB and scope them to relevant repositories, then treat industry averages as a default benchmark. When a team shows red against that benchmark, the response isn't judgment but investigation.
The choice of statistical measure matters. Sudan prefers P90 over averages because it exposes the slowest portion of delivery performance rather than hiding struggling areas behind flattering aggregate numbers. "We want to find out the slowest 10%," he explains. "We don't want an average because it flatters us." If a team is doing well overall, averages quietly absorb the areas that are actually struggling, and those problems never surface.
Quantitative signals like cycle time, review time, and rework must be combined with qualitative context from Jira, Slack, and survey inputs to explain why patterns appear. The numbers tell you what's happening, but the people tell you why. This layered approach prevents metrics from becoming a gamified scoreboard and instead positions them as diagnostic signals that help teams learn from the data.
Sudan warns against treating metrics as targets. When a metric becomes a target, it stops being a useful measure, a phenomenon described by Goodhart's law. The right use of data is asking what's worth learning from it, not optimizing for the number itself.
Review time is the biggest delivery bottleneck
For many engineering organizations, review time represents the most significant cycle-time bottleneck. Unlike deploy time or build time, which can be addressed through infrastructure investment or pipeline optimization, review time is primarily a people and process problem, making it more complex to diagnose and resolve.
Initially, Sudan's team assumed that large or complex merge requests were the main cause of review delays. It's a common hypothesis, and one that high-level platform data seemed to support. But when they joined LinearB's trend detection with deeper repository analysis, pulling reviewer activity, comments, touched files, and other granular signals from their Git provider, the data told a different story.
Size and complexity accounted for only 5 to 10% of the slowdown. The real culprits were time zone differences, team silos, and reviewer availability. Engineers in one region were opening merge requests at times that didn't align with the code owners for those areas, who were largely in another region. The work simply sat waiting for a reviewer who wasn't online, regardless of size.
"That's the hidden bottleneck that these silos were masking," Sudan notes. Proving this hypothesis with data transformed it from a hunch into a real, data-backed problem that leadership could act on. The fix Kraken is implementing: expanding the pool of code owners so reviews aren't gated to a single time zone.
This analysis also reinforced the importance of earlier stakeholder feedback and cleaner handoffs from prototypes. When late-stage rework piles up, it doesn't just slow down individual merge requests, it creates cascading delays in the review queue, making every subsequent change more expensive to process.
MCP connects engineering metrics to root causes
The Model Context Protocol (MCP) offers a practical way to connect high-level engineering metrics with the lower-level evidence that explains them. Sudan describes this as a two-layer approach: LinearB provides the high-level trends and scoping, while Git provider data supplies granular signals like reviewer activity, comment volume, and file changes.
"The two-layer approach works really well: LinearB high, then your repository provider low, and then AI is the part that moves fluidly between each other and enriches it."
The key is starting with selective context rather than flooding the model with every low-level detail. LinearB identifies the relevant merge requests, repositories, and teams at a high level. Only then does the analysis move to repository-level data for deeper investigation. Dumping every granular detail upfront would muddy the signal, increase token costs, and degrade the quality of insights.
This layered strategy extends beyond code repositories. Tools like Jira, Slack, and calendars provide additional context that helps AI analyze the broader delivery environment rather than code events in isolation. At Kraken, Sudan's team uses these sources to understand not just what's happening in the codebase but why patterns emerge, whether it's scheduling conflicts, unclear project priorities, or communication gaps.
Sudan notes that while MCP distribution helps less technical users access structured engineering context, CLI-based access can sometimes be preferable for reliability and control. "I actually prefer to MCPs 'cause they're not as flaky, their authentications are long-lived, and sometimes they even have more capabilities," he explains. The choice depends on the workflow and the user's technical fluency.
The real value of MCP-based integration goes beyond data access, it's the ability to move fluidly between layers of abstraction without requiring manual stitching. AI becomes the connective tissue that enriches high-level trends with low-level evidence, and that's where the biggest time savings come from.
Evidence-driven leadership turns engineering data into business action
Evidence-driven engineering leadership represents a shift from anecdotes and intuition toward explicit hypotheses, measurable signals, and validated conclusions. It's a scientific approach to decision-making, and it requires leaders to challenge AI-generated explanations, test their own assumptions, and accept when the data disproves an initial hunch.
AI is never going to decide what questions you should ask. That has to come from the leader. AI helps with the analysis, but the quality of what you get out of that assessment is bounded by the quality and context you provide. Sudan emphasizes the importance of having hunches, testing them rigorously, and being open to being wrong. In one instance, the team's initial hypothesis about merge request size turned out to be incorrect, but because they approached the problem scientifically, they discovered the real issue.
"It also pays to be scientific in this rather than just trusting the hunch. Too many people trust AI, and AI is a yes person."
Translating engineering measures into plain business language is critical for connecting delivery health to roadmap speed, investment needs, and risk. Non-technical stakeholders won't act on metrics they don't understand or trust. "You can't just throw metrics at people and expect them to understand them," Sudan says. "If they don't understand it, they don't trust it, and that's completely worthless."
Within his organization, Sudan describes cycle time as "how long it takes us to ship something in front of our clients" and links it to outcomes that stakeholders care about: faster project execution speed, quicker response to customer needs, and reduced time to revenue. This translation helps product, design, and operations leaders understand why engineering metrics matter and unlocks prioritization for engineering-driven initiatives like tech debt reduction.
Sudan warns against comparing teams through simplistic leaderboards. Weak signals usually reveal friction or underinvestment rather than low capability. "A team at the bottom is usually one suffering from friction," he explains. "They need engineering investment. And it needs non-technical leadership to give it that time, that backing to fix it, not the chopping block."
Building the full chain
Finally, Sudan emphasizes the importance of separating experimentation from production. Proof-of-concept work should happen in isolated environments with relaxed repository rules and lightweight deployment scaffolding. This resists the temptation to promote pilots straight to production and ensures that production systems maintain higher standards, including agentic testing, proper architecture, and scalable design.
The chain is clear: data, context, insight, action. Most teams stop early, collecting raw metrics without the context to enrich them or the insight to drive action. Leaders who commit to the full chain, and who put measurement in place on day one, position their organizations to operationalize AI effectively and prove its value over time.
To hear more of Nik Sudan's insights on AI measurement, cycle time bottlenecks, and evidence-driven leadership, listen to his full episode on the Dev Interrupted podcast.
