Are you confusing a skyrocketing AI token bill with actual engineering value? This week on Dev Interrupted, Kraken's Engineering Operations Lead, Nik Sudan, joins the show to break down the harsh realities of moving agentic AI projects from pilot to production without compromising code health. He unpacks why raw AI adoption is a flawed vanity metric, detailing how his team uses tools like the LinearB MCP server to combine high-level engineering metrics with granular repository data to uncover hidden workflow bottlenecks. Finally, Nik reveals his exact playbook for translating complex data, like P90 cycle times, into a clear, business-driven narrative that secures vital buy-in from non-technical stakeholders.
Show Notes
- LinearB MCP Server:Learn how to chat with your engineering data and uncover hidden bottlenecks atlinearb.io/platform/mcp-server.
- The APEX Framework:Read LinearB's guide on the operating model for AI-era engineering teams atlinearb.io/resources/apex-framework.
- Website / Follow Nik:niks.space
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: And today's guest is Nik Sudan, engineering operations lead at Kraken. And in this session of LinearB's AI Enablement interview series, Nik's going to share his insights and structural lessons about scaling AI maturity across engineering teams, including their hands-on experience deploying tools like the LinearB MCP server.
[00:00:18] Andrew Zigler: So Nik, thanks so much for joining us today
[00:00:21] Nik Sudan: Hey, no, thanks for having me here. It's been, uh, it's been really awesome to be here
[00:00:25] Andrew Zigler: Awesome. Well, I wanna go ahead and dive in about, uh, something that's really top of mind for a lot of folks that are working with AI tools is they start with this phase of just, like, adoption and then experimentation, and then ultimately there's a mandate or a need to operationalize it and reach production. It can be really hard to ship AI-powered ideas from the pilot to production in order to have, like, long-term value. So, like, in your experiences at Kraken, like, what were the things that broke down during, like, transitioning from pilots to production, and how did y'all overcome being able to operationalize with [00:01:00] AI?
[00:01:00] Nik Sudan: Yeah, great question. So I love shipping stuff really quickly, and you know, in this day and age, I love proof of concepts as well. And everyone at Kraken loves it. Even before this whole AI renaissance or slopageddon, however you wanna phrase it, right? Um, but we have always been moving really fast, right? We always value that kind of startup vibe, try and keep teams lean, you know, stuff like that.
[00:01:23] Nik Sudan: And every day we look for ways to execute faster. And AI is... It's made it extremely easy for everybody, right? Um, including non-engineers, which is where this can definitely, have a greater effect than how it was before, right? We don't wanna block anyone from building out kind of pilots proof of concepts, right?
[00:01:43] Nik Sudan: So transitioning from pilot, uh, proof of concept to production has always been quite tricky, and I think today it's even easier now. And it breaks down in a few predictable ways. So I wanna call out some of the problems, 'cause I think knowing the problems will help mitigate it. But then I'll also [00:02:00] give, um, some examples of what we're trying to do to avoid it as well.
[00:02:04] Nik Sudan: So number one, spending too long on your proof of concept, on your pilot, right? It should be a quick and dirty couple of project, uh, quick and dirty thing, take you a few days or a week maximum, right? Get something out the door. That's why it's called a pilot, a proof of concept. You know? You're trying to get the point across.
[00:02:25] Nik Sudan: So don't overthink architecture or design. You know, engineers will spend a lot of time on architecture. You know, designers, more visual people will do design. Don't, don't... Just, just for what is the kind of core essence of what you're trying to get across. You know? Like, you know, when showing the BlackBerry proof of concept, you know, the dumb, a dumb phone just to show investors.
[00:02:45] Nik Sudan: Like, that's what you're trying to do, right? A- and just because we can build stuff in software doesn't mean that we should, you know, build something that's five times as good. Still just keep it simple. Spending weeks polishing a proof of concept, then you've, I think you've lost the plot. So [00:03:00] secondly, treat your proof of concept as the foundation for your real thing.
[00:03:04] Nik Sudan: Don't do that. You know? Don't treat it as the foundation. If you're building a scalable product, you'll need proper architecture. Proof of concept doesn't need any of it, right? Just goals are completely different. Promote proof of concept straight to production, and you're probably gonna get a lot of pain later on, right?
[00:03:20] Nik Sudan: So don't be afraid to work on throwaway code projects.
[00:03:25] Andrew Zigler: Yeah,
[00:03:26] Nik Sudan: yeah.
[00:03:26] Andrew Zigler: about like showcasing what could be the end result more than about trying to lay every brick of the path to get
[00:03:32] Nik Sudan: 100%, yeah. And again, I think a lot of engineers try to think about this and stuff, and I guess the mistake is promoting that to your production product, right? Um, and this is where I guess the third thing I would like to bring up, which I say is, I would say is more of a newer trap. So, you know, now that everybody has AI, everyone can build something fast.
[00:03:52] Nik Sudan: You know, you go on X, you know, someone, someone's shitting their startup idea, and then a week later they tweet, oh, sorry, they post, " Oh, [00:04:00] this broke. My secret leaked to production. S- I guess I won't be committing them." It's like, yeah, for us it's like, no s**t, of course. But but um, you know, with AI and building something fast as well, there's a temptation to think it, you know, the speed, just the high output scales consistently all the way to production, right? So you can iterate quickly, get things in, in front of people quickly, but making something that's scalable production ready system, you're still going to need to take time with it. AI can help and will help you make it faster, but you take out all of the thinking, you get agents building it, it's gonna, it's gonna break.
[00:04:39] Nik Sudan: You know, pressing one button telling, you know, turning on auto mode on Claude isn't going to ship you a money maker. You know, it might ship you something that looks like it does, but it doesn't, right? Um, so I guess some examples of what we're doing at Kraken as well to prevent this as well. So [00:05:00] testing, specifically automated testing as well, um, and agent-based testing, right?
[00:05:05] Nik Sudan: Um, and also, you know, the tried and tested just internal testing with, with your real users. The bar for something being done has to be deliberately higher than, you know, this worked in the demo, right? Especially with AI now. As a, as a engineer myself, I spend less time coding properly now. I will prompt stuff.
[00:05:28] Nik Sudan: I'll have a whole bunch of agents doing stuff, and I'll verify it works and then, you know, send it over. But that's, you know, my, my trust in that has dropped significantly. So relying on testing es- and especially, you know, you run a company with hundreds, thousands of engineers, you're not gonna keep on top of it.
[00:05:46] Nik Sudan: So investing in testing, works out a lot there. and AI, having AI in the loop now, um, you know, where you have mo- more non-deterministic behavior, right, means, um, pass it like, uh, passing a unit test isn't the same [00:06:00] as like a system, right? So I think having, uh, incorporating, you know, agentic testing, that works out well.
[00:06:06] Nik Sudan: Um, another thing, isolating your proof of concept from production. So, um, we build proof of concepts in separate environments to resist that temptation to kind of, "Right, let's actually turn it on." So throw away repos, you know, very light scaffolding for deployments and stuff like that. So peop- you know, maybe relaxed, kind of repository kind of push rules and stuff, so people can just iterate quickly without dragging production's concerns and stuff.
[00:06:34] Nik Sudan: Um, our designers, so, um, our designers are a lot more hands-on now with coding and such. Um, and they're working on kind of building out their vision for certain projects, but they're working in a separate repository. Like we have our, all our mobile app, our web app repositories, but, you know, we've given them an environment where they can quickly iterate, get something, it's removed from the production app, um, and then when it's time to productionize, [00:07:00] they just hand over the repository, the files to the next engineers.
[00:07:03] Nik Sudan: And that's actually a lot easier to work with than a Figma file or a PDF or an image, right?
[00:07:10] Andrew Zigler: Right
[00:07:10] Nik Sudan: and they, and that engineers then go, "Okay, this is how it looks. Let's productionize it, integrate it properly." You know, they know how the design system works. They know how to build things effectively, cache things effectively.
[00:07:22] Nik Sudan: You know, designers are good at design, so let them work, focus on the visual part in that separate environment. And I think, um, lastly as well, be explicit about what that pilot is, right? Um, because I think many stakeholders, many reviewers aren't clear that this is the proof of concept. You show it to them and they're going, "Oh, so can we expect it in production next week?"
[00:07:46] Nik Sudan: It's like, "No, no, no, no. This is something different," right? You need to make it very clear that this is not your final production product. Um, value in there is to prove the point, not to ship. You know, show a working product instead of [00:08:00] writing long AI-generated briefs, running endless meetings, right? Showing something is worth a thousand words, especially for people who are really busy all the time, right?
[00:08:10] Nik Sudan: Prove that you can build the thing that they want, the thing that they envision in their heads. Get... Grants them a lot more confidence, and they'll give you feed- really, really good feedback there and then. ' Cause if you can... If you give them documents, if you talk to them endlessly, they'll be like, "Uh, I'm not really listening.
[00:08:25] Nik Sudan: This sounds all right." But you show them something, they'll give you feedback immediately, right? So this is... And just making that very clear for them, right? You know, I think everyone who's built products, you know, you... It's like a week before shipping to production and some stakeholder comes in and it goes, "Actually, this isn't right."
[00:08:44] Nik Sudan: And it's like everyone panics like, "Oh God. Oh no," right?
[00:08:47] Andrew Zigler: It's like, who
[00:08:47] Nik Sudan: Ev-
[00:08:47] Andrew Zigler: who let them see it before we shipped it?
[00:08:49] Nik Sudan: Yeah. Yeah. But it's like, well, why didn't they see it earlier? And it's
[00:08:52] Andrew Zigler: Why
[00:08:52] Nik Sudan: 'cause they've been busy. Yeah. Yeah. So, and this is where the value of it comes in, right? Get the feedback there and, you know, [00:09:00] having feedback at the end is expensive, right?
[00:09:02] Nik Sudan: It takes time. It, it causes, uh, regressions as well 'cause you're ch- if you're changing massive stuff last minute, the chance of it not working in production, right? And especially with, you know, with LinearB, you can measure the kind of rework rate and stuff like that. You know, you want to keep that kind of stable, right?
[00:09:18] Nik Sudan: And to kind of maintain that rework rate, you need to make sure that you kind of get things right at the start as soon as you can, right? You know, so so just kind of making that kind of clear line. This is proving the idea, and then this is how we make it real, right? That's kind of the ethos around it.
[00:09:35] Nik Sudan: Um, but yeah, that's kind of my take on that
[00:09:38] Andrew Zigler: I-- So what I hear about some of, like, the shape of how y'all think about AI, here's what I, here's what I hear, is like it's almost like this, um, permeable barrier. Inside of it is all of this experimentation and rapid iteration, and inside of this world, you're not creating production-level foundational code for what's gonna be-- what drives the, the front of the sleigh for the next year.
[00:09:59] Andrew Zigler: Instead, [00:10:00] you're creating the, the visions, the interfaces, the hooks, the things that you can show to non-technical
[00:10:05] Nik Sudan: Yes
[00:10:06] Andrew Zigler: the technical leaders, and to your fellow engineers to all coalesce around what is-- what, what are we trying to ship? And so you get this rapid bubble. I call it a permeable, uh, barrier because ultimately from that, something has to exit and now start to enter, you know, the SDLC, the agentic, uh, DLC, however, whatever you wanna label it as for your particular org. And so, you know, you mentioned, for example, looking at, uh, you know, when engineers are working, keeping the rework rate stable and tracking that through LinearB, for example, helps you keep a handle on overall code and organizational health as things cross out of that barrier. So that's kinda what I wanna s- I wanna zoom in on next and try to understand is when y'all are, as engineers who have built the way to go from that vision, we're all on the same page, so now we're going to lay the foundation [00:11:00] brick by brick and get there, uh, in a really stable way and ship it. Uh, you know, how do you start to partner with, uh, like your tooling to not only all get aligned, but then also, like, to understand, like, is this working, um, and are the practices that are leaving the barrier and starting to enter our ADLC, um, are they effective? Like, what are the kinds of ways that y'all look to that and make sure that your engineering pipeline stays healthy?
[00:11:28] Nik Sudan: Everyone at Kraken is using AI right now. It's whether we like it or not. Um, it has been a complete game changer when it comes to software engineering. I guess the, the myth of like it replacing, you know, software engineering isn't really coming about, and to be honest, it shouldn't because it is a partner, it is a tool, right?
[00:11:47] Nik Sudan: Kind of, you know, going from assembly language to kind of typed codes, uh, to frameworks, all that stuff. AI is just kind of another level of that on top. But rather than it being very deterministic, it's agentic, right? It... [00:12:00] And this is kind of how we interface with the modern world right now. And, you know, everyone at, not just at Kraken, but I think everywhere, right?
[00:12:07] Nik Sudan: They're jumping on AI to win on kind of throughputs, on output, 10x their impact. Um, but um, the spend is, I, I think one of the biggest, um, things here, right? Um, to the point where it's close to or matching the salary of a developer. So y- how... So the question here is, you know, if, if it gets more expensive, um, you know, when, you know, with like, you know, Fable coming out now and, you know, people have been like, "Oh, I asked it one question and there goes all of my tokens," you know, stuff like that.
[00:12:41] Andrew Zigler: waiting for a token refresh. I totally understand
[00:12:44] Nik Sudan: have not switched over to Fable yet. I saw the pop-up and I'm like, "I, I'm, I'm gonna stay with Opus for now," you know?
[00:12:50] Andrew Zigler: Y- you're missing out, Nik. All
[00:12:51] Nik Sudan: Yeah. Yeah. I don't, I don't need it to, to do that for many things actually. And I guess that's one thing as well, which I'll get into a bit, but just,
[00:12:58] Andrew Zigler: Yeah
[00:12:59] Nik Sudan: [00:13:00] effectively like what's the best model for your work as well?
[00:13:02] Nik Sudan: But yeah, cost, whether we like it or not, is the biggest thing about AI impact now and whether, uh, we get a value out of it. 'Cause at the end of the day, that's like time is money, right? And,
[00:13:15] Andrew Zigler: like the value of an engineer is like the cost of the tokens
[00:13:18] Nik Sudan: Exactly. Yeah. Yeah.
[00:13:20] Nik Sudan: And so I even think AI's gonna get even more expensive over time. Um, and businesses who aren't thinking about that regarding ROI will be starting to think about it now, this week in particular, right?
[00:13:32] Nik Sudan: Rather than assuming that, you know, today's costs are, this is how it is gonna be in the business, you know? So what my take is in regarding how to measure the value of using AI with your SDLC and such, so, I think most leaders actually make one of these two mistakes. So the first is, you know, AI usage is, uh, or spend, you know, is proof of value, right?
[00:13:54] Nik Sudan: You know, so the assumption is if, you know, cost is high for an individual, for a team, we're [00:14:00] getting our money's worth, right? And then measuring adoption, you know, like how many seats are being filled, percentage of engineers using AI, adoption dashboards, tracking all of that stuff. They look great, but they don't tell you anything about whether the work gets better, right?
[00:14:16] Nik Sudan: Um, AI usage isn't a metric that is great for effectiveness. It's one signal across many. But by itself, it doesn't prove anything regarding SDLC and the value of AI. So what you would actually want here, and what my recommendation is, and what we're starting to do at Kraken is you have to balance a series of metrics across throughput, quality, stability, right?
[00:14:43] Nik Sudan: So with throughput, merge request output, frequency, and the maturity and the size of those merge requests, all that you can see within LinearB, right? Quality, so like, you know, what's the rework rates of the changes? What about bug escape rates? You know, how deep are the reviews that actually happen? And then when it [00:15:00] comes to stability, you know, is the thing that people ship to production actually holding up, right?
[00:15:05] Nik Sudan: Any incidents, any problems, regressions. And all of it has to be contextualized as well. You know, what kind of projects are people working on? What are they contributing to? What's the real impact of those projects when it comes to the company and the products, right? I think for me, the most useful measure is, you know, cost per contribution, AI spend divided by contribution.
[00:15:27] Nik Sudan: You know, it's a very simple one. You could compute it as spend per merged merge request or per engineer, per sprint, or something like that, right? As people use AI more, their output should increase. But for genuinely valuable work where the cost per contribution stays stable, that's what you're looking for, right?
[00:15:47] Nik Sudan: Um, and then this only works if you weigh by impact as well, right? If you measure it raw, you're gonna reward whoever ships the most trivial low changes, right? I guess the best way to describe this with an examp- example is [00:16:00] this. You take two engineers who have the same output and the same impact. One spends thousands of dollars and the other hundreds.
[00:16:10] Nik Sudan: Who's the most effective engineer here? Maybe the one who's burning thousands of tokens has more raw output, right? But the quality and impact of their changes is poor, then the output is worthless, right? What we want to have is people using AI effectively, you know, using tokens well, knowing which model to reason for, as we were talking about, caching, optimizing the usage with context, knowing when to use deterministic kind of, um, thinking versus agentic.
[00:16:35] Nik Sudan: AI isn't a replacement for this. It, it is an enhancer, so we need to make sure that we measure it like that. And yeah, I don't think people out there are really thinking about this too much. So that's my take on that.
[00:16:49] Andrew Zigler: Yeah. And you get like, uh, everyone's over-rotated and over-focused on like how do we get in the code layer and get in the IDE and help make sure we're steered and we're all aligned on like we have, uh, [00:17:00] configs and, you know, an agents.md and best practices, whatever. Like, all of that stuff is super helpful.
[00:17:06] Andrew Zigler: But what you've just described is how there's a higher order problem to solve to be effective. It's your entire engineering process need to harness. In your case, it's like LinearB's that harness because it gives you this through- this like view into all of these different health signals and things like quality and throughput across all of the code you're shipping so that when you are, you know, asked or presented and you're looking at the numbers of like your token consumption and are we getting value of this?
[00:17:34] Andrew Zigler: You can actually... You have a contextualized narrative for like why you are getting value out of this. And, uh, that kind of like bridge, bridges into the next thing that I wanna talk about, which is, connecting like data silos and actually using that to get really good velocity as a team because, um, it, it sounds like from what you're describing, you have a lot of signals that come into one place and then it helps you understand how ef- effective, [00:18:00] um, are coding processes, how effectively are we shipping.
[00:18:02] Andrew Zigler: But then what's the health of it downstream, which is a really important problem to always have in perspective. Uh, so like one example is, uh, you know, when we first were chatting about this, you, you mentioned that like Kraken was using MCP server from LinearB to, to contextualize th- the information, but also combine it with these other sources and tools to create a really effective, uh, things that drove, uh, decision-making internally.
[00:18:29] Andrew Zigler: So like how are y'all thinking about using MCP, for example, to, to distribute this engineering knowledge to folks?
[00:18:37] Nik Sudan: So taking a couple of steps back, this boils down, all boils down, MCPs, AI, boils down to this one thing, which is the thing that AI is really good at right now, which is data analysis, right? So data analysis isn't new. People are in that field for many years now. But I think the difference [00:19:00] now, um, and the relevancy to the question right now is AI lets anyone be an effective data analyst, right?
[00:19:06] Nik Sudan: It's the fact that, you know, now that everyone has a camera in their pocket, anyone can be a photographer. You don't need crazy equipment. Anyone can be a data analyst now, right? Good data, anyone can analyze it, but they need to be asking the right questions. You know, just because you have a camera in your pocket doesn't mean a really blurry photo of like, you know, a crooked angle and stuff is, doesn't make you...
[00:19:27] Nik Sudan: It makes you a photographer, but it makes you a horrible one. Oh, yeah. So it's the same with data analyst as well, right?
[00:19:33] Andrew Zigler: sure. Yeah, I
[00:19:34] Nik Sudan: yeah. So quite a lot, lots of air quotes there for sure. Um, so step one, uh, at least with all these data silos, the MCP is to get the data in front of you and as rich and as sensibly as can be.
[00:19:50] Nik Sudan: Not over-enriched, 'cause then you're gonna be spending more and more with context and stuff as we talked about before. But with it at a point where every data point you think might come in [00:20:00] handy is there. So timestamps, you know, names of people who reviewed it, comments, how many comments were left on merge requests, you know, what files were touched, anything that might explain kind of skew or size of the change, right?
[00:20:12] Nik Sudan: So LinearB is great at providing the high-level stuff, um, here. And then when you... So when you're using your kind of, um, Git provider here, so GitHub, GitLab, Bitbucket, you know, that's where you can dive in deeper, right? And this is where the bridge comes in there. So example here, we were asking probably one of the most common questions that people get, "Why does review time take so long," right?
[00:20:38] Nik Sudan: Um, and tools like LinearB help surface it, uh, and you have to do a bit of digging in to find out why. But it's not a simple answer when you have, you know, thousands of engineers contributing tons per day across different types of services, types of programming languages, types of like... There's so many factors, right?
[00:20:56] Nik Sudan: And if you just get present like, "This needs to improve," [00:21:00] you need the data, right? Um, so review time in my view is arguably the most important part of cycle time, and it is the current bottleneck for us and probably the bottleneck for many companies right now. And the one thing that AI can help out with a lot, because it's a lot more like agentic with the way of thinking.
[00:21:18] Nik Sudan: There are many different possibilities. It's not as simple as like, "Oh, if the deploy time is broke, that means we need to speed up the pipelines. There are a number of runners available," or something like that, right? we'll reduce the number of dependencies. But review time that it, it's, it's a whole black box of like, it could be this, it could be that, right?
[00:21:36] Nik Sudan: A people and process problem rather than a tooling one, right?
[00:21:39] Andrew Zigler: Yeah
[00:21:40] Nik Sudan: So with the structure, with the structure bit, right, as we were talking about, uh, before. Um, so yeah, use LinearB to aggregate all of that at a pure data level. Um, you know, so right now you're not using the AI to, you know, aggregate or kind of make decisions, but the MCP is really good to get that data out.
[00:21:59] Nik Sudan: The MCP [00:22:00] can help answer questions, but one data source alone, specif- especially as LinearB is focused more on the high-level stuff, um, that's the entry point, right? High-level themes, trends, and it shouldn't pull in the granular themes at this point 'cause it's gonna get expensive, it's gonna get unnecessary.
[00:22:17] Nik Sudan: It gives you the opening path to kind of make the decision, kind of go like, "Right, this could be an avenue, this could be an avenue," right? if, if you dumped every low-level detail in, in it, it would just muddy the signal as well. So high-level trend is very important to first to identify. So in this case, review time is slow.
[00:22:36] Nik Sudan: We find out the relevant merge requests, the repositories, the teams. LinearB, is really good at that, right? 'Cause we, like, we ingest thousands and thousands every week. At that point then, once you've got kind of the high-level data, then we use our, um, repository, uh, our Git provider, MCP, to then... And al- and also maybe not even MCP at this point, [00:23:00] 'cause if you plug in a CLI, which I actually prefer to MCPs 'cause they're not as flaky, their authentications are long-lived, and sometimes they even have, um, more capabilities.
[00:23:11] Nik Sudan: But whatever you use here in the agentic interface, this is where you can join, right? And the two lay- layer approach works really well. LinearB high, you know, uh, then your repository provider, the low, and then AI is the part that moves fluidly between each other and, and enriches it, right? So there's no human stitching it, making the connections itself, and that's, that's the biggest time saver right now.
[00:23:36] Nik Sudan: And it also pays to be scientific in this rather than just trusting the hunch. Too many people trust AI, and AI is a yes person. It's never g- and, and it's, it's generic as hell, so you need to be thinking critically. You need to be challenging it, and you need to have hunches, you know? Our hunch was wrong, actually, right?
[00:23:55] Nik Sudan: So first of all, we assumed that big, complex merge requests were what [00:24:00] was slowing the review time down. That's what most people say. You know, LinearB gives those signals as well, you know? Review time bad, merge request large, size must be the cause. But we joined up the data, and we looked at it, and size and complexity accounted for maybe 5 to 10% of the slowdown, right?
[00:24:19] Nik Sudan: 70 to 80%, something like that, came from time zone differences, a hunch that we didn't even think about too much. Team silos, reviewer availability, engineers in one region opening a merge request at a time that didn't line up with the code owners for the areas. Um, and it was really hard to analyze all that type of stuff, and who are largely in, in, who were largely in love with so that, you know, the work just sits waiting for a reviewer who, who wasn't online, um, regardless of size, right?
[00:24:49] Nik Sudan: And that's the hidden bottleneck that, you know, these silos were masking. And now that we could prove it, it stopped being a hunch, you know? Became a real data-backed problem [00:25:00] that leadership became aware of, and the fix we're working on right now is to expand kind of the pool of code owners, you know, so reviews aren't gated to a single time zone and, you know.
[00:25:10] Nik Sudan: It's, it's a complicated thing, you know, people and process problem. But having the data to prove the hypothesis, like, you know, being scientific about it, that's, that's the thing. That's the structural shift, you know? It moves leaders from the anecdote driven to evidence driven, right? Instead of, instead of acting on the feeling and maybe some high-level metrics or, I think, you know, when you're having one-on-ones, if you're like an engineering manager, you might get someone going, "Ah, yeah, no, I've had some slowdowns working with this team," or that.
[00:25:40] Nik Sudan: And you might just be like, "I'll pencil it in. I'll put it in for later." But that, maybe that's the hunch to start digging a bit deeper, right? Find out why that's happening, right? Form the hypothesis, test the data that actually supports this, and be open if it doesn't, right? So the one thing that AI doesn't do for you is it doesn't decide what you should ask.
[00:25:59] Nik Sudan: That has to come from [00:26:00] you. Presents, it helps with the analysis, but the quality of what you get out of that assessment is bounded by the quality and the context that you give it as well, right? So that's a little bit of how I use it and how I recommend kind of people use multiple sources as well. You know, use Jira, you know, that's really good for knowing the context as well.
[00:26:20] Nik Sudan: If you're using Teams or Slack, you know, get the conversations in as well, 'cause sometimes it's not as simple as, you know, at the end of the day in engineering, everything comes kind of code change, but there are so many factors as well, right? This is just a very simple example of that, right? Um, you know, plug in schedules, calendars, every- like all sorts of little nuances, right?
[00:26:39] Nik Sudan: There's so many different connections now that are supported with MCPs that you'd be crazy not to use them.
[00:26:45] Andrew Zigler: Right. It's about, it's about contextualizing all of the data that's available to you so that
[00:26:50] Nik Sudan: Hmm
[00:26:51] Andrew Zigler: you can take your high-level questions that you yourself as the expert in the domain in which you serve value to your end users can pose these [00:27:00] really complex questions that before would've taken a team of data scientists and maybe a few weeks to crunch for you, that now you can do really quickly by assembling these tools.
[00:27:08] Andrew Zigler: And I totally know what you mean about like sometimes MCP can feel like it's in the way, and for certain workflows it certainly is. And for yours, you know, you might benefit more from like a CLI-based approach for a lot of the stuff you do. But one thing that MCP solves really well that another, other toolings and ways of sharing context with AI haven't is distribution, because that MCP server can be really relatively simple to put into the hands of somebody who is less technical and equip them with abilities to take their high-level queries and actually get the crunch of the numbers.
[00:27:39] Andrew Zigler: But therein lies the trap that you've very, very smartly identified and stepped around, that, you know, it, the, one, the AI is a yes man, and two, you have to have that understanding of what you're doing to ask the right questions that ultimately extract the right stuff you want. And if you drown it in data, you're not really going to get...
[00:27:59] Andrew Zigler: You're gonna [00:28:00] get a, a facsimile, like a mirrored, uh, version of what you're asking instead of like the core truth, right? So, a, something I wanna ask you about is, for example, how you equip those stakeholders to be able to understand what's going on in the engineering world through tools like the LinearB MCP server, for example, 'cause you can distribute that or otherwise make it accessible.
[00:28:23] Andrew Zigler: So like, uh, for example, you've talked a lot about cycle time in this conversation, about how, why that's important to you. Like, do, how do you use this, uh, these kinds of tools and, and distribution of context to get stakeholders that are non-technical in the, the conversation and really understanding what's going on?
[00:28:42] Nik Sudan: So when it comes to contextualizing, I think these metrics, these kind of things, they shouldn't come top up. They should come from the ground level. Um, and anything that's important for an engineering team should be important to that higher level audience. [00:29:00] I think most times this isn't the case because of that contextualization, that translation, you know, into the non-technical terms and, you know, this is why MCPs are really good right now.
[00:29:11] Nik Sudan: So to give a bit of an example as to kind of like how we translate stuff at Kraken. So, you know, we prefer P90 to average, for example, right? Um, because we want to find out the slowest 10%, you know? We don't want an average because it flatters us, right? And if a team is doing well overall, it quietly absorbs the areas that are actually struggling and you never see them.
[00:29:32] Nik Sudan: And P90 surfaces that slow tail, which is exactly what us as an engineering want when we're trying to improve. We hold a high bar for quality and technical ability, and P90 is where we want to go and hold everyone rather than hiding behind a comfortable mean. and so by doing that, it's to stakeholders who are higher up, they might go, "Oh, this is really slow," but it's like, "No, no, no, this, this is what we want to do," because we...
[00:29:56] Nik Sudan: A metric shouldn't like, you know, when a metric starts becoming a [00:30:00] target, it's not the target, you know, Goodhart's law in this case, right? So a metric should be helping you inform. So, the aggregation is one thing. Um, but then also it's not just about, you know, a single goal or a company-wide P90 for us.
[00:30:15] Nik Sudan: Different teams work in different ways on different services and different repositories with different repositories and, sorry, different dependencies, right? And some are slower for entirely legitimate reasons. Um, so comparing all of these across one global number, ultimately meaningless, right? Um, and this is where kind of this like, you know, "Hey, here's an average for the whole company."
[00:30:39] Nik Sudan: Like that's not how... Uh, m- you know, maybe if you're a really small team, but if you're-- But once you start to productionize things, once you start to-- start being a, a company with hundreds of thousands of engineers, you need to understand the different domain contexts as well, right? So identify the right development groups upfront, isolate them [00:31:00] in LinearB, you know, sometimes at the team level, sometimes subgroup, like a front end or a back end or mobile, which scope to relevant repositories. And then treat the industry average as sort of like the default benchmark, you know?
[00:31:13] Nik Sudan: And if a team comes up red against it, um, we don't just go, "Oh, so that team's rubbish." No, look into it, 'cause sometimes it might be explained like, "Hey, our way of working just, it doesn't a- it doesn't work with this stuff." And that's a fair outcome, right? And that's the context, and that's what you need to identify before.
[00:31:33] Nik Sudan: Or, or, or the, before you kind of move over to stakeholders, non-technical stakeholders, like, you have to do this groundwork to contextualize it. Um, you know, what is their green line? Set that goal, right? So that's the contextualization preamble. But then you have to translate it, right? And this is where you can't just give that to non-technical stakeholders and such.
[00:31:58] Nik Sudan: Um, you can't just throw metrics at [00:32:00] people and expect them to understand them. You know, I think many people, especially like at C-levels and stuff, director level, get many dashboards which are just what, like a bunch of numbers, red and green. Like, you know, if they don't understand it, they don't trust it, you know?
[00:32:15] Nik Sudan: And that's, it's completely worthless. And, and if they don't trust it, um, they won't act on it, right? So what, you know, what is the whole point by doing that? So just for me, describe everything in plain language. Cycle time, I won't assume people know what it is, even if they're an engineer or not, right? Um, 'cause cycle time actually, depending on what, um, metrics, uh, aggregation do, it could be different.
[00:32:38] Nik Sudan: Some people go to merge, some people to deploy, you know, what, what does it mean? So, um, describe it in plain language. I want, uh, so I'll say basically, look, how long it takes us to ship something in front of our clients, and link it to things that we care about. You know, like cycle to a higher cycle, uh, well, a lower cycle time equals faster project execution speed.
[00:32:57] Nik Sudan: And it's like, "Oh, okay. Right. Right." [00:33:00] So like, you know, there's, there's the technical term, but then there's like the non-technical term. And also the non-technical term helps out engineers as well 'cause, um, translate anything to basic language, um, and help it be contextualized, then it, you know, it all becomes clear, right?
[00:33:17] Nik Sudan: So what does this mean in practice? How do you do this actually? So, you have to make it concrete. Um, so, you know, product design, operation leaders, uh, this number is high for this team, which means this project is going slower. Like get-- actually sit maybe in a call or in them with them, just walk through them.
[00:33:35] Nik Sudan: No stupid questions. Bring it to that table. Many people might dismiss it as like, "Uh, we don't need to run through these engineering things," but, you know, this project is moving slower. They don't wanna hear that. You need to make sure you translate it into that terms. And then you say why as well. And don't just stop with a diagnosis, right?
[00:33:51] Nik Sudan: Call out specific areas that may need improvement, so you can give actions and insights rather than it being red light, right? And [00:34:00] doing this, kind of helping with these leaders, bringing to them, being that translation, kind of Google translate for them, if you will, unlocks prioritization of like engineering driven initiatives, getting non-technical stakeholders to understand the reasons behind the metrics will actually allow you to get time to work on tech debt, right?
[00:34:18] Nik Sudan: And engineering initiatives, get them onto a roadmap that's already full of, you know, product initiatives, or we need to build this, we need to build that, you know. Every company knows how that is. Um, so that's how you do that. If leadership understands red, what it means and why, you can make the case for carving out that time, right?
[00:34:34] Nik Sudan: And without that transition, they're not gonna give... They'll be like, "Oh, they don't need to work on that. We need to ship that. That's gonna make money." But if you can conceptualize it, translate it saying, "But this is money," you know, that's it. That's why product have all their KPIs and stuff. I know as en- engineers, we don't...
[00:34:49] Nik Sudan: We, we hate hearing, you know, KPI, OKR, all these acronyms. But you know what? If we wanna get them on board, we have to speak their language, right? Um, and then [00:35:00] underneath all of it, you know, trust and adoption. You know, don't compare teams to each other, gamifying it. That's the wrong move, and that's the first move that many people make, right?
[00:35:08] Nik Sudan: Evaluate each team against their own benchmark, not a leaderboard of like best versus worse. Kind of the message to each team is like, "Hey, this is for you. This, these metrics are for you." Um, you know, if you're green, great, nothing to worry about. And if you're not, it's a signal to kind of step up. And the right use of the data is like, what, what's worth learning from it, right?
[00:35:30] Nik Sudan: Is it the internal team working under the same constraints, which beats the generic industry fix every time like You know, just trying to, you know, make it contextual for each team. Um, and I think this matters most with leadership who are often quick to look at, you know, who's performing worst and then, you know, cut them.
[00:35:51] Nik Sudan: You know? That's wrong. Um, you need to make sure that's translated to them as well, right? 'Cause a metric is a metric, but then the rationale as to why that's happening. A team at the bottom is [00:36:00] usually one suffering from friction,
[00:36:02] Andrew Zigler: Right
[00:36:03] Nik Sudan: or they need engineering investment, you know? And it needs non-technical leadership to give it that time, that backing to fix it, not the chopping block, 'cause if you chop it, you're just gonna make the problem worse, right?
[00:36:16] Andrew Zigler: It, it highlights the opportunity because
[00:36:18] Nik Sudan: 100%.
[00:36:19] Andrew Zigler: the end of it, then now there's a new end of the problem 'cause you haven't addressed the core of why that, that might be red, right, for
[00:36:26] Nik Sudan: Yeah
[00:36:27] Andrew Zigler: like, I, I really... It's really brilliant, Nik, how you think about bringing everybody into the conversation, not only like translating all the technical jargon to the engineers, which is like what we're really...
[00:36:38] Andrew Zigler: Or to like the non-technical stakeholders, which is like a really big driving function of LinearB, right? But also to using it to contextualize those higher level decisions and things like KPIs and OKRs to the engineers and help them map the work that they're doing to that direct impact, to what we're shipping and to why that matters for our [00:37:00] organization.
[00:37:00] Andrew Zigler: And ultimately, like, by using a platform like LinearB to have an informed, shared conversation, you can build trust because now, like you said, those leaders can trust and understand their dashboard. They might not be as fluent as you in talking about the healths and the metrics of their, of their conversation, but y'all now have like a pigeon language you can operate in.
[00:37:24] Andrew Zigler: You, you have like a, a way to c- get each other's point across and then also effectively get the work done. And for a lot of engineering teams, that this is a completely missing in- ingredient. It was before, it is now, and now we're trying to go even faster. And as you've rightly called out, most of the problems we're bumping into are people and process problems.
[00:37:45] Andrew Zigler: You need to have everybody equipped with the same language and ability to operate on the same kind of data, right? So I, I really love all of these callou- call-outs. I think it's a really, like smart way of understanding problems, uh, even just like things [00:38:00] like, uh, using this, uh, MCP data layer to curate your own like P90 benchmark layer, right?
[00:38:05] Andrew Zigler: So you can understand these specific points because that was a priority for you. And the, the beauty of this is it can be arranged to be a microscope onto any kind of problem that you might think exists in your org. And if you are con- fluent in the data, then you're not gonna have this whole, uh, you know, can I trust the hypothesis situation because it's like grounded in the truth of your engineering org.
[00:38:27] Andrew Zigler: It's all a really great kind of, um, I think like a playbook for how to use LinearB to, to like ship things at scale, but then also have, uh, conversations where everybody in that Zoom meeting or whatever, they all understand what we're talking about because for the
[00:38:43] Nik Sudan: Yeah
[00:38:44] Andrew Zigler: all now fluent in like the health of our org. Um, so really great kind of like way of putting that all together. I do wanna say, um, you know, we've covered a lot of ground today, and just before we wrap up, are there any, you know, last parting thoughts or, or words that you have for [00:39:00] folks, um, that maybe who are listening to this conversation and wanna have a bite of that success as well?
[00:39:05] Nik Sudan: One word, four letters, data. You know, it's everything. If you're not measuring AI effectiveness, adoption cost, and relating to that as well, you know, the quantitative engineering data like code throughput, DORA, then it's all worthless, 100% worthless. And this has to happen on day one, because if you're not doing it already, like, you know, it's going to be incredibly costly later on.
[00:39:28] Nik Sudan: Start it now. Like, prioritize this. Like, after listening to this, just do it. Um, you need to be able to prove your point with data, um, and adopting kind of AI, adopting all these tools is not as simple as like, "Hey, we're gonna procure a new code editor or a license for some framework." It's a whole new way of working, right?
[00:39:46] Nik Sudan: And you have to, you have to measure it like a, a new way, right? But it's not just AI specific, right? It's a chain, like data, context, insight, action, right? That's the layer. Um, [00:40:00] e- each layer kind of gets built on it before, and most teams actually stop early. So, you know, raw metrics are step one, data ingestion.
[00:40:07] Nik Sudan: Good. You've got it. You pull most of it where code lives, and tools like LinearB help support this, right? Um, core metrics, throughput, cycle time, review time, et cetera, you know. This is the part most teams are doing right now. Um, but the context part, this is the second part, derived from the foundation.
[00:40:23] Nik Sudan: You're bringing the context sources that enrich that quantitative data, you know, could it be qualitative? You know, like what are people saying? What projects, what are the actual projects, the company's priorities, you know? What's the expected end outcome of all of this? Financial impact, you know, adoption impact.
[00:40:37] Nik Sudan: Metrics will tell you how fast your engine's moving, but context tells you, you know, are we going in the right direction, right? And then with that data and context gives you insight, right? Someone using AI more doesn't mean they're great. You know, as we said before, anything, heavy usage is a signal to go and look what they're doing, right?
[00:40:56] Nik Sudan: You know, are they actually doing it? Is everyone using it well and saying it's brilliant, but [00:41:00] you don't see a business impact, something's going wrong, right? Uh, people hacking away on their own little projects to improve their own little processes is fine, but needs to show results, right? You know, it's so tempting to work on like, "I'm gonna revolutionize how the company works and roll it out to one person", which is yourself.
[00:41:15] Nik Sudan: No, it's not. You know, if it's not doing that, rethink it 100%. And then this is where kind of the intentionally, intentionality matters, right? Someone using AI more doesn't mean they're great. And I think with that, keep it tight as well. Leave room for experimentation, but make sure generally, it's generally, you know, pushing your engineering organization forward by finding out what is working and what doesn't, and back up all of that data, all of that qualitative data with the quantitative data.
[00:41:45] Nik Sudan: Surveys like space surveys are pretty good for this as well. That's what I recommend, and there are many other frameworks out there like, you know, DORA is something, but although that's more quantitative. And then, plenty of reading materials to look at. The numbers tell you what's happening, but then the people tell you why, [00:42:00] right?
[00:42:00] Andrew Zigler: Mm-hmm.
[00:42:01] Nik Sudan: you know, if I was advising someone, the, the, the real kind of final line here, measurement in place day one. Make sure it spans both the raw metrics and the context that enriches it. Keep it all intentional and build towards that insight and action rather than just looking at the numbers and stopping at the numbers, right?
[00:42:20] Nik Sudan: The tools then will thrive with all of that.
[00:42:23] Andrew Zigler: Well, I think this has been a really great view into how y'all operationalize with AI as well as how you think about working with data. You know, data is really the, the king here. It's gonna allow you to, to operate, uh, and soar at this level. So thanks for breaking it all down for us, Nik, and, uh, we're-- you know, it was really great chatting with you.
[00:42:40] Andrew Zigler: I hope to have another conversation with you in the future, and thanks again for joining us today
[00:42:45] Nik Sudan: Thank you very much.
