Capital One SVP Abhijit Bose, who leads the company’s enterprise AI and ML platforms, has spent more than 20 years in AI and ML, and his answer is to make governance the platform’s job instead of the developer’s. This week on Dev Interrupted, he joins Dan Lines for LinearB’s AI Enablement Series to walk through how every harness change at Capital One now runs through evals that measure token cost, latency and quality before it reaches users. Abhijit also breaks down MACAW, the bank’s multi-agent framework behind Chat Concierge.
Show Notes
- Chat Concierge: Driving the future of car buying with agentic AI. The multi-agent car-buying assistant that runs on Capital One’s GenAI platform.
- MACAW: Capital One at EMNLP 2025. Capital One’s introduction to MACAW, the multi-agent framework behind Chat Concierge.
- VulnHunter: Announcing VulnHunter. The open-source agentic code security tool Abhijit points to when talking about AI and cybersecurity.
- Connect with Abhijit: LinkedIn
- Live Wed Nov 18, 10am PT: Dan Lines and Ilan Rado walk through LinearB’s new Agentic Studio on real data in “Deliver outcomes when human attention can’t keep up.” Andrew Zigler hosts. Register here.
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Dan Lines: Hey, what's up, everyone? Welcome to Dev Interrupted. Today's guest is Abhijit Bose, Senior Vice President and Head of Enterprise AI and ML Platforms at Capital One.
[00:00:13] Dan Lines: In this episode of LinearB's AI Enablement Series on Dev Interrupted, we're picking back up the story on Capital One from earlier this year when we had Ameesh Palasia on the show to talk about AI adoption. Abhijit is going to share how Capital One takes an AI-driven approach to building software, uh, governing and instrumenting its own multi-agent systems on one enterprise platform, and the lessons learned that take a pilot to lasting value.
[00:00:47] Dan Lines: So Abhijit, thanks for joining us today.
[00:00:50] Abhijit Bose: It's,
[00:00:50] Abhijit Bose: it's a great pleasure, Dan. I'm a big fan of the show.
[00:00:54] Dan Lines: Yeah. Oh, thanks for saying that, man. Awesome to have you on. And, you know, [00:01:00] we looked up, obviously, a lot of background about you and the really, I think, interesting stuff that's going on at Capital One, and obviously, I think the whole world, like all engineering leaders, VPs of engineering, CTOs,
[00:01:15] Dan Lines: we're all going through this AI transformation one way or
[00:01:18] Dan Lines: another.
[00:01:20] Dan Lines: And so I just wanted to start us off by asking, you know, what's kinda top of mind for you right now as you change how your teams build with AI, and what did that, like, uh, same type of work look like maybe a year ago compared to now?
[00:01:37] Abhijit Bose: So first of all, you know, I have been doing AI ML for like almost more than 20 years, and I have never seen a time like this as, as we all know.
[00:01:49] Abhijit Bose: Um, you know, AI is a huge, uh, business transformation for us. Every company is going through that, that phase change. for us, you [00:02:00] know, when it comes to AI, it's like how do you scale AI in a regulated, um, enterprise like, like banking? That is really important. How do you-- Like we have a saying i- inside the company, like how do you stay at the frontier of AI with the speed of well-managed?
[00:02:17] Abhijit Bose: And, you know, you don't wanna get slowed down, uh, but at the same time, how do you transform your basic, you know, risk and cyber systems? As you know, like if you have to unleash a bunch of agents within your environment It's not just enough to create a runtime, you know, to let the agents work, but you have to worry about which tools they will access, you know, what modifications they can make to the file system and all that.
[00:02:43] Abhijit Bose: So we have to think through a lot of those things to really scale AI within the company. just between last year and this year, personally, you know, my workflow has changed. I think most people's
[00:02:57] Abhijit Bose: workflow
[00:02:57] Abhijit Bose: have, uh, workflows have changed. [00:03:00]
[00:03:00] Abhijit Bose: We
[00:03:00] Abhijit Bose: are seeing so much AI first, you know, kind of mindset.
[00:03:05] Abhijit Bose: Like, a- and, and often the work itself has changed. You know, like code generation, for example, it's way more automated than, you know, even we w- where we were last year. Uh, but now human judgment is still quite important. Like, how do you pick the right intent? How do you experiment more? You know, where do you have the developer still engage with the AI?
[00:03:29] Abhijit Bose: For example, designing the harness properly, designing the agentic loop, uh, guardrails. You know, a lot of those things are more important than last year. So it's been a profound change. Like, I think it's gonna change, like, how we build products, how product managers, engineers work together. You know, the lines are even blurring in some cases.
[00:03:52] Abhijit Bose: So very fun and interesting times that we are living through.
[00:03:56] Dan Lines: That's super cool. I've... That y- that you've been doing this, like, 20 [00:04:00] years, right? Yeah, I think that's what you mentioned, about, like, 20 years, and probably in the last, like, two, three years. I don't know maybe what was happening for the first 17 years, but there's been- Oh, first 17 years.
[00:04:11] Abhijit Bose: Absolutely ...
[00:04:11] Dan Lines: like, rap- probably some rapid, you know, developments in the last, I don't know, two years or so. And you brought up so many great topics. Let's see where we dive in because you talked about, okay, speed versus quality, right? You also, I mean, and, and I think what's pretty cool, we have a whole topic on this, is you're in a regulated industry.
[00:04:33] Dan Lines: Like, a lot of the, the customers that I work with, yeah, different banks or financial institutions, we call them kinda like these large regulated enterprises, which a lot of our, uh, listeners are in. It's almost like you have an extra constraint that we're gonna need to dive into it too as well there, right?
[00:04:52] Dan Lines: Yeah. But I think, maybe where, where we'll start is let's start on that maybe, like, speed and quality [00:05:00] side or, or something around there. Like, where do you see the biggest gap today maybe between where engineering organizations might think they are with AI versus the reality? One thing, and I'll,
[00:05:15] Dan Lines: uh,
[00:05:15] Dan Lines: one thing that I just wanna throw in here, at least the ones that I'm working with and some of the data, a lot of the companies right now, there's a ton of code generation going on.
[00:05:25] Dan Lines: Co- code generation, code generation. I, I think a lot of us are there now. But one of the gaps when I'm talking to engineering leaders is I'll ask them, "Well, how much of that is actually making it out to production?"
[00:05:37] Abhijit Bose: Absolutely.
[00:05:38] Dan Lines: And how much is it making out, like, in a high-quality way? And sometimes they'll have a tough time answering that question.
[00:05:46] Dan Lines: They know that there's a lot of, uh, agentic code generation going on, but we don't know exactly what's happening after that, and that's where I see a bottleneck. What are you seeing in terms of like, okay, kind of like, [00:06:00] uh, the reality versus perception, um, with the AI SDLC?
[00:06:06] Abhijit Bose: So, you know,
[00:06:08] Abhijit Bose: last year or even, like, earlier this year when we turned on, you know, like, one of these agentic coding tools, you know, Claude Code, for example, we saw a huge rise in AI-generated code and everybody jumped in.
[00:06:23] Abhijit Bose: I think the biggest gap today are actually in few different places. One is the outcome. You know, so we are-- Yes, we are generating a lot of code. Um, every company, you know, uh, all the engineers are probably more productive. They're experimenting more. But does it, you know, result in faster product delivery?
[00:06:45] Abhijit Bose: You know, does it result in more features, you know, like, uh, enriching, you know, enrichment
[00:06:51] Abhijit Bose: of the,
[00:06:51] Abhijit Bose: of the product itself? I think those are the things that we are starting to measure. Uh, like, what's the-- How do you tie the [00:07:00] outcome of, you know, uh, SDLC or PDLC, uh, with the all of the AI enablement that's happening today?
[00:07:10] Abhijit Bose: Uh, so tokenmaxxing is not the right, you know, for example, like, the, the right metric .
[00:07:16] Dan Lines: Yeah, we're beyond that now. I think we moved pa- We moved past that, right? We were at token at tokenmaxxing for a bit. I think we moved- Yeah ... we're beyond that now, hopefully, as in,
[00:07:24] Abhijit Bose: you know. Well, I, I think we are beyond. Uh, it's actually- Yeah
[00:07:26] Abhijit Bose: the reverse problem now. We are all trying to figure out, you know, how to manage-
[00:07:29] Dan Lines: Save the money ...
[00:07:30] Abhijit Bose: optimize, right, the, the cost of, uh, tokens. But, uh, I think the outcome is gonna become, like, a really solid, you know, measure, uh, that every company will be, um, you know, probably putting in place. So we are probably a little bit ahead of other companies because we are starting to put those systems in place, like measurement systems.
[00:07:54] Abhijit Bose: The second gap that I see is that a lot of companies, [00:08:00] probably developing a lot of AI applications, you know, including they are deploying a lot of agents, but they are not Putting the time or resources into evaluation frameworks. Uh, so the evals, you know, have become like the core focus area for, for This year, you know, we have been very serious about putting together the evaluations frameworks and eval infrastructure. We'll continue to invest more next year, uh, to the point that, you know, these days, um, in our software enablement framework, we have all the coding tools, you know, all the different harnesses that we put in.
[00:08:40] Abhijit Bose: every change now goes through an eval with our both custom datasets for benchmarking as well as industry, you know, benchmarks that we run. So that way we can do good trade-off between different models, you know, different, uh, harnesses. Uh, and we wanna codify that, [00:09:00] you know, before we put in a particular harness, uh, into production, into the hands of our users, I want to know like what it'll cost me in terms of tokens, you know, what the latency would be, what the quality threshold would look like.
[00:09:15] Abhijit Bose: And so we are getting better and better at that. So I feel like that's, that's not always, it, it has not always been true for us also, but I feel like those are two big gaps.
[00:09:26] Dan Lines: Yeah. I mean, I... And, and it sounds like, you know, you are-- Yeah, you're probably more mature in the way that you're thinking about it, I think, than, than most, especially with the evaluation framework.
[00:09:39] Dan Lines: I think we were probably previously at a time that it was like, "Hey, let's just make sure AI is generating a bunch of code." And, you know, I, I... It sounds like from talking to you here, you went through that phase maybe a little, a little while ago. I would love to hear about the evaluation framework because that's maybe the next step [00:10:00] here, and hopefully we, you know, you probably have quality in mind when you're thinking about it.
[00:10:05] Dan Lines: But tell me about- Yes ... tell me about the evaluation framework and then also tell me, you know, what metrics are, if, if you have any, do you need to report on? Like do you report, you know, the results or like the efficiency and the cost back to the business?
[00:10:21] Abhijit Bose: So with AI cost, we have a slightly different, uh, approach.
[00:10:27] Abhijit Bose: You know, we, we don't, you know, like, we, we are not like top-down in terms of like, "Oh, you have to manage your AI cost, you know, this way." Or, you know, in, in some places, you know, there is like a strict limit on how many tokens you can use and all that. So we are not there yet, um, because we feel like this is a long-term investment, and we are in a phase where we need
[00:10:52] Abhijit Bose: to do
[00:10:52] Abhijit Bose: a bunch of experiments to try out different models, you know, different ways of working.
[00:10:58] Abhijit Bose: Uh, even to train [00:11:00] all of our associates, you know, with, AI, we need to have those investments today. So we are not tracking like the cost as a, as a strict measure, but we do, of course, track our token usage and all that, and we try to understand why some people are using, you know, so many tokens. Like, what's the outcome of that?
[00:11:21] Abhijit Bose: Or is it just, you know, taking an entire GitHub repo and putting it into the context window? Uh, so the-- One of our recent products actually came out of, uh, you know, because of that thing that we saw, that a lot of people are not using the context window efficiently, so we started building our own context retrieval engine that we are gonna roll out to the entire company now.
[00:11:45] Abhijit Bose: so but going back to the eval, we have the Gen AI platform. Uh, this was, you know, put in place about two years ago now with the RAG as the only use case, the first use case Now we have a full set [00:12:00] of capabilities on it. Uh, one of the things is to be able to do evaluations. Um, part of that, we have our own SDK that you instrument an application end-to-end.
[00:12:13] Abhijit Bose: Um, and that puts all of the data from that application, from the FM hosting, like LLM hosting, multi-turn agentic loops to, data coming from the tools. All of those things get, put into a one-- like a lake, um, that we manage on behalf of users with tracing IDs throughout so that you can concatenate all of those events into an end-to-end view of how the application is doing.
[00:12:44] Abhijit Bose: And then for each application, we define a set of metrics. If it's a, you know, if it's a document summarization agentic workflow, there are metrics like how many documents, you know, what's the retrieval accuracy? you know, if it's [00:13:00] reading like OCRs, um, you know, there are like, precision metrics that we can come up with for that particular use case.
[00:13:08] Abhijit Bose: And then we have a, uh... So there are like bunch of pipelines that people have to build for that application with our, you know, starter kits that take the data and, and build those insights. And then we have-- We are not fully there, but we have a visual way of seeing all those metrics on a dashboard. and then we use a third-party packaging, you know, like package software to do that, uh, as your kind of front door, uh, to see how your metrics are doing, um, both during designing the agent, like offline before you deploy, and then once you deploy, it gets the data from the production system.
[00:13:48] Abhijit Bose: So You know, it, it seems like it, it's quite advanced, but I will tell you that it's a work in progress. You know, we have done this for a few major use [00:14:00] cases that we have shipped. Now we have to democratize it for the whole company so that my team and I, we are not in the, you know, in the, in the user fl- u- user journey here.
[00:14:10] Abhijit Bose: It should be all self-service. So that's the work that we have to do over the next few in terms of- I
[00:14:16] Dan Lines: love it ...
[00:14:17] Abhijit Bose: yes.
[00:14:17] Dan Lines: I love it. That's so cool. Actually, I mean, Capital One sounds like a good place to work right now. This sounds fun, right? It
[00:14:25] Abhijit Bose: is, it is, it is quite fun. Yeah. We have been very lucky, by the way, with our, with our talent.
[00:14:30] Dan Lines: that's awesome. Yeah, I mean, w- first of all, I, I like what you said around-- I mean, you gotta spend money to do these experiments. Like, AI is, uh, everyone is trying to save right now, but AI costs money. If you wanna change a workflow and to be more, like, AI-driven, like, you gotta pay to play.
[00:14:48] Dan Lines: A
[00:14:48] Dan Lines: lot of our, our customers come in, they wanna, you know, talk to
[00:14:51] Dan Lines: me
[00:14:52] Dan Lines: about the cost of different models, comparing the different models, the cost of tools.
[00:14:57] Dan Lines: How do I bring down [00:15:00] the cost? That's usually, like, one part of our, our conversation that we're, uh, I'll help with. But yeah, on, on the flip side, you do gotta spend if you want to innovate and get to where you are and do these tests and that type of thing. And you were just talking a lot about context. You said you were building, I think you're, like, almost building a context layer or maybe a data lake.
[00:15:22] Dan Lines: one thing that we're, we're doing at LinearB, at least for our customers, is actually sending their agent context. We usually send things like incidents found in production. Is there a live incident going on? What's the rework rate of the code area? Is there anything gonna get merged? All this, like, additional context that we're able to look at, and these are the things that allow an agent to say, "Okay, maybe I don't need to work as hard.
[00:15:48] Dan Lines: I'll select a lower effort," or, "I'll change my model." And that's been pretty effective. Like, I think that's going super, super well. I was wondering for you, what type of [00:16:00] context do you, if you're, if you're able to tell us, like, what type of context do you put into your data lake? Are you trying to send to your, your agents to be more optimized right now?
[00:16:11] Abhijit Bose: So there are two distinct kind of groups of use cases you can think of, uh, in, in, in terms of context. One is, of course, software development. You know, that's where we have GitHub, Jira, Confluence, you know,
[00:16:26] Abhijit Bose: all
[00:16:26] Abhijit Bose: the Google Docs, PDF files for, you know, PRDs to technical specs and all, all of that. Uh, and more recently we are seeing that, you know, Slack actually becomes a lot of source for good context.
[00:16:41] Abhijit Bose: Um, a lot of the Slack channels that people put in for, you know, like SRE functions for a platform-
[00:16:48] Dan Lines: Yeah, like all the alerts, alerting- You know, yeah ... and information that gets sent there. Yeah, yeah.
[00:16:53] Abhijit Bose: And, and kind of knowledge sharing between, you know, SREs, you know, they're using Slack channels. so all of that, [00:17:00] you know, for software.
[00:17:02] Abhijit Bose: For other types of agents, you know, for like, um, traditional document processing, back office, risk, um, you know, a lot of other multi-agentic frameworks, they are-- it comes in two forms. One will be the APIs that they are calling, you know. So all of the information, uh, we are lucky to have a central point, you know, where all of our APIs are registered and, and orchestrated. So, uh, we can index all of that information beforehand and, you know, essentially only supply the needed information to the agent on the APIs and tools they, they will be calling. and then the other information is the business logic, like the business policy. Like if you're in a call center, and you are deploying an agentic AI application in your call center, there are so many operating manuals, you know, where the business logic policies are encoded, [00:18:00] and some of them are compliance.
[00:18:02] Abhijit Bose: You know, like how do you handle a particular call or a situation and all of that. Uh, so all of that also get, get indexed. Uh, and then, you know, we have this multi-stage ranking system that takes all of this information, does a re-ranking over all of the chunks that it retrieved, and then finally the, the top end chunks are sent to the agent and then that eventually the harness packages for the LLM to, to process.
[00:18:32] Abhijit Bose: So that's kind of the way it works, you know, among the different use cases.
[00:18:37] Dan Lines: That's super cool. I mean, I, I think we all know, but it's like context is king. These agents, they work so much better when they have the better context, can save money, quality increases. So I really appreciate you kind of bre- breaking down how you think about it.
[00:18:53] Dan Lines: I think that's really important for the audience. I have a note here, that you've built your [00:19:00] own multi-agent workflow.
[00:19:02] Abhijit Bose: Yes.
[00:19:03] Dan Lines: we'd love to hear about that, and also in particular, what parts of your SDLC are actually running agentically today? You know, are, do you have agents talking to each other, passing off work?
[00:19:18] Dan Lines: And also, you know, on the more humble side, what do you not have, you know, running with agents? Like, what's been really hard, hard for you?
[00:19:27] Abhijit Bose: Yeah. So that, uh, three-part question. Um, so let me answer, you know, the multi-agentic framework, what we call MACAW. Uh, and frankly, I think, um, I think we were the first company to actually put a mul- multi-agentic framework in front of customers back in 2024, you know, when there were not too many multi-agentic, like, you know, customer-facing applications.
[00:19:55] Abhijit Bose: So it's a, it's a very general, general framework, where there is [00:20:00] an understanding agent that, uh, works with a planning agent, and then these two agents work with the user to disambiguate the intent. You know, so they go back and forth between, between the user and the understanding agent and the planning agent to, to fully create a plan.
[00:20:20] Abhijit Bose: And then there is an evaluating ag- evaluation agent that actually verifies the plan, and it runs some sandbox to check the tools that the plan, you know,
[00:20:31] Abhijit Bose: generated
[00:20:32] Abhijit Bose: to, to be called by the LLM, for example, of the agent. and then finally, there is the actual, you know, like response agent that'll either go back to the user or actually take the action on behalf of the user.
[00:20:46] Abhijit Bose: Uh, you know, for example, if I say like, "Book this, you know, time with my, uh, with the, with the dealership," like for a car dealership, for example, uh, it will go ahead and try to call the, [00:21:00] you know, booking API with the dealership to, to set a time and date for your visit to the dealership. So we built a bunch of applications actually, uh, with this generic framework.
[00:21:13] Abhijit Bose: The first one was the, in our auto business, you know, like called, uh, Chat Concierge. So a lot of dealers in, you know, in North America now have this thing on their, either on their app or website. It looks like an assistant. You can browse cars, you can ask anything about, you know, cars, different cars, different choices And you can say like, you know, "I want to test drive this particular car, you know, please give me a time and date," and it'll book the car, book the time for you and all of that.
[00:21:46] Abhijit Bose: Um, not many people realizes that it actually runs on our genAI platform on the back end.
[00:21:51] Dan Lines: No, that's cool. I had
[00:21:53] Abhijit Bose: no idea. Automatic interaction, you know, happening in real time. Uh, and when we first shipped it, I think 12, [00:22:00] uh, second latency. so literally 12 second will go by, you know, before the, before you got a response back.
[00:22:07] Abhijit Bose: Uh, but nowadays, um, it's, you know, we are targeting for less than a second, uh, latency. So we have come a long way. Uh, that's the other point I was gonna make, is that deploying an application, you know, is easy, but to scale it and to make it, like, really useful to our users, uh, with multi-agentic frameworks, uh, with multiple LLM calls, it's actually very hard.
[00:22:31] Abhijit Bose: Uh, so to the point that we had to build our own model and we had to build our own inferencing stack to actually make it possible end-to-end.
[00:22:40] Dan Lines: Yeah, I mean, this one's... That's interesting. This one is the, at least the example you gave, there's a user waiting on, it's real time. There's a user- It's real time
[00:22:51] Dan Lines: waiting on the other end, right? So 12 seconds, you know, it maybe doesn't sound that much like- You can imagine the wait
[00:22:56] Abhijit Bose: in the early-
[00:22:57] Dan Lines: Yeah ... early
[00:22:58] Abhijit Bose: version of our product. Yeah. [00:23:00]
[00:23:00] Dan Lines: 12 seconds is kind of a lot when you're thinking, "Okay, I, I'm waiting for, for the response," right? Yeah. So you have to scale that. Okay.
[00:23:08] Dan Lines: And I didn't even realize that the way that you've built this, it's more like a generic system, right? That you can apply to... You gave one use case of, of the book- It can be many different use cases. Okay.
[00:23:20] Abhijit Bose: Yeah. Yeah. So we, we have done this for document summarization, you know, document, uh, processing. Uh, it's becoming a qu-quite a, quite, quite a nice generic framework for us. yeah.
[00:23:34] Dan Lines: Got
[00:23:34] Abhijit Bose: it. Actually, I would say like Jensen in 2025 actually featured this in, uh, GTC as one of the, you know, few frameworks that are out there as a generic, um, multi-agentic framework.
[00:23:49] Dan Lines: That's, that's awesome. Are you using this? So I understand like, okay, hey, this can be used in an actual app, you know, interacting with it.
[00:23:58] Dan Lines: Are you using this [00:24:00] in your SDLC as well, or no? Not yet in our SDLC. Like for for your internal development? Okay.
[00:24:05] Abhijit Bose: Yeah, not yet, uh, SDLC, but a lot of the customer-facing use cases and even internal other use cases, non-software like back office-
[00:24:14] Abhijit Bose: we have been using it. Um, yeah. To, to come to your other question, you know, like whether there are multi-agentic frameworks in SDLC, there I have one example where we have built, you know, essentially a pretty elaborate code review system- that does a bunch of things, you know. Where one agent will look at your, for example, architectural integrity of your code. Um, you know, another one will do a s-cyber review.
[00:24:47] Abhijit Bose: Uh,
[00:24:47] Abhijit Bose: another one will do stylistic checking of your code. Uh, and these agents also have verifier agents. Um, you know, uh, they will check, and then finally your [00:25:00] PR will be approved, um, you know, or at least get the first pass through the system.
[00:25:06] Abhijit Bose: Um, and then, um, eventually, you know, it'll go to a human. We are still in the human-in-the-loop, for our software deployment. but
[00:25:15] Abhijit Bose: you know, we, we do a AI first review of the code base, uh, particularly the PR, um, um, before it even goes to a human.
[00:25:26] Dan Lines: This is what I'm seeing, actually what you just said, for a lot of the, I would, I would call them like the enterprises or maybe the regulated, uh, industries were right there where you said, "Hey, there's a human in the loop," which means, okay, we probably used a bunch of AI to generate the code.
[00:25:45] Dan Lines: Now we do our AI code review. Maybe I have something built in-house, maybe I'm using one of the popular review tools. But right there is where I see a lot of companies saying maybe we're not ready to go fully [00:26:00] agentic. We still need a human in the loop. Now, for you, is that, is that something to do with the governance or the sensitivity around being a re- in a regulated like financial, or do you see it as something that, hey, we can't fully trust AI yet?
[00:26:20] Dan Lines: Like where does that all sit for you that, you know, you feel like you need the human in the loop.
[00:26:25] Abhijit Bose: I, I don't think it's a trust issue, but, uh, you know, we do have a higher bar, you know, when it comes to, um, governance. That we want to be absolutely sure of something, you know, that would be operating in an automated manner.
[00:26:41] Abhijit Bose: So we have to convince ourselves that, you know, we have taken into consideration all manners of risk, you know. And there are areas where we may not, you know, always have an autonomous system. You know, those things will always be human in the loop. But the biggest gap that [00:27:00] we have today is that when you have a piece of code and you want to deploy to production, what is the risk of putting that code into production?
[00:27:11] Abhijit Bose: You know, would it cause an outage? You know, um, looking at similar code bases, uh, looking at similar outages, uh, looking at that type of application, you know, how many users are gonna be impacted by that, that particular change, you know. Those things, depending on the scale of it, will have a higher bar.
[00:27:31] Abhijit Bose: In some cases, there might be a human in the loop. In some cases, we might let that go automatically to deployment, maybe behind a feature gate, and then the feature gate would be the last, you know, human in the loop, uh, stage. But rest of it would be automated. so that's where we are. Like we are, we are literally doing a bunch of experiments to figure out what's the risk, you know, uh, level associated with different types of tasks that we [00:28:00] do, and what's the best way to mitigate that.
[00:28:03] Abhijit Bose: And, you know, in some cases, as I said, always human in the loop. In some cases, you know, there could be opportunities to automate, you know, certain deployments.
[00:28:12] Dan Lines: That's so funny that, that you said that, 'cause yeah, at LinearB, that's exactly what we're working on. Like the pain point for most of our customers is what you said, "How do I do an impact and risk assessment of this code that I'd like to get to production?"
[00:28:30] Dan Lines: I'd love-- Most of our customers would love to be in a situation where there doesn't need to be a human in a, in the loop. But oftentimes, depending on the risk score, I may need to route it to a human for review or to a security team or maybe multiple people. Um, that's where I think like the, I would say, the next pain point is, uh, for the industry.
[00:28:55] Dan Lines: It sounds like you're doing something maybe built in-- Like you're doing your own risk assessment or- [00:29:00] Yeah ... impact score and then rou-routing it. Is that- Yeah, because- Are you doing a build in-house? Yeah.
[00:29:05] Abhijit Bose: We, we are, we, we are building that risk scoring infrastructure right now.
[00:29:10] Abhijit Bose: Um,
[00:29:11] Abhijit Bose: you know, and then when you couple that with long-running agents in agent runtime, for example, on the cloud It's not just the deployment, you know, because the agent has to run that risk scoring depending on the step in the process, you know.
[00:29:26] Abhijit Bose: And it's not just deployment, it's a step in their workflow might come back to the human to escalate, to approve something and then, you know, it can go about its way to do the next, you know, task for a few hours. So we looked at this problem holistically, and we realized that, you know, th- there needs to be a risk scoring thing here, you know, whether it's a long-running agent, short-living agent deployment.
[00:29:52] Abhijit Bose: Uh, so for us it's all about risk scoring. And we, we are a bank, so we, we understand risk. Uh-
[00:29:58] Dan Lines: Yeah.
[00:29:59] Abhijit Bose: So [00:30:00]
[00:30:01] Dan Lines: This is your specialty. A lot of, uh- It's been our area ... IP around risk, risk scoring, yeah.
[00:30:07] Abhijit Bose: Yeah. So, w-
[00:30:08] Abhijit Bose: and, and you know, like when you have good telemetry, you can bring in a lot of data To bear into that risk scoring process, like what exactly happening, you know, in our production environment.
[00:30:20] Abhijit Bose: You can be really, uh, good at, predicting risk of, of something, the next step in the, in the process.
[00:30:28] Dan Lines: Yeah, I mean, that's the context, right? That all comes back to context. Yeah. Do I know what's happening in production? Are there any live incidents? What's the sensitivity of the code?
[00:30:38] Dan Lines: Who, what or who built the code? There's a... That's where I think if the, the teams that I'm seeing that are focusing on getting more and more context to each decision point, better outcomes tend to be happening. It's kind of like a Yeah, yeah ...general- Yeah, observation that I have. Yeah. And it seems like you, you kind of have the same mindset.
[00:30:57] Dan Lines: If I know what's going on in production, I [00:31:00] know what's suppo- the more live data that you have coming in, probably your risk score will be better.
[00:31:06] Abhijit Bose: Yeah.
[00:31:07] Dan Lines: Or
[00:31:07] Abhijit Bose: more accurate. But, you know, what I have come to understand, you know, throughout, through this process is that if you have central platforms, that's like a, that's such a blessing.
[00:31:21] Abhijit Bose: Um, you know, like a lot of companies, you know, have two or three or three of the same thing because of different lines of business or just historically, you know, they may have acquired companies or- Yeah. Mergers and
[00:31:32] Dan Lines: acquisitions.
[00:31:34] Dan Lines: Yeah,
[00:31:34] Abhijit Bose: exactly. Lots
[00:31:34] Dan Lines: of different, yeah.
[00:31:35] Abhijit Bose: Yeah. So at Capital One, we have most of the major, you know, kind of work is done through central platforms. Uh, So our founder basically calls this a platform company. and that actually really helps when you are deploying agents in an AI, because you have to do telemetry once, you know, you have to do the integration once. [00:32:00] Um, so I think that's a big sloper, um, in, in AI.
[00:32:05] Dan Lines: Yeah, I totally agree. I mean, instrumenting, let's call it like the software factor, instrumenting is tough.
[00:32:12] Dan Lines: Yeah. Especially when you have disparate tooling. Mm-hmm. The more centralized, the more centralized you can get, the, it still might be tough, but it'll be easier- Yeah ... than if you have lots of different system, lots of different pla- lots of different tools.
[00:32:27] Abhijit Bose: Um- And, and being on the cloud, you know, like we are an AWS, you know, infrastructure, you know, all, all of our infrastructure runs on, on the cloud.
[00:32:36] Abhijit Bose: It'll be very hard to do these things if you have mainframes. you know, like you have to literally probably bring the data into the cloud and do all of this, but it's kind of like a indirect way of doing it, you know, not instrumenting directly with those systems.
[00:32:52] Dan Lines: I saw a few startups that are focused on the mainframe problem now.
[00:32:57] Dan Lines: Let me like ... Yeah, yeah.
[00:32:58] Dan Lines: But that's, you know, it, it's a, [00:33:00] and that, that's pr- ... migration and- ... obviously a real thing, yeah in terms of, uh, governing and instrumenting, but maybe we'll, we'll stay on governing ano- another minute. So what I've learned from, from where you're at so far is that it seems like you're, you're pretty, pretty much AI first almost end to end.
[00:33:22] Dan Lines: You're on that review phase problem in terms of when to bring a human in the loop and when not to, that we just spoke about. Are there other areas of the software development life cycle that you have to, I don't know, do something in particular because you are regulated, or areas that you worry about more or less worried about?
[00:33:48] Dan Lines: Like, you know, what's coming next, I guess?
[00:33:51] Abhijit Bose: So we want to make risk and governance and well-managed, you know, really at a, as [00:34:00] the responsibility of the platform so that our users and developers, you know, can build an application and all of the controls are automatically applied on their behalf. I'll give you an example.
[00:34:14] Abhijit Bose: We have deployed a bunch of coding tools in the company, you know, like, like many companies. But every one of those tools we had to work with the vendor, you know, whether it's Anthropic or, you know, OpenAI, et cetera, to follow a particular architectural pattern. You know, for example, we have an AI gateway in the company that we built ourselves, and a lot of our controls are there, like prompt injection, you know, guardrails are there.
[00:34:41] Dan Lines: Gotcha.
[00:34:41] Abhijit Bose: You don't have to worry as a developer. It's automatically invoked in
[00:34:45] Abhijit Bose: e- every, you know, input/output. That, uh, is a game changer, because you can actually then focus, as a developer, I can only focus on the task, um, and not have, have [00:35:00] to worry about the, the individual cyber or well-managed controls.
[00:35:05] Abhijit Bose: Um, that's, that's one thing that I s- you know, we will continue to make progress on, that how do you basically make your platform central to risk management and, you know, um, controls.
[00:35:20] Dan Lines: Yeah, I love that. I mean, no, no developer wants to be worrying about that stuff ever.
[00:35:26] Abhijit Bose: Yeah.
[00:35:26] Dan Lines: That would, that would suck. Then it's like, "I can't even really do my main job."
[00:35:31] Abhijit Bose: Yeah.
[00:35:31] Dan Lines: So it's not-- okay. Yeah. No, great tip there. I mean, yeah, having that like, uh, taking that off the developer's plate is- Yeah ... uh, I think crucial. Makes
[00:35:41] Abhijit Bose: Yeah. And, and we are training our developers to, you know, be more like how do you run agent teams or multiple agents, you know, like the, the other, like those aspects of software development and, you know, uh, MACAW and all of that. But not like how to [00:36:00] figure out prompt injection, you know. That, we want to automate that as much as possible.
[00:36:04] Dan Lines: If I shift our focus now, we'll go into our last topic here, and the last topic is around going from, let's say like a pilot to maybe first deploying something a- agentic into production to that lasting value.
[00:36:24] Dan Lines: So once an agentic, agentic system is in prod, what does your team do in the months after launch that turns it maybe from like, "Okay, we just launched this," to Scale and lasting, like what comes after you get something into production?
[00:36:44] Abhijit Bose: The pilot phase, um, is mostly for us to experiment and learn.
[00:36:50] Abhijit Bose: We wanna get as much, you know, offline, uh, and some real, you know, production data, uh, with, with the pilot users, uh, [00:37:00] to do some evals. But then once the system goes to production, we start to slowly scale down the, the number of users. And scaling has its own set of things that it surfaces that, that you don't always find out, you know, like when you are designing or developing an application, uh, with a small number of users, you know, in the pilot stage.
[00:37:23] Abhijit Bose: So that's one thing we do. We slowly start to ramp and see, you know, is the latency going up or down, you know, or, or going up, for example. Or, you know, do we see more turns, you know, for some reason the agent is taking? where are things dropping, you know? Uh, like when we don't get the right expected output, um, from the, from, you know, for like the right tool calling, for example.
[00:37:48] Abhijit Bose: That's one thing we do. The second thing is, um, we, as I said, like eval is the, the new kind of like mantra for us, you know. So we, [00:38:00] we double down on evals to actually use evals to improve the system. So, you know, whether it's the using a different LLM, using the, you know, tightening the harness, br- bringing better context, every part of the system, you know, uh, we try to improve using evals from these, you know, from that scaling in production data.
[00:38:25] Abhijit Bose: So that's where the engineers focus on, um, engineers and product managers and, you know, our data scientists once the application goes from pilot to, to production.
[00:38:36] Dan Lines: I mean, all of that is about telemetry, right? Yeah. Absolutely. You gotta have, be measuring everything. You gotta have all of that to input back into your system.
[00:38:45] Dan Lines: That's what it sounds like to me.
[00:38:47] Abhijit Bose: Uh, absolutely. Yeah. So I mean, one of our, you know, distinguished engineers basically took it upon herself to make sure that every part of our platform is emitting [00:39:00] the right telemetry with the right tracer ID, for example. You know, again, this was, until you really deploy the platform to a number of use cases and start to scale, you don't always, you know, see those things.
[00:39:13] Abhijit Bose: So now, you know, every part of our platform has a, has an ID that stitches all of these data together for a particular session, for a particular app, app ID and, you know, et cetera. So, um, makes it easier now it as well.
[00:39:30] Dan Lines: Super cool stuff. We should have you on for another hour. We could probably keep going.
[00:39:37] Dan Lines: You've been doing this, you know, kind of in this world, ML, AI, you said, you know, let's say 20... I mean, you've been doing it a long time now, which is great. Now looking ahead, let's go one year into the future. Do you see any trends or shifts on the next change of how engineering teams will work? What will [00:40:00] AI, you know, consume next?
[00:40:02] Dan Lines: What are you, what are you seeing over the next year going forward?
[00:40:05] Abhijit Bose: I think the, the models. I'm, I'm really excited to see what, you know, the, the future of open weight models and US sovereign models, you know, like in-- just in couple of months, uh, we have had some great news of, you know, new models coming on.
[00:40:25] Abhijit Bose: So the, I'm, I'm really watching that, that space, like open weight models and even, you know, uh, the frontier models will continue to improve. Uh, I mean, we are all excited about what they have done in math, you know, for example, more recently. so I think there will be lot of pleasant surprises and step changes in the model side.
[00:40:49] Abhijit Bose: The, the scary part is what it will do to the cyber security side. That's one thing we are also watching. You know, I think you have seen our recent [00:41:00] announcement of VulnHunter, uh, that we obs- open sourced, a product, you know. so the ability to find, you know, um, like very complicated attack surfaces and the ability to patch them very quickly, that's something we are also very, very much focused on, and that's where the platform controls come in.
[00:41:21] Abhijit Bose: You know, you don't want to patch every application out there. That's just too much work. But if you find certain platform-related, you know, controls that you can patch quickly, you know, I think every enterprise has to be like super, uh, careful, you know, and on top of like cyber threats in the age of AI.
[00:41:42] Abhijit Bose: I think that's gonna continue to become, uh, a topic for us, you know, in the coming months. And then I'm really, curious about where RL will take us. So the combination of our own models, the frontier models, all the deep telemetry we [00:42:00] do, you know, and then if we put RL on top of it, how much more can we push the boundaries of what the agents can do inside the company?
[00:42:10] Abhijit Bose: Uh, I think that's one area, again, um, we are really interested, and I think that's gonna shape up, you know, a lot of our agenda, uh, next year.
[00:42:21] Dan Lines: Amazing. Thanks for sharing, especially on the security side, which is something we don't talk about that much on this show. But yeah, I mean, anytime there's a revolution in tech, the attackers always use it, uh, to try to get access to whatever they need access to.
[00:42:37] Dan Lines: So yeah, that's actually- And, and the window- Yeah. Yeah ...
[00:42:39] Abhijit Bose: and
[00:42:40] Dan Lines: the
[00:42:40] Abhijit Bose: window is getting, you know, smaller and smaller. A lot of companies used to have, like, one-month SLA, you know, you patch something in open source. I mean, you can't do that anymore. It's like you have to patch it, you know, almost the, the same time, you know, it's detected.
[00:42:54] Dan Lines: Abhijit, there's one question that I've been [00:43:00] asking each guest that comes onto Dev Interrupted, so I'm gonna ask you the same question. What's one piece of advice that you'd give another engineering leader who's in the middle of their, their own AI transformation for their organization right now? One piece of advice that you would give them.
[00:43:25] Abhijit Bose: I think it's the growth mindset or be curious and learn. You know, like, we are all learning. I started my career with traditional machine learning. My PhD thesis was in machine learning, you know. I-- Then I picked up AI. Um, I took coursework in AI, and then there were no jobs, you know, when I was studying AI.
[00:43:45] Abhijit Bose: And now we are into the age where we are, you know. So I feel like if I look at o- at my own journey, it's been just continuous learning. Uh,
[00:43:54] Abhijit Bose: that's, uh, that's probably gonna be the number one skill, you know, or, [00:44:00] or kind of like the self-development, you know, journey that everybody has to go through in the next, you know, 10, 15 years
[00:44:08] Dan Lines: Well, Abhijit, looks like you made the right bet a long time ago of where to focus your career on. I'm sure it's paying off now. Thanks for coming on here and sharing all of your knowledge, your background, your information, uh, where Capital One stands. Like I said, it sounds like a really fun place to come and innovate and work. And, uh, yeah, just thanks so much for joining us on Dev Interrupted today.
[00:44:37] Abhijit Bose: Thank you so much, dan, uh, for inviting me, and I, I really enjoyed it.
