This week, Andrew sits down with LinkedIn Distinguished Engineer Karthik Ramgopal to explore the reality of deploying agentic platforms across a massive organization. Karthik unpacks the mechanics of AI memory, spanning procedural and episodic structures, and explains how to build durable engineering primitives that actually last. Finally, the two discuss the enduring importance of system fundamentals and why LinkedIn is restructuring its internship program into AI-native pods to foster a new culture of two-way mentorship.
Show Notes
- Follow Karthik on LinkedIn: Karthik Ramgopal
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: Joining me today is Karthik Ramgopal Distinguished Engineer at LinkedIn. And Karthik
[00:00:05] Andrew Zigler: LinkedIn is obviously becoming the, or has been, the primary platform for the professional world to understand how the world around us is changing. It's a, it's a place where we can share ideas and collaborate, and behind the scenes is an engine that keeps it all humming.
[00:00:23] Andrew Zigler: It's defining even the next generation of professional tools and professional people and what they expect out of the networks and communities that we build at work. And at Dev Interrupted, we've been talking a lot on the show about an effect called the agen- the agentic halo. This is a transformative effect where an AI-powered leader or individual uplevels and upskills those around them. Instead of being a singular 1000X engineer, they turn everyone around them into 100X engineers. And those types of leaders are, are really [00:01:00] redefining the future of software, and I know that you've been at the center of this at LinkedIn and their agentic platform. And so today we're gonna dive into how you've been exploring these concepts, uh, in a massive production environment and what it's been for you to be like to, to scale that effect, uh, beyond just raw compute, also addressing it with other, uh, parts of the AI stack, including the people part, including the technology parts that sometimes get neglected when all of the uz- other buzzwords are flying around.
[00:01:32] Andrew Zigler: So as you can tell, Karthik, there's so much spinning through my head ready to talk with you about, and it's so great to have you here on Dev Interrupted.
[00:01:39] Karthik Ramgopal: you know, like, thanks for the opportunity, and let's jump right in. I'm excited to talk to you too about all of this.
[00:01:47] Andrew Zigler: Great. Well, let's start at the top. How about we learn a little bit about what has been happening at LinkedIn with its agentic platform for its engineers that are building and servicing and developing the tool [00:02:00] that, the platform that we all use. Uh, when you've been shifting into not only being an agentic engineer and having agentic engineers around you but also now providing AI features through the LinkedIn service to your customers and users, what have been some of the fundamental architectural shifts that you've made in your day-to-day?
[00:02:20] Karthik Ramgopal: So I think the word agent platform is overloaded. We effectively have, I would say, two incarnations of it. A lot of technology is shared, but the usage is, like, obviously different. The first is the agent platform for internal productivity for engineers, but not just for engineers, for a variety of other job functions as well.
[00:02:41] Karthik Ramgopal: Product, design, legal, marketing, business operations, bunch of people use this stuff, internal productivity, right? AI is pretty, uh, universal in that way. It's like a rising tide which lifts all the boats up. It's the same way for AI. It's not just engineers, it's everyone else. So there's that. [00:03:00] And then you have the agent platform, which we use as the framework for building a bunch of our production agents, either on the consumer side or on the enterprise side to serve our members and customers.
[00:03:13] Karthik Ramgopal: Again, as I said before, a lot of the technology is shared. Uh, some of the core building blocks here are stuff around basic stuff, right? Prompt management. How do you do inference? How do you abstract away, uh, common orchestration operations when, like, building agents? How do you do context management? Again, we started off very simple.
[00:03:34] Karthik Ramgopal: We had chain-based systems. We have graph-- Like, we had graph-based systems, and we have more harnesses right now. But one of the things we have realized through this entire journey is that a very important thing for an AI system, which is very different from a traditional software-based system, is agency and personalization.
[00:03:53] Karthik Ramgopal: And personalization, especially over the long term, requires investment into a form of, like, memory, [00:04:00] right? So that it remembers, it gets sort of like better over time and truly understands you. So we've also been making a lot of investments into our cognitive memories, uh, stack.
[00:04:10] Andrew Zigler: Right. So I, I love how you frame the problem, the, the opportunity even for teams that are working with these tools, that it's not something that's locked within an engineering world. What it does is that it allows you to bring deeper action, insight, and leverage to all kinds of roles within your organization, and in doing so, bring them closer to the problem, uh, space that y'all are all working in. It's not just the engineer's responsibility to be customer obsessed and be really close to the conversations that, that their customers understand. You know, we try so hard as organizations to bridge those two very often, like very, um, like gapped kind of parts of the org. But it's also like, um, everyone else's responsibility to understand how they can have a role in this.
[00:04:56] Andrew Zigler: What are the new skills they need to develop?
[00:04:58] Karthik Ramgopal: [00:05:00] Absolutely.
[00:05:00] Andrew Zigler: best practices with folks around them? So you had to... Y- you saw this as an opportunity of, if I can create this internal ecosystem where people can come together and share, then I trust that the coming together and sharing will happen.
[00:05:13] Andrew Zigler: And then, and then it did. It was kind of like, you know, if you build it, they will come kind of deal.
[00:05:18] Karthik Ramgopal: Yes.
[00:05:18] Andrew Zigler: how, how did, how, how did you assess that problem and, and even start? 'Cause I know a lot of o- like leaders, they maybe see, they get hungry for like, "I wish I had that ecosystem pin, that, that innovation thing."
[00:05:30] Andrew Zigler: How, how did you start to tackle that problem at an org so big?
[00:05:33] Karthik Ramgopal: So I think that the first thing to understand is in order to reap the, like, benefits out of AI, everybody needs to be able to use it to a reasonable level. Otherwise, you end up creating bottlenecks. It's almost like, uh, you know, you have an acceleration in one place because you have built a freeway with N lanes, right?
[00:05:53] Karthik Ramgopal: And after that, the freeway suddenly ends, and you get onto a dirt road, and everybody slows down. The freeway doesn't help you, right? It's the same [00:06:00] thing. If you just accelerate the engineering function, and even within engineering, a lot of people make the mistake of acceleration being limited only to coding.
[00:06:08] Karthik Ramgopal: It's way more than coding. It's how you write your s-specs, including design specs. How do you write your docs? like engineering docs, design docs, collaboration docs, whatever. It's also stuff around operations. How do you deal with alerts? How do you deal with deployments? How do you deal with production issues?
[00:06:25] Karthik Ramgopal: Even just looking at engineers. You also have product management. You have design. You may have questions for legal, right? So everybody needs to experience the acceleration so that you can actually see a business acceleration. Otherwise, if you only accelerate the engineers, everybody else slows down or is operating at the same pace, it results in frustration as opposed to acceleration.
[00:06:48] Karthik Ramgopal: So you need to start with that, like, mindset of, "Hey, how do we accelerate everyone?" The second thing is that there is a lot of hype, right? A lot of different tools are coming in every day. It's like a Tool A, Tool B, [00:07:00] Tool C, et cetera. I think the real change required, though, is a, an attitude change and a mindset change.
[00:07:07] Karthik Ramgopal: It is sort of, you know, like less about the tools, and it's more about how you use these tools to accelerate your productivity. A very important thing to note here is that whatever tool you buy, most likely it's not gonna know about the internals of your organization, right? Every organization has a bunch of proprietary stuff.
[00:07:28] Karthik Ramgopal: Uh, sometimes it's proprietary data, proprietary business practices, how people do things, et cetera. Sometimes it's also proprietary tools and workflows which people use, which require more of software integrations. At the end of the day, this boils down into a giant context management problem. So we at LinkedIn have invested a lot in recognizing that even as the tools advance, and they will advance, we will continue to get limited by the context we provide them.
[00:07:57] Karthik Ramgopal: Unless you provide them context, it's just [00:08:00] useless, right? So an equivalent is you hire the smartest engineer on the planet, let's say, and if they aren't able to access your systems, useless, right? You, you aren't harnessing them. It's in the same way for AI. Uh, and as I said, it's pretty universal for other job functions.
[00:08:16] Karthik Ramgopal: Then the question comes, hey, you do invest in this context curation, context sort of like management experience, but how do you do it in a way such that as the tools advance, you don't have to keep reinventing this? Which is why we've invested a lot in exposing this stuff via open standards like MCP, the Model Context Protocol, for example, so that, hey, as tools evolve and, uh, as they start supporting MCP, you just sort of get this for free.
[00:08:43] Karthik Ramgopal: We've also seen other evolutions. For example, a lot of folks are doing CLIs right now. There's been this entire thing around progressive disclosure and, uh, the skill.md standard. So as these standards evolve, we are also evolving alongside these standards. But it's very [00:09:00] important that we stick to the open standards for exposing our proprietary context to the tools, so that as the tools evolve, we keep up and we aren't left behind.
[00:09:09] Andrew Zigler: Exactly. It's about making the smart bets on the durable
[00:09:13] Andrew Zigler: primitives and where those primitives are being built and managed. I think that's exactly right on the head. And I, I, I know a lot of engineering leaders are in the position or, or maybe even like from last year, right? We're all coming into this year with like a huge baggage of, of budget of all of these tools that we bought over the last year, just kind of like, "Oh, maybe this will work, maybe this will work.
[00:09:35] Andrew Zigler: Maybe, maybe we'll try this, this, this." And so people have built up a lot, and I think that that's gonna ultimately come to a reckoning. And the teams that just bought those tools and didn't build durable learning practices, didn't figure out how do we actually interface with this tool? What do I have to bring to this tool every time?
[00:09:52] Andrew Zigler: That like context engine that you're
[00:09:55] Karthik Ramgopal: Absolutely.
[00:09:57] Andrew Zigler: now that they're gonna be at the end of this road, they [00:10:00] didn't get, you know, literally that now they're on a dirt road or whatever the case is, and they're not getting the benefits that they were expecting, and maybe the tool churns or they go to a different tool. The problem just repeats itself because they're not building the durable internal, uh, ways of managing how they interact with that. And so then that becomes the responsibility of the org to create this kind of a open standard, open, platform that's... And it's managed by standards that are gonna be durable over time.
[00:10:30] Andrew Zigler: And that's a bet both for you as the organization and your engineers in keeping the lights on, maintaining whatever context engine you build. But that's also a smart bet for the tools you will buy, because chances
[00:10:45] Karthik Ramgopal: Yes.
[00:10:45] Andrew Zigler: they're gonna conform to the spec that you're conforming to because they know what you're investing in. So now your procurement process is easier. It's a win-win.
[00:10:54] Karthik Ramgopal: Our procurement process, our security, uh, and validation processes, [00:11:00] and also honestly evals. See, a lot of AI products, uh, people try it almost as a, like, vibe check. Oh, this looks good. But, like, vibes aren't reality. So unless you have a robust set of evals, you really don't know how well is it gonna perform in the real world.
[00:11:17] Karthik Ramgopal: So generally, when we build these integrations, it's not just exposing the data in an agent-friendly way, in a secure, compliant way. Those things are very important, yes. But it's also building the evals, because at the end of the day, you're dealing with a non-deterministic system, and unless you have the evals, you don't have the guarantees that, hey, this is actually worth it.
[00:11:39] Karthik Ramgopal: And I do second your point on build versus buy. These are decisions we are continuously making here. Hey, do I build this or do I take some open source project And use it or do I buy some product? And also for the open source and the buy part, do I use it as is or do I build some adaptation [00:12:00] like layer on the top to adapt it to my internal systems and internal processes?
[00:12:04] Karthik Ramgopal: So these are decisions we are continuously making. Space is moving very fast and sticking to standards helps with all of this.
[00:12:11] Andrew Zigler: Exactly. It just helps you make smart, uh, investments, but also be able to move quickly and benefit from the best and the greatest. As soon as it hits the market, you're already readied and positioned to take advantage of it. You don't have to have an entire, you know, onboarding cycle a quarter to figure out how do we use this tool again. And in this world, you start to get a lot of benefits on top of benefits. It's like this is where the exponential gains of being able to work with these tools in a fluent way really starts to pay off. What is, you think, like an-- maybe like what people would consider like an old school engineering discipline that maybe they think AI's replacing it, but now in this new world is more important than ever, and you maybe think they're underestimating or something?
[00:12:55] Karthik Ramgopal: Strong knowledge of systems fundamentals for engineers. A lot of [00:13:00] engineers treat AI as this magical like black box, and they expect that even if they do not have system fundamentals, they go prompt the AI, and the AI can do like magic for them. The AI is great at doing things, but it's also great at fooling you, right?
[00:13:17] Karthik Ramgopal: So unless you have strong system fundamentals, you cannot make the right judgment calls on is what the AI doing correct or not? And sometimes even if it is okay, correct, is it the most elegant way to solve problems or not? So those system fundamentals, ironically, are even more important because AI is like super smart, so you've got to be smarter than it, right?
[00:13:42] Karthik Ramgopal: You cannot keep up with its throughput because you are human and it's sort of a machine, but you have to keep up with the nuance, the intelligence, the smartness, the judgment quality that is solely on you, and even more so for more junior engineers who are entering the workforce right now. This is incredibly [00:14:00] important.
[00:14:00] Karthik Ramgopal: Otherwise, it's going to result in skill atrophy.
[00:14:04] Andrew Zigler: Yeah. If, if you, if somebody was in a position of I need to build system fundamentals for a new system I don't understand or for systems I'm still trying to understand, how do you think they could use AI to, as a learning or an upskilling tool instead of a magic black box?
[00:14:21] Karthik Ramgopal: So the first is, uh, this is an incredibly important point which you made, using AI as a tutor and not just AI as a doer. Jensen Huang, I believe the NVIDIA CEO, said this a while back, that everyone has a teacher in their pocket right now or something like that, right? And he was alluding to this, using AI as a tutor.
[00:14:41] Karthik Ramgopal: Asking AI the right questions is important. The first is the attitude to ask it these questions Right? This is important as a practice and important as a validation step itself instead of blindly trusting what it produced to ensure quality of results, right? The second is how do you [00:15:00] know how to ask it the right questions?
[00:15:01] Karthik Ramgopal: Like, uh, th-there is an art to asking it questions, right? What parts do you want clarity on? How do you get the AI to provide citations or, like, references which you can go and independently validate to confirm that it did not hallucinate? And also when you are starting off with things, again, this is more so for early career engineers.
[00:15:22] Karthik Ramgopal: So sometimes it helps to just take a step back, maybe go to a whiteboard or d-do it on a piece of, you know, okay, pen and paper to write this out automatically and ask, "Hey, can you explain... Hey, have I understood this right? Has, is what they explained really making sense?" The other interesting technique is you can ask the AI to test you.
[00:15:45] Karthik Ramgopal: You can literally ask the AI to test you, right? It's almost like you have exams at school. You can tell the AI, "Hey, ask me questions about this topic." And you give it answers, and after that it essentially judges you and it says, "Hey, did you get it right or wrong?" [00:16:00] And if you got it wrong or right, you can again ask clarifications questions for more nuance.
[00:16:05] Karthik Ramgopal: It's like a highly personalized tutor. Again, you've got to spend the cycles though to invest in this. I know, uh, there is a lot of push for productivity everywhere, but I keep telling people that learning cannot be sacrificed at the altar of productivity. Durable learning is what really keeps us in the game long term.
[00:16:27] Karthik Ramgopal: So we have to invest the cycles in ensuring that we, like, learn things and just don't keep mindlessly producing things in a cycle without understanding what we are really doing
[00:16:39] Andrew Zigler: Exactly. It's like constantly investing in your education is investing in your productivity, because at the rate in which these tools change and the ways we have to use them evolve, if you aren't actively learning and investigating and asking questions and maybe wondering if you should throw away that practice that you [00:17:00] did two weeks ago now, then you're going to end up with a lot of baggage, misunderstandings, misconceptions that are going to add up over time and ultimately hamper your ability to get the most out of the new way of work. And I, I loved how you framed that it has to come from being curious and asking the right questions, slowing down and asking, you know, "What do I not know?" And trying to get a, uh, a better glimpse at things. But then also making it more deterministic in a sense of like, you know, quiz me. Like, like, like actually help me prove to myself that I understand this.
[00:17:39] Andrew Zigler: Give me the citations, the go read further. And that just comes from being deeply curious, which I know most engineers, you know, everyone who listens to this show by definition is deeply curious. And so it's a practice that's already natural to many, and it's just that they have to operationalize it, [00:18:00] right?
[00:18:00] Andrew Zigler: They need to, uh, accept that that's not a passive thing that, uh, I could do on the weekends. It's actually something I need to make a part of my, my regular da- even daily practice because it's the way I work. And so if maybe for someone listening to this, if maybe they really relate with how you're, you're thinking about solving this context problem, this memory problem internally, that way you can interoperate with, you know, the future. Um, if they were trying to maybe express this and start this at their own org, where do you think that's best started? How do you get stakeholders? How do you prove the value? How did you get early investment in this idea?
[00:18:40] Karthik Ramgopal: So a lot of early investment in this idea essentially came from providing like negative examples, which is here is where things break down without the external-- like without the internal context, we're just using the external tools. I almost call this as a poking the [00:19:00] balloon moment, right? Because otherwise there is so much hype.
[00:19:03] Karthik Ramgopal: The hype is like a giant gas balloon, and you're like, "Yeah, just use it." But I-- you gotta basically, I mean, like poke it to show that no, in some cases it, it doesn't work. So you start with the negative examples, but you cannot continue on the trail of like negativity because at that point you're not being, okay, constructive.
[00:19:20] Karthik Ramgopal: So then you try to build Prototypes or showcases to show that, hey, here I did this context injection, and it actually became, like, better. So then the question gets asked, hey, is it useful for everyone to do this again and again and again? Or do we build a repeatable set of durable primitives, abstractions, platforms around this?
[00:19:42] Karthik Ramgopal: And then the conversation becomes, who does it? Is the central team doing it? Is it gonna scale? Or should this actually be the decentralized responsibility of any team? So this was the journey at LinkedIn, and right now it's almost like all of our platform teams have [00:20:00] two kinds of users. You have the human users, and you have the agentic users.
[00:20:04] Karthik Ramgopal: And the primitives you build for them are very different actually, because the access modalities are different. But it's great that it's become decentralized because I, as the owner of a platform team, understand my system the best. So when I evolve it, when I change it, I ensure that I do it in a way that my users, in this case, both the agent users and the human users, also evolve in like tandem, right?
[00:20:31] Andrew Zigler: Right
[00:20:33] Karthik Ramgopal: context awareness problems for the humans building and maintaining these systems, maintenance overheads, et cetera, simply doesn't scale. So this is the journey
[00:20:43] Andrew Zigler: Yeah. Uh, it's really critical to, for, to understand that you have to have that same shared layer that becomes that con- that, that fluency layer that, that which through stuff passes, you and the agents. And you're right that, like, the agents are going to ingest the information way [00:21:00] differently and, and probably use a whole bunch of esoteric CLI to commands to pull out, you know, stuff in JSON formats that we're not going to read.
[00:21:07] Andrew Zigler: And we might want very nice, uh, web dashboards or a connect integration into Asana or whatever our favorite tool is to, to interop with it our way. But the key is to figure out how do I connect all of those dots so that all of that information can flow back and forth with as little friction as possible, with as much cohesion.
[00:21:27] Andrew Zigler: And I loved how you gave, like, the, the recipe for starting. It's so simple. You find, find the bad examples. Like, find, like, the really awful ones. Find the ones that are causing problems at the highest stakes, and then you literally make some evals for them. What would make them better? What's the problem I'm trying to solve?
[00:21:43] Andrew Zigler: And then you go throw together some prototypes. You throw some agents at it, and then you show that to people. You find an owner for that problem. Whose life, whose job is that gonna make easier and better? Who owns the outcome of what this fixes? And then you rinse and repeat. And then by doing that, you build [00:22:00] those durable, those durable internal practices and mindsets.
[00:22:04] Andrew Zigler: Now suddenly you get people raising their hand, "Karthik, over here. We got, like, all these problems that's coming out of our AI system that we are just been hitting like a whack-a-mole for months. Like, how do we, uh, build a system to survive that?" And I-- Part of this too, and I wanna, I wanna switch to talking about memory, because you've hinted at that a few times as being a really big part of how you solve that.
[00:22:25] Andrew Zigler: And I think most people stop at just context. Like, "Oh, I can just go get the context. I can structure it. Now we're fluent back and forth." But where does memory come into play? How do you think about memory in terms of how an organization like yours interacts with the tools?
[00:22:40] Karthik Ramgopal: At the end of the day, everything is about context management, right? Memory is primarily helping with context, context management beyond a session, okay? Th-that is the way to think about it. So when we look at the memory, we decompose it into different forms. You have the working memory, which is essentially your [00:23:00] raw conversation history, which is tied to a session.
[00:23:04] Karthik Ramgopal: Now, the reason you need it is because sometimes these sessions can go very, very long, way beyond the context window of a, of a model. So you need a way to retrieve the right parts from it, right? Sometimes you do some very naive things like, hey, I just get most recent N in terms of history. Sometimes that's not good enough.
[00:23:24] Karthik Ramgopal: You need a semantic lookup in order to retrieve the most relevant portions, et cetera. Sometimes you use smaller models because you're trying to save on cost, and smaller models have smaller context windows. So having this, uh Having this kind of a, like, memory system augmented will actually help you reduce cost and reduce latency because you don't need a larger model with a giant context window because this external layer is actually managing the context for you.
[00:23:51] Karthik Ramgopal: This is the relatively easy part though, because it's mostly systems engineering, except maybe a little bit of, you know, embedding-based retrieval [00:24:00] for that semantic, uh, like retrieval stuff, which is not so much systems, has a little bit of science as well. The harder part where the memory comes though is with what I call as more longer term like memory, where you are inferring things based on what users are doing, either in your agentic application, outside, et cetera, and you're collecting various sort of signals.
[00:24:24] Karthik Ramgopal: We decompose this into three different kinds of memories. We have procedural memories, which is where we try to understand what are people doing, right? And how do we learn what people do so that the agent can start automating those things as opposed to having humans do it. Or let's say if the agent is doing a few things and the human is not happy about how those things are being done and they wanna give feedback, great.
[00:24:47] Karthik Ramgopal: All of those fits into procedural memory. Again, it's important to remember these things, right? Otherwise, it kinda gets irritating. I told you, "Here is how I like it," you keep repeating the same things again, not a pleasant experience. So, that is procedural memory. [00:25:00] Then we have stuff around episodic memory, where across sessions or even within a session, we try to capture episodes which describe specific intents or specific jobs you're trying to accomplish, and try to understand preferences associated with that episode. And then you have the long-term memory, which is, hey, there are just few things which are your preferences regardless of the task or job you're doing, just as a person. How do I infer these preferences? How do I store them? Again, a lot of this long-term like memory processing happens asynchronously. Some of it happens offline, some of it happens nearline, because processing it online is actually very expensive.
[00:25:42] Karthik Ramgopal: It's a lot of data, and it's very expensive to sort of crunch and munch in a short period of time. Some of it has to happen online just given the nature of the product, but a lot of it happens offline and like nearline. And it all gets updated asynchronously into our, into our, like, memory agent, right?[00:26:00]
[00:26:00] Karthik Ramgopal: Now, you may think, "Oh my God, you know, okay, memory itself is an agent?" Yes, because when you're given, uh, an abstract query, you need to know where to look in the memory, how to reconcile potential conflicts, and how to return the right answer. So we've built a bunch of systems infrastructure and AI infrastructure for this and exposed it as an agent.
[00:26:20] Karthik Ramgopal: Again, using standard protocols which all of our other agents can talk to, to almost use this as a sub-agent to enrich their own context, and improve the Like improve the user experience, both for our external products and our internal products. Now, how these, how these, uh, kind of like memories are actually structured is going to be very dependent based on the use case, right?
[00:26:43] Karthik Ramgopal: We have hosted agents running in the cloud, which needs access to distributed memory stores with consistency primitives, et cetera. You have your local agents running on the machine, which actually would do very well with file system-based memories because it's just on your own like machine, et cetera.
[00:26:59] Karthik Ramgopal: [00:27:00] So again, the architecture, the system fundamentals slightly differ. The trade-offs you make slightly differ based on your needs, but the overall principles are the same.
[00:27:10] Andrew Zigler: Wow. That was a really cool glance at how your org has built around that huge, like, uh, problem. Like really, uh, what it-- this does is it allows-- it bridges that last little bit of the gap in the context between it just being like, "Oh, yes, it kinda gets us. Oh, yes, it's like an intern in our org. Oh, yes, it, it is, like, aligned, but doesn't know every little bit of what we do," to actually covering that last bit of a gap of like this is a skilled and-- this is a skilled member of our organization.
[00:27:41] Andrew Zigler: This is somebody that can be trusted with this ti- kind of scope, and it unlocks a new kind of way of working with the tools in a reliable and repeatable manner. And so, you know, because that is such a big unlock, you invested in being able to understand the different layers of what [00:28:00] memory is. And you walked through some of them, and they're, uh, these are just like really amazing concepts to understand of like, you know, even coming back to how we think about when we provide context for an agent, you might have like the system prompt, you have the assistant prompt, you have the skills that you give it, you have the tools that you provide it.
[00:28:15] Andrew Zigler: And all of this is like creating some sort of like, you know, bit of a mash of some kind of like memory process or like, uh, best practices for that tool. But the reality is, is that like, uh, like There's a lot of nuance to how those things are layered. The same thing is true for when you're going to approach memory.
[00:28:35] Andrew Zigler: You have to understand, how do I cut the edges of these episodes and define them? What even matters for us as an episode? And those are a lot of, like, internal alignment conversations, is a lot of build it and figure it out. It's a lot of understanding the nuance of how did I get that result, and having really good evals to sniff out all of the weird edge cases you're introducing in this new memory system. [00:29:00] But the biggest, uh, thing that here that you said that really stuck with me is kind of how then it gets exposed as an agent. You know, this is actually a kind of way of working with, AI and agentic memory that I myself use in my own personal practice. Obviously at, like, a minor scale, but I don't necessarily have some sort of place where I off-board all of these memories to for processing. But I certainly have a queryable graph-related assortment that is structured data of the sessions I do, the tasks that those agents do. And all I did in the beginning was just set up a skill so that they knew how to maintain that over time, and then you just get this rich corpus. The same thing becomes true of memory.
[00:29:41] Andrew Zigler: If you teach agents how to use the memory, you teach that agent how to serve the memory, you can build a really durable thing. I'm curious, like, what were the, it, the timelines on building that? It's like there's a lot of talk right now around episodic memory for agents, and I know people are calling it, like, sleep mode.
[00:29:59] Andrew Zigler: It sounds like y'all [00:30:00] already had a whole sleep center or, like, a agentic sleep institute already built by the time people like Anthropic were maybe even talking about having a sleep mode for Claude. So, like, was that an early unlock for you as you were building with context? Like, where did that-- how, how did that come as a need?
[00:30:16] Karthik Ramgopal: I think it was a early unlock when we pressure tested using the idea I said before. Where does this break? What kind of repetitive patterns are people putting in order to solve this problem? And also looking at human parallel processes like you were describing, right? Like you said, "Hey, all this stuff goes into a prompt."
[00:30:36] Karthik Ramgopal: Yes. Do you want every human to be repeating that process? Do you want them to be making the same, like, mistakes? No. Which is why you build a platform and infrastructure around it to make everyone's lives easier, so that a select group of experts who understand this deeply go and build this infrastructure, and then everyone sort of benefits.
[00:30:57] Karthik Ramgopal: The other huge learning we had here is that [00:31:00] you don't always work inside your agentic system. Let's take an example of coding. You use a coding agent, you are also giving human reviews on your PRs. You as a human are reviewing code, leaving comments. Now, if you only took the input inside the agentic system, it's not comprehensive.
[00:31:19] Karthik Ramgopal: You also have to look at the human reviews on the PRs. Very strong signal. If an experienced developer is leaving a comment about something then your agent should learn it so that the next time it incorporates that feedback, of course, in a context-appropriate manner, and does not repeat the mistake. So the data sources for ingestion go way beyond the conversational memory of your particular agent, right?
[00:31:48] Karthik Ramgopal: You could even extend it to other areas. Let's say you have a Slack channel or a Teams channel where you're discussing stuff about your project. Somebody provides input, there's consensus. All those data sources can be gleaned in order [00:32:00] to learn. The other big insight we had is that there is a lot of like noise in this data.
[00:32:06] Karthik Ramgopal: So analyzing this data to extract durable signals is an expensive process. So you have to build specialized infrastructure in order for you to do it, right? And also the traditional thought here, if you look at it about a year, year and a half ago, was that, oh my God, anytime you need something like this, you need to do like model fine-tuning.
[00:32:26] Karthik Ramgopal: It's one way to solve the problem, but it's a very expensive way to solve the problem, uh, especially as the frontier models keep advancing, which is why we relied on like, you know, memory as a way to solve it. For our external products, a lot of it came from customer feedback, which is, "Hey, I want this product to feel more personal.
[00:32:45] Karthik Ramgopal: I want it to feel less like an AI and more like a companion, right? Which is helping me in the job. I'm still in control, but it's helping me in the job. It needs to understand me as opposed to being a tool." [00:33:00] So a lot of this was informed by those learnings and what we need to do to deliver product at the quality levels which our users were expecting.
[00:33:10] Andrew Zigler: Exactly. It's, oh, I g- it's like I'm just so excited about the way that you think about this, Karthik, 'cause it's rare that you get an engineer on here who has both the skills to enact it, but then also just, like, the mindset to upskill and to teach and to explain why those problems are so important. And h- honestly, it's like everyone has a part to play in creating this true, uh, partner AI that is able to meet us where we need it. And part of that too is also involved in upskilling and training the humans and the operators of these systems and, and approach of the way that we use these tools. I'm curious, like, how has the rise of AI kinda changed your personal philosophy on how you mentor engineers and people within your organization, and how you [00:34:00] think it maybe even redefines the role of leaders within organizations now?
[00:34:06] Karthik Ramgopal: So, uh, it's been a journey, right? I think that people fall into two, two camps broadly when it comes to AI. You have the Extreme proponents who are like, "Hey, I will use AI for everything," and sometimes take it to extreme levels where they just, uh, almost use AI with vibes, right? Where they don't really do engineering, which has, uh, a lot of cons, especially for production systems.
[00:34:33] Karthik Ramgopal: Great for prototypes, not very good for production systems because you end up with AI slop at the end of the day. Then you have the detractors, and sometimes this former group is what ends up creating detractors because they see all this, all the slop which is being produced, and then they're like, "You know what?
[00:34:50] Karthik Ramgopal: This is really bad. I don't wanna have anything to do with it," right? The goal of leaders is to find the happy middle. Basically, [00:35:00] try to coach people into using AI with the appropriate guardrails and the appropriate context to increase their productivity. That is our job as leaders. So then the question is, if you encounter a problem, the first question is, how do I accelerate productivity using AI?
[00:35:18] Karthik Ramgopal: What AI tool do I use? What kind of context do I inject? But more importantly, what sort of validations, guardrails, and evals do I have to ensure that th-this is producing a result at acceptable quality? So generally, what I tell people is, if you use AI to do something, you should use AI if you also know how to do it and you're using the AI just as an accelerant.
[00:35:42] Karthik Ramgopal: Otherwise, you cannot judge its quality. If you don't know how to do something, please don't use AI for it. First, learn how to do it, and after that, use AI to accelerate it, right? Because otherwise you can't write the validation guardrails and the, uh, evals and all of it for it because you don't know how to do it personally.
[00:35:58] Karthik Ramgopal: That is the first part. [00:36:00] The second part is that your judgment bar for using AI should be the quality of the output should very closely resemble, apart from very minor, almost facetious differences as to what you would have produced had you, had you done the work manually, right? So that is the quality bar.
[00:36:20] Karthik Ramgopal: Now, once this mindset shift comes in, a lot of the detractors become way more comfortable, and honestly, some of their skeptical approaches are useful in ensuring that quality is not compromised at the expense of productivity. And it also starts, you know, encouraging the folks who are AI proponents to go out and experiment with more things.
[00:36:43] Karthik Ramgopal: Uh, you're not curtailing their curiosity, you're just putting the guardrails around it. So it's almost like a kind of push and pull. Now, once you do this, the next step is how do you ensure that people are not solving the same problems over and over? So you need to have a culture [00:37:00] of open learning, information exchange.
[00:37:04] Karthik Ramgopal: M-Make it okay to ask questions Including what sound like very dumb questions. There are no dumb questions. Everybody expresses a curious mindset. People ask questions, people unblock each other. So in our team channels, when somebody doesn't know how to do something, they ask. Again, we have a, we have, I mean, a bot in the channel which is indexing all of this so that, you know, the next time somebody asks a question, we don't wait for a human to respond, the bot responds.
[00:37:32] Karthik Ramgopal: And a human can correct it, of course, right? So we try to reduce the effort involved also in answering these questions, right? Now, for the more junior engineers, as I said before, if they start blindly trusting the AI, they start getting into this skill atrophy. So it's very important that as leaders, we create the space and the structures for them to go and experiment and learn.
[00:37:58] Karthik Ramgopal: And even for the more senior [00:38:00] folks who are learning new areas or new ways of doing things, right? But especially more for the junior engineers, you have to, you have to deliberately engineer these moments. These things aren't gonna happen by chance or by providence. They may, but then you're just counting on fate, right?
[00:38:15] Karthik Ramgopal: You have to deliberately engineer these things so that people have a sense of learning and growing while producing high quality outcomes.
[00:38:25] Andrew Zigler: The deliberateness is the important thing. I think what's happening and what's happened, especially over the last years, you have a lot of leaders like what I said, you know, they buy a lot of tools, they have a lot of patchwork processes, and they maybe just kind of sit on their hands or just try to, like, work with what they got, but then they don't ultimately arrive at what they need.
[00:38:43] Andrew Zigler: And parts of that is, like, it gets broken down into lots of little, uh, problems. One of them that you pointed out is that, like, you know, if you don't know how to do something, don't use the AI necessarily to do it. Use the AI to learn how to do it. Use the AI to upskill around what it would mean to do [00:39:00] that. But don't just blindly use the AI to do it, because the truth is, is that for you and everyone else out in the field, you know, the floor has never been higher ever bef- it's higher than ever before. It's really easy to get something that like, "Oh, yeah, that looks okay. That looks great." But you still have to maintain that incredibly high bar for your ceiling.
[00:39:19] Andrew Zigler: It has to clear that ceiling before it represents the work we do. So your s- your, your point of that a leader needing to acknowledge that, you know, we want the output of that AI to be what you would have done
[00:39:33] Karthik Ramgopal: Absolutely.
[00:39:34] Andrew Zigler: maybe, like, facetious level edits or, like, you know, those surface level, like, those little tiny, like, those little tiny th- tweaks that, like, you might do.
[00:39:40] Andrew Zigler: But even then, let's figure those out and solve those too. It really needs to match what you would do. And so then by accepting nothing better, nothing less than that excellence, you then kind of like- Bring all of this fresh air into a, a, a discourse that might be polarizing within your org. Of you have the skeptics that are [00:40:00] not getting the usage out of it, you have the people that are, and there, there's no open learning, there's no crossing that gap.
[00:40:07] Andrew Zigler: Instead, now you're, you're saying to the, to those skeptics, "You're so right. It shouldn't be-- the output shouldn't be that bad. It shouldn't be that unreliable. We have these standards for what good looks like, and that's why you don't use the tool." And then y- at the same time, to the folks that are getting those gains out of it, we challenge them to how do we create then like an eval for everything?
[00:40:33] Andrew Zigler: Just like how in our, in our engineering org, you know, we would have, hopefully, a lot more lines of test code than production code 'cause we test everything. Now the same has to be true of your leadership organization. You need to have evals for every little type of way that people are interacting with the tool, including that little, you know, chatbot that, that might, you know, sometimes say the wrong answer and someone corrects it.
[00:40:55] Andrew Zigler: But now, because you have an eval, you have a learning loop, you have a way of improving [00:41:00] it. And then if you have a memory system, a context adjustment system, now that thing that you learned and improved has a place to live, and it's
[00:41:08] Karthik Ramgopal: Yes. So this all a
[00:41:10] Andrew Zigler: matter
[00:41:10] Karthik Ramgopal: virtuous...
[00:41:10] Andrew Zigler: you use.
[00:41:12] Karthik Ramgopal: It's all a virtuous cycle. If you engineer it carefully, it'll be a virtuous cycle. The other interesting demographic shift which I'm noticing is that a lot of junior engineers, folks coming fresh into the workforce are AI native by default. They've just grown up with this tech.
[00:41:33] Karthik Ramgopal: So there is this different paradigm of two-way mentorship. Traditionally, senior engineers mentored more junior engineers. Right now, senior engineers are also learning new ways of doing things from junior engineers. Someone like me who's been in the industry for 15 plus years, no matter how hard I try to embrace this new wave, is still ossified in certain ways of doing things and ways.
[00:41:59] Karthik Ramgopal: So when [00:42:00] I look at a more junior engineers grown up in this way, there is a lot I can learn from them. So in- so transparently encouraging that culture of two-way mentorship, putting all egos aside is also incredibly important in this moment
[00:42:17] Andrew Zigler: Exactly. It's like everyone should remember that right now for when, you know, you're thinking about your intern class for the summer or the fall, that you're gonna be learning just as much from them as they're gonna be learning from you. How do you create an environment where that flow of information, that upskilling's gonna happen?
[00:42:33] Andrew Zigler: Because if you don't capture it, then you're going to miss it. And for-- to those entry-level engineers that are more AI native, there's so much opportunity for them to define what these systems of tomorrow look like because they're not operating with fossils in their brain like you and I, and so they get to reimagine what the world looks like without all of the baggage that you and I, you know, just assume might always be there. And that, that kind [00:43:00] of, uh, uh, way of challenging the status quo, the way we work, is something that has to get fostered, I think, like everywhere.
[00:43:07] Karthik Ramgopal: Absolutely.
[00:43:10] Andrew Zigler: say that you're in, a large org and you have those-- you have the camp of the AI skeptics, right? We've talked about them and maybe how we bring them back in.
[00:43:19] Andrew Zigler: But let's say that you have a whole bunch of like siloed different like AI wizards in all these different parts of your org, and like you've maybe identified them at this point. You got like the wizard of the North, you got the wizard of the East, like whatever the case, it's like, but they're all very separate. Like, how do you bring them together and then be like, "We need the context layer. We need the memory layer." Like let's-- this is actually the most important problem to solve. Is that just like what you have to do, is just say it as
[00:43:45] Andrew Zigler: That's how do you start?
[00:43:46] Karthik Ramgopal: Exactly what So firstly, you g- go back to this old, uh, human mode of communication called as like talking to people. You talk to people and you understand what are you doing, what are your [00:44:00] problems, and then bring everyone together to discuss are we solving the same problems over and over again?
[00:44:06] Karthik Ramgopal: At the end of the day, all these wizards are also engineers. Nobody wants to solve the same problem over and over. As long as you can give them a solution which will satisfy them and reduce their work, and ironically increase their productivity in other areas because they're able to solve higher order problems, they will acknowledge it.
[00:44:26] Karthik Ramgopal: See, this organically happens when a huge technology shift like AI comes in. Initially, there is a lot of chaos, and s- sometimes it's good to let that chaos, uh, like reign for some time. So that you don't curtail it prematurely, but you see what people come up with. But you cannot let it reign on forever because otherwise you end up creating a lot of tech debt and unmanageable chaos.
[00:44:52] Karthik Ramgopal: You have to rein it in at some point and rein it in in a very specific opinionated way. I'll give you a very [00:45:00] real world example here, right? when you unleash AI agents for development, they are going to work with N different systems, right? Let's say you're working with a AB testing system, you're working with a configuration management system, you're working with a secret key vault.
[00:45:15] Karthik Ramgopal: Uh, in order to do your development job, let's say you need to work with all these systems. If the context for all these systems wasn't accessible in a uniform way, you're doing so much more integration work on your coding agent, on your, like, memory agent, everywhere else, right? So if you were a human, you are capturing all these information through your five senses.
[00:45:39] Karthik Ramgopal: That is your standard API for interaction with the world, with the tools. Now imagine we-- all of us had completely different senses, how hard would it be? It's the same thing here, right? This is where that platform layer and that investment really helps. But you need to give the confidence to people that I'm not going to, [00:46:00] you know, curtail your ambitions by building an overly restrictive platform, because then people will start to resist.
[00:46:08] Karthik Ramgopal: So you need to draw the boundaries around the, around the platform in a way such that it standardizes thing, but it does not curtail ambitions.
[00:46:17] Andrew Zigler: Yeah. And you're absolutely right to identify that those, you know, siloed wizards or whatnot, they do have that shared incentive of not having to do the same things over and over again, repeat building these little mini primitives instead of durable ones over and over again. And frankly, if they're siloed, that's what they're already doing, and they're probably already looking for a way to stop doing that.
[00:46:37] Andrew Zigler: So it's about having that real human conversation about it. Uh, I, I love that. And kinda going through what we talked about a moment ago, we, we had mentioned that, um, or rather, we talked about the, the, you know, the interns coming in, right? Folks that are more AI native that have been in the engineering world for a short amount of time or are still in school. Like, you know, they-- we might feel like there's so much that we can [00:47:00] learn from them. But flipping the table, they might have a lot of anxiety about the world that they're stepping into, about the skills that they invested their early life into building. And I just-- There's so much palpable anxiety, uh, with hiring groups, you know, hiring classes that come out of university about what their roles would even look like.
[00:47:19] Andrew Zigler: Are they gonna be automated away, and is the ladder even there anymore? So, like, what would you say to those engineers that are about to start their career or at the very beginning of it?
[00:47:32] Karthik Ramgopal: It's really hard to predict the future, especially in a transformative like moment like this. But just looking at how humanity has evolved with any technological change, humanity has always found a way to adapt and create new jobs Right, which embrace the change. So I am an optimist, personally speaking, that new kinds of jobs will emerge.
[00:47:53] Karthik Ramgopal: People will have to adapt, but I don't think jobs will just go away in, like, in their entirety. Some jobs will, but [00:48:00] other jobs will emerge to replace them. That is the first part. The second part is that for interns or anyone graduating afresh, I admit this is a very challenging market. Uh, it would be dishonest of me to deny that, right?
[00:48:14] Karthik Ramgopal: But there are skills which are still in value. As I said, if you have strong system fundamentals and you know how to harness AI in the right way to increase your productivity, there is a lot of opportunity right now, It requires effort, it requires investment. N-now, with regards to the interns at LinkedIn, something we are doing this year is that we are reimagining our internship program.
[00:48:41] Karthik Ramgopal: Traditionally, internship programs have followed a, like, one is to one, like, model, where you have a single intern with a single, like, mentor/manager who's sort of guiding the intern through their journey. It has its advantages, right? Predictable execution. It's a time-tested model. You give them a constraint problem, and they do it.
[00:48:59] Karthik Ramgopal: [00:49:00] But this time we are trying something new, right? A subset of interns will do what we are calling as AI native pods, where we are taking three to five interns, putting them under a TLM, a tech lead manager, who will be responsible for leading the pod as well as, you know, like, managing them. It's a more experienced person, right, than what we would ordinarily put.
[00:49:20] Karthik Ramgopal: And then we are like, "Hey, we'll give you a very ambitious problem. Use AI and use this pod, as we call it, of three to five interns plus the TLM, to actually go solve it." You can lean in on each other. You can lean in on your TLM to sort of navigate this moment together. You have grown up in a AI native environment, you as in the interns, so you can educate or, you know, almost, I mean, okay, mentor the TLM in some way.
[00:49:53] Karthik Ramgopal: And the TLM has experience, real-world experience, and then they bring that. So you... What we [00:50:00] are expecting to see on the basis of what we've seen with some of the other teams at LinkedIn is this culture of two-way, like, mentorship, learning, and compounded productivity to solve ambitious real-world business problems.
[00:50:15] Andrew Zigler: That's an exciting way to reimagine the program, and I... You know, bringing in more interns as well and giving them more opportunities and more exposure to high-level problems is really exciting. Like, I know that that's how, what I would be, what would, I would be enticed to do. Give me a really hard problem.
[00:50:31] Andrew Zigler: Just let me see what I could do without all of the assumptions and baggage of the way that y'all work, because that's really the two-way learning street that you get from
[00:50:38] Karthik Ramgopal: Absolutely
[00:50:39] Andrew Zigler: and there's so much opportunity for that now. And so that's, like, such a smart, uh, rein- reimagination of it. And, uh, just kind of to bring this to a close, you know, Karthik, we've covered so much ground today about how you are thinking about agentic transformation within your org, but then also how you enacted it, the primitives that you think will scale, and the bets that you're [00:51:00] making on how you build your internal tools to scale alongside them. But then also too about, like, what skills look like in the future, what the future of that workforce might even evolve to do. You know, and I'm sure folks will have a lot more questions about what they learned today and wanna go and, and follow you and, and, and learn more about what you're doing. Uh, where can folks go to, to stay in touch with you?
[00:51:21] Karthik Ramgopal: I actually post most of my content on, like, LinkedIn, so just follow me on LinkedIn. And yeah, shocker, right? Yeah.
[00:51:31] Andrew Zigler: Okay, perfect. Well, then we'll make sure the links get into the show notes today. And to those listening, you know, if you're not already following Dev Interrupted on LinkedIn or Substack, be sure to do so. There's a newsletter that accompanies this episode that comes out every Tuesday with our expert guests, so you can learn more about what we talked about today as follow...
[00:51:49] Andrew Zigler: As well as follow the links to, uh, LinkedIn, where you can find both Karthik and I and continue the conversation. Uh, and be sure to follow Karthik as well so you can stay in touch with his content. And Karthik, [00:52:00] thank you again for coming on Dev Interrupted. It was a total blast to have you here, and I can't wait to have you back sometime.
[00:52:06] Karthik Ramgopal: Thank you so much for the opportunity. It was great chatting about all of this stuff. Thank you.
[00:52:10] Andrew Zigler: Thank you.
