What if you stopped treating observability as a simple insurance policy and started viewing it as a profit center? This week, Andrew sits down with Honeycomb CEO Christine Yen to explore how observability, data science, and product development are colliding in the agentic era. Christine explains why production signals must become compiler inputs for autonomous agents and how MCP tools are democratizing telemetry for entire organizations. Finally, the two discuss Honeycomb's latest Innovation Week announcements and the exact strategy for reframing observability from basic risk mitigation into a clear revenue accelerant.
Show Notes
- Honeycomb Blog: Read deep dives on SLOs athoneycomb.io/blog
- Honeycomb Innovation Week:Explore the latest announcements
- Required Reading: Check out the bookObservability Engineeringby Charity Majors, Liz Fong-Jones, and George Miranda.
- HumanX Interview:A Codebase Is No Longer the Source of Truth
- "Production is a Compiler Input": Chad Fowler'stake on the future of code generation.
- FollowChristine on LinkedIn
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:43] Andrew Zigler: We're in an impossible battle.
[00:00:45] Andrew Zigler: And traditional modering, monitoring tools are bursting at the seams. The techniques that we used yesterday might need to be
[00:00:52] Andrew Zigler: flipped
[00:00:52] Andrew Zigler: on their head. So we'll explore what this means for developers and the pressing conversations the industry needs to be having right now. [00:01:00] But particularly, we're going to explore how the worlds of observability and data science and engineering and product are all colliding into this one big, new thing.
[00:01:10] Andrew Zigler: Christine, welcome to the show.
[00:01:14] Christine Yen: Thank you so much for having me. I'm excited to be here.
[00:01:17] Andrew Zigler: Honeycomb is a, a big, a longtime friend of the Dev Interrupted show as well. your co-founder Charity has been on here many a time, um, even predating me. She's probably been on more episodes than even I, the host, have in some regards. There was, there was a period of time in my early days where that was certainly, uh, the case.
[00:01:36] Andrew Zigler: So it's really great to have you here and, and, and get your perspective because, uh, I'm a really big fan of all of the, the thinking and the leadership that's coming out of Honeycomb right now. And I really wanna dive into the things that you're, uh, seeing and you're understanding from your role as a CEO of this company of, of really s- looking at the landscape and how it's been changing kind of like right underneath you. You're not an
[00:01:58] Christine Yen: Yeah.
[00:01:58] Andrew Zigler: native company. You [00:02:00] were talking about understanding technology before, uh, all of this transformation happened. So, uh, you know, what, uh, what is like highlighted right now for you in this big wild world and how it's getting shaped, uh, by agents? And how do you think observability is, is fitting into that?
[00:02:17] Christine Yen: I think one of the things that strikes me, um, you started this off by touching on data science. Um, and when I think about observability, w- you know, there's many definitions out there. I just think of it as trying to make sense of what your systems are doing by using data, right? Which is a very broad definition.
[00:02:39] Christine Yen: And when you think about what mechanically that requires, it does sound like a lot of very parallel industries. Business intelligence, data science, product analytics. There's all these adjacent things where we're trying to do very similar things, and I've always been, uh, surprised, you know, historically how [00:03:00] high and thick the walls between these adjacent disciplines tend to be, how little idea sharing there tends to be.
[00:03:06] Christine Yen: And I'm seeing, and we can, and, and imagine we can talk a little bit more about that later if you want, but I'm seeing a version of that happening with talking about agents and agentic development. Whether, whether it's I have Claude helping me write code, or I'm trying to build my customer-facing agents.
[00:03:24] Christine Yen: Like, people are really, really trying to talk about this as if it's something entirely new and entirely different, and we leave all of our traditional software things over here, and we're doing all of our AI things over there. And maybe it is because we've been doing this since before ChatGPT, but I look at it and I'm like, " Guys, it's software."
[00:03:48] Christine Yen: Like, it's weird software. It's non-deterministic software. It's more autonomous software than we're used to. And, um, you know, uh, there's a lot of the, the, the, the shape of how we serve that software has [00:04:00] changed, right? We don't have clean request-response cycles anymore where, you know, you have, um, clients talking to servers and no state's shared in between.
[00:04:10] Christine Yen: Um, the shape has changed, but it is software, and it's interesting to see different organizations sort of working through that idea, um, and figuring out for themselves which ideas to carry over from how they, serve software systems, how they think about reliability of that software, how they think about observability, um, and where they're like, "No, truly our-" Practices from the past do not serve us in this new world, and we need to take this sort of step change, uh, evolution in our practices.
[00:04:41] Andrew Zigler: That, that's, that's definitely, uh, something we've heard a lot from leaders on this show in terms of, really what AI does is it makes more dramatic and [00:05:00] more
[00:05:47] Andrew Zigler: just the, uh, the things that we need to carry forward might need to be in a different arrangement. You might need to repack your s- repack your suitcase, but you're still, you're still taking the same thing, you know? so
[00:05:58] Christine Yen: I think that piece is super interesting, [00:06:00] right? Um, I-- one of the, my l- one of my light bulb moments years ago now with AI, um, was realizing that, like, it was, I was at some event and someone was talking about using AI to generate bedtime stories for their kid. Um, and how their six-year-old was actually really good at being like, "I want a bedtime story about dragons and witches and my left sock, and, um, I want it, I want them to go on an adventure together."
[00:06:27] Christine Yen: Right? Um, and hearing them talk about this, like, what has been hard about communicating with humans forever? Well, it's been about communicating expectations and assumptions upfront, being explicit about what you need, um, and your intent. And when I think about engineering practices, when I think about observability, there is something really exciting.
[00:06:50] Christine Yen: honestly, when I, when I'm talking to a customer about, um, observability in their practices, one of the most valuable conversations I can have with them and then they can go have with their teams are the [00:07:00] conversations outside the tool themselves. It's what does delivering great service to our customers look like?
[00:07:07] Christine Yen: What are the signals that really matter for, you know, to the business where we should page an engineer, um, if something, if something looks off? It's probably not every little, you know, it's probably not CPU hits 70% utilization, CPU hits 75, CPU hits 90, CPU hits 95, right? That- Okay, let's step away from that pattern.
[00:07:27] Christine Yen: It probably is conversations like, um, if our, if we are an e-commerce platform and our checkout rate drops and sustains that drop for some amount of time, right? These conversations that are at a higher level of what matters and, um, what is good are ones that we've-- I, I see teams having all the time now because AI forces those conversations up front.
[00:07:52] Christine Yen: Um, and it is necessary or important for using, you know, AI code generation. and it is [00:08:00] increasingly, I think should be, it both should and is happening, part of using tools like Honeycomb. Um, and so there's, you know, AI taketh, AI one hand taketh, one hand giveth. Uh,
[00:08:15] Andrew Zigler: Give
[00:08:15] Christine Yen: I don't know. It's, there's, there's some really fun shifts happening in how we use our tools and how we do what we are here to do, which is deliver great service to our customers.
[00:08:27] Andrew Zigler: Exactly. Really what the technology has done for many teams is allowed them to close a loop always open where there were people and processes and idea of this is what our North Star, this is what we're aligned to, and we're gonna push, and we're all gonna do really hard, and the product people are gonna do the product stuff, and the design people are gonna do the design stuff, and we're all gonna move this wagon along somehow, right?
[00:08:50] Andrew Zigler: But at the end, we're all gonna then evaluate it and understand it and be like, " Where did we land? What's this good? What's this bad?" Go through our, our goals as a team. Now, because [00:09:00] all of that can also flow into a system, a, a, you know, a, a, a probabilistic system that can do a lot of high impact things at scale because all of that information can flow into that same system. But that same system needs to be a- aligned to rewards and like what you're saying, like objectives and like this is, you know, this machine that can do 100x, a, 1,000x more of this analysis impact than the, than maybe the individual running it could. They now need to know where to point that bazooka very, very accurately, um, in terms of their time and, and their token cost.
[00:09:34] Andrew Zigler: So it really causes like what you're calling, um, like this transformation, you know, that happens within companies where they, uh, have to have those conversations a lot sooner, and then they're better for it because everyone is now like, "Oh, wow, we're really aligned, and my-- me and my, all of my agents are also aligned, and I don't have to exist in this fuzzy space of trying to convince them otherwise."
[00:09:58] Andrew Zigler: And so, uh, [00:10:00] that's really the, what like the big unlock I think that's happening, and observability is definitely the key to closing the loop. Oh, observability is really what is handing the information and allowing that loop
[00:10:10] Christine Yen: Absolutely.
[00:10:13] Andrew Zigler: I like, for me as a go-to-market engineer, like I build agents, I, I work with them constantly, and they're only made possible by observability.
[00:10:22] Andrew Zigler: Observability is a key underpinning for how they work because I won't even begin to build them until people can tell me what they have to do. And so, um, and, and then you work backwards from, uh, from, you know, observability from there. So you're... It, it's like there's a really great advantage that you get to be in to understand then how all of these teams are handing that information across that loop.
[00:10:43] Christine Yen: Well, thank you for setting me up so well. Yeah. I mean, people love to talk about AI as like, "Again, it's this AI problem. It's so new. It's so different." Um, and yes, but for, for that verification piece, for what AI needs to be able to do anything, [00:11:00] um, you know, there are a lot of people who are like, "AI SREs release will be the s- or the second coming, and I never need to do this, need to do this work again."
[00:11:06] Christine Yen: What is, what do those AI SRE agents need? Well, they need data. It, it, everything falls down to a data problem. Everything falls down to where are those agents getting the information about what is normal, what is not normal, what's happening, what, um, should be happening? And all of that is driven by observability tooling.
[00:11:23] Christine Yen: And I think it is, um, I think it is both really exciting to just like crack open this world of what is possible and what can we automate and what can we make faster with humans driving and, and all of this. and also not to, not to go back to my whole ev- it's all just software, but, um, every data problem means, uh, or reducing it to a data problem means that there's like two, two things that really need to be true of the thing that's, is providing the data to your agents, and it's the thing needs to be really fast, especially at scale as we produce more and more telemetry than ever before.
[00:11:57] Christine Yen: And it needs to be able to support, [00:12:00] uh, it needs to be able to provide the agent as much context as the agent can possibly use to figure out what, where to look in the system and, and where, you know, what to investigate. Um, and that context means really rich, you know, schemas or really rich, um, you know, data formats so that you can provide it information about, you know, if you're that e-commerce platform, how your checkout process is doing.
[00:12:27] Andrew Zigler: Exactly. It's like all of those things become those little bridges that are closing the loop, and they
[00:12:31] Christine Yen: Yeah.
[00:12:31] Andrew Zigler: existing within this observability space, um, that, that, that you work within. And, something that also that's happening for these teams that's really kind of, um... It has a side effect that's also then relevant for you and the observability world, and that's that you know, uh, A- AI agents are doing a lot on the knowledge working side, AI coding, AI coding agents, AI in the, in the CLI, in the IDE.
[00:12:57] Andrew Zigler: They are also churning out huge amounts of work, both [00:13:00] in like durable, uh, in the cloud environments that are running 24/7 at scale. We're all seeing these autonomous like coding factories and, and, and groups that are doing this at scale, but then also on the machine. So there's more code than ever before.
[00:13:13] Andrew Zigler: GitHub is literally bursting at the seams, and they're posting a report about it all the time. And if you look at the charts of every Monday morning of everyone opening up their laptop and starting Claude at 8:00 AM Pacific Time, it's
[00:13:24] Christine Yen: Yep.
[00:13:25] Andrew Zigler: uh, staggering. And so we're in this world where like the There's so much code that that code base itself kind of becomes, like, not much of a source of truth. And this is something that I heard you say when you were recently at HumanX. and you spoke there briefly about this, and, and, and there was this clip on YouTube that we're, or we'll include in the show notes as well so our listeners can, can go check it out. You called out this, like, really great observation that because there's so much more code, like, um, that code is not, like, uh, as important as what's the impact of that code.
[00:13:57] Andrew Zigler: And that, that flip has been, like, really hard to do [00:14:00] testing around to understand about, like, how do we build it and understand our software's impact. So I, I wanted to dive more into, like, what, what you've thought about that since you've said it.
[00:14:08] Christine Yen: I stand by it. Um, uh, early on in talking about observability, someone used the phrase, um... Someone started talking about the mental model that we have of our systems, and I, I, I was like, "That is a, that is a great phrase." Because anytime we are trying to map our code base to what is happening in reality, where it's passing through our mental model of like, I'm gonna read the code, put it in my head, and then, and then try to, um, figure out, you know, is what I'm seeing in production aligned with what it's supposed to be doing or is it different?
[00:14:45] Christine Yen: Um, and that was always hard. And that was always, um, something that, uh, I think as systems got more complicated, like just, just left, increasingly left the realm of reality. But now, as you say, or I guess as I [00:15:00] said, the rate of change and the, volume of code is just so much that, um, any mental model we try to build is gonna be immediately obsoleted.
[00:15:10] Christine Yen: Um, there's a blog post that I loved, that I still have open in a tab, uh, by Chad Fowler titled "Production Is a Compiler Input." And I love that sentiment, right? That as we're moving so fast, part of how we maybe should have always worked, but now certainly should be working, is taking these production signals and having that be a part of how the agent is making decisions about how to write code going forward.
[00:15:38] Christine Yen: What real-world conditions are being seen and should be factored into making sure that the, the next set of changes that are pushed out, um, you know, ensure a great experience for your users.
[00:15:49] Andrew Zigler: Yeah. That's the challenge right now is compressing that gap that, that it, uh, bridging that fuzzy context. Fuzzy context is a lot of what I call this, like where humans like you [00:16:00] and I, like we, we, we go to meetings or we have chats and, and tasks and threads and emails and a lot of context that like you could argue in all sorts of different ways that an AI could access. But could they sophisticately, sophisticatedly understand the relations of those, the timing of them? becomes much more of a challenge. So like because the consumption of that fuzzy context is really difficult, it becomes like this, And, and because unlocking that is the key to scaling like a, an engineer's impact or a team's impact beyond an individual and to, to orchestrate further with agents.
[00:16:35] Andrew Zigler: Bec- because of that, the observability tool also then becomes like the learning and teaching mechanism by which the human and the agent figure out how to do that dance. I'm curious like how you've seen... I know at Honeycomb that, that, that y'all are very agentic and, and you exist in an agentic world, like how you've seen the abilities of working with obs- observability to actually then transfer directly [00:17:00] into like engineering competence around being able to orchestrate and work with the tools.
[00:17:04] Christine Yen: Yeah. Um, I think that we-- I think I take pride in is how thoughtful, uh, the Honeycomb team is about where the human goes in the loop, how to not attribute skills, how to, um, you know, we, we have some really great, processes and, and actually blog posts about how we do things like incident reviews to really focus on learning rather than, you know, slap a Band-Aid on it, move on.
[00:17:29] Christine Yen: Um, and I think that that really carries through how we're trying to build, Honeycomb out as a product for our customers. I think that there-- The thing I, I, I think I have liked to say since the beginning is that debugging is inherently a collaborative process, even if it is just present you, past you, and future you, right?
[00:17:50] Christine Yen: 'Cause past you d- wrote something, made some decisions that present you is trying to figure out, and you're trying to, like, m-mitigate, you know, the, the problem for future you. It's--
[00:17:59] Andrew Zigler: [00:18:00] you.
[00:18:00] Christine Yen: Yeah. And, and, and th-there's this question of, again, I, I used this word in this talk track before AI made it a thing, but, um, those, those different versions of you are passing context between each of you, and you're trying to figure out, "Okay, how did I debug this last time, and what signals are interesting?"
[00:18:17] Christine Yen: Um, I think those questions, uh, again, talking in a pre-AI world, so you're onboarding a new team member. Knowledge, like, there's such a role that observability tools can play in that sort of knowledge transfer. Um, the naive approach of this is like, "Here. Hey, um, you know, you're the new engineer on the team.
[00:18:34] Christine Yen: Here's my dashboard that I use. You should use it too." But I think that there's a lot more, um And, and this is something that Honeycomb's always tried to sort of bake into the fabric of our, uh, to mix metaphors, um, to weave into the fabric of our product, this knowledge of other people on your team and, uh, sort of the, the, the trails that they've worn into the grass and how to surface those sort of [00:19:00] investigative cues.
[00:19:02] Christine Yen: Now, in an agentic world that we have these incredible building blocks, um, that can try to do some of these in- investigations for us, um, it, it becomes like a really exciting thing to explore. Uh, again, like a naive approach of this is, oh, document everything that your humans have ever done, put it in a runbook, and then feed it to the agent to follow.
[00:19:23] Christine Yen: Like, that's possible. Um, we think much more interesting is like, okay, when two humans or, or, you know, when you're actively in an incident or you're in some sort of investigation, incident implies an urgency and severity that I don't think applies to everything that's interesting worth investigating.
[00:19:40] Christine Yen: Anyway, so say you're in an investigation, um, how can agents connect dots that you might not have been able to? How can agents help you explore multiple hypotheses at a time? Um, how can... And then how can a human teammate look at what your agents are doing to maybe spot, uh, to spot trends or provide some [00:20:00] context?
[00:20:00] Christine Yen: What if they bring their own agent? What does that look like? Um, what does the collaboration surface then become if you're all, you know, d- does it, does it come out of Slack? Is Slack the primary, um, primary collaboration surface? Uh, by the time this podcast is released, we will have announced, uh, some, some pretty interesting points of view on what that surface area should look like and what that flow should look like.
[00:20:25] Christine Yen: Uh, and, you know, it's sort of just the beginning, but it is off the back of years and years and years of thinking about this collaboration. And, I am contin- I continue to be bullish on humans. I think that humans are always gonna bring some context and judgment and, you know, lateral thinking that LLMs aren't gonna be capable of.
[00:20:45] Christine Yen: but LLMs certainly raise the floor. LLMs certainly like help write down and document things, um, in a way that a tired version of you can at least come back later and, and reference. And I, I think it [00:21:00] is, it is going to be really interesting to see the range of solutions out there, um, over the next few years as vendors explore the surface area of where, where to encapsulate and shield a user from.
[00:21:16] Christine Yen: I see a lot of vendors out there making magical promises to their users about never having to do XYZ again, and where to make those boundaries more porous because there is value to building up, building up knowledge and building up skills in the humans, uh, and humans learning how to use these tools maximally effectively
[00:21:36] Andrew Zigler: Right.
[00:21:37] Christine Yen: Little bit of a ramble, but I, I, I love
[00:21:39] Andrew Zigler: I lo-
[00:21:39] Christine Yen: area.
[00:21:40] Andrew Zigler: I, I mean, I, I love that. That's really the-- That's-- Okay, so that's like what you just hit on the very end there. That's like a lot of the, the danger and the glossing over of just kind of like mainstream kind of consumerism, I think, around AI. Like, "Oh, it's this black box that does these things for you, and it's just so great, and you never have to worry about this again." that's not really the case. It's an
[00:21:59] Christine Yen: Yeah.[00:22:00]
[00:22:00] Andrew Zigler: to sharpen a really, like, specific and personalized tool. But that's gonna take work, and that's gonna take understanding what you even want out of it in the first place, because you can't offload that level of, of engagement in the problem. And so, like, I, I loved w- how you called out of, of it enables this collaboration.
[00:22:20] Andrew Zigler: Uh, well, that's what observability does, rather. It enables this collaboration between, the, the developers, between their agents, between the context. It allows this transfer that is really crucial that not a lot of, a lot of other tooling allows us to do.
[00:22:34] Christine Yen: Yeah, it's like how you orient yourself in a system. It's-- Any observability tool, you're coming in, you're like, "What, what does normal look like?"
[00:22:41] Andrew Zigler: What does
[00:22:42] Christine Yen: I think that the collaboration piece is not a thing that all collab- all observability tools try to answer, and that's
[00:22:47] Andrew Zigler: No. No. The collaboration piece is not. Some pl- some places are, are, are definitely or, or some, some tools are definitely gonna be more focused on, as much signal noise for analysis as possible. It's not really digging [00:23:00] into the impact, the, this hit this objective that we had around it. That's much more subjective, and that's kind of like a, it becomes its own evolved domain of that, right?
[00:23:10] Andrew Zigler: And y-you're kind of, um, you're, you're, you're ju- you're, you're hitting at, like, something I wanted to ask about, kind of like in this world where there are all of these signals, and you have these amazing abilities to transfer context and skills up between the agents and down into the... Or u-up into the humans and down into the agents and sideways. You, you end up in this world where there's a lot of noise. There's a lot of data. There's a lot of information. And for me, as a, as a, as a builder, as a, as a, a worker, as a knowledge worker, as somebody in this space, like, there's a lot, a lot of signals. So then it becomes, like, how do I then cut through that noise and find the highest impact things that I can leverage, that the people around me can leverage, uh, or like how do I just keep it all from drowning out?
[00:23:55] Andrew Zigler: So I'm curious, like, to know from your perspective how you've seen observability where it is more of a [00:24:00] fire hose, how people can then find the, the, like, the, the learnable moment from it.
[00:24:07] Christine Yen: Yeah. I mean, I think you're, you're asking the, um, how do I know what matters,
[00:24:14] Andrew Zigler: Yes.
[00:24:16] Christine Yen: And that's-- It's true. When you're, like, in it, when you're just, like, handed a giant pile of telemetry and you have to go sift through it, that problem in the abstract sounds terrifying. And, for most people, the challenge of investigating a system is not a single moment, like single contained moment in time.
[00:24:36] Christine Yen: It is a longer period of time where you have, uh, some, uh, you know, influence over the system. You have some ability to say, "Hey, I'm gonna emit this telemetry because this telemetry will be useful for, um, investigating." You have the ability to do an investigation, and then you have the ability to feed it back.
[00:24:55] Christine Yen: Um, again, in the naive case, this has resulted in a lot of people building it, [00:25:00] you know, emitting some telemetry, building a dashboard, having an incident, um, having as a postmortem, build more dashboards, and then, then you end up in a sort of vicious cycle where you just have too many dashboards. Uh, I always try to reframe it for our users and customers about having a virtuous cycle around improving your telemetry instead of building dashboards and artifacts, right?
[00:25:19] Christine Yen: Dashboards are just, I think dashboards are frankly also gonna become obsolete in an AI world. Um, if you interact with most of these tools via your own MCP or via their own MCPs and your own agent, like who cares what dashboards are pre-canned? Like you're always gonna ask bespoke questions. Um, which again, points it back to a telemetry problem.
[00:25:38] Christine Yen: Cool. Now that we're all looking at the telemetry, what can we, what fields can we add? How will we know what matters? Well, if you start thinking about, if you, if you start thinking about it from a data point of view, and you think about, um, what does it mean to deliver great software? You know, your, your boss or your boss's boss is probably gonna start saying something like, "Well, you know, our h- high priority customers need to have great [00:26:00] experience in the platform."
[00:26:01] Christine Yen: Great. How do you define high priority customers? How do we capture those bits and store it in our telemetry? What does it mean for g- them to have a great experience? What parts of the product are really important to be, um, you know, low error, h- uh, low error, low latency, um, things? What are the contracts that have been implicit in how we've, we're building our, our s- system that might not be expressed as a, a simple red metric?
[00:26:28] Christine Yen: An example with Honeycomb, we are, you know, a- at our core, a data processing system. Um, we... An internal metric that we look at really closely and have always looked at really closely is how long does it take between the time that a, some, some blob of telemetry hits our API to when it is queryable? That's something that touches across the whole system.
[00:26:52] Christine Yen: That is something that directly impacts your user experience if you're trying to query something that you just sent in and you want a really real-time response. Um, [00:27:00] these questions around what matters then drive how do we capture what matters, um, store it to, to query, and then build this flywheel of making sure that we can capture all the bits that might be useful.
[00:27:17] Christine Yen: One last caveat here. A lot of I imagine a lot of your listeners right now are rolling their eyes. Ah, another observability vendor telling me to capture all the data that serves their business model so well. You know, I've, I've experienced, I've tried to throw... I've accidentally thrown an IP address into my metrics and seen things explode and had people get mad at me like, "You're not gonna catch me again, Christine."
[00:27:37] Christine Yen: And for, for those folks, um, if you're feeling that, I really encourage you to interrogate how that observability vendor, uh, stores your data because, again, to reduce everything to a data problem, um, you know, for an engineering audience, y- [00:28:00] it, it is, it is not so hard to imagine that any data tool is only as good as the data store.
[00:28:05] Christine Yen: And, uh, we are really in the middle of, um, I think a, sort of an in- an inevitable transition from the previous generation of ways that we used to store telemetry data in like a log store over here and a metric store over here and a trace store over here towards sort of the next generation of columnar stores that are meant for really fast analysis, that are meant for you to send lots of metadata that you might need in a way that is really cost-effective.
[00:28:36] Christine Yen: In Honeycomb, it's effectively free. Um, and this is, this is the shift from, you know, the tools that have those silos that re- rely on pre-aggregation, um, you know, cardinality limits that you've run into in, in practice, and then you read about them in documentation after the fact when you're trying to figure out why your cluster fell over.
[00:28:57] Christine Yen: Um, you know, a lot of the problems in the [00:29:00] previous generation around data fidelity and data quality and context are things that this next generation, Honeycomb included, have really taken into account and built for. So for anyone in the audience rolling your eyes, feeling deeply skeptical about observability vendors telling you to capture, uh, richer data, take another look, um, because things are changing.
[00:29:23] Christine Yen: Things have changed.
[00:29:24] Andrew Zigler: At the top there you gave us a really powerful strategy for when you're in a, uh, like you're, you're drowning in data and signals and you're trying to understand, "What am I supposed to be learning from this?" You need to interrogate. You need to flip the, the script actually, and you need to start asking questions.
[00:29:41] Andrew Zigler: Because by asking all of these uncomfortable, investigative and, and, and really deeply like core to the, the, the, the consumer and the product and, and the, and the, and the audience of your, of your product, uh, really asking those questions is the only way that you're going to then figure out what is the signal that I [00:30:00] should be measuring to, to finally find something in this sea of, of noise that you and other people can align to, that then you can turn into, you know, nom, nom, nom, delicious agent food and feed to them and get them on the same page.
[00:30:13] Andrew Zigler: And that's all gonna be possible through treating your relationship with observability and telemetry as like the first class citizen of your organization, as the
[00:30:24] Christine Yen: Yeah.
[00:30:24] Andrew Zigler: highway, uh, through which all of the other a- areas and developing things are going to get built and through all of this is going to be dispersed.
[00:30:31] Andrew Zigler: Because, uh, even when I, when I was at HumanX, I hosted a panel with some really great engineering leaders, and it was about go- taking, uh, AI prototypes from demo to production, particularly around like manufacturing and factories and, and
[00:30:48] Christine Yen: Cool.
[00:30:49] Andrew Zigler: you know, 10- like a, a 10% failure rate is like no- nowhere gonna cut it.
[00:30:54] Andrew Zigler: Like, you know, we're talking about serious levels of nines of security and uptime. And, [00:31:00] um, right at the top, one of the, the opening quotes that came from Robert Nishihara, he's the co-founder of, uh, Anyscale. He said, "I've never heard anyone say they over-invested in observability." And that like set the tone for the wh- whole rest of the talk, because everyone it was framing as engineering leaders about how they got to this level of, you know, t- nine nines of sec- of, of accuracy on this particular workflow or dependability in these, in these factory floors by having a lot of these smart protect- protective ways of understanding and engaging with how the problem was solving th- or how the product was solving problems for their customers, and they all were underpinned by observability, by having this really rich, source of telemetry, um, that just like what you said, that was performant, that was modern, that was aggregated, that was, uh, kind of like cut down a lot of the, uh, traditional baggage that was preventing them from access, uh, accessing that information.
[00:31:59] Andrew Zigler: And so with [00:32:00] all of that in one place, it becomes this super tool for product leaders to investigate and understand about what they, what, matters to their customers and what the c- business should care about. But then this also becomes amazing the things that you can align your agents or, uh, the, the workflows or the tools or all of the undercurrents of your-- of the modern, software engineering organization.
[00:32:22] Andrew Zigler: You can align them to, this data as well. And by asking those uncomfortable questions and surfacing those signals that are doing you good over there, also doing them a favor over here because now you have everything you need to map their, uh, objectives to, uh, impact that they can measure and trace and sift out of that data way better than you ever could. Uh, and now you have a virtuous, uh, cycle. So it's like really like the whole, the whole strategy on a plate and, and, and going back to the whole idea that obviously observability is the, the key underpinning of this.
[00:32:54] Christine Yen: Absolutely. ROI question of, like, investing in observability, um, [00:33:00] I imag- I-- There are probably many CFOs that would say that they, that their teams have over-invested in observability, right? And, um, but that's because they're looking at it from a single, you know, one, one lens. Uh, my, my co-founder Charity has a lot of, um, a lot of posts on our blog around how to talk, think about the cost and the ROI on ob- on observability.
[00:33:22] Christine Yen: I think she has a whole chapter in her, the upcoming Observability Engineering book.
[00:33:26] Christine Yen: Well, she's, she's rewritten it for, like, business leaders and talking about this. It's very cool. Um, but one of the, the phrases that has always stuck with me is at, at the core, do you talk about engineering as a cost center or a profit center, right?
[00:33:43] Christine Yen: Is for, especially a lot of companies in tech, the answer is, like, obviously engineering builds the product that we sell. And so obviously there's a tie between, um, the work that we do and profit. And, and yet there are some, there are some classes of tools [00:34:00] that are viewed as risk mitigation or, you know, insurance almost, and so put in the cost category.
[00:34:08] Christine Yen: And there's a lot of, um, arguments to be made that tools like observability tools that ena-enable faster feedback loops, that enable more confident development, that enable more change to be made, more frequently, more confidently, actually is Th-those tools should be moved into the profit center. They should be moved into accelerants.
[00:34:32] Christine Yen: Um, and when I think about the customers that I've seen, you know, use Honeycomb really well, they're tracking things like, you know, they've moved past the have we reduced our downtime, and they've moved towards things like how many of our pull requests or GitHub issues reference Honeycomb URLs?
[00:34:52] Christine Yen: Because that becomes a signal that our engineers are thinking ahead. They are updating their mental models of what's happening in production. They are, [00:35:00] um, building to the reality we can see right now, and that is, like, a really cool way to sort of turn, turn that measurement or, or think about impact in a way that aligns with velocity and, um, you know, positive investment in, in, in engineering org.
[00:35:20] Andrew Zigler: So I have to ask you as well, I'm curious how you think about
[00:35:24] Christine Yen: Hmm.
[00:35:51] Andrew Zigler: Who can update the website or, you know, a marketing copy or, or an engineer who's, who's going into those customer conversations. Uh, that's actually I think one of the biggest [00:36:00] unlocks recently that people have been doing that we all should have been doing all along
[00:36:03] Christine Yen: Yeah. I,
[00:36:09] Andrew Zigler: is getting used. We're using that now because it was a, it's a shortcut to kind of, uh, getting like, you know, cheap and easy and, and raw context just right out of the person's head, and then you just go deal with how you're gonna turn that messy data into action.
[00:36:22] Andrew Zigler: It's not like the highway of observability, but it is a bandage. you're seeing like skills broaden and, and the kinds of folks that are kind of like entering the, the field are much more, uh, diverse. I'm just curious like what you think about how y- like your, your own skill set has changed, how you've seen the skills of engineering leaders, product leaders at Honeycomb change, um, if...
[00:36:44] Andrew Zigler: and, and, and what you think that has done for the whole organization.
[00:36:48] Christine Yen: uh, I'm going to start with something that is, uh, very classic Honeycomb to say, but I promise I'll bring it into the AI, AI era. Um, our whole origin story is Charity was an ops person. I was a [00:37:00] developer. I broke production. Charity would be like, "What did you do to production?" And, you know, I wouldn't understand the graphs that she was putting in front of me Uh, 'cause they weren't built for developers.
[00:37:09] Christine Yen: And so you could say all of Honeycomb is about reducing this gap between development and production. Building this shared, you know, th- building a shared language, building the shared understanding of like what is actually, how are changes we're making to the system impacting our users? Um, and so I think that the, the, the change we've always been trying to drive, that AI in some ways is accelerating and making easier, and what you're, you're sort of describing, is people being more aware of production.
[00:37:35] Christine Yen: Thinking of production and what your users are actually seeing as like a core input into what all of us are doing.
[00:37:40] Andrew Zigler: Mm-hmm.
[00:37:41] Christine Yen: certainly what you're describing, I mean, the example of an engineer being, being on a customer call is really just the qualitative version of what they should be doing with production data already, right?
[00:37:51] Christine Yen: It's just getting it with mouth words instead of telemetry data points.
[00:37:55] Andrew Zigler: raw, raw and just like you figure it out kind of context. Here you go.
[00:37:59] Christine Yen: The [00:38:00] other really exciting and, and interesting, um, shift I'm seeing though, uh, that is a little bit, uh, sort of outside what we are trying to drive, but is an happy effect. Um, it has always been a below the line measure of success in an account for Honeycomb if we see people outside the engineering org reaching for an observability tool, right?
[00:38:21] Christine Yen: 'Cause we b- we aspire to be, you know, the truth about what's happening with your users in production. Um, how are people actually using the software? When we used to see s- uh, support engineers or product managers start to be able to ask, "How's the user using this? What does normal look like?" Before, you know, identifying work or spinning off a ticket, awesome.
[00:38:42] Christine Yen: Today, though, now that MCPs and LLMs have just eliminated the barrier to entry to using these tools, we have one of the heaviest users of our dog food MCP inside Honeycomb is our head of sales. Because who else [00:39:00] cares about how our users are using our software? Well, it's the people who are driving customer conversations.
[00:39:04] Christine Yen: We actually have a customer, um, a sort of a leading inference provider I can't name, um, but you, you'd recognize the name. yes, their engineering team uses Honeycomb, and their whole sales team does. Before they go into a conversation with a customer, they're using Honeycomb, "Hey, how is this customer using our tool?"
[00:39:19] Christine Yen: And they can-- They don't have to figure out how to use, they don't have to figure out how to interpret the graph, they don't have to figure out how to navigate the S- list of SLOs. They can just ask their Claude or their Codex or their whatever to hook into Honeycomb MCP, pass in the name of the customer or ID or whatever they use, and just say like, "Tell me how their traffic has changed."
[00:39:40] Christine Yen: And this, you talked about, you know, all, all these disciplines collapsing. Like it is so cool to be the source of truth for what is happening with your software, which it more and more is just like the re- like what is our reality? And, um- You know, Honeycomb's not gonna turn around and sell to s- sell to sales leaders anytime [00:40:00] soon.
[00:40:00] Christine Yen: But it does feel really natural and exciting and right to have, you know, marketing teams doing gut checks of how healthy is a customer or, um, you know, "Hey, hey product, how is this feature being used?" Or, um, there's all sorts of conversations around the org that can now be served by this source of truth.
[00:40:24] Christine Yen: Um, and it's, it's so fun to think about how relationships and roles continue to evolve When there's no longer a, like, skill, skilling up necessary to use a certain set of tools.
[00:40:36] Andrew Zigler: exactly. When you can be skilled by the harness or the, the tooling around you that you're using that
[00:40:42] Christine Yen: Yeah.
[00:40:42] Andrew Zigler: on your level, because context belongs to everybody,
[00:40:45] Christine Yen: Yes.
[00:40:46] Andrew Zigler: That's matter- that matters is inside of a lot of different people who are all at different levels of investment in the process, including some of them completely outside of the process.
[00:40:54] Andrew Zigler: They're external.
[00:40:56] Christine Yen: And that, you're right. You're right to call out that that actually puts more of the burden on the vendor [00:41:00] to have, uh, to make sure that what we are returning m- is good even with a less, less perfectly formed question coming in.
[00:41:11] Andrew Zigler: Exactly. And so really now every company's job and every employee's job is to become a data scientist,
[00:41:17] Christine Yen: hahaha
[00:41:18] Andrew Zigler: to really understand all of these signals and what matters to me, breaking that
[00:41:23] Christine Yen: Yeah.
[00:41:24] Andrew Zigler: When, and when you start thinking about inputs and outputs at that level, at that level of granularity and capture, uh, and record, uh, you start to be able to really actually form almost, like, more scientific, like, hypotheses.
[00:41:36] Andrew Zigler: You c- you can run your
[00:41:37] Christine Yen: Yes.
[00:41:38] Andrew Zigler: run what matters to you and your customers, like, b- like, uh, business experiments that are just ha- that have just as much of a rigid methodology and, and research data underpinning them as, as anything else that you would call science. And so you're just applying it towards your particular business.
[00:41:55] Andrew Zigler: And, and everyone, I think, has a responsibility to kind of, like, figure out what that means for them. It doesn't mean that everyone [00:42:00] now has to be, like, some, know, very seasoned data researcher, but everyone needs to be able to be fluent in the data that they're in.
[00:42:07] Christine Yen: I would suggest, I think saying that it's everyone's job to become a data scientist carries a lot of like, "Well, what does that mean?" And all of that. I would, I would turn that again maybe 10 degrees and just say it's our job to ask good questions. And like that, in a way, that's always been our job, but now we have more tools and, and, and more powerful things to, and, and more access to more data to ask those good questions on of ourselves, of our systems, of each other.
[00:42:34] Andrew Zigler: So true. You're saying that too. I'm like sitting here, I'm like, host of the show, I'm like, my job is to literally ask questions. I'm like, I, I agree with this so much, but then I also agree it on dual end. It's like I find myself now on an agentic world and the enablement that I've been able to kind of provide myself and my team, I find myself now asking way more questions than I ever did before.
[00:42:56] Andrew Zigler: And it's not even that because my level of like involvement or commitment or [00:43:00] dedication changed. It was just that my ability to leverage those questions into solutions has dramatically shifted. And my
[00:43:08] Christine Yen: Yeah.
[00:43:08] Andrew Zigler: then identify what matters to us most right now, that's all that matters, is figuring out what matters to us the most right now.
[00:43:15] Andrew Zigler: And then I know I can trust that I have this system I can take immediate action on. Because you like eliminate that lag in
[00:43:22] Christine Yen: Yes.
[00:43:23] Andrew Zigler: an idea, you become so much more bolder and brave to think outside of the box and to try new things. But you can only do that if you have like this, you know, this highway of, of data, this, this support, this platform that you can trust and that's verifiable, um, and that, you know, frankly is surfacing signals from everywhere so that you can run all sorts of crazy experiments.
[00:43:44] Christine Yen: Yeah. When the, when the cost of answering a question drops, you can ask more questions, thus building up the reps and ask bet- to ask better questions, which then if the... Like, you just have this beautiful flywheel of being able to go in a bunch of different [00:44:00] directions and think more and do more, um, hopefully build up your, your intuition of what, what hypotheses are worth exploring.
[00:44:08] Andrew Zigler: Absolutely. I'm, I'm really excited to see this all takes us. I've had an amazing chat with you learning about your perspective on how observability is driving all this transformation and the opportunities that are available to everyone right now with the data that's at their fingertips. Uh, it, it aligns so much with things that we talk about on the show, and I know I'm gonna continue thinking about it and sharing things from what I learned today. Um, but just as we wrap up, Christine, where can folks go to learn more about you and your work at Honeycomb and, and all the stuff that we chatted about today?
[00:44:40] Christine Yen: I am not terribly active on the socials anymore. Um, so if someone is interested in finding me, you can probably find me on LinkedIn. Um, but the Honeycomb website, um, and our blog. Our blog, we have a lot of our engineers who are incredibly thoughtful and also great writers. It's kind of unfair. Um, and there's a lot of, uh, there's, [00:45:00] there's just a lot of really great posts about, what does a healthy SLO process look like?
[00:45:04] Christine Yen: What does, um... Or like, how do you determine great SLOs? How do you build a process around it? How do you do incident reviews? Um, a lot of the socio part of the sociotechnical systems we're all trying to build. Um, highly recommend that. I also, by the time this podcast is announced, um, we will have... be on the other side of our innovation week, uh, where, and I believe if you go to honeycomb.io/innovationweek.
[00:45:27] Christine Yen: We'll, we'll drop the actual link in the show notes.
[00:45:29] Andrew Zigler: Yep.
[00:45:30] Christine Yen: uh, there will be a whole bunch of announcements that, um, uh, again, I think nod to how we think engineering teams Should work together to, uh, should work together with AI agents to do these sorts of investigations and, and ask the sorts of questions that we've talked about so far, as well as how to use observability tools to make sense of the new shape of software that we're building today, um, that, you know, the, as we move away from request response, uh, client [00:46:00] server to something a little more complicated.
[00:46:02] Christine Yen: So there's a lot of cool stuff, um, I'm excited to share with the world.
[00:46:05] Andrew Zigler: Well, we love a good engineering blog here at Dev Interrupted, so we are gonna
[00:46:09] Christine Yen: Great.
[00:46:10] Andrew Zigler: All of the great thinking coming out of Honeycomb. And to those listening, if you are not following Dev Interrupted, definitely be sure to follow us wherever you're listening to us today. Don't forget to check us out on LinkedIn and Substack where we publish a weekly newsletter that will be accompanying, uh, this, um, episode with Christine. And you can find both of us, uh, on LinkedIn to continue the conversation. If anything today sparked, uh, a, a thought or a question or tickled your fancy or you really disagreed with it, we would still wanna know, and come find us and let us know. um, while you're at it, definitely be sure to follow Christine and, and check out the Honeycomb blog, which we're going to include. Uh, so Christine, thank you again for coming on the show. It was super fun to have you. We'll have to have you back sometime.
[00:46:52] Christine Yen: Of course. Thanks again. Have a great day.
[00:46:56] Andrew Zigler: Thanks.
