Before Backstage became the industry standard for developer portals, Spotify’s engineers relied on spreadsheets to navigate their massive microservices ecosystem.
Tyson Singer, Spotify’s Head of Technology and Platforms, joins us to trace the evolution of their internal developer experience from a necessity for order into the open-source giant Backstage and its new SaaS evolution, Portal. We dig into how they use golden paths to align autonomous squads and how their new AI Knowledge Assistant (AiKA) reduced internal support tickets by nearly 50% while protecting developer flow. Finally, Tyson shares his philosophy on sustainable innovation, explaining how to train an engineering organization to run a marathon at a sprinter's pace.
Show Notes
- Spotify Engineering Blog: engineering.spotify.com
- Spotify Portal: backstage.spotify.com
- Confidence: confidence.spotify.com
- Connect with Tyson: LinkedIn
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: Today we're joined by Tyson Singer, Spotify's head of Technology and Platforms. He's the person responsible for turning engineering chaos into harmony for teams and Tyson's fingerprints are on everything at Spotify's internal developer experience. From backstage to the open source platforms that reshaped to how engineering orgs operate across.
[00:00:21] Andrew Zigler: The industry. And he spent nearly a decade steering Spotify's massive tech ecosystem. And now he's doing it again with Portal, a next generation internal developer platform built from hard earned lessons from this whole experience. And we're gonna dig into some of those learnings today as where is where your team could take it.
[00:00:40] Andrew Zigler: And Tyson, welcome to Dev Interrupted.
[00:00:43] Tyson Singer: Thanks Andrew. Appreciate, uh, you inviting me and I'm excited to have this conversation.
[00:00:47] Andrew Zigler: We're really excited too. Uh, Spotify comes up a lot in our discussions with engineering leaders. It's a widely accepted, adopted leader and, and, and tool to look towards, uh, for getting teams aligned [00:01:00] with their own internal practices, which is something that we talk about constantly on the show. As well as like making, uh, forest superior developer experience.
[00:01:07] Andrew Zigler: So, you know, we're gonna kind of dive into the journey of how Spotify ideated and created and scaled and delivered this kind of experience. Um, But we're gonna start by looking a little bit at the past from where this kind of technology even comes from, because inside Spotify, before backstage even existed, Engineers were living in effectively like whisper networks, right? You're asking around for docs and for services. Who owns what? Where's this resource? How do I get started with this? So how every good journey starts. How did that chaos evolve into what then becomes now such a widely influential platform model for developer experience?
[00:01:49] Tyson Singer: Yeah. So, one thing to maybe call out in that question is that. Even today, Spotify is actually a very network oriented company. It's really [00:02:00] important that we stay synchronized, uh, across, kind of everything we do and that has a very human element to it. But we are so large now, the weight of all the connections between all the people just doesn't scale very well.
[00:02:14] Tyson Singer: So if we look back to about 20, sometime in the 2010s, what was going on in the software industry was, uh, in order to enable scale, and I'll say scale on a couple different dimensions. One from a, just a compute perspective, like the number of consumers sitting in your platform, and then the other from like a people perspective.
[00:02:34] Tyson Singer: Uh, we were shifting, uh, from these sort of large monolithic software, uh, approaches where all the dependencies were resolved, kind of build time to much more smaller independent scale software components where things were resolved more upon time. To maybe oversimplify it. And, uh, so at Spotify that was really happening [00:03:00] kind of on steroids.
[00:03:01] Tyson Singer: It was, Spotify's always been a very extreme, uh, context. We were growing fast and we had a very microservices oriented architecture. So thousands of engineers, tens of thousands of software components, and, uh. You know, we had to make sure that the system was reliably running and and scaling. We started doing that, you know, kind of using, you know, sticks and wires, uh, AKA spreadsheets and to kind of map out how, how things really work together.
[00:03:31] Tyson Singer: And, uh, that, that was sort of fine initially. But what we found was that you know, as we scaled out, we had to figure out. Who was behind the software, and that was changing really rapidly. And so if you're dependent on a component, you need to, you need to talk to the person who you're dependent on to make changes.
[00:03:53] Tyson Singer: You have to find the, the relevant documentation. Uh, to understand things. [00:04:00] And, uh, what we also saw is because we had a really, really autonomous culture, uh, trying to hire the best and the brightest, everybody came in with their own background and how they wanted to develop software and wanted to do things independently.
[00:04:14] Tyson Singer: So we saw a lot of like. People duplicating capabilities in our ecosystem. 'cause they couldn't quite see like, has somebody already built that? Uh, or can I reuse it? So we kinda went from this like, all right, we have a bunch of spreadsheets to, we need to be more organized about this. Have like a single managed repository for our entire software catalog.
[00:04:34] Tyson Singer: That provided this both, uh, sort of a build time and a runtime visibility into the dependencies and most importantly, the people who were involved in that. So that we could really manage this rapidly changing ecosystem. We had, you know, we, we have this idea of a squad team and squads would split and the software they own would move from here to there, and we wanted a system that could.
[00:04:56] Tyson Singer: move at the pace that we we needed. So that's, that's [00:05:00] sort of the starting point. I'll, I'll pause to see if you have like some more thoughts about that.
[00:05:04] Andrew Zigler: No, I, I, I love how you painted the scene. You took us back in time for a minute to is the 2010s you're doing what every large company is doing with monolithic software. You're breaking it up into micro services, you're, you have hordes of engineers that are owning all these small services and one-off, uh, you know, types of implementations of things and you're trying to reduce people duplicating and kind of reinventing the wheel to, so to speak, while still allowing a high degree of autonomy.
[00:05:32] Andrew Zigler: And ownership over the engineering decisions that are making you best in class at the time. That's why all those best amazing engineers wanted to work there still do is because you were creating this environment for them. So I like that you, you know, that's a very relatable experience, I think to any large company that they were, they, we were all doing that in, in the 2010s.
[00:05:52] Andrew Zigler: And so, uh, it sounds almost too that. From the beginning, you tackled it not just as technical debt, but as [00:06:00] a type of cultural debt too. People are coming in and they have misaligned expectations, or they wanna do things that are specific ways um, they've assumptions that are maybe frozen, right? And so you're trying to create this cohesive experience so that now this, now, now that starts to evolve from, from like a.
[00:06:19] Andrew Zigler: I, I net, I guess, from a network. And you say it still is, where does it go from that point onward.
[00:06:26] Tyson Singer: Yeah, so, those were some of the sort of the initial, uh, challenges that we saw, but we needed to, as you just pointed out, we needed to get people more aligned to a similar way of doing things. And so we had a, a number of different tools to do that. One of the, you know, most powerful tools we had at that time was because we were growing very fast is we had our onboarding on onboarding boot camps, uh, and.
[00:06:51] Tyson Singer: We basically indoctrinated people right then, uh, with our, uh, golden paths, our golden tech, our our tech standards, our quality [00:07:00] standards. And so this was like new people coming in trying to get them aligned to things. Of course, all the existing people didn't necessarily have that was same expectations 'cause we hadn't set them.
[00:07:09] Tyson Singer: And so you're trying to change the expectations for, for people coming in, still dealing with like legacy expectations. But what you, we couldn't see really well without more tooling was where people were diverging, uh, from our expectations and from those golden paths and the golden state and the golden technology.
[00:07:29] Tyson Singer: And since, you know, we sort of, the engineering leadership couldn't see it. Well, the people who are doing the work couldn't see it either, so we needed to bake in visibility. Uh, so that everybody could see, oh, okay. Like what am I even trying to manage to, and so having a tool that allowed for effective visualization of that, really putting it at the fingertips of the engineers to say like, okay, this is actually the, the easy way to do things.
[00:07:56] Tyson Singer: This is the right way to do things and you can see it, [00:08:00] uh, and you can visualize it. So, you know, it wasn't just about, you know, getting that linkage between the software catalog and the people, but it was also getting that linkage between, uh, incentives, uh, and. Awareness and visibility on what people were doing.
[00:08:16] Tyson Singer: That then kind of all eventually got bundled up into this product. We eventually called Backstage to sort of match with our, our music focus, uh, culture.
[00:08:25] Andrew Zigler: And so backstage ultimately arises. From the internal curation of. That whole developer experience from having the catalog of tools to having the network of talented engineers, but then also having the golden paths in the very visible, replicable ways of picking the, picking the tool out of the toolbox, so to speak.
[00:08:45] Andrew Zigler: And it's already ready to go. You can go and tackle today's problems. And so, uh, you know, I, I think a lot, a lot of teams can get to that point, you know, uh, to give them some credit. A lot of, a lot of teams can think about what are those communication issues that are stopping us? [00:09:00] How do we align our developer catalog of what we have to accomplish?
[00:09:03] Andrew Zigler: And then how do we put it in front of everybody and repeat it at every town hall meeting? Right? Any, I, anybody, anybody who's making software could do that with a large team. But then how do you then take it from just being the, these, this loose. Uh, group of ideas into being backstage, you've ended up productizing it.
[00:09:22] Andrew Zigler: It becomes a tool that other teams can pick up off the shelf, break open, and then be like the, these are all of the parts we need to have. A successful developer experience where all these things are um, laid out for us. Right. was that a conscious decision that you and other Spotify tech lead, like engineering leaders found yourselves in early or was it at the end and you were like, oh, this is something that's ready to productize?
[00:09:47] Andrew Zigler: Like how close was it to being at that point?
[00:09:50] Tyson Singer: Yeah, there were, there were a number of points where it was quite a conscious decision. Uh, when I started at Spotify, what I noticed was I had this [00:10:00] amazing, team of smart folks, uh, a lot of the. M were very focused on infrastructure capabilities and doing that really well, and maybe a little less focused on, uh.
[00:10:11] Tyson Singer: The sort of customer needs in, in this case, I'll, I'll refer to the customer as sort of the feature, uh, teams across the organization. And so, you know, building out this platform was kind of like nothing special in one sense. It's just like standard product development. To understand your customer needs and workflows, you have to build your quantitative and qualitative insights on their, their behavior.
[00:10:31] Tyson Singer: You have to develop your hypothesis and your value propositions, and. You know, kind of pre treat it as, as anything that you would do since, you know, Spotify is a consumer centered company, or very data and insights oriented for that part, it was pretty easy to kind of take those skills and add them towards, uh, an internal customer.
[00:10:53] Tyson Singer: And I think that's what a lot of folks weren't doing, at that time, they weren't saying, oh, I can take all these same patterns and I can apply them [00:11:00] to my internal customers and build out internal platforms at that time. So, but that's what we did. Uh, and we bolstered our product management and our design team and our insights function to ensure that the work that we were doing to support the effectiveness of our engineers and our developers and the, the ecosystem really had a high ROI. That was sort of an important part of just setting the, the framework there. One of the things that really worked for us to drive adoption, uh, very well was to make the platform something that everybody could participate in. So, uh, backstage was built with this idea of sort of adding capabilities into it dynamically.
[00:11:42] Tyson Singer: Uh, so yes, my teams would build out the, the core of it. Uh, they build out a lot of the, the core capabilities and workflows, but then in a feature team, anybody could come in and contribute. They could say like, oh, like, I really need to understand this aspect of my software and I want to visualize it and measure it, and, [00:12:00] and do those sorts of things.
[00:12:01] Tyson Singer: And so they could do that. So the capability was there in order to get people to engage with that capability. We set up some hack weeks. Uh, I think this is probably in. You know, 2019 um, where we said like, Hey, go and hack on this. See what, see what you can do to make this platform like yours. The end of that year, I think we had more than a hundred different plugins, uh, from across the organization contributing into that.
[00:12:28] Tyson Singer: And so it's sort of also applying these techniques for building a platform where you really engage your user base to contribute back to it. And since our user base was developers, they could contribute back to that in a way that a lot of users can't. They can literally, you know, shape it and, and, and build it out. And, you know, eventually we took that further because we open source backstage allowed the community to engage in that, follow that same pattern sort of flow back into the, the value ecosystem and just sort of continue that, that journey with our [00:13:00] commercial product as well, uh, a bit later.
[00:13:02] Andrew Zigler: so Spotify becomes community zero for backstage. You get everybody on the platform contributing to the platform. You have these hackathon days, uh, structured around, you know, just putting everything else down and just working on our developer experience, by the way, a really common tactic of really high performing teams with excellent developer productivity and experience who are able to tie that work back into results like you mentioned.
[00:13:30] Andrew Zigler: OI that was something that's very important from the whole time building this experience is you're not making it just to make it, you're not, you know, picking out the nice, beautiful cutlery to put on the table just to set the table nice like you're there to for a purpose. And so, everything had a high degree of utility while still allowing people to exercise freedom.
[00:13:49] Andrew Zigler: and of course all of this becomes an open source tool that other people can use. So in, in, in that world. We, we talked a bit about the past. [00:14:00] That's like the environment that made that possible. All of the levels of iteration from like the generations of Spotify leaders and engineering folks who have been there, have been building on, on, on backstage the beginning.
[00:14:12] Andrew Zigler: But now we're coming into the present and, Backstage itself and what backstage is needing to be is evolving pretty rapidly. The engineering world that we are in this year is very different from the engineering world that we were in last year.
[00:14:26] Andrew Zigler: And so, what are those hard learned lessons of backstage that you're carrying now with you into portal into the other things that we're gonna be talking about Spotify is working on. You built that foundation and then how has, how has the last year broken that foundation and changed your expectations?
[00:14:45] Tyson Singer: Yeah. So, uh, maybe it's good to explain a little bit of, of where we came from. So we, we decided to open source backstage, uh, probably 2020. And part of the reason we did that was because [00:15:00] we could see that, you know, this process, uh, platformizing, The product was very valuable for our context and that other folks would be able to to do that.
[00:15:09] Tyson Singer: But it was also to some degree, like a selfish decision, which is, we can see that when there is a very, very clear need that everybody has, then someone's gonna come along and solve that. Uh, they're either gonna do it commercially or they're gonna do it in open source. And we had deeply learned experiences that replacing things, uh, and migrating to new products is really expensive.
[00:15:35] Tyson Singer: And so we basically made the bet that it would be less expensive for us to share our solution, to make it the industry standard, uh, for, uh, what eventually became defined as, uh, internal developer portals than to try to like. Go out and, uh, replace it at some point. Uh, so that was sort of almost like a, just a, or a high in an investment, uh, decision from, from one perspective.[00:16:00]
[00:16:00] Tyson Singer: Uh, but you get a lot of value back from it. You get a lot of value from the, the developer community. Uh, they. They, they engage with backstage in particular, we had one of the highest engagements of any open source project from external parties. So typically when you open source something, it's really mostly the open sourcer who does all the work.
[00:16:18] Tyson Singer: But we had very, very different uh, statistics, like 40% of the contributions were coming externally. So that was, that was really validating on the strategy overall. And it worked out. It, it played out well, but we of course made some. So I, I don't know if that we, we made mistakes, but we built in things that didn't necessarily work for the community.
[00:16:39] Tyson Singer: So we put a lot of effort initially in the front end of those plugin, that plugin ecosystem that I mentioned before. And then kind of said like, great and integrate it with your backend system. Turns out that didn't really meet everybody's needs in the community. Uh, and so, uh, we spent basically a year, uh, rewriting that [00:17:00] backend, uh, plugin ecosystem with the community.
[00:17:04] Tyson Singer: Uh, and that was pretty painful, like when you have to kind of do a lot of that low level, uh, plumbing, uh, to, to like rework things, uh, so that, you know. Uh, more people can participate in the ecosystem and in, in more you know, flexible ways than, than we started out with. So that was, that was really good for the community.
[00:17:22] Tyson Singer: I think over time it was also better for us because usually when you have the opportunity to rebuild something, you build it better. Uh, we have rebuilt backstage a few times before we, we got it out the door as well. And so those were all useful iterations for us. And then also useful as we started go towards our commercial product.
[00:17:38] Tyson Singer: And because, it provides a more sort of no-code, uh, approach, low-code, no-code approach for the full plugin ecosystem inside of Spotify portal. So those things, you know, come back in, in, in goodwill, uh, to, to, to your approach as well, uh, later sometimes.
[00:17:56] Andrew Zigler: Yeah, imagine being Spotify and being so big that you can just be like, oh, [00:18:00] you know, it's actually cheaper for us to just define IDP as a category than it is to change or move to anything else. I love that as a conclusion.
[00:18:07] Tyson Singer: It's not that hard to imagine. Uh, we, we did open source, uh, a project called Helios, which is our, uh, our orchestration capability. And, uh, Google did it the same thing like a day before us. Uh, and we know how that went. Like Kubernetes, uh, one replacing our system with Kubernetes has been, uh, extremely expensive.
[00:18:28] Tyson Singer: So we, we, we know the reality of it,
[00:18:31] Andrew Zigler: Absolutely.
[00:18:32] Tyson Singer: not to be a startup at that point, to, to be able to do those sorts of things.
[00:18:35] Andrew Zigler: Absolutely. And so take those lessons and you, I mean, you, you also described this anecdote of having to rebuild the back end of some parts in order to make it more broadly applicable. That's a, you know, that's a definitely a hard lesson. It's way easier to already have the plumbing in place than to be up on the stage where everyone's watching expectantly waiting leave for you to ship the plumbing, watching you pick up one [00:19:00] pipe and everyone's going, no bad pipe, bad pipe.
[00:19:02] Andrew Zigler: And you put it down and you pick up a different shape pipe and more people are yelling, bad pipe, bad pipe, you know, it doesn't feel like it's a good, uh, solution. So you had to work through this, like fixing it in the open. But then you get through it. And so, whereas then portal come from in these, in, in this evolution because it sounds like, oh, you know, you're checking all the boxes.
[00:19:21] Andrew Zigler: Backstage is perfect. Why? Why started a new product name called Portal.
[00:19:26] Tyson Singer: Yeah. So, uh, on our journey, uh, through, uh, evolving backstage. first we open sourced it. Uh, we got a lot of community engagement, a lot of positive feedback. One of the, you know, strongest, uh, open source projects out of the gate and then just grew from there. And we said, okay, this is a cheaper than doing a replacement, but it's not cheap.
[00:19:47] Tyson Singer: So, uh, it's great if you can, uh, sort of amortize some of the cost of that development work. And we launched, uh, what we call our commercial bundle of plugins. Uh, so basically a set of plugins [00:20:00] that weren't things that we put into the open source ecosystem, but we thought like, great. Uh, this will help people go faster.
[00:20:05] Tyson Singer: This will be a revenue stream back into, to our ecosystem to help cover our, our costs and, and potentially more. And that has been a really acceptable, uh, really well, uh, accepted by the community. It's really grown. A lot, lot of people have, are interested in, uh, that capability. And then we started to see that there were more challenges that folks were out and having out there.
[00:20:26] Tyson Singer: So, you know, for us who created. backstage in the, the category of IDPs. Like, we had a bunch of sort of assumptions like, yeah, sure, this is easy. Like we can, you know, set this up and manage it and do all these things in, in the Spotify specific ways. And a lot of other companies were like, yes, you can.
[00:20:44] Tyson Singer: Well, uh, we, we don't really wanna do that. Like, that's not our core business. Could, could you do that for us? And we said, yeah, we could probably do that for you. Uh, and so that led us to sort of a SaaS-based, uh, product, which we called a portal. We can't actually call it sort of backstage portal [00:21:00] because when you open source something, you give away your rights, uh, to, to the naming and the branding.
[00:21:04] Tyson Singer: Uh, so it is our, uh, our version of Backstage in its sort of commercial form. Uh, and so that's why we call it portal.
[00:21:13] Andrew Zigler: Okay, so then you arrive at the, at, at the SaaS offering and you're, and you're offering it as a solution for other, other teams, other engineering. The orgs, the pickup off the shelf use as a service through you. So in, in those conversations and those relationships, how does Spotify show and measure and, and, and, and help team under teams understand the impact of those tools and to, to get it fine tuned?
[00:21:40] Andrew Zigler: Because obviously it's so flexible. Every team that picks it up is gonna have to wildly probably change it to make it fit them for the exact needs that they came to you for. In the first place. So, you know, how, how, how do you go about thinking about those conversations? What do you, what do you measure?
[00:21:53] Andrew Zigler: What's most important?
[00:21:55] Tyson Singer: Yeah. So. What we do in, uh, our approach is we, you know, talk with a [00:22:00] potential customer and we try to understand, you know, their context, uh, the outcomes they have, their culture, how it differs from our culture and explain to them how the product can be customized for their context. So we have a plugin called soundcheck.
[00:22:16] Tyson Singer: Soundcheck is a way to, uh, help people align to whatever technology, quality or other standards you would like in your organization, and to display that and visualize that and give the, the, the team members who are, you know, doing the actual work actionable ways to, to move on that. And so out the door we provide like, basically a whole bunch of defaults, which we are like, these are the kind of the Spotify way.
[00:22:43] Tyson Singer: But then we realized that, of course, no organization is like Spotify. Every organization is unique and so they need to be able to customize and adapt that. So that's sort of how we integrate, you know, uh, interact on the, with the customers in those type of contexts. And it just kind of goes off [00:23:00] for the different capabilities we have, uh, across portal.
[00:23:04] Andrew Zigler: Yeah, I like that you acknowledge that when teams come in and they adopt it, it's largely up. It's part, you know, a, a cultural adoption in terms of you, you, you look at it and you, you, you study them and you understand what are their needs? Why are they coming to us, what are they looking to achieve? And it, it really acknowledges the fact that each engineering org, it's kind of like its own living organism and each kind of need their own different kind of treatment and help, you know,
[00:23:28] Tyson Singer: Yeah, go. Going
[00:23:29] Andrew Zigler: is like.
[00:23:29] Tyson Singer: to your question about sort of hard learnings, like, sometimes we kind of, you know, not very humbly think, wow. We're like, we're so unique and great. Uh. But then we forget that we are unique. I, I won't say anything about the great part, but like that we are, we are actually unique and that other.
[00:23:47] Tyson Singer: Companies have different requirements and needs, and we hadn't necessarily understood the variety of those things. So you go through this process of building out a product, you learn those patterns, and then those patterns become [00:24:00] useful to other people because there are actually sort of a more finite set of patterns, uh, of like, this is how companies operate.
[00:24:07] Tyson Singer: Spotify, for example, is a company that is in sort of a less regulated space than some of the folks who who use portal. We have regulatory requirements. We have soc requirements for a certain part of our software pipeline, et cetera. So we have that, but like some customers have that in spades. And so they need to apply those patterns more aggressively.
[00:24:28] Tyson Singer: They need to do more of the aggregate metrics things and feature rollouts and, you know, a little bit more of those stronger guardrails. like they're, it, it's a different requirement. One's not better than the other. They're just different. Uh, and so having a, a platform that allows people to do that is really, really important.
[00:24:48] Andrew Zigler: You know, you're, you're wise to draw circles around the groups of users and find the cohorts and then understand that those cohorts kind of have similar needs to each other, obviously at different levels of intensity. But, even [00:25:00] just by grouping them in those loose ways, understanding like, oh, this person's in a regulated or less regulated industry.
[00:25:05] Andrew Zigler: This typically impacts the, you know, how they use our product in this, this, and this way. I think that's a smart learning for anybody who's building a SaaS offering and, and who's dealing with a large array of customer sizes is the cohorts and understanding why they come to you, especially for something so, ubiquitously, broadly customizable portal is, you know, you're talking about a, uh, everyone's engineering org. Everyone has one at this point. Uh, but they all look incredibly different. And um, speaking of looking incredibly different, I think it's time in our conversation to. Talk about how AI is impacting some of this stuff. And it's actually kind of refreshing because this has been a nice long conversation where neither of us have really brought up ai, and those are pretty rare these days.
[00:25:50] Andrew Zigler: But you did it. I, you can't possibly think I was gonna let you get away from this one, Tyson, I, I do have some questions about how AI is coming in and impacting this because you [00:26:00] know, first and foremost. You, you get to the point that you're at, you've built backstage, you've built portal, you have this community, you have so much momentum behind this experience, and AI kind of comes in like a bus and hits it, you know, on the side.
[00:26:14] Andrew Zigler: How are y'all thinking about where AI fits into all of what you've built?
[00:26:19] Tyson Singer: Yeah, so I'll give you maybe just a, a few examples to sort of share the directionality of that. one of the, the primary areas where an IDP like backstage really, uh, helps, uh, people with is, uh, discovery. And so you're specifically trying to discover your software ecosystem, how it's linked together, how it's linked to the, to people.
[00:26:46] Tyson Singer: Where, like, and, and all the documentation and all that sort of stuff that is, uh, related to that e that ecosystem. So as sort of gen AI swept through, uh, the world, uh, what [00:27:00] happened sort of back in 2023 is we're like, oh, right. Look at this like, amazing potential for expressing and solving that problem.
[00:27:09] Tyson Singer: Just like a better way. How can you instead of making someone go across the documentation and searching for it, just they can just ask the question, it can come back with a really high quality answer. That sounds very simple. Uh, but it actually required a fair amount of context engineering to look at all the different knowledge sources that were important in that, that that context.
[00:27:33] Tyson Singer: And distill that and represent that in, in a really effective way. Uh, first some of our first iterations on that, like the quality of the responses came back not so great. And, uh, I'll talk about in a second, like, what are the metrics that we, we tried to focus on? Uh, anyways, we ended up building this product that we call AI Knowledge Assistant or AiKA.
[00:27:53] Tyson Singer: And, uh, sort of another plugin into our ecosystem, uh, that tries to, uh, [00:28:00] really consolidate all that knowledge, uh, in a very effective and context specific way. And so as we thought about this particular problem, we, we thought about, well, what's the, what, what are we trying to solve? And so one of the biggest opportunities for improvement in the developer ecosystem is to improve the flow for developers.
[00:28:21] Tyson Singer: So I'm doing something. I'm dependent on some other component. Uh, I reach out to you, Andrew, and say like, Hey, I don't understand that thing. Can you help me? Uh, I just Interrupted your flow. My flow was Interrupted too because I couldn't figure out something. Uh, I had to wait for you to wake up 'cause you're on the west coast of the us I'm in, uh, London.
[00:28:39] Tyson Singer: Like, all these things are like flow dis disruptors. And so what we wanted was something that could solve part of that problem at least. And so with AiKA, we actually were managing to those type of outcomes. And what we saw over time, after a few iterations was a basically a 50% reduction, 47 to be precise [00:29:00] in our internal support requests.
[00:29:02] Tyson Singer: Uh, we saw employees using this basically, every day with, you know, more than a thousand employees on a base of about, you know, 4,000 employees. Uh, and, uh, weekly active users at about 85%, 86% of our, our developers. So that told us like, great, there's some like, clear adoption, uh, metrics. Some clear like task, uh, avoidance, uh, type metrics, uh, that really were driving very concrete and specific value so developers could stay in flow.
[00:29:33] Tyson Singer: Both um, sort of both sides of the equation. So really like impactful metrics coming from what on a surface seems like great, it's another chat bot. But there's, you know, a, a lot more context that goes into that in, in terms of having, uh, something that really allows AI to, to have a, a meaningful impact.
[00:29:55] Andrew Zigler: How did you measure and understand you, you, you cited [00:30:00] 47% of like, interruptions prevented effectively, like you're, I I, I like how you define success for AiKA as like protecting the flow state of our engineers. Our engineers are flowing towards whatever goals we've set for them. We already have them aligned for ROI.
[00:30:16] Andrew Zigler: We need to reduce their interruptions, reduce the blockades that they, you know, that they hit along the way. Was this like a cohort? Like did you start AiKA small with a group of people and do before and, with surveys, survey, and I ask this specifically because these, these experiments are things that people are running constantly in their engineering orgs, but everyone kind of emerges with a different level of, like an empirical way of understanding what happens.
[00:30:40] Andrew Zigler: So I'm curious to know Spotify's.
[00:30:42] Tyson Singer: So, so that is sort of roughly what we do. I mean, one of the starting points is that we have added a, a awful lot of instrumentation into our developer ecosystem so that we can see what developers are doing. And that can sound a little creepy. Uh, and I've certainly talked with some large companies that I'll not name, [00:31:00] who've done this sort of stuff and had a really backfire on them.
[00:31:03] Tyson Singer: So for us, we've. Always been super, super, uh, rigorous about saying when we collect metrics, we always represent them at the squad or higher level. We do not expose individual metrics, uh, outward because, you know, that can change incentives and make people feel fearful and all that sort of stuff, and kind of mess with your culture.
[00:31:24] Tyson Singer: And we've been just rigorous around that. So then you have this base where you can actually get high value signals into the ecosystem and then, you know, you can run your experiments. Uh, you can target a, a, a segment of your, your ecosystem. We have this other amazing product that we use for our, sort of our consumer business, uh, which is, uh, we internally call our experimentation platform.
[00:31:46] Tyson Singer: We came up with a much clever, more clever name from a commercial perspective. It's called confidence. And uh, so we can apply that on these type of problems as well. It's, you know, not. It's not tuned for these sort of problems, uh, because it's tuned [00:32:00] for large scale, uh, consumer problems. But we can apply that as well and we can do our experiments on that.
[00:32:04] Tyson Singer: And that's what we've been doing with a lot of the AI tooling that's been coming out, is we basically pick a cohort of people. We say, great, you could use this tool. Uh, we're gonna run a experiment for three months. We're gonna compare it to something else. We're gonna look at a whole, uh, host of metrics because as you know, like.
[00:32:21] Tyson Singer: It's, it's very hard to understand impacts on efficiency and effectiveness. So you're looking at all these sort of proxy metrics and you're trying to offset it with quality metrics. AI is definitely. Know, having, you know, interesting impacts on things like our maintainability, uh, index and our PR commit sizes.
[00:32:41] Tyson Singer: Uh, as, so we're kind of watching all of those things, uh, in the context of these experiments. You know, it's, it's always a little bit messy, so you're trying to read the signals out of a, a lot of noise.
[00:32:53] Andrew Zigler: So you tackle AiKA by, by making it as a, a plugin that plays nicely within the ecosystem, the territory you've already [00:33:00] built and created, and then you, you run and experiment your AI tooling through cohorts because like you said your measurements already work on a team level and rolling upwards.
[00:33:11] Andrew Zigler: So you find a team, you isolate it like you're the team that's gonna use this tool. You're the team that's gonna use something else. We're gonna look at, you know, all of these leading and lagging indicators. We're gonna look at the quality of your code later on, we're gonna do like a full analysis is kind of what I'm
[00:33:25] Tyson Singer: Yeah. To varying degrees sometimes we, we, we can't be quite so precise in saying like. Uh, we're gonna take these five teams and, and, uh, look at them in isolation. We have to just kind of pick individuals out of the ecosystem, uh, and then aggregate them. So we don't, we, we kind of show the results of, Hey, we did the experiment on 60 people.
[00:33:45] Tyson Singer: Not, this is how it, it impacted these five squads. So like, sometimes it's just like a reality of like, who wants to participate and can we get enough density, uh, in, in that, uh, that ecosystem. Uh, it's a little different than when you're doing [00:34:00] experimentation on, you know, hundreds of millions of users.
[00:34:02] Tyson Singer: You have. You, how about bunch of different levers you can use for that.
[00:34:05] Andrew Zigler: Yeah, you have lots of experiments, I'm sure, for scaling and testing things scale, given your consumer user base on top of, you know, your developers and, and, and the ecosystem that you've built. I'm wondering what are the lessons that you think you and Spotify have taken from that about your pacing and how Spotify does pace its innovation for Indu, the industry for its consumers.
[00:34:28] Andrew Zigler: AI I think is tempting for everybody to speed up into a sprint. Including companies that maybe haven't sprinted in a while. And so, that can be a really, uh, you know, hard learning experience for them to kinda start moving again, like an AI native company. how is Spotify evolving and changing to meet the market and all of the opportunities available in it right now?
[00:34:48] Tyson Singer: A lot of folks use the sort of, the analogy of sprinting versus marathoning, and like his sprinting, is it sustainable? We all feel like in this, this current ecosystem that like, you need to [00:35:00] sprint, you need to really go. The way that I try to think about it is my goal as a, a leader of, uh, of the platform organization is to figure out how we can.
[00:35:11] Tyson Singer: Get Spotify moving at a sprinter's pace most of the time. So if you look at the best mar marathoners, I mean if you've ever seen some of these videos of like their pacing relative to like the average person, like their normal pace is faster than say, you and I could possibly sprint. But it's because they've created this sort of the rigor and the training in the systems to allow them to.
[00:35:33] Tyson Singer: To do that. And so that's like my goal at Spotify, which is how do we train the organization so that we can always be in the world class marathon, run our category for our development ecosystem. And so that means really, I think you actually said it pretty well, the, the beginning, which is it's not just about focusing on building scalable systems like backstage, which we do, which is super, you know, important, but also encouraging this like [00:36:00] set of of healthy behaviors and processes in, uh, the culture as well. So investing in, you know, the guardrails early is a very like, concrete way to, to do that, to help people with their, their rigor and their training. Uh, helping people visualize the opportunities. Uh, it's just like. You hire smart people, you show them like what could, what what you want them to do in very clear ways.
[00:36:27] Tyson Singer: They do it. You don't have to do too much more. Now layering on like a, a management system on top top of that to set targets, uh, helps as well. Uh, but overall, like the, the point is you're trying to align your culture and your system and the incentives to. Sort of keep that active training going on. 'cause you also have to kind of change that training like over time if you're, if you're really trying to improve, you can't just do the same thing over and over again.
[00:36:52] Tyson Singer: Like you, you have to be constantly tuning it. So that's how I think about it, which is we, we kind of wanna always be sprinting. But how [00:37:00] do you make sprinting for a full marathon sustainable? And so that's what we're always trying to do.
[00:37:04] Andrew Zigler: I love all those metaphors. It makes me wanna like be in a gym right now. It's like you have to like work out all of these different muscles. You have to rotate through them to be strong. You have to build core strength before you can do feats that people. You know, can't do, but more importantly, you know, it takes persistence and grit and a repetition.
[00:37:24] Andrew Zigler: And, and I think that's a lot of what, you know, developers, they, they show up to do their best work every day. And if you create the right environment for them, like you said. That's gonna happen. You hire smart people to make smart decisions and build smart, amazing things. All you have to do is provide the right kind of environment.
[00:37:40] Andrew Zigler: And this has been a really insightful walkthrough for how Spotify frames these experiences for their own developers, how you've built it into a, a world class tool that's open source that you also then provide as a service to, to other technology companies. Anyone else looking to replicate your [00:38:00] org success, which is a, a huge testament, I think, to the engineering culture that you have built.
[00:38:04] Andrew Zigler: And for those that have listened and I'm sure taken their own lessons for how they can experiment and build with these tools. You know, where could they go to learn more about you, Tyson, and what you're building at Spotify. Uh, maybe even look at portal. I know you mentioned something called confidence.
[00:38:19] Andrew Zigler: where can they go to kind of dig into this stuff?
[00:38:22] Tyson Singer: Yeah, so I would point folks first to our engineering blog, it's engineering@spotify.com. Uh, we have, I think, uh, you know. Not, not being humble again, uh, like a really good engineering blog. There's, there's great, uh, content on there. And then from there, uh, if you're interested in some of the, the products like, portal or confidence, you'll find the links backstage@spotify.com.
[00:38:44] Tyson Singer: You can find out more about, uh, portal. Uh. There are some, you know, interesting things, uh, coming up in the future. Uh, you asked about me. I tend to like to sort of meld into the background. I'm not like a, a super in front of the, the world sort of, uh, person all the [00:39:00] time. But we are doing a, a, a nice documentary on backstage on how, like the whole thing grew from the, the ground up.
[00:39:06] Tyson Singer: you can see all about the, uh, the backstage team and, and everybody in the community who's been, uh, working on that. So that hopefully will be great. And those are, those are some things people can check out.
[00:39:18] Andrew Zigler: Amazing. Well, we're gonna include notes too, the, the first things that you mentioned in the show notes so people can check it out. I am gonna be staying, staying tuned for the documentary. That sounds really cool. We'll definitely be talking about that here on, on the show as well. thanks for joining us in this conversation.
[00:39:33] Andrew Zigler: If you have anything you'd like to add, please come and poke us on LinkedIn. Leave a comment below on Substack, wherever you're listening to this. Uh, we want to continue the conversation and hear what you thought about it. And that's it for this week's Dev Interrupted. See you next time. And Tyson, thanks again for coming on the show.
[00:39:49] Tyson Singer: Thank you so much for having me.
