Hate interruptions? Ever feel like you’ve lost your ability to focus on coding?
Katie Wilde, VP of Engineering at Ambassador Labs, knows your pain and she’s on a crusade to help devs everywhere reclaim their focus.
Spoiler: She's got a message for managers who can’t meet with devs whenever they want: “Don’t like it? Well, suck it up.”
The thing about productivity is, you can’t have it both ways. You can either protect your devs’ ability to focus by providing them meaningful time for creativity or you can call them into meetings all day long for constant feedback - but you can’t have both.
These concepts don’t have to be at odds with each other, though. On this week’s episode of Dev Interrupted, Katie details exactly what managers can do to foster a harmonious, productive environment between themselves and their devs. She also talks about the importance of “shifting left,” the dangers of doing it too quickly, and why “hiring great people and getting out of their way” may not be a good mantra.
Episode Highlights Include:
- (5:31) Defragmenting a developer’s calendar
- (8:16) Why video games optimize for flow
- (10:59) It takes 23 minutes to get into a flow state
- (16:58) The problem with being solutions-oriented
- (26:44) When “shifting left” goes wrong
- (36:33) Katie's smallest investment with the biggest ROI
How to Reclaim Your Dev Team’s Focus w/ Ambassador Labs' Katie Wilde
SAT, 05 MAR 2022 03:00:00 -0800 ◦ 40 MINUTES
---
Dan Lines: Host
Katie Wilde: VP of Engineering at Ambassador Labs
---
[Music plays]
Katie: [0:00] As a dev, if you're saying, “Oh, I wish my manager was listening to this podcast.” So what you can do is you can say to them, Hey, I really want to meet with you and tell you all of the things and have the one to one, but I also want to do a lot and be really productive for you. You can have both if we can move the meeting to this other time, and you don’t ping me all the time, or you can ping me all the time and we can have the meeting, but I'm not going to be super productive for you. Or I guess I can be super productive for you, and we maybe don't meet but like what do you want? Try to respect those preferences as much as you can as a manager. And that might mean that as the manager you have a little bit weirder hours. I hate to say this but, kind of suck it up.
Sponsor: [0:43] Over $30 billion in engineering wisdom will be at your fingertips at INTERACT on April 7. Join engineering leaders from Netflix, Slack, Stack Overflow, American Express and more at INTERACT, a free virtual community driven engineering leadership conference. One day, twenty speakers, all selected by the thousands of engineering leaders in the Dev Interrupted community. If you are a developer, team lead, VP or CTO looking to improve your team, this is the conference for you. Go to DevInterrupted.com/INTERACT to register today.
Dan: [1:14] Hey, what’s up, everyone, welcome to Dev Interrupted. I'm your host, Dan Lines, and today I'm joined by Katie Wilde, VP of Engineering at Ambassador Labs. Katie is also going to be joining us for the INTERACT conference. So welcome, Katie.
Katie: [1:29] Hey, Dan, it's great to be here, and definitely check out INTERACT as well. It's gonna be great.
Dan: [1:33] Awesome. Yeah, thanks for the shout out. Let's start by giving our audience the opportunity to know you a little bit better. Can you tell us about your career background, how you got started in engineering? And how you became a VP of engineering? It's an amazing accomplishment.
Katie: [1:51] Yeah, thanks so much. Well, I'm currently VP of Engineering at Ambassador Labs, a cloud native platform leader, so we make dev tools that allow it to be very easy for you to work with Kubernetes, a variety of things. Previously, I was the VP of Engineering at Buffer, and that was where I, you know, first became a VP. I learned that it's very good to just say yes to opportunities. I would say that's how, you know, my career went when there was a need for an engineering manager. I was like, yep, put my hand up. I'm sure I can do this, you know, just jump in, when our CTO quit. And it was, “Okay. Katie, well, do you wanna- do you want to step up? Do you want to try this?”, you know, was a bit daunted, but I said, yes again, and that's been really great for my career. So I would say that's really how I've ended up in this position, I was of course lucky to find people willing to take that bet early on with those first moves into management, those first moves into being an exec. But I also kind of threw myself into it and you know, thought, “Well, I'll try it out. What's the worst that could happen?” Right? Like the worst that can happen is you fail, at least you tried. I think that's paid off so far. So that's what I'd recommend.
Dan: [3:00] Were you ever afraid to say yes? Did you ever have a moment where you were debating it? Or was it always just full [confidence] for you?
Katie: [3:09] Oh, no, like, always, lots of- the thing is, it was always very scary so I just kind of got used to it. I had this experience that I'm sure a lot of the listeners here will relate to. you know, when you're called into, like the CEOs office, and you think like, okay, I'm either going to get fired, or I'm going to get promoted, and that's how it kind of always was, you know, because I would just try to figure out, what is it that needs to happen and try to make that happen? And I would worry that okay, maybe I totally did the wrong thing, or I overstepped the mark, I felt very anxious, and of course, getting promoted very stressful, you know, they say, well, we'd like to promote you to X, and you’re thinking I could barely do the previous job. Now it's gotten even more of a responsibility. So no, this isn't a case of like, just utter confidence all the way through, I think it's more important to not let fear stop you from things.
Dan: [4:00] Yeah, if you're listening, I subscribe to the same philosophy. Let's say if you have an opportunity, and you're a little bit afraid to take it, that's probably the best indicator that you should say yes, because you're kind of challenging yourself. and like for you, you're you been the VP of Engineering, you are a VP of Engineering at these amazing companies. That's the type of things that can happen in the career path.
Katie: [4:24] Totally. I mean, I felt nervous with every step. I mean, I felt nervous to join, you know, a dev tools company coming from social media. I felt like oh, that's a leap, I'm gonna have to, you know, really figure out a lot about this industry. I'm used to SAS and now we're gonna be shipping system software. That's very different and I'm so glad I did. You know, it's just it's incredibly fun. I love dev tools. I'm finding my groove, but at every point, it's been a case of, Wow, I'm a little scared of this opportunity, and those have always been the good ones.
Dan: [4:58] That's perfect. We have an interesting first topic that we're going to talk about here around protecting dead focus. I know you think a lot about productivity and efficiency, we talk about that all the time here at LinearB, we're always trying to do things to save developers time or keep them in flow. It's also about creativity, which I know, you know, we've talked a little bit about behind the scenes, how do you think about protecting developers’ ability to focus?
Katie: [5:30] Well, I think it's incredibly important, because you need to have these sort of chunks of time that you're not interrupted to get into flow, and to be able to be creative. And that's really, really important. So things I think about are, you know, defragmenting people's calendars. Try not to have a lot of meetings, you know, everyone will say that, and make sure that your meetings are very efficient. But something that was a game changer for me was defragment, the calendar so you can put the meetings together, if there needs to be you know, a stand up and a one to one and a skip level, try to have them go, bam, bam, bam, nod, stand up, okay, now we've got two and a half hours, then we got a one to one, now we've got you know, 45 minutes, then we've got a skip level, then there's like two hours, you know, because you just can't really get that deep focus time in those little chunks. So that's something that is really important. With most of our teams here being to some degree remote, a lot of communication is happening in places like Slack, you know, group chat, and that can be a huge time suck for that focus time, because people feel, gotta respond to those Slack pings. I don't want very important decisions being made without me, if somebody pings me and I don’t respond, maybe they're not going to trust that I'm working. So you need to make sure that you're looking at how you're using Slack and is it okay for your team to not be responding for, say, a three hour chunk? Because if it's not, if everyone's always responding right away, like I can tell you one thing, they've got Slack open in one window, and they're kind of reading that, and then they still may be coding a bit and like me, there's going very well
Dan: [7:08] Right, I deal with that as well. I always feel obligation, I guess, to respond in real time as possible. It's very difficult to be like, no, I'm in my focus block, or I gotta get into a flow state. For our listeners who do not know, can you describe like, what is this flow state? What does that even mean?
Katie: [7:30] Yeah, so flow is a technical term coined by the psychologist and behavioral economist, Mihaly Csikszentmihalyi, and I believe he might have won a Nobel Prize for this work, right. So like, it's not just me making up this, like flow vibe. This is like a very serious scientific topic. Flow is characterized as this experience where the task that you're doing is perfectly matched to the skills that you have. So your brain is in this optimal state of engaged challenge. You're not feeling so challenged, that you're stuck, and you're not feeling bored like you could be scrolling Twitter at the same time, right? And video games optimize for flow, why do humans like to play video games? Because video games are flow machines, they stick you in that and it is one of the most satisfying experiences that we can have, as humans, like our brains are hardwired to want flow. So a video game is tuned to give you that flow state where you know, you'll start a game, say and your character will be not very powerful and now there's like the baby boss fight, right? So you're trying to get it and then as you get more skilled, the game gets harder, and it's very, very tuned so that every moment you're in the state of flow, and you can have that experience where you know, you look up and five hours have passed, and you've had a great time. So that is flow and programming is one of the things that also allow us to experience flow. So it's incredibly satisfying. The human brain is hardwired to seek it. It's that optimal problem solving creativity zone, where the brain is just sort of like open and relaxed and engaged and everything is just flowing together. And so this is what we really want, you know, as devs this is what we want to experience for many of us, you know, myself included. this is why we started coding, because we're like, this is fun. This is scratching an itch for my brain, I like it, and as managers and leaders, this is the state that we want our teams to be in. This is that like, highly productive, really creative state. But it means that you have to be in the zone, you have to be able to go deep on these tasks. You're not multitasking, you're not having Slack pings. I mean, imagine trying to like play a video game, but like constantly be responding to Slack, like you just wouldn't be playing it right and it's the same thing with coding.
Dan: [9:58] Totally, I think our audience can read relate to the video games. I’ve been playing a lot of Pinball Infinity Gauntlet, it's a Marvel pinball game, PS4. Great game if anyone has the-
Katie: [10:10] Can’t even get a PS5 too, it’s such a sad state. I'm extra depressed because the New Horizon Zero Dawn is out today and I really wanted to play it on a PS5 and now I don't know, do I do it on PS4? Do I wait? I dunno.
Dan: [10:27] These are tough- we’ll have to do a different pod [crosstalk] [10:29] just because it’s a tough decision of what to do.
Katie: [10:29] Tough decisions, we’ll need an entire podcast, tough tradeoffs, yeah!
Dan: [10:33] But getting into flow. Okay, here's a few things I know about managers, they want to meet with me. They want to do a meeting with me. Yeah, messes up my flow and messes up my calendar. My calendar is becoming defragmented because managers want to know things-
Katie: [10:49] Well, it’s become fragmented.
Dan: [10:50] Oh yeah, fragmented. Yeah. I need to defragment it [crosstalk] [10:54] instead of fragment it.
Katie: [10:54] Defrag that calendar. Yeah.
Dan: [10:56] So like, how long does-I don't know if he even know, but how long does it take to get into a flow state? I assume it's not instantaneous.
Katie: [11:06] I mean, it's random that I know this, but 23 minutes.
Dan: [11:09] You must be studying? [Laughs]
Katie: [11:11] Yeah. I mean, it's very specific answer. Yeah. So if you get if you get pinged, you know, say you got a Slack ping, and you're like, oh, I'll just ask the question. How long does it take you to like, find the thread again and get back? What's that total interrupt time? It's 23 minutes, like that's been measured.
Dan: [11:31] So okay, if it's 23 minutes which, a lot of calendars work in like half hour increments at least, and we're talking about defragmenting calendars, which means getting a lot of space to actually work. But there's something that you're doing proactive. And we also know, like I just said, managers want to get on engineers’ calendars, whether you're doing proactive at your company to ensure that your developers are getting time to get into flow and actually do coding work.
Katie: [12:00] Yeah. Okay, so we have all of our meetings on Mondays and Tuesdays, those are like meeting days. So we encourage our managers, you know, the recurring meetings, those happen on Mondays and Tuesdays. Wednesdays, Thursdays, Fridays, we try not to have meetings. That doesn't mean you cannot have a meeting, you can pair with someone if you want. If your team needs to have a team sync, and they decided, oh my god, I like get on a zoom, this release is going badly, of course, you could do that. But we do not have these recurring meetings on, you know, Wednesday, Thursday, Fridays. And then the second thing is, if you're a manager, and you're scheduling, when you're scheduling, don't look at your calendar, and then find a time and then see where you can slot the engineer in, because that's a very expensive mistake you're making. Look at the engineer's calendar and see, where can you tack the meeting on that it is after another meeting, or it is maybe at the start of the day, the end of the day, their lunch break? And like ask them. You know, some people say, oh, yeah, not very productive at the end of the day, I'm sort of, that's a fine time for me to have a meeting, that's when I like to wrap up and I like to check in on things. While some people will say, you know, it takes me a little bit to get started as the morning. I like starting with a bit of a stand up, bit of a chat, drink my coffee, but then I want to, you know, roll on through the afternoon by myself. So try to respect those preferences as much as you can as a manager, and that might mean that as the manager, you have a little bit weirder hours. I hate to say this, but kind of suck it up? There’s no way to get around that, you know? You might have a little bit of a weird calendar. Then as a dev, if you're saying, “Oh, I wish my manager was listening to this podcast.” So what you can do is you can say to them, it's a menu, you say, hey, I really want to meet with you and tell you all of the things and have the one to one, but I also want to do a lot and be well equipped for you. You can have both if we can move the meeting to this other time and you don't ping me all the time, or you can ping me all the time and we can have the meeting, but I'm not going to be super productive for you. Or I guess I can be super productive for me for you and we maybe don't meet but like, what do you want? And you let the manager pick what they want, you know, and if they want to order the expensive option, if they want the “I want meetings all the time,” and you don't do any code, like, I guess they can order that, like they can have that. But you've surfaced that to them, and you know, I will tell you like. managers want devs to be more productive. So if you say to your manager, hey, you want me to be more productive? Here's an easy win. Like there's a very good chance that they'll pick that option, if they at all can.
Dan: [14:36] Yeah, I think it's good advice. It's kind of like as a manager, you got to think outside yourself. Put yourself in the developer’s position. I don't think it's anyone's doing it on purpose and saying I'd really like to break my developers flow today. No, you're just- Yeah, if you're a developer, you can approach your manager explain it. I need 23 minutes to start my flow then I can get an hour. I actually didn't ask you how long you can be in flow after- I used to think about developing in at least two hour blocks, something like that, I got to get into the mindset, I got to do a cognitive reload on like, what problem am I trying to solve? Then I start coding, then I get an hour or whatever in there. I don't know if there's any data that says it should be like three hour time blocks, do you know anything on that?
Katie: [15:23] I don't have data to hand, but that sounds about right anecdotally. Two to three hour chunks is like a good deep work chunk. There's actually a book, Deep Work by Cal Newport, so if you want to nerd out, get the Cal Newport book, I'm sure it's in there somewhere, I just don’t remember the exact number.
Dan: [15:39] Yeah. Do you have any other tips to encourage these blocks for developers to actually make them happen at organizations? We talked about as a manager, making sure that you're not scheduling in bad places, anything else that you've learned of how to encourage the states of flow?
Katie: [15:59] Another thing that often works really well is, we have every six weeks, there's a two week developer, we call it a cooldown, every six weeks and devs can do whatever it is that they're choosing to do so that's another way.
Dan: [16:13] Awesome!
Katie: [16:14] Yeah, which is really nice. You know, at Buffer, we had hack weeks, again, all recurring meetings are canceled, like there's we’re just gonna hack on staff that can be whatever you want. So there's a lot of ways. There are a few cons, change the day to day mechanics of the organization, you can bring in some of these methods where then at least you can have a high flow week, every so often, you know, whether [crosstalk] [16:38] that happens every six weeks, every month, every whatever it is.
Dan: [16:38] I love that.
Dan: [16:42] Yeah, very cool. Thank you so much for sharing those. I want to move us on to our next topics. Our next topic is around why devs need to change from being solution oriented to problem oriented. So great ideas are always a solution. Can you talk to us about what having an engineering background makes you solutions oriented? And why sometimes that might not always be a good thing?
Katie: [17:08] Yeah, absolutely. So I mean, as engineers, we typically want to come up with things that are answers, we want to build stuff, we want to solve problems, we want to come up with solutions. Often that's why we became engineers and that's why we've become good engineers, and that's often what gets us promoted, you know, at least initially, in the career. You become a senior engineer, because you come up with more solutions, right? You become a manager, if you're good at solving problems with solutions. And the problem that we run into is that, you know, as you start going up the ladder, either on the icy track itself, becoming like a principal engineer, or becoming like a senior manager, or director or a VP, coming up with great solutions is no longer what is the most valuable thing. The valuable thing is getting the problem right. I don't know how many of you might have had that experience where you, you know, built some amazing platform, and it's great, and you're like, this solves everyone's needs. Here we are, we've got this lovely thing, and then no one wants to use it. I mean, this is particularly common in internal tools. You know, I've heard just so many stories of people that come up with like, a great solution. Here are wonderful developer environments, everyone's using them, right? And it's like, no, no, they're still just gonna like, ping that one person that they know, that helps them, you know? Oh, you need to change a config, right? Well, here's a great, easy way to do that, you know, there's like a little GUI. Do people do it? No, they don't, and the problem here is that you haven't understood the problem. And so it's a little bit tricky when you are trained and rewarded for coming up with clever solutions, and then suddenly, you get promoted, you're a team leader, or a manager or you're a soft engineer, or whatever it is. Suddenly, solutions don't actually matter all that much. Like maybe the solution is like a Google form, or like something really random, you know, or maybe anyone can do the solution. The thing that you need to do is define the problem that you need to solve, and that is a skill, and the problem that we have as an industry is it's a skill. It's a skill that we do not train, and it's a skill that we usually don't tell people that they need. So I’m gonna say that you need to get better. I don't know who you are, but I know this about you, you need to get better at defining the problem space. So you must know that and then secondly, that's a skill to build, so you should build that.
Dan: [19:41] I will double down, that's definitely not a skill that was taught to me. I was-you know, I have a computer science background developer and that's what you get rewarded for your coding. You solve a problem. It feels awesome. Was there a certain time in your career that like this hit you or this switch went off? Like, how did you realize this, that you needed to change?
Katie: [20:03] Well, I realized that, you know, all of the mistakes that I was making, were coming up with totally viable solutions that were like, very great to problems that did not exist, you know? So like, oh, let's decouple our pricing product and like unbundle it, and we can have all these microservices, and here's how it'll work and allow for autonomous teams. It was like a lovely, organizational architectural solution to a problem that our customers did not give a, you know, flying hoot about. They did not want to buy things in an unbundled manner. They just wanted to like, pay for the thing, I didn't really think about is solving a problem that we actually have, because I was so carried away by like, well, this is such a great idea. This is such a great solution, and you know, I've made that mistake many times. And it was really when reflecting on what are the mistakes you've made? The strategic mistakes that you would look back on your career and say, those are failures? What are they? And someone asked me this question, or she was my current boss asked me this question, and I was like, tell them all the failures. I’m like, these are the things I'm going to learn from my mistakes, growth mindset, and he asked me, okay, well, what is the common theme in all of those failures, and solving the wrong problem? All of them, like 100% of them solving the wrong problem is the single straight up most expensive mistake you can make, and I thought he was gonna say, okay, well get better, Katie. And you know what he said? He said, “That is the same for me, and I have made much bigger and more expensive mistakes than you have.” And, you know, so like, it really is something that I think all of us as technical people, whether we become leaders, whether we become architects, solving the wrong problem, it's the most expensive mistake we make.
Dan: [21:47] Yeah, awesome. It’s something that I'm like thinking about all the time. You're a leader, making sure that everyone-you always have to provide the solution, making sure that everyone knows what is the problem that we're trying to solve. So my last question in this area that I want to ask you, as a leader, when you start framing the problem, or you ensure that you're talking about the right problem with your team, what impact does that then have on everybody else, like what happens then?
Katie: [22:16] Well, if people understand the problem very well, there's two things that happen. The first thing that happens is that they're going to be more motivated to come up with a solution, because they're going to feel that they have that sense of ownership, you know, and I mean, I hate being told what to do. I'm sure you all hate being told what to do. So if you're told here is the problem that needs solving, it's a much more motivating experience. And so people will engage their brains more and try harder and be more productive. So that's all great. The second interesting thing that will happen is they will come up with a different solution to what you might have come up with. If you’ve gone on and on about, here's the thing we need to do solution wise, they might come up with a different solution, and might be a significantly better solution. I'm almost embarrassed to say this, but I can think of many times when I've defined a problem, and I've known what the solution was, I was so sure, this is what we got to do, and luckily didn't say it and was wrong, and the solution presented was like clearly way better. It's a humbling experience when that happens to you as a leader, but that's one of the best experiences because you have in that moment, actually, like tapped into the power of your team. But this is not a case of just, you know, define a problem and then walk away and like, see what happens. You do you want to maintain alignment, right? You want loosely coupled implementation, but tightly coupled alignment. So you want to stay with them, where you're continuously revisiting, okay, what are the various areas the problem? How does your solution work? What are you running into, you know, one of the mistakes that people make is this kind of attitude of hire great people and get out of their way, you know, or like, just define the goals and then walk off and your team is gonna, like, do the thing, and like, I think we all know that that doesn't quite work, but we don't want to say that because it sounds like we're suggesting micromanagement. We're not. We're saying yes, define the problem. Tell them what they want. Hire the great people. But then stay aligned with them. Don't just leave, make sure that they are still going in the correct direction, because it's amazing how fast that understanding of the problem will decay.
Dan: [24:30] Yeah, I love it. I can see that happening. Even if you state the problem, kind of the first time, and everyone's on the same page. When you start getting in, now they're trying to do solutions and your brain can get confused like, wait, what are we trying to do again? Totally am with you, reiterate or realign and make sure that they're solving that can really help the team. I love that. Another topic that I wanted to hit on is around owning production. It's very easy to say that your organization uses modern software development practices or other buzzwords, but it's another thing to actually do it, such as everyone says, yeah, we want to own production and things like that. How do you think about owning production? How does your team do it?
Katie: [25:16] Yeah. So of course, you know, we currently at Ambassador, we make tools to make this an easier process. But it's really difficult because we have the situation here as an industry where I think we've figured out that the cloud is not going to go away. I think we've decided as an industry that Kubernetes has sort of won, and so we're going in this direction. And we figured out that throwing stuff over the wall to you know, the DevOps organization to just, you know, add more servers isn't really working very well for us. So we're in this moment of shifting left, and to steal Netflix's term, encouraging our software does to become full lifecycle developers, where you no longer are just working on your little widget, and then you're like, great, my widget is great, and there's some other team that needs to like, get that widget into production and look after it into production, you now need to think of what do I have as the dev environment for my widget? How does my widget interact with the other micro services? How do I deploy my widget? Now I need to know about Canary deploys, now I need to be on call for my widget in the middle of the night. This makes sense for the business, because if I write an infinite loop, in my widget, there's no amount of extra, you know, machines that a sysadmin can throw that is going to fix that, like they're gonna have to wake me up and be like, “Katie, there's something wrong with your actual widget here,” And I will need to go “Oh shoot!” and you know, go and fix it. So it makes sense, organizationally. However, for engineers that are being asked to do this and being asked to shift left, it's really difficult to do, because there's a huge learning curve, there's a lot to learn. It's fairly stressful, and there's also some pretty dangerous things that can happen. So I'll tell you a story. We did this early on, you know, it was 2016. And we just adopted Kubernetes. In hindsight, I think that was a little early, right, that we adopted Kubernetes, and, you know, we were shifting left and doing the great things, you know, we're on it. Devs, were owning these micro services that were breaking up the monolith, and somebody deployed something, and everything went down. And it's not supposed to have-everything, everything we could think of. we're running around in circles, and the CEO is like, “I thought the whole point of this new system was added wasn't everything that went down.” And this person had deleted the Kubernetes namespace, you know, that we were using for all of prod because they had access to it, because shift left, you know, do the thing, you got the access. And it was called a namespace. So they were like, cool, let me just rename this thing, because that's kind of a dumb name, and they just deleted the namespace. They didn't realize, like you have deleted like everything, you know, like, this is a disaster. So once we figured that out, like, oh, wow, you know, Systems team ended up figuring that out, but that was something that was quite dramatic that can happen when you're just, you know, forcing people to shift left, and there's so much to learn and there's so many valid assumptions that you can make that turn out to be really dangerous. Really dangerous, incorrect models. So that's difficult. Then the other interesting story that I've had, you know, challenges was shifting left, where I had a dev team that petitioned me to, I kid you not, return to the PHP Monolith, and it PHP 7, okay, which is a lot better, by the way. But they petitioned me, they were like, “We want Monolith”, and I was stunned because we were doing all the cool tech, everyone had their micro service, they could own it, they could do what they want. They're not waiting on some other team to like, update a config for three days. And it turned out that they had absolutely lost all sense of flow completely, because they're in a dev loop. They're sort of time from like making, you know, making a change, you know, committing and seeing what that change was. That loop, that little loop that you want to be second, it was ours, they needed to get sort of seven, eight sort of interlinked services all running at the same time. Docker for Mac wasn't being able to run all these containers, that computers didn't have enough memory. They had these like really kitted out laptops that was sitting on bags of frozen peas, one of them had like totally just flipped the table and got a Linux machine, you know, because Linux is better with containers. And eventually they were like, this is just so horrible., we just want to go back to a PHP Monolith because then we could just make a change and see it, and so you know, there's real challenges when you're shifting left that your team runs into where A: they can do very dangerous things and B: they can kind of have this organ rejection where they’re just saying “No, this doesn't work. We are just so unproductive, it's really difficult.” And yet it still makes sense as an industry for us to do this. So we have to figure out ways to make it easier. And I really care about this, I mean, this is why I ended up going to Master because they solved the problem where my previous dev team flipped the table and said they wanted to return to the PHP monolith, like, Ambassador had a solution to that. That sold me. I was like, okay, I deeply appreciate what you're doing.
Dan: [30:25] Yeah, I mean, it sounds… In theory, of course, shifting left is good, and owning production end to end is good, but for probably a lot of teams, especially if you didn't start out that way, getting there could be maybe like a traumatic experience. Do you have any baby steps that teams could take that are trying to get there that you've learned? Do it the right way?
Katie: [30:52] Yeah, absolutely. I would say that the first-first baby step is get your teams comfortable having good monitoring and being very familiar with what those sort of metrics for their service, their code, all the rest looks like, that's like the first baby step, just getting them comfortable, you know, what are your red metrics? You know, one of those four golden manual errors, latency, traffic saturation, you know, watching that, knowing that, that's very helpful. And then the second thing I would say is, you do want to start rolling out on call for these services, but do that very carefully. Absolutely horrible if you just throw people on call, and they don't know how to do this and they're not systems engineers. So I'd say make sure that you've got really nice run books before you do that. Something that worked well for us was, we call it the shadow on-call schedule, where two people are on call at the same time. It's the new product dev that is like shifting left, you know, like whether they want to, we’re kind of making them but like, it's this person, and somebody who's very seasoned, who can walk them through the incident, who can show them what to do, and that works quite nicely having that kind of shadow on-call system. And then you've got to give them good tooling that actually works, you know? I hate to plug Ambassador, but we have a lovely product called Telepresence, that makes it, you know, click a button, you can have a dev environment for whatever service it is that you want. Very, very easy, you can actually stick your laptop straight into your Kubernetes cluster, I mean, not physically, but like, you know, using this Telepresence intercept, and debug really, really easily, and it makes the experience a lot more similar to how things worked, when you maybe had, you know, my SQLite running on your machine, and you could kind of like, see what's happening, and you've got a local setup where everything is on your machine, it's a similar experience. So I mean, I genuinely do really like that tool, like, I like it to the degree that I joined the company to work on that, I do think it helps a lot.
Dan: [32:57] So it's kind of this multi-faceted thing, which is why it's probably difficult to do. You're saying, first of all, understand the metrics, just understand what you should be looking for. Another thing is, have the right tooling, like what Ambassador's providing. If you don't have the right tooling, your developers will be so frustrated, they're wanting to go back to a monolith, or they'll, you know, have the right-and probably like, third thing have the right architecture that actually make it happen. I don't know if you're gonna know the answer to this, and they can probably vary, but let's say that I'm running a team, I have twelve engineers, and we are throwing it over the wall still, before, you know going to production. Should I be thinking about my shift left journey for my team in like, months timeframe, six month timeframe, year timeframe?
Katie: [33:48] Well, I mean, you’ve got twelve people, I would say that that sort of shift left movement is probably going to be you know, months, because you've got, you're a small enough group that you can kind of like all do it together as a group, you probably all know, you know, who is your sysadmin person, you can kind of do it all together. But I would say try to make sure this is as much as possible, a bottom up move, and in a small company, you really want it to be a bottom up move, you know? In a large company, I don't think you can do bottom up, because how do you get two thousand people to organically, like decide something? I mean, I know you can, like we've had revolutions in human history, but yes, like foment a revolution, that's hard. So in a huge company, these efforts are typically more top down, but they work much better when developers are understanding the problem and are coming to the conclusion that this left shift is a good solution for them. That works a lot better. And then the other thing I would say is, as a PSA, I don't think everybody needs to be, you know, running Kubernetes with different micro services and cloud native and distributed everything. If you have a relatively straightforward, simple website, product, whatever it is, like, maybe consider how complicated do you need to make your architecture. Having this sort of distributed system set up, it solves a lot of problems, but it might be solving problems that you don't actually have. So don't just take this as, oh, all the cool kids are doing the thing and I should do it because that's the good thing to do. Caveat and I wish my former self could have heard this, just check that it's solving a problem that you've have actually got, you know, because any new technology, it's going to bring you more problems as well with it. So hopefully it's bringing you problems and solutions. It's like a lot of teams, and I've known a lot of VPs, and they've made the mistake of thinking, well, because Google does X, I should do X, because Google is-it's successful. It's, you know, they think it's state of the art. It's highly technical. It's what we want to aspire to. But it's very, very unlikely that you have Google scale problems to solve, and so bringing in a Google scale solution for a scaling problem you do not have, it's using a chainsaw to cut bread and like a really terrible way.
Dan: [36:06] Gotta have the awareness of where you're at and what you're doing. Yep.
Katie: [36:10] Yeah!
Dan: [36:11] Awesome. Thank you so much for all this wisdom that you're bestowing upon us in this area. I wanna slow the-
Katie: [36:19] Despite the mistakes I’ve made, I’m very happy to share them.
[Laughing]
Dan: [36:21] It’s all of your mistakes, exactly. Well, that's how everybody learns, right? Yeah. I want to kind of close us out here just with a fun question. What's the smallest investment you've made that's created the biggest ROI?
Katie: [36:40] Yeah, I love that, because I like that it's the smallest possible investment, and as a leader, it's this thing I do called My Week in Review, and I do it like, probably every two weeks. I write a little note. You know, it's kind of like, I don't know, maybe like five, seven hundred words, and it's got just some stuff I'm thinking about, some stuff I'm hearing, like maybe some comments on like, a cultural topic that's popular right now. You know, it's like, just some thoughts. I never thought anyone would read it. I still-I write it every two weeks, and I'm like, that seems pretty random. Here's Katie pontificating about random stuff. Every time I posted, people love it. They feel connected to me, they tell me it's great. It's like one of the things people appreciate most about me as a leader, and it's just really weird, because honestly, every time I write it and I think, what am I going on about? I don't know, who wants to read this, like hodgepodge of things on my mind? But it's really surprising to me, the impact that it has, when as a leader, you're kind of just sharing what you're thinking about, you know, people crave that. They crave that access, and it's not that I don't want to give people access, I just didn't think it was particularly interesting, you know, but it is, it really is. And so I do that little Weekend Review, and I now do it in video format, as well, where I send a little loom video every so often, and people can see my face and hear the message, and it seems to have a really, really high return for a very, very small time investment. You know, it's probably one of the most useful things that I've stumbled upon. I think it was Laura Hogan that told me to do this? If I think back so Solara is great, so much good advice, but like that's one of the things that it's absolutely outsized bang for buck.
Dan: [38:25] I want to steal that. Thank you so much for sharing that.
Katie: [38:29] Sure thing, yeah!
Dan: [38:31] Okay, great. Well, Katie, this has been such a cool podcast. Thank you so much for coming on. Really excited to have you joining us on INTERACT on April 7. I know that you are going through a large hiring phase. If people are interested in joining you and joining your team. What can they do?
Katie: [38:53] Yeah, if you are interested in building to have tools working on open source projects, we have multiple projects be donated to the CNCF. If you're interested in Kubernetes and writing go, please do apply. You can go to getambassador.io You can find me on Twitter, I am @GoKatieWilde, but you can email me, any of these things work. I’m hello@katiewilde.com. Please reach out, I have loads of open roles. I think right now I've got seventeen open racks just open right now. Because we're going fast, and we're doing some really, really interesting things that hopefully provide a lot of answers to many of the problems we chatted about here with Dan and that you might be interested in working on. And if you haven't worked on dev tools before, you know, neither had I before I joined, don't let that stop you. It's so fun. I will never go back to any other kind of product. So you know, hit me up.
Dan: [39:48] Ambassador, we talked about the great culture that Katie's providing, keeping developers in flow, and you also get to actually work on the dev tool that will help the community, so amazing opportunity there. We'll include all the information of how you could apply in the links for the pod. A quick reminder for our listeners, [Music fades in] if you haven't already rated and reviewed the show on your podcasting app of choice, particularly Apple Podcasts, please do so. Reviews are really crucial for us to get discovered. Make sure that you're there for INTERACT 2.0 on April 7 2022. Again, Katie's gonna be joining us, and Katie again thanks so much for coming on today.
Katie: [40:29] Thank you so much Dan, this has been really super fun. I'm looking forward to INTERACT and seeing all of you there