The wrong internal tools can hold your team back. So how do you find the right ones, and how the heck do you get engineers to adopt them once you do?
On this week’s episode of Dev Interrupted, co-host Conor Bronsdon welcomes Debo Ray, co-founder & CEO of DevZero, to discuss the challenges developers face due to inadequate tools. With a keen sense of developers' needs, Debo explains why many companies fail in this domain, squandering precious dev time.
Debo also offers a peek into the future of cloud development, the work he's doing at DevZero, and the nuances of marketing tools to developers.
Episode Highlights:
- (2:25) Why are engineers held back by internal tooling?
- (5:00) Where are companies wasting dev time?
- (9:05) Developer experience has to come from leadership
- (16:00) Debo's work at DevZero
- (17:00) Moving development to the cloud
- (21:15) Combining local tools with remote compute
- (23:00) Getting devs interested in a new tool
Episode Transcript:
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
Conor Bronsdon: Hey, everyone. Welcome back to Dev Interrupted. This is your co host, Conor Bronsdon. And today I'm delighted to be joined by Debo Ray, co founder and CEO at DevZero. Debo, welcome to the show.
Debo Ray: Thanks so much for having me, Conor.
Conor Bronsdon: My pleasure. Before you founded DevZero, you had a really fascinating career as a staff engineer at Uber, where you worked in infrastructure and product security.
You've also founded two other companies before starting DevZero. So you're a perfect guest for us, particularly because And I want to give you a bit of a shout out here. You graduated from my alma mater, the University of Washington. Go Dawgs!
Debo Ray: Yes, best school ever.
Conor Bronsdon: Amen. It's clear that you are someone who's very passionate in everything you do.
Whether it's, your work at Uber, or dev tooling, something that you've worked in deeply, both at Uber on the infrastructure side. You're on a mission to unlock developer potential is how we've seen it defined for you.
And that makes you a kind of kindred spirit for us because at LinearB, we think a lot about these concepts And we both agree that when you get the tooling right, You make the developers lives better and you make them happier, which the great thing is that helps engineering leaders, engineering teams produce higher quality work, produce things faster.
So I want to kick off this interview with something you said before the show started, which is engineers join companies to do good work and help the business, but are held back by internal tooling.
Debo Ray: Conor, like us developers, we join companies because we care about the mission.
And also about the learning opportunities, that we'll gain in service of that mission. And fundamentally, as developers, we also have a pretty high bar for quality. There is this inner gauge that we have, which tells us whether a product or a feature or a service that we have shipped, if it's indeed good or not.
At the same time, we are also very impatient as people. And usually the path for me to launch and get some feedback on this new feature is often a pretty arduous one. And I have all of this context in my head and I just want to, I've written some code and I want to see this thing working end to end.
And as developers, normally in companies, we have to go through these like almost toilsome dev cycles where I'm constantly like changing my mental context. And this is quite challenging and often unproductive in my opinion. And this is generally what we call the inner loop of us. of software development.
I want to constantly be in this state. That is where magic and creativity, everything's happening. But we end up spending most of our time in the other drudgerous aspects of software development, right? Be it debugging issues related to the environments that we are working in and all of the inconsistencies there, or being able to access some data so I can actually test this feature end to end, or actually trying to figure out, What the best way to call their sister teams, products or services and or the worst one is when I actually have to get into a meeting and drive consensus with fellow teams.
Conor Bronsdon: Yeah, it's tough because those are obviously crucial things for a development team, but. Most devs have been trained to be this kind of like focused individual problem solver and as we get into larger companies where you're relying on a lot of other people to get big products done, which is, how you achieve incredible things, great teams, people don't necessarily have the tools, training or time to make some of these team dynamics work the way they want to.
And companies often don't do a good job of actually ensuring that devs have the processes, tools, and help they need to be in the best position to succeed on these. Where do you see companies wasting the most dev time?
Debo Ray: In my opinion, developer productivity, I think it's a fundamental part of any tech company's core product line.
When you don't invest in like internal, tooling, a couple of issues start to, prop up over time. There's usually this gradual increase in the friction of shipping new product updates, right? And then we also see successive product iteration cycles. They take longer and more delays start to show up.
Early on, when a company is just starting off, it's just a group of people that have gathered together around a certain project. And often testing is now happening in production. It's risky. But the developer experience can be pretty good early on, but right after that, as you gain more users, you start to care more about reliability and immediately there's this drift starts to happen and you end up making a somewhat divergent copy of production for your CI for your dev.
And normally as engineers will always try to keep the app layer consistent because that's the layer that our end users usually get to interact with the most. But the underlying infrastructure. Fundamentally is forced to diverge now, and this is a gap once introduced, it gets consistently harder and harder to reduce over time as the company just continues on its life cycle.
Conor Bronsdon: So it sounds like what you think that if the foundation that is laid around dev tooling and internal processes for engineers. is problematic. It's really hard to restart that later down the road once you realize it's wrong.
Debo Ray: And usually in most companies, you'll end up seeing that it takes a significant investment later on when the problem reaches a point where essentially a shipping product comes close to a standstill.
Conor Bronsdon: Are there particular companies that are doing a great job on this that you know, folks listening can learn from?
Debo Ray: Yeah. So in my opinion, and obviously it's a bit of a partial one at Uber. I thought we did a pretty fantastic job with this. We had a consistent. It was almost a culture of introspection into our dev tooling and processes.
We made a lot of mistakes along the way, but we were continually questioning whether we were shipping fast or not. And I think that, and we obviously learned from our peers over in Google or Facebook now called Meta at how they were also approaching this problem. A few of the lessons from our peers and industry were very helpful.
Conor Bronsdon: So it sounds like you think there are definite lessons that companies can learn from some of these massive Mega cap companies that have invested in platform infrastructure teams in internal tooling. What are some of the specific lessons that someone, say, working in a startup or Maybe a scale up can work.
Debo Ray: Yeah, so as I was saying, the constant focus on whether we were shipping fast enough, I thought was a very important one. And often this will fall down to one of the super easy to measure metrics, right? Everyone normally starts with the number of commits that are being put out, the number of lines of code, etc.
Which kind of start to treat software as tangible goods. Although as a starting point, I don't think that's a terrible place to be in, right? Yes, that shows the beginning of introspection and one important thing to note over here is like software falls into the class of intangible goods and its value lies in its novelty.
So by shipping more features or capabilities of obviously thoughtfully applied, you actually stand a chance of providing more value than you currently provide to them.
Conor Bronsdon: Because if you're doing it right, you're iterating positively.
Debo Ray: Exactly. So caring a little bit about the cycle times. And am I actually shipping enough changes into production that are reasonably sized?
Conor Bronsdon: What's the reverse for these large companies? What can they learn from startups about how they're approaching tooling and enabling devs to have positive developer experiences?
Debo Ray: A company will always start as a small group of people working on a project. And at this point in time, you fundamentally care about the day to day experiences of all of your colleagues.
And you're consistently also building these little tools to fill the gaps that you're spotting on a day to day basis. And communication channels are, as a result of being a small company, they're Way simpler. As the company starts to scale, this strategy does not scale that Yes. Having developer productivity owned in some way is much better than not having it be owned at all.
But as the company grows, it will often become a community funded initiative, right? This is an approach will end up falling prey to the fallacy of the company.
Conor Bronsdon: Absolutely. There needs to be executive sponsorship or clear importance put to productivity and the experience developers are having. Otherwise you risk these major challenges coming to light down the line and not enough focusing put on it because in the end it's so easy for a business to say.
Oh, yeah, we want to build this tool for internal productivity to, help us on the long term. But we have a short term goal. We need to focus on that. And let's shift that team over. They're working on a side product. We need to make sure we ship this for our customers. And that might be fine once, even twice, over time, you're going to devalue the tools and the platform that your engineering team is using.
And you're gonna, to your point, bring your product organization to the standstill.
Debo Ray: Actually, to that point, Conor, I was reading this thing recently. There was a comment from Dylan Field when Figma was very early on in its life cycle, and apparently they had an outage. The team was, the engineering team was working on a variety of features at that point in time, which they deemed to be more important for their end users.
And during this outage, Dylan was like, why aren't we fixing this right now? And the engineering manager over there was like, we have more important things to work on. Dylan's we have users. The engineering manager is not really. Dylan's, we have 10 users and some of them might be using us for their real work.
So we have to fix this right now. So a lot of this is also cultural and like engineering reliability and the developer experience it has to come from the top.
Conor Bronsdon: That's a very illustrative example. And I think it speaks to another ethos with engineering teams, which, I've been guilty of myself, which is.
We usually think we're right. We're solving these problems. We're doing the research. We're on the front lines of this creative process. That is delivering the end value. And we sometimes maybe become disconnected from the actual users, the customers. And I think that's a really important piece of business alignment that We're doing.
Talk about, but sometimes don't put enough emphasis on and, I think a corollary of that to focus in on tooling is that as engineering teams, if we don't want to do something, if we're like, I don't really this tooling, it doesn't make sense for me. My workflow. Initially, it's hard to get us to actually adopt that tooling.
Even if it's something that's crucial in the long term. So I'm curious from your perspective, how do you think engineering leaders Should get their teams to adopt tooling that they know is going to be important in the long run. Even if in the short term, they maybe don't see the benefit.
Debo Ray: Yeah, I think taking to your previous point an engineer is never wrong, even when they are.
And so you, if you ever ask an engineer to use a tool, they won't, they don't like, they won't do it. If you ask them to turn left, they'll turn right. So at that point in time, it's very important to first explain what value propositions the tool will actually bring to the engineering team, right? And then you you all need to be very focused on making these tools very simple and easy to use.
Let the engineers derive value from it. And just fundamentally right, especially now being a vendor that provides a developer tool. What do you see as developers? Always hate talking to salespeople, right? I'm as an engineer myself. I can completely relate. So let them think Yeah, as an engineer, just let me tinker.
Let me see value. Make sure that Engineers are able to do whatever they want to with your product And once they see the value and actually derive the workflows They will end up being your biggest champions.
Conor Bronsdon: So let's say I'm a salesperson and I'm coming to you, and I'm saying, look, Debo, you're a former staff engineer, you're CEO now, you're leading this team, and I really think you need this dev tool to help your team.
How should I pitch you?
Debo Ray: I will very rarely try out a tool just because a salesperson asked me to. I might find myself going to the website and actually trying to sign up myself, connecting my source whatever else to see how the tool works. A bunch of examples, guided walkthroughs are usually helpful.
I'll take a look at the documentation. And then, If I have a burning need at that point in time, great. If I don't, it's something that will stay in the back of my mind. And probably in a few months, weeks, whatever, when that problem arises in my life, I will remember that tool, if it left a good, lasting impression.
Conor Bronsdon: I think that to your point, you need to meet engineers where they are and say, Hey, I just want to share this information about something that can help solve a problem.
I think you might have. And if you have that problem, great. Then maybe you'll go pursue it and you'll investigate it. You can dive into hopefully great documentation, a great walk through, free sign up flow. But companies and people who choose to wall that off are doing themselves a disservice and making it much less likely that Okay.
Engineering teams will pick up their tool because we want to tinker.
Debo Ray: Yeah. And that's the other thing, right? And I find, because I use a bunch of different tools on a day-to-day basis, one thing I've noticed is just there is this creative bit of my mind that I like to exercise whenever I'm doing something.
Cause I don't know, as engineers, I think we always like to discover things and then solve problems. And if something seems a little too magical, You end up questioning it a lot, as engineers, we always make sure, it first it needs to be a value add to my life and then I also need to understand how exactly this thing works underneath in order and my sentiment is this gives me a sense of reliability that okay, when I use this on a day to day basis, I have a general idea about how things are working underneath that tool itself.
Okay, now I can use it.
Conor Bronsdon: So let's switch gears a bit. Okay. Can you catch the audience up on the work that you're doing at DevZero?
Debo Ray: Yeah. So at DevZero, we build cloud hosted development environments, primarily aimed at software engineers to do their day to day writing, testing, building code. Our goal is to ultimately provide a production symmetric environment, for all engineers in a company, their very own environment, to do their everyday work in.
Our belief is that this is the highest context environment possible, and this is the closest you'll be, able to get to literally coding in production. And in environments like these, turns out you don't have to wait as long for your code to build or compile or for your tests to run. And turns out you can actually run your end to end tests right within your Dev environment itself. And you can also reuse these for CI if you want.
Conor Bronsdon: Why is it that most development hasn't moved into the cloud yet? Do you think?
Debo Ray: Yeah, so most of the tools that we use on a day to day basis be it our communication tools or like the various collaboration tools that we have, all of those have moved to the cloud.
But the only notable exception to this trend has been the tools that. Developers literally use to write and run their source code, our IDs, etc. My sentiment is that this has happened because of two large classes of issues. One is the internet dependency. There is the perception of a lot of latency, and what happens if I have to code when I don't have access to the internet.
So that's one class of issues. And the other is just the overall complexity. Because at the end of the day, I am moving my localhost to the cloud now. And how am I going to get all of the dev tools that I'm already used to using to work on this non localhost network that I'm already used to? And most of us engineers, we don't have a very deep understanding of how networking, et cetera, works underneath, right?
The various, Potential value propositions that might be available as part of this, like just the first hurdle of being able to jump through that barrier might seem a little off putting.
Conor Bronsdon: Yeah, that initial friction can be a big turnoff for folks who would otherwise make the jump and If I'm an engineer who has a local ID, like I'll say I do, I'm my instinct is to keep using what I have here and expand my tooling and other areas.
Maybe because it's a comfort thing. Maybe because there's just these like. Blockers. So given these challenges, why should engineering leadership choose to push into the cloud for actual ID usage, coding, etc.
Debo Ray: The thing here Conor is like the end of the day. Developer productivity is something that is usually owned by a CTO or a VP of engineering or an infrastructure or platform engineering leader, right?
And they care about things like the performance of environments. Are developers slowed down by their local environments or from using various shared tenancy dev clusters? How easy is is it for us to on board, off board engineers to switch environments? How easy is it for an engineer to switch teams?
They also care about things like security and compliance. And so my fundamental belief is that In order for an organization to truly adopt cloud based development practices, it has to be a top down motion in some way, driven by a couple of these priorities.
Conor Bronsdon: So it sounds like you envision a future where, these tools take the Microsoft Word Google Drive route of not necessarily having software locally on the computer, but Trying to maintain a constant connection so that you can reduce local build times and offload those to the cloud and take these other approaches
Debo Ray: Yeah see at the same time Conor your point, right?
You have your IDE your local computer. Your local laptop feels like home to you. And when you open it, you just open up your IDE snd stuff just works changing these fundamental developer workflows. I don't believe engineering leadership would ever want to do that because it doesn't make sense. You have to retrain your entire workforce and that comes at a pretty significant cost.
So what will work in this space? My belief is that a concept of Combining local tools with remote compute. Net engineers use the IPAs, their, all of their aliases, dot files, et cetera, exactly how they use internet. The, to our previous, conversation around points of friction, remove all of, all points of friction to the engineer adopting cloud development environments.
Have everything be already set up and ready to use for them. And I think pairing this, idea of local tools with the flexibility and the power of remote compute is going to be the trick.
Conor Bronsdon: And how are you trying to solve that problem at DevZero?
Debo Ray: Yeah, so for us, we look at environments in a couple of different ways, right?
There is obviously the whole idea of a hermetic environment primarily paired around one repository. We do that pretty well. But as an engineer, especially nowadays, everyone's using microservices in some way or form, whether it comes out of a monorepo or a bunch of different smaller repos. And at this in this world, we take a dependency on, having upstream or downstream services around.
So at DevZero, we try to make that world possible as well. So we have this, templating language at DevZero that enables A DevOps or a platform engineering, or an infrastructure engineer to go and configure what dev environments might look like for a certain sub team or large team, et cetera, with all of your dependencies, et cetera, around you.
And then we just, we have a bunch of, Essentially, pipelining work in the background that makes sure dev environments are always ready to go. So when an engineer shows up, you just click a button, you connect your local IDE to one of these environments, and suddenly it'll everything looks and feels the same.
It also feels like my IDE has just moved into a production ish environment, where I can just start to make calls to arbitrary downstream services, which in the past was only possible in my dev cluster or in a staging environment of some sort.
Conor Bronsdon: And we've talked a bit about how we don't want sales folks to pitch us to get on these tools, but given that, you're a former staff engineer yourself, you're now a business leader, you obviously need to pitch other engineering teams on why they should check out your service.
What's the approach that DevZero is taking about how to share information and interest developers and getting on board?
Debo Ray: Yeah, so as we were talking about, engineers do not like being sold or marketed, right? But I also believe engineers like well marketed products. That's this overall aspect. I have a problem.
I have discovered a potential solution to my problem. Here is how I have implemented a solution to this problem, and as like vendors we just have to be at the right place at the right time. Cold calling or cold emailing isn't something that will work to get essentially an engineer onto a product.
They just need to be able to find you in the right places, be it a hacker news or when you're doing a Google search. You have to show up.
Conor Bronsdon: If you have any tips on how to make sure we're on Hacker News all the time, we'd love to get the podcast on there more.
Debo Ray: I need to find someone that knows that trick.
Conor Bronsdon: Yeah, I think we all do. Fantastic. Debo, thank you so much for coming on the podcast. It's been a great conversation. Where can folks follow the work you're doing at DevZero or check out more about your personal story?
Debo Ray: Yeah, so I have started writing on Substack. It's called Debo's Newsletter. And in parallel, we are obviously launching a bunch of new features within DevZero.
Primarily to solve a lot of these pretty complex replication challenges. And these are continually going out of our Usually in two to three week cycles. So feel free to sign up on our website at devzero.Io. Take DevZero for a spin. Follow us on LinkedIn, on Twitter.
Conor Bronsdon: Fantastic. Debo, you should absolutely come write a guest article on why cloud tooling is important for dev teams on our sub stack.
So for those of you listening who don't know, we've got about 14,000 folks as of this recording. Who are now subscribed to our Dev Interrupted sub stack. We put out a weekly, a deep dive article on Thursdays and on every Tuesday we'll release our podcast episode plus some of the top content we're seeing in engineering leadership and Debo we'd love to both feature you in the podcast whenever this comes out and let's maybe see if we can work together on an article about cloud development. I think it'd be really interesting to do those in concert.
Debo Ray: Absolutely. I'd love that.
Conor Bronsdon: Great. Thank you so much for coming on the podcast and, thanks everyone for listening. We'll see you next week.
Want to cut code-review time by up to 40%? Add estimated review time to pull requests automatically!
gitStream is the free dev tool from LinearB that eliminates the No. 1 bottleneck in your team’s workflow: pull requests and code reviews. After reviewing the work of 2,000 dev teams, LinearB’s engineers and data scientists found that pickup times and code review were lasting 4 to 5 days longer than they should be.
The good news is that they found these delays could be eliminated largely by adding estimated review time to pull requests!
Learn more about how gitStream is making coding better HERE.
