You can't automate a planning process that only some of your teams follow. This week on Dev Interrupted, Newsela CTO Dee Wilcox joins Dan Lines, LinearB's COO and co-founder in our AI Enablement interview series. She shows how Newsela brought dev cycle time down from four days to closer to two, and why she treats code review time as a measure of how long engineers wait for feedback. They close the conversation on proving AI ROI to the board, where Dan makes the case for cost per effective PR, a metric that only counts the PRs that ship without causing incidents or needing rework.
Show Notes
- Learn more about Newsela: https://newsela.com
- Accelerate Everyone: Dee's Substack on engineering leadership https://accelerateeveryone.substack.com
- Connect with Dee: LinkedIn
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Dan Lines: Hey, what's up everyone? And welcome to Dev Interrupted. I'm your host, Dan Lines, and today's guest is Dee Wilcox, CTO at Newsela.
[00:00:09] Dan Lines: And in this episode of LinearB's AI Enablement interview series, Dee is going to share how she's leading Newsela's engineering, data, and ML organization through an AI-first transformation, and what she has learned about getting an organization ready before the tools arrive. Dee, thanks for joining us today.
[00:00:32] Dee Wilcox: Thanks for having me
[00:00:34] Dan Lines: Yeah, awesome to have you on the show. And as I told you, uh, before we started recording, I looked up, um, all of your background, really impressive, career. So you're doing a lot of, great things, so awesome to have, like, your expertise on the show today. And where I wanna start is kinda around this, like, uh, ex- AI experimentation and maybe some [00:01:00] of the bottlenecks.
[00:01:00] Dan Lines: So I'll hit you with this kinda first question. So many teams right now are experimenting with AI in their SDLC.
[00:01:09] Dan Lines: Uh, what has that experimentation looked like so far for your team, and what has successfully transitioned to full production?
[00:01:18] Dee Wilcox: That's a great question. Um, we started with a more traditional, um, ML AI team. Newselas that team for years. Out with just data scientists, and then we had one of our folks say, "Hey, I really wanna go the ML route," and started working with LLMs before it was cool, when people still had to look it up and understand what that meant.
[00:01:39] Dan Lines: Yeah.
[00:01:40] Dee Wilcox: so we started building features with LLMs a few years ago and felt really good about them, started putting evals in place. So we had some competency around that, like how to use it in product development, but not really as tooling for engineers or for QE or what that meant for the data team. So I wanna say [00:02:00] last spring, we kinda moved forward very cautiously. We started out testing Copilot and testing OpenAI and testing different options. Then this fall, we really, really just ramped up our efforts. Andrew Parker joined Newsela, really, really drove that as well. He works across all of our organizations. And we just found this, like, time to get moving, time to take off some of the guardrails and just embrace it, and then take the learnings, which was great.
[00:02:30] Dee Wilcox: So we decided rather than moving forward very cautiously, just to lean in and take those foundational principles around iteration, moving rapidly, running tests, and, and asking our team, like, "What is working for you? What is not working for you?" Um, so we started out with, of course, in our QE process, of course, in our code review process and AI-assisted development, all of those things. But the thing that really helped is we already had baseline KPIs, so we had a baseline to measure, so we [00:03:00] could say, "How are you feeling? What are you noticing? And also, here are our metrics, and here's how this is changing our SDLC and our delivery." so that was really useful to look at both, like, how are we making progress against the roadmap? Where are we creating new bottlenecks? What are we gonna do about those? Um, so it's been just very rapid learning. I, I feel like every week we talk, and we're learning, and we're tweaking, and we're changing. We have some teams regularly using Harness, a couple of other teams saying, of our workflows are so different, one Harness won't work for us," but then taking some of the principles.
[00:03:36] Dan Lines: That's super cool. Lot of like, uh, good information there. One of the things that I wanted to ask you, 'cause you said you kind of had this team before, you know, LLMs and like AI was cool. So let's say that you probably had it for a few years.
[00:03:52] Dee Wilcox: Yeah
[00:03:53] Dan Lines: is that a centralized team now that kind of helps out all of the other [00:04:00] development teams?
[00:04:01] Dan Lines: Does that team stick together or did you kind of like, uh, disband that team? I'm just int- interesting in like the organizational dynamics and the setup there.
[00:04:09] Dee Wilcox: Yeah, they are a centralized team. They do embed on product engineering teams. So let's say, um, we are enhancing something in a recommendation service. One of our ML engineers will go and embed with that team and work on that feature
[00:04:24] Dan Lines: That's cool
[00:04:24] Dee Wilcox: make sure the evals are right, make sure we're thinking about performance, we're using the right models, all of that.
[00:04:30] Dee Wilcox: And then also saying, "No, this is a data science problem. We need to not-- It-- This is deterministic. We need to take a different approach." Um, they do embed, but we also leverage them for, like, that, um, expertise in how to think about a problem. So they don't necessarily set all the standards, but we do pull them in very, very frequently.
[00:04:52] Dan Lines: That's really cool. I, I love that model, and I, I... You know, I... That's not- I don't think it's something, like, specific to [00:05:00] AI. Like, this is probably dating myself, but back in the day when I was leading, you know, like VP of engineering and all of that, when we would have, like, a new concept, whether it was, "Hey, let's do more, like, DevOpsy stuff now," or, "Let's go to the cloud," or whatever it was, kind of having, like, a centralized team to help out or kickstart or, like, the mindset, I think goes a long way.
[00:05:21] Dan Lines: So that, that's really cool that you're doing that. The other thing that, that you brought up was KPIs.
[00:05:28] Dee Wilcox: Yeah
[00:05:29] Dan Lines: for me, just, like, a little bit on my background at LinearB, I work with a lot of our customers, and obviously with LinearB you get a lot of these KPIs. Um, and what I find when I'm working, uh, with our customers is, let's say that, the nice way to say it is we're- they're in different, uh, maturity levels.
[00:05:49] Dan Lines: Let's say some are, like, more advanced and some are not as advanced. And what I mean by that is some of them are kind of just in, like, the early adoption phase of, like, [00:06:00] experimentation. Other ones are saying, " Hey, you know, actually, you know, all of our engineers are using AI. We're even using it to build stories, but, you know, our quality's not so good," or, like, "Our review process is not so good."
[00:06:14] Dan Lines: Or maybe even it's just the deploy process, and these metrics can kind of show you, okay, where, where, where should we be starting?
[00:06:22] Dan Lines: I was in- interested to see, like, kind of what you me- you measure, and also was there something from the data that allowed you to say, because I think you might have said you started with quality, but, "Hey, let's start here because the data kind of tells us to do so"?
[00:06:38] Dee Wilcox: Yeah. Uh, several years ago, I met with my engineering managers. I really wanted to show our delivery health for the board, for the CEO to say, like, the investment we're making into R&D is paying off, right? We're all having the ROI conversation. Um, but Newsela is also very, uh, cost optimization-focused, so where we [00:07:00] invest.
[00:07:00] Dee Wilcox: We're ed tech, so where we invest really matters, and I wanted to show, you know, we're being wise with those resources, all of those things. So I met with our engineering team, and we aligned on how we measure health as a team. Like how does a team know that it's healthy? What's true when that's true? And where we landed, and LinearB is great for this, what is your dev cycle time? If your stories are too big, your dev cycle time goes up. If there's a lot of ambiguity in the requirements, your dev cycle
[00:07:28] Dan Lines: Yeah
[00:07:28] Dee Wilcox: up. You go through a lot more cycles, right? So we set, uh, I believe we started with a four day dev cycle time. We reduced it to three days. We're now closer to two days on our dev
[00:07:39] Dan Lines: cycle time. Nice. Wow.
[00:07:41] Dee Wilcox: Our code review time, to me, is a proxy for someone who's waiting for feedback, and the longer someone is waiting for feedback, the more they're context switching, the more, like, when they come back to that problem, they have to start over, and it's just a, a time sink. It really, really is. So we set that at two days. Last [00:08:00] year, we were closer to one day, and I was thrilled with that. Right now, we're closer to four days. So there are other metrics that I've started to pull in that didn't matter as much to me then. One was PR size. We kind of had some loose guardrails when it was just human review, like don't make giant PRs.
[00:08:16] Dee Wilcox: Do one PR per JIRA ticket ' cause
[00:08:18] Dan Lines: Yeah
[00:08:19] Dee Wilcox: integration. I've got one team shipping, like, 1,500-line PRs, and their code review is through the roof, and every engineer on the team is like, "Do you know how long it takes to review a 1,000-line PR?" So, um, so we've introduced a few new ones as we're measuring.
[00:08:35] Dee Wilcox: Like I ask the team every month, "So this code review time keeps going up, keeps going up," and so, uh, yeah, it gives us that baseline to bring that back and, and iterate on our process.
[00:08:48] Dan Lines: That's super cool, and I, I love that you kind of have all those different, uh, metrics, 'cause it's kind of like a lens into,
[00:08:55] Dee Wilcox: Yeah
[00:08:56] Dan Lines: you know, what's going well and what's not, not going so well. But the other thing [00:09:00] that I really like that you said, and this has been my experience as well, anytime that, uh, I- let's just call it a new technology or a new way of working is introduced,
[00:09:10] Dee Wilcox: Mm-hmm.
[00:09:10] Dan Lines: with AI it happens to be something, like, as big as, I don't know, the Industrial Revolution or something, like, massive, um, sometimes you can have, like, a cycle time that's been really good and then, oh, new technology, new way of working, it can start changing and st- start going to that four days, five days.
[00:09:30] Dan Lines: So
[00:09:31] Dee Wilcox: Mm-hmm.
[00:09:32] Dan Lines: just one s- one thing that I wanted to highlight there, I think, for the audience is once you start measuring, it doesn't mean that the metrics will always be great, especially when you're introducing something so new. And as long as you can keep an eye on it, you can adjust, and it seems like you're doing a great job at that.
[00:09:49] Dee Wilcox: It really, really helps, right? Because then you change your expectation and you've got that feedback cycle. The other one, um, that really matters to us is release defects, right? Because it's the [00:10:00] same thing. You introduce a new technology or a new feature or a brand new service and your defects can change.
[00:10:06] Dee Wilcox: And then also for us, our ability to respond faster though has been great.
[00:10:11] Dan Lines: Amazing. You know, something else that you, you brought up, I think you started talking a, a little bit about value to the organization or board meetings or, you know, in, in, in my experience, you know, working with CTOs like yourself, there kind of is that obligation, I would call it, back to the ... It's part of the job, right?
[00:10:32] Dan Lines: Back to the business of, "Hey, we're gonna try something new here. We're gonna spend some money." Even the experimentation probably costs, you know, you gotta pay for tokens and all of that. And then there's kind of this inherent expectation of AI is gonna make us better in some way, shape, or form. Usually, and again, this is with my experience, usually it's like, and especially for, like, the non-engineering folks, it's like we're gonna deliver way more features.
[00:10:59] Dan Lines: We're gonna [00:11:00] beat our competition. The customers are all gonna be happier. Everything's gonna be instantaneously great. What
[00:11:07] Dee Wilcox: the right.
[00:11:07] Dan Lines: Yeah, everything is up and to the right. Always only show graphs that are up and to the, the right if you're a CTO. But what ha- Yeah, I guess we kinda wanna pick your brain there. Like, what has been your experience in communicating ROI back to the business, back to the board, anything around what you're tracking or expectation setting there?
[00:11:27] Dee Wilcox: I have to say we have a phenomenal board, a phenomenal board, and they really pushed us early to experiment and invest. They believe heavily in R&D, so we're really fortunate for that. Um, I still feel accountability for my team and for the business. And so I will say we're still iterating on that and what really means something financially to our CFO and to the board, like how do we say that we are, really, really getting something meaningful out of this investment? We started tracking our [00:12:00] software assets in a different way. So we do capitalize software, lots and lots of people do. But we started keeping a roster of new software assets that were created or were getting more investment than they would have in years prior. Because typically, if you're like, "Oh, we need this 18-month platform investment project to build this thing that will serve our development team," it's really hard to get buy-in on that,
[00:12:24] Dan Lines: Yeah. 18 months sounds like an eternity probably too.
[00:12:28] Dee Wilcox: Yeah, nobody has 18 months.
[00:12:29] Dee Wilcox: That's the fastest way to get a project killed.
[00:12:31] Dan Lines: Yeah.
[00:12:32] Dee Wilcox: And so instead to say, " Hey, we've got a principal engineer," or, "We've got a few hack days in a row and somebody's gonna work on this," and you've got something to show in a few weeks or if it's like a string of hack days, maybe it's a few months of somebody saying like, "Last month I worked on this, and this month I picked it back up," that has been amazing.
[00:12:52] Dee Wilcox: So our register of software assets is higher. Our internal tooling is so much better, and those things support the [00:13:00] development process, right? So you get to your end goal a lot faster. Our ability to, um, I wanna say consume more of the roadmap, complete more projects, tackle big initiatives that used to be scary, where the team might say, "Ugh, that might take us six months, so we're just not gonna do it this year," teams are far more optimistic in really tackling those hard challenges
[00:13:22] Dan Lines: Yeah. I guess one thing that remains the same is iter- like iteration, iterative concepts, no 18-month projects, like especially when you're communicating Doesn't even have to be the board, but let's say that, like you're communicating to the business, your executive peers. Like, I, I think everyone still appreciates that.
[00:13:41] Dee Wilcox: Yeah
[00:13:41] Dan Lines: and, and then the other thing that, that I'd say, like some of the larger, I'll call, like the enterprise customers that we're working with, they're... It's very KPI driven across the business, not just, uh, software development. And so I would ju- just say like if you do need to show some North [00:14:00] Star metrics, everyone likes North Star metrics.
[00:14:02] Dee Wilcox: Mm-hmm.
[00:14:03] Dan Lines: There's two that I usually, uh, recommend for like an AI transformation, and I, I don't know if you're, uh, we didn't talk, so I don't know if you're doing this, but one is cost per effective PR. So that's kinda saying here, here's like the dollar amount for every PR, um, that gets merged and delivered, and the effective side of that is you gotta make sure that they don't cause incidents, those don't count, or if they're reworked, those don't count, so there's like a quality factor.
[00:14:31] Dan Lines: That one work- works really well. And then the o- the other one, and this is more like a classic one, we, we call it predictable delivery, but it's basically are we meeting our sprint or our project delivery on time? Not even like more volume, just,
[00:14:46] Dee Wilcox: Mm-hmm.
[00:14:47] Dan Lines: "hey has AI helped us be on time more?" Be on time. So I don't know if either of those resonate with you, but that's usually what I'm seeing with like the la- the larger enterprises I'm working with.
[00:14:57] Dee Wilcox: Yeah, we do measure on-time [00:15:00] roadmap completion for sure, but I heard you mention cost per effective PR. So I'm glad you brought it up. I was like, what does effective mean? But that quality cycle for
[00:15:08] Dan Lines: It's a quality cycle, yeah. 'Cause you don't, don't just wanna push out more. You have to have the effective... That word means a lot in that little, uh, KPI sentence, yeah.
[00:15:19] Dee Wilcox: And it's not just lines of code, which I really like. I-- The more that's come back lately, I'm like, "What are we doing? Lines of code is nothing." G- yeah. But we all need something, so that's great.
[00:15:30] Dan Lines: The other thing that, that I wanted to ask you, and it's kinda going back to that experimentation, uh, 'cause I think you've been experimenting with tools, I, I would say earlier than most, let's put it that way, kind of on like the adoption curve. Are you sensing any gaps between maybe like your expectations of what some of the tools would produce, like whether it's like a code review tool or like a coding assistant or...
[00:15:56] Dan Lines: And you, and you don't have to mention like, vendor names [00:16:00] or, or something like that. But is there like... 'Cause usually they're all like pr- pretty the same if you're working with it. But is there any part of the ADLC, SDLC that you were like, "Oh, I thought AI would really crush here, but really I think it needs to like catch up a little bit before like my expectations would go up again," or something like that?
[00:16:17] Dee Wilcox: it's so much better than it used to be, but I think there's still opportunities around consistency. Um, for better or for worse, you run the same security review agent on the same repo in different sessions, you're gonna get different output. If you run it locally versus in a different, you know, production environment, you're gonna get a different result.
[00:16:39] Dee Wilcox: That's part of the non-deterministic piece of it. But there are certain things in engineering where you have to be very sure, right? You have to have a high degree of confidence, and I think we're lacking on that in some of those pieces. also thought we'd be further along on the design side. There is so much that just you can tell when it's AI generated. We've [00:17:00] started, um, pulling in our own design system, which has been helpful, but it still isn't as, as mature as I would like to see. So I think there's a lot of opportunity there, and I do see some of the, the big players working in that space more and more.
[00:17:13] Dan Lines: Yeah, I, I mean, I, I've heard similar. It's kinda like the places where it needs a lot of cont- I mean, the, the thing I, I think that's not deterministic, which is very, like, anti-engineering brains, if you've been in the industry for a while, is when it needs, like, a ton of context from many different places to get a good design.
[00:17:33] Dan Lines: Whereas a human, like, that's been working in the place for a while kinda knows all the bits and pieces. At least for me, that's where I've seen, like, you can get really varying results, uh, still
[00:17:44] Dee Wilcox: Yeah, and it feels weird to say, but in that particular scenario, a human is less prone to forget. Where I feel like our agents were constantly reminding or saying, "Check all of these things. Make sure you've looked at all of these." I heard one of your other guests talking about the [00:18:00] register of resources that they make available to their agents.
[00:18:03] Dee Wilcox: We do something similar, but you have to tell the agent, "Go and validate against all of these sources." Um, and that, that is tough. It's kind of like that sometimes I'm like we're working with a junior dev, and then other times it's so advanced, it's phenomenal. So yeah, we're gonna continue to see those, like, different levers move up over time, just like we have in the last year especially.
[00:18:27] Dan Lines: It's like, "Don't forget all this information that I sent you previously. Please double-check to do it again." Yeah.
[00:18:33] Dee Wilcox: I know. It's like, "Please reference my last email from back in the day."
[00:18:37] Dan Lines: Um, uh, awesome stuff. Uh, moving on to, like, a, a topic that I think is, again, it depends on what industry that you're in, but I think mo- most anyone, you know, wants to make sure, let's say, that the results are good, the output's good, what's going to production is of high quality. Let's put it that way. [00:19:00] Um, in some businesses, I don't know, let's say if you're in finance, healthcare, it's not, like, necessarily a life or death situation, but it's like, hey, you gotta make sure that this is right.
[00:19:11] Dan Lines: Let's put it that way. Do you have anything that you've done with, like, governing, you know, what goes out or any, like, policies or checks that, checks and balances that you've put, put in place?
[00:19:25] Dee Wilcox: We do. Um, so our product engineering teams are mostly pretty small. We have, uh, coding style guides. We have code review standards for each team. Have local code review. The developer is expected to test in their local environment, and then we have code review from a peer, AI-based code review. And then once something passes, it has to go to staging, and we run automation tests against it there. Sometimes we'll also run them in, like, a dev environment as well. So we have certain gate checks there, and then we [00:20:00] also have, uh, on the data side, data governance in place. So there's, like, multiple phases there.
[00:20:07] Dee Wilcox: We run automation in production as well, but we're really trying to check through multiple gates before we get to prod
[00:20:14] Dan Lines: that makes sense. It's kinda what I'm hearing from most. We, we just did a panel, I don't know, panel interview or, like, a panel show the other day, and a lot of the audience through the chat and the speakers were talking about, like, these policies, exactly what you were saying. What I've seen with our customers, and usually it's more so around when the PR is, like, created or ready to review.
[00:20:39] Dan Lines: Like, first of all, make sure you have, like, automated assignment. Some of these PRs are now created by agents end to end, and what we were finding in the data is, like, no one picks them up for review, and they, like, never even make it, so that's like the most basic.
[00:20:52] Dan Lines: the most basic But other times, like, okay, if you're changing code in certain repos, like, [00:21:00] we still have to have two reviewers.
[00:21:02] Dee Wilcox: Mm-hmm.
[00:21:02] Dan Lines: in other situations, like, hey, if it's just, like, okay, we added a ton of tests through AI, okay, maybe you don't need a reviewer. But I do think it's important still to have some types of, like, checks and balances in place, and then maybe change them over time, and to have them be automated. Because if they're not automated, what I've seen is the amount of small...
[00:21:24] Dan Lines: You talked about small PRs. The amount of small PRs that are coming through systems now are increasing, which is great. We want a lot of small PRs. But, like, we need some way to gover- govern them, govern these changes. So I think it's just, like, for, for the audience, really important to have a plan in place so these PRs either, one, don't just go out to prod and you have a bunch of incidents, or two, they just sit there forever and it's just like they don't do anything
[00:21:50] Dee Wilcox: Right. I think the other piece of that is accountability. So our KPIs are across the org, but I also have team breakdown. So whenever I'm seeing [00:22:00] something funky, like your code review cycle time is up or your released defects are really up, then I'm going back to those engineering managers and tech leads to say, "Hey, have you looked at this?
[00:22:10] Dee Wilcox: There's something going on here," right? Because SRE has the pager and they're like, "Hey, hey tech lead, whatcha doing?" You know? And so we try to keep that cycle really active so that we're, I don't know. Yeah, it's the KPIs and driving that accountability. We have the gates, but if you're bypassing the gates, if you've got someone doing a quick looks good to me, then, um, yeah, it's not as effective as it could be, and it creates a lot of time for other people to go track it down, figure out what that bot did, all of those things.
[00:22:42] Dan Lines: Yeah. I lo- I love it. Gr- great advice there. So if we're looking a little more as we get towards, like, the end of this, uh, session together here, this pod together, a little more towards the future, like, what's your plans over the next three months in terms of, like, [00:23:00] evolving your ADLC? Where, where are your- where's your focus at?
[00:23:03] Dee Wilcox: Yeah. So the bottlenecks we're seeing right now are in the discovery, planning, and design phase, right? Building that alignment, building really, really strong use cases. I feel like the fundamentals are the fundamentals, and wherever you are strong, that's really playing well right now. And if you were weak in an area, that's biting you, and that's true for us, where we got a little bit loose with our, um, product and design and requirements process, where, like, some people did a PRD, some people didn't.
[00:23:35] Dee Wilcox: Some people did user stories, some people didn't. And it's very hard to automate that when you're really inconsistent. trying to be more consistent on the early planning side of things. our design team is phenomenal, so kind of bringing them along with us in that process. And then upstream of that, we have a really great QE and automation team, but working with SREs so our [00:24:00] services can be more autonomous, our teams can be more autonomous, and we have those guardrails and gates so we can release with confidence without an SRE team member saying, "Okay, I got paged again," you know?
[00:24:12] Dan Lines: Yeah
[00:24:13] Dee Wilcox: So it's both sides. I feel like development, code review, QE, looking pretty good.
[00:24:18] Dan Lines: Yeah.
[00:24:19] Dee Wilcox: always more to
[00:24:20] Dan Lines: It's the ends.
[00:24:21] Dee Wilcox: Yeah, it's the other sides of that.
[00:24:23] Dan Lines: That's super cool. We'll have to bring you on again to see, you know, how did that go. But yeah, you're, you're right. I think a, a lot of... It's like the middle is where a ton of innovation's happening, and it's actually surprisingly awesome, but kinda the beginning and the end is where it's a little more experim- experimentation also.
[00:24:42] Dan Lines: We'll have to see how that works out for you. Uh, the last question that I have for you today, and this is something that I'm doing with all of the guests, uh, in this series,
[00:24:52] Dee Wilcox: Okay
[00:24:53] Dan Lines: what's one piece of advice you'd give another engineering leader who's just [00:25:00] starting their AI transformation, so maybe not as far, as long as you are, or, uh, something you wish someone had told you before you got, uh, started in terms of advice?
[00:25:12] Dee Wilcox: I wish someone had told me not to let all of the news be so distracting. It's so like you just feel like you have to chase this new release and this new model and this new thing and new way of working. The fundamentals really are the fundamentals, right? We have to get more comfortable with change.
[00:25:30] Dee Wilcox: That might be my advice. I'm a cautious person. The quality of our releases really, really matters to me, but knowing that we can respond faster just creates more safety, right? So I think getting really comfortable with change, leaning in, and then making sure your fundamentals are strong. I think that's the main thing
[00:25:50] Dan Lines: Don't get caught up in the hype cycle.
[00:25:52] Dee Wilcox: Yeah.
[00:25:53] Dee Wilcox: Don't get caught in the hype. Yeah
[00:25:55] Dan Lines: Awesome. Well, well Dee, this has been incredible. Uh, [00:26:00] all of the knowledge that you've shared here and your advice, we all really appreciate it, so thanks so much for joining me today on the show.
[00:26:08] Dee Wilcox: Yeah. Thanks so much, Dan. It's great to talk with you.
