This week, Dev Interrupted takes a field trip all the way down to the GPU. Leading the way is Together AI’s VP of Kernels Dan Fu, a co-author of FlashAttention who also runs a research lab as an assistant professor at UC San Diego. He joins Andrew to talk about how ideas now move from the lab into production models within months, and what it means to build a cloud for AI. Dan argues that AI is an accelerant for individual engineers, and that the more domain expertise they bring, the faster it lets them go. They close the conversation on how DRIs and managers keep track of ownership and AI spend when your star players get even faster
Show Notes
- Together AI: together.ai, the AI cloud where Dan’s team runs the inference stack
- SandyResearch Lab: sandyresearch.github.io, Dan’s research lab at UC San Diego
- FlashAttention: arXiv, the paper Dan co-authored
- GPT-6 Astra and looped models: The Information, the report Dan cites on looped models reaching production
- Perfetto: perfetto.dev, the trace viewer Dan’s team built its own versions of
- Together AI and Moonshot AI: Partnership announcement
- Together AI and MiniMax: MiniMax on Together AI
- Together AI and Cursor: Customer story
- How to turn your 1000x engineer into a 10x everyone | LinkedIn’s Karthik Ramgopal: Listen
- Connect with Dan: LinkedIn | X
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: Welcome to Dev Interrupted, brought to you by LinearB.
[00:00:03] Andrew Zigler: I'm your host, Andrew Zigler, and today we're going on a field trip all the way to the GPU and beyond. Joining me today is Dan Fu, VP of Kernels at Together AI, and an assistant professor at UC San Diego. He's the co-author of Flash Attention, and his team at Together writes the kernels that decide how fast everything above them actually goes.
[00:00:24] Andrew Zigler: And seriously, if you've run a model in the last few years, you've almost certainly touched Dan's work in some way. And what I find the most interesting about Dan is he wears a lot of hats. He runs a research lab, he's a professor, leads an engineering team. So today we're gonna talk about what happens when research and engineering stop being separate departments, start becoming a harmonious loop, and what it means to build a cloud that's designed for AI from the start.
[00:00:49] Andrew Zigler: So Dan, welcome to the show.
[00:00:52] Dan Fu: Thanks so much for having me. It's, uh, it's really fun to be here and, and looking forward to a great conversation
[00:00:56] Andrew Zigler: Amazing. Well, we're super excited to host you. And, you know, I really gotta [00:01:00] start with the part of your very unique kind of assemblance of roles and responsibilities that you do. I think that really gives you a, a unique perspective. I wanna frame from the start. Could, could you tell me a little bit about, like, your background in, in being a, like, a professor and, and being in a research lab, um, as well as your capacity as VP?
[00:01:21] Dan Fu: Yeah. Uh, yeah. So like you said, I, I wear, I wear a few hats. Um, at Together, our team runs, uh, the, the whole inference stack. You can imagine as soon as a request enters data center, um, all the way down to the GPU, um, and, and everything in between. So we, we, we think a lot about how to get more tokens out of the same hardware, um, so that when you're, when you're trying to use it for coding or chat or, um, you know, ChatGPT, tell me what plant this is or, or whatever, um, uh, uh, all that, all that stuff, we, we wanna make it so that, uh, all those capabilities are, are available to, to a lot of folks. Um, separately, I run a lab at UCSD, uh, [00:02:00] and there I, I have a few great students who are, um, doing all sorts of research on, uh, you can think of it as next generation architectures and, and things like that. Um, so yeah. It's, it's a lot of fun, uh, a, a fun intersection of, of the research and, and the engineering cultures.
[00:02:16] Andrew Zigler: And did you, did one come first or did one come lead to the other or did they emerge at the same time?
[00:02:24] Dan Fu: Hmm, let's see. So I've been at Together for about four years now. Uh, started part-time during my PhD. Near the end of my PhD, went on the faculty job market, um, and, uh, you know, got, got the position at, at UCSD and, and started running a small lab there. So I guess they, they kind of rose in parallel with each other, um, which is, which has been fun.
[00:02:48] Andrew Zigler: No, I've definitely... I, I've been meeting folks like yourself that are, like, dually armed going down this, like, academia path and this, like, applied engineering applied, like, in the industry path.
[00:02:59] Dan Fu: Yeah. Yeah
[00:02:59] Andrew Zigler: [00:03:00] And I think actually there's a lot of opportunities now to bring that kind of work, which traditionally has been so bridged, uh, closer together in terms of just accelerating each other, building off of each other's, uh, progress.
[00:03:12] Andrew Zigler: And, and so, um, like, what does it look like, I guess, then bringing that kind of mindset into your role at Together? Um, how does your, uh, i- ideas around those things, like, then impact the actual engineering that y'all end up delivering?
[00:03:29] Dan Fu: Yeah, yeah. The way I think about it is the, the ideas, the research ideas that are, um, a little bit more out there that, uh, could work one day, that, that could have, that, that could
[00:03:40] Andrew Zigler: The moonshots.
[00:03:42] Dan Fu: Mm-hmm.
[00:03:42] Andrew Zigler: Yeah
[00:03:43] Dan Fu: Um, but that won't necessarily, um, you know, translate to something immediately. Uh, those are, those are the ideas that, that I tend to, um, explore more in the lab.
[00:03:54] Dan Fu: So, um, uh, I mean, I-- So the, the funny thing is it, it's a, it's actually a [00:04:00] lot quicker of a loop than it was even when I was doing my PhD. So, um, one set of ideas that one of my students has been looking into at UCSD is this idea called looped models, where you take... So traditionally, you think of a model as you have some input, goes through all these layers of a deep learning transformer. Um, a basic idea of a looped model is instead of just going one after another, somewhere in the middle, you're gonna go back and, and do this, do this big loop. Um, and he, he was really interested in this idea, um, found that it could, you know, add another axis of scaling. You can get the, the same performance with a lot smaller model, um, stuff like that.
[00:04:34] Dan Fu: Um, and you know, in my mind I'm like, "Okay, th-this is a, this is a cute idea. It's, it's really interesting. Um, maybe someday, somewhere in, in a few years, someone will scale it up." Um, but then I think some news piece, like the information leaked that GPT-6 Astra might be a looped model, so I'm like, okay, you know, the, the, the, the, the time from cute research idea in the lab to someone reads it [00:05:00] and goes and implements it, uh, in, in industry at the, at, at, at a, in, in, in the biggest, the biggest cases seems to be a lot narrower than, than it was before. Um, So, uh, you know, the, the line is kind of blurring. Um, but yeah, the it's, it's, it's, it's very... I think the, the, the ideas that for me are just driven by pure interest happen in the lab and then at, uh, together with the, with that team, it's a lot more driven by some big open source model, how do we make that one in particular more efficient?
[00:05:32] Andrew Zigler: Totally. It changes the math on like to the traditional kind of R&D idea and like what you're saying about like you bring the idea to the lab where it's like really early in its infancy or it's not relevant in the pathway of immediate application yet, but you can bet or you can foresee, you can hypothesize that it will be, and it will take X amount of time to figure it out.
[00:05:53] Andrew Zigler: And if we figure it out now, then we won't have this bottleneck later. So like that's an evaluation you're constantly making in the lab of like, what do [00:06:00] we pick up? What's gonna unblock us? Like, because w- w- really what's happening, especially in engineering, like teams are working so quickly, uh, producing so much more code.
[00:06:09] Andrew Zigler: The cognitive load of what they're like putting out day to day is, is so massive that they usually can't even get a grasp on it, and this then causes a whole bunch of bottlenecks, right? Like, you have the review bottlenecks where like code review is just turned into a big ceremony. No one reads it anymore.
[00:06:26] Andrew Zigler: You have an ownership issue with the code and who wrote it. Oh, an agent wrote it, but who prompted the agent? Where did that agent come from? What,
[00:06:33] Dan Fu: right
[00:06:34] Andrew Zigler: what is it, what is it building on? So all of these problems, uh, there's so many of them, and so teams kind of get like scattershot about like, what do we attack and what do we, what do we solve?
[00:06:43] Andrew Zigler: And by having a lab, you can be more strategic about like this is our menu of
[00:06:49] Dan Fu: Yeah, exactly.
[00:06:50] Andrew Zigler: and
[00:06:51] Dan Fu: Yeah
[00:06:51] Andrew Zigler: these are the pain points that we need to apply deep thinking, and not only thinking, but we literally need time. The access of time is [00:07:00] not scaling at the access of capability from the model, so we literally need time to bake these ideas.
[00:07:06] Andrew Zigler: I can't make time. You know, I can't make more time. And
[00:07:08] Dan Fu: Yeah.
[00:07:09] Andrew Zigler: uh, it, and so it really becomes like symbiotic between the two, the research and the, and the lab, yeah.
[00:07:16] Dan Fu: Yeah, I agree. Yeah. I, I think the, the real thing that differentiates the lab research is that it has to be really driven by, by personal, by personal interest and, and student interest. Um, it's like, uh, I, I always tell my students, like, "If you pitch me an idea and you tell me why it's useful," I'm like, "Mm, maybe not.
[00:07:35] Dan Fu: But if you tell me why you're interested in it, then that's something that you wanna go spend, spend a
[00:07:39] Andrew Zigler: Totally. Totally. It's
[00:07:41] Dan Fu: it's a, it's a very different value function and, and selection function there
[00:07:46] Andrew Zigler: And, and you call out, uh, how it becomes a quality in a person. Uh, like I was at a talk, it was a, it was a conversation hosted by Dwarkesh and Dr. Fei-Fei Li, and it was in, [00:08:00] uh, Work OS was hosting it. It was really cool. And, uh, something that she said she admired in her students, in her lab is intellectual fearlessness of just being
[00:08:10] Dan Fu: Mm-hmm.
[00:08:11] Andrew Zigler: completely fearless about being curious about something and not giving up on that curiosity and not getting discouraged along the way.
[00:08:19] Andrew Zigler: And that for her, like when she had a student that had the, that pairing of that intellectual fearlessness and they came to her framing it exactly how you did. Like I r- I'm really curious about this. I wanna know about this. I think this is important. It transforms like the whole research process. Um, and the timelines, like you said, become so compressed where you're like actually now this once upon a time moonshot could be a realistic thing to shoot for.
[00:08:45] Andrew Zigler: Um, and, and how does that, how does that impact like how you th- even think about like hiring and skills you look for? Uh, both like in the lab, but really I guess moving to like thinking of applying this into the engineering realm where a lot of our listeners, um, th- that's where they, that's where they work every day and [00:09:00] they're wondering how to apply this kind of thought.
[00:09:01] Andrew Zigler: Like how do you, how do you bring that into bringing in really good engineers?
[00:09:05] Dan Fu: Yeah, yeah. Um, I'll, I'll, I'll address the, the lab piece quickly. I think when I'm, when I'm looking for folks for the lab, um, I, I think that, that phrase intellectual fearlessness is, is a really good one. So there's, you know, in the history of research, there's countless ideas that, um, started with, you know, "Oh, I had this idea but my advisor said no."
[00:09:23] Dan Fu: Um, and or like, "I, I had this idea but someone somewhere who should've known better, better than me d- discouraged me." Um, but the, the really f- great thing about a PhD is that, that you have the time to, to explore that and you have the time to, to go down all those paths that, that may not make it. So,
[00:09:38] Andrew Zigler: No until you can't go down the path is what the PhD is all about. Yeah.
[00:09:42] Dan Fu: yeah, yeah. Um, it's, it's almost the, the naiveté of like, uh, you know, "Nobody told me that this thing was impossible un- until, until I did it."
[00:09:50] Andrew Zigler: Until I did it, yeah
[00:09:52] Dan Fu: exactly, exactly. Um, so yeah, that, that, that, that's certainly one quality I look for in the lab. Um, when I'm building the team out together, [00:10:00] uh, I think it's, it's a, uh...
[00:10:02] Dan Fu: You know, that intellectual curiosity is, is also some- something that, that I look for. But, um, a lot more of it is kind of, um, who's willing to get down and dirty with the details. So the systems type of work is very much, um, you know... I, I think one thing that, that those of us who spend a lot of time in it is that, uh, you can actually go in and understand all the details.
[00:10:23] Dan Fu: You can actually go all the way down to the hardware and ur- understand what every single little thing is doing. Um, there's this part of AI machine learning research that's kind of the models are too big, there's too many parameters. Uh, it's like during-- doing neuroscience or something to try to understand why the model said A or B.
[00:10:42] Andrew Zigler: Mm-hmm.
[00:10:42] Dan Fu: Um, but when you're just optimizing the, the, the runtime of a model or optimizing a attention module or a kernel or something like that, uh, it's also very complex. There's a lot of moving parts of course, but you actually can go down and say, "This is what's happening on the GPU," all the little pieces. [00:11:00] Um, and oftentimes you need to go and un- and, and look under all those stones to really understand what's happening. Um, so that, that is the quality that I select for more in the engineering side, just that, um, leave no stone unturned
[00:11:15] Andrew Zigler: Exactly
[00:11:16] Dan Fu: Um, and then of course I think if you-- you have to enjoy the, the, the
[00:11:20] Andrew Zigler: The system, the deep systems work. The person you're describing is turning over every single stone. It- We're, we're operating in a very, like, type safe kind of reality. And so it's like, it's, it's real- it's really kind of, uh, funny to see how the shape of the kind of tooling that you work on really impact, like, the actual, like, skill sets that you need for very specific kinds of engineering, especially when you get so close to the hardware.
[00:11:45] Andrew Zigler: Because there's many, there's many engineering leaders right now who would argue that, like, the engineer they're looking for has this fusion of, like, product craft and understanding the customer need and talking with them. But [00:12:00] it also still speaks to, like, the deep engineering work that's still required to scale all of that opportunity, which is what y'all focus on at Together.
[00:12:10] Andrew Zigler: It's kind of the idea of being in, like, the cloud category right now, and you see a lot of folks call it, like, the neo cloud because it's really about a mindset shift, a reset about how we think about those resources and scaling them. And since we're accessing them and leveraging them in new ways, now suddenly, uh, we have to, um, do deep research to understand, uh, how do we get, um, every little bit out at the scale that we're going.
[00:12:36] Andrew Zigler: So
[00:12:37] Dan Fu: Yeah. I
[00:12:37] Andrew Zigler: how do you think about that? How do you think about navigating that world in, in the cl- in the cloud world now?
[00:12:42] Dan Fu: Yeah. It's, it's very-- I think the type of work that we do is very deeply technical, um, and it is the type of thing that I, I think, you know, when you talk about the engineers who have that product sense, certainly, like, we, we look for that. We, we, we love having folks who can talk to customers and, and understand those bottlenecks. [00:13:00] Um, but, uh, once you do that, it ends up boiling down to two or three numbers. And then, then you have to actually go and, and make the, the thing, like, uh, get more... Like, it's gonna boil down to cost or, like, some quality thing or some, some TPS. And then once you, once you understand that and, and what's happening on the other side, which is, uh, if you, if you do it right, it's one or two calls, then you have the rest of your week, the rest of the month to actually go and, and hit those, those challenging technical goals.
[00:13:29] Andrew Zigler: Right
[00:13:30] Dan Fu: I think it's, it's interesting. So we've certainly started using AI a lot to accelerate our workload. So, uh, we can write kernels way faster than we were able to, uh, you know, a year ago. We're able to build entire parts of the inference stack a lot faster than we were able to a year ago. Um, but one thing that is definitely, uh, I think more true now than, than it was before is that the more domain expertise you have in something, the faster the AI [00:14:00] tooling lets you go.
[00:14:01] Dan Fu: So if you are coming to it and you don't have a really strong understanding what's happening on the hardware, what's happening in this, in this complex software stack, uh, the AI will let you write a lot of code, but it might not be the right code. It might not have the right design. It might not be doing the right-- might not even be correct at be- because of some ABC, um, uh, different, uh, uh, edge cases. Um, so when you're, when you're in that position, I think what we found is that the more domain expertise you have, the more you deeply understand something, the more effective you are, the faster you can go. Um, yeah.
[00:14:41] Andrew Zigler: Yeah, absolutely. It's, it, it speaks to AI as the amplifier in this case. Like, if you don't have those skill sets, if you don't have that, that curiosity, if you don't have that, uh, you know, you can lack the domain expertise. But if you also lack the reflective skills with an agent to build that [00:15:00] expertise and the pathways of building that expertise, like, outside of a chat, right?
[00:15:03] Andrew Zigler: Then you kind
[00:15:04] Dan Fu: Yeah
[00:15:05] Andrew Zigler: s- starve in, in terms of being able to, like, leverage that for decisions. And so, like, for t- people who are working on, like, deep system stuff, there's a lot of partnership still between agents and humans to get stuff across the finish line, especially because it all comes from a realm of innovation, which agents have spiky, you know, things within, but is still ultimately, like, requires a bit of a spark that it can be hard just, like, to expect to appear in a loop somewhere.
[00:15:37] Andrew Zigler: Um, and, and, and another thing too that I've noticed is that, like, AI enablement came for everyone unequally, and, like, for people that were in the product world, the SaaS world, the customer-facing world, writing the Reacts and the front ends of, like, their ecosystems and their industry and their, the tooling they used every day evolved [00:16:00] so rapidly and so quickly over and over again that just, like, fundamentally the way that they were lev- able to leverage the tools were fundamentally different.
[00:16:07] Andrew Zigler: Meanwhile, you have folks that were on a lot of, like, uh, embedded lang- like, embedded systems and, like, deep languages and, and, and, and things that did not have as much, uh, code in the training data that all these foundation models are trained on, that did not have those same experiences. And so they would use those same tools and get, like, crappy results, or they would not get anything near what their peers are talking about, and then you get, like, a, a distaste.
[00:16:31] Andrew Zigler: You get a divide. So, you know, uh, I think it's always been about, like, within that realm, where if you've been, like, shunned away by AI because it was really crappy at your domain, having, like, the persistence of pushing it and being like, "Why is it bad at my domain? What can I give it? What's the scaffolding I need to do?"
[00:16:50] Andrew Zigler: I imagine that, like, y'all have just ended up building this, like, you know, you know, probably a wild-looking factory in terms of how you push the research along between each other as [00:17:00] researchers, but then also across the line into, like, the engineers. How do you think about systematizing it as a, you know, a systems thinker?
[00:17:07] Dan Fu: Yeah, I think, I think it's, it's really interesting. So, I mean, first of all, the AI ability to generate really fast front-end apps and things like that. So half of, half of understanding what's happening to these systems is building the right visualizations, understanding, and understanding the data deeply, um, and doing that.
[00:17:24] Dan Fu: So even, even, you know, before it was able to write, uh... AI was able to write kernels at all, the ability to build those visualizations was hugely impactful,
[00:17:34] Andrew Zigler: Yeah
[00:17:34] Dan Fu: for, for folks', um, acceleration. So we, we built our own, like, Perfetto viewers, our own trace viewers, um, our own tools for tracking, uh, how fast the, the engine is going, how much, how much data it's processing, how many tokens.
[00:17:49] Andrew Zigler: You made windows. You made windows, like you had this like big black box where all this stuff was happening and you, you punched some windows into it. You created for the first time a way to share and talk about it.
[00:17:59] Dan Fu: Yeah. Yeah. Yeah. [00:18:00] I thought you were talking about Windows the operating system at first. I'm like,
[00:18:03] Andrew Zigler: no, no. Literally, li- no, lit- no, literal- you punched literal holes in the wall of the black box, and you said, "We have the opportunity to not only go really fast inside this black box, but wouldn't it be cool if we leveraged some of that to visualize and to share language on, like, what is happening?"
[00:18:19] Andrew Zigler: And I imagine this kinda helps kind of tackle bottlenecks too, 'cause now you kinda have language.
[00:18:25] Dan Fu: so that ability to-- it, it gives you a new microscope almost to, to go in and look at what's happening and, and understand what's happening. Um, that, that tooling was hugely impactful. Uh, the, the AI models, the coding models still kept improving to the
[00:18:39] Andrew Zigler: Yes
[00:18:39] Dan Fu: I think a year ago they were able to write some kernels but, but not a ton.
[00:18:43] Dan Fu: Now, now they can write, um, a lot of simple kernels and if, if you equip a strong kernel engineer with, with a, with a good coding model, you can, you can generate a lot of kernels. Sorry, and for, for the viewers who, who don't know, so kernel just means GPU program. It's
[00:18:58] Andrew Zigler: Hmm
[00:18:59] Dan Fu: it's a fancy word [00:19:00] for GPU program.
[00:19:00] Dan Fu: I, I don't know why, why there's a different word for it. Um,
[00:19:03] Andrew Zigler: Just, it just has to be special. It's just a little building bl- it's just a little building block of leveraging the literal GPU hardware to, to the bridge into the realm that we're trying to all build on top of.
[00:19:15] Dan Fu: Exactly.
[00:19:15] Andrew Zigler: why that's, like, in conversations like this, you know, I, you're not even, like, the first, like, kernel researcher that we've talked to on this show.
[00:19:22] Andrew Zigler: Like, ultimately, so much of this stuff boils down to this level because this is about as fundamental as you get before you start just moving into, like, raw power inputs and things. And so, uh, it, it, it becomes where the problem, uh, has, like, the highest opportunity for folks that, like, really eke out a small advantage.
[00:19:39] Andrew Zigler: That advantage scales all that way up. And so that, that becomes, like, why the opportunity's so big. And speaking of the opportunity being so big, you know, Together is expanding. Uh, y'all have, like, are continuing to get more partnerships. Your surface area of your platform where people are training and running models gets bigger, gets more higher [00:20:00] performant.
[00:20:00] Andrew Zigler: Um, what are some of, like, the big bets that y'all are making, uh, in the future, like, as you think about, like, defining the AI cloud category?
[00:20:12] Dan Fu: Yes. Yeah. So I think-- so the-- when, when I think of AI cloud, I, I, I think a couple-- there, there's a couple analogies I think of. So first is, uh, you know, what is the analogy to, to the old traditional digital cloud? Um, if you look at AWS, it's not just machines that are running CPUs, it's the databases, it's the logging, it's the, um, you know, all, all those other pieces to it. Um, and I think the AI cloud, we are building up similar sets of infrastructure, um, as, as we grow. So, um, it's not just, not just GPUs for, for training your models, um, but it's once you've trained the model. So training is a one-time cost in, in some sense, but then, um, the only time that anybody ever makes money in, in AI, the only time when you actually deliver value is at the inference part of it because, um, you [00:21:00] know, that-that's when people actually use the thing
[00:21:01] Andrew Zigler: That's the consumption endpoint where you charge them. Yes
[00:21:05] Dan Fu: So we're, we're spending a lot of time thinking about inference, um, scaling inference. We have partnerships with great model providers like Moonshot and, and Minimax and, and Cursor, um, who, who are of course spending a lot of time training really strong models, and then we partner with them to, to serve inference.
[00:21:21] Dan Fu: So that's, that's gonna be a, a major strategic bet for going forward, um, where we're going and looking at how do you give training, uh, you know, expose training to, to more people. So,
[00:21:34] Andrew Zigler: Yeah
[00:21:35] Dan Fu: the training side of the business we've worked with very deep AI native companies where, you know, it's the, it's the AI PhDs who, who dropped out of their PhDs to go build a model company. Um, there's a lot of folks who, who are just, just getting into it and, and es-es-ex, uh, especially as AI expands in the economy, there's, there's a lot of other folks who are thinking, "How can I [00:22:00] use this great new technology in my own business?" Um, and you might need to customize it or, or post-train it or, or fine-tune it.
[00:22:07] Dan Fu: So we're, we're thinking about th-those folks and, and how do you enable those folks, um, to, to, to, to, to use these, these new capabilities that, that are co-going, uh, that are being developed and, and how do you, how do you get them, get them to work? Um, so th-those are a couple bets and, um, you know, we're, we're just seeing the demand for AI grow in such, such crazy ways, so we're, we're just, um, you know, a lot of it is a supply game, a compute game.
[00:22:35] Dan Fu: How do you get more compute? How do you get more supply so that you can help more tokens for the world?
[00:22:42] Andrew Zigler: Yeah, absolutely. I think the, some of the biggest challenges I think are about definitely taking this tooling, this infrastructure that used to be highly specialized for a very sub, small subset of people or use cases, and now we have, we've totally cracked that open. We've acknowledged that we're on [00:23:00] this jagged frontier where even if we stopped innovating today on all things AI, there's still so much to go back and implement across every industry, all these work surfaces.
[00:23:10] Andrew Zigler: There's so much of, of that work to be, to be done, and we haven't even really started taking big bites out of it. And so, like, I, I think that's, like, gonna be a huge opportunity, and so that just then swells up where all of that is going to run, like, what's gonna support that, but also what's going to enable these companies, these people who now for the first time have to approach the, "Oh, we need a model that's a little specialized on this," or, "We need to figure out how to take this very specific combination of domain expertise and fuse it with a model to, to...
[00:23:44] Andrew Zigler: A- and then serve it to our customers." 'Cause our customers now, that's what they expect. They don't wanna come to my app. They don't wanna come to my website and log in and click around and look at dashboards. They have their own dashboard world that they live in where all of their stuff flows in, and they need to [00:24:00] be able to...
[00:24:00] Andrew Zigler: They, they need to, on a consumer level, being able to plug into that. And it's really interesting to start to see, like, major players in, like, the market start to tackle the consumer grade AI agentic assistant kind of thing. You have, like, Muse, which is making a big stir, and everyone has a different opinion on it.
[00:24:18] Andrew Zigler: And, and so I think that just represents, like, the very beginning of what's gonna be, like, a vast amount of enablement. Uh,
[00:24:27] Dan Fu: Yes.
[00:24:28] Andrew Zigler: more like, uh, we've talked- joked about it on this show before, kind of comparing it almost to, like, the eternal September of, like, AOL and Newsnet of, like, the idea that once AOL started sending out, uh, internet access emails to everybody in the mail and people got online for the first time, things like, uh, the, the, the user net boards were just expanding with people saying hello for the first time.
[00:24:50] Andrew Zigler: And that used to just be when college would start, 'cause those were the only people who had access to the internet, were college people and students. So,
[00:24:58] Dan Fu: Right,
[00:24:58] Andrew Zigler: like, all of a sudden, [00:25:00] everyone, every- this whole world that college lived in is occupied by people. That's why we, that's why we exist and tolerate all the weird things about ma- like, the mail that we do.
[00:25:10] Andrew Zigler: Because then all of consumer mail was built on top of the pre-existing mail structures. And also, um, I think about, like, news groups and user groups, and about how, like, that's been reflected into how we, like, communicate on the web. And all of that comes out of, like, places like academia, specialized computer researchers tinkering and building systems, thinking about how this might scale, and then it scaling explosively out of their control, and then all they can s- do is, you know, make, make changes and, and modify things over time.
[00:25:41] Andrew Zigler: Uh, it... And that's kind of a parallel that I see that we're, like, on a bit of a precipice with, with, with, with this kind of compute
[00:25:49] Dan Fu: Yeah. There, there, yeah, there, there's a ton of interesting parallels there. So my favorite one is if you go back to some of the old papers from the '60s that, uh, kind of produce or propose [00:26:00] email for the first time,
[00:26:01] Andrew Zigler: Mm-hmm.
[00:26:01] Dan Fu: they didn't call it email, um, but they, they kind of said, "Oh, you could have these two computers, and they could have a network between them, and you can send messages from one to the other, and you can have a, a, a list of messages that you read on the other side, and it can be just like sending letters o- over, over the, the, the old-fashioned
[00:26:17] Andrew Zigler: They didn't even have the language to,
[00:26:18] Dan Fu: the
[00:26:18] Andrew Zigler: express it
[00:26:20] Dan Fu: uh, at the very end of some of those papers, there's a section that's like, "You could also say, so, you know, since now these are, these are all virtual, so you could send one, one piece of mail, send it to many people, and that could be really interesting to communicate to a lot of people. But you might get this thing where people get a lot of unwanted, a lot of unwanted mail in, in their inboxes." And it was like this, this, this, this paper from the '60s that was, uh, predicting spam email. Um, and it's like, yo, uh, it's, it's, it's really interesting that you can go back and, and, and see some, some of those parallels.
[00:26:53] Andrew Zigler: Yeah, you can extrapolate then what happened against the reality. Like, it's so funny to think, like, [00:27:00] the, so, uh, obviously, you know, some PhD was thinking about, well, when you scale up this mail thing and you make it accessible, and then, then, uh, that hypothesis becomes an all too unfortunate reality that we're all just incredibly aware of.
[00:27:13] Andrew Zigler: I think that's, like, the fun thing about research right now is, like, because this technology will be so widely transformative, those little, like, unassuming turn over the rock and find something interesting things could have really profound implications. So being like, there's a lot of value right now of, like, being out there with the pickaxe and digging around and trying to find stuff and dig things out of the ground.
[00:27:36] Andrew Zigler: Um, I've had a lot of, uh, like I- I've really enjoyed, uh, working more on research papers, collaborating with, like, researchers. I've done a lot of that in the last year, um, including even participating in a conference where I wrote a paper 'cause it's just, like, I think there's so much opportunity to, like, bridge stuff from the engineering world and, and all of, like, the things that we, the [00:28:00] tooling we use, and share them more broadly.
[00:28:02] Andrew Zigler: And it just turned out that, like, um, there was a whole conference set up for people sharing those same kinds of ideas as me, and that's kind of the opportunity I think that folks need to have now if, if, like, they're really curious.
[00:28:14] Dan Fu: Yeah, especially I think, especially in a, you know, when there, there's been an interesting thing going on, I think, in, in math academia where everybody got access to these really powerful models, um, and suddenly these models could do proofs that were just your-- that you're trying to work on for, for a
[00:28:28] Andrew Zigler: we haven't even talked about the math world. They're having a total meltdown over there. We've all... Yeah.
[00:28:34] Dan Fu: yeah. I think, I think there's something nice about sitting down and writing a paper yourself though, because it, it gives you the opportunity to, you know, really think about what you wanna say and how you wanna say it and, um, what, what, you know, when you're talking about research paper, like you, you have some set of ideas, some perspective on the world.
[00:28:53] Dan Fu: How do you, um, how do you test that out for yourself to s- to see how true it is? Or, or maybe there's, there's something [00:29:00] that you didn't quite understand. Um, and then how do you communicate that? How do you present it in a form where people really, people really get it and, and understand it and,
[00:29:10] Andrew Zigler: And they care. Like, I think that's what makes,
[00:29:12] Dan Fu: Yeah,
[00:29:13] Andrew Zigler: the research paper really stand out is, you know, you read the beginning of it, you're, you're understanding what they're going to explore, what the abstract is, and maybe the conclusion is solid, but, you know, I want the conclusion to really matter or to have some sort of resonant kind of impact, even if it's not a, a technical one, even if it's just like a, an implication of something to think about.
[00:29:34] Andrew Zigler: That's, that, that for me is what I, what I always reach for. Um, it's like, what is the, what's the, what's the h- the reason in here, um, that I would care about what this person learned in this paper?
[00:29:45] Dan Fu: Exactly. Yeah. That, that connection between like when you're writing, you're really trying to understand and, and really get to the, to the root of your own thoughts. And then, um, like the, to, to, to spend a lot of time writing something and to, and [00:30:00] then to put it out there for people to, to read and consume and react to and, uh, and, and give you feedback on is I think, you know, you know, part of, part of what makes this, this whole research thing so fun, part of, you know, being, being in this community.
[00:30:13] Dan Fu: Um, and yeah, I'm, I'm glad that, that we still have that, that capacity and that, that capability to do it. Um, yeah
[00:30:21] Andrew Zigler: Yeah. So I, I want to take an opportunity to pivot just a little bit here and talk about more on the engineering application world
[00:30:31] Dan Fu: Mm-hmm.
[00:30:31] Andrew Zigler: of, uh, u- using and scaling AI, particularly on a team level, because this is really the nut we've been trying to crack on Dev Interrupted. Maybe you can help us crack it. And, and, and the problem is, is that as you start to get more agentic as a team, you get pockets of maybe, like, one really agentic person or f- or a really agentic team, and you have, and you have, uh, team leads and your CTO, CFO asking, like, you know, "We're spending all of this money on [00:31:00] AI.
[00:31:00] Andrew Zigler: We're trying to measure the impact, the outcomes of that. There's clearly some agentic AI acceleration stuff going on, but w- you know, how do we talk about it? What's actually getting shipped? Why does that actually matter?" All of these become, like, really big questions that traditional processes in engineering were built to solve, like code review and sprint planning and, and that kind of stuff kind of crumbles at the scale that gets unlocked by the way that we're, a lot of teams are working now.
[00:31:32] Andrew Zigler: How have y'all thought about building an engineering org AI first and, uh, preventing those bottlenecks from appearing? How do you think about ownership within, with code as it moves through the, the SDLC? These are kinds of questions that a, a lot of our listeners are, are, are grappling with
[00:31:53] Dan Fu: Yeah. I think, so here, maybe, maybe I'll, I'll separate my, my answer into, into a couple parts. So, [00:32:00] um, I think there's a question in there about tracking cost, like how do you-- and how you do attribute, we've been spending all this AI spend, uh, is it actually accelerate- accelerating us? Is it actually making things better?
[00:32:11] Dan Fu: Um, there's a portion there around, um, you know, ownership of different features or, or different ideas. Um, and then there's, I think, a little bit in there about, uh, how do you, like the nuts and bolts of how do you make, um, it easier for AI to accelerate your, your coding workflow. So, um, let me start, you know, a- addressing those, um, um, one, one piece at a time.
[00:32:36] Dan Fu: So I think, um, at the highest level, so going after the first one about that, uh, that, that cost piece and the how do you attribute acceleration or non-acceleration to, to AI, um, coding piece, uh, to, to A- an AI spend. Like how do you know if this thing was, was worth it when, when the CFO co- comes knocking? Um, I think for, for us, uh, [00:33:00] almost the traditional metrics of how do you evaluate people, um, how do you evaluate, you know...
[00:33:06] Dan Fu: Like how, how do you know if your engineer is good, right? How, like how, how do you know what, what they've been producing? What, um, like is it time for a promotion? Is it time for a raise? Um, do you need to have a conversation about, uh, about their output and, and where they're spending their, their energy? I think, um And actually connecting back to the ownership piece, the world-- that part of the world is still pretty similar.
[00:33:27] Dan Fu: So like there's things that, that you need to get done. Um, you have to think deeply about how you split up the responsibilities. You have to think about the organizational pieces of who controls or who is responsible for which parts of the stack. Um, all that, all that thought, all that effort still needs to go into it. Uh, just because people have AI and like you've, we've heard people talk about AI agents, you're managing a team of AI agents and things like that. Um, you're a manager or a manager of an organization, you're running or- an organization, you still have a [00:34:00] responsibility to really think deeply about what your people are doing. How do you help them grow? How do you make sure that they have opportunities to show off, um, their, their, their capabilities? When somebody is ready for it, you-- like, how do you, you know, know when to give them more responsibility for something that, that is key to the business? Um, I think all those pieces are, are still the same.
[00:34:23] Dan Fu: So, uh, the way that we do it, if there are key projects, if there, if there are key features that need to be implemented, um, we still have people who are DRIs. We, we still have people who are, who are responsible for it. Um, and the tracking, the management of that progress still looks very much like, like it did ev-even before AI. Uh, I think, uh, a major difference is just, um, the velocity of how fast those things get done. So, um, you know, we, we have to accelerate our planning. We have to change our timelines. We have to think about, um, you know, how, uh, how many people do we need and, and how much time do, do, do we need? So, [00:35:00] so those things change, but like, um, your star players are still your star players. usually clear to everybody in the organization who the star players are. Um, and, uh, and you can see, like, if you, if you look at where they were now or versus where they were six months ago or a year ago, if you've been following their progress o-over that time, you, you, you can see the improvement as, as the AI spend goes up. and the flip side of that is, okay, at some point somebody ta-takes a listen, it's like, okay, here's the AI spend of everybody in the organization. Uh, and then you go and you, you, you bring it to the managers who, who knows who's doing what, um, and who's tracking kind of, kind of the impact, the same way that you do at a perf review cycle.
[00:35:41] Dan Fu: And you're like, "Okay, this person, this amount of money impact, super great. Okay, thumbs up, thumbs up." And most of the time it matches. Um, occasionally you have a few people who's like, "Oh wow, you spent that much money on AI?" And I don't know what you have to show for it, then you have to go and, and, and sort [00:36:00] of have, have conversations with those people. Um, a, it's a, it's an increased leverage point. So like if someone, um, you know, in, in some sense, uh, you know, sometimes you may have underperformers who in the past would've just sat there and then at, at perf time you're like, "Okay, what did you do? Let's try to figure out where you're spending your time."
[00:36:19] Dan Fu: Hopefully if you're a, if you're a good manager, you're having those conversations earlier because, um, you're hopefully checking in with your people more, more frequently. Um, I think there's, there's, there's a new failure mode of, okay, person who I thought was kind of sitting around, maybe I need to have a conversation with them.
[00:36:35] Dan Fu: Oh, it turns out they spend this much money on, on AI spend. So that, that's kind of a new failure mode. Um, I personally haven't really seen the one where it's like a really high performer that is also using too much AI spend. Usually they're pretty calibrated. Um, maybe it's because, you know, in, in, in our company and in our industry, like everybody kind of knows like what the spend correlates to on, on the other side. Um, but in, in many ways it's [00:37:00] still the classic engineering manager roles where you're like, um, trying to, trying to help your people grow, trying to figure out, um, you know, what's, what's the right way to split up work and, and responsibility. So, so a lot of those pieces, um, uh, stay a- actually quite similar, um, both in terms, terms of ownership and in terms of, um, uh, tracking the spend and, and attributing the spend to different, uh, to, to particular things that, that show up, um, at, at, uh, at the bottom line at, at the end. Um, I'm sure as you scale up much larger organizations, uh, that this becomes a much bigger fractal problem.
[00:37:40] Andrew Zigler: Mm-hmm.
[00:37:40] Dan Fu: I think even then, you know, that, that was still the case if you were running a 1,000-person organization before, you still had to answer to, what are my 1,000 people doing? Um, now it's 1,000 people plus the AI spend, um, and, and hopefully fair- failure correlated.
[00:37:55] Andrew Zigler: Yeah
[00:37:56] Dan Fu: there was another piece in there about how do you actually make, it [00:38:00] easier for, for AI agents to, to operate on your code, um, and, and accelerate things. I think there we've, we've invested more time in, uh, kind of like, uh, markdown agent readmes about what's going on so that an agent can pick up a, a works, uh, a, a piece of work and, and really start running with it. Um, and there's, I think there's, there's a new set of best engineering practices that is developing around coordinating, orchestrating using agents.
[00:38:29] Andrew Zigler: Yeah. I am actually really quite intrigued that you used the word fractal in there to describe it, because that is actually quite an in- a really sh- accurate way to describe what happens to the SDLC as folks get more and more agentic. Because, you know, earlier in the year we had, like, the whole Gastown transformation.
[00:38:48] Andrew Zigler: You had people starting to figure out, like, "How do I do this whole agentic engineering orchestration by myself at scale? I have, like, 20 sub-agents or, and one orchestrator. Like, how do I [00:39:00] build this?" And that person did experience a lot of gain. We, we see this is visible, like, in our benchmarks at LinearB.
[00:39:07] Andrew Zigler: You can see at the beginning of the year the high AI usage, which was registered by just amount of their consumption usage compared to the lower users. They just completely divided in their, uh, PR output and their lines of code output, like, more than 2X in some of those cases. And so that was kind of, like, the transformation that happened for folks.
[00:39:29] Andrew Zigler: And w- when th- that, when, when, like, the, the Gastown individual transformation came, it then started to call into question a lot of the, the parts of, like, what was traditional in the SDLC, and one of them being, like, reviews. Like, you called out, like, m- in major, major features, stuff that is still, like, has a, an owner.
[00:39:50] Andrew Zigler: We, we, you know, I w- definitely I think most leaders on this show would agree with you. Everyone says that these, that the PRs, even the ones that are agentically driven and created, they still [00:40:00] have owners. But then what happens is in this, going back to your fractal comparison, is that, and really what's happened is now the engineer themselves has become a mini SDLC because they have all of these agents, and they have all these repos, and there's processes.
[00:40:14] Andrew Zigler: And we're not really living in a world where it's like my Gastown's gonna call your Gastown. Like, actually merging into main, getting nice and clean up there, it, we're still figuring a lot of that out, which is, I think, really fascinating. A, a, a big problem that we've seen pop up, too, is, um, you get an uneven distribution in that review load.
[00:40:37] Andrew Zigler: So it does become important to, like, pay attention to who, uh, who cognitively owns, who's the DRI on, like, a, the biggest bulk of agentic space, and maybe they themselves aren't pushing and, and, and, and writing a lot. Maybe they're not even agentically driving the creation of that code, but if they are the DRI, how big is that surface area getting?
[00:40:57] Andrew Zigler: Are they able to stay, um, at the [00:41:00] same mental pace? So th- these become the new challenges, which are totally I think foreign for a, a lot of folks on these teams. So fractal is definitely a fascinating way to, to think about it. Apply the challenges of up high to solving the ones down low. Um, and I, I wanna ask too, like do you think about it in terms of a factory metaphor?
[00:41:20] Andrew Zigler: We see a lot of people talk about things like now as a software factory. So how, is that kind of a language that y'all apply internally or do you think about it differently?
[00:41:28] Dan Fu: yeah, we, we, we, we haven't used factory, I think, as a, as a metaphor before. I think, um, you know, that's, uh, it's, it's not really assembly line type of, type of work, I think. Um, it's, uh, I, I view it as a, a accelerant of individual people or, or individual engineers who, who are working on, um, particular features.
[00:41:50] Dan Fu: And, uh, the-- what I've seen is just that, uh, the, the capabilities of the 10x engineer become, become even greater. You know,
[00:41:58] Andrew Zigler: Yeah, they turned, they turned like [00:42:00] 1000X engineer. I, I talked with,
[00:42:02] Dan Fu: yeah
[00:42:02] Andrew Zigler: we had Karthik Ramakrishnan. He's a distinguished engineer at LinkedIn Engineering. He built over at LinkedIn their, like, memory layer that enabled all of LinkedIn Engineering and, and LinkedIn in general to, like, have a shared contextual layer with their AI agents really early.
[00:42:18] Andrew Zigler: So he identified this opportunity that was gonna be a big leverage move for them. He's very much like a, a lab scientist thinker like yourself, and, uh, that was, that was a, a real keystone to that conversation. He was like, "I don't wanna be, like, the 1000X engineer. I don't wanna be that super agent. I want everyone to be a 10X person that they were yesterday.
[00:42:37] Andrew Zigler: You know, I wanna distribute, um, those gains." Because if you just end up with, like, the 1000X engineer, the 1000X engineer, what actually happens is they become black holes. The, like, there's an event horizon around them where things enter, and then you're not sure what the state of it is on the other side.
[00:42:55] Andrew Zigler: And so, uh, how do you distribute that, uh, uh, enablement to, like, [00:43:00] everybody is a challenge, too, for folks.
[00:43:02] Dan Fu: Right. Yeah, yeah.
[00:43:03] Andrew Zigler: Yeah.
[00:43:03] Dan Fu: I think s- p- uh, part of that becomes also on the engineering manager role, where, like, I think the, the ideal engineering manager, uh, is, uh, you know, spending time, of course. Like, there, there's the people managing of, like, you know, what, where, what is... where's everybody in their career?
[00:43:21] Dan Fu: How do they want to advance? What are they thinking about? But then there's also the technical side of things, like understanding what people have built, um, understanding what, what those features are and, and how, how you, how you might want to think about the roadmap for that. And I think that role become, becomes very important.
[00:43:40] Andrew Zigler: Yeah
[00:43:40] Dan Fu: and you know, it was already important before, and I think it's, it's still important. It's, it's
[00:43:44] Andrew Zigler: You, you-- Do you hear us listeners talk to each other? Engineers out there building things, swivels to your left, swivels to your right. Share what you're building. Get folks involved in the process, and, uh, we're moving so fast, it just kind of helps to just kind of be constantly giving everybody, [00:44:00] um, an update and pulling them along for the ride as well.
[00:44:02] Andrew Zigler: Especially when, you know, the, the work that you do especially, it allows so much to be leveraged. There's a lot of curiosity, um, so much that we've explored in this conversation. It's so evident about how passionate you are about what you work on, and so it's been a real delight, Dan, to chat with you and to get inside your head about how you think applying the kernel world to the engineering one.
[00:44:22] Andrew Zigler: And just as we wrap up, where can our listeners go to learn more about you and your lab and the work that you're doing at Together?
[00:44:30] Dan Fu: Yeah, yeah. Um, first, thanks so much for, for having me. It's, it's been a really, really fun conversation and, and really gra- glad to hear, uh, your perspective too on, um, how, how folks are, are adapting to, to these, to these new, to these new worlds. Um, so I'm on Twitter, uh, uh, uh, I'm RealDanFu on Twitter, so, so you can follow me there.
[00:44:50] Dan Fu: I,
[00:44:51] Andrew Zigler: Love it
[00:44:52] Dan Fu: lab puts, puts stuff out, um, you know, I'll, I'll tweet and, and, and, and, and release that. Um, uh, together we, we [00:45:00] also will put out research blogs and, and different things, um, at, at different various cadences. So, um, yeah, you can follow me there on Twitter. That, that's probably the, the easiest way to find me.
[00:45:09] Andrew Zigler: Amazing. We'll, we'll include links in the show notes, and I'll definitely be paying attention to the research lab as well for stuff coming out of your mind. And I, I appreciate, again, you, you lending us some time to give us a chat. And, uh, to those listening, obviously if you made it this far, you loved this conversation.
[00:45:24] Andrew Zigler: You're a total nerd about the intersection of research and engineering. So you should go find wherever you listen to this, whether it's on a podcast or a newsletter, give us a like or a comment. Come find Dan or I on social media, give us a poke or let us know what you thought about today. Uh, and Dan, thanks again for coming on Dev Interrupted.
[00:45:43] Andrew Zigler: It was a pleasure to host you
[00:45:45] Dan Fu: Thanks so much for having me. Uh, it was, it was a lot of fun to join.
