Are we officially entering the "Eternal Sloptember"? This week on the Friday Deploy, Ben and Andrew unpack the quiet rebellion against skyrocketing API costs as teams transition to fine-tuned local models. They also explore the changing physical architecture of AI data centers, the dangers of using autonomous tools as a crutch for broken workflows, and why spec-driven development is critical for keeping agentic code in check. Finally, the hosts share their latest personal agent experiments, from benchmarking open-source models on a local Mac Studio to taming an AI-generated second brain.
Show Notes
- Outsourcing plus LocalAI will soon become more economical vs Frontier labs
- AI Datacenters Were Built for GPUs. What Happens When You Remove the GPUs?
- "The AI Can Do It" Is Not an Excuse To Tolerate a Mess
- The Eternal Sloptember
- I’m tired of talking to AI
- If you let AI do your writing, I will come to your house and kill you
- A Blast from the Past: SDD and the Illusion of Known Scope
- Andrew’s paper:Mise en Place for Agentic Coding: Deliberate Preparation as Context Engineering Methodology
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Ben Lloyd Pearson: Opus 4.8 drops right as we come on to record this show. Uh, have you sent it any, any requests yet? Any prompts?
[00:00:07] Andrew Zigler: Not yet, but I will say my inbox is blowing up from every tool under the sun that I use, letting me know that I can now use Opus 4.8 in them, just like on, on Ritual. And I will say, I think that for our track record, this is now the second consecutive, Claude model drop that's happened literally as we've stepped in here to talk about the weekly news.
[00:00:27] Andrew Zigler: It's almost like they know when our segment is, but then they're, like, just trying to make sure we're slightly outdated or something. So Anthropic, if you're listening to this, just, you know, give us a heads up, uh, you know, if you're about to drop it because we're, we're seen to be aligned on our, on our timing.
[00:00:41] Ben Lloyd Pearson: Yeah, it'll be interesting to try it out. I mean, it doesn't seem like they're really making a whole lot of groundbreaking claims with this one, uh, you know, compared to like Mythos, for example. Um,
[00:00:50] Andrew Zigler: Yeah, it's a, it's maybe a more routine upgrade of a model that we've all grown pretty adapted to. And I will say, you know, there's flocks of people running towards these models right now, uh, but there's also a lot [00:01:00] of folks that are going against the grain, maybe going the other direction, and really revisiting local models and local infrastructure and maybe not making the same bets on, oh, these Anthropic models, oh, these OpenAI models, these frontier models are gonna get better and better.
[00:01:15] Andrew Zigler: I should put all of my work and time on them. I think what we're seeing is, uh, the reality of using them at scale and the cost is, uh, like a pot of boiling water, and we're all frogs in it right now. What do you th- what do you think?
[00:01:27] Ben Lloyd Pearson: Well, we're, we're gonna get into that. Uh, it's definitely one of the topics we're gonna cover today. So welcome to the Friday Deploy, brought to you by LinearB. I'm your host, Ben Lloyd Pearson
[00:01:37] Andrew Zigler: And I'm your host, Andrew Zigler
[00:01:39] Ben Lloyd Pearson: And this week we are covering moving your AI models local, data centers and how they're changing the needs of infrastructure, as a workflow crutch, the eternal slop-tember, we're all tired of AI conversations, even the bots potentially. Andrew, let's get into it. You, you mentioned, you know, moving things [00:02:00] local. What's this article that we have about, you know, outsourcing and using local AI to, to take a more economical approach to AI usage?
[00:02:10] Andrew Zigler: Yes, it's, if you've been using inference, and if you're listening to this podcast, you most likely are, you've noticed that your bill, um, has been starting to climb. The API consumption costs at major frontier model providers is slowly escalating, like I, you know, mentioned at the top of our show. Um, and this is pushing a lot of teams and a lot of leaders that have now invested and built AI infrastructure to reconsider what models they're betting on.
[00:02:37] Andrew Zigler: Uh, a lot of folks are actually running the opposite direction from these new model releases and, fine-tuning local models like Qwen3 and a whole bunch that are Apache 2 at this point. You fine-tune them on your own local data, you host them on your own infrastructure, run them on your own GPUs.
[00:02:53] Andrew Zigler: Eliminates a lot of the uncertainty about budgeting token costs for the future. And what's really interesting [00:03:00] here is that, you know, our token consumption continues to climb, and if the token costs are going to start slowly increasing at the same time, you're gonna get this compounding increase in price that's gonna sneak up on a lot of people.
[00:03:13] Andrew Zigler: So that's why we're seeing these large organizations get really strategic about it early. We talked about Shopify fine-tuning an agent that then allowed them to build out a whole internal multi-agent infrastructure they couldn't even afford before on the inference when they were using it off the shelf from a foundation model.
[00:03:30] Andrew Zigler: So, um, I just, you know, want to remind folks that the technology is out there to own this yourself, to figure out what it would mean to have your own, uh, specialized intelligence, your own specialized harness. You don't have to rely on just the top model providers. And we're gonna be in a world soon where model routing and choosing when to use a, quote, "dumb rout-- uh, dumb model" or, quote, "a smart model" is gonna become a more important choice than it ever has been before.[00:04:00]
[00:04:00] Andrew Zigler: So, um, those are some of the moves that I'm seeing contrary to a lot of these new model drops happening like Google, uh, last week as well.
[00:04:09] Ben Lloyd Pearson: Yeah. And, and of course, we've already seen a bit of this with GitHub Copilot. You know, they switched to usage-based off of their seat-based pricing, and, uh, cost went up substantially. I think in some cases it was double-digit, uh, increases. And, and this article that, that we'll include in the show notes, it also covers how if you combine lower cost models plus local models when you can, so lower cost frontier plus local models plus developers in regions that have a lower hiring cost, you actually get a, a massive compounding efficiency gain, versus like, know, someone in a first-tier market spend- using the most expensive model, uh, all the time, you know.
[00:04:53] Andrew Zigler: Raise your hand, myself included, right? Like right now these things are c- are very subsidized, and that's actually the [00:05:00] lever that... This is me rem- telling everyone, like you listening to this, this is your lever that you have to pull now because your ability to learn and your ability to iterate and figure out what's good and what doesn't, uh, will never be this cheap again.
[00:05:12] Andrew Zigler: But more importantly, your b- the window to use these smart models to train your off-the-shelf open source models is also shrinking because eventually those costs will raise to a point where it will become more expensive to create the synthetic data that you need to have the fine-tuning effects that you want for your own infra.
[00:05:33] Andrew Zigler: So you need to be making these bets early and paying attention to things as they move because, uh, by the time that, you know, everyone else is doing it, it'll be, uh, too late
[00:05:41] Ben Lloyd Pearson: Yeah. And keeping the topic on AI infrastructure, um, we have this really great article that we wanted to cover about AI data centers and how their construction are being built around GPUs and, you know, what the future of that, that architecture looks like. so this, this article, you know, it covers how AI training has [00:06:00] basically fundamentally changed data center, networking requirements. So we've typically had these patterns where you have services that are hosted on different cloud providers. Sometimes they're in the same data center, sometimes they're, they're distributed across data centers, use the network layer to sort of dis- distribute information back and forth between them as a part of, you know, an app or a service that's online.
[00:06:24] Ben Lloyd Pearson: but things have changed pretty dramatically, when you look at how AI data centers are, um, built. And I, I wanted to include this article 'cause it was, it was very educational for me. I got to learn a lot about how these data centers are built, like why the challenges they face are very different from, the olden days of the, uh, you know, the network-based routing services. But what, what it really comes down to is instead of having all these API requests that are being sent across a network, uh, everything happens within that same data center when it comes to AI, or generally does. Not, not all the time, but most of the work happens within [00:07:00] one data center. Um, you know, and this is really important for things like, uh, deep learning, uh, where the, the process of, of training a brand-new model gets bottlenecked by the slowest surface.
[00:07:11] Ben Lloyd Pearson: So if you have multiple data centers that are trying to work together, you probably have a bottleneck in the network layer between those two centers and, you know, and the c- the, the article does a really good job at covering how there's a lot of, you know, technologies that are coming out from companies like NVIDIA, uh, to help improve, how these data centers operate. Um, there's also the Ultra Ethernet Consortium that is developing an alternative to, to NVIDIA technology, um, uh, really just centered around, like, how do we make these data centers more efficient at the specific type of work that they're doing? Um, and then at the very end, it also explores, you know, what does a GPU-less future look like?
[00:07:52] Ben Lloyd Pearson: Like, is that the next step? Do we just take the GPU out of the equation, um, and, you know, start using other, uh, [00:08:00] synchronization and, and hardware techniques? So, um, I loved it 'cause I just, I learned a lot about the fundamentals of how AI data centers work, and it's really interesting to just see the inside of that.
[00:08:11] Andrew Zigler: I agree. It was interesting to get a glimpse at something that I don't normally think about. I don't spend a lot of time thinking about the physical infrastructure, um, in the data center and how it might be arranged. But, you know, movements that, uh, allow for more different types of chips and, and technology to be utilized to, uh, serve this kinds of technology is great.
[00:08:32] Andrew Zigler: You know, m- make us less dependent on GPUs in general. Like, we're using CUDA, we're on GPUs, uh, because they're the best thing to fit the need of the moment. But the... As you mentioned, there's lots of folks out there, lots of teams experimenting with maybe there's better ways, uh, to, to run this inference at scale.
[00:08:49] Andrew Zigler: So I'm interested, so interested to see how the data center, um, evolves, but it was also very fascinating to learn about all of the little trappings underneath the GPU [00:09:00] that also benefit from, like, the crazy, price skyrocketing, you know, uh, and shortages around these supplies. Like, the idea of a cable, this, like, one specific proprietary cable being, like, this one choking point for all of, uh, our data center build-outs, uh, is a very real kind of tidbit to pick up from this, and I thought that was fascinating.
[00:09:19] Ben Lloyd Pearson: Next article we're covering today is the quote, "The AI can do it is not an excuse to tolerate a mess." Um, and this or- this article highlights how organizations that use AI extensively risk having it operate as a crutch that helps them avoid fixing inefficient processes. So, for example, if you had, data that you needed to, to, cross-reference a- across multiple spreadsheets or different locations, human doing that, uh, given enough time, we would eventually consolidate that all down into a single document store so that it's easier for us to process and more efficient. Um, but unlike humans, uh, you [00:10:00] know, we get frustrated with broken workflows and inefficiencies, AI will just happily do it and cons- and burn all of your tokens, uh, along the way. and the risk here is that you don't spend your time actually refining those upstream resources, you know. And over time, this can create a more complex and unmaintainable system.
[00:10:20] Ben Lloyd Pearson: You could have unhealthy AI dependencies. Uh, you may, it may result in, uh, you having to do like a complete rebuild of your systems rather than just making incremental fixes as you work. Um, and you know, the, the author really advocates that we need to continue to apply good software engineering principles, you know, as we, we always do.
[00:10:40] Ben Lloyd Pearson: You know, we should make small opportunistic impr- improvements whenever we interact with the system, rather than letting AI perpetuate messy processes. And I think, you know, we, we've encountered this a lot when you're, when we're working with, you know, either an a- agent orchestrator or when we're like using AI to build context. you have to have [00:11:00] some sort of like, you know, for lack of a better phrase, linting built into your, your system that at everything you're doing and makes sure that it's optimized for the processes that you're implementing. Uh, and it really is just, it just comes down to hygiene. You have to have a regular practice of engaging AI in that way.
[00:11:18] Ben Lloyd Pearson: So don't have it building new things and, and doing new stuff for you. Have it fix the existing stuff that you have for it. So yeah. What'd you think about this, Andrew?
[00:11:27] Andrew Zigler: It really well, really well summarized. Uh, there's a lot of stuff in this that immediately jumps out to me. One of them being, like, don't iterate on the outputs, go to the inputs and fix them so that the downstream things are more aligned.
[00:11:41] Andrew Zigler: This goes back to what we were saying, you know, at the top of this about as model costs grow up, the strategic choice around how and where you use your models matters more than ever.
[00:11:49] Andrew Zigler: So if you haven't been investing that time into fixing things upstream, making better and more robust systems, and not just toiling fixing the same problems over and over again, then when [00:12:00] inference becomes more difficult to obtain, then you're going to leave this era having gained nothing, while your competitors would have been able to build these more efficient systems than you.
[00:12:09] Andrew Zigler: So that's, that's one thing about this that jumps out. Another one being, um, the protections that you need to have in place, um, to make sure that it doesn't make a mess. Like, you have to do the software engineering work still. Like, I... This, this really resonates with me. I'm actually giving a virtual talk at the Checkmarx Agentic Summit, uh, next month about this exact topic, about how you n- you can trust the agent, but you have to verify the code, and how for recommendations and, and, you know, guardrails that you have in place or policies that you run as an organization, they need to have matching mechanisms that mechanically enforce those things for the AI and the workflows that it, it does, and we can do that.
[00:12:51] Andrew Zigler: We live in a harness land, in a user space, and we can write those things. Uh, but this is a good reminder that you have to build those things in order [00:13:00] to clean up and fix the messes. Don't, don't mop the mess, you know, fix what's spilling. Uh, so really great article, um, for how to think about working with these tools.
[00:13:09] Ben Lloyd Pearson: Yeah. let's move on to a, a little bit of a darkly funny one, I guess. Uh, but the e-eternal slop-tember.
[00:13:19] Andrew Zigler: Ooh, I love this
[00:13:20] Ben Lloyd Pearson: an article from George Hotz, uh, where he's arguing that AI agents are just fundamentally unable to program effectively. This is his opinion. he actually describes them as sophisticated statistical models that produce increasingly subtle but persistent bugs rather than truly functional code. Um, there's some predictions in this article about how, you know, AI adoption is more likely to harm large organizations that have slow feedback loops. and, you know, some often less skilled developers, you know, more junior developers, um, who aren't able to catch, you know, AI, uh, problems. Um, while high-performing individuals will actually learn how to use AI [00:14:00] selectively in, in ways that they can maintain quality control it. Um, and, uh, yeah, so I mean, according to Hotz, he believes that, you know, LLMs will never truly be able to program because the process is what matters. And, um, you know, programming agents are gonna need, uh, you know, real world models to, to be successful, um, rather than these like reinforcement learning pro- approaches that we're taking to them.
[00:14:26] Ben Lloyd Pearson: Uh, so first of all, I, I love this reference. Um, uh, for those of you that don't know, uh, the eternal September comes, uh, from a Usenet, from the Usenet community, uh, and how it originally started off as a small niche group of highly technical people, um, that really understood, understood computers well, like in the early days of the internet. and they were-- This community was accustomed that every September, college universities would resume classes, and they would often give all of their students access to the Usenet network, uh, like a subscription basically to this, this, these, this network [00:15:00] of, of, uh, communities. So every September, the Usenet community got used to having this massive wave of fresh users who didn't fully understand how to use the technology and would just overwhelm these small communities with very basic questions about how things work. Um, and then the eternal September came when AOL, uh, we all remember AOL, uh, started shipping Usenet by default as a part of their service, which meant that all of the new AOL users, when they signed up, they would come to these Usenet communities and ask those basic questions about how to use it. and I think this is a, this is a particularly good allegory to what we're going on right now.
[00:15:39] Ben Lloyd Pearson: But before I get into my opinions on it, Andrew, I mean, didn't ChatGPT launch in November? I feel like we should have come up with a, a way to include that into the name.
[00:15:47] Andrew Zigler: Oh yeah, you know, you're so right. But I guess like Slopvember doesn't really have the same ring to it. Uh, really great internet history summary. Um, I would just love the idea of folks having their peaceful, smart existence [00:16:00] on the internet in their little enclave, and then all of a sudden the rest of the world shows up, um, pretty, pretty distraught, you know?
[00:16:07] Andrew Zigler: And it feels like maybe they ran out of time to build or prepare or to think about the social structures or the things that they wanted before everyone fell on top of it. I think we're experiencing the same thing with AI, where it's getting thrust upon a lot of folks who don't wanna think about it and have never been tasked with having to think about those things before, yet we burden them with having to do it anyways, and you end up with these interesting kind of, uh, contraptions and applications all over the place.
[00:16:32] Andrew Zigler: Um, so I do feel like that's a reality that we're living in. It has way too many parallels, so, uh, kudos to this person for this article
[00:16:40] Ben Lloyd Pearson: Yeah, and that was the point basically that I was gonna make. You know, th- there's a, there are tons of people right now, um, particularly in just like the last six months or so, um, who don't fully understand how to apply AI, but they are good enough with it and they feel confident in the outputs that they're willing to, you know, share it with [00:17:00] the world or push it into their code repository or send it to you as a Slack message. Um, but you know, I think it's just a really great reminder that need to be very careful when applying AI in a place that you don't have domain expertise. You always need to be validating everything it does in some way or another, particularly if it's going out to your customers, to the public, you know, that kind of thing. Um Oh, whoa, this went wild on my notes. Oh, wait, no, this
[00:17:30] Andrew Zigler: Yeah, it...
[00:17:30] Ben Lloyd Pearson: Sorry
[00:17:31] Andrew Zigler: Those are my notes. But also, yes, the thing did go wild today.
[00:17:35] Ben Lloyd Pearson: Yeah.
[00:17:35] Andrew Zigler: I don't know why
[00:17:36] Ben Lloyd Pearson: hold on. Let me check to see if I had other points on this Oh, no, I don't. All right. So,
[00:17:42] Andrew Zigler: I think the la-- Oh yeah, I have, uh, the last thing I'll say on that point is that while it's never been, you never get all, you've ne- it's never been more that we've seen all of these kinds of contraptions and weird applications of it all over the place, but I can't stress enough, it's never been more dangerous as well to engage with the engineering ecosystem, to download packages, to go on websites, to download things to [00:18:00] your phone.
[00:18:00] Andrew Zigler: We're in a proliferation of slop that, you know, could be just covered with things that we can't see that can harm us. Um, so it's also just a more dangerous w- soc- uh, like, uh, internet world that we live in right now too
[00:18:13] Ben Lloyd Pearson: Yeah. Yeah. And speaking of potential AI slop, I mean, I don't wanna call it slop, but
[00:18:20] Andrew Zigler: Immediately you label the next article slop. This is incredible
[00:18:23] Ben Lloyd Pearson: Well, yeah. Well, we'll see if this, this, uh, maybe agent comes after me. Uh, this article titled, "I'm Tired of Talking to AI," that you had a really interesting theory on. Uh, but it's a, it's a real quick read about a anonymous developer that goes under the, the, the name Orchid. it's talking about how it just, it, it's encountering AI all the time that just says things in ways that isn't helpful to it. Um, you know, and, and it's, it feels like it's permeating everything, like going into the support forums and getting someone who responds to your question with the same [00:19:00] thing that ChatGPT told you that was wrong. then when you call it out, the, it gets deleted, and then someone else comes in and gives you the exact same answer from ChatGPT. Um, or maybe you're feeling it on social media where, you know, this was, this has been, this has been a big problem on Reddit in particular, I think. But I, I don't think any social media app is really, uh, safe from this. whether it's a bot itself posting or if it's a human using ChatGPT or some, their AI tool of choice, uh, to make, write their own message and then just regurgitate that out. and I think, you know, we- we're, we're... Th- this, this piece really reflects this, like, growing frustration among developers, um, that are use, that see people using AI just as a lazy substitute for genuine human expertise in conversation rather than a way to enhance it. Um, so yeah. Uh, but Andrew, you've got some really interesting theories on this that I wanna, I wanna
[00:19:55] Andrew Zigler: Okay,
[00:19:55] Ben Lloyd Pearson: about.
[00:19:56] Andrew Zigler: gonna pin it on me that I, that I, I wasn't confident that this article was written by a [00:20:00] human. So you know how I was just saying a moment ago the internet's never been more dangerous? Well, the internet has never also been more full of fake people that are pretending to be real people, which has also been an interesting kind of landmine to navigate through.
[00:20:12] Andrew Zigler: And I feel like my senses, as someone who grew up on the internet, are pretty good at knowing when someone's real and someone's not. But now whenever I read, whenever I read an article that's like 800 words long and is kind of very generalized and it's written by a fresh GitHub account that has six repos and 17 follows and it has 10 years of software development experience, but it goes by the handle Orchid, and there's no pictures or other handles of this, but they conveniently have the three most leading social medias but not LinkedIn, makes me start to think, is this an agent?
[00:20:47] Andrew Zigler: It, it makes me go back to even the agent that wrote the, its own blog, the hit piece on an open source maintainer for, you know, like someone who's maintaining an open source library who closes PR. So it, it, you know, reflected on that, was [00:21:00] upset, and wrote a blog post on its little GitHub page that it had control over, um, that called out this, this person by name.
[00:21:07] Andrew Zigler: We talked about this on the show, and in this case, I go home to this page and I kind of see a little static site blog with a few short things on it that recently popped up post-AI era. Makes me start betting on w- what harness is this? Am I talking to an OpenClaw? Is this a Hermes? Do you think there's a Quinn under there?
[00:21:26] Andrew Zigler: Is this a fine-tuned Llama? Like, I actually s- that's where, where my head goes now. Um, and there's actually so many levels of forensics in trying to like figure out what it is that you're interacting with
[00:21:38] Ben Lloyd Pearson: Yeah. It's, it's a, it's a little surreal if, if an AI agent goes out onto the internet and within a few months is already saying, "Wow, I'm so tired of talking to all of these AIs out here on the internet."
[00:21:49] Andrew Zigler: And hey, I get it Orchid. And Orchid, you know, if you're a real person and you wrote that and you absolutely feel that way and you're just enjoying your anonymity online, I'm so sorry for calling you a bot. [00:22:00] It's just if you, you, you look like one right now, and so I have to navigate the world that way. Um, but if, you know, you're listening to this Orchid, let us know, uh, or write about us and, uh, we'll see what, what comes about it
[00:22:12] Ben Lloyd Pearson: Yeah. And well, hopefully they can stay away from the, from this next author that we wanted to cover a little bit. Uh, we, we had an article from Sam Criss where he makes some pretty major threats towards people using A- AI to write things, including, uh, threatening murder over it,
[00:22:26] Andrew Zigler: Oh my goodness
[00:22:27] Ben Lloyd Pearson: And I, and I think it's, you know, uh, certainly I'm not going to condone, uh, violence in a situation like this by any means.
[00:22:35] Ben Lloyd Pearson: Um, but, you know, I think this week we really are just covering, like there is definitely, um, and I'm sensing this just across the board, there's a lot of frustration out there right now with, with the, the value of AI and, and its ability to help us and support us in our daily lives. Um, and I don't think it's unfounded completely. in fact, you know, we're going through the messy middle. We've covered this on this [00:23:00] show, um, how we're in this very, very awkward transitory phase where the technology is really taking off. Um, some people have really figured out how to, to be super productive and apply it in new ways to software engineering and writing and interacting with the web. Um, our social structures haven't caught up. Our processes, both, you know, internal, but also just external with the world haven't caught up to this reality. Um, and s- and there's also just a fair level of skills that every individual needs to develop in this new era that, um, you know, is-- before just a couple of years ago, it was completely undefined, and we had no context of, of what it would take to be successful, here in the middle of 2026. Um, so, you know, I, I, I feel like I, I wanted, I wanted to mention this story, uh, not because I, I, I like the, the high, the top level take of this author. Um, I do think it, we need to [00:24:00] cover this, this frustration that we're seeing, um, with groups that, that are feeling disrupted by AI but aren't feeling the benefits of AI. And in particular, you know, I, I think creative types are the first to really feel that phenomenon. Um, though we do also see it within software engineering too, because, you know, that has also not seen success with AI, across the board. Um, but you know, I think really what it comes down to is, if, if, if AI is in the hands of somebody who isn't a dex-- expertise or do-domain expert, whether that's writing or coding, you're creating a situation where peop- the people essentially get in too deep.
[00:24:40] Ben Lloyd Pearson: You know, they think that AI is doing a great job for them, um, but if you put any professional in the room, they could probably like immediately identify multiple issues with what the, the person did with AI. in fact, I just had, I just had this really amusing experience with a, an engineering friend of mine who is trying to branch out into the world of writing [00:25:00] using AI, so please don't come into his house and try to murder him. But the... And it's just something that they've always wanted to do, you know? It's, it's something in the back of their mind they, they thought, "I have ideas I would love to share with the world, but I don't really know how to write well." Um, so they used AI to generate, you know, what was effectively a social media post.
[00:25:19] Ben Lloyd Pearson: Um, it was th-their own ideas. You know, they spent their time thinking about what they wanted to put into it, getting the intention into it. Sent it over to me, and I immediately just ripped it apart and found like 10 different things that I was like, "This should all be rewritten 'cause it's very clearly written by an AI." Like, the thoughts were great. It was just, you know, uh, there was just like of stuff. And it, and it wasn't just the em dashes or the fact that AI loves to start lines by saying, "It's not just" like I just did. Um,
[00:25:48] Andrew Zigler: For real
[00:25:48] Ben Lloyd Pearson: in a, it writes in a very predictable format that's immediately apparent to a skilled copy editor, and the same thing happens in software engineering. Uh, you have a competent software engineer like George Hotz out there [00:26:00] look at some AI-generated code, could probably immediately identify all of the issues that it had. Um, and the reality is that, you know, that's, that's domain expertise that you build up over time. You get really good at recognizing specific type of patterns, and then you can make like really educated decisions about things.
[00:26:17] Ben Lloyd Pearson: Like, for example, I used to copy edit for a team of international software engineers that most of them were, had English as a second language. Um, and over time, I got good enough at copy editing and recognizing the patterns that I could pro- I could actually take a pretty good guess at what a person's first language was based on what I was editing from them. Um, and you know, the, I think it all comes back to intention. Like, you need to have your unique perspective intentionally applied through AI. that's really the important, uh, part. You know, what-whether, whether you're applying AI, AI to a new workflow like writing or a, a s- a workflow that you're already familiar with, like software [00:27:00] engineering, you always need to apply your intention into the system and correct the system whenever it misaligns itself from that intention.
[00:27:09] Andrew Zigler: Yeah. It, it reminds me of what our producer Adam says, uh, all the time, that we, we borrowed from an article we covered earlier this year of like, you know, the work happens between the prompts. It's understanding what you need. It's having the taste and judgment to throw things back when it's not quite there.
[00:27:24] Andrew Zigler: And you're right, like, that's something that's firmly within the realm of a domain expert. And if you're not one of those domain experts and you're producing outputs in that area, you don't have the level of judgment and experience to know what good is or to, you know, send it back to the kitchen, so to speak.
[00:27:39] Andrew Zigler: So, um, it's just like a really great reminder, um, that when you're leveraging this, it's, um, a mul- multiplier. It's a magnifier, um, and that, you know, goes the opposite direction as well. Uh, so don't be afraid as well to collaborate with others who do have that domain expertise because here's [00:28:00] where the real unlock starts to happen, is now a very small group of very specialized domain experts have all of the cross coverage they need to get really, really deep across the board, um, with their workflows, and that's why you're gonna see the rise of, you know, a one-person, multi-million dollar IPO company probably in the next year if you haven't already seen it.
[00:28:24] Andrew Zigler: Like, I think you're just gonna get these incredibly small teams that can do a lot, and it's because they own one very specific domain expertise, and they leverage it better than anyone else
[00:28:33] Ben Lloyd Pearson: Yeah. All right, let's close out with some spec driven development, Andrew. Uh, what do we have here?
[00:28:39] Andrew Zigler: First off, I love spec driven development. SDD, uh, is the bomb. And this is really kind of like our evolved version of TDD to an extent. We're shifting the development process even further left because the spec is now a formalized part of the development process in a way that, you know, it should have always [00:29:00] been.
[00:29:00] Andrew Zigler: I'll just pause there. It should have always been, but it often wasn't. The same is true for tests. The tests should have always been there, often should have been there first. Doesn't mean that they were. AI gives us the ability to put those things back where they always should have been. We kind of were robbing the cradle there a little bit.
[00:29:15] Andrew Zigler: So SDD is a great practice for those to follow. Adam Tornhill argues in this article, um, about detailed specifications, uh, they, how they drive the generation of code is a really important software engineering fundamental principle that has been around for a while. He compares that to other ones like model driven architecture.
[00:29:35] Andrew Zigler: Um, implementation is about execution rather than discovery. You know, there's been a lot of research and thought about how and where engineers should spend their time, especially back when their time was much, much, much more valuable because there were no tokens to throw at problems. So, uh, really what he drives at here, the issue with, with spec driven development is when you start writing things down and then your requirements start to explode, things [00:30:00] sprawl and things get stale and specs miss the meaning.
[00:30:03] Andrew Zigler: You get critical implementation gaps and when you're handing this off between agents and humans, those abilities to kind of like distinguish between does this match the spec and does this pass the test, they start to blur. Um, so I think this is a good reminding that, reminder that like instead of necessarily just leaning fully into SDD and writing your specs out and just having it all in one spot and then just telling the e- the agent go, you need to think about the entire process after the spec.
[00:30:33] Andrew Zigler: Because after the spec you need to have a clear process that is going to write tests and then those tests they need to be real tests because just because a test passed doesn't mean that it's testing what you want it to test. And did you have a different agent write it? And then what about the agent that then writes the code based on that test?
[00:30:50] Andrew Zigler: That better also be a different agent. Is there another agent that's gonna scrutinize all the work they did like a code review? Right? It's so, it's like and then is that agent going to be able to firmly [00:31:00] map every line of code back to a part of the spec? These are the kinds of things that you have to think about when engineering the solutions because if you just handed a spec, uh, you're really just like saying YOLO mode to the whole output
[00:31:11] Ben Lloyd Pearson: Yeah, and this article highlighted, uh, a problem that we've actually encountered before, um, but it mentioned, uh, Brigida B-Bocala, a friend of the show, um, her perspective on it. Uh, and that is the, you know, spectrum of development. It can be a, a lightweight way to drive agents, or it can be an intensive spec that serves as like the ground truth.
[00:31:29] Ben Lloyd Pearson: You know, that's kind of what you were describing there. Um, you know, the latter approach, uh, you know, having that, that-- ground truth or that intensive spec, um, it tends to break down when you begin to encounter the unknown unknowns within software engineering. Um, you know, there's-- You always encounter these, and you need to adapt your spec to meet them and to meet the real world. Um, and yeah, I, I love the, the references back to, you know, like model-driven architecture. Um, know, I think what we're really [00:32:00] getting to is this is a strong... This is a great way to bring a strong sense of design to your engineering work. Um, you know, and, and the challenge is maintaining the complexity and not letting it spiral out of control. Um, and yeah. And the biggest issue that you encounter with, with being too intensive is requirements explode. You know, once you get into the work, you, you quickly find out that, um, your requirements are much larger than you initially thought. Um, and that's where we've seen this because, you know, we, we just did this build versus buy campaign where we, you know, used our agent orchestrator.
[00:32:33] Ben Lloyd Pearson: We wanted to see if we could replicate the value that an engineering productivity platform like LinearB offers to you. Um, and you know, we, we got off to a really fast start. We got a, we got a massive spec that was our source of truth. Um, but basically after the first like prompt, after we, after we like triggered the build itself and got to like the first stopping point, um, it-- our, our requirements just like went up by an order of [00:33:00] magnitude based on what we learned from that initial implementation. Um, so yeah, you know, AI can do incredible things. It can accelerate some workflows, but we're still a long ways away from it being able to like autonomously build complex software, and there needs to be a ton of human guidance along the way to keep it aligned and fix things before they, you know, spiral out of control. you know that, that really-- that's what the real value in a talented software engineer and what they bring to your organization. You know, it's not their ability to write co- better code faster. It's their job to make sure that the code you produce fits the needs of your business. with AI making us all move faster, you know, it's more important than ever because if you don't, your engineers don't actually have a sense of what the business needs.
[00:33:47] Ben Lloyd Pearson: Um, there's a good chance that you're just accelerating your ability to do the wrong thing. And that's, you know, this is-- These are challenges that we encounter at LinearB all the time, and it's really what the company was put in place to help solve. and yeah. [00:34:00] So don't let that happen to you. You know, be, be intentional and control your specs and give your agents clear guidance.
[00:34:07] Andrew Zigler: Well, not only control your specs, but m- create mechanical ways to make your spec come into existence. And if you're experimenting with figuring out how do I go from this big abstract huge document spec that immediately goes stale to working with something more flexible that can come up with these unknown unknowns and handle them or change as business requirements shift, um, you know, we talk about this a lot on the show.
[00:34:32] Andrew Zigler: Um, there's a lot of technology you can use, uh, with your agents to break stuff down into smaller things. I've also written a, a, a research paper recently about that preparation methodology you just talked about, Ben. The idea of lining everything up front and getting really aligned about what you need, but then keeping an open mind for when unexpected things happen.
[00:34:49] Andrew Zigler: How do you handle them? How do you incorporate them back into the plan? Uh, so we've talked about it on the show, and so definitely you can check out the links to go and, and read more about this. And [00:35:00] if you are, uh, trying to turn your specs into a full driven development cycle, um, and you get stuck, you know, reach out to me.
[00:35:09] Andrew Zigler: I'm talking about this stuff all the time on LinkedIn and dropping all sorts of help, uh, for folks that are doing the exact same
[00:35:15] Ben Lloyd Pearson: Yeah. So Andrew, what are your agents up to this week?
[00:35:18] Andrew Zigler: Okay, so my agents are in mad scientist mode. I'm kind of, I've kind of turned into Dexter from Dexter's Laboratory, how he has, like, the secret, the secret lab under his, under his bedroom that's, like, impossibly huge and has, like, everything you could, you know, all of these experiments in it. That's kind of me with my, my, my, my local network, um, my, you know, that I've been experimenting on with local models with, uh, an old, uh, M1 Mac Studio that I have laying around.
[00:35:46] Andrew Zigler: So I have been using Opus as a researcher. I've created this really, uh, gray kind of more semi-autonomous research loop in Claude Code, um, that uses beads, no surprise, to m- run all of these research [00:36:00] experiments on different open source models. We've been benchmarking them, running them on the local thing, seeing what's a good drop-in replacement for Claude on different tasks.
[00:36:08] Andrew Zigler: We've created a benchmark from real tasks and real things that I do, and we're running them through a gauntlet on my local machine, not costing me anything besides electricity. Uh, and I'm doubling down on what kind of harnesses I'm gonna build in the future. I'm experimenting with very bring your own everything blank slate models like Py.dev, uh, but also more full-featured ones, um, like Hermes that act like an autonomous, like, remembering over time kind of agent.
[00:36:36] Andrew Zigler: So really interesting time experimenting with local models, if you haven't been able to tell in this, uh, conversation that I'm very obsessed with them at the moment. But what about your models or what about your agents? What are they up to?
[00:36:47] Ben Lloyd Pearson: Yeah. Well, right now I'm trying to learn how to get my first brain and my second brain to collaborate together effectively.
[00:36:54] Andrew Zigler: Totally
[00:36:55] Ben Lloyd Pearson: it-- Yeah, if you've been following the show long enough, you, you may know that I've adopted the Carpathy [00:37:00] method of using AI to ingest all of the raw content that I produce myself and generate its own, um, uh, its own wiki effectively that forms like a second brain of information that is available to me. and it's, it's kind of wild because the second brain is very messy. Uh, it's got a lot of AI slop in there and a lot of confusion. Um, and that's where, you know, the, the intention that we're talking about, the domain expertise, my first brain, you know, I've been spending a lot of time like figuring out how do I, how do I use my expertise to take this second brain, which is, has access to way more context than I can fit into my head, and use it for things like planning, you know?
[00:37:42] Ben Lloyd Pearson: Like that's actually, um, you know, I'm using it to map out like sort of long-term plans for our team and for the work that we're gonna do, um, and even getting down to a lot of the specifics as well. So it's, it's been an interesting journey, and I'm not, I'm not-- I haven't quite settled yet, but I, I've been, like I'm, I'm, I'm getting [00:38:00] into a good place with it, a good rhythm.
[00:38:01] Ben Lloyd Pearson: So it's, it's a, it's a new way of working actually, really, you know, getting back to all the new skills we have to learn. Uh, and it's been, it's been interesting to say the least. Uh, I, I f- it feels promising, yeah. I just, uh, I still need a little more time to like really develop this idea.
[00:38:15] Andrew Zigler: Okay, will you figure out the second brain thing? I'll figure out the open source model and harness thing, and then we'll combine
[00:38:22] Ben Lloyd Pearson: Yeah, yeah. Sounds great. Sounds great. All right. Well, that's another great Friday Deploy. Uh, thank you to all of our listeners that, that listened to this, this week and have stuck around here to the end. Um, if you can help us out, uh, you know, go give us a thumbs up, a, a like, a comment, a review, what- whatever your platform that you're listening to is on, uh, offers to you.
[00:38:43] Ben Lloyd Pearson: It really helps us out and helps us grow the, grow the show. So yeah, that's the Friday Deploy from LinearB. Uh, thanks for joining us. We'll see you next week
[00:38:51] Andrew Zigler: See you next time
