Does it feel like your favorite AI tool is declared dead one week, only to be resurrected the next? This week, Andrew sits down with Bryan Bischof, Head of AI at Theory Ventures, to explore the hidden levers of inference systems and the industry's obsession with prematurely writing off useful tools. Bryan shares his experiences with why prompt optimization is mostly a dead end, the secret to building high-performing data agents, and how his team builds operational software for VCs. The two also break down the origins of the satirical rip-grep.com and drop hints about Bryan's highly anticipated next AI game show experiment.
AI Council 2026: Catch Bryan's track on inference systems and get tickets.
Show Notes
- Events:Catch Bryan's track live at theAI Council Conference.
- Satire & Writing:Check out the April Fools manifesto atrip-grep.com
- Firm:Learn more about the work happening atTheory Ventures.
- Connect with Bryan
- X/Twitter:@BEBischof
- LinkedIn:Bryan Bischof
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: Today I'm joined by Bryan Bischof, the head of AI at Theory Ventures. Bryan has a PhD in Pure Mathematics. He teaches AI and data science at Rutgers, and he's built teams and lead projects at places like Hex, Weights and Biases, Blue Bottle and Stitch Fix.
[00:00:16] Andrew Zigler: And Bryan, I've really been looking forward to having you. On our show because I got to see your work in action just last year at your America's Next Top Modeler hackathon, uh, which is an amazing name. We're gonna dive a little bit more into the things behind it as well. It was an amazing time, uh, and I learned a lot that's really in influenced my own personal journey and the things that we've talked about.
[00:00:39] Andrew Zigler: On this show, so it's really great to have a mind in the mix behind all of that. That kicked it off, and we're gonna get into some of your work at Theory Ventures as well. Uh, but first of course, we have to talk about what's happening right now at the AI Council 2026 conference, where you and I are both on site.
[00:00:58] Andrew Zigler: And Bryan, I'm thrilled to have you [00:01:00] here. Congrats again on being named a track chair as well at the conference.
[00:01:04] Bryan Bischof: Thank you. Thank you. Yeah. Uh, I'm really excited about AI Council this year, or FK Data Council. Um, some of us still think of it as data council. Let's just be
[00:01:14] Andrew Zigler: Totally.
[00:01:14] Bryan Bischof: Um, this reminds me of an interesting debate that I was just like. Is it AI and data? Or data and ai? I think it's a really interesting like, you know,
[00:01:23] Andrew Zigler: Hmm.
[00:01:24] Bryan Bischof: question.
[00:01:25] Andrew Zigler: I think you gotta say AI and data that kinda rolls off the tongue a little better, doesn't it?
[00:01:30] Bryan Bischof: I think that's probably right. AI and data. Data and I, I, I can't
[00:01:35] Andrew Zigler: You can't. You can't even say it. You can't even say it.
[00:01:37] Bryan Bischof: the answer is AI and data with, I can't speak the other one out loud. If the utterance is unutterable, then the decisions have been made.
[00:01:45] Andrew Zigler: It is just not gonna catch on.
[00:01:47] Bryan Bischof: amazing conference of data and no AI and data. Um, yeah, I'm really excited. Um, Pete likes to throw me a challenge, and this year his challenge was inference systems. [00:02:00] Last year it was foundation models, but his question to me was, I'd like to put together something about inference systems have fun and um, like the challenge, but I also, every time he throws down one of these gauntlets, I find myself thinking like. I don't know anything about this. um, that begins the journey. Um, and so I research it, sort of like how I would research an article or how I would research like a book topic. And I basically just like dig in to start trying to learn things. In this case, way that I thought about it is not so much like I wanna learn all the things about inference systems during this research, but what are the things that I would try to learn if I were to try to learn this topic anew? Then what I do is I basically jot down like, okay, these are the major holes in my understanding. then I just ask some of my friends who are very smart and very knowledgeable, how much they know. And I try to figure out [00:03:00] like, am I the only dumb one in this particular area? And for many of these, the answer is yes.
[00:03:04] Bryan Bischof: Everyone else seems to know except for me. And in other ones it's kinda like, nah, I don't really know much about that. Or like, ah, you know, I like kind of understand how that works, but not in like a deep way. And so one of the benefits of this approach is because I have all these like smart friends that I can ask, I kind of naturally get this exposure to what is known, is known by like basically everybody.
[00:03:27] Bryan Bischof: So like I know it. And then what is known by sort of like the, the really strong people, but not experts in those areas. And then I find the things that are missing from that. And I say, okay, now I need to go and hunt the people for that. And so that's basically how I engineered both of the tracks the last two years. And so this year it was very much like a lot of people had questions of what does it really mean to optimize at the inference layer? What does it really mean to like think about cost in a more sort of like holistic way. [00:04:00] exactly are we doing when we are sort of choosing an optimizer for training? Things like that. These are questions that a lot of people felt like they had like a little bit of knowledge, but not as deep as they wanted. And so then I went and I hunted down the best experts I could find.
[00:04:18] Andrew Zigler: It's really key insight on kind of how you think about organizing, um, a track and, and all of the talks within it is treating it as if it's like a high level research project. Exactly. You have a thesis that you're exploring. You're trying to plug the gaps. You have a, a wide and educated network that you can draw from as somebody in this vantage.
[00:04:37] Andrew Zigler: And in doing so, you start to surface these obvious gaps that there's a lot of smart people in the industry and that, you know, and, and that are really dedicated to the problem, seem to have. And then you get the shift into this like, uh, uh, information headhunter. You're like, who is this out here in the world who could teach us all about this gap And help can make this [00:05:00] puzzle a little more complete.
[00:05:02] Andrew Zigler: And I think like in such a, a frontier kind of environment as AI where there's just so much innovation, there's so much, uh, opportunity to learn every single day in every kind of thing that it touches that there's a lot of gaps. So I think like there's so many pitfalls and so many obvious. Uh, things that, uh, a track chair or somebody in that position could fall into of like, oh, I have a topic.
[00:05:23] Andrew Zigler: I'm gonna create a narrative instead. You, flip that on its face. I'm not gonna tell a narrative. I'm going to get, uh, the breadth of this field and I'm gonna find these pitfalls, these these hidden dangers that we all need to be more educated about. And then I'm gonna bring us all together to learn about them.
[00:05:37] Andrew Zigler: And, and so, you know, what do you think ultimately is the story that's emerged from those gaps that you found in the people that you've lined up to teach us?
[00:05:45] Bryan Bischof: Yeah, so I would say two things here. So like, one, what is the point of going to a conference? Like, I, I've gone to a lot of bad conferences over the years. Um, one's where I found myself wondering like, why am I here? [00:06:00] It sounded like a good idea and I, I don't know if it was a good idea to even like, spend this day doing this thing.
[00:06:06] Bryan Bischof: I saw some people I like, but like, man, is it just a social activity? And that's fine if that's like why you're gonna conferences. Like respect. I get that. I went to a conference in like, um, Denmark, a really long time ago that was basically just a social activity. Um, but like what I'd like conferences to be is educational.
[00:06:26] Andrew Zigler: Mm.
[00:06:26] Bryan Bischof: And from that perspective, I was like, okay, you know, like here's how I wanna make sure that everybody has something to go home with. To answer your question about sort of like what kind of narrative did I find myself in? I think it's exactly your point where I was not trying to construct a narrative, but I did find myself asking sort of like, what is this like structural thread that pulls things together it's kind of coalescing into this idea that. We sometimes talk about the cost of intelligence chart. my friend Charles Fry, he made this like amazing [00:07:00] graphic a couple years back of like the cost of intelligence chart. And it's basically sort of like how well the models are doing on one particular famous benchmark, which is now basically like oversaturated and dead, versus the cost of tokens or sort of like how achievable is a certain level of intelligence at a certain time, there's been 18 billion remixes on this chart. Like by now, it feels very easy to find the one that you remember in your head, and you're probably picturing one right now that you've seen most recently. It's probably not Charles's. Um, but Charles, as far as I know, did the first one, and this one has been really influential for a lot of people. And the reason I bring it up is because Charles plot essentially has three dimensions. It's plotted in two dimensions to put on your screen, but it's effectively representing three. those are basically sort of like the metrics that everyone is optimizing to. [00:08:00] It's the thing that you're trying to produce on the other side of the function, but I've been finding myself thinking more and more as a mathematician. Okay. There's the, like the target of this function, or there's the sort of like, know, the, the, the range of this function where we start getting observations. What about the domain of these functions? Like what do we put in and what do we change? You sometimes hear about this in sort of like data science and business ops as levers, like what kinda levers can you pull? Also, I don't know if your audience says levers or levers. I've gotten like criticized aggressively for calling them levers, so I'm being very conservative by calling 'em a lever.
[00:08:42] Andrew Zigler: You covered your bases.
[00:08:44] Bryan Bischof: Um, so, so these levers that you can pull, so one lever that sometimes people pull is they're like, buy big GPU. That's great. That's totally reasonable. Another lever that people pull is sort of like the way that they connect that hardware or the way [00:09:00] that they optimize where that hardware lives. But then we, we hear about some like obvious things. We hear about things like quantization. Quantization is actually another lever that you can pull. You can take your model and quantize or reduce the precision of the numbers in the sort of weight matrix to make your model take up less memory. That is a thing that you can choose to do. There's some reasons why you would expect that to have an effect on the range of this function, but there's other things too. And there's actually quite a lot of these levers and some of them get talked about a lot. We get a lot of talk about optimization, but we don't get a lot of talk about optimization at large scale inference. And so as an example, you know, I think a lot of your listeners will be familiar with Adam. If you just hear the word Adam in the context of neural networks, they're probably like, oh, I know that's an optimizer, but what about shampoo? [00:10:00] And what about other ones? And so there's this other question around, okay, well what about optimization? At different stages of the pipeline you can optimize for serving inference. That's neat. But what about training inference? There's optimizations to do there too, and that also has an effect on these functions that we've been talking about. You can optimize things like sort of like what does your job look like? A lot of people are now familiar with KV caches. This is a thing that like now is a little bit like, you know, familiar for our everyday parlance and some people may know about sort of like. those KV values for subsequent conversations. They may know that when you ask Claude code 15 questions in a row, that some of those vectors during the sort of like forward propagation are stored and they're stuck there so that when you call it the second time, you're using less input tokens. That's neat, and I'm glad that people know that now. [00:11:00] But what if you have. 150,000 LLM calls that are all kind of similar and they only share part of their prefix. What about that? Can you do anything there? The answer is actually yes. And so I think these are the sorts of things that, like they're all hiding behind this,
[00:11:19] Andrew Zigler: Hmm.
[00:11:20] Bryan Bischof: this chart, this chart of Charles', Charles' chart and it's, it's getting into like a Nickelodeon TV show, but like, you know, uh, this, this amazing chart is just the sort of like the range of this function, but there's so many, like dimensions in the domain that people don't talk about. And so if I had to say like, now post hoc, what's my revisionist history of like, yeah, I'm real smart and here's my track theme, it's this. Um, and of course like, you know, like, I didn't think of this ahead of time.
[00:11:51] Bryan Bischof: This is stumbled into by the things that I didn't understand. And then my friends didn't understand as much as they wanted.
[00:11:57] Andrew Zigler: Right. And so [00:12:00] you're really calling to light the fact that like, not, uh, not all of the levers get their time on stage, get their limelight. Some of them are just way more sexy and talked about than others, and others are just still undiscovered because maybe they're less glamorous or they're more like an edge case.
[00:12:16] Andrew Zigler: And so you also dually saw your, uh, opportunity to bring people together and educate. Like you said, as a, as like a responsibility to shine a light on some of these less talked about less understood things because ultimately then you could start, then, you know, a whole new cycle of deeper study and investigation and discussion and, uh, ultimately, uh, instead of recycling ideas and new ways that we've already been talking about.
[00:12:43] Andrew Zigler: Going out and finding some of these new unknowns, which to your point, there are just so, so, so many of i, it's impossible really to go out right now and use these tools and not feel like everything you're doing is like some mini research project to an extent. You just feel so compelled to study and like [00:13:00] record everything you're doing because there is so much net new to learn.
[00:13:04] Andrew Zigler: So in in that, in that role, as someone curating that, that kind of education. And then we zoom out and, you know, we're at, we're at this really great conference. There's a lot of other tracks here where everyone is, is gonna come on site and learn about these very dedicated things. What are, uh, what's maybe like some of the other stuff at the conference that like, you've really got circled or like that's like the next big thing.
[00:13:24] Andrew Zigler: Anything that's like caught your attention, like what's on the horizon that we're all gonna be talking about that you think will echo beyond, you know, data council, AI Council as it is.
[00:13:34] Bryan Bischof: so. Last year I was a track host and a speaker, actually, it's kind of a weird coincidence, but I, I had a talk that I was really excited to give and that was called Failure as a Funnel, and that was talking about sort of like evaluation for data science agents. And as you know, I am hell bent on data agents and so, since last year where we did get some discussion of data agents at Data [00:14:00] council, there have been even more developments. We all, you know, in this crowd, we all know that like there was a step change in capability in the models, especially around coding. With that step change in the models, we didn't see quite the same step change for data agents, but we did see. Sort of like some of this, um, we could draft off of it a little bit, as you know, because a lot of data agents are ultimately coding agents wearing some, you know, additional armor so. What I am really keen to see this year and the sort of talks that I'm gonna make a point to go to are the ones that are talking about sort of like data agents and what's changing so rapidly in the area. the other sort of sub theme that I'm looking for is more and more people are coming to the sort of awareness that designing for agents is different than designing for humans. When I was at Hex, I used to tell people a lot. What's great for agents is great for [00:15:00] humans, and what's great for humans is great for agents.
[00:15:02] Bryan Bischof: And this was to remind people that, you know, AI is not a silver bullet or a magic sword. It is this thing that like you have to put work to make it great. And that was true for quite a while. That Maxim is still helpful to remind people put in work to give the agent great context like you would a human and you will have more success. But if I were to put a little like asterisk on that maxim that I used to wi like wield, the asterisk is okay, but also you can do some sneaky shit when you're designing for agents and that sneaky shit can make the agent have even more superpowers.
[00:15:48] Andrew Zigler: Say superpowers.
[00:15:49] Bryan Bischof: crazy shit that you would never expect a human to do. But if and only if you actually give it that capability. As a [00:16:00] quick example, you hear about search and you hear about search for agents. This is sort of like an emerging trend where people are starting to realize that the way humans search for things is quite different than the way agents search for things. can you start to. I, I sometimes call this ergonomics like agent ergonomics. How can you build the agent ergonomics in a way that the agent feels really, really natural to do this shit that you would never do but can really level things up? So that's a sort of another emerging trend that I am very excited about, and I'm specifically looking for talks that address that.
[00:16:36] Andrew Zigler: Yeah, the find those like secret little hidden things that we're discovering about how these like rocket fuel things that you can hand to the, to the agent to, to give it some superpower that maybe is a non-obvious or is something that a human, if it were given that same tool, would not be able to wield that same way.
[00:16:53] Andrew Zigler: I think that those are the biggest opportunities. What, what you've highlighted and like this, like almost like with this like devilish glee of [00:17:00] like, and if you give it this one small thing, it's just a whole night and day difference. I encounter that all the time. There's been things like, um, understanding how we as humans, um.
[00:17:10] Andrew Zigler: Make tasks and collect information in those tasks and then utilize and drag them around and understand them is so spatial and it's so burdensome in a way that it obstructs agents from being able to effectively work in that space with you. So it's like non-obvious that all, if you strip that back to its bares base forms and you make it something small and fast and light and predictable, deterministic for the LLM to use, they're great.
[00:17:35] Andrew Zigler: At being right there in that task management space with you, they just need to be given, like what you said, the right ergonomics and, and you're, and you're so right that like not everything that works for humans ultimately ends up transferring to them. And so there is a bit of research and discovery.
[00:17:50] Andrew Zigler: Like what I, what I'm talking about at the conference is, uh, about like from my background in teaching and like pedagogy method. Methods for having a classroom and creating environments where [00:18:00] students can learn in structured ways together and by themselves in a way where you ultimately remove and remove support and how this mirrors what's happening with harnesses, with agents, how we're creating these environments for them.
[00:18:11] Andrew Zigler: We're setting them up for success, giving them the tools to learn and to adapt and slowly taking away the support. And so, you know, there's a lot of. You know, there's hundreds of years of education research that can maybe be used as a platform to build what is ultimately finding these new non-obvious things for agents.
[00:18:27] Andrew Zigler: So, um, really great mindset. Yeah, I'm, I'm, I'm stoked to give it, it's gonna be a blast. Um, you know, obviously all of this gets up on YouTube so people can check out your, your whole track. They can, they can see what I talked about as well. Um, so I'm definitely excited to maybe shine some light on some new ways to work with agents.
[00:18:45] Andrew Zigler: I've definitely been doing some, uh, you know, uh, when I share it with folks like some, uh, non-traditional environments for how I get stuff done.
[00:18:52] Bryan Bischof: That's awesome. I just wrapped up teaching my AI engineering class at Rutgers, and this is a real [00:19:00] applied AI engineering class. And I can tell you if you ever want like a real adrenaline rush, try to teach a topic that's really only been around for two years and changes every month to graduate students that want jobs in that domain at a sort of like Normy R one University.
[00:19:17] Bryan Bischof: Have fun with that. Uh, it's a real, it's a real trip. We'll say that. Um, I
[00:19:23] Andrew Zigler: I bet.
[00:19:24] Bryan Bischof: Like rewrite lectures because like something new happened in the industry and I'm like, well, shit, like in two weeks I'm supposed to teach this and this is kind of outdated. Now allow me to update this. The the MCP slash tool lecture was,
[00:19:39] Andrew Zigler: even get us started here. How many times has MCP come alive and died? And we're gonna talk to be talking about that here in just a moment too. So I'm, I, I can only imagine. You know, we actually have a similar pain sometimes here on Dev Interrupted where I, we, we'll be like. Recording something for our Friday news segment.
[00:19:54] Andrew Zigler: Uh, and like literally while we're recording, like the biggest news of the week will drop and now suddenly we have to like [00:20:00] completely re-pivot what we're talking about. It's hard to think about that on an, on a even bigger magnified education scale. Like I have to equip this extremely smart and extremely capable person who probably knows more about this tool than I do to go out into a world that's changing faster than I can understand it.
[00:20:15] Andrew Zigler: Like talk about a challenge.
[00:20:17] Bryan Bischof: It is fun.
[00:20:20] Andrew Zigler: So let, let's get into a little bit about, you know, in the industry, the trends that are happening, the thing the currents under the water of, you know, what's happening right now, ai, uh, at AI council. And so, recently, I want to pivot to this piece that we talked about a bit on Dev Interrupted did, we covered it on a new segment.
[00:20:35] Andrew Zigler: Um, and, and you talked about like our software next. Epic, uh, our investment into rip grep. And it, it, you know, this, this is kinda like a, a, a part science part, part data science, part satire manifesto about the state of the industry and where things are going. And, you know, uh, we've, you've been talking about as well on LinkedIn pretty extensively about tracking the death of technologies and our industry and the way that we talk about them in the news cycle.[00:21:00]
[00:21:00] Andrew Zigler: You know, I, I really wanna ask you, Bryan, 'cause you feel like you're the expert on it, so. Why is it that in AI, a technology can die 12 times a single year, but still be the main topic of a conference?
[00:21:10] Bryan Bischof: Yeah, so I think ultimately, the. The blessing of AI's is that there is the likelihood that more people will benefit from these new changing technologies. It's not kept in the ivory tower. It's not locked behind an extreme price, um, moat, people like my mom and my sister might interact with AI and actually get benefit from it. The curse its visibility is that the tam is so large for attention that everyone wants a piece. [00:22:00] And so, one thing that I talked about, and this is a little bit on the spicy side, um, back in like 22, 23 and even a little bit into 2024, I was sort of early on this opinion that dev rel was killing software that like developer relations and specifically the craft around like attention optimization, and I don't mean Adam, um, was really like harming the industry. You see these like amazing examples of people like Kelsey Piper. Like everyone can look at Kelsey Piper and be like, he did so much for us and we owe him so, so deeply. then you can see sort of like a new generation of people at his level, like Charles Fry, who I mentioned earlier as like a standout case. Unfortunately, they're not all like that. And I, I, I think there's a lot of people that are focused [00:23:00] on rage bait and rage bait gets clicks. I think no one can deny that if you are an aspiring derell with low morals, I got a playbook for you. and that's rage bait. And so rip grep was meant to be rage bait, the rage baiters.
[00:23:16] Bryan Bischof: Um, that was very much the intention with that like piece, but it also did come from a very earnest position of curiosity, which is
[00:23:25] Andrew Zigler: Yeah.
[00:23:26] Bryan Bischof: What, like it seems like everything's dead all the time. Are things actually dead all the time? Like am I just in some filter bubble that like makes me feel like a myopic moron? Kind of. Yes. But also not only, yes. Um, there's also like a very real thing, as you can see in the data. Like there's a lot of people calling a lot of things dead. There are a lot of things that people call dead very rapidly and then fall off that. And you know, you made this call out to like dying 12 times in a [00:24:00] single year.
[00:24:00] Bryan Bischof: You know, whether that's a mathematical aberration or just a reflection of the fact that like, we do see these hype cycles and we, it, it, this is not new, like hype cycles are everywhere, pervasive, blah, blah, blah. I'm the last person on earth to be an expert on sort of social media influence. But the one thing I will call out is I do feel like it is hard for me when I see my students me questions in the middle of me presenting tools lecture we care about function calling anymore, because now coding agents can just use the CLI to do everything. Now you and I and a lot of your listeners are going to understand the nuance. We're going to know that when people say CLI are really good for agents and sandboxes, like we know that like, yes, the sandboxes can be valuable.
[00:24:55] Bryan Bischof: And yes, CLI are really powerful because coding is a [00:25:00] verifiable task. And so training models to get back at coding, getting better at coding is, is a very tractable thing. But we also know that there's a shitload of applications of language models where plain function calling is super valuable. And then separately, sort of like these like connections of tool use.
[00:25:19] Bryan Bischof: And then MCP is more about authorization and distribution, and we understand this nuance, but unfortunately attention is being highly fought over, and so everything is pushed into these extremely spiky statements like MCP is dead. And so even some of the people that I used to really like admire in terms of the way that they presented tech in sort of like on Twitter, for example, I think now are making regrettable comments that piss me off.
[00:25:50] Bryan Bischof: Like levels IO saying MCP is dead because he doesn't have the MCP shaped problem. Come on. So, [00:26:00] um, I think ultimately I think what I wished people took away from both rip grep and find all. And if you haven't found the Easter egg of Find all, I encourage you to go to rip-grep.com. Um. The kind of idea here is to find people talking about things and sort of desensitize yourself a little bit to these hype cycles, but then also have an opportunity to understand the dialogue in a more meta perspective.
[00:26:30] Bryan Bischof: Look at the whole field because remember, if you say, ChatGPT killed Grock, and then a week later somebody else said, crock Grock killed Claude. That gives you this sort of like teach the whole like, uh, you know, discussion or argument perspective. And so if there's one real like goal with rip grep, it's to give everybody a moment to look at the entire situation.
[00:26:56] Bryan Bischof: I.
[00:26:57] Andrew Zigler: Yeah, absolutely. And I think you're really calling out [00:27:00] here the, the commoditization of information within our industry, the social media ification of all industries. On platforms that we post on, that we share discoveries and you know, it's unique to our times and I think sometimes we forget that like if you were to take the same social media world that we have and plop it onto the.com bubble, you probably would've gotten a lot of cycles of a lot of people calling, a lot of things in the early web dead.
[00:27:25] Andrew Zigler: And you're exactly right that what this does is it erodes. All of the nuance of the conversation so that you get the polarized, like you said, spiky view, um, the, like most magnetized on either end of that spectrum kind of thing you could say. And like I said, it erodes the nuance. It, it prevents us from having constructive conversations.
[00:27:46] Andrew Zigler: Uh, and it ultimately. it kind of like dampens the ability to share. And so it's like a really tough environment sometimes in that world to like actually navigate like what's real and what isn't. What can I act on and, and [00:28:00] what can I not? Um, and so, you know, I think this is a really healthy reminder.
[00:28:04] Andrew Zigler: For everybody that you know in any kind of thing that you're reading or discovering. If it's constantly a hyperbole of the situation, then you're probably being robbed of the nuance of what's actually happening, which you can only really learn by maybe trying to find counter sources or or supporting arguments, or frankly, at this stage that we're all in.
[00:28:21] Andrew Zigler: I really just beg everybody just go to arXiv. Just read the white paper that we're all talking about because they're not that long. And, uh, and honestly, in some cases they're like, you are gonna get much more out of like the abstract than you would from any scroll by on LinkedIn. Um, so all of that, you know, said, I, I don't even want to get into how you got rip-grep.com
[00:28:42] Andrew Zigler: Um, but it's like, I, I, I, if it's like, do you, where do you, could you even get that URL from? Surely that was taken.
[00:28:51] Bryan Bischof: Uh, shockingly, no. Uh,
[00:28:53] Andrew Zigler: That is crazy
[00:28:55] Bryan Bischof: on this, so rip-grep.com, both the domain name and the [00:29:00] original like idea of
[00:29:01] Andrew Zigler: Yeah.
[00:29:02] Bryan Bischof: for this were Adam Conway's, like brainchild. He texted me and was like, okay, so this thing we've been talking about, like here's how I think we could do it, and then he's like. rip-grep.com is available. I'm like, buy it now.
[00:29:19] Andrew Zigler: Immediately.
[00:29:20] Bryan Bischof: with Buy it now. And then his first version uses used Google Trends and I think just he and I very rapidly in about one week went from this is a cute, I we gotta do this right? Oh my God. April Fools is around the, oh, let's get it now. um, yeah, it was a little bit of a mad dash. Um, so for just in case people don't know, so rip-grep.com we launched on April Fools. It is, is accompanied by a very satirical post from our flagship like Employer Theory ventures. Um, we followed up about a week and a half later with Find All. Which is if rip-grep.com [00:30:00] is blah, blah, blah is dead, find all is blah, blah, blah is all you need. Um, because again, like the oversimplification is kind of cute. Um, but someone pointed out to me that we made a huge blunder with find all. And the blunder was that we didn't launch it on Easter.
[00:30:17] Andrew Zigler: Because then it would've been coming back.
[00:30:21] Bryan Bischof: Yes.
[00:30:22] Andrew Zigler: Yeah.
[00:30:23] Bryan Bischof: Idea that all these things are rising from the grave was such a good pun that I didn't think of, and I'm like, it's missed
[00:30:31] Andrew Zigler: You were just in this cloud of humor and you're like, dang it, I just missed the way to optimize the joke. Well, you know. I, you know, I definitely work, gonna be putting the links in the show notes for folks to go check this out. We've also covered it recently on our new segments. We'll make sure our listeners have, uh, routes to go check out all this stuff that we're talking about.
[00:30:47] Andrew Zigler: you know, just before we move on from this, you know, we've talked about eroding away all of the nuance and how social media and conversations like this just, you know, push all of the conversations to the extreme. So here I am doing it and I want to ask you, Bryan, what [00:31:00] do you think is actually dead?
[00:31:02] Andrew Zigler: Do you think there's anything actually dead in what we've been talking about that's really not coming back?
[00:31:06] Bryan Bischof: So one thing that I think is to my eyes and dead to my evals. Like just a reminder, like I have a lot of evals for both, like real work stuff and personal stuff. I've been running personal evals since 2023 on just like random shit that the models can't do. I keep them private 'cause I'd rather them not get saturated like, um, sorry.
[00:31:27] Bryan Bischof: I'd rather them not get saturated. artificially is
[00:31:31] Andrew Zigler: Yes.
[00:31:31] Bryan Bischof: way to say that because if they get art, if they get un artificial or, you know, naturally saturated, that's great news 'cause that's an indicator of progress. But we'll keep that to the side to my evals, I think one thing that is kind of dead is this like extreme massaging of the system prompt, and bring this up because it's something that I believe very strongly, but I've also been contradicted this week, so. [00:32:00] I really believe this, but it's also came out this week that I'm wrong about this. And so I just wanna like, you know, uh, air my dirty laundry. So I've, for about the last six to nine months, found almost no impact in random stupid shit in the system prompt about like, don't do this and like, speak this way.
[00:32:18] Bryan Bischof: And that just like doesn't work anymore for me. A lot of sort of like context engineering stuff still works and prompt engineering is not dead. I'm, this is not the claim, although maybe that's the like, headline for the thumbnail. Uh, but like, you know, I think I don't see a lot of value in, don't talk about goblins, and that's, I think a perfect example of me being wrong, where like, OpenAI literally has to put twice in the system prompt, don't talk about goblins. So. You know, uh, whether or not like I'm wrong or not, I think we all should let the goblins decide, but what I will say is this is a thing that I genuinely do [00:33:00] believe is mostly dead. The other thing that I don't think is dead, but I think has never worked, and I continue to not find success with it. When I get people into like a tight enough, like, you know, booth at a bar and they know that no one's listening, they will admit to me. And that is prompt optimization. I like, it's everybody's favorite, like, you know, hobby horse, that like prompt optimization will get you there. And I continue to try over and over and over on this task and that task. There are some really, really like rigid classification type things that I can get some improvement, but just last week, over the weekend, I had a Classifi classification task that would be annoying to train a normal classifier on. So I was like an LLM classifier is gonna be ace for this. I'm stoked to do [00:34:00] it, and I let it optimize the shit outta my prompt. I said, let's go. I made it. Give me a little CLI based, like data labeler.
[00:34:10] Bryan Bischof: I laid a hum, labeled a hundred examples. I like ripped through it with my keyboard shortcuts. That cloud code was so happy to make for me. I'm like, this
[00:34:18] Andrew Zigler: You did everything you were supposed to.
[00:34:20] Bryan Bischof: I did, I did. Let's calibrate the shit outta one another. uh, it ran and the performance was not that good. It was like 65%. And I'm like, maybe this is a really hard task. Okay, let, let's do some error analysis. Well, I don't know where Hamel is, but he's like sweating suddenly. we're, let's do some error analysis. I did some error analysis. Oof. Those are some, like really annoying errors. Let's, let's make some suggestions to the model to update the prompts about that error analysis.
[00:34:50] Bryan Bischof: Oh, why don't you do some error analysis on your own model. Why don't you keep iterating on this? Hey, let's be careful. Let's take a hold out of examples that you're not allowed to look at while you do your own [00:35:00] iteration loop. Let's do some prompt optimization. Let's fucking get RLMI said RLM. Um, let's do it.
[00:35:06] Andrew Zigler: Just put it, put it all in the pot. Put it all in the pot.
[00:35:09] Bryan Bischof: Yeah, let's go. And we get to like 68% and I'm like, man, is this task that hard? It doesn't seem like it should be. So I do the thing, I pull up the prompt. I look at the prompt with my eyeballs. I'm like, what the hell is this? This is insane. I'm like, these are insane things to have in the prompt. I'm now five models deep. They all tried to optimize this shit. I've like scanned, you know, this like parallel grid search over different models and different thinking modes. Trying to keep it at a certain level. 'cause I got a hundred thousand of these things a label, so I can't be using GPT 5.5. It didn't exist last weekend, but you know, get the point
[00:35:51] Andrew Zigler: Yeah.
[00:35:51] Bryan Bischof: so. Long story short, like I'm like annoyed. And so I just write the damn prompt and suddenly we're at 78%. I'm like, for [00:36:00] fuck's sake. And so, you know, is like prompt optimization dead? No, I never saw it work a single time, and I, this is like, you know, I, I remain annoyed.
[00:36:10] Bryan Bischof: We'll just leave it at that.
[00:36:11] Andrew Zigler: Oh, your poor soul. So you just got trapped in this like, um, uh. Optimization loop that just was, ended up being wasted cycles. It didn't actually do what it was supposed to be. And there was something maybe born out of hype and out of tools that were supposed to be supporting, uh, what your ultimate goal was.
[00:36:28] Andrew Zigler: But it, but it didn't and, and it really challenges too, like maybe once upon a time that worked really well. For a certain class of models, and that's just like a vestigial thing that we've continued to carry forward of like, oh, it worked, then it would work Now, um, I, I've definitely been in your shoes before where like, I go crack open what Disp did, and I'm like, what is happening in this room?
[00:36:46] Andrew Zigler: And so, um, I'm like, what did I just. Walk. What did I just walk into? Very relatable. When you, if so, if you know you're using these kinds of tools, it's a good reminder to go put your human eyeballs on the stuff that your agents are, are putting their agentic eyeballs on. [00:37:00] Uh, it actually can be the root of so many of your problems.
[00:37:02] Andrew Zigler: You think that you're on this like hyper optimized same page as your agents and it just turns out they're reading, um, something in one language and you're reading it than another. So
[00:37:12] Bryan Bischof: Totally, totally.
[00:37:13] Andrew Zigler: very relatable.
[00:37:14] Bryan Bischof: The multi-language thing because
[00:37:16] Andrew Zigler: Okay.
[00:37:16] Bryan Bischof: of the error cases was some of the data was in, uh, foreign language and I had had a semaphore for basically like. These are foreign language, like, um, like examples and I want them kind of their own little class. And it was not learning that at all. And that was like a very simple fix. And like, that's actually one of the sort of things that I found by looking at the data.
[00:37:40] Andrew Zigler: Yeah, exactly. You know, I, I wanna now to talk a little bit about. What you do kind of day to day and the vantage that you do have in the industry. We talked a bit about like your background in, in teaching, but then also curating, um, information and education tracks and about, you know, doing your April fools pranks as well.
[00:37:58] Andrew Zigler: But, you know, the other [00:38:00] 364 days of the year. I wanna peel back the, the, the hood here and really understand like what it is that like you're most focused on as the head of AI at, uh, you know, Theory Ventures. It's a really unique vantage point to be in. So, you know, what kind of, um. Things are like kind of like top of mind to you in terms of like how you operate in your role and what you see your responsibilities as being
[00:38:21] Bryan Bischof: So ultimately my job is to build software. Um, my team's responsibility is to build software that makes the firm more effective. So we've
[00:38:32] Bryan Bischof: got
[00:38:32] Bryan Bischof: these amazing investors who are all brilliant and they're doing intelligence work. They're doing work that uses resources and research and a lot of sort of like document driven workflows, and they're trying to learn things in a very deep way, in a very short amount of time, and then understand how big of a bet they're willing to put on their understanding. [00:39:00] Now these people are extremely good at research. These people are extremely effective at sort of like high order thinking and sort of like trade-offs and things like that. They're also really great at finding people to connect with, to learn from. That's where their time should be spent. Should their time be spent like massaging documents to put them into like a particular structure? Probably not. their time be spent doing sort of like the obvious kind of document research or going and hunting down the same type of sources that we always want to use? Probably not. Should they be monitoring the situation? Ideally not. Um, and so there's a lot of work that goes into being a great investor that is not edifying for them and is not candidly like what makes them special. They add on to that they have this sort of like marginal value that they add to each of those steps. But [00:40:00] actually the most alpha that we sometimes refer to it as that they can provide is by being focused on the most deep, hard, intellectual plus social problems in the job. Well, it turns out the past couple years we've had this massive rise of technology that is very good for doing routine tasks to intelligence and documents and information. So now's a great time to build software. Um, so long story short, we're building research agents and document management systems and context management systems, and a lot of things around how do you make a business operationally completely driven by language models. How do you say our business operates via documents?
[00:40:49] Bryan Bischof: Like documents are like the structural thing that allows us to operate. If you, if that is true of your business, how do you build software that makes that feel [00:41:00] insanely rigid and strong and effective? That's what we build. We build a lot of different sort of experimental software. We're constantly sort of trying the new things to make sure that we are taking the most advantage of language models. This is the kind of thing that you could have tried to build five years ago. It would've been really tedious and it would've sort of had like a glass ceiling,
[00:41:25] Bryan Bischof: and if you Yeah.
[00:41:25] Bryan Bischof: to build it five years before that, there were a lot of things that would've been like really net new and sort of like none of the infrastructure existed for it, and you would've just like been constantly around. But man, is it a fun time to build this kind of software. And so that's what my team does. In addition to that, we obviously help with diligence here and there, so like when it's time to talk to some founders and help the firm understand sort of like their approach and what might require 10 years of engineering experience to grok deeply. We can help. Uh, and then [00:42:00] also we do some portfolio support. So if you are one of the lucky, uh, members of the portfolio and you say like, Hey, we've got this challenging AI problem, any ideas? Sometimes we have ideas. And so we try to step in and really kind of close the gap where we can, as you know, like I am terminally online.
[00:42:20] Bryan Bischof: And so given that I am terminally online, sometimes I just have a deep understanding of what's new.
[00:42:26] Andrew Zigler: Yeah, the way that you put it, it's so, uh, like net new in terms of, uh, the things that you build and things that you do. I think it's a really exciting time because I think in the world of knowledge work and the world of running business right now, we're all in this discovery mode of like, how can I create the most effective, the most optimized, the most high leverage business that I can?
[00:42:46] Andrew Zigler: And it's not about doing. More with less people. Just about doing more for, because I have this information because I'm able to surface these insights that before were never able to get close to my fingertips. And then even once they were, my ability to act upon it was so [00:43:00] removed. There were so many layers between me and that.
[00:43:02] Andrew Zigler: So you make it sound more like you're like a venture engineer. You create these environments, this situation, this leverage for in investors, uh, and their companies to be able to both, uh, get more out of that relationship and to better understand each other. And also, like what you said, one of the biggest benefits of being at as a VC network is that you get the benefit of all the other smart founders that you're, you're lined up with in their portfolio.
[00:43:28] Andrew Zigler: And right now. How amazing would it be to have a portfolio that can effectively spread over all of these, you know, going back to what you said earlier, gaps in what we're understanding and, and making smart bets on smart people that are solving those, those problems, you know, and, and making them a reality now.
[00:43:47] Andrew Zigler: And so, uh, as somebody in your position who like works across that kind of like domain and that kind of expertise, obviously you have to be terminally online and read all the polarizing opinions from folks like me on LinkedIn you [00:44:00] also have to be teaching, uh, students and then, uh, compiling research together.
[00:44:04] Andrew Zigler: Uh, and, and so. Sometimes though, you know, when you're not terminally online, you're terminally in person and you're hosting really interesting events as well. Like when you had a earlier I in this, uh, in this discussion I mentioned America's Next Top modeler, uh, competition that you had at the end of last year.
[00:44:21] Andrew Zigler: Hamel was actually there, uh, and, uh, Hamel big fan of his work and, and I learned so much about observability from him and at this event, like you challenged all of us with, uh, a pretty kind of kooky premise. You know, I maybe if you wanna just remind us, me and my listeners, uh, you know, like what you challenged us to do when you summoned this group of like 200 people to this venue in sf.
[00:44:45] Bryan Bischof: Yeah, for the BoJack fans in your audience, um, data agents, what do they know? Do they know things? Let's find out. Um, and so that was ultimately the premise is just, yeah, like are data agents any good? And if so, like is there some [00:45:00] secret sauce? Is there a sort of way that you could build a data agent if you had one specific data set that was really good at it.
[00:45:07] Bryan Bischof: There's, you know, like Texas equals been dead before. Many things were dead. Um, like I think that was declared dead very early on in the, in the epoch. But there is this weird in the community of like, Texas Equal is certainly dead. The models are so good at sequel. candidly, all of data science is totally trivial for models. I can't get them to do fucking anything, right? ever like every single data analysis question I ask the model to do it screws up in a creative way. Um, but there is this like middle ground where a lot of great products exist and you can get value out of products that. Like have been engineered to be useful and like to be clear, I'm talking my own book a little bit [00:46:00] because I worked on one for two years, uh, and I arguably the first one.
[00:46:04] Bryan Bischof: And so I think, yes, I am biased. I do believe that there is a value in a data agent, I'm also biased in the other direction, which is I hear that this has completely been obviated by smarter models and I don't see that or don't feel that. And so my big question was, okay, show me if you think it's so trivial to do this, then show me. And so I don't know if you know this, but I actually invited a lot of like people that work on these things that claim that theirs is. A star 10 out of 10. And I said, on in, like, let's see it. Um, we had some people that were very confident and then didn't show up. We had some people that were very confident, did show up and then quietly snuck [00:47:00] out.
[00:47:00] Bryan Bischof: We had some people that were a little bit more sort of like confrontational of their failure. Um, and then we had some. People that came with some confidence and did quite well. And so I think, you know, I wrote a, a blog post about this afterwards that I think is a great place to go and see my full take on the event of sort of like what works and what doesn't.
[00:47:23] Bryan Bischof: But ultimately what I wanted to understand is just the simple question, like if wanna build a data agent, a, is there some secret sauce that a few growth stage companies have figured out and you just buy that product? B are the models so damn smart that you shouldn't even like think about it and you should just like, let it rip.
[00:47:41] Bryan Bischof: Like just open up a sort of like coding agent and it's friendliest sandbox connected to your snowflake and move on with your life some other thing where you should build some harness. You should think about the sort of, you know, the way that you're doing harness engineering [00:48:00] and really put like effort into this. Is that the sweet spot? Or maybe you're an ultimate pessimist and you're just like, Nope, they suck. Move on. Like don't try it. don't think the last one is correct. I also don't think. The first two are correct, I believe in harness engineering. Maybe that's the like, you know, like, um, t-shirt that I need to wear to data council.
[00:48:21] Bryan Bischof: I believe in harness engineering. Um, but like I, I think that there is a lot of interesting work to be done here. I think the problems are still really interesting and really hard. Um, and I haven't given up my obsession yet. We'll say that.
[00:48:36] Andrew Zigler: Amazing. So you treated us like a real. Science experiment. You brought us all together. Uh, and you're definitely right. There was a huge mix of confidence and ability in the crowd. And you even hired a real, like actual human, like meet LLM analyst to be there and like, and like study the the stuff and be like, what a traditional analyst do this better than the agent.
[00:48:58] Andrew Zigler: You really were throwing [00:49:00] everything you could at this experiment. Uh, so are there plans for like a season two? Do you have a second experiment that you're looking to summon together the masses for? Like what, what can people expect for you to be planning next?
[00:49:13] Bryan Bischof: Yeah, so it's not been announced yet, but I can preview for your audience, um, that there will be a new event. It will. The name has not been it. The name will not be announced just yet. Um, we are still cooking, as they say.
[00:49:29] Andrew Zigler: Okay. Okay.
[00:49:30] Bryan Bischof: but
[00:49:30] Andrew Zigler: you have high standards live to, because before this you did a, so you think you can prompt. That's also pretty hard to beat. You managed to beat it with America's Next Top modeler, so I am seated for the third title.
[00:49:42] Bryan Bischof: Yeah, I mean, I think a lot of people are kind of wondering like. What's the next game show? Like what, like what bad TV does Bryan watch?
[00:49:53] Andrew Zigler: Truly.
[00:49:53] Bryan Bischof: so, um, I can say a couple things about the next one. So it is gonna be data agent themed, but it's not going to [00:50:00] just be data agent builders. So I think that I am interested in hearing from that crowd more, but I'm also interested in hearing from the other side of the table. So, the basic thing that I've been obsessed with for a couple years now and I've been talking to Hamel about consistently is, uh, data science is AI engineering. And AI engineering is data science. Fun fact, I posted a job posting for an AI engineer before the blog post about AI engineering was written. And sometimes I look back and I'm like, I should have just called it a data science role. Um, but long story short, I, I believe that there's a lot of overlap between doing data science and doing AI engineering. And this intersection of data science for AI engineering is really interesting. And I think it's like it's produced a lot of like my thinking, but the other direction is a little bit. Underappreciated in a way that boggles my mind. [00:51:00] how many great talks have you seen about sort of how to use AI for data science? Weirdly, sparse. I've seen 10 times as many about using data science to build ai, and candidly, just like building AI for data scientists. Everyone seems obsessed with like building the tool.
[00:51:25] Bryan Bischof: I think this is a little bit of a sort of a second order effect on picks and shovels. Everyone's like, oh my God the most interesting topic is how you build a data science agent. That's interesting, but it's also interesting to use an agent to do data science. I do it. I do it all the time. I'm doing it on this laptop over here that you can't see. I'm looking at this computer, but there's another computer and candidly another one over there as well, and they're both doing data science right now and they're doing data science as agents. What are they doing? How are they doing that? That's [00:52:00] something that I'm very interested in and so there is an event coming that will dig into both sides of this relationship of AI for data science and data science for ai.
[00:52:11] Andrew Zigler: Well, we're all excited to see what you end up unveiling and uh, we'll definitely be sure to, to share it here so that folks can connect all the pieces of this story together. Um, but Bryan, thank you so much for joining me today, taking some time out of your schedule to, to chat about all the things happening so rapidly in our industry.
[00:52:30] Andrew Zigler: It's really exciting to be on site with you and looking forward to chatting on and around the event, but you know, where can folks go to learn more about you and your work at Theory Ventures and all the stuff that we talked about today?
[00:52:42] Bryan Bischof: Yeah, so I think the easiest place to stalk me is on Twitter. Um, as we already mentioned, I'm terminally online. Um, so check me on Twitter. It's be bi, which will almost certainly make it into the show notes. And
[00:52:56] Andrew Zigler: Yes.
[00:52:56] Bryan Bischof: on LinkedIn you can also follow me there. I would say I'm a little bit [00:53:00] like toned down, not all the way toned down. Um, occasionally I still get great messages about my LinkedIn header. Um,
[00:53:08] Andrew Zigler: It's pretty great.
[00:53:09] Bryan Bischof: firsthand what that LinkedIn header is. and those are the sort of like main channels, but you should also check out Theory VC um, so theory vc.com. Um, we are an early stage venture firm. We invest in data, crypto and AI companies.
[00:53:25] Bryan Bischof: We are really motivated by sort of like technological changes that lead to new opportunities. We're a highly concentrated fund. We make very few investments per fund, but they're big ones. And so, um, yeah. Uh, looking forward to hearing from you if you're a sort of like founder or a really excited, sort of like, know. ready to build the next change. The last thing I'll say is, uh, earlier you said there's only one day you're, that's April fools. And I just wanna say like I disagree. Every day can be April Fools if you try [00:54:00] hard enough.
[00:54:00] Andrew Zigler: You know. That's fair. That's fair. I think we could all try a little more to be a, a little more April Fooly, especially on places like LinkedIn. Uh, but we're definitely gonna be including links to all the stuff we talked about today, uh, for folks to go and continue the story. Um, like Bryan said, and he's
[00:54:15] Andrew Zigler: terminally online I am as well. You can find me predominantly on LinkedIn. Uh, but if you want to continue these conversations, please come find us, bug us, shake us, you know, drop in our dms. Uh, we love hearing from y'all. And if you're listening to this, be sure to go to our LinkedIn or substack. You can subscribe and get the newsletter that comes along with this issue where you can find a lot of these helpful links.
[00:54:35] Andrew Zigler: Um, and as well, be sure to find, uh, follow Bryan wherever you can find him and wherever, whatever platform is most resonant for you. Uh, we're big fans of his work here at Dev Interrupted. We're gonna be continuing to showcase it and you know that we'll be here for season three of the great AI Data Showdown.
[00:54:51] Andrew Zigler: So Bryan, thanks again for coming on the show. It was so fun to have you. Uh, and let's do it again sometime. Thanks.
[00:54:56] Bryan Bischof: Awesome. Thanks man.
