# AI adoption delivers when engineering leaders prioritize impact | LinearB Blog > OpenAI technical staff member Louis Brandy breaks down a pragmatic framework for engineering leaders navigating the AI hype cycle. Discover how to evaluate engineering impact using the "shiny things" framework, design time-boxed AI experiments to build team judgment, and properly measure AI coding assistants before scaling adoption. [Blog](https://linearb.io/blog) / AI adoption delivers when engineering leaders prioritize impact # AI adoption delivers when engineering leaders prioritize impact ![Photo of Ben Lloyd Pearson](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/blp_headshot_1_ee25d527aa?_a=BAVMn6ID0) By [Ben Lloyd Pearson](https://linearb.io/blog/open-ai-louis-brandy-ai-impact-engineering-frameworks#ben-lloyd-pearson) | July 10, 2024 ![Blog_AI_hype_cycle_2400x1256_3decdf7253](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/Blog_AI_hype_cycle_2400x1256_3decdf7253?_a=BAVMn6ID0) [Louis Brandy, Member of Technical Staff at OpenAI](https://linearb.io/dev-interrupted/podcast/AI-hype-cycle-how-to-find-real-and-practical) and former VP of Engineering at Rockset, has watched AI evolve from early machine learning systems to today's landscape dominated by large language models and retrieval-augmented generation. His perspective on AI adoption is refreshingly pragmatic. He is neither dismissive of the technology's potential nor swept away by the hype cycle that surrounds it. He firmly believes that AI is not all hype, but he is equally convinced there is an incredible amount of froth that organizations will have to sort through to [figure out what is real and what actually sticks](https://linearb.io/blog/ai-measurement-framework). ## **AI adoption pays off when leaders match investment to real demand** For engineering leaders, the critical first question isn't whether AI matters. It is understanding your organization's relationship to the technology. Brandy distinguishes two fundamentally different contexts: companies betting their product strategy directly on AI capabilities, and teams responding to external pressure to simply "do something with AI" despite operating in unrelated domains. These situations demand completely different approaches and levels of investment. The evaluation framework Brandy recommends works from both directions simultaneously. Outside-in analysis examines market demand and customer needs. It asks what problems customers are actually asking AI to solve and where genuine demand exists beyond the noise. Bottom-up exploration looks at internal ideas, technical possibilities, and team curiosity about what AI might enable. Both perspectives matter because AI's impact is likely to be horizontal rather than vertical. "Two years from now, it's going to be shockingly good at something you did not expect it to be that good at that quickly." This horizontal disruption pattern makes AI particularly difficult to predict. Rather than cleanly replacing entire job categories, the way mechanical looms displaced weavers, AI is more likely to affect significant fractions of many different roles. A recommendation engine might become dramatically better at real-time personalization. A documentation chatbot might suddenly handle complex troubleshooting queries. The specific capabilities that mature fastest are genuinely hard to forecast, which means leaders need enough understanding to recognize strategic opportunities when they emerge without [overcommitting resources to speculative bets](https://linearb.io/platform/resource-allocation/features). The practical challenge is maintaining this balance by building sufficient organizational knowledge to move quickly when AI capabilities become relevant, while avoiding the trap of chasing excitement for its own sake. ## **Shiny AI projects rarely deliver the most impact** Brandy's management framework for evaluating engineering work cuts through the AI hype with remarkable clarity. He argues that every project exists in a two-dimensional space defined by how shiny it appears to engineers and how much business impact it is likely to deliver. The mathematical reality of this framework is sobering. Truly shiny and impactful projects are almost never real. If something were genuinely both exciting to engineers and clearly valuable to the business, it would already be done. The work that remains tends to cluster along a diagonal where the less glamorous a project appears, the more likely it is to deliver meaningful impact. This creates a specific leadership challenge around AI initiatives. Engineers naturally gravitate toward shiny work without managerial encouragement. If you tell a software team that AI is strategically important, they will run full speed toward it regardless of whether the specific application makes business sense. The manager's job isn't to amplify this enthusiasm, since it is already self-sustaining, but to provide the counterweight. "As a manager, you should be deeply distrustful of shiny things." The most valuable engineering investments are often comparatively unglamorous. Foundational quality improvements, [delivery pipeline optimizations](https://linearb.io/blog/how-to-measure-your-software-delivery-pipeline), and test coverage expansions increase execution speed across the entire organization. These initiatives do not generate the same excitement as implementing a novel AI feature, but they frequently deliver far greater business value. Applied to AI specifically, this framework demands that leaders make impact explicit before approving work. Enthusiasm for new LLM capabilities or vector search infrastructure is simply not sufficient justification. The fundamental question must be whether the specific AI application addresses a real business problem, serves a genuine customer need, or removes a meaningful constraint on organizational effectiveness. The same principle extends to individual career development. Engineers maximize their impact and advancement by aligning their strongest skills and deepest interests with business-critical problems. Chasing novelty in isolation, or working on cutting-edge AI just because it is exciting regardless of whether it matters to the company, is a surefire path to frustration and stalled growth. ## **Time-boxed AI experiments build judgment without wasting resources** Given the difficulty of predicting which AI capabilities will prove valuable, Brandy recommends a specific experimentation model of short, bounded sprints that encourage broad exploration without creating open-ended commitments. Rockset runs periodic hack weeks where engineers can work on any project they choose, with the only requirement being a demo by Friday. During AI's current hype cycle, these weeks naturally attract significant AI experimentation. The time-box is the critical element because it limits downside risk while still pushing teams beyond superficial familiarity with the technology. Brandy points out that even if nothing comes of a project, the team is no longer operating at a surface-level understanding of AI. By diving two or three levels deep, they transform how they evaluate future AI opportunities. After hands-on experimentation, engineers can assess demos and proposals with genuine insight, identifying what won't work and recognizing what modifications might make a concept viable. They can distinguish real technical challenges from solvable implementation problems. The time-boxed approach also creates a [psychologically safe way to fail](https://linearb.io/blog/engineering-mental-models-happier-productive-teams). Killing a three-month project because it isn't working feels demoralizing and wasteful. Building something ambitious during a hack week, discovering it's terrible, and erasing it at the end feels energizing. Teams preserve momentum and morale while learning extensively about what doesn't work and why. Sometimes these experiments surface genuinely durable opportunities. Rockset's first implementation of vector search, now a formal product capability supporting retrieval-augmented generation architectures, originated in a hack week. Engineers built a prototype demonstrating that vector search could integrate cleanly with Rockset's existing infrastructure, which eventually justified full investment. The broader architectural context matters here. Retrieval-augmented generation depends on fast, fresh data access to augment what large language models know intrinsically. An LLM trained on static data cannot answer questions about current baseball scores or recommend live streaming channels without real-time retrieval. This makes infrastructure choices about how quickly data moves from source systems into searchable indexes newly important for AI applications. Time-boxed experiments help teams understand these dependencies and identify where infrastructure investments unlock new product possibilities. ## **Measure AI coding tools before scaling them** When it comes to AI tools in the software development process itself, Brandy's stance is remarkably measured. Rockset has enabled tools like GitHub Copilot for experimentation, but they are not yet treated as essential infrastructure. Brandy notes that while they have people trying out Copilot and similar tools, he cannot say they have been wildly successful so far. This doesn't mean AI coding assistants have no value. It simply means their impact varies significantly by context, and the cost is very real. Before a broad rollout, Brandy emphasizes the absolute importance of measuring actual effects on developer productivity and experience. Some engineers find substantial value in AI-generated code suggestions, while others find them distracting or low-quality. This massive variance makes blanket mandates premature. The next capability frontier appears to be tool-using AI agents that participate more actively in engineering workflows, from writing code and generating tests to running those tests and iterating based on results. This extends an existing automation spectrum that already includes linters, static analysis tools, and bots that open issues or comment on pull requests. Modern AI coding assistants aren't a complete break from prior workflow automation, but rather a more sophisticated point on a continuum. ## **Measuring impact before making a commitment** This perspective helps contextualize both the opportunity and the risks. One practical concern Brandy raises is [review capacity bottlenecks](https://linearb.io/resources/6-bottlenecks-ai-development). If AI or bots generate code faster than humans can meaningfully review it, the result isn't accelerated delivery. It is a brand new constraint that slows everything down. Brandy observes that larger companies are rolling out these tools to 70 percent of their developers, often driven by top-down directives to stay ahead of the curve. However, the most effective leaders also approach this with a mindset focused on measurement, wanting to [gather data and report back to the business](https://linearb.io/blog/how-to-use-team-data-for-engineering-resource-allocation) before fully committing. This measured, data-driven approach reflects the current state of AI coding adoption across the industry. Most organizations are experimenting and want to deeply understand the impact before making AI assistance a required part of software development. The data is starting to come in, showing significant variance based on developer role, application type, and codebase characteristics. For now, engineering leaders should approach AI coding tools as exploratory rather than essential. They are valuable for some contexts, less so for others, and absolutely worthy of rigorous evaluation before significant investment. To hear more of Louis Brandy's insights on the AI hype cycle, managing engineering experiments, and finding real business impact, listen to his full episode on the Dev Interrupted podcast. ## Improve developer productivity with LinearB Find us on [](https://www.linkedin.com/company/linearb) [](https://devinterrupted.substack.com/) ![blp_headshot_1_ee25d527aa](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/blp_headshot_1_ee25d527aa?_a=BAVMn6ID0) ## Ben Lloyd Pearson Ben hosts Dev Interrupted, a podcast and newsletter for engineering leaders, and is Director of DevEx Strategy at LinearB. Ben has spent the last decade working in platform engineering and developer advocacy to help teams improve workflows, foster internal and external communities, and deliver better developer experiences. ### Connect with [](https://www.linkedin.com/in/benlloydpearson) [](https://substack.com/@benlloydpearson) ## Your next read [![Cover image for Deterministic guardrails must evolve to control probabilistic AI systems](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/Blog_Post_Name_2400x1256_5_f5375661ec?_a=BAVMn6ID0)](https://linearb.io/blog/launchdarkly-cameron-etezadi-deterministic-guardrails-probabilistic-ai-darkfactory) AI [Deterministic guardrails must evolve to control probabilistic AI systems](https://linearb.io/blog/launchdarkly-cameron-etezadi-deterministic-guardrails-probabilistic-ai-darkfactory) LaunchDarkly CTO Cameron Etezadi explains why traditional engineering pipelines are breaking down in a probabilistic, AI-driven world. Discover how to safely... [![Cover image for Slack turns channels into the context engine for agentic AI](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/Blog_Post_Name_2400x1256_077524ba8b?_a=BAVMn6ID0)](https://linearb.io/blog/slack-jaime-delanghe-mcp-agent-context-channels) AI [Slack turns channels into the context engine for agentic AI](https://linearb.io/blog/slack-jaime-delanghe-mcp-agent-context-channels) Slack Chief Product Officer Jaime DeLanghe breaks down how channels serve as the foundational context layer for human-agent collaboration. Learn why Slack is... [![Cover image for How leading teams turn AI into an engineering productivity edge](https://assets.linearb.io/image/upload/c_limit,w_2560/f_auto/q_auto/v1/Blog_Post_Name_2400x1256_be95b2d258?_a=BAVMn6ID0)](https://linearb.io/blog/ai-productivity-edge) AI [How leading teams turn AI into an engineering productivity edge](https://linearb.io/blog/ai-productivity-edge) Developers using AI heavily are merging PRs up to 2.2x faster than a year ago. See the mid-year benchmark data on the widening productivity gap, and how to...