"Sidekick's vision is that it's basically your co-founder or your e-commerce expert that doesn't judge you. You can feel comfortable to ask any question, get expert results... Having somebody who you can reach out to answer any question you want, ask anything."
First, there was vibe coding. Now, get ready for "vibe entrepreneurship."
Andrew McNamara, Director of Applied Machine Learning at Shopify, joins us to explain how his team is making this new era of business a reality. He shares the vision behind Shopify Sidekick, an AI co-founder designed to empower merchants by acting as their on-demand e-commerce expert. Drawing on his 16-year journey building AI assistants, Andrew reveals what it truly takes to create an AI tool that customers can trust with their livelihood.
He shares a critical insight from his applied research background: the hardest and most important part of building production-ready AI isn't the model, but the evaluation ("eval") system. Andrew breaks down Shopify's innovative approach of using an LLM-as-a-judge to measure how well they're fulfilling user goals, not just executing features. This conversation is a definitive guide to creating high-trust AI systems and offers a powerful glimpse into the future of commerce.
Show Notes
- Follow Andrew on X
- Follow Andrew on LinkedIn
- Andrew's Talk at ICML
- Learn more about Shopify Sidekick
- Follow Fatima on LinkedIn
- Read GitLab's: The Economics of Software Innovation
Transcript
(Disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
[00:00:00] Andrew Zigler: We're back on Dev Interrupted and joining me is Andrew McNamara the Director of Applied Machine Learning at Shopify and his impressive background. It includes so many great things like applied research at Microsoft after a Maluuba where he built and scaled machine learning algorithms was acquired in 2017 and. That really sets the date for you because he's been been building human-centered AI since before
[00:00:24] Andrew Zigler: it was cool. And we're actually gonna get to that a little later, but until recently it was more just called machine learning. And right now, Andrew is bringing Shopify sidekick to e-commerce with the goal of unlocking the full power of the platform in everyone's hands using only natural language.
[00:00:41] Andrew Zigler: And with Sidekick, we're not just talking about automating tasks. We're watching the rise of something new, a generation of merchants that can almost run a vibes first business. And no spreadsheets. No MBAs. Just good ideas, great products, and the right AI tools, and I love this idea because it's so quintessentially [00:01:00] 2025.
[00:01:00] Andrew Zigler: So I'm excited to dive into this with you, Andrew. Welcome to the show. Yeah,
[00:01:03] Andrew McNamara: Yeah. Thanks so
[00:01:04] Andrew McNamara: thanks so much.
[00:01:04] Andrew McNamara: Uh, I'm excited to be here and really looking forward to our chat.
[00:01:07] Andrew Zigler: Great. So, uh, starting off, you know, I, I wanted to pick your brain about, uh, this like personal assistant to AI pipeline that's been your career. And we talked about this a little bit when, when we first met, and I wanted to kind of explore that here with you today.
[00:01:22] Andrew Zigler: So could you maybe tell me about your time, like building assistance for the last 16 years, um, at Maluuba and Microsoft and beyond?
[00:01:31] Andrew McNamara: Yeah,
[00:01:31] Andrew McNamara: Yeah.
[00:01:32] Andrew McNamara: just having, you know, gone from that scrappy startup back in, you know, 2010, even starting the personal assistant before Siri came out, uh, which was crazy. So when Siri did come out, we were thought we were in a lot of trouble. But, you know, being pretty scrappy, we ended up getting some pretty big licensing deals, with Samsung, Microsoft. Or not Microsoft, Samsung, lg, their phones and TVs, and even Blackberry at the time. Uh, and then, you know, through that we kind
[00:01:59] Andrew McNamara: you [00:02:00] started.
[00:02:00] Andrew McNamara: lab and then eventually got acquired, like you said, and we became, you know, Microsoft
[00:02:05] Andrew McNamara: Microsoft
[00:02:05] Andrew McNamara: And then we got back into, you know, so we were doing research
[00:02:09] Andrew McNamara: research
[00:02:09] Andrew McNamara: and natural language.
[00:02:10] Andrew McNamara: Then got back into the product side of things with the chat with, you know, Sydney now called co-pilot. Um, back then it was called Sydney and Bing Chat, but. Yeah, the mix of research and engineering has just been like super helpful from that whole scrappy startup to big company and then bringing both kind of experiences here to Shopify.
[00:02:29] Andrew Zigler: And it's really interesting how your background spans this applied research world. And you know, it's not every day that you really talk with somebody who comes from applied research. And often when I do, they fall onto one side of like, they're really strongly in the research camp or they're really strongly in the like applied camp, the building, What about you as an applied researcher? We've had, we've had many of them on Dev Interrupted. I'm kind of curious like what, how, how it is that you approach your work philosophy.
[00:02:54] Andrew Zigler: Yeah, I
[00:02:54] Andrew Zigler: Yeah. I think one of the big things from being both
[00:02:56] Andrew McNamara: the, you know, the
[00:02:57] Andrew McNamara: you know,
[00:02:58] Andrew McNamara: research side
[00:02:58] Andrew McNamara: side and Microsoft[00:03:00]
[00:03:00] Andrew McNamara: and then, you know, the scrappy engineering and applied research side in the startup at Maluuba was, uh, like
[00:03:06] Andrew McNamara: like just the value of valuations,
[00:03:08] Andrew McNamara: that, how that's really shaped, kind of
[00:03:10] Andrew McNamara: everything.
[00:03:11] Andrew McNamara: done moving forward.
[00:03:12] Andrew McNamara: Valuations are big in the research
[00:03:14] Andrew McNamara: They're also big on the
[00:03:15] Andrew McNamara: product side
[00:03:16] Andrew McNamara: but they're
[00:03:18] Andrew McNamara: they're hard to get right.
[00:03:19] Andrew McNamara: in both cases. And I think with LM these days and, and vibe coding, you know, and whatnot, proper evals seems to be kind of falling off a lot of people's radars, uh, honestly. So I think it's just like a huge value that sometimes people are missing these days.
[00:03:35] Andrew Zigler: I'm really glad you're touching on evals because this is a topic that I've been learning more about recently, more in depth, and it's something that I know many folks building in the space are really caring about as they start to build these products, you know, production ready applications, uh, understanding the decisions that they make to, to quote unquote improve it.
[00:03:54] Andrew Zigler: You know, does it actually improve it, does it make it better? And what about it? Uh, did improve. And why? And you know, Andrew, you [00:04:00] recently gave a talk actually at the International Conference on machine learning about building production ready Agentic systems. And it touched on this need for like a robust eval system for tools.
[00:04:10] Andrew Zigler: And so I wanna know from you, you know, what is VIBE testing compared to what you are talking about and how can folks like level Up to reach those real evals?
[00:04:18] Andrew McNamara: yeah, it's a great point. And, and yeah, that talk at ICML actually gave, there's a lot of great discussions afterwards, both in the talk of people sticking around and then coming by the Shopify booth after and just having lots of deep discussions on it. So I think it's. is a huge topic these days and, and it's super important. It's a big part of machine learning. And LMS have really changed a lot about machine learning specifically around, you know, validation, testing and evals. And in my opinion, like LMS have almost polarized evals, like everyone used to have a standard way of doing evaluations. You know, you split
[00:04:53] Andrew McNamara: Split your training data,
[00:04:54] Andrew McNamara: or you
[00:04:54] Andrew McNamara: split up your data into training and validation,
[00:04:57] Andrew McNamara: and
[00:04:57] Andrew McNamara: and then you
[00:04:58] Andrew McNamara: and then you
[00:04:58] Andrew McNamara: train, and then you test, and then you have [00:05:00] this new
[00:05:00] Andrew McNamara: or golden set. That would be
[00:05:02] Andrew McNamara: would be hitting developers. Then you test that on
[00:05:04] Andrew McNamara: after the model was done, training and, and
[00:05:06] Andrew McNamara: and see how you're doing.
[00:05:07] Andrew McNamara: kind of optimize,
[00:05:09] Andrew McNamara: Optimize
[00:05:10] Andrew McNamara: But these
[00:05:10] Andrew McNamara: days
[00:05:11] Andrew McNamara: LMS have completely changed evals. You can
[00:05:13] Andrew McNamara: make insanely
[00:05:14] Andrew McNamara: evals with, you know, LM as a judge or even agents as a judge, which was a paper that I saw at ICML. but you can
[00:05:21] Andrew McNamara: Can also get away with
[00:05:22] Andrew McNamara: testing and just, asking agent,
[00:05:24] Andrew McNamara: Hey,
[00:05:24] Andrew McNamara: you rate this from
[00:05:25] Andrew McNamara: can this from one to 10
[00:05:26] Andrew McNamara: LMS are so good that. just try a couple things and it works, and so you kind of ship it. So we've gone from like this
[00:05:32] Andrew McNamara: variable?
[00:05:33] Andrew McNamara: Principled approach in classic ML to LM enabling super powerful evals. then LMS also work so good that some people kind of just vibe, test and ship. So yeah, it's, it's an interesting spot that we're in right now. Very polarized.
[00:05:47] Andrew Zigler: Yeah, very polarized. And you're, you're kind of calling out some of the, the, I think the pitfall here of, you know, you can use, you can leverage AI so much to get so much of that done, uh, that along the way you kind of lose the rigidity of [00:06:00] like the benchmark and what it's looking for. It's like, sure, you're getting that result faster, but what is like the efficacy, the accuracy of that result that starts to get thrown.
[00:06:09] Andrew Zigler: Into question. Right. And so, uh, this actually brings me to interesting question because you're working on a product that's, you know, it's bringing ai, it's bringing agent work to really close to a huge customer base. To a huge user base. Right? So what's like the hardest parts of getting a system like that to be production ready, is it the evals?
[00:06:28] Andrew Zigler: Is it something else?
[00:06:30] Andrew McNamara: yeah, I think evals are probably the most important thing to get a production ready and probably the hardest because. Uh, it's, it's pretty easy actually to get an agent up and running these days or any kind of product. Uh, with LLMs, like for example, what we built in the first two years at Maluuba, I could literally do in an afternoon, like a day at the most, I could rebuild Maluuba, which is like insane on how far we've come in the last 15, 16 years. Uh, but for it to be in production, you have to have very high [00:07:00] trust that. What you change is making it better across the board, and you really need good evals, uh, to do that. And it's really, it's
[00:07:09] Andrew McNamara: Especially difficult
[00:07:10] Andrew McNamara: conversational setting.
[00:07:12] Andrew McNamara: But I think
[00:07:12] Andrew McNamara: It's.
[00:07:13] Andrew McNamara: you know, it was a big part of the talk at ICML conference on how to, you know, really make your systems production ready by having, uh, very strong evals.
[00:07:22] Andrew Zigler: And you know, I'll make sure we include a link to that talk. Um, if we can share it, you know, to our audience that we folks can go check it out. Uh, just because I, I, I know people are gonna want to try to. Learn from this to replicate for their own tools that they're building. A lot of, a lot of folks who listen to Dev Interrupted their, building, they're experimenting in this space.
[00:07:40] Andrew Zigler: And, you know, ma many of us are experimenting internally in terms of improving our own workflows and seeing how it impacts, like things like software delivery. Uh, but there are many of us who are also building and shipping AI products and I know that, understanding the impact of the changes that you make, especially when we're talking about natural language prompt changes and.
[00:07:59] Andrew Zigler: Calibration over [00:08:00] time. This eval system is such a critical component, so I know everyone is just like really hungry to learn more. and moving on from that like, like problem space of like understanding the impact of the tool. And let's move into a little bit about the tool itself and talk about your work building sidekick.
[00:08:15] Andrew Zigler: And I wanna start at the top. Uh, you know, for some folks listening to this, this is probably their first time hearing about Sidekick. What's the vision behind it?
[00:08:24] Andrew McNamara: Yeah, sidekicks
[00:08:24] Andrew McNamara: Kick's vision is that it's basically you're
[00:08:27] Andrew McNamara: co-founder or your
[00:08:29] Andrew McNamara: eCommerce expert
[00:08:30] Andrew McNamara: doesn't judge you. You can feel comfortable to ask any question get expert results, I think like
[00:08:37] Andrew McNamara: like around why
[00:08:37] Andrew McNamara: Why it
[00:08:38] Andrew McNamara: started,
[00:08:39] Andrew McNamara: how it
[00:08:39] Andrew McNamara: started.
[00:08:40] Andrew McNamara: that kind of led to this vision was, I've heard Toby, it was kind of before my time at Shopify, but I've heard Toby, uh, talk about this, uh, before, which was, uh, there was, and I'm not gonna get the numbers exactly correct here, but basically studies or
[00:08:56] Andrew McNamara: User questionnaires
[00:08:57] Andrew McNamara: are
[00:08:57] Andrew McNamara: signing,
[00:08:58] Andrew McNamara: on whether they,
[00:08:59] Andrew McNamara: they have
[00:08:59] Andrew McNamara: [00:09:00] know, have an e-commerce or an entrepreneur expert
[00:09:03] Andrew McNamara: in their
[00:09:03] Andrew McNamara: they can
[00:09:03] Andrew McNamara: they
[00:09:03] Andrew McNamara: questions to or not. When they're
[00:09:06] Andrew McNamara: when they're starting a
[00:09:07] Andrew McNamara: And then when
[00:09:08] Andrew McNamara: when.
[00:09:08] Andrew McNamara: start the store, if we check back on that set of users that, that answered this question or are part of the study, how
[00:09:13] Andrew McNamara: How successful
[00:09:14] Andrew McNamara: right now? Are they still
[00:09:15] Andrew McNamara: is store right now,
[00:09:16] Andrew McNamara: Did they get their first sale, et cetera. And it
[00:09:18] Andrew McNamara: and it was
[00:09:19] Andrew McNamara: like
[00:09:20] Andrew McNamara: like very telling results,
[00:09:21] Andrew McNamara: the numbers aren't exact here,
[00:09:23] Andrew McNamara: but
[00:09:23] Andrew McNamara: it was
[00:09:24] Andrew McNamara: it was something like
[00:09:24] Andrew McNamara: 80% of
[00:09:25] Andrew McNamara: people.
[00:09:26] Andrew McNamara: had. An entrepreneur expert in their life or an e-commerce expert that they could reach out to, still had stores running and successful stores going. I think, like almost al, almost everyone who didn't have that in their life, like never finished completing their store,
[00:09:43] Andrew McNamara: Store they never
[00:09:45] Andrew McNamara: made
[00:09:45] Andrew McNamara: made for sale
[00:09:46] Andrew McNamara: or their
[00:09:46] Andrew McNamara: or their has shut down.
[00:09:47] Andrew McNamara: been shut down.
[00:09:48] Andrew McNamara: Uh, again, so I don't have the exact numbers, but it was it was a pretty telling story that
[00:09:53] Andrew McNamara: Having somebody who
[00:09:54] Andrew McNamara: somebody who
[00:09:55] Andrew McNamara: out to answer any question
[00:09:56] Andrew McNamara: question.
[00:09:56] Andrew McNamara: want, ask anything, feel free, like you can ask anything. It was just like [00:10:00] so critical to, to people starting a business. And that's, that's what Sidekick is.
[00:10:05] Andrew McNamara: That's the vision behind it. That's what we wanna provide with Sidekick.
[00:10:08] Andrew Zigler: Wow. So that's like an incredible signal. Looking at those who get the help of an entrepreneur or a resource in their life that's familiar with business workings, how much more successful they are, the discrepancy between them and those who can't. And so that sets the scene perfectly for Sidekick being this.
[00:10:25] Andrew Zigler: AI assistant within your selling platform that is an expert on not only the platform itself, but specifically your business. And it becomes, like you said, that kind of co-founder. So, you know, we're talking about Shopify here. Shopify has a huge user base and a huge, Spectrum of users, right?
[00:10:43] Andrew Zigler: From anyone selling like homemade candles to, you know, in some cases multimillion dollar direct to consumer brands. So how do you design a tool like Sidekick to work for both of those groups? sidekick.
[00:10:55] Andrew McNamara: has, uh, it is an interesting problem to solve and Sidekick does have a massive range [00:11:00] of skills ranging from helping you design your logo, generating your theme, basically from scratch, which is something that we just launched a couple months ago and is like, new merchants are getting huge, huge value outta that. Uh, all
[00:11:12] Andrew McNamara: All the way to like complex
[00:11:14] Andrew McNamara: analytics,
[00:11:15] Andrew McNamara: which
[00:11:15] Andrew McNamara: can run on or get
[00:11:17] Andrew McNamara: get data
[00:11:17] Andrew McNamara: any single piece of data that's related to your business. So we find users are
[00:11:22] Andrew McNamara: like ramping up
[00:11:23] Andrew McNamara: their stores way quicker and getting to their first sale much faster.
[00:11:27] Andrew McNamara: and then we also
[00:11:28] Andrew McNamara: big merchants, you know, are talking about how they're getting these very deep and
[00:11:32] Andrew McNamara: actionable insights about their store.
[00:11:34] Andrew McNamara: That was virtually
[00:11:35] Andrew McNamara: impossible.
[00:11:36] Andrew McNamara: to get before because, you know, there's the preset graphs, you can run custom queries, but. Like just
[00:11:42] Andrew McNamara: Just having this
[00:11:43] Andrew McNamara: know, agentic loop and reasoning and planning able
[00:11:45] Andrew McNamara: to have you know,
[00:11:48] Andrew McNamara: access to these low,
[00:11:48] Andrew McNamara: level tools that can run queries for you and, and get these insights has just been huge.
[00:11:52] Andrew McNamara: So we're seeing
[00:11:53] Andrew McNamara: incredible adoption
[00:11:54] Andrew McNamara: from both new
[00:11:55] Andrew McNamara: new users.
[00:11:56] Andrew McNamara: to first sale quicker, and massive companies and [00:12:00] businesses getting like super deep and actionable insights on their store.
[00:12:03] Andrew Zigler: Yeah, it's really interesting 'cause like for a small business, it becomes their concierge. Like, I don't know how to change this thing in my store. You go do it for me. You're the expert on the platform. And then like, for the big, company, you know, they're like, they get the, a free data scientist out of the box.
[00:12:16] Andrew Zigler: You know, tell me about this kind of user or, or this kind of customer. Like, help me understand this slice of, of my. Already pre-existing user base or a customer base, which is pretty cool.
[00:12:25] Andrew Zigler: Hmm.
[00:12:26] Andrew Zigler: And, and you know, there's a lot of challenges, I think too, in, in, in building something so close to like an e-commerce workflow.
[00:12:32] Andrew Zigler: You're talking about like monetary transactions, building a business and a livelihood, and people's, you know, lives are invested in this and, you know, how do you, uh, how do you bring engineers close to that problem at Shopify? How's that part of like your engineering culture?
[00:12:45] Andrew McNamara: uh.
[00:12:45] Andrew McNamara: question too. I think like one of the biggest challenge we have is how do we know that we're providing the right value to merchants? And then like you say, how do we make sure that the engineers on our team are working on the rec right thing to, to bring that value you know, not sound like a [00:13:00] broken record.
[00:13:00] Andrew McNamara: But I, I think there's two things here. Like one is definitely evals because like, you know, if you go and watch that ICML talk on, on evals, one important thing is a ground true set. Which is where we, which is what we're calibrating our judge to. You can think of it, you know, I won't get super deep into now, but you can think of it as like basically your specs. and then, so we're
[00:13:19] Andrew McNamara: We're trying to, the judge to pretty much look at a conversation and say, this is,
[00:13:24] Andrew McNamara: perfectly or not? And did a good conversation happen? Um, and
[00:13:28] Andrew McNamara: and when we created the L judge,
[00:13:31] Andrew McNamara: A very
[00:13:31] Andrew McNamara: very important part of it. People,
[00:13:33] Andrew McNamara: a lot
[00:13:33] Andrew McNamara: a lot of people realize, especially
[00:13:35] Andrew McNamara: ICML, I was talking to people and they kind of had a different view on this, but
[00:13:38] Andrew McNamara: I think it's critical that your LLM judge or your
[00:13:41] Andrew McNamara: evaluation does
[00:13:42] Andrew McNamara: not actually know.
[00:13:43] Andrew McNamara: What features you support.
[00:13:44] Andrew McNamara: So yes, it'll look at your specs and judge conversations based on that, but it
[00:13:48] Andrew McNamara: Also
[00:13:49] Andrew McNamara: it itself doesn't know
[00:13:52] Andrew McNamara: what
[00:13:52] Andrew McNamara: you support and what you don't. Because we
[00:13:54] Andrew McNamara: we want.
[00:13:55] Andrew McNamara: To look at like, did we fulfill the user's goals? Was the merchant happy?
[00:13:59] Andrew McNamara: And [00:14:00] uh, if we just
[00:14:01] Andrew McNamara: if, if
[00:14:01] Andrew McNamara: sidekick,
[00:14:02] Andrew McNamara: says uh, we don't,
[00:14:03] Andrew McNamara: support that uh, and we can't, grade that as basically,
[00:14:07] Andrew McNamara: a high mark.
[00:14:09] Andrew McNamara: So conversations that people are having that
[00:14:11] Andrew McNamara: we don't do what they want, marking that as low, and then as we add more and more features, Suddenly. these scenarios that were marked as low by our evaluations
[00:14:20] Andrew McNamara: are Suddenly.
[00:14:21] Andrew McNamara: being marked as higher. So then we're seeing the score go up. And we're moving the score up in a positive direction with every single change we do.
[00:14:28] Andrew McNamara: So we know that we're adding more and more value to merchants and supporting the right things, basically. So, yeah.
[00:14:35] Andrew Zigler: really cool. It's almost like you're pinning user stories right there. When they come in through the experience and the eval flags, it was like, it's failed, wasn't because of whatever, it's because you know, there's, it's not on the platform, there's not tools for it. And then as the engineers build and fill out those problems and you have a catalog of things you need to build, right?
[00:14:51] Andrew Zigler: These are user stories that people wanna do, and then you build them and then you get out of the box free evaluation of. That user story is now in flight, and you can see the eval of it [00:15:00] because you identified it that way in the first place.
[00:15:02] Andrew McNamara: yeah, yeah. And
[00:15:03] Andrew Zigler: Yeah.
[00:15:03] Andrew McNamara: to build like lowest level tools possible, like natural language to, you know, query language that can access your data, natural language to something else that can access your store, and then people end up using it. In, you know, creative ways that we didn't predefine, but just because we gave low level access to tools, they, you know, they're doing cool things.
[00:15:21] Andrew Zigler: You're giving 'em like the primitives of the Shopify system to be able to manipulate. And so what are, um, let's talk about that. Like what are some surprising use cases? You're seeing people get out of it right now when they, when they turn on sidekick?
[00:15:33] Andrew McNamara: I think just
[00:15:34] Andrew McNamara: Yeah. Think.
[00:15:34] Andrew McNamara: research and getting insights out of it and like one of the things we see. Happening, you know, on Twitter and social media is that people are sharing these huge, massive prompts with each other and like getting it to do deep research, like very long prompts. Like it's insane.
[00:15:51] Andrew McNamara: Like you think it's attached system while they're just talking to it, but they're writing massive, massive prompts and then they're sharing it with each other. And it's even evolved to somebody made a [00:16:00] repository so that they have all these sidekick prompts in it, and then they're sharing this repository with people
[00:16:04] Andrew McNamara: Yeah,
[00:16:05] Andrew McNamara: can go in and copy, paste and put in sidekick. So then like from our
[00:16:09] Andrew McNamara: our end.
[00:16:09] Andrew McNamara: merchant access obsessed and building in the open, we just launched a feature where you can share a prompt and it just gives you a short link. And then
[00:16:18] Andrew McNamara: Then you
[00:16:18] Andrew McNamara: that short link and it'll open your store, pre-fill it, and then you can look at the prompt, personalize it for your store, and then just easily run it.
[00:16:26] Andrew McNamara: So it's this, you know, this back and forth, like we're learning how people are doing it, we're learning what merchants are doing, how, what their journey is like, and then. We're reacting to that and building for them. And so it's just this great relationship we have with merchants, you know, to kind of build what they need and, and keep, you know, keep facilitating them and, and helping them on their journey.
[00:16:45] Andrew Zigler: Yeah, it speaks to the high value of what they get out of the system. And it's funny that, so out there you're saying there's like an, an awesome sidekick, you know, whatever repo that, that has all of the prompts. It's, it's really cool when you see the, that not only the emergent use cases, but then the emergent prompt [00:17:00] engineering context, engineering community around it.
[00:17:02] Andrew Zigler: Uh, you know so much you can learn from that kind of customer base.
[00:17:06] Andrew McNamara: to see. Yeah,
[00:17:06] Andrew Zigler: Yeah. And so, you know, it, we, we can't talk about all of this in your time at Shopify without touching a little bit about what it means to be, you know, all in on AI at Shopify. This is something we, we hear from, from, from the top at Shopify and been in the news many times.
[00:17:21] Andrew Zigler: And I wanted to ask you, just like as someone who works at up applied position, applied research right on top of this problem set, what does being all in on AI look like for you at Shopify? Yeah.
[00:17:32] Andrew McNamara: Yeah, I think just,
[00:17:33] Andrew McNamara: You know,
[00:17:34] Andrew McNamara: you know, the best
[00:17:35] Andrew McNamara: the best way, what it looks like to me, even if I just look at Shopify,
[00:17:38] Andrew McNamara: people, it's, it's
[00:17:39] Andrew McNamara: it's about how they,
[00:17:40] Andrew McNamara: reflex re reflexively they reach for ai. Like is ai. Anytime you have a task, you know, you gotta make a tweet. Maybe you're using ai, you gotta write an email, you're probably using AI to at least like read it over and slightly change it. Or there's tons of MCP tools out there that you can use to, your, whatever LM you choose to use, which like all of them are available at [00:18:00] Shopify. And then you can plug in different tools to it. And just like almost
[00:18:03] Andrew McNamara: Everything you do
[00:18:04] Andrew McNamara: you
[00:18:04] Andrew McNamara: reach.
[00:18:05] Andrew McNamara: AI first, and
[00:18:06] Andrew McNamara: That's,
[00:18:07] Andrew McNamara: that's how I
[00:18:08] Andrew McNamara: how I operate the day.
[00:18:09] Andrew McNamara: And, uh, and a
[00:18:10] Andrew McNamara: And a lot of people at Shopify
[00:18:12] Andrew McNamara: know, not just tech people, but you know, everyone is, is being very reflexively reaching
[00:18:17] Andrew McNamara: reaching,
[00:18:18] Andrew McNamara: And I think
[00:18:18] Andrew McNamara: that's the
[00:18:19] Andrew McNamara: that's the biggest
[00:18:19] Andrew McNamara: biggest indicator.
[00:18:20] Andrew McNamara: of how all in, you know, we are on ai. And I think, you know, even at what we call Summit, which is where the whole company got together. a week long conference, there was like, you know, Shopify AI
[00:18:33] Andrew McNamara: ai.
[00:18:33] Andrew McNamara: Village, you know, there was talks throughout the whole week on how people are using ai, how they can help each other, how teach other people how to using it. It's, it's a It's a great.
[00:18:43] Andrew McNamara: here at Shop Shopify with like, using AI and, and trying to make it more effect, make you more effective and more efficient.
[00:18:49] Andrew Zigler: That's really great. I, I, I love hearing about people's experimental cultures within other engineering orgs or just their orgs at large about experimenting and trying new stuff out, especially like the [00:19:00] sharing great stuff. I've, we've talked with a lot of guests recently, uh, who've really shared about like that, that internal like AI thought group that's like presenting every week about those cool use cases that they find and there's so much value there, um, in having those conversations that can get applied to.
[00:19:14] Andrew Zigler: Even things that you're building like right now with Sidekick, just really cool to be able to double down on all of that. And you know, this whole conversation has just like made me think, gosh, I wanna like, I wanna open a Shopify store and just put a robot in charge of it. It's like it's inspired, it's in, it's kicked off a lot of ideas, but also it's like the way that it mix and matches its tools is so interesting.
[00:19:33] Andrew Zigler: It almost kind of bridges into what you could call like vibe entrepreneurship and it's a funny phrase that. Reflect on, especially in a world where like, vibe coding is already so polarizing.
[00:19:44] Andrew McNamara: Yeah.
[00:19:44] Andrew Zigler: Uh, but I think that there's a lot actually to talk about there. and I, I think it's interesting to think about how, uh, these tools can interact with like, the future.
[00:19:53] Andrew Zigler: So how can AI tools like Sidekick enable a new generation of entrepreneurs? Like you talked about that survey [00:20:00] group, right? The ones that did and didn't have access, you know, uh, how is Sidekick gonna change things for them?
[00:20:07] Andrew McNamara: Yeah, I it's just so much easier become A merchant and to start a business these days than to get to your first sale, like measuring of like can people how their first sale? I think like Sidekick is making a massive impact on that. So, you know, Horizon theme, which is a theme we launched a couple months ago, and how easy it is to use Sidekick to just create customized store for you through just vibe entrepreneurship or vibe, you know, creating your store, you know, even helping you make a logo or even helping you decide what products to sell and how to start your business, um, or what your business should even be. Like. All
[00:20:43] Andrew McNamara: these things are now on the table for vibe entrepreneurship and like you
[00:20:47] Andrew McNamara: said.
[00:20:47] Andrew McNamara: Popular for Vibe coding.
[00:20:49] Andrew McNamara: But, uh, yeah, I think vibe entrepreneurship entrepreneurship and, uh, you know, is, is is something that is, is coming especially with, you know, tools like Sidekick.
[00:20:59] Andrew Zigler: [00:21:00] Yeah. And so where do you think like this, this trend could kind of go, like you built something like sidekick. What, what eventuality do you think that that would take us to when something like Shopify, do you have any ideas?
[00:21:12] Andrew McNamara: it again, it's one of those things where like there's gonna be emergent use cases where you, where you, you
[00:21:17] Andrew McNamara: don't even,
[00:21:18] Andrew McNamara: it could be a possibility
[00:21:19] Andrew McNamara: yeah.
[00:21:20] Andrew McNamara: You know, it's helping you start your business, run your business, uh, and do everything. I don't think it'll ever get to like a completely thing.
[00:21:29] Andrew McNamara: I think the humans are like, humans are so important to this. It kind of makes me think, like at the I CML talk there was like, you know, we were doing reinforcement learning for training and it basically reward hacked that and optimized on something we didn't want. But it was hard to like see that. And I think like the same thing could happen if you go like completely. Vibe entrepreneurship. Um, you never know like what it'll consider high rewards that may not be best in line with your business. So I think
[00:21:56] Andrew McNamara: I think it's like, it's definitely balancing you know,
[00:21:59] Andrew McNamara: having [00:22:00] human
[00:22:00] Andrew McNamara: entrepreneur mindset, that creativity, that judgment is so critical.
[00:22:03] Andrew McNamara: But then having AI
[00:22:04] Andrew McNamara: AI sidekick.
[00:22:06] Andrew McNamara: Is just like helping you jumpstart your business and really, you know, move closer to running it. As autonomous. Um, you know, I don't, I don't, I don't think getting autonomous business will be super easy, but, uh, you know, helping you get started and out what to do and then making judgment calls on it, I think is just like an incredible opportunity that we're in right now.
[00:22:28] Andrew Zigler: Yeah. I, I especially think too, for the interesting thing about something like an autonomous business, even if that's outside the realm of what something like sidekick is, is, you know, then you're talking about plugging AI into creating like value. Like societal value or monetary value. And a lot of times when on Dev Interrupted, we, we focus really specifically on plugging AI in to get that engineering value to, to ship that better product, faster, safer, more secure, and more impactful for customers.
[00:22:55] Andrew Zigler: Right. But it's just a, a. A big eyeopener that like, you know, [00:23:00] we're solving these problems here in the engineering world, AI can solve a lot of problems in a lot of realms, including things like businesses and, you know, generating capital or whatever they may be. But I agree with you that humans will always be vital in that loop and.
[00:23:12] Andrew Zigler: You know, there are things that get optimized for that we always have to keep our eyes open for. So, uh, it really sets the scene for an interesting future. But Sidekick has named Sidekick for a reason. It's gonna stay. I think your, your, co-founder, your assistant, your technical, um, you know, purveyor for the time being, but, uh, you know.
[00:23:32] Andrew Zigler: This has been like a super fun conversation to talk about how you are approaching building like business assistance, especially since your background and applied research spans, you know, talking about and building these tools. Before they were really more household names and people cared about them. So we really got to see that perspective of how you've, of, of what matters to you still all these years later, what matters now more than ever.
[00:23:54] Andrew Zigler: And so I think the, the bottom line of this talk is. Evals, evals, evals for what people should care about. So we're gonna drop [00:24:00] resources for that. But, uh, I, I, I also now want to go back to the drawing board and, uh, come back up with a Shopify story idea with, with Sidekick. I gotta see what this thing can do.
[00:24:09] Andrew Zigler: Um, but before we wrap up, you know, where can our audience go to learn more about what you're working on and, and what you do, Andrew?
[00:24:15] Andrew McNamara: I mean, I'm pretty active on Twitter and I love building out in the open, and I love, you know, I think probably every tweet that's mentioned, like Shopify and Sidekick in it together in the last eight months, I've basically read and either interacted with in some way. So, you know, I, I love talking to merchants.
[00:24:31] Andrew McNamara: I love seeing what they're building, what they're doing with Sidekick and, and I love building in the open. So if you. You know, maybe share my Twitter link or something and people can follow along and, and see our updates and, and what we're doing and what we're thinking about.
[00:24:44] Andrew Zigler: Yeah, no, we'll definitely share them with our listeners and,
[00:24:47] Andrew Zigler: so you, our listeners, thanks for, for joining us for this conversation. It's been really interesting one for sure. So be sure to go follow and join this conversation online. You know, we're having it right now in your ear, but we're gonna continue talking about it on LinkedIn.
[00:24:59] Andrew Zigler: Uh, you can [00:25:00] check out Andrew's talk as well that he recently gave ICML, uh, that's kind of the grounded. For a lot of the stuff we covered today. Uh, but more importantly, go check out Sidekick and, uh, you know, see what it can do. I know I'm gonna go, uh, explore it after this and I, if I kick off a store, Andrew and, uh, have my little robot co-founder try to sell something with me, um, I might, I might loop you back in to gimme some, uh, tips or otherwise point me in that right direction.
[00:25:22] Andrew Zigler: So I appreciate you, uh, you teaching us about it. It'll be cool. So, uh, you know, thanks for listening, uh, to Dev Interrupted, and we'll see you next time.
