Your early tech stack decisions won’t ensure long-term success, but they can certainly set you up for long-term failure.
Pragmatists rejoice! This week’s episode of Dev Interrupted features Sam Lambert, CEO & President of PlanetScale, a tech leader known as the ‘Oracle of Pragmatism’.
In a winding conversation that touches on Sam’s time at GitHub, where he helped the then 40th most-trafficked website in the world run on just 2 servers, to his experience working at Facebook where he learned that you don’t need to sacrifice quality in order to move fast, this episode has the insights you need to make straightforward, no BS decisions about your tech stack.
Or as Sam says, learn how to avoid “the new hotness” in order to build a culture that reflects the often boring decisions that make or break a company.
Episode Highlights:
- (2:18) Lessons in pragmatism from GitHub
- (6:49) Sam's experience at Facebook
- (10:40) Moving fast without sacrificing quality
- (17:12) Building teams that are passionately pragmatic
- (26:07) Why people are scared of their database
- (29:19) What is the serverless movement?
Episode Transcript (disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
Dan Lines: Hey, what's up everyone. Welcome to Dev Interrupted.
I'm your host Dan Lines. And today I'm joined by Sam Lambert, CEO of PlanetScale. Sam, welcome to the show.
Sam Lambert: Hi, thank you for having me.
Dan Lines: Awesome to have you on here. You've had a really successful career working at places like Facebook and GitHub, where you were the VP of engineering. You ran a team of 350 people through it all.
You've been called an Oracle of pragmatism, coolest name ever for your ability to identify what you call sensible tech and stick. In fact, you've said that most successful companies are boring. So that's where I want to start us out today. I think you have a little story about your time at GitHub, some of the pages were, trafficked by tons and tons of people, but maybe, only running on a few servers or something like that.
What do you got for us?
Sam Lambert: So GitHub was a very pragmatic company when it came to our tech, the majority of what people think of as GitHub is a Ruby on rails app talking to a, my database and then servers at store Git. And we tried to keep it very simple. The story there is that, GitHub pages, the actual product that hosted millions and millions of individual developers websites for the longest time ran on two servers that ran EngineX with a multimillion line file. And at one point I think the page GitHub dot IO domain name was like github.com is the 40th most traffic site in the world. And GitHub pages was like the hundredth.
And so we were running a top 100 site off of two servers, essentially because we allowed people to build static sites and we deployed it really simply. And we had this culture in the early days of being pragmatic and only really building what is necessary to iterate ship and get stuff done and not over complicating things.
And we had to shut out a lot of noise. It was like microservices and like over architecture and all these kind of things. People were obsessing over and we just kept it ridiculously simple. And we managed to take, the site to the scale of 70 million users, , running plain old, my sequel, a bit of a test and Ruby on rails.
It really taught me a lot of lessons about the power of pragmatism and how quickly you can ship. And when we started a plant scale, we did the same thing. Plant scale is a rails app. People talked to us all the time talking to a customer the other day, and they were like, by the way, you ship faster than any other product we bought.
Why is that? And I was like, again, it's the combo of Ruby and rails of my, and. A bit of magic and just iterating super quickly. So yeah, very I'm obsessed with pragmatism, keeping things important, understanding the minimal viable thing that needs to be done.
Dan Lines: Yeah. That's amazing. Is that GitHub where you got your name?
The Oracle. If you're the Oracle of anything, that sounds good to me. I don't care. I, what, you're the Oracle of, where did that come?
Sam Lambert: I don't know, this is new to me. It's nice that people are calling me some form of article.
Dan Lines: I try made an impression.
Sam Lambert: Somebody I affected somebody out there which feels, great.
Dan Lines: Yeah. You brought up a really, I think interesting point when you started talking about planet scale, actually. And you talked about delivery or momentum, or how fast you're able to. Can you give us like a few more insights about that and how it relates to, some of that simplicity that you were talking about from an engineering perspective,
Sam Lambert: it starts with culture.
Our engineering team has a great culture, fantastic people. I can't take credit. Any credit for that shipping is they do so much intentional work, whether it's cultural or technical, to keep that going, you have to have it almost dissatisfaction with things slowing down and things getting in your way. And we have people in the engineering team that kind of rather than wanna astronaut big, long running products projects they wanna.
We just wanna get something out there and iterate on it. And we just think to ourselves, like, how do we, descope something, how do we boil something down to its essence so that you can get something out there in front of users, get the feedback and iterate. And I read like competitors' blog posts about how they built something and it's they started with this giant vision for this thing.
And just slowly ground towards it, without truly knowing that it was something that would be good or useful or viable for their users. We take the opposite. Like our backend is incredibly stable, incredibly robust, and has supported some of the larger sites on the internet. We don't, we're not fast and loose with the back end.
The thing that serves the queries, but when it comes to the, like the product, the automation and the final experience, we are experimental. We build quickly, we push quickly and it comes down to a lot of our culture and just the way we are and what we build and just being dissatisfied.
Dan Lines: Yeah.
You, you also spent some time at Facebook, which from my understanding essentially runs on PHP. My sequel. , is that where you picked up the, this, pragmatism, the sensible, type decisions? Like where did you gain that from?
Sam Lambert: So I was, I had a short student at Facebook and I wouldn't say I gained it there.
I think I really, it really became clear to me. At GitHub was probably where it had the most effect on me. , but Facebook definitely. Yeah. So they've done a lot to make PHP scale. They've done a lot to make my SQL scale, but essentially, yeah, those are some of the fundamental technologies. And for a long time, Facebook has the motto of move fast and break things.
It's now move fast with stabled infrastructure, but they still have this spirit, which is opposite. Like it's very different to. Sort of fan companies. They still have this very scrappy culture of building and getting going and moving. And they still continue to iterate really quickly. Like I was there when the pandemic started and they wanted to do like group video calls and most companies like that would be a half a year.
Like they Z said, let's do this. Six weeks later, it was shipped to three and a half billion users. That's just incredible work. Yeah. At that scale deploying across millions of machines and and a just huge scale. And again, it just comes back to that scrappy hacker culture and being fundamentally being hackers.
And you are always reminded even the street that headquarters Facebook's headquarters are on is hack away. It's just, it's always. Hacking and being scrappy and moving forward, even at a huge scale. And that is the opposite of a lot of other cultures that don't move fast and don't build things for their users.
Right.
Dan Lines: We might use Google possibly as, a different type of, we've had a few people from Google on here. Amazing. Super smart. Always thinking about scale. Always having scale on the mind, but maybe not as much as the move fast and break things type mentality. How would you compare these two different mindsets or is it really a blend?
Sam Lambert: So go, I love Google. Google has built amazing things, but as end user, it is frustrating. How things slowly like Google calendar it just hasn't gotten better. Same with Google docs. Things move just painfully slowly. I think if you read about their SRE culture and their culture in general, I don't think there's any gradient in their minds between.
Ship something small to a small amount of people and it's gonna be the next Google search, right? Like it's either at scale or it's not. And even you hear about I, I knew the guy that worked on a bunch of their internal apps and even the app they use to book massages at their offices has to run in three data centers.
So it's over, it's architectured more robustly the most. Primary company's apps. And that just doesn't feel pragmatic to me that doesn't, there's one way of doing most things at Google, because there is just this ingrained sense that everything is at scale. And that means things move so much more slowly because you have to get it on, into the big shipped into the mothership.
, even though it could just get killed next week.
Dan Lines: And from a developer standpoint, there's a ton of talk now about developer experience, developer, toil all that type of things. And what do developers wanna do? They wanna merge their code? They wanna ship their code. , so a lot of the people.
That are out there talented people, I would think tend to go towards that. Okay. Am I working at a place or am at, maybe I'm at a company like planet scale, that's able to move quickly. This is amazing, but you also have to keep quality in mind. , how do you start thinking about that? Blending those two aspects together.
We wanna move fast, maybe we don't wanna break everything.
Sam Lambert: I think you can move fast without sacrificing quality when it comes to scope. You can reduce what you're doing and do it well, and do it at high equality. It's one way of doing, but I think it's a lot of it comes from what happens when you're not programming.
Like, how are decisions made? Does someone like people are planning? Like we shipped a feature last week. I love this feature. Okay. A common thing people do is they drop tables from their database, right? You wanna clear up some data you're not using this table anymore. You wanna get rid of it.
And a very common source of outages is people thinking they've cleaned up all of the code. That's accessing that table, but they haven't and they drop the table and it goes wrong. I've caused this out, I've done this, we've all done this. So we ship this feature that warns you. When you're about to drop a table, when you're about to deploy your schema, our plants go, you deploy schema as if it was code again, very oriented around shipping.
, it warns you. If that table has been queried recently and tells you, you probably gonna break those queries. If you drop this table, we shipped that feature in a day because developers felt empowered to wake up and say, I wanna ship this today. And. They knew who to talk to your decision was made and it just got done at most companies.
You'd have to write a project documentation and talk to those stakeholders and try and pitch your bosses, who don't have a clue, how computers work anymore and try and explain it and then talk to a product manager and convince them. And but if you can keep it light and everyone up the management chain knows the value of this, the innate value, they understand the product, they understand what we're here to do.
You can move so fast. So we shipped it one day. We, so we build it one day that overnight they created a quick blog post to get it ready. The marketing team picked that up. The design team designed ahead of, for it within an hour and boom features out in two days. Yeah. Database, no other database company shipped that quickly without level of quality.
It starts with having great people and it starts with those people feeling. They understand the, what it takes to get stuff done at most companies. It is this. unable, like Rube Goldberg, machine of process and mess and stroking, egos of people that feel they have to be informed and involved. And you have to drive that out of your culture.
Dan Lines: There's a few interesting things there. The first thing that you brought up is having your engineers feel like they can make a decision with the product. They know it's a good decision. They understand the product, they understand. It will save developers time. Like you're saying we've all been through that scenario.
So it was a no brainer thing to do, but it's also, it's all of that with a combination of, for example, you being the CEO okay, does Sam have to note, does Sam have to sign off on this? No, just move forward. So what I see is like a little bit of both there, some culture, but also engineers, good decision making with the product.
Sam Lambert: Exactly. They showed me they were gonna do it. I was like, hell yeah, it's amazing. Boom, go on. It's done. They knew I'd love it. Because we make it very clear. What we're here to do. The mission of our plan scale is very clear and that's how you build good developer experiences. You empower developers.
We all use those products that start off being amazing. When the company is like five developers, just building what makes sense to them. And they grow up and through failures of leadership, they just end up with bloated product teams of people that don't understand the work of developers who are allowed to boss around their engineers.
A term. I like when you hear PMs and people saying my engineers, my it's just greats me like crazy. You have this, you have this culture of misdirection towards getting towards the product. Hurts developer experience. It hurts building a genuine product. We obsess over there's other developers that use our product and the people that build it to make sure we're building things that make sense for them.
Yeah.
Dan Lines: When you're thinking about, you're building a product for developers, do you have any I guess I would call them principles or tenants for example, the feature that you just said was super obvious. I love it because I know it will save time for developers. Some, any developer that's using this, they're gonna get time back.
Do you have any way thinking about what would be. for a developer.
Sam Lambert: No, we just, we know what developers want. We are developers. We don't write, we haven't written down our kind of manifestor of what we think it takes to build this stuff. It's like taste. You can't really sum it up. You either have it, you know it, or you don't right. You look at it and it feels right or it doesn't feel right.
And it's more We have people that we know have good taste and are proven. And the way we build things ensures that we spend that time thinking about it. So our product design team are very technical. They code, they build the, the basic versions of what we are gonna build and they are, again, unreasonable enough.
So you'll see this kind of negotiation of like the backend engineers will produce how a feature works. And the front end kind of the product designers will be like does it need to be this many steps? How do we fit that into this? And it's no, it doesn't actually have to be.
And we refine it and ground it down to the simple, , story we want tell our users and the flow we want them to go through and you put it together. And developer experience is a real buzzword. Now and people think they get it. It's just it's tell everyone it used to be like, you were only cool if you had an API or whatever.
And then people would just ship an API and they didn't know why and see APIs for just random tools. And everyone wanted to try and do it, but it's unless, I think it's, you have to, it just has to be part of culture and you have to value taste, and you have to give people with those roles, a seat at a table and listen to them and empower.
Dan Lines: Yeah. When it comes down to it for you, I hear it's a lot about the people and the DNA and the people that you're bringing in and you talk about taste or having a good instinct. Hey, this is something I would use, or this is something that, you know, really. Hurt me in the past, something like that.
And we know that, you're also very into being pragmatic. So how do you build teams that are passionate about being pragmatic, at your company?
Sam Lambert: I think it goes with the technical choices you make. If you try, if you build a culture that just wants to mess around with the new hotness, unproven, whatever tech, then your culture reflects that. And then pragmatism is out the window. If you build on the stack, that is prob from default. You attract people that want to do that, right?
Like generally if they pick mature tooling and they've worked at places that have done, so they probably come with that knowledge, but we definitely test for it in, in GitHub with our interviews, we used to ask people to build solutions as part of the interview. And it was.
Sometimes people would just roll up with a simple shell script. And that was sometimes just the right answer. Or some people would build this heavily abstracted kind of mess and you'd know that they were not right for the role. You just sense it out.
It's also partly what you reward, like what gets you your bonus or your pay rise at performance review process. If it is someone building demo where that's all like.
A big mess around project that has no impact and you, and that person gets rewarded for that. It reflects in the rest of the culture, like people, joke at these fan companies that you have to do some, build a build system at Google or whatever, to get recognized, or, implement some paper or write some paper if you reward that.
That's how it happens. If you reward people, shipping, measuring the impact of their shipping and moving the needle from the company and you reward that, then. It reinforces and it becomes this kind of virtual cycle. What we do at planet scale is our head of HR. At the end of every review period, she analyzes all of the reviews and picks the traits.
The people who exceeded expectations, we only aim for less than 10% of the company can exceed expectations. If you have massive of the company exceeding expectations, then it doesn't mean anything. It has to be truly special. Yeah. It's only a, we're only a 95 person company. So nine people give or take can get that rating.
So really there's only one or two from every org or team or whatever. And she analyzes what traits and behaviors they exhibited that got them into that position. And she writes it up and post it for the whole company. So essentially the company gets an overall report of what we valued and what we saw in the performance review process that got the highest rate of people, their rating.
Does that means we then reinforce that types of behavior and it helps become part of the culture.
Dan Lines: Yeah, totally makes sense to me. A lot of the companies that, we're working with, , they are valuing shipping now, how often are we able to merge? How often are we shipping? It's actually something that is, measured.
And I love that you have that reinforcement within your culture coming from HR, that's incredible.
Sam Lambert: it's why our product looks the way it does, by the way, our obsession was shipping. When we came out to market. Tell me about that. No database company had ever done branching.
we were the first company to data branching. Or deploy requests for deploying your schema planet scale acts almost like GitHub when it comes to deploying your schema. Schema changes get in the way of shipping at every company. Some features GitHub would take 22 schema migrations done over the course of a month or two months to get something into production.
When we interview users and we talk to developers and companies, their biggest pain. Ask schema changes. In fact, one of my wife's friends, he was staying with us and he's a product manager at our company and he said, what's, what's your product do? And I talked to him about schema changes and he was like, oh, schema changes.
They are the number one reason I get told our products are delayed from our engineers. We went to solve the problems with schema changes first, and you can deploy your schema fully online, no time, no locking. And you can also reverse it. You can redeploy your schema without data loss. And so really the first database has ever been able to truly get into the Dev workflow.
Like everyone else's databases are just this big blob of state that everyone creeps around and doesn't like, wanna touch? We push. Five six schema changes a day as we push and build stuff. And you just don't think about it. And that's the number one feedback we get from our customers is my God we're shipping so much faster.
Developers just, they don't even talk to infrastructure. People to get schema changes done anymore. They just push them into production and hit merge and they know it's gonna work. They just walk away. And so that's why we focus on that. And no other company had tried to do that before. And that's what makes us really special is.
Most like we have the most scalable database out there. We've proven that with a slack and YouTube and GitHub that use the backend of our technology, but we also obsess over the daily lives of developers. Everyone else is just thinking about DBAs. Backend folks. And they're thinking about the guts of it.
And we've gone a level above that and thought about everyone who has to use this thing.
Dan Lines: Yeah, that's an amazing example. , you've said something in the past, like until the last couple of years, it's either been, you can focus on scale or you could focus on velocity, but you can't do it at the same time.
And now that's changing. Can you talk to us about how that's changing and then. Few more examples would planet scale.
Sam Lambert: Yeah. It's a moment in time, really? Where, so with developer experience, it starts with things being approachable, like easy to use, easy to get going, but it's only maintained with the technology actually being able to scale, if you can't scale, then developer experience just falls down, right?
It's all good. That cars are fast, but they have to be safe. Otherwise they portray the user in the end. And so historically in the world of databases, you either had something that was quick and easy to get going, but was ultimately a toy and would drop data and wouldn't scale. So then you lose them the most precious middle gross years of your business, replacing your database, which is as hell nobody wants to do.
Or you start with the scalable databases that are out there that are much slower to use. And I don't mean slowest in terms of query performance. To build against and iterate against. So it was an either or choice and both were wrong. You, we all talk about like getting a job done, right?
For all, for the job you shouldn't over architect, you should just use what makes sense to you and your teams. But thanks to plant scale in this moment in time. that moment in time comes from. The back end of plants go is for te which is the open source project that we maintain. It was built at YouTube and open source.
It was YouTube's main database. Everything of YouTube com comments counts, everything was sawed in the test. It was then open source and adopted by slack square, blah, blah, all it's all of these companies who have then contributed and GitHub. And that's how I got to know technology. Who've contributed back to make it better.
So we have this incredibly scalable backend. There's no other database out there that you can buy that isn't span or on some specific cloud that you would be stuck on for the rest of your life. There's no other database out there that scales and is proven to scale to a degree we can. And there's no other database as quick as I get going and build and iterate against.
So it's really is just this moment or time where the trade off no longer exists, which is very exciting.
Dan Lines: You mentioned a few interesting features, , that allow a developer to build and iterate against the database. Is there anything else that's, unique in terms of working with the database and moving fast with the database?
Not from a query perspective that you'd like to highlight.
Sam Lambert: Rewind. I think plant scale rewind is a thing that means that you shouldn't be scared of your database anymore. It's so funny. When we did user interviews at the beginning, the word fear came up a like fear. People are fearful, like scared of their database.
Yeah. It came up a ridiculous amount. Like no developer raised their hand to go and modify the database because when you get it wrong, it's danger. Things go very wrong. It's danger. At best at sound time, when something goes wrong at worse, it is data loss. Both those things are terrible. Loss of reputation, loss of business, loss of trust.
And so no one wants to do it, but PLAs go rewind, which is proprietary to us, gives you the ability to literally rewind your schema change so you can go and drop a column. For example, you imagine, you think you've cleaned up this model and that, you don't need that column anymore. You go in and you drop it.
So you do a deploy, just like I get, however, you do a poor request. It rolls into production. We drop the column, it's gone suddenly you're seeing errors. What would you do in a normal scenario right now? You would like, you would be paging people. You'd be trying to restore from a backup. That's gonna take forever.
That's not quick. You have to restore it onto a different. You have to wait for the backup, then you have to pull like you maybe add the column back, that's empty. So the queries stop failing, but now that's getting filled up and a backup has a different copy of the data. Like chaos is happening right now, while this is going on.
Plan scale, you press the button. And within one second, that column is back and all of the data that's happened in between those two periods is. We create a replication loop from the old version of the table from it's a new version without the column. So we ensure this continual loop and that there's no data missing and you come back.
So what would be hours of downtime is now a second blip of some queries failing. End of story. And that to me is incredible. I have lived through that downtime scenario. More times than I would like to remember. And so as every DBA, so as every backend engineer, we've all done this, and the fact we've reduced that now to a single button press.
It's amazing to me. I feel I just love it. And so that's, it's that kind of thing that we're really working on to make it super easy to be managing and shipping and building against the database.
Dan Lines: I love that gives a lot of confidence back. Nobody likes that panic moment. Or the fear factor.
Even when you're just telling that story, my heart was racing because it's not a fun moment to go through. So I really love that. Yeah. The last area here, Sam, just, to get, a few of your thoughts what's going on with the serverless movement.
Sam Lambert: Yep. Serverless is a really great movement.
At first I was cynical to it being a grumpy grumpy, old person being like, of course there is service. What's this about? Yes, there is servers, but I really thought about it. And servers serverless is a rebellion against how complicated and gross cloud and SAS software has become.
Before I came to plant scale, I went and looked at the competitors and what they did. And one of the darlings at the time I logged in, I set up my account. First of all, I got automated messaging from their sales team immediately trying to get me on a call that was self annoying, but I had to select how many nodes I wanted a cluster.
How many VCPS I needed how much memory I needed, what region, what cloud? I, they made me fill out a form to just get a database. And if I've got the idea for the next Stripe in my head, why do I need to care about this stuff? What you are supposed to take? You are a high price piece of SAS software.
Shouldn't you be taking this away from me? If you are doing Dropbox, if you drag a find a drop. Doesn't ask you how many storage node do you want that to be placed on for high availability, be insane from a consumer product, but why do we stand for it? In cloud technologies like SaaS, SaaS products and service for me is a movement that is really thinking about what is the core and essential job to be done of these products.
And how do you deliver that without the bloat, without the abstract, like without leaking senseless abstractions to the users?
Dan Lines: Yeah, absolutely. I think it's a really good way to describe it, but I love about the conversation that we have, had here. Sam, I can tell that you have the right mindset.
When you're building your company, the technology that you're building, moving fast, shipping fast, re reducing these annoying things and also scary things that are happening over and over again. , highly, , appreciate what you're doing here. If someone wants to get started with planet scale, how can we do it?
Sam Lambert: Hit up planet scale.com. Sign up, get going within second, you will have a database available to you. You can start putting your data and your schema in. If you are an Amazon RDS user, you can migrate to plant scale fully online. We can attach your RDS instance. We can pull the data in no dumping data, no messing around.
We were fully bringing. You can deploy your app to point to us, we will proxy the connection back to IVs. You hit the switch button, we will switch places and you will be fully online migrated. We have customers. Migrate in without a second of downtime, which, when have you been able to do that with a database migration?
So you can get going and move super fast and you can get online and just get on client scales, start building things with your dreams.
Dan Lines: Awesome. So everyone listening. Definitely check out planet scale.com. I also wanna say thank you to the more than 3000 of you who are now subscribed to our weekly interruption newsletter.
We bring you articles from the community inside information and weekly pods. And the first look at interact 3.0 on October 25th. , if you haven't already been to interact, definitely recommend it. It's a ton of fun. It's free. It's all online. You can register today. Of course we'll have any of the interesting information, the description and the links below and Sam again, thank you, for coming on here and having the right mindset of what we need to do around databases.
Sam Lambert: Thank you for having me. It's been wonderful. Really enjoyed our conversation.
Want to cut code-review time by up to 40%? Add estimated review time to pull requests automatically!
gitStream is the free dev tool from LinearB that eliminates the No. 1 bottleneck in your team’s workflow: pull requests and code reviews. After reviewing the work of 2,000 dev teams, LinearB’s engineers and data scientists found that pickup times and code review were lasting 4 to 5 days longer than they should be.
The good news is that they found these delays could be eliminated largely by adding estimated review time to pull requests!
Learn more about how gitStream is making coding better HERE.
