How Open Source Enabled Kubernetes’ Success

This article is based on an interview with Kubernetes co-founder Brendan Burns who appeared on the Dev Interrupted podcast.

The success of Kubernetes was never preordained - it took years of work.

While today it has grown to be one of the largest and most popular open source projects and become the go-to API for building cloud-native applications worldwide - it almost wasn’t even open source. 

Since its start in 2014, Kubernetes (an open-source orchestrator for deploying containerized applications) has transformed from a tiny project co-founded by three people, Joe Beda, Craig McLuckie, and Brendan Burns, into a production-grade infrastructure that powers large-scale production applications in various fields, from machine learning to online services. 

Created with the intent to simplify the task of building, deploying, and maintaining distributed systems, it allows people the world over to achieve new levels of velocity, agility, and reliability. Over the years, Kubernetes has grown into a large, successful open-source community - but it was a long journey getting there.

What is a container?

Before jumping into the history of Kubernetes, let’s first define a “container” because it often has a broad set of meanings. 

Fundamentally, a container is the process of taking an application and packaging it, building a binary representation of the pieces that make up that application like the software, configuration files, etc., and having a protocol for distributing it around the world. 

There are three pillars of what became known as containers: 

• Packaging everything so it's exactly the same, for instance, on a laptop as it is out in the cloud
• The distribution of step 1 needs to occur easily and around the world
• Creating an environment where a bug in one component of the process doesn’t affect another component

How it started

When we interviewed Kubernetes co-founder Brendan Burns on the Dev Interrupted podcast, he told us that as an engineer, he found himself gravitating towards spaces with opportunity. While working at Google, he noticed that Cloud had a lot of white space and thought it would be an attractive space to work in. At the time, he led a small team of about seven engineers and decided to transition his team over to Cloud. 

At the same time, his eventual Kubernetes co-founders Joe and Craig created Compute Engine, the virtual machine product for Google Cloud, and the three of them began to work together in Google’s Cloud division. While Joe and Craig focused on compute, Brendan worked in config management on things like CloudFormation and Terraform. 

Ideas were starting to coalesce, and the three of them were witnessing the challenges people faced trying to adopt Cloud - a far too difficult process. There were also some internal systems at Google, in particular a system called Borg, a cluster manager that’s still used today, that served as the initial inspiration for the three developers as they dreamt up Kubernetes. 

However, none of it would have been a reality without Docker.

Docker changes everything

As a prerequisite to the functionality of Kubernetes, people needed to be motivated to build container images and run containers. Docker was the startup that came along and convinced people why they should care about containment. All of a sudden, a use case existed, and an amazing opportunity presented itself. 

Docker didn’t have a lot of experience at scale, and they were focused on one machine, with a container and daemon on that machine - what they were lacking was orchestration. If a system was built that could act as a container-orchestrator, it represented not only a massive opportunity to change the market but to change the entire cloud landscape. 

When you think about what it takes to deploy an application up to the cloud or even to an on-premise collection of machines, it’s a lengthy process. It requires you to package up an application, distribute it, keep it running, have load-balanced traffic between the various pieces of the application, and an API to tie it all together. 

Prior to Kubernetes, some of these systems were in place, but nothing like today. Kubernetes is responsible for mainstreaming the idea of a distributed systems application environment for building and constructing distributed systems that span machines far and wide.

With the need for orchestration realized, the next step was selling the idea to executives.

Selling open source

Convincing people that it was possible and a good idea was pretty straightforward. There were folks at Google who understood what Joe, Craig, and Brendan were trying to do. The real battle was fighting to make Kubernetes open source. As Brendan shared in our interview, they had a lot of internal arguments at Google about it being open source. 

Mostly it came down to money and control. From a business perspective, if a product or system is massively successful and you’re the only one who can sell it, then you’re in a great position. Conversely, Brendan told us that he always felt that Kubernetes would only be massively successful if it had an ecosystem, and the best way to foster an ecosystem was to make it open source. 

This viewpoint is centered around the community that comes together to build the software. An amazing community formed early on of people who helped build docs, who helped build tutorials, who would talk about their work at conferences, and then an ecosystem of companies that were betting their whole business on the success of Kubernetes. Startups began popping up, saying things like, “Well the prerequisite for using my monitoring software is that you have a Kubernetes cluster.” All of the attention and goodwill formed a sort of virtuous cycle around Kubernetes.

Success has a way of looking easy

Soon enough, Kelsey Hightower, principal engineer for Google Cloud and Brendan’s co-author of the book Kubernetes: Up and Running: Dive into the Future of Infrastructure, came along and started doing a ton of evangelism and driving attention towards Kubernetes

It can be easy to look back and assume that it was easy, because Kubernetes just took over. It's present in every major public cloud at this point, people expect it to be in new systems. But the truth is that in those early years, it took a lot of hard work to build and evangelize Kubernetes. 

Brendan shared with us that his hope for the future is that the bits of Kubernetes sort of fade into the background. It’ll be there, and it’ll be important, but it won’t be talked about or thought about from day to day because, as he puts it, “There’s so much more that needs to be built.”

Thanks for reading! If you want to stay up-to-date with what top engineering leaders are thinking about, subscribe to the Dev Interrupted Substack.

You'll get access to weekly articles, podcasts, and news from the best engineering leaders in the industry. Enter your email below to subscribe and be a part of our growing community!