On this week’s episode of Dev Interrupted, we talk to Liz Rice, Chief Open Source Officer at Isovalent, and author of the book Learning eBPF: Programming the Linux Kernel for Enhanced Observability, Networking, and Security.
Liz is an expert on open source, containers, and cloud-native technologies, and joins us to discuss her book, what she describes as some of the eBPF "superpowers" people are talking about, and some of the fascinating projects surrounding eBPF like Project Kepler.
Liz also gives advice to engineers looking to try their hand at writing a book.
Episode Highlights:
- (1:38) Liz's background
- (6:20) What is eBPF?
- (12:30) Advice for engineers who want to write a book
- (15:35) What is Cilium?
- (18:09) Security & visibility
- (27:27) Project Kepler
- (31:50) The future of cloud-native
Episode Transcript (disclaimer: may contain unintentionally confusing, inaccurate and/or amusing transcription errors)
Dan Lines: Hey everyone. Welcome to Dev Interrupted. This is your host, Dan Lines, co-founder and COO at LinearB.
And today we're joined by Liz Rice, author and chief open source officer at Isovalent. Liz, welcome to the show.
Liz Rice: Hi. Thanks for having me, Dan.
Dan Lines: Yeah, super great to have you on with us. You have a really interesting, great technical skillset, with a background in system software engineering. You're also a regular speaker at conferences like Reinvent, Velocity, DockerCon you've earned this reputation for making kind of these complex concepts, more accessible and more understandable, which is really great because today we're gonna talk about your most recent work as an author.
you have a book titled Learning, eBPF, programming the Linux kernel. For enhanced observability networking and security. And in the opening line of the book, you describe eBPF as one of the hottest technical topics of recent years, in the cloud native community and beyond. We're really excited to dive into your book and what is eBPF and all of that kind of stuff.
But first we wanna start out with you and your back. I looked you up on LinkedIn before we talked and you had a really, I think, interesting career path cuz I see all these different titles and you're involved with these different like organizations and companies. So can you give our audience a little bit about who you are, what your career path looked like, and how you kinda got to where you are?
Liz Rice: Yeah, so I always knew I wanted to work with computers, since I was a kid, it was always gonna be something in software. My, first job out of university was for a company that, it, it's taken a very long and securous path, but it's now part of Microsoft. But we were doing networking stacks first foray into professional programming was all about portable networking stacks.
And that made for really boring party conversation, my twenties. After a while doing that, I moved into I, I moved to Skype for a bit and, that was something that everybody had heard of, so that was brilliant. And then I, yeah, moved from there to, I spent. Bit of time working on, recommendation systems.
I worked for a company called Last fm, which did music recommendations, and then I did some startups around TV and movie recommendations. And then, and this was all taking me further and further away from like the nuts and bolts of the technology. It's a long way from a network protocol, from a network packet to.
A movie recommendation. And at some point I was having a conversation. We were actually in a startup accelerator. Somebody mentioned Docker to me and I thought that sounds, interesting. Maybe we should look into that. And I ended up co-founding a startup that, did sort of container scaling way before its time.
Terrible business idea, but got me back into this, much lower. Into the weeds of the technology got me involved in the container space and then the cloud native space. And I guess as part of that, I learned I was doing a lot of presentations. I think, if you are working for a startup you spend a lot of time pitching your products and talking about what you do I somehow learned that I was quite good at explaining these complicated things.
It's really nice for you to mention that in the introduction cuz I think it's something that, it took me a long time to learn about myself, but I think I am quite good at understanding how to break something complicated into a storyline that people can build up and follow and, take people on a journey from knowing nothing to actually having a pretty good understanding of how something quite complicated.
Whether that's a container or now eBPF, I hope.
Dan Lines: Yeah, absolutely. One of the things that you mentioned your startup and a lot of startups are before their time, right? You have to have the perfect timing and a lot of things need to fall into place, to get like a huge success.
But one of the things that I've learned about startups is you really do need to hone your storytelling skills. That's what people understand, especially us. engineers like coming from that background, at least for me, like I was never like trained in storytelling or something like that, but I do find that skill to be very, and yeah, thanks for sharing that wonderful background.
Now, the book that we're going to be talking about, so it's called Learning eBPF. Now, is that your second book?
Liz Rice: It is, yeah. It's my second kind of proper book. I've written a couple of other little report things for iri, but this is the second book that you can, buy at your bookstore. The first one was about container security.
Like I mentioned I've got quite into containers and, the container security book I really wanted to explore, not just in fact, not really. The tools that you use much more about how do containers work and what about them is potentially insecure and how could you attack and what are the different, mechanisms that people could use to attack a container and what are the different mechanisms you can.
To defend your container, and it was through thinking about container security that I really got interested in eBPF and actually seen my now colleague Thomas Graff back in 2017 talking about the Cian project. We were actually both talking at the same DockerCon and he was talking about how sills built on.
Amazing technology called eBPF and I just thought that's really interesting. And at the time it was a little bit, felt like this is cutting edge kernel, you practically need to build your own kernel to use this. But over the years I've kept interested in eBPF.
We started using it on a project for container security. And then a couple of years ago I ended up. Joining Thomas at Isovalent and really focusing on eBPF, that's what my fo main focus is now.
Dan Lines: That's great. Let's actually define eBPF. Like what is it? What does that mean?
What does it stand for?
Liz Rice: Yeah, we have to, because it's a set of letters, we have to say what it stands for and it stands for Extended Barclay packet filter. But now you can erase those words from your mind because, Although it has its roots in packet filtering, it is capable of so much more that the acronym is pretty much meaningless now.
So we just think of it as a term eBPF, and what it allows us to do is run custom programs inside the kernel. I nearly said the Linux panel. It is also possible these days to, to, run eBPF in Windows as well. So it's becoming a adopted by other operating systems, but the sort of history of it and it's certainly, most, most widely adopted in Linux.
And that's where I'm most familiar as well. Yeah, so it allows us to write custom programs, load them dynamically into the kernel, attach them to. Any event on the system. And then our program could be collecting information about the event. So we might use it for observability. We can even influence the way the kernel behaves.
So we can use it to do things like, modify network packets or redirect network packets to, to build networking functionality or potentially make policy decisions for security reasons. Whether that's dropping network packets or. Permitting or denying certain activities to happen from a security perspective, that having the ability to modify how the kernel behaves and modify it dynamically gives us superpowers.
And that's why I am really excited about it.
Dan Lines: Yeah, no, that sounds amazing. In terms of like the type of person that would be interested in this or who you would recommend to. Catch up on your book? What types of developers or like any person, like who, who is the audience really here?
Liz Rice: So the slightly ironic thing is I don't really think very many people will need to write their own eBPF code. So I've written a book that I think nobody needs to read, but a lot of the time you do things not because you need to, but because you're interested and you want to, and I'm the sort of person who doesn't, I don't learn very well from, pictures and diagrams. I need to feel the thing working. I need to, try it out for myself and, I need to see the code. And that's what I've tried to do in the book. So explain for people who are interested in what is this eBPF thing that people are talking about, and how does it achieve all these amazing superpowers that people are talking about.
In the book I'm trying to show you, and build that up. And I think the best way to, to explain it is through code. So if you do then subsequently want to go and write eBPF code, hopefully it's a good starting point to do that. But I think for most of us it will be okay. This gives me a feel for how the concepts work.
It gives me a mental model, can understand. How other tools that are built on eBPF, how they're working and hopefully get some interest from that.
Dan Lines: So it sounds like for the book, if I get it right, like I can really get my hands dirty, like maybe I, am I able to actually write my own eBPF thing?
I'm making an assumption here, but is this the type of thing where, companies are gonna standardize this so it's like more out of the box for me? I'm not gonna have to write my own thing or is it still something like if I wanna use it, I'm on my own. Like where does it live in the wor world of like maturity right now?
Liz Rice: Yeah. So the book will absolutely gives you tons of examples that you can try out for yourself. And, I've also put together, a Lima VM configuration so that, because one of the problems. E V P F and in many sort of environments is, you need to have the tool chain set up and you need all the right things in place.
Yeah. So I've tried to make that as easy as I can so that people can, spin up a virtual machine that has all the right things in the right place and that, then you can follow through all the different examples in the book. Now whether or not everyone will need to do that, I think most people's experience of eBPF will be through other projects and products. Whether that's, I'm very involved in the cilium project and Cilium uses eBPF extensively. And I think, we see a lot of cilium users who are, they're interested to understand how E V P F works and they're interested to kick the tires and play with it a bit.
And, You'd have this the knowledge of okay, how can I expect my system to see what eBPF programs cilium is using? And, just try and get maybe a bit more knowledge of the eBPF tools that you are using. But yeah, I don't, I think most people will find themselves using and there are plenty of tools out there that are based on eBPF, a ton of command line tools, TCP dump.
If you've used Setcom, that's using a sort of a form of bpf F So a lot of us have been using eBPF to some extent, even if we didn't realize it.
Dan Lines: Gotcha. That's how it usually goes, right? It's all it if you're listening to this pod, it's depending up to you, like how deep do you wanna go?
Do you really wanna understand how it works? Or you wanna be more on the surface level that, that's u that's usually where things go. More the surface level I'm trying to achieve. Maybe something insecurity, whatever it. We're gonna dive into all of that, cuz I do wanna get to some of the more like specific takeaways.
But before we go there, one thing that is really cool about you is that you have written these two books now. And so for our audience of engineers, if they are in a situation where they're like passionate about a particular topic or they feel like they have a cool te. Can you give us pull the curtain back at all on what it takes to write a book or get your work out there?
If that's something we wanna do.
Liz Rice: Yeah. I have actually written myself a note to say, Liz, never write another book while you are also doing a full-time job. Cuz it's a lot, And I think both times I've written a book I've had, I've felt like I had quite a lot of material already from doing conference talks.
And the nice thing about a book is you get this ability to take, a talk that maybe is half an hour or 40 minutes and really dive in a bit more into the details and the things that you have to cover at a pretty superficial level you can really dive into. And you know that in a book, if people find.
Detail boring. They can just skip over it and move to the next section. You have that kind of, as a reader, you have that choose your own adventure aspect of reading a book. So I, I felt like I had quite a few talks. That I could use as a basis for, and I had a rough idea what the structure of the book was gonna be.
What actually happened was I kept uncovering more and more interesting information and things that I thought I understood. And it turned out that while I was writing my examples, I'd learned things from myself. Which, it is all part of the fun. But and I think that's, it's one of the nice things.
Whether it's a conference talk or a blog post or a book or whatever it is that you are creating, if you're trying to teach something to somebody else, in your heart whether you really understood it as well. And I find that part really satisfying.
Dan Lines: How long did it take?
Liz Rice: It was probably not quite a year between me first writing the proposal and getting the book physically in my hand.
But it was definitely in chunks. Like I spent, a couple of weeks in the summer where I just took a couple of weeks off and pretty much wrote three or four of the chapters. And I had another chunk of time in the autumn where I was like, okay, I'm gonna have to get some chapters turned out, take some time and.
Dan Lines: Okay. Kinda gives us like a estimate of what it really takes. That's like a, at least a year project commitment to get that going.
Liz Rice: Yeah, really it is. Yeah. And if you work backwards from publishing, prior to publishing, there's various editing phases and feedback cycle and I got some incredibly.
Really good feedback from people involved in eBPF, both on technical aspects and also, just helping structure my thoughts. something that you do when you're writing a book that you don't maybe do so much if you are just writing a blog post, is really getting that in depth feedback and that sort of sense of somebody saying I felt.
I would've preferred you to introduce this concept earlier, and that's really helpful.
Dan Lines: That's amazing. Thanks for sh sharing that. It's a super useful information for anyone that wants to go down that path. Now, if we go back to eBPF and you've mentioned cilium a few times here and you've also mentioned, hey, eBPF, like this technology is actually.
I don't know, in these projects, or you might already be using it. What is cilium, catch us off like on that and like how is eBPF used there?
Liz Rice: Yeah, so Cilium is probably best known in the Kubernetes world. It's, networking plugin for Kubernetes. Although we do also have some people using it in.
Networking environments standalone from Kubernetes, but probably, 90% of the users right now are using it with Kubernetes or is in Kubernetes. And it provides that networking layer. It also provides network security. So I, I mentioned before, this ability to use eBPF. Get network packets at various points in the stack.
Cilium uses that ability to, optimize the way that we connect and pass packets between different entities and Kubernetes. Do things like encryption enforce network policies by, using eBPF to compare packets against policies and decide whether or not they need. Forwarded or dropped if they're out of policy.
Yeah, so it's a very powerful platform for connecting your cloud native workloads. We also have, the observability aspect. So it's one thing to have networking. It's another thing to understand, to be able to debug your networking if something goes wrong. And so the Hubble component, which.
Observability is a really important part of the project and you get this really amazing sort of flow of network packets. You can get really cool metrics. We've got a, a bunch of really nice dashboards that you can put in Grafana for seeing. How different aspects of your networking are performing, different latency characteristics, how many packets are being dropped by network policy, all kinds of different aspects that you can visualize.
And all of this is built on top of the fact that we can extract this information from the kernel using eBPF. Yeah. And and in a really performant.
Dan Lines: Like a few minutes into our conversation and I already know I'm not as smart as you are, so I'll try to ask a few questions that will help, orient me a bit here.
So one of the things that I usually ask myself when there's this new, amazing technology, sometimes if you look back in. There's amazing technologies, and then it takes a while to get them to be something that's like practical for consumers or like practical for the world. And it seems to me like, some of the things that you're focusing on here is like security, right?
So security is one of them and observability usually goes along with that security, component. But what's the difference between, cuz there's a lot of observability companies out there, there's a lot of security companies out there. What would be the difference if I like, do not have this eBPF technology?
Versus if I do have it in the world of security?
Liz Rice: I think there are a couple of things that I would highlight. So one is because eBPF is sitting in the kernel, however many processes you have running on that machine, how, whether they're in containers or not in containers, whatever is running on that virtual machine or physical machine.
There is one kernel, and if we can instrument that kernel, we get visibility over everything that's happening on that machine, which is really powerful, particularly in a cloud native environment where, traditionally we've had to instrument our applications by using CCAR contain. So the whole point of containers is to isolate them from each other.
So that kind of by design means one container can't really observe or interfere with another container. And so we have to have this sidecar, injecting sidecars into the same pod as a container so that they can observe and interact with each other. Whereas if we can use the kernel for the instrumentation, We just automatically have this ability to see and influence what's happening across all the applications on that machine.
Dan Lines: Oh, that's cool. So it's a more like centralized location where everything is happening. That's what it sounds like to me. Containers by design, they're designed to be, exclusive from each other. That's one of the pros. You're saying, okay, now with the kernel, we can do like our, I don't know, security inspection there.
And everything flows through it. Is that like a Okay. Exactly.
Liz Rice: So whenever your application is doing anything interesting, whether it's. Sending a network message, reading something from a file, writing something to a screen, even, accessing memory or, whenever permissions get changed, all of these things require assistance from the kernel.
The kernel is involved when you're doing basically anything interesting. And so that's a really good, opportunity to look at things from a security perspective. Should this application be allowed to access a file? For having the control, and this is nothing new. Having the ability to control this from the kernel.
We've had things like, app Armor and SE Linux and Setcom that I mentioned before, that are using interfaces in the kernel. But the difference with eBPF is we can customize that and, have very dynamic policies and we can, extract information and pass it to user space in.
We can build innovative interfaces, we can extract that information and send it to a sim or send it to a Grafana dashboard or whatever it is that we want to, however we wanna format the information. We have so much flexibility by being able to program the kernel.
Dan Lines: that's really cool.
Is there anything around performance that comes along with that, or is performance still the same? How do you think about that?
Liz Rice: The nice thing is that it is typically really performant way to gather information. Yeah. Because. If you are you don't have to worry about the transition between Userspace and Kernel, which it was interesting.
I actually tried to find some data on it's a sort of known thing that this transition between kernel and USERSPACE is expensive. How expensive? It depends. So I just have to gloss over that and say it is expensive to transition between user space and kernel space.
The fact that you can with eBPF see something happening in the kernel. Store some metrics about store whatever information you want about it without making that transition. Yeah. Makes it much more performant really than anything that requires you to access it from user space. So typically it's gonna be a, an efficient, anything you can all, you can then write code that's really inefficient or you can do something to make things not perform well, but at least in theory, eBPF programs are typically really lightweight. They're avoiding these transitions, so they're not gonna have a huge effect on performance.
Dan Lines: Yeah, that's really cool. Going back to the security side of things, totally understand that now it's like in a more, it's in a centralized location, so it's easier.
You have all the, it's a great area to do all of your inspection. Sometimes when, let's say like a new security company would come out, they would say, okay, like email security is now a new thing, like back in the day and we're gonna, provide more email security or like against phishing attacks or against this type of attack or that type of attack.
Is there anything specific with eBPF that there's like a new type of security, detection, or is it more like doing it easier, more performant, that type of thing?
Liz Rice: I'm gonna say it is an opportunity to build a whole new, approach to security so quite often. In the security world, we think about things in terms of network security and then like runtime security.
So we are very used to the idea that firewalls should drop packets. If you see traffic that you don't wanna handle, you drop the packets. That's normal behavior for runtime security. We typically see people being a bit more cautious about. Okay. If I see something that looks like malicious activity now, there, there are ranges of this.
We probably say we're gonna have permissions on a set of files and we are gonna lock down permissions to, what files people can access, but, We typically see, and I used to work for a security company which had the ability to do both audit and enforcement of runtime security.
Nearly all customers just wanted the audit and then they wanted to get all the information about events into a sim and then analyze. After something's happened, they've got all the data and they can do the forensics. And I think some of the reasons around that are performance, and some of them I think are more to do with how hard it is to write a meaningful policy that doesn't break your application.
You might say I don't want, I wanna stop my application from, it's only gonna access a certain set of files, but if I got that set of files wrong, Then maybe my application would break and then, everything would be awful. And we don't want security to get in the way of the application.
But I think a lot of that is because the profiles are really hard to write. If people are writing se Linux profiles or app hammer profiles, it's just too hard for people to get them. Correct. I think there's an opportunity for a much more sort of expressive and meaningful way of talking about these.
And the reason why I think eBPF is gonna be key to this is we can express policies, and this isn't gonna be an easy problem to solve, but we can do the sort of filtering in the kernel. There's actually a sub project in cilium called Tegan that. It allows you to, observe security events and also you can also, actually prevent, violated events that violate a policy.
You can prevent them from happening. And the fact that we can filter those events in the kernel gives us this opportunity to be really high performance, to not stand in the way of the application. But I think there's still this aspect of being able to, as a developer say, here is my app. It talks to this service, that service, and the other service.
It expects connections from here, it writes to these files. it doesn't do anything crazy about escalating privileges. So if we see anything, It suddenly wants to, get capsis admin or it suddenly wants to start initiating network connections to some unknown destination. Those things should be blocked, and I think being able to express those rules in a way that really makes sense to developers is gonna be the next green field for security applications.
Dan Lines: Thank you for explaining that. That sounds amazing. The last question that I wanted to ask you in, in this area of the show is, we have di dived in to what I would consider like the practical nature of eBPF. Is there anything more futuristic that it, that comes to mind for you?
It's not being used, that way now. Maybe you found it in your book. I don't know, like any like future dreams you have for it.
Liz Rice: It's something that, is already in progress, but I think it speaks a little bit to the breadth of things that we can do with eBPF. There's a project called Keppler that's being used to measure the energy efficiency of software.
So it's hooking into. various points in the kernel to measure essentially how much CPU time, how much, I actually interviewed, one of the maintainers of the project and he was explaining how, every time you retrieve, information from cash, Or if it's not in cash and you have to go and actually hit memory, then that is more energy intense.
Just being able to measure these really fine grained, like the CPU cycles and the cash misses and turning that into a representation of how efficient your software is or how much energy your software uses. I think, for those, that's super cool of us who are concerned about energy consumption.
Yeah, that's really. Really interesting, really nice application.
Dan Lines: Definitely a hot topic right now. When you were talking, it made me like go into a daydream about, okay, we have all of these computers running all over the world. What if I could see a dashboard in real time of like energy consumption across the planet?
It's probably a lot like you. Oh, it is. It's a lot of machines running.
Liz Rice: Yeah. And I think there's lots of stuff that we're doing in Cloud Native that actually lends to energy efficiency. Things like scaling applications in response to demand and that's really, I feel like there's lots of things about Cloud Native that can help us address this sort of the giant amount of energy that is being used and having different tools, just to make sure we're not being.
Just ridiculous about how much energy we're using. I think that's really good idea.
Dan Lines: I love it. Anything else like that?
Liz Rice: Oh, I wish I had a whole kit bag of other future ideas. Yeah, no I think there's so much potential for innovation. because it is a platform, eBPF isn't, it's not a product.
It's this Yeah. A platform on which people will build all sorts of interesting things.
Dan Lines: That, that's another good reason to, check out the book. I think, if you're someone that wants to learn a technology and do something different maybe than like businesses, the how the world works in capitalism, they.
eBPF and of course, apply it to the things that will make money, but maybe there's other things out there that we don't even, know about yet. Like that energy project or something like that. So I think that's like a good reason to dive into the book. What has the feedback been on, on the book so far?
Liz Rice: So far really lovely positive feedback. I've heard people who've been, working through the examples which is always good. I was very excited to get it takes a while before the book, between the book being available and you start seeing reviews on Amazon or whatever.
So I was very excited when I started getting, really nice positive reviews and the five stars and everything.
Dan Lines: Yeah. I see five star reviews, so that must feel good.
Liz Rice: Yeah, that is really nice. And there's a bit of a, a bit of an emotional rollercoaster where I think all the time you're writing the book, you're like, yeah, okay, this is, yeah, this is fine.
This is fine. And then at the point where you deliver it, you have that horrible sense of, what have I missed? What have I forgotten? What if it's terrible? And then you get feedback from your reviewers and you know that's encouraging. But then when you deliver the final manuscript, Then it takes several weeks really before it gets actually into production.
And it's this sort of sense of, I don't know what's gonna happen, what are people gonna like it? Are people gonna even notice that it's there? So it's, it is amazing when you finally, people start saying, I've got your book. And they post pictures and, and when I finally got my physical copies, it really, very exciting.
Dan Lines: That's amazing and some inspiration for everyone to, get their own work out there, whether it's a blog or a book or whatever. Yeah. Getting that feedback always feels like amazing after you work so hard on it for a
Liz Rice: long time.
It does. I
think, even, yeah, like you say, even if it's just a blog post, somebody reacting to something you've written is so encouraging.
It's the sense that you. Given somebody something to think about or they've wanted to respond to you in some way. I think it's incredibly rewarding.
Dan Lines: Awesome. The last section that we have on here is anything around the future of cloud native, where is it going, or some of the most interesting applications or anything around security.
Can you catch us up? Any of your knowledge there and what you think the future?
Liz Rice: Yeah, we're definitely seeing a new generation of infrastructure tooling, so anything that's around observability, logging, tracing, monitoring, the security aspects, which can be the network security, runtime security, there's.
Security observability. There's a lot of interesting aspects and interesting tools being developed. It feels like every day. There's, like you said, there's a lot of security companies out there, and it feels every day somebody's coming out with an idea for, doing their tool in eBPF.
And then also, in networking of the networking is an area that, you computers have been connected to each other for quite a long time. How much innovation can there be? And it turns out people are constantly coming up with new, problems to solve in networking. And eBPF can be a really, really efficient way to handle, handle network packets.
Yeah, we're seeing lots of things like, telcos, people in Telco particularly, they have some very specific. Networking requirements and they handle a lot of networking traffic and really interesting to see them embracing cloud native, which we've been doing for a while, but also really seeing the benefits of using eBPF alongside that.
Dan Lines: Liz, thanks so much for coming on the podcast today. It's been a really fun conversation.
Thank you so much. Before we go and sign off here we'd like to give our guests an opportunity to close out the pod, with any type of, call to action. I know you might have some labs content or something like that, but what do you wanna say to our audience?
Liz Rice: Yeah, so obviously I'm hoping people will either, buy the book or you can download the book from is isovalent.com.
Where we also have under is isovalent.com/labs, A whole series of hands-on labs where you can try out some eBPF examples we're working on putting some of the examples from the books onto that lab site, and there's also tons of examples around c. And really nice way to get your hands dirty, play with the labs again, without having to worry about setting up the environment.
It's a lot of fun.
Dan Lines: That's great. Thank you, Liz, and thanks everyone for listening. We'll see you all next week.
A 3-part Summer Workshop Series for Engineering Executives
Engineering executives, register now for LinearB's 3-part workshop series designed to improve your team's business outcomes. Learn the three essential steps used by elite software engineering organizations to decrease cycle time by 47% on average and deliver better results: Benchmark, Automate, and Improve.
Don't miss this opportunity to take your team to the next level - save your seat today.
