Episode 63: EBPF Networking Isovalent with Liz Rice – Chief Open Source Officer
Intro
Mike: Hello and welcome to Open Source Underdogs! I’m your host, Mike Schwartz, and this is episode 63, with Liz Rice, Chief Open Source Officer at Isovalent, the software startup behind Cilium, an eBPF-based Networking, Security and Observability project.
This episode was recorded in early February at the inaugural State of Open Source Conference or SoCon, which was held in London at the QEII Center in Parliament Square. The force of nature behind SoCon was Amanda Brock, CEO of Open UK and editor of the essential book Open Source Law, Policy and Practice, 2nd edition. Check it out on Amazon if you’re an open-source founder. Don’t miss SoCon next year in 2024, especially if you’re already in Europe for FOSDEM.
If you think eBPF or enhanced Berkeley Packet Filter sounds like a geeky low-level technology that you don’t need to know about – well, you’d probably be wrong. It enables developers to safely write code that runs in the Linux kernel. And safely is the key word here, because if you crash the Linux kernel, everything on the whole server goes down, all the containers, and everything else running on that server.
However, by exposing the power of the Linux kernel, developers can write code that runs faster and consumes less energy, and faster and cheaper has always been an attractive feature. Cilium combines three products into one. It’s like an old-fashioned firewall, an API Gateway and Wireshark, and it’s Kubernetes pod aware. It’s used by a number of successful products like Teleport for access management or Solo.io Service Mesh.
Simply said, eBPF is going to fundamentally change our infrastructure.
I met Liz at the SoCon conference, and after learning a little about Cilium, I was really impressed, and I asked her if she would come on the podcast, and luckily, she said yes. So, here we are with the interview.
Mike: Liz, thank you so much for joining me today.
Liz: Thanks for inviting me.
Tech Overview
Mike: As I understand it, Isovalent leverage’s a kernel technology to build a product called Cilium Enterprise. The upstream Cilium project on GitHub has over 22,000 commits and 14,000 stars – these are really impressive numbers for a project that started in 2016. How did this happen and how does this relate to the origin story of Isovalent?
Liz: Yeah. So, Cilium is built on a platform called eBPF, which is the kernel technology that you referred to. And eBPF allows us to run programs that are triggered by events that happen in the kernel, and those events could be Network packets, they could be a system call being made by user application – pretty much any sort of event in the kernel can be used to trigger an eBPF program.
Cilium was the first networking project to take advantage of eBPF. And it was always designed with the idea of container networking in mind. And the folks who started it are the founders of Isovalent, as well as being the originators of the Cilium project. So, Thomas Graf, Daniel Borkmann, who’s a kernel maintainer looking after eBPF, within the kernel.
And eBPF and Cilium, particularly eBPF in Networking and Cilium, kind of grew hand in hand since 2016 thereabouts, as we – the many, many contributors to the Cilium project – as it grew and as it gained functionality, sometimes that’s required additional capabilities in eBPF.
So, it’s been really almost like a long game. I think when Daniel and Thomas and Dan, the CEO, when they were first thinking about using eBPF, it was such a cutting-edge kernel technology – nobody was using it in production.
You know, when we add something to the kernel today, people won’t be using it in production for probably three, four, five years to come, so really, anticipating what the future was going to be.
I first saw Thomas presenting Cilium and the underlying eBPF technology back in 2017, and at the time I thought, “Well, this is revolutionary, this can change so many things.” Because not only can we see Network packets being manipulated by eBPF programs, we’ve also got this incredibly performant way of observing those Network packets and reporting on them that we can use for observability tooling. And like you mentioned network policy – we can implement network policy in eBPF.
Just making policy decisions about whether an individual Network packet is permitted or denied by policy, based on Kubernetes identities – this is the other real strength of Cilium.
It knows the Kubernetes identities, the labels of every pod. And so, you’re no longer just looking at network flows in terms of IP addresses and the port numbers you’re actually looking at them in terms of “this is a flow between service X and service Y.” It is so much more meaningful for a Kubernetes’ user.
Why the name Cilium
Mike: Just out of curiosity, do you know what Cilium means?
Liz: I think they’re little hairs in the inner ear – I’m not entirely sure why that was used as the name for the project.
Origin
Mike: I understand the eBPF technology is mind-blowing – Cilium is quite a project as I said. I mean, you’re not one of the co-founders, but do you know anything about how did it become actually a business?
Liz: I think pretty early on, as Cilium, the project, was getting established, and this sort of understanding that eBPF was going to be a really great foundation for efficient networking. That idea of building a company around this technology was probably in Thomas’s mind right from the get-go – I don’t know that for sure, but I imagine it was. And he and Dan Wendlandt, who I mentioned earlier – this is Thomas Graf and Dan Wendlandt – Dan had the background in software-defined networking, he’d worked at Nicira.
And I think they really saw the future of container networking being built on eBPF, so it was kind of natural to build a company. But, for the first few years, really the focus was on building the Cilium open-source projects, getting that really well-established and really well-known in the Kubernetes community.
It’s now been adopted by the CNCF, so we’ve actually contributed the project to CNCF, we’ve recently applied for graduation status there. It’s probably the most widely adopted in production networking plugin for Kubernetes now.
That kind of path from open-source projects, we really need to see this widely adopted, and then, a business that can provide, not just support, but also some Enterprise features that really large adopter is going to need. And just makes a lot of sense.
What does a Chief Open Source Officer do?
Mike: Your title is Chief Open Source Officer, and that’s a title I’ve never actually heard before. How is that role defined at Isovalent and why were you so excited to take on this mission?
Liz: It’s a particularly interesting title in a company where the vast majority of the engineering is open-source engineering, but I don’t run the engineering teams. My role is much more about how do we continue adoption of the open-source project, and how do we interface with the foundations, the community – I do a lot of work with the CNCF as well. How do we both act as good citizens towards that community and do the right thing in the open-source world. But also make sure that we’re taking advantage of everything we can.
You know, foundations like this offer us a lot of roots to speak to people who might become users and how we can do that in a way that is beneficial for people who want to learn about Cilium, or who want to learn about eBPF. So, that kind of educational role also falls within my team.
Open source v. Enterprise
Mike: This may sound like a silly question because Cilium was so powerful, but from a business perspective, what would you say are the main value propositions of the software?
Liz: So, from the open-source perspective, it’s a highly performant networking solution with built-in observability and security features. And we could dive into more details on what those are. From our perspective, it’s fantastic. If people are satisfied using the open-source version of the code – that’s great – we never want to make it such that — we don’t want to curtail the functionality, so that it always wants to be useful to open-source users.
That said, there are some features that particularly larger Enterprises are particularly interested in that you won’t need if you’re not a big Enterprise. So, for example, integrating with Legacy workloads. Some high availability features that you don’t really need unless you’re at a certain scale – those are the kind of features that we provide in the Enterprise distribution at Cilium.
Isovalent v. Sysdig?
Mike: Do you see yourselves competing with a company like Sysdig?
Liz: On the security front – yes. There is an element of competition there. I think we’re sort of speaking with slightly different customers there. Because, to my understanding, Sysdig is very much a security focused solution, whereas Cilium really applies more to a platform team who’s establishing, I would say Networking first, with this incredible set of security capabilities that you can then show to the security team, these amazing capabilities that they’ll get all that they already have by using Cilium.
I think we’re probably talking to different people within our respective customer organizations, but there is a certain amount of overlap around particularly the kind of runtime security, which we have a sub-project of Cilium called Cilium Tetragon. And there’s the ability to create profiles for the kind of things like accessing sensitive files or running certain executables, privilege escalation, suspicious network activity – these are the kind of things that we can detect at runtime using eBPF.
Why contribute project to the CNCF?
Mike: You mentioned that Cilium was contributed to the CNCF. What was the reason you brought the project to the CNCF? Also, what does that mean for the governance of the project?
Liz: It’s a big step to contribute a project. Because we hand over the intellectual property to the CNCF. That is something that Isovalent used to own and no longer owns. And the governance of the project really needs to be in the hands of the community. So, Isovalent remains the most prolific contributor, but – and this is again part of my role – encouraging more people and more organizations to get involved in not just code contributions and not just documentation contributions, but also the kind of broader evangelism of what Cilium is and the advantages of Cilium.
So, yeah, we’ve really embraced that community. And I think the phrase that we’ve used internally is “paved the world with Cilium”.
And the best way to pave the world with Cilium is to give it to as many people as possible, and the CNCF gives us a really great route to reaching all those people who are using Kubernetes. It gives those people confidence that it doesn’t matter what happens to Isovalent, the Cilium project is in the hands of a much, much bigger organization at this point.
And then, you know, that subset of people who are using Cilium, but then, find themselves needing Enterprise features. We won’t necessarily be the only Enterprise distribution, but there’s no doubt in my mind that we have the greatest expertise. So, hopefully, we will be the obvious choice for someone looking for Enterprise features or Enterprise support agreements around Cilium.
Trademark
Mike: This actually leads into my next question, which is that CNCF actually owns the trademark for Cilium, but your product, the Isovalent product is called Cilium Enterprise. And so, hypothetically, another company could make a product called Cilium Pro. I mean, I looked at the contributors and I went down eight contributors, they all work for Isovalent, I didn’t have time to go any further, but, obviously, your company has a lot of expertise, but still, the prospect that company spent a lot of money defending their trademarks, I almost never heard of anything like that – is it sort of terrifying, though?
Liz: I mean, at one level, yes, it is kind of terrifying. And Cilium is a brand name that is better recognized today than Isovalent is. And that’s a challenge that we have to embrace. And there are rules around what you can and can’t use – I think that there are probably still a few instances of documentation and use of the word Cilium, which we’re not really allowed to do any more, that we haven’t managed to tidy up everything.
There’s limitations on what you can and can’t use around a name based on what is now a Linux Foundation trademark. But everybody understands there’s a transition between us having a trademark and then giving it to the foundation. It obviously takes a little while to tidy up all that options around that, yeah. So, Isovalent Cilium Enterprise is the Isovalent distribution of what is a CNCF-owned community project.
Outside Contributors
Mike: I mentioned that there’s a lot of Isovalent engineers who are contributing code, but are there other engineers who are also contributing?
Liz: Absolutely! Google is quite a prolific contributor, Cilium is actually used in Google’s Dataplane V2, we have maintainers from Datadog, again a huge adopter who has been using it. Enormous scale – there’s some really good talks from Datadog talking about the scale of which they’ve deployed Cilium, we have contributors from Palantir.
Yeah, there’s several what we call committees, so maintainers of the project, who come from lots of different organizations. And then we have – I think it’s around 700 contributors in total. Isovalent today is just over a hundred people. The contributor base is much, much wider than just Isovalent. That said, we probably have the largest group of people working full-time at Cilium.
Market Segmentation?
Mike: On the commercial side, for infrastructure, the marketing is very horizontal, but have some natural segments worked out in terms of the customers who convert from open source to a commercial relationship with Isovalent? And are you figuring out that there’s any ways to segment the market here or the messaging?
Liz: I think that’s something we’re learning – I have just mentioned that we’re about a hundred people now, so we’re growing in our capabilities for how we target different customers and different verticals. We’ve had a lot of success in financial verticals media, quite a few transport, strangely enough. Yeah, so there’s a pretty wide breadth of Enterprises who have adopted this. I guess, the prerequisite for nearly all cases is that there are Cloud Native Kubernetes users, or that we do have some users who are using Cilium in a standalone load balancer scenario.
Have we figured out how to market to all of these different types of businesses? We’re absolutely still evolving and learning. But I think the fact that we’ve for many years had this very community-based focus, a very community-based approach, means that we can establish relationships and have trusted sharing expertise on a technical level that then encourages those engineering teams to recommend us internally.
And when it comes to making a choice about an Enterprise product or whether they need commercial support, those engineering teams already know who the experts are, and have potentially already had help from our team through the open-source community.
Team Location
Mike: Is there an Isovalent headquarters office where engineers go in, or is everyone like spread around the world?
Riz: We are fully distributed. We do have offices in Zurich, where Thomas is based, and in the Bay Area, where Dan is based. And I think that the timing, you know, really around the pandemic, just at the point as Isovalent was growing was sort of around the same time as the pandemic hit. So, inevitable that we were going to be remote based.
And as people have joined, they joined from countries all around the world. We have people from as far as long as Japan, or Alaska, Australia, throughout Europe and across the U.S. So, our team is really now fully distributed, and the culture has to embrace that. So, we’re very much focused on being remote first.
We do get the team together, and we try to get the whole company together, at least once a year. And we have a lot of encouragement around getting teams together in what we call hive time. Because we’re all about kind of bee-related metaphors.
Monetization: What features are enterprise?
Mike: I’m curious about monetization. It sounds like it’s open core, and what are the extra bits that you’re offering, I guess, in the Enterprise? And how do you decide what to make open source and what to add as an extra feature in the Enterprise distribution?
Riz: I see that the term open-core can sometimes come with a bit of a negative connotation. Sometimes people think of it as an open-source software that’s got some kind of, you know, been cut off at the knees, and that’s absolutely not what we believe in.
We absolutely believe in the open-source product being genuinely usable, and there are some pretty large organizations who continue to use just the open-source version. The kind of things that people will come to us for will be — there are some high availability features, there are things like BGP support for connecting into your legacy data center workloads, some Telco specific protocols that we’ve worked on – we very much don’t want people to feel that there’s something that’s core to their basic use case that they can’t do with Cilium.
Unless they are big enough that they’re the kind of organization that wants to pay anyway. You get to a certain size of organization, where you really don’t want to be just relying on open source with no sense of who’s going to support it when anything goes wrong. And they may come to us for features, they may come to us because they just want to know that somebody will be there to help them, you know, with a contract in place, should anything be needed.
Features for Growth
Mike: We mentioned that Cilium is a really broad product. Is there one particular product feature that you see driving the most growth, going forward in the next couple of years?
Liz: That’s a really great question, because we do have you know really, really powerful features in a number of different axes. So, for example, we just did a partnership with Griffon, where we’re building some really great dashboards, again a big part of this is available, completely open source.
There are also going to be some additional Enterprise features here. Perhaps the thing that strikes people is that they get this amazing visibility. And you know, that could be the moment when they realize, “Huh, look at the power of Cilium!” And the fact that we can see all these latency metrics or security information being shown in a visual way. So, that could be one thing that really drives growth.
It could be Service Mesh. We have a very efficient approach to doing sidecars Service Mesh in Kubernetes. Service mesh is one of those features that when it first started being talked about in probably 2018 – huge hype, huge excitement – the reality of people adopting Service Mesh, they found that it’s actually quite resource-heavy, there are issues, instrumenting all your workloads with these Service Mesh sidecars.
I think some of the realities of deploying Service Mesh had not quite lived up to the initial expectations. And then, last year, we announced the sidecarless approach that Cilium can bring. And mostly through the power of eBPF, it’s incredibly efficient. We can shortcut a lot of the path that a network packet has to take through the Service Mesh.
So, I think that’s another area that can be a real driver for growth, as people realize they can get all the benefits of Service Mesh, but without the overhead that they’ve come to associate with it.
And then, finally – security. I think I mentioned earlier the runtime security tooling that we’re able to provide through eBPF and through the Tetragon project, combining in a really performant, efficient security tooling. At the moment, everybody’s focus in security seems to be on supply chain, but they also still have firewalls. I’m quite a big believer that we have runtime security, everybody has runtime security in the form of firewalls.
We just were on the cusp of people understanding how powerful this new generation of runtime security tools can be to essentially firewall, not just Network packets, but things like bad executables or unexpected privilege escalations, that kind of thing.
Mike: Does the breadth of the product ever feel like a curse? Wouldn’t it be so much easier if there was just one application, and we can focus the marketing message and the sales, and all is just this one thing?
Liz: I’m sure the marketing team tasked with coming up with a tagline would find it a curse, yes.
Lessons for Open Source Startups?
Mike: So you’ve been in the techs business for a long time, taking off your Isovalent hat for a second and just as an observer of the startup scene, and other than the open-source scene in, do you have any advice for particularly entrepreneurs? Because this podcast is really designed first for founders, any advice for founders?
Liz: Yeah. This is actually something I’m getting increasingly interested in and I’m working with the CNCF on how we can encourage businesses on how to operate and be successful with open-source based businesses. There’s two sets of vendors who I would say have quite a lot to learn, particularly if they come into like a Cloud Native community audience.
There’s one class of vendor who is open-source based, they have an open-source project that they’re building their business around. The second class is people who are not open-source, but they have a product that they want to sell into the primarily open-source based Cloud Native community.
I think for both those sets of people, really understanding how powerful community is, Cloud Native community is kind of where I’ve lived for the last, I don’t know, half a dozen years. And it’s incredibly powerful, the relationships that you can build up – not just between individuals, between organizations, can be a really solid foundation for the business relationships that you then build on top of that.
And I think the real lesson for a lot of vendors is: don’t just expect to turn up at an event, pay for a booth or a table, and expect people to come and buy your software. Invest in time as well, invest in contributing, get involved in our project, get involved in the cigs and tags.
Don’t just expect people to immediately think that your open-source project is the one true amazing solution. Take the time to learn what other people are doing around that, and then, have those conversations about why your solution is great and what its strengths and potentially weaknesses might be. Learning to get involved in a community is really, really important.
Closing Notes
Mike: Well, I think that brings us to a close. Liz, thank you so much for sharing and best of luck with Isovalent and Cilium.
Liz: Thank you so much.
Mike: Again, special thank you to Amanda Brock and the whole open UK team for launching the State of Open Conference, where we recorded this episode. Cool graphics from Kamal Bhattacharjee, music from Broke For Free, Chris Zabriskie and Lee Rosevere.
Remember how Liz said that eBPF and Cilium are really good for Service Mesh? Well, remember that, because next week’s guest is Idit Levine the founder of Solo.io.
Until next time, this is Mike Schwartz, and thanks for listening to Open Source Underdogs.
