Containers, Processes, And The Future Of Security With Liz Rice

[00:00:41] ANNOUNCER: Hi. You’re listening to The Secure Developer. It’s part of the DevSecCon community, a platform for developers, operators and security people to share their views and practices on DevSecOps, Dev and Sec collaboration, cloud security and more. Check out devseccon.com to join the community and find other great resources.

This podcast is sponsored by Snyk. Snyk’s developer security platform helps developers build secure applications without slowing down, fixing vulnerabilities in code, open-source containers and infrastructure as code. To learn more, visit snyk.io/tsd.

On today’s episode, Guy Podjarny, founder of Snyk, talks to Liz Rice, Chief Open-Source Officer with eBPF pioneers Isovalent. At Isovalent, she works on the Cilium project, which provides cloud native networking, observability and security. Prior to this role, she was the VP of Open-Source Engineering with security specialist, Aqua Security. She is the chair of the CNCF Technical Oversight Committee, co-chaired the KubeCon and CloudNativeCon 2018 events in Copenhagen, Shanghai and Seattle, and is an Ambassador for Open UK. She’s also the author of the O’Reilly book, Container Security.

Liz has a background in systems software engineering, and loves writing code, and helping other people understand complex technologies. We hope you enjoy their conversation and don’t forget to leave us a review on iTunes if you enjoyed today’s episode.

[INTERVIEW]

[00:02:24] Guy Podjarny: Hello, everyone. Thanks for tuning back in to The Secure Developer. Today, we have a great guest that I’ve actually sort of been waiting to have on the show for a while now. We have a million things we could talk about, but we’re going to talk about two primary ones, which are eBPF and the CNCF and its role in security. To talk about those, I like to welcome Liz Rice, who is the Chief Open-Source Officer at Isovalent, as well as the chair of the Technical Oversight Committee at the CNCF. Liz, thanks for coming onto the show.

[00:02:53] Liz Rice: Thanks for inviting me. It’s a pleasure.

[00:02:56] Guy Podjarny: Liz, before we dig into the sort of the meaty topics we have ahead of us. Can you tell us a little bit about what it is that you do and a bit about the journey that got you here into security or sort of worked on security more directly maybe before than now? But tell us a bit about the journey.

[00:03:11] Liz Rice: Yeah. I mean, I’ve been in software forever and ever. I guess maybe about five or six years ago, I was working on a startup that was basically failing. While that was failing, some friends of mine were saying, “Hey! This containers thing is looking really interesting. Have you come across docker?" And they persuaded me to start a new startup where we were looking at container scaling technology. That startup also didn’t go very well financially, but I did learn an awful lot about how containers work and kind of got involved with the community, and met a ton of really great people and I knew that I wanted to stay in the container space.

From there, I joined Aqua Security and that’s really where I got more involved in the security side of containers. I really approached that with the mindset of curious engineer, which is what I am at heart. I wanted to understand how containers work and why – how an attacker could go about compromising a container. I really wanted to get into the details of that. I ended up writing a book, the Container Security book. I guess after four very fun years at Aqua, I then moved into more of the eBPF space. eBPF is the technology – we’ll talk about this more shortly I’m sure, but it’s something that had really caught my eye a couple of years ago. The opportunity to join Isovalent came along and I thought, “Yeah! These people really know this eBPF space.”

eBPF has got some interesting capabilities with regard to security, so I wouldn’t say that I’m not at all in the security space, but it’s not 100% of what I’m doing now.

[00:05:01] Guy Podjarny: Yeah, understood. Liz, sometimes security is this daunting thing when you’re not in it and before you get into the industry. I guess, how did you find getting into sort of a security focus surrounding when you get into Aqua? Did it feel like a very steep learning curve over there? Was it scary? Was it actually not so much?

[00:05:23] Liz Rice: Yeah. I think security does look pretty daunting from the outside and there’s lots and lots of acronyms and lots of jargon and terminology. The way I approached it was really – I don’t particularly care about the acronyms. I’m much more interested in understanding what’s really going on underneath, how did this attack work or how is this attack surface vulnerable? I really enjoy kind of digging into those things and really getting into how these things work. My approach was really to be a bit, I suppose, a combination of naïve and sort of just determined to get to the bottom of this.

I tend to kind of – when people use a jargony term, I’ll just ask what that is. It’s like, “I don’t know what your three-letter acronym means. I want to understand what that means.” I think it’s actually not so difficult when you kind of get to the fundamentals. You just have to sort of scrape off this veneer of what all these acronyms mean. I mean, there’s plenty of complexity there, but there’s plenty of complexity in engineering in general. I would say, anybody who’s interested in software, there can be some really interesting security. It’s like problem solving, that’s why we’re all in this kind of engineering space in the first place. We want to kind of understand how these things fit together, and how we could change them and how we could solve problems with them. Security feels to me a bit like a very problem-solving angle of software engineering.

[00:07:00] Guy Podjarny: Very cool. I mean that’s kind of my sentiment as well on it. Did you find, I guess on the flipside of it, you came with a sort of rich understanding of DevOps and kind of cloud and container technologies coming into it. I guess, when you got into this, the industry after all, right, sort of talking to other people in security space. I guess, what was your sense of familiarity with those worlds. What is a high hurdle to get people familiar with them? Is there actually kind of a lot of openness and interest in learning them on. I guess you were in container security, so there must have been some amount of selection bias, but was it hard to teach the other way around, kind of teach that audience about containers in DevOps.

[00:07:42] Liz Rice: I think the biggest challenge and I still say it today is, just people having maybe not quite the right mental model around what containers are. Containers are essentially just processes. They’re not really – especially from the Linux Kernel’s perspective, they’re not any different from a normal process. But sometimes, people come with this idea that because they kind of look and feel a bit like a virtual machine, maybe they are a virtual machine. They’re not. They’re much, much closer to just being regular processes.

I feel like a big part of the education that I’ve been trying to provide over the last few years is explaining to people why they should think of it more as a process on a shared machine. They got a common kernel, what the implications of that are. Just really fundamental things like, “Yeah! If you root inside a container, probably root on the host as well,” unless you’re running with user namespaces, which most of us aren’t today. Yeah, that’s the same root. Don’t be relying on that root being different in a container. That’s going to get you into trouble potentially.

[00:08:53] Guy Podjarny: Yeah, it’s a very convenient –

[00:08:54] Liz Rice: That kind of misconception I would say is the sort of thing that I like to try and clarify for people.

[00:08:59] Guy Podjarny: Got it. Yeah. It’s always easier to sort of relate it to an existing mental model, I guess [inaudible 00:09:07]. I guess, this is containers. I think most listeners are sort of familiar enough with them whether they think of them as a VM or as a process. Let’s talk about this sort of cutting-edge tech world that you live in today. Tell us a little bit about eBPF for starters, like a bit of a primer, like what is eBPF?

[00:09:24] Liz Rice: Yeah. It stands for Extended Berkeley Packet Filter. I usually say that’s really not terribly helpful. It comes from a background in filtering network packets and attaching kind of custom code to decide what to do with those packets. But now it’s been extended to basically any parts of the kernel. You’re not just filtering packets, you’re also able to hook into any function core entry or exit, or a trace point, or a number of other different points across the whole Linux system and run custom code to decide what to do, or maybe to extract data for observability purposes. Because we can hook into anywhere in the Linux machine, but most typically within the kernel and because the kernel is shared across a whole virtual machine. So all of your containers running on that virtual machine have this kernel in common.

That means we can use eBPF hooks to see what’s happening in all of our processes, whether they’re containerized or not, and it gives us this really powerful mechanism for observability and for security, as well as networking if we get back to the kind of packet filtering side of things. Yes, it’s also very performant because we can run it in the kernel. It’s opening up a whole new world of ways to implement networking, and security and observability tooling at runtime.

[00:11:03] Guy Podjarny: What happens to – like eBPF has been getting attention recently. Is it that it’s new tech? Is it old tech that’s been rediscovered? I mean, what kind of brought about its renaissance.

[00:11:17] Liz Rice: BPF has been around for years and years and years. Things like seccomp filtering use BPF. The thing that’s happened is, as BPF and eBPF has evolved in the kernel - new features have been added, new capabilities, new places that we can hook into - this is happening inside the Linux Kernel and it takes quite a long time for a new Linux Kernel to make its way into the distributions that people are actually using in production. We finally crossed the watershed now where most people’s production kernels have pretty decent eBPF capabilities. This wasn’t true a couple of years ago. Almost nobody could run – well, that’s a little bit of an exaggeration, but a lot of people couldn’t run eBPF tooling because they just weren’t running up-to-date enough kernels. Now, most people are, so that’s why we’re seeing this explosion of interest in eBPF tooling.

[00:12:16] Guy Podjarny: Got it. It’s now applicable. You can actually sort of rely on its presence, on your ability to –

[00:12:21] Liz Rice: Yeah. You’re not going to find that you try and install something and it just says, “No, sorry. It’s not supported on your machine.”

[00:12:28] Guy Podjarny: Before we dig into kind of the practicality of what you can do with it, who’s you in this context? I mean, if you’re running eBPF, what type of hat are you likely running? Are you an infrastructure engineer, an application developer? Who typically deals with choosing to install eBPF or with writing code that runs in it?

[00:12:49] Liz Rice: In terms of who’s going to be installing eBPF tooling and eBPF programs, it’s most likely to be infrastructure teams, platform teams. Essentially, people who you would allow to have root access or the ability to reinstall a whole machine. Because we are talking here about changing what’s running in your kernel. Absolutely you don’t want anybody, probably not even any application developer just sort of changing what’s happening in your production cluster at a kernel level.

Yeah, in terms of installing, choosing tools, most often it’s going to be a platform team. the results of those tools, the kind of observability that we’re able to sell offer for example could very well be applicable to the application developer. I think we’re quite likely to see that information being shared more and more with developers, and particularly as we see more tooling being developed in making this. These eBPF programs are quite often running in quite a sort of esoteric kernel context with lots of kind of kernel-related data.

[00:13:59] Guy Podjarny: Talking about acronyms...

[00:14:01] Liz Rice: It needs to be turned into something that’s meaningful for application developers most often.

[00:14:08] Guy Podjarny: I guess, maybe let’s dig into those bits of functionality. What for example can eBPF do and maybe or maybe like which tools or functionality would it replace. What can you run on it?

[00:14:22] Liz Rice: Some of these tools, a lot of people will have come across particularly at the command line. If we go back to – there’s a framework called BCC, which a lot of tools have been written in. There’s a lovely diagram that illustrates the kind of scope of – it’s basically a picture of a Linux machine and it’s got boxes for networking and inside that there's TCP, and IP and sockets and network packets. There’s a similar box for storage. There’s a similar box for user space. There’s memory management. All these different parts of a Linux operating system. Then there are all the tools that have been written in this BCC framework to allow you to get observability data from these things.

Things like TCP dump - that’s a tool that a lot of people would have used to take a look at network traffic and that’s using an eBPF to hook into TCP/IP stack and provide information that we can find very useful for seeing what network traffic is flowing and potentially debugging network issues. Those kinds of command line tools, a lot of them are actually available. You can install them and use them on the command line and have been able to for a while. We’re now also seeing more cloud native implementations. Another example would be – it’s a tool called Inspector Gadget, which takes those BCC tools and hooks them into the world of Kubernetes. You can say run TCP dump, but in the context of this particular Kubernetes card or service.

Then of course, there’s the Cilium which originates at Isovalent and Cilium is using eBPF to provide Kubernetes networking. An example of how we’re using eBPF technology there would be for network policies. Cilium is connecting all the different Kubernetes endpoints, all the parts, all the IP addresses that are associated with these different endpoints. It knows which IP addresses are associated with these different Kubernetes identities. Then we can look at each packet and see whether or not it passes any network policy very, very efficiently because we can drop those packets very quickly in the kernel if they don’t meet network policy. Then we can also provide information so Kubernetes aware network flow.

If you look at TCP dump, it’s all in the context of IP addresses and ports, but using something like Cilium, we can associate that with the Kubernetes identity information.

[00:17:23] Guy Podjarny: And do routing or sort of make decisions based on Kubernetes primitives.

[00:17:27] Liz Rice: Exactly. Yeah.

[00:17:29] Guy Podjarny: Is it fair to think about this not to reduce it again to you were saying, network is just a piece of it, but is it fair to think of it as sort of a kernel level network router, sort of a network switch. Like a lot of the functionality you’re describing is really something that historically would be in some sort of switch or router that sorts of sits in your network.

[00:17:50] Liz Rice: Yeah. Absolutely. I think that is a good way of thinking about it. If we think about the context of container networking, I find this pretty interesting. In a virtual machine – well, in a Kubernetes host, you've typically got one networking namespace per part. You’ve also got a networking namespace on the host. Each of those networking namespaces is running its own stack. What that means is, in a traditional sort of non-eBPF networking environment, a packet arrives from the outside world for one of your pods. It has to go all the way through the networking stack or the host to then get passed through the virtually connection into the networking stack in a pod to reach the container that it’s destined for.

It always had an IP address. It always had its destination IP address. What we can do with eBPF is we can look at that packet the moment it arrives at the host. So before it’s gone through the host networking stack, we can see it’s destined for a pod or essentially to a container on this machine. We just send it straight there and we can completely bypass the host’s networking stack. It’s still going through networking on the kernel because the containers stack is also using kernel primitives, but we’re bypassing a whole chunk of it. That’s one of the reasons why it can be really efficient.

[00:19:26] Guy Podjarny: I think back maybe to your comment before about – to an extent you’re sort of turning that container into something that’s a little bit more a virtual machine, in that context, kind of moving it –

[00:19:36] Liz Rice: Yeah. I guess you could look at it that way. The other thing that’s really interesting and I think the future of this is for service mesh. Because again, we’ve got this awareness of Kubernetes services, so we can still use things like Onvoy to do kind of layer seven inspection. But we can be much more efficient about this, we don’t need to have the sidecar model, because sidecars are used because you typically need to be in the same namespace as the pods that you want to observe or instruments with the sidecar.

If we use eBPF for this instrumentation, we don’t have to have one sidecar per pod, because in the kernel, we’re aware of everything anyway. Using that Onvoy as an example, we can just have one instance per host, which is a lot more efficient.

[00:20:39] Guy Podjarny: This is just to sort of call out what I think are the limitations, like that does require running an operating system. If you’re running Lambda or a serverless platform, can you still run eBPF, like it’s not your machine at that point?

[00:20:52] Liz Rice: Yeah. You wouldn’t have access typically to sort of running eBPF functionality in that kernel. I don’t know whether if we look into our crystal ball, we might see a future where people will be allowed to attach particular- It would make sense to me to say, even though we can – if I’m a cloud operator, I might say, “Well, I’m controlling the kernel and I’m going to be very specific about what I allow you to do.” But maybe I’m going to allow you to turn on certain eBPF functionality, or load certain eBPF programs.

[00:21:27] Guy Podjarny: Right, yeah. Or implement maybe some of general capabilities through sort of eBPF themselves.

[00:21:33] Liz Rice: Yes.

[00:21:34] Guy Podjarny: I think eBPF sounds somewhat all powerful. Everything we talked about until now were really around networking capabilities and around maybe more performance or usability around those kind of Kubernetes primitives. When you think about security, how does – what are sort of examples of what you can do, where eBPF can benefit security.

[00:21:57] Liz Rice: I mentioned network policy. That’s one angle of kind of run time security enforcement. Another area that’s pretty interesting is executable and file access security. Some of the things that we might traditionally do is APAM is SELinux. Controlling which processes can access which files for example. It’s pretty complicated to set up those profiles. APAM or an SELinux weren’t originally designed for a container environment, although it’s quite complex. But if we’re in a microservices environment, typically, our software is doing – that microservice is doing a pretty limited set of things. It’s only expected to access a certain set of files, or run a certain set of executables.

Quite often, the timing of running those executable– if you’re running a microservice, you might have some initialization and then you pretty quickly, it’s running one thing. If couple of hours later, it suddenly started running something new, maybe that’s a sign of compromise. Maybe we look at what that executable is and it turns out to be a cryptocurrency miner or something. This would be bad. We can pretty easily think and build a mental model of what we expect our services to do in terms of file access, and network access and executables. eBPF gives us this ability to see that file access. We can hook into reading and writing files. We can hook into network connections. We can hook into running executables.

We can start to use eBPF to police that. Already, there are tools today like Falco and Tracy from my former colleagues at Aqua that will look at the executables that are running and match that against the known profiles. There is more work going on in the eBPF world to not just kind of observe that these potentially malicious things are happening, but also to be able to put controls in to say, “No, you can’t run an executable, unless it matches a set of –

[00:24:24] Guy Podjarny: [Inaudible 00:24:24]

[00:24:25] Liz Rice: Yeah, of that policy.

[00:24:26] Guy Podjarny: I guess kind of graduating from network policy to execution policy or sort of other kernel commands.

[00:24:33] Liz Rice: Absolutely, yeah. Just really restricting what each of those microservices is allowed to do in quite a powerful shrink-wrapped way.

[00:24:44] Guy Podjarny: Yeah. One of the things I find interesting around eBPF is trying to kind of wrap my head around the – is this user space or is this sort of platform space, is it – I mean, writing for starters, you mentioned there’s the permissions element of it, you wouldn’t want just anybody modifying eBPF on your system if nothing else, because it runs from the entire system and all the containers that run on it. And also, similarly, like it’s not easy to write eBPF, right? I mean, it sounds like pretty, hairy, low-level codes you need to create.

[00:25:20] Liz Rice: It is. It is a bit of a sort of specialist’s endeavor, and it’s quite fun. Fun in the engineering sense, because – one aspect that makes it quiet fun: verification. When you’re running code in the kernel, it’s very important that it doesn’t crash or get into an infinite loop. There’s a verification step that happens before any eBPF program is loaded. The kernel will verify and what that verification step is doing is looking for loops and it’s looking for memory access, making sure that all the memory accesses is safe and as expected. That means you have to, for example, every time you dereference a pointer, you have to explicitly check that it’s not nil. If you don’t have a check for nil pointers is, then your verification step one will pass and you won’t be allowed to load that program.

The verifier helps make sure that these BPF programs are safe to run and that they’re not accessing memory outside of the scope of what they’re supposed to, but it can be a bit of – there’s a nice quote from NCC group. It’s something about the verifier being a fickle beast or something like that. It certain can require a little bit – sometimes just moving a line of code somewhere in your – just reordering it can persuade the verifier that it’s safe, even though it looks to my brain of sort of exactly the same. That can be a little bit fun.

You also can quite quickly end up needing to understand the kernel context. You’re running this custom code that you’re writing in the context of something that happened typically in the kernel. You might have a kernel socket buffer to look at and you’re going to have to know what that socket buffer, how it’s structured. You’re pretty quickly getting into some nuts and bolts there, some pretty detailed nuts and bolts.

I don’t expect many people, most application developers are not going to also be writing eBPF code. They’re going to be using eBPF tooling, but I don’t think they’re necessarily going to want to – and it's fun! I don’t want to put anybody off if they’re interested in that level of getting into the weeds there.

[00:27:57] Guy Podjarny: Such as that sort of engineering bone over there. I think that that’s an important part I guess kind of aspect to grok here, which eBPF sounds super powerful. I think for most of us, it wouldn’t be because we’re going to find ourselves replacing JavaScript for eBPF. That’s not the sort of the right tradeoff or to think of it.

[00:28:20] Liz Rice: Yeah, exactly.

[00:28:21] Guy Podjarny: I guess go, – maybe you’re sort of starting to go rust, you’re kind of getting a little bit closer, but still, it’s even lower level than that even more involved. But I guess it’s an important capability to help all of us understand the art of the possible. It might imply things, would be some order of magnitude faster or closer or where is it that you can apply. So it is kind of like sidecar. Maybe that’s the best way to think about it, eBPF sort of unlocks possibilities like sidecar unlocked possibilities because suddenly you can install tools in this fashion, you can apply rules and apply capabilities in this fashion. Just like sidecar has become, it comes from something that none of us ever knew. Maybe a few more people are writing sidecars and eBPF, but it unlocks possibilities. eBPF is doing something similar.

[00:29:11] Liz Rice: Yeah. We can certainly think of it as a similar way of stepping up the level of, I guess what we can see. We had that sidecar model to allow us to inspect what’s happening inside a pod. I guess eBPF allows us to say, actually, let’s take that. We’re going to step it up a level and see everything across the whole virtual machine.

[00:29:36] Guy Podjarny: Yeah, for sure. It’s interesting a little bit to see the tensions may be between on one hand a desire for some to move to more of a serverless surrounding, try to forget their servers under there and move away, versus the opportunity with eBPF, which is sometimes – well, if you remember, there’s a server under all of these containers, you can actually do some fancy stuff.

[00:30:00] Liz Rice: Exactly. As an end user, you may be completely oblivious to what’s happening on those servers, but somebody’s got to manage those servers and somebody has to keep those servers safe, and maybe they’re using eBPF even if it’s not made accessible to you as the serverless user.

[00:30:22] Guy Podjarny: Yeah, absolutely. I think eBPF is fascinating. Thanks for kind of giving us here sort of a bit of an overview of how to think about it, where are the capabilities. People definitely should sort of check out Cilium, and I guess Isovalent is the commercial backer there.

I want to sort of use the time, there’s so much to talk about. Use some of the time over here as well to sort of switch to a very different topic, a very different hat that you are wearing. You are also, as I mentioned at the start, you are the chair of the Technical Oversight Committee at the CNCF. I’d like to think that most listeners are sort of familiar with the CNCF, but do you mind giving us a couple of words about CNCF and what is the Technical Oversight Committee.

[00:31:06] Liz Rice: Sure. Yes. CNCF stands for Cloud Native Computing Foundation. It’s part of the Linux Foundation and it's the entity that owns the intellectual property and provides a home for projects like Kubernetes, and Onvoy and Linkerd. There are just dozens and dozens of projects now. They are all in the realm of cloud native.

[00:31:32] Guy Podjarny: The biggest foundation I think within the Linux foundation sort of –

[00:31:38] Liz Rice: It is, yes. I think it’s something like the fastest-growing foundation of any open-source foundation I believe. Certainly, hundreds of thousands of contributors across the project. As the Technical Oversight Committee, we’re there to set or perhaps also enforce the technical direction of the foundation, so kind of what do we mean by cloud native. If a project wants to become part of the foundation, is it cloud native does it meet those attributes. We have a pretty light touch as a foundation, so we don’t require any particular governance model. We don’t do take all those governance models have to be, but we do want to see open governance in every projects. We do exercise our level of kind of policing around, making sure that projects are well run.

With the TOC, we’re responsible for assessing whether new projects should join the foundation. We also look at the level of maturity whether they’re experimental and should really be in the sandbox through to graduated projects where we’re saying, “Yeah that’s a well-ran, well-established project with lots of adoption. As an enterprise, we think you’re not taking any significant risks to use those projects.” I think like Kubernetes, Prometheus and Onvoy and several others fit into that category. In between, we have incubation where they’re being used by at least some people in production – potentially, there are some things we would like to see improve about governance model. We want to see a bit more track record of collaboration, but they’re nonetheless pretty well-established projects at incubation level.

[00:33:35] Guy Podjarny: Right. In the collaboration, you’re sort of implying you don’t want sort of a single commercial party to be kind of dominating any of these projects, because they’re meant to be an industry collaboration. Is that what you’re referring to?

[00:33:47] Liz Rice: That’s right. I mean, we want to – I think it’s important that the CNCF – it’s part of a community and that community has everybody from individual contributors up to giants, the big cloud providers, and in between, lots of independent software vendors as well. We want it to be an environment that allows everyone to be successful, but I think that does mean the CNCF isn’t there just to give a vendor’s project, a leg up in marketing. It has to be able to be a truly community-oriented project. We prefer to see multiple collaborating entities, whether they’re vendors or end users. We like to see lots of organizations being involved in a project.

[00:34:36] Guy Podjarny: Yeah, makes perfect sense. I think from a security lens, the CNCF and maybe specifically the Technical Oversight Committee, have a dual role, like there’s an aspect of, how do you help CNCF project be secure. And there’s maybe the question about CNCF’s role in helping all of us build secure applications on top of sort of CNCF or secure platforms, how people use CNCF’s products securely. Is that correct? Maybe I’m over interpreting things. How do you think about security in the context of the committee?

[00:35:18] Liz Rice: I mean, it’s definitely something that I wouldn’t say is completely – it’s not finished. There’s more that we can do in the world of security. For the TOC, we – there were 11 people on the TOC, but we’re a much bigger community than that. We have a set of technical advisory groups that we kind of rely on and outsource for their expertise. One of those tags is security. That is actually I think the most active of all the tags. If there are people listening and interested in getting involved in the CNCF’s security work, definitely reach out to the security tag. There are some amazing people and some really good projects that you can get involved with.

They’ll do things like provide a security buddy for a project that – a lot of projects don’t necessarily have security expertise on board already. But the security tag can help match them up with someone who can take a look at maybe some of the security aspects of a project. That’s a relatively new capability that the communities kind of pulled together to offer its projects to help them be more secure.

[00:36:32] Guy Podjarny: The primary objective over here is to help the project themselves to be secure, like minimize and then address if they happen from our abilities in the projects, secure access to the code, et cetera, et cetera, as opposed to their security capabilities necessarily that they –

[00:36:48] Liz Rice: Exactly. Some projects have nothing particularly security related about them. But they –

[00:36:55] Guy Podjarny: But all of them should be secure.

[00:36:57] Liz Rice: Exactly. Also encouraging things like a responsible security reporting process. That’s something that we’ve added into the graduation party recently and something where we become conscious that, “Yeah. Actually, projects do need to have that root for people to report potential security issues.” Also, the CNCF for graduating projects provides security assessments that are done by a third-party, full kind of pen testing assessments. That’s another, I guess service that the CNCF is offering to the user community. You can look at the assessments that have been done on those graduated projects. I think not all of them have been done yet. It’s a work in progress, but it’s a pretty significant investment.

[00:37:51] Guy Podjarny: Yeah. To an extent and I know security isn't the specific focus over the top. But it’s a sort of a massive volume of developers. You mentioned before, over a hundred thousand contributors, developers, they’re building systems that in turn are massively prevalent oftentimes by definition of being a part of this foundation. Yet, you’re nobody’s boss in this context. Like there is nobody that’s sort of telling anybody what to do. I guess, what’s your sense of how that’s working out. It’s to an extent, the very essence of empowered developers, sort of building those out. But also, boy is it a high-impact surrounding if you slip up. How do you manage that or how does the CNCF manage that?

[00:38:43] Liz Rice: I think it’s incredible how these hundreds of thousands of people have been able to pretty much self-organize in a pretty successful way to develop in all these different projects. I think that that self-organization didn’t come from nowhere. It came from some really intentional thinking by people in the early days of the CNCF and in the early days of Kubernetes, thinking about how contribution should be sort of managed and contributors should be onboarded.

I’m thinking about people like Sarah Novotny and Paris Pittman and the people involved today in the contributor strategy tag that’s part of the CNCF. People are thinking about how these groups of essentially volunteers can manage and what’s the sort of best practice ways of providing ways for people to grow in those roles and go from their very first contribution through to leadership roles in open source. Yeah, it didn’t happen by accident. It happened because people put some really careful thinking into it. I’m thoroughly grateful for that.

[00:39:57] Guy Podjarny: Yeah. I mean, I think in my mind another, probably a dedicated podcast topic on its own there.

[00:40:04] Liz Rice: Absolutely.

[00:40:05] Guy Podjarny: We’ll dig into that. One more question on CNCF before the closing one here, because I think we’re going pretty long to so many things. I guess, as anticipated. How do you think the CNCF is building the foundation or much of the foundation for cloud native computing indeed? Historically, it’s been much more ops oriented, sort of DevOps oriented, not necessarily app dev. There have been some works on sort of application, working groups and things like that. When you think about the CNCF and where it’s progressing towards, do you think more projects there would be app dev oriented? Do you think the CNCF’s role is really still in the infrastructure layer at the DevOps layer? How do you see that?

[00:40:48] Liz Rice: No. I do think that application development and delivery aspect is very important. We see projects like Backstage that came from Spotify, which is a developer platform. I think we will see a lot of other projects and there are several other projects in that area, whether it’s about how applications are built, things like build packs, how developers interact with that platform. The whole world of GitHubs and how development codes get into production and how that sort of pathway’s management. I think these are all areas where we will see more projects, some consolidation I think in much the same way we standardized on Kubernetes and we have a pretty good common understanding across industry on how we manage Kubernetes clusters. I think we might start seeing more standardized ways of how we deliver application code on to those clusters, yeah.

[00:41:56] Guy Podjarny: Yeah. I think that sort of is consistent with what I’m seeing here and I think we’ll be building a lot more. Liz, thanks for this sort of view into maybe the foundations on top of what you’re building. I know I think eBPF from an exciting new evolution and what we can do with it. The CNCF is maybe where a lot of that gets coordinated and packaged up for us to consume.

Before I let you go, one last question I like to ask all guests coming on the show. If you take out your crystal ball and you think of someone in your shoes five years from now. I don’t know which shoes those are, maybe you’re indeed sort of orchestrating. You have a lot of things going, but thinking about maybe this reality from an industry leadership, from a – whether it’s the open-source activities or the open standards. What do you think in five years would be most different about your reality?

[00:42:49] Liz Rice: Five years is such a long time, isn’t it in this space. Five years ago, Kubernetes was barely – it was just a new thing. We didn’t know –

[00:42:58] Guy Podjarny: Like eBPF was.

[00:43:00] Liz Rice: Yeah, absolutely. Setting aside whether or not we’ll have the climate apocalypse in that timeframe, let’s hope that doesn’t happen. I mean, we’re seeing such rapid adoption of cloud technologies. I think it will be table stakes for any company to be just using the cloud. We won’t be talking about early adopters anymore for sure. It’s so difficult to say. I’m sure we will see huge advances in observability. We’ll see massive scale. We’ll see the proliferation of things like edge and the Internet of things. So many more entities that we’ll need to sort of monitor, and secure and control. I think that will add a whole new layer. I think things are pretty massive scale already, and I think that’s only going to explode over the next few years. We’ll all have our work cut out for us, building the tools that will manage that, and secure that and allow us to keep an eye on what’s happening.

[00:44:13] Guy Podjarny: Yeah. Sort of seeing it getting messier before it gets cleaner, I guess in the evolution. More and more moving parts.

[00:44:20] Liz Rice: Yeah. I mean, everything is always up in about an arm’s raise, isn’t it? Things get worst and then people build their solution and then things get worst and people build a solution. I suspect that will be true with the scale of Internet-connected devices.

[00:44:33] Guy Podjarny: Indeed. Liz, thanks for coming on the show. This was a great conversation. I do fully believe we probably could have spoken for another hour and still kind of scratch a lot of things. But thanks for coming on and sharing more knowledge and views with us.

[00:44:48] Liz Rice: Absolutely a pleasure. Thanks for having me.

[00:44:51] Guy Podjarny: Thanks everybody for tuning in and I hope you join us for the next one.

[END OF INTERVIEW]

[00:44:59] ANNOUNCER: Thanks for listening to The Secure Developer. That’s all we have time for today. To find additional episodes and full transcriptions, visit thesecuredeveloper.com. If you’d like to be a guest on the show, or get involved in the community, find us on Twitter at @DevSecCon. Don't forget to leave us a review on iTunes if you enjoyed today's episode.

Bye for now.

[END]