The Future Of Software Supply Chain Security

EPISODE 129

Jim Zemlin: “I think in five years software supply chain security will be significantly improved, order of magnitude improved. And here's why. I'm pretty optimistic that there's an industry will, and I want to thank you for your leadership in being a part of that, to go and solve these problems. 

I think in terms of building automated tooling to enable software bill of materials metadata to freely flow in a very easy automated way throughout version control systems, package managers, build systems, and into an end consumer software manifest, is going to happen in the next couple of years, if not sooner. I think we can go train software developers on how to write secure code over the next few years, that will significantly decrease the number of unintentional vulnerabilities that go into software every single day.” 

[INTRODUCTION] 

[00:00:59] ANNOUNCER: Hi. You’re listening to The Secure Developer. It’s part of the DevSecCon community, a platform for developers, operators and security people to share their views and practices on DevSecOps, dev and sec collaboration, cloud security and more. Check out devseccon.com to join the community and find other great resources.

[EPISODE]

[00:01:21] Simon Maple: Thank you for joining us for part four of our Software Supply Chain Miniseries. So together, we went on quite the journey on dissecting the complexities of software supply chain, with the help of experts, we really had the pleasure of speaking to on the podcast. We defined what software supply chain really means, and why we must pay attention to it in 2023. We also looked at the key players and the initiatives to know about and also the strategies to implement in your organisation, to tackle the software supply chain security threat.

In 2023 and beyond, software supply chain will continue to be a priority for organisations, leaders, and the developer and security communities. So, what can we expect to see in the future for software supply chain? What will 2023 and beyond hold for us in this space?

To kick things off, we have insights about the challenges and future of open source software from Emily Fox, Security Engineer, who also serves as the co-chair of the CNCF Technical Oversight Committee. And Aeva Black, Open Source Hacker at Azure Office of the CTO, Microsoft. Brian Behlendorf, General Manager of the Open Source Security Foundation, and Guy Podjarny who you all familiar with, Founder of Snyk, and host of this podcast.

[00:02:29] Aeva Black: Really address the threats that I see today in software supply chain security, particularly in open source, we need engagement from open source package managers, and distributors, everything from how Debian and RPM packages are built. And there's work happening on reproducible builds to the distro publishers. PyPI or NPM.

We also really need companies to be more involved in funding this work. It doesn't directly relate to revenue for most companies, but it's critical to safe and successful operation and revenue generation. So, we need more funding for security work to happen upstream. The corollary there is federal government, who really drives a lot of the policy, at least in the US, and the European Commission in Europe, for how incentivised the industry is as a whole to do this. Companies are going to do as much as they need to, and no more for themselves. But when the government says you have to reach this bar, and we saw that starting last year, at the executive order on supply chain security. A whole lot of companies are doing more to meet that bar.

And that was a net positive for the industry. And a little bit of confusion at the mass, as everybody rushes to do it. But I think it'll be a net positive and I'd like to see more of that, more engagement between companies. Open source foundations and governments around the world. 

[00:03:49] Brian Behlendorf: There's a lot of this kind of collective understanding that I think needs to come about to help address some of these issues, and I never want to get to the point where we're like charging people to use open source. You really can't do that, legally, anyways. But you can start to, I think, send messages that say your enterprise is at risk, unless you treat this a little bit more like a grey box than a completely black box. If we can be helpful in organising some of that upstreaming effort, I think we've done some good.

I think there really is hard to come up with metrics to understand what's the maturity level of organisations as they approach both companies that use and produce open source code, but really the foundations that are managing this. But I would say, if we could – one metric, of course, will be best practices badges, security scorecards, those kinds of things, and hopefully we'll be able to track adoption of those and improvements and fewer of the Log4j style messes rather than, “No, we can't guarantee zero. But hopefully we can see fewer of those.”

But generally speaking, if we saw more of the major open source foundations out there, and obviously there's some inside the Linux Foundation and some outside. It’s on us to fix. But the more that we see the foundations out there taking security seriously, spending as much on security as they do on marketing, for example, putting some dedicated time towards it and making it an expectation on the part of developers that if you're going to write code, it's not enough just to write something that scratches your own itch, and then you throw it over the wall. Think about your user community. Think about the risks that you put people through. If we saw more of a culture of that, then I'd be happy.

[00:05:16] Emily Fox: The other part of it is getting more individuals approved to be able to contribute to open source security. That's one of the ongoing challenges that we have in open source, and it's one thing to be able to find a vulnerability and report it, but it's an entirely different thing to also provide a fix for it. And that in my mind is the mark of a strong engineers that you're capable of finding the problem, but also coming forward with solutions to be able to select from and solve it at the start, instead of waiting.

So, the Open Source Security Foundation’s Open Source Mobilisation Plan does have streams that kind of cover a lot of this. There's been ongoing discussion about how do we incentivise and recognise individuals that are doing the chop wood, carry water work of making these kinds of impactful changes on a project by project or community by community basis to solve some of these problems. There's still a lot of work to be done to folks who are interested. I highly recommend joining the foundation's Slack channel and seeing how they can get involved there. 

There are items such as a security incident response team for foundations, which is just one way in which we can provide that kind of human capital, to be able to assist projects as well as security tooling being written to automate a lot of this.

[00:06:34] Guy Podjarny: So, in the ideal world, every open source component, every component upstream will track its activities and will attest to it, and give you some statements as I conform, I attest, that I conform to this policy in a trustworthy fashion, maybe even with some standards like SOC 2 or like FedRAMP, in which you said, “Well, this has conformed to this ISO like standard that can now cascade downstream.” That, as an organisation, if you want to consume some open source component that does not have those attestations, you know that and you can make the decision of saying, “Are you okay with that risk or not?” Maybe in some systems, you wouldn't be at some systems, you wouldn't, similar to what we have in the world of physical goods.

This is a complicated and powerful eventual reality. It is very far from us, even if all the open source maintainers in the world joined forces today and tried to do this, the tooling is lacking. There's no alignment and what's the right way to do it. And also, of course, in practice, it's not necessarily the top motivation, the top priority for every single person. But hopefully, it's the eventual destination that we are aiming towards.

When you look at the broad ecosystem, another future reality that we might aspire to is around making dependencies more declared. Today, when you consume some open source piece of software, you kind of need to reverse engineer, which dependencies may or may not be inside of it. When you're trying to look at some piece of software and to try to determine which dependencies it would use, you oftentimes again, need to sort of approximate, or sort of assess those things. Software never dies. That would probably not go away, because components will stick around. But the hope is that we evolve towards a place in which when there is some package that you are downloading, it is much more declarative around the dependencies that it wants to use, and that those cascade down the process.

Now, this is not as easy as it sounds, because people think about dependencies, simplistically, but if you're using component A and component B, and each of them come with their own graphs of dependencies that they're using, there's some intersection between those graphs. If A and B both use some version of library of C, you need to do resolutions. And so, in practice, you can’t really know all the dependencies that a component would have, without looking at their combination, their holistic. But still, there's a lot that we can do in the ecosystem to be more declarative, around the components we're using. It will just make it easier to deal with this complexity downstream.

Interestingly, the concept of attestations of software, and everything from downstream can actually be applied to the fake news problem as well. If there was a trustworthy validation on every component on every piece of social media that attested to the fact that it came from the source, and that it is been peer reviewed and has been validated, you can cascade that information downstream. And when you consume news, you'd be able to sort of see, has this been verified by the right compliance view downstream, so that you would know that it's trustworthy to that degree. In practice, probably entirely improbable that that would happen. But it just another example of how kind of fake news and software supply chain securities are really parallel problems.

[00:09:43] Simon Maple: The prevalence and consumption of open source software as a key challenge for software supply chain security. This will drive many initiatives and motions in the future. As Aeva, Emily, and Guypo shared, there will be a need for more engagement for the open source community, and this includes the maintainers, distributors, but also the large companies, that are the consumers. The government will have to continue to play a pivotal part and the right recognition and incentivisation will be at the core of this. There'll be maturity in the space of open source attestation and dependency declaration.

But it's not just about the integrity and support for open source software security, that is imperative, as Adrian Ludwig, Chief Trust Officer at Atlassian, and Lena Smart, CISO at MongoDB share, SBOMs will continue to play an important part and not just know what components you have, but know that they're up to date, and what compromises are contained would really be equally crucial.

[00:10:37] Lena Smart: This is a great opportunity for us as companies to look and see how we can better supply SBOM information for the government. But also, there should be a way we should always be looking to streamline this process and keep them as clean as possible. Because my worst nightmare as just for talking sake, we've got 100 government agencies, and there's going to be 100 versions of our SBOM, because every one of them want a separate one. Whereas the perfect world would be, there's one storage space, and it's secured. The 100 government agencies have a log in, and they just say, “Oh, I want to see the SBOM for MongoDB. And I want to automate it by doing, blah, blah, blah, this API.” And you've got your one source of truth, you've got your one SBOM per company, and this secure MongoDB repository being created, continually monitored by Snyk. I mean, there we go, just solve the SBOMs in two minutes. Only if it was that easy.

[00:11:38] Adrian Ludwig: As we've seen it so far, like we tend to think of ingestion of open source projects as just a win. And what we lose track of is, that there's a long-term management costs associated with that. So, the more we can automate that management, the better. In my world right now, the way that looks, is getting to the point where having the most recent, most up to date version is the default. So, figuring out how to manage that as source is also probably important.

But regardless, we have to flip the models so that the default is that it's up to date, as opposed to default is not up to date. Legitimate security boundaries between those components. Being able to have confidence that any exploitation is isolated and contained only within those areas. Maybe again, maybe that's base image, and the base images are actually isolating them out. I don't know, in a serverless kind of model. Maybe it's something else. But that level of isolation would be amazing. If you could both know that your components are up to date, and that any compromise of them is contained within just that component.

[00:12:36] Simon Maple: There's a lot of work to be done to better consume and secure open source software to track our pipelines, to optimise SBOM utilisation, and bring everyone together in this effort. So, what would software supply chain look like in five years’ time? We fielded these questions to Jim Zemlin, Executive Director at the Linux Foundation, Brian Behlendorf, Adrian Ludwig, and Aeva Black.

[00:12:59] Adrian Ludwig: It's shocking to me that some of the basic exploitation mechanisms, buffer overflows, have been around for 50 years. Heck, I've been exploiting them for 30 years. Yet, we continue to write core parts of our technology in these languages that have problems and using methods that are known to have problems. So, I think longer term transition away from known risky languages and methods to less risky. Rust, I think is really attractive in that regard. It's odd to me that we prioritise performance, so high and at the same time, have layer, upon layer, upon layer, upon layer of abstraction that just destroy performance. I think that as a long-term bet, is going to be really, really valuable. Getting clear security contract baked into the languages and tools that people are using.

[00:13:46] Brian Behlendorf: My hope is that in five years, people's very high default reservoir of trust that they have for open source doesn't lead us to another crisis, that it actually can be substantiated by the open source community getting its act together. How we measure that is still a work in progress.

[00:14:03] Aeva Black: But I hope that a lot of the projects that are happening right now, in and around the OpenSSF, and Cloud Native Computing and reproducible builds and observability, in build process, I really hope those all continue to grow and take root. If they do, in five years, I think we'll have a much better sense of security around downloading software from the internet, Curl-pipe bash install things is a really unfortunate anti pattern that so many places are still using. Install something off Docker Hub, or just download the build file off GitHub and run that inside your company with no auditing, another anti pattern. So, a lot of work is going into kind of close those gaps. Those gaps only emerged in the past five or eight years as open source sort of grew in success. Now, we have to play catch up. So, I would love to say that in five years, we will have closed all those gaps. I think there will still be some. Hopefully, in five years GitBOM will have succeeded in that work and it'll permeate the build tooling.

So, when you build something with GCC, or Go, or Rust, it'll just automatically contain this sort of chain of fingerprints of all your dependencies, and that'll make discovery of issues and remediation of vulnerabilities so much faster. So, I think we'll have much better tooling in five years.

[00:15:24] Jim Zemlin: I think in five years software supply chain security will be significantly improved, order of magnitude improved. And here's why. I'm pretty optimistic that there's an industry will, and I want to thank you for your leadership in being a part of that, to go and solve these problems. I think in terms of building automated tooling to enable software bill of materials metadata to freely flow in a very easy automated way throughout version control systems, package managers, build systems, and into an end consumer software manifest, is going to happen in the next couple of years, if not sooner. I think we can go train software developers on how to write secure code over the next few years, that will significantly decrease the number of unintentional vulnerabilities that go into software every single day.

I think we're going to have much better tooling. Your product gets better every single day. Your competitor’s products get better every single day. And there's going to be a lot better tools available to software developers to catch bugs, before that software gets distributed into critical infrastructure. I think we're going to go look at sort of the software that the industry has depended upon for decades and retire a ton of technical debt. The industry is already stepping up to resource developers to go and fix vulnerabilities in the Linux kernel and Log4j and for other critical packages. I think that's going to increase.

I think we've got work already happening in better security around software distribution and package managers through things like Sigstore that provide for cryptographic signing, already being adopted. I mean, you put all these things together, and at least the incredibly low bar attack vectors will be a lot harder to exploit. That is going to be a big deal. The biggest difference this year than last year is a serious collective industry and societal will to get this stuff done. What will happen, is in five years, we will all be collectively better off. But unfortunately for companies like Snyk is going to happen is attack vectors will be way more sophisticated. The burden on tools like Snyk to secure this code will be even higher. But I'm also confident that companies like yours and others, are pretty darn innovative and can keep up with the bad folks and coming up with creative ways to secure software as they come up with creative ways to attack us from a security perspective.

[00:18:15] Simon Maple: So, it sounds like collectively, we would be better off as long as we as a community of software professionals and developers, we continue to take strides in the right direction, and those strides would come in the form of community initiatives and collaborations, supporting engagement with maintainers, contributors, developers and companies with the right incentives and recognition. Innovation in technology and security solutions paved the path for auto attestation, security verification that builds on the trust for open source. We'll have to figure out how best to use SBOMs to really know what we're consuming, and the journey all the components have had.

There is a wealth of knowledge that has been shared in this miniseries and a lot of great work being done in this space. And we hope that through this miniseries, we've shared some great foundational pieces that you can continue to build upon in 2023 and beyond. We would love to hear your feedback on this miniseries, whether or not you'd like to see us revisit this topic later in 2023. Thank you for joining us on this miniseries as we deep dived into software supply chain security. Until next time, stay secure.

[OUTRO]

[00:19:29] ANNOUNCER: Thanks for listening to The Secure Developer. That's all we have time for today. To find additional episodes and full transcriptions, visit thesecuredeveloper.com. If you'd like to be a guest on the show, or get involved in the community, find us on Twitter at @DevSecCon. Don't forget to leave us a review on iTunes if you enjoyed today's episode.

Bye for now.

[END]