The State Of Open Source And Docker Security With Liran Tal

"Liran Tal: When we asked developers, who do they think should own security for their application obligation? Eighty-one percent of them said developers should own that. The DevOps thing is kind of empowering developers to own their infrastructure. So as they take more ownership around security, that's kind of the good security posture that you want developers to have when they build applications. When we ask maintainers, how do they find out about vulnerabilities. Half of them find out about it when other people are opening public issues on the project. There is a good intention, but not enough action on it to mitigate the security problems.

[INTRODUCTION]

[0:00:36] Guy Podjarny: Hi, I'm Guy Podjarny, CEO and Co-Founder of Snyk. You're listening to The Secure Developer, a podcast about security for developers, covering security tools and practices you can and should adopt into your development workflow.

It is a part of The Secure Developer community, check out thesecuredeveloper.com for great talks and content about developer security, and to ask questions, and share your knowledge. The Secure Developer is brought to you by Heavybit, a program dedicated to helping startups take their developer products to market. For more information, visit heavybit.com.

[INTERVIEW]

[0:01:09] Guy Podjarny: Hello, everybody. Welcome back to The Secure Developer. Today, we have Liran Tal, joining us here. Welcome, Liran.

[0:01:15] Liran Tal: Hey, Guy, thank you for inviting me.

[0:01:17] Guy Podjarny: Thanks for coming on the show. We have a lot of topics, we have a data-packed element here. Liran, you work at Snyk here, and you ran a couple of security reports we're going to talk about.

[0:01:28] Liran Tal: Indeed.

[0:01:29] Guy Podjarny: These are the ‘State of Open-Source Security’, talking about open-source security as a whole, and then one on Docker security. Before we dig into sort of the details, I guess on those. Tell us a little bit about yourself. How did you get into this one full world of security?

[0:01:41] Liran Tal: I guess I've been doing that since an early age. Security has been a large part of my life since then. From the early days of 2600.com, and IRC channels and stuff like that, I've been kind of running my own BBS, just getting acquainted with the digital world at that point in time. So, been doing that for a while. Then, I think in the last maybe couple of years, I've been diving more into Node-specific security. I managed to write a book about it, kind of put a lot of best practices and guidelines for developers how to secure their Node applications. A lot of Dot and OWASP involvement, and stuff like that. That got me into working full-time helping secure the system at Snyk.

[0:02:25] Guy Podjarny: Cool. Well, that's basically go from nerding out on the infrastructure, to sort of getting to securing the apps on it. Cool. Let's dig into the report. First of all, we're talking about these two sort of state of X security type of reports. What's the motivation for doing these types of reports in the first place? Why bother?

[0:02:44] Liran Tal: A lot about it is just exposing this information out, so sharing a lot of those insights, and just exploring the state of the ecosystem around security. A lot about the report, actually, the stuff that I really like about it is, it really covers kind of a 360 view of the ecosystem. It's not just focusing on application libraries or operating systems, it's combining all of those different data points by serving people and asking them questions, maintainers, and developers about their practices in security. A lot about it is kind of measuring the state of the security in open-source ecosystems and communities. And of course, the software that we have. Everyone are building software based on open-source, kind of, so you can't run away from it. It's good once we have the visibility into what's going on there.

[0:03:30] Guy Podjarny: Yes. I guess there's kind of a lot to unpack there. One aspect is, you can't optimise what you can't measure, I guess. If we put some measurements on it, then we're able to maybe make it better. Have you indeed seen much conversations that flowed from those conversations, from the stats that were published?

[0:03:47] Liran Tal: Right. I think the Docker stats really surprised a lot of people. The idea, the notion that people are surprised by the amount of vulnerabilities they are shipping with Docker containers to production is something that I think really hit vulnerability points around social media. Because, people were really surprised about the amount of vulnerabilities present in their Docker images, what they build their containers on, and that really made a good hit.

[0:04:15] Guy Podjarny: I guess the – this is kind of a bit of a recurring theme on this show, which is the importance of visibility and security. Security is naturally invisible. So, doing things that just raises it a little bit to your awareness, the fact that there is a security concern, or let alone a security breach is kind of worthy of highlighting. Now, let's kind of switch into the actual report. First of all, how did we kind of get to this point. What data gets collected, how are the reports kind of built up and structured?

[0:04:49] Liran Tal: We looked at different data sets. One of them is the data elements of it are basically looking at Snyk-related data from customers and all the users we have, looking at how people or developers actually use software in their projects. How many of that is vulnerable? What is their kind of versioning schemes around it? We also looked at just open-source repositories, things like the ecosystems, the registries, and GitHub-related data itself. Lastly, had a human element to it, which asking several hundred people in a survey what is their security posture around open-source security specifically. Kind of trying to measure a lot of the human element and blend that with the data that we collected.

[0:05:36] Guy Podjarny: Was the data fairly consistent, where there are sort of discrepancies between the data that we saw, and the answer is people said?

[0:05:45] Liran Tal: Not really a lot of those. I mean, the general trend being that there's an uprising trend in terms of more vulnerabilities that we find in open-source libraries, in terms of application libraries, and operating system libraries, the trend is just up. Meaning, we are finding more and more each year more vulnerabilities in the ecosystem. There's an interesting trend around the survey itself; whereas, developers or maintainers state that they want to own security. On average, maintainers have 6.6 out of 10 skill level of security. That sounds like a medium. That's fairly high to rate yourself, on average.

Then, at the same time, when we asked maintainers on, how do they find out about vulnerabilities. Half of them kind of find out about it when other people are opening public issues on the project. There's kind of the effect of maybe people rate themselves as having high-security knowledge. But at the same time, they don't really find out about the vulnerabilities themselves.

[0:06:50] Guy Podjarny: Yes. The actions don't necessarily speak for that. Well, I guess we start with good intent, and then we proceed from there.

[0:06:57] Liran Tal: True.

[0:06:57] Guy Podjarny: Okay, interesting. Sort of putting aside the fact that we, as humans, are pretty lousy at sort of assessing our skill levels. We have these data sources, so you run the survey, you run the analysis on a bunch of data. You collected some data from Snyk usage and collected all of those into these two reports. I know the reports are fairly kind of intermixed a little bit in the data, and maybe the insights that they have. But let's still try to kind of break them apart, let's start with this ‘State of Open-Source Security’ report. What are the key insights? We're not going to go through the full data set from there but give us a highlight or three.

[0:07:35] Liran Tal: Sure. I think, as you said, open-source software maintainers, they have good intent, but they're not well equipped to handle it, so they do need to do a little help with tooling, and stuff like that to help them raise this stuff. Some highlights, for example, only 30% of maintenance we asked, say they have a high level of security knowledge. As we said before, only half of them just find out about issues when users are opening security issues for the repository. There's kind of a discrepancy there, between how users kind of rate themselves to what really happens.

If you think about it, when we asked maintainers, how do they audit their codebases to leverage information about vulnerabilities, to expose it. One in four maintainers don't even audit their codebases. That's truly a high number. Just go through your Packer JSON or whatever project, count one out of four dependencies. That probably didn't go through a security review, through a security audit, code review best practices, and stuff like that. That's the kind of virtually not even there. It's a bit of a terrifying stats to have, I think, especially in terms of the tooling that we have in the ecosystem. Specifically, for Snyk, that makes it really easy to find those vulnerabilities. We'd want to see more people just leverage the kind of developer-friendly tooling that we have to raise those.

[0:08:58] Guy Podjarny: Yes, it's a scary stat. I mean, somehow, not entirely unexpected. It's great to see maintainers care about it. But we have such high expectations from open-source maintainers when you build almost single-handedly, or like a very small group, you build a piece of software that gets downloaded millions and millions of times a month. Then, the bar is kind of high, your impact, you're doing a lot of good, but that also means that the responsibility on your shoulders is pretty heavy. It's good to sort of see the intent there, but it's indeed a little unsettling to sort of hear about security practices not necessarily being there.

[0:09:33] Liran Tal: Yes, it's good you mentioned like, we're going back to the intent part. Because, I think the positive right now, the optimistic stats to take out of it is when we asked developers who do they think should own security for their application, 81% of them said developers should own that. I think that's, that's a really good trend. That's really high number in terms of the statistics, and showing that I feel that full stack development, at least that term, that terminology kind of brought to it itself a lot of responsibility into developers. Meaning, they don't just focus on the frontend. Now, we have the full stack at the frontend and backend.

The DevOps thing is kind of empowering developers to own their infrastructure. By product, a lot of those obstructions on software, like owning a Docker file, but not the actual infrastructure allows developers to take more ownership. As they take more ownership around security, that's great. That's kind of the good security posture that you want developers to have when they build applications. Just have that mindset.

[0:10:40] Guy Podjarny: Yes, indeed. Well, I guess, you have to start there. Otherwise, nothing will happen. That's kind of insight one, on the maintainer side, sounds like, maintainers are well-intentioned. They do accept that them as maintainer and as developers should own this responsibility. They potentially slightly overrate their sort of security skills, but still, like a bunch of them, only about three in 10, if I understood correctly, rated as high. Hopefully, over time, we see improvement there. We did have Marten Mickos from HackerOne on the show, talking about the internet bug bounty, which I guess, maybe they can use those.

I believe in last year's report, there were a bunch of these examples and recommendations, probably repeated this year around having a disclosure policy, and the likes to check out the report for concrete advice as a maintainer on what you can do.

[0:11:26] Liran Tal: Maintainers and developers.

[0:11:27] Guy Podjarny: Yes, maintainers and developers as a whole. Indeed, let's kind of shift a little bit. This is a highlight of insight on the maintainer side. What about the consumer side? What have we learned? What did you find out on the consumer side?

[0:11:39] Liran Tal: I think as developers, we try to get insights into, once developers develop their apps, they have a lot of CI infrastructure to run tests, run accessibility, stuff like that. An important part of that is being able to address security as well at CI, shifting security to the left, or kind of trying to raise vulnerabilities much sooner than finding out about them in production, or, God forbid, as a data breach.

One of the questions was, whether security testing is being used in CI. Thirty-seven percent of developers are not even at all using any kind of security testing during their CI. We also asked about Docker images scanning, for example, out of which, only 14% of developers said they actually do Docker scanning.

[0:12:28] Guy Podjarny: Fourteen percent?

[0:12:29] Liran Tal: Fourteen percent. It's a fairly low amount. I think it correlates a little bit with the ownership that developers have around DevOps. As in, they haven't truly embraced the idea of owning the DevOps pipeline, Docker file, managing that. Even if they have good intentions of owning that, there isn't yet enough credibility around what that entails, in terms of, security risks and stuff like that.

Other interesting information is, as we have asked them about, how do they find out about vulnerabilities in general in their CI. Not just about scanning is, we learned that 36% of users use a dependency management tool to handle scanning, which is a fairly good amount of that. Yet, 27% of them have no automatic or proactive way of raising the information around vulnerabilities. Basically, there's still a large amount of users that even though tooling exists to help them find out about vulnerabilities and CI, they're not naturally using any kind of that in their CIs.

[0:13:34] Guy Podjarny: Yes. Interesting. Basically, like only about one in four. I guess, if a new struts vulnerabilities sort of pick on the infamous struts vulnerability or anyone of those or – I guess, there was actually recently a jQuery, one that came out. Only about one in four developers would automatically be notified about one of those. I guess, a bunch of others would either not know about it or would have to do some proactive exploration to find that out. What about the data a bit? I know one of the aspects in the reports talked about, when you consume open source, most of the vulnerabilities came not through the libraries you consumed, but rather indirectly. Do you want to tell us a bit more about that?

[0:14:10] Liran Tal: Yes, that's a really interesting insight that we were able to figure out. That is, as you develop, there are dependencies, and libraries that you directly consume, directly using your app. But to the nature of open source, those libraries, maybe you're building on other libraries that other people have written. Out of that, we were able to conclude a lot of data that Snyk has on users, scanning their project. We're seeing an interesting thread, where most of the vulnerabilities are not coming from the libraries that you're explicitly using, but those indirect dependencies that they're bringing with them.

For NPM, for example, I think that percentage is around 75%, 78%. Meaning, as Snyk detect vulnerabilities for npm projects, 78% of the time it's going to find nested dependencies and indirect dependencies. That you didn't explicitly bring into your project, but they got brought in by other dependencies through the dependency chain. That's kind of a worry around managing dependencies in general.

[0:15:16] Guy Podjarny: Yes, you have to dig deep. That is indeed – I guess we talked about the high bar for maintainers. But we kind of expect us developers to sort of do a fair bit as well. Not only do you need to care about the dozen or so libraries that you put in your package, JSON file, or your pom.xml. Actually, those vulnerabilities in those would only account for, again, every fourth vulnerability that might be disclosed. Really, three out of those four would be in libraries that you never really ever asked for. This library –

[0:15:43] Liran Tal: Are just coming in.

[0:15:43] Guy Podjarny: Yes, those libraries in turn pulled in. A little bit biased here, but I'd sort of say a little bit of the realm of the tools. On the consumption side, we have this sort of insight from the data side, or on vulnerabilities coming more from indirect. We have – it seems like a not a terrible, but highly exciting amount of CI security testing, a little bit more daunting in the darker security side. Maybe indeed, let's kind of use this as a bit of an opportunity to shift to the Docker security report. Before we move off, if somebody wants to read the full set of data for the ‘State of Open-Source Security’, where should they go?

[0:16:20] Liran Tal: So it's on snyk.io/stateofossecurity?

Guy Podjarny: Okay, cool. Let's switch to this Docker-specific insight. You took this data, you collected, or you took the survey data, you took some additional analysis of data, and you released this state of Docker security report, or really container security, I imagine. Tell us what were the key learnings from there. You already alluded to a couple from the broader report, but what else have you learned?

[0:16:45] Liran Tal: Vulnerabilities-wise, we just scan through 10 of the top most popular images on Docker Hub, which is a fairly popular kind of central place to find your Docker images. We went through that, just going by the amount of popularity of images, took the top 10, scanned all of them, their default tags for that, which is what people intend to use. We found out that for each of them, every one of those top 10, they include at least 30 vulnerabilities inside them when you use that.

[0:17:19] Guy Podjarny: Which is to sort of echo that back. Every single one of the top 10 had at least 30 vulnerabilities on it. How come? These are the top 10 most popular images on Docker hub, why would they not have zero vulnerabilities on them?

[0:17:40] Liran Tal: Certainly, good question. Choosing a base image is a crucial part of building your Docker containers on. As we're learning, simply Docker images built themselves. They have a lot of vulnerabilities inside them. These could be general-purpose images, for example, the Node image has a little bit more than 30 vulnerabilities. There's something like 580, just a bit more. It hurts me as I'm a Node guy.

[0:18:13] Guy Podjarny: Well, if I'm not mistaken, also the Node security working group.

[0:18:16] Liran Tal: True. We're trying to get that, of course, solved. But the rule starts, or that, as you scan the default no tag, it's bringing 580 or something like that. We'll monitor vulnerabilities inside them. That's true, the idea that the base image, the kind of the default layer that the image is built upon, brings with it a lot of libraries, and tools, compilation headers, tools that you may not need. That's kind of leaking back, or dropping into your base image, for whatever app you're building. Because, as kind of a default thing for you to do is, build your app, and start the Docker file “from node:10” or something like that. This is where I think visibility is coming into play. You're not really understanding, or people may not be truly aware of what they're bringing in when they're doing something like a “from node:10” and they're bringing in a default image that has over 500 vulnerabilities.

[0:19:15] Guy Podjarny: It's interesting. You're kind of describing a bit of a Russian doll of these base image elements. On one hand, the fact that these top images have vulnerabilities is something you as a consumer of those base images should know. Like you're probably pulling down a whole bunch of vulnerabilities, you should know what's there and address it. But you're also saying that those base images themselves, one of their key reasons for being vulnerable is because they in turn use other base images that also have vulnerabilities. Is that how we got to that sort of 580 number in Node world?

[0:19:47] Liran Tal: Indeed. There's a technical part of that. The Node one is built on a popular image that's called build pack depths. That's basically an image that is intended to build other base images upon, so it's kind of something that gives you a collection of cross-compilation headers. You can go and do npm install, or pip install, or gem bundle, whatever. You can do all of that by default. The idea is to kind of give you an out-of-the-box, good experience when you build a base image. All of these tools kind of work. Similarly, you don't need to install anything else. As it turns out, the default Node image, at lease is based on distro, which is Debian-based, and brings with it all of this goodness of tools. But at the same time, a lot of vulnerabilities that are pulled into them as well. There are better choices in building base images that we could make.

[0:20:41] Guy Podjarny: What are those for instance? The kind of two questions come to mind. One is, what do I do? I still want to use Docker; I still want to use Node. I felt like Node, using the Node-based image sort of appears to be a fairly reasonable decision. How do I kind of help myself out here a little bit? Then, maybe like a related question is, are there examples of base images that do it well. That maybe, dare I say, even get it down to zero.

[0:21:08] Liran Tal: Right. Some considerations that you could do is, try to see which tag brings with it lesser vulnerabilities. Meaning, there's node:10, but then, there's a collection of tags that exist in this Docker repository, so to say. That you can pick and choose, kind of like cherry-picking your own specific flavour of the OS that Node is running on. For example, if you were to choose node:10-jessie, you would get these amount of vulnerabilities. But if you would choose node|:10-sid, or maybe even alpine, you would get lesser vulnerabilities simply because those base images are built differently in terms of the dependencies and the libraries that exists in them.

[0:21:49] Guy Podjarny: Okay. Cool. Step one is just choose the variant of your base image that suits you best.

[0:21:55] Liran Tal: Choose wisely.

[0:21:56] Guy Podjarny: Choose wisely, indeed. How do I find out how many vulnerabilities in each of those images?

[0:22:02] Liran Tal: You have it in the report, and you can use the tool to scan Docker images. Snyk is a way to scan your Docker images, allows you to get this visibility into the amount of vulnerabilities you have across the image. Even give you an idea of which other very similar base images that you can choose based on what you have, but like just different tags, that would allow you to just choose, and use a different one that has lower vulnerability discount. It could be a smaller change for you to make, or major one, depending on your preference of both mediating the risk, as well as making sure that your application actually doesn't break, and runs well.

[0:22:37] Guy Podjarny: Cool. I think Docker Hub kind of gives you some stats about vulnerabilities as well.

[0:22:42] Liran Tal: It does.

[0:22:43] Guy Podjarny: That's when you sort of scan. Snyk will sort of show you the specific, hey, you're using whatever Node, you can switch to Node Alpine, that would reduce your vulnerabilities to so and so. But then, also Docker Hub itself would at least show you. If you browse around, it's kind of labour manual. But if you browse around, you can also sort of see some of this data on the hub itself.

[0:22:59] Liran Tal: Yes, they do expose some of that as well. I'm happy you're bringing this up, because there is, however, an interesting aspect of that. That is containers, long-living containers and stuff like that. Let's talk about it a bit because it kind of relates to the visibility of data. If you go and scan your Docker image, see that it has maybe zero vulnerabilities, maybe a low amount of them, and you're good without, you're shipping that out. That's just one aspect of mitigating the risk. The thing is if you're just using that Docker image in production, you deploy it, it's running there for a week or two. During that time, and new vulnerability surfaces that affect tools, or libraries in that base image, you may not have the visibility into that.

Kind of the key takeaways around Docker images is, if you can, rebuild that image often. Meaning, monitor it. See that there aren't any new vulnerabilities surfacing on that specific image tag that you're using. As you do that continuously, you're going to release new containers into your production environment that have less and less vulnerabilities. So, we're not only releasing containers due to tests and fixes, or features that we develop, but also due to the idea of lowering the attack surface due to vulnerabilities that we want to mitigate against. One key takeaway is definitely, scan your Docker images that you have in production in development throughout the lifecycle, and then be able to deploy a new version that you can rebuild the image upon.

[0:24:30] Guy Podjarny: Okay. This is – kind of quoting some stats from the report, correct me if I'm wrong here. One is, we talked about sort of choosing the stats from the report, talk about the vulnerable images, kind of select wisely, using the tools at your disposal. Two is, rebuilding often, which would get rid of – how often was it? That if a vulnerability was found, it can be fixed by a rebuild?

[0:24:52] Liran Tal: Like, 20% of the vulnerable Docker images can be fixed by a rebuild. That's what we found. You can just mitigate that by rebuilding the image.

[0:25:00] Guy Podjarny: It's amazing. It's probably a topic for a whole conversation on its own. It actually is a topic for many, many outside security about the non-determinism of Docker builds. But sounds like a pretty high ROI that if you just rebuild often, you can fix 20% of the vulnerabilities that you encounter. Okay, cool. That's sort of a second takeaway. I guess, kind of a third was, to scan in the first place.

[0:25:22] Liran Tal: Right. Obviously, try and scan in the first place. I think we're seeing kind of a positive trend from the survey around Docker images as well, as we've seen around developers embracing security. Just as a stat out around that, 60% of developers believe that they should be responsible for container security. So that, good, positive trend all around. Yet, 50% of those don't scan their OS layers for vulnerabilities. So kind of the same conflict in terms of, there is a good intention, but not enough action on it to mitigate the security problems.

[0:25:56] Guy Podjarny: Yeah. I mean, to me, all in all, that's the way change happens. You start from appreciating that it needs to happen. Then, you go from there. I guess, you can take the glass half full, or glass half empty a little bit here. But, cool. To read up the full Docker Security Report, check it out, also on the Snyk website. Before I let you go, I like to ask every guest that comes on the show, if they have one sort of security pet peeve or security tip, a bit of security advice for a team looking to level up their security, what would that be?

[0:26:29] Liran Tal: I think mine would be, try to leverage, coach someone from the team to be a security champion, and really empower them to take actions on that. I found that previous experience leading teams, I found that very helpful and impactful to have someone from the R&D team, owning security and loving it. And, trying to help the rest of developers through doing a lot of AppSec work through the teams.

[0:26:55] Guy Podjarny: Cool. Yes. That's, I think security champions is a really, really good, good theme in a way that it's been mentioned many times on the show. Liran, thanks for coming on the show.

[0:27:04] Liran Tal: Thank you for having me.

[0:27:05] Guy Podjarny: Thanks, everybody, for tuning in and join us for the next one.

[OUTRO]

[0:27:11] Announcer: That's all we have time for today. If you'd like to come on as a guest on this show, or get involved in this community, find us at thesecuredeveloper.com, or on Twitter, @thesecuredev. Visit heavybit.com to find additional episodes, full transcriptions, and other great podcasts. See you next time.