Uncharted territory – discovering vulnerabilities in public Helm Charts
Helm is a popular package manager for Kubernetes. As well as being used by developers to package their own applications, the official Charts repository contains 100s of Charts you can use to install third party software like Jenkins, PostgreSQL and lots more. This saves time, but you also benefit from the expertise of the people packaging the software and making it easily configurable.
Let’s take a quick look at a sample Helm chart, in this case for Linkerd.
$ tree . ├── Chart.yaml ├── README.md ├── templates │ ├── NOTES.txt │ ├── _helpers.tpl │ ├── config.yaml │ ├── daemonset.yaml │ ├── ingress.yaml │ └── service.yaml └── values.yaml
1 directory, 9 files
In the values.yaml file we find details of a few container images, which may contain vulnerabilities:
image: buoyantio/linkerd:1.1.2 image: buoyantio/kubectl:v1.6.2
Image vulnerabilities in Helm charts
Like any repository of third-party content, vulnerabilities in popular Helm Charts can pose a risk to many users. Helping developers use third party content securely is what we do at Snyk. We already provide developer tools to help secure open source dependencies for popular package managers (for instance for Java, .NET, Python, Node,js, Ruby and more), as well as providing tools to detect vulnerabilities in container images.
When looking at the challenges of securing third party content in Kubernetes, one of the questions we asked was “how do I know what vulnerabilities exist in the images from this Helm Chart?” This turns out to be a fairly manual process at the moment:
- Determine which images the Chart is using, this probably involves reviewing the rendered templates and identifying the specific image versions
- Scan those images individually, which probably involves running several commands and aggregating the results manually
This is further complicated by the fact that Helm Charts can take settings that may affect the images being used.
Announcing the Helm Chart Security report
In our new report we look at the state of vulnerabilities in the public Helm Charts repository. Our intention isn’t to call out Helm as being insecure any more than any popular third party content repository is insecure. Rather, our intent is to start a conversation about better ways of securing Helm Charts so even more people can benefit from Helm’s ease of use.
We found that:
- 68% of stable Helm Charts contain an image with a high severity vulnerability
- 64% of the stable Helm Charts could reduce the number of vulnerabilities by updating to the latest published images
- 6 images (out of a total of 416) account for half of the instances of vulnerabilities
We’ve been sharing this work with the Helm community too. Matt Butcher, one of the Helm Org Maintainers said:
“For the cloud native ecosystem to reach its full potential, we need to collectively improve security. And that means applying a set of patterns, techniques, and tools that focus specifically on cloud native technologies. The Helm Project is deeply interested in the security of our core software, our charts, and the practices of the community. And we eagerly welcome a new breed of tools that help us identify and fix vulnerabilities proactively.”
The full report contains detailed information and analysis. Please click here to read Uncharted Territories: The Untold Tale of Helm Chart Security.
Introducing the Snyk Helm plugin
As part of putting together the research above, we created a plugin that makes checking your own Helm Charts for vulnerabilities as simple as everything else in Helm.
You can install the Snyk Helm plugin using the built-in commands in Helm.
helm plugin install https://github.com/snyk-labs/helm-snyk
To use the plugin, you’ll need a Snyk account, available for free at https://snyk.io/signup/. Once you have an account, you can test your charts like so.
helm snyk test <path-to-chart>
For instance, here’s an example testing the Redis Chart from the public Charts repository:
$ helm snyk test stable/redis Image: docker.io/bitnami/redis:5.0.5-debian-9-r181 Testing docker.io/bitnami/redis:5.0.5-debian-9-r181... ✗ Low severity vulnerability found in tar Description: CVE-2005-2541 Info: https://snyk.io/vuln/SNYK-LINUX-TAR-105079 Introduced through: meta-common-packages@meta From: meta-common-packages@meta > email@example.com ✗ Low severity vulnerability found in systemd/libsystemd0 Description: CVE-2019-9619 Info: https://snyk.io/vuln/SNYK-LINUX-SYSTEMD-442642 Introduced through: systemd/libsystemd0@232-25+deb9u12, util-linux/bsdutils@1:2.29.2-1+deb9u1, procps@2:3.3.12-3+deb9u1, firstname.lastname@example.org, systemd/libudev1@232-25+deb9u12, email@example.com+deb9u1 From: systemd/libsystemd0@232-25+deb9u12 From: util-linux/bsdutils@1:2.29.2-1+deb9u1 > systemd/libsystemd0@232-25+deb9u12 From: procps@2:3.3.12-3+deb9u1 > procps/libprocps6@2:3.3.12-3+deb9u1 > systemd/libsystemd0@232-25+deb9u12 and 4 more...
Helm charts often have properties that can be set at installation time, and some of those may affect the images the chart will use. The Snyk Helm plugin uses the same interface as the rest of the Helm commands to allow you to set individual properties to test variations too, for example:
helm snyk test stable/mysql --set imageTag=latest
With the Helm Chart security report we’re helping the Helm community find vulnerabilities and making fixing them easier. The Snyk Helm plugin makes it easy to quickly test your Helm charts, whether you are investigating a third party Chart or running checks regularly as part of your CI/CD pipelines. Look out for additional tools in the future from Snyk to help developers use third party content more securely.