Uncharted territory - discovering vulnerabilities in public Helm Charts

Similar to our report on Docker image security, we wanted to take a look at the state of vulnerabilities in the public Helm Charts repository.

Helm is a popular package manager for Kubernetes. As well as being used by developers to package their own applications, the official Charts repository contains 100s of Charts you can use to install third party software like Jenkins, PostgreSQL and lots more. This saves time, but you also benefit from the expertise of the people packaging the software and making it easily configurable.

Let’s take a quick look at a sample Helm chart, in this case for Linkerd.

$ tree
.
├── Chart.yaml
├── README.md
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── config.yaml
│   ├── daemonset.yaml
│   ├── ingress.yaml
│   └── service.yaml
└── values.yaml

1 directory, 9 files.

In the values.yaml file we find details of a few container images, which may contain vulnerabilities:

image: buoyantio/linkerd:1.1.2
image: buoyantio/kubectl:v1.6.2

Image vulnerabilities in Helm charts

Like any repository of third-party content, vulnerabilities in popular Helm Charts can pose a risk to many users. Helping developers use third party content securely is what we do at Snyk. We already provide developer tools to help secure open source dependencies for popular package managers (for instance for Java, .NET, Python, Node,js, Ruby and more), as well as providing tools to detect vulnerabilities in container images.When looking at the challenges of securing third party content in Kubernetes, one of the questions we asked was “how do I know what vulnerabilities exist in the images from this Helm Chart?” This turns out to be a fairly manual process at the moment:

1. Determine which images the Chart is using, this probably involves reviewing the rendered templates and identifying the specific image versions

2. Scan those images individually, which probably involves running several commands and aggregating the results manually

This is further complicated by the fact that Helm Charts can take settings that may affect the images being used.

Announcing the Helm Chart Security report

In our new report we look at the state of vulnerabilities in the public Helm Charts repository. Our intention isn’t to call out Helm as being insecure any more than any popular third party content repository is insecure. Rather, our intent is to start a conversation about better ways of securing Helm Charts so even more people can benefit from Helm’s ease of use.We found that:

• 68% of stable Helm Charts contain an image with a high severity vulnerability

• 64% of the stable Helm Charts could reduce the number of vulnerabilities by updating to the latest published images

• 6 images (out of a total of 416) account for half of the instances of vulnerabilities

We’ve been sharing this work with the Helm community too. Matt Butcher, one of the Helm Org Maintainers said: 

“For the cloud native ecosystem to reach its full potential, we need to collectively improve security. And that means applying a set of patterns, techniques, and tools that focus specifically on cloud native technologies. The Helm Project is deeply interested in the security of our core software, our charts, and the practices of the community. And we eagerly welcome a new breed of tools that help us identify and fix vulnerabilities proactively.”

Introducing the Snyk Helm plugin

As part of putting together the research above, we created a plugin that makes checking your own Helm Charts for vulnerabilities as simple as everything else in Helm.

You can install the Snyk Helm plugin using the built-in commands in Helm.

helm plugin install https://github.com/snyk-labs/helm-snyk

To use the plugin, you’ll need a Snyk account, available for free at https://snyk.io/signup/. Once you have an account, you can test your charts like so.

helm snyk test <path-to-chart>

For instance, here’s an example testing the Redis Chart from the public Charts repository:

$ helm snyk test stable/redis

Image: docker.io/bitnami/redis:5.0.5-debian-9-r181

Testing docker.io/bitnami/redis:5.0.5-debian-9-r181...

✗ Low severity vulnerability found in tar
Description: CVE-2005-2541
Info: https://snyk.io/vuln/SNYK-LINUX-TAR-105079
Introduced through: meta-common-packages@meta
From: meta-common-packages@meta > tar@1.29b-1.1

✗ Low severity vulnerability found in systemd/libsystemd0
Description: CVE-2019-9619
Info: https://snyk.io/vuln/SNYK-LINUX-SYSTEMD-442642
Introduced through: systemd/libsystemd0@232-25+deb9u12, util-linux/bsdutils@1:2.29.2-1+deb9u1, procps@2:3.3.12-3+deb9u1, sysvinit/sysvinit-utils@2.88dsf-59.9, systemd/libudev1@232-25+deb9u12, util-linux/mount@2.29.2-1+deb9u1
From: systemd/libsystemd0@232-25+deb9u12
From: util-linux/bsdutils@1:2.29.2-1+deb9u1 > systemd/libsystemd0@232-25+deb9u12
From: procps@2:3.3.12-3+deb9u1 > procps/libprocps6@2:3.3.12-3+deb9u1 > systemd/libsystemd0@232-25+deb9u12

and 4 more...

Helm charts often have properties that can be set at installation time, and some of those may affect the images the chart will use. The Snyk Helm plugin uses the same interface as the rest of the Helm commands to allow you to set individual properties to test variations too, for example:

helm snyk test stable/mysql --set imageTag=latest

Conclusion

With the Helm Chart security report we’re helping the Helm community find vulnerabilities and making fixing them easier. The Snyk Helm plugin makes it easy to quickly test your Helm charts, whether you are investigating a third party Chart or running checks regularly as part of your CI/CD pipelines. Look out for additional tools in the future from Snyk to help developers use third party content more securely.