Everything you wanted to know about addressing security vulnerabilities in Linux-based containers

Think about the most important container image that you have running in production right now. How did you choose its base image? Do you know how many vulnerabilities that base image has? Wouldn’t you like to know?

Here at Snyk we try to make the process of choosing the most secure base image smarter, smoother and most importantly, more data-driven.

But first let’s start with some background about Linux distro security, since we focus on Linux-based images in this blog post.

Linux distros in a nutshell

A Linux distribution (distro) is an operating system (OS) built from a software collection, comprising a Linux kernel (the core of the operating system), a package management system, and GNU tools and libraries, additional software, as well as documentation—all bundled together and modified by its maintainers in order to maximize its capabilities to meet the needs of a variety of platforms and devices.

Distros are typically made available in two ways:

• as compiled binaries (packaged images), available for download from image repositories such as Docker Hub; images are typically packaged from the libraries where they are maintained as source code.

• as source code, enabling modifications to the original software. This enables great flexibility, especially since most of the software included in the distro is usually open source software.

Today, there are hundreds of Linux distros available, most of which are actively maintained. With great availability and flexibility in functionality, they are used across platforms and devices, including for containers.

Similarly to any other software, this means that containers are also susceptible to attacks by way of any vulnerabilities they may contain in their code.

How does security differ across different Linux distros and their releases?

There are differences in how individual distros are built and maintained. These differences can include the way the components are packaged together, the way they are managed, and more, and account for varying distro size, package managers, and susceptibility to security vulnerabilities.

A security vulnerability that appears in a component can have a severity score that changes across different distros. A component that is common across distros could be considered safe in one distro whereas it could receive a high CVE score as part of another distro—all due to the different way each of the distros is packaged.

Certain distros are also typically safer than others due to mechanisms their owners have baked into their maintenance. Commercial and well-established distros (Debian, Ubuntu, CentOS, Oracle) have dedicated security teams. These teams are responsible for checking CVE scores, prioritizing and implementing security fixes, and maintaining the general security health of the distro.

Even then, there are differences amongst all of the options. Take a look at the table where we’ve delineated the differences in the way these major distros are managed—a factor that also affects how secure they are:

*Alpine has a different approach to maintaining images; they regularly upgrade their packages, adding security fixes for a small subset of their major packages (curl, …).

How does a Linux distro differ from a language ecosystem?

Language ecosystems store packages in registries and package managers (such as npm and pip) are responsible for keeping these packages updated. Likewise, distros are stored in package repositories for re-use in different projects as dependencies, however they are packaged as images and then stored separately in package repositories designed for images from where they can be downloaded for immediate use in other projects; the images are stored in their own package registries after being packaged for download and immediate use, and from those registries they are updated by their package managers separately from the libraries from which they were sourced originally.

In either case, it is the package manager that is responsible for regularly downloading the package, updating it,and then uploading it back to its storage.

This is where the similarities end, however.

Maintaining packages in language ecosystems

For packages in language ecosystems, life is pretty simple: a package registry, at least one package manager and an exclusive and straightforward relationship between the two—the manager downloads and uploads the packages to and from the registry.

This can be visualized with the following diagram (a general example on the left representing any package repository, and an example on the right for npm specifically):

Maintaining packages in distro repositories

For distros in image repositories on the other hand, things are a bit more complicated. A distro can use multiple repositories, multiple package managers and multiple upload tools. The choice of which repository, which manager and which upload tool depends on the release number of the distro, the scope of the distro (main, contributions, non-free, community) and more.

This can be visualized with the following example (a general package repository for any distro with any version on the left and an example of multiple Debian releases on the right):

Each Debian release uses the repository that suits the specified release.

Packages and their versions are not always available from all of the different repositories. Packages can be backported on some distros (package versions only available for one release type can be installed for another) and some packages can be added to a distro version directly from an external repository such as from a personal package archive (PPA) for Ubuntu, rather than by the native package manager. This complicated relationship between packages and distros is in contrast to the simple relationship between package registries and package managers, which is much less sensitive to the versioning issues described.

The complexities in addressing container security issues

Most distro security teams maintain security fixes for only some of their components. For example, Debian only officially provides security fixes for their main components, as outlined in their documentation. This is due to the complicated process of triaging the different images over time.

As we indicated, images are packaged from their source code, typically stored separately as open source libraries. Once the image is packaged and then stored and maintained in its separate repository, the image continues to progress through its own versions separately from the source code library, which too continues to progress in its versions at its own pace. This means that while a vulnerability might exist in the library, it might not be in the image. The vulnerability may have appeared in an upstream version of the library, in which case it would never affect the already packaged image. The same could also be true vice versa.

When a security vulnerability is discovered in a library upon which an image was built, then it is the job of the Security teams to triage the image accordingly, verifying whether the vulnerability even exists, and whether it is considered a vulnerability based on the composition of the image itself.

Versions in most of the older, more established distros are not upgraded dramatically, and can be expected to get minor updates and security fixes for new vulnerabilities generally. 

Very old and end-of-life (EOL) distros as well as brand new distros (edge versions that may not yet be under the responsibility of the security team yet) are often considered unmaintained by the majority of the distros (Debian 7 for example is EOL). When a vulnerability affects version X of a package, which is available for only EOL releases or is brand new, it might not be available as part of the distro security advisories. Because of that, the security support level of the distro for EOL and bleeding edge releases are not necessarily reflective of the actual security health of the distro. In these cases, the information available is not sufficient for Snyk to detect all vulnerabilities and some critical vulnerabilities could be missed even.

Why isn’t my Linux distro supported?

Every distro-based image that we support is scanned against our vulnerability database according to its distro and release. Once we scan the image, we list the vulnerabilities with details about the vulnerability and its source, whether it’s fixed, which distros and releases are affected by it and from which version it is fixed.

In our vulnerability collection process, we do our best to keep up with every new release and have our roadmap prepared for it. We also track our users’ activities and needs very closely in order to enrich our database accordingly, and we’re also always open to chat with users through support@snyk.io and our Twitter account. Additionally, we make a special effort to support LTS releases by the providers we’ve reviewed in this post. Having said that, we don’t support every distro under the sun. If your image is based on a very old or very new distro for example, it’s likely that Snyk doesn’t support it.

How does Snyk know which secure base image alternatives to suggest?

We also regularly collect data for a wide range of repositories, pulling all tags per repo several times a day from Docker Hub in order to stay current. With this information, we can also provide you with base (or parent) image remediation advice, assuming you’ve linked your Snyk projects to the associated Dockerfile.

If you’re familiar with Snyk’s application scanning, you may be used to getting just a couple to a few dozen vulnerabilities per repository that you monitor or test. In the Linux ecosystem, however, we’re dealing with different numbers: a few hundred vulnerabilities is pretty average.

This can be overwhelming and that’s why we try to give you more actionable advice and insights about the image and the vulnerabilities. To help us with that, we ask you to provide the Dockerfile, which is often part of a container project.

When using Snyk to scan your images, you can add a link to your source code where you’ve stored the Dockerfile. Once we scan the repo, we can then provide remediation advice based on the results. You can read more about how to link to your Dockerfile in our docs.

Once we have the Dockerfile we detect the base image, according to which we know how many vulnerabilities originated in the base image and how many from other layers introduced through the Dockerfile. That’s useful for both deciding whether to amend the other layers introduced in the Dockerfile and for deciding whether to swap the base image with a different more secure image.

To suggest relevant upgrades, we pull and scan dozens of repos and hundreds of tags continuously. Then we query our database of images against our security vulnerability database to find the most secure image available that contains up-to-date results. We then suggest tags of the repo for your base image, including suggestions for:

• a minor upgrade—as it sounds, a tag of a minor upgrade from your current tag. Sometimes by just a minor upgrade you can eliminate dozens and even hundreds of vulnerabilities, so why not check it out at least?

• a major upgrade option—upgrade by jumping to the next major release

• an alternative solution—we recommend other unrelated images that could be suitable for your container

We consider an alternative tag to be an upgrade candidate according to the following conditions:

• It has a smaller number of high severity vulnerabilities or the same high severity but with fewer medium severity vulnerabilities, and the same number or fewer low severity vulnerabilities.

• The OS distro of the tag for the candidate image is neither too old, bleeding edge, non-LTS nor unstable.

• Since Alpine distros security advisory are only available for a small subset of packages which were patched, we considered Alpine based images lacking in security visibility, which is why we don’t recommend Alpine-based tags.

Alternative upgrades may use tags from different distros and are therefore usually a much riskier change; however, we want to provide you with more information about as many options as possible to secure your image.

Conclusions

Containers are different than libraries in ways that make security and triaging a more challenging practice without question. At the same time, containers are everywhere, given their obvious and important benefits. This doesn’t mean you have to release dangerous containers however! With a little bit of planning and the implementation of good security tools, such as Snyk, you can ease the process of securing your containers considerably.

Either way, it’s important you triage your containers regularly to ensure they do stay as safe as possible. When you discover a vulnerability and need to fix it by replacing your base image, consider these points in order to keep you container safe without breaking it:

• Check if you have any dependency on an OS package, and if you do, make sure the upgrade does not break it.

• Consider the size of the image.

• If you consider using an alternative upgrade, you should also check any specific distro commands or features that you’re using and update them accordingly as well.

Try scanning your images with Snyk today.