Tips for hardening your container image security strategy

In the first part of this blog series, we looked at security best practices for the base images which you might be using. But what happens to container image security when we add other things to it? Perhaps we’re installing additional software from upstream, and we’ve got custom applications of our own which might have their own dependencies also being installed. These things that we’ve added are under our control, and so we need to take responsibility for fixing vulnerabilities we have introduced.

Create a baseline

It’s a good idea when building your container images to scan just your base image before adding any modifications. In this way, you establish a container image security baseline and can easily see what vulnerabilities were in your base image, and which you have added in subsequent layers. If you’re building images with multiple additional layers — perhaps a custom middleware layer as well as your application — it can also be good practice to baseline at each of these points and maintain those images separately, giving you a clear indication where vulnerabilities are coming from.

Establish priorities

It’s probably not realistic (except for very simple applications) to expect complete container image security, meaning no vulnerabilities at all. Since we don’t have endless resources available to spend on fixing things, we need to prioritize and make judgement calls on which ones to fix and which ones we might just accept.

However, it can be very difficult to make those decisions if you are dealing with images which your scanner reports have a large number of vulnerabilities. There is a danger with security scanning that the amount of information that scanning provides just overwhelms the viewer, leading to ignoring or disabling scanning altogether.

And vulnerabilities themselves are not a zero-sum game. A particular vulnerability may only be an issue under very specific circumstances, or on a specific architecture or platform. Without reading the details of every vulnerability, how can we possibly decide what is an issue in our environment or not?

Prioritization is not an exact science and can be based on a number of different factors. Severity alone does not give us very much information other than potential impact. The CVSS score, which takes into account things like exploitability and impact, does give us more context - but we can also use information like the maturity of available exploit code and most importantly if a fix is available. High severity vulnerabilities which have an exploit and an available fix are probably a no-brainer to fix first. Snyk’s prioritisation scores take all of these factors into account when presenting vulnerabilities so that developers have clear information available to base decisions on.

Decide on a container image security strategy

Security is almost always a series of trade-offs, particularly between effort and risk. How much effort is involved to fix something versus the risk of it being an issue in my particular environment? When considering how to prioritise fixing security vulnerabilities, defining a strategy is usually the first port of call. A very simple example of this might be:

• No high CVE’s in production

• Nothing with a mature exploit

• Apply patches if there are any

If you followed this, it would likely drastically reduce your overall vulnerability count in most cases.

Risk mitigation: Every environment is different

Whilst scoring gives you an idea of seriousness, there are obviously elements which are subjective. For example, you may decide that a high severity vulnerability which requires local shell access is lower risk in your particular environment because you have other controls in place to protect against that access. You might use distroless containers which do not include a shell and these controls would mitigate the risk.

Understand your security boundaries

For some environments, you may consider that the network is trusted therefore anything vulnerable facing into that network is less of an issue. This is a problematic scenario in most modern computing environments since connectivity is so complex that it’s very difficult to draw the line except in totally air-gapped environments. This becomes especially problematic in cloud environments. This also fails to protect you against internal bad actors, which can be a significant risk, particularly in bigger organisations. A safer position is usually to consider all of your networks untrusted.

Know your tools

This is where understanding the tools we use to detect these things comes into play. Most image scanning tools will allow you to filter the output of the tool itself, so that you can configure what information you are most interested in. This is particularly important when building security scanning into your software delivery pipelines. You really don’t want to continuously be failing builds or deployments based on information that’s not relevant, or where you have decided to mitigate risk in other ways. Snyk provides the --fail-on flag to control the output status of the CLI scan, so that the CLI only returns failure under certain circumstances. For example:

1snyk container test purpledobie/mattj-goof --fail-on upgradable

This will only exit with a failure if it finds at least one upgradable vulnerability in the image, but not if it finds vulnerabilities which are unfixable or patchable.

The Snyk CLI also provides output in JSON for post filtering. Using the JSON output, it’s possible to construct complex filters by using tools like jq:

1snyk container test purpledobie/mattj-goof --json | jq '.vulnerabilities[] | select(.CVSSv3 | test("AV:N"))'

This very simple example will only return vulnerabilities which are exploitable from the network, by filtering the vulnerability list based on the Attack Vector in the CVSS score being set to Network. By building filters like this, you can implement complex, policy-based decision making into your scanning.

Automate remediation

Once you’ve established your strategy and configured your tools, enabling automation in remediation can reduce the cognitive load on developers (by requiring fewer decisions to be made), as well as reducing the resource required. Automated pull requests for issues which fall clearly into the overall strategy and have fixes available make it much easier for development teams to continuously remediate, whilst reserving the manual resource for more complex edge cases.

Container image security in depth

Container scanning may not pick up things like binaries outside packages added during the build process, and so container image scanning should not be your only protection. This is why scanning your codebase and Dockerfiles is important as well. For production workloads, you almost certainly want to be adopting the principle of security in depth and include other security checks as well, such as runtime anomaly detection and endpoint scanning.

Keep your containers secure

There are quite a few considerations when it comes to dealing with container image security, but hopefully, this post has helped clarify some starting points. To see how secure your current container images are, set up a free Snyk account and run a scan.