From image security to workload security
October 31, 2019
0 mins readThis is the third part of a four part series about building your Kubernetes AppSec strategy. Find part I here and part II here.
In one of our previous posts, we discussed how packaging of applications is shifting to developers as organizations embrace containers. But it’s not just packaging that’s moving from systems administration to development, it’s configuration management as well.
Kubernetes and the challenge of configuration
The Kubernetes API is a powerful abstraction for building cloud native systems. But an unintended consequence of the rich API has been developers authoring by hand large amounts of configuration, mainly in YAML. For example, take this description of a Kubernetes Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
These configuration files are often stored in source control systems, sometimes alongside the application code and other times separately. There are more than 1.5 million Kubernetes configuration files public on GitHub alone.
Insecure by default
Unfortunately, the large surface area for configuration with Kubernetes opens up a number of potential security issues. The presentation The Path Less Travelled from Ian Coldwater and Duffie Cooley does a good job of summarizing some of the problems with the Kubernetes defaults, from a security perspective. Some common configuration properties that often go unconfigured include:
CPU and Memory limits | Limiting the expected CPU and Memory limits has operational as well as security benefits. In the context of security, this is about limiting the impact potential denial of service attacks have in affecting the app, rather than the node, and potentially the whole cluster. |
runAsNonRoot | By default, containers run as the root user. This property prevents that at the container runtime, meaning, if an attacker was able to execute a command in the context of the container, they would only have limited permissions. |
readOnlyRootFilesystem | By default, the file system mounted for the container is writable. That means, an attacker who compromises the container can also write to disk, which makes certain attacks easier. If your containers are stateless, then you don’t need a writable filesystem. |
Capabilities | Linux capabilities control, at a low-level, what processes do in the container—from writing to disk, to communicating over the network. Dropping all capabilities and adding in those that are required is possible, but requires understanding the list of capabilities. |
These configuration properties aren’t vulnerabilities themselves but they generally make it easier for an attacker to exploit a vulnerability in an image. If an exploit occurs the configuration properties can possibly amplify the damage. Your level of risk exposure isn’t just the vulnerabilities you have, but the context in which they are present.
Configuration throughout the SDLC
As we shift more security responsibilities to developers, similar to image vulnerabilities, it’s interesting to look at this secure configuration challenge through the lens of the SDLC.
Stage | Description | Feedback | Completeness |
Local | Local tools that help author security conscious configuration, from integrating with unit test workflows to prompts in IDEs. But solves individual rather than team or organizational problems. | Fast | Low |
CI/CD | Quickly fail the build for potentially insecure configuration files or those that don’t meet some internal policy. However, it needs implementing in all relevant pipelines. | Fast | Variable |
Repository | Configuration is currently mainly stored in a source control system (although it’s worth noting Helm 3 adds support for storing images in compatible OCI registries). How do we help developers write secure configuration between submitting pull requests and creating branches? | Medium | Medium |
Admission | Kubernetes admission controllers block API requests and provide a means to gate against prohibited insecure configuration. Different clusters possibly have different policies, however, and this impacts new requests, not existing workloads. | Slow | High |
Production | The Kubernetes API represents the running configuration — here is where you should pay the most attention to configuration. But issues here can affect real workloads, and feedback cycles to address those issues may be slow. | Slow | High |
Similar to when looking at testing container images, we see different tradeoffs at different points. Testing only at one stage may provide fast feedback but less control, or vice versa. Similar to looking at image vulnerabilities, a modern and mature security process probably involves testing configuration at multiple stages of the process.
Conclusions
Our applications aren’t just the images we use to package them, but also the configuration we use to run them. At best we’ve seen these as two separate domains to secure. But, in most cases, conversations about container security for developers have focused exclusively on image vulnerabilities. As responsibility for application security shifts to development teams, it becomes increasingly important to understand the relationship between image vulnerabilities and configuration, and to have tools that help secure both.
Get started in capture the flag
Learn how to solve capture the flag challenges by watching our virtual 101 workshop on demand.