Critical Arbitrary Code Execution Vulnerability Found in Kubernetes

On December 3rd 2018, a severe vulnerability was disclosed to the kubernetes community, which marks the first critical CVE found on the kubernetes project (based on a CVSS v3 score).

Patched versions were released and made available for end users and cloud providers. Make sure you upgrade to a fixed version, if you haven’t done so already. We suggest the following patched releases: v1.10.11, v1.11.5, v1.12.3, and v1.13.0-rc.1. If an upgrade is not possible, other workarounds and mitigation options have been suggested in the aforementioned GitHub issue by Jordan Liggitt, from the kubernetes project.

The critical vulnerability, identified as CVE-2018-1002105, was discovered by Darren Shepherd of Rancher. It allows an attacker to gain remote access to backend services that reside in the kubernetes cluster and execute arbitrary commands on them, potentially resulting in privilege escalation for the attacker.

The attack is made possible due to a vulnerability that exists in the kubelet API service, which is one of many components that make up the kubernetes project.

The kubelet API service is a kubernetes provided interim gateway that allows backend services, also known as aggregated API servers, to register themselves to the gateway. Therefore, any requests to these servers, such as API calls to /apis/<apiGroup>/<apiVersion>, would be proxied through the kubelet API service and on to the aggregated API destination inside the kubernetes cluster.

An example use-case of an aggregated API server is a health check service for load balancers that query internal backend servers for health status. Another example would be the built-in metrics service for kubernetes.

The vulnerability

To exploit the vulnerability, a user must have access to the kubelet API service, identified as kube-apiserver. This opens the way to escalation of privileges and gain further access control.

The vulnerability exists due to the way in which the kube-apiserver proxies requests to the internal backend servers within the cluster. This opens up a direct tunnel between the backend servers and the user. More specifically, this occurs during the handling of requests for connection upgrades, which are relevant for websockets communication. The resulted open TCP connection between the client and the backend server could then be utilized for arbitrary requests directly on that server.

Normally, this kubelet API service will be blocked by anyone outside of the cluster. Except that by default, access to this API is enabled for anonymous users to allow for service discovery and health checks.

Another attack vector exists whereby a user accessible service can interact with the kubelet API service in a way that allows a remote attacker to take control over the interaction, and thus opening a path for manipulating the connection to the kubelet and exploiting it in a similar manner.

Interestingly enough, this is not the first kube-apiserver related incident. An analysis published in March revealed how a publicly exposed kube-apiserver led to a cryptocurrency mining malware in a kubernetes cluster. Tesla’s own cloud infrastructure was also a victim of a similar cryptocurrency attack due to an insecure kubernetes administration console.That said, this recent vulnerability is the most severe vulnerability disclosed to the Kubernetes Product Security Team to date.

At Snyk, our security team added the vulnerability to our database on 5th December 2018.

Key takeaways:

• Insecure defaults - both authenticated and unauthenticated users are allowed to query the kubernetes API

• Insufficient logging - malicious or suspicious activities were made over an existing API connection. This meant kubernetes did not capture the actual activity in their logs,except for the initial API call.

How to protect yourself

Back in June we announced our developer friendly docker image scanning solution that you can use to detect if any of your images contain any of the vulnerabilities mentioned in this post and many more. Furthermore, Snyk will also provide remediation advice, suggesting which base images you could upgrade to that contain fewer vulnerabilities and to mitigate this specific vulnerability.

If you haven’t yet tried it out, you’re welcome to test your projects that containDockerfiles and we’ll start scanning and monitoring them for vulnerable packages that are installed on the Operating System base image.