Four steps for hardening Amazon EKS security
In the first part of this blog series, we explored deploying Amazon EKS with Terraform, and looked at how to secure the initial RBAC implementation along with securing the Instance Metadata Service. In this second post, we’ll look at more best practices to harden Amazon EKS security, including the importance of dedicated continuous delivery IAM roles, multi-account architecture for Amazon EKS cluster isolation, and how to encrypt your secrets in the control plane. And finally, we will show how to incorporate static analysis tooling in your CD pipelines to catch these issues before they reach production environments.
Keep track of who deployed the cluster
The Amazon EKS service integrates with the Identity Access Management service to authenticate users to the cluster. The authorization decisions are performed by the native Kubernetes RBAC model. RBAC configuration is mapped to the IAM identities via aws-auth ConfigMap. However there is a special mapping that is not visible in any configuration file or setting. The IAM entity used to create the cluster is automatically mapped to the system:masters built-in Kubernetes group. This is well documented in the user documentation. However, it still has significant implications on auditability of the cluster permissions.
In order to showcase the impact of this configuration we will delete all Roles and ClusterRoles from our demo cluster.
dev@pwnbox:~$ kubectl delete clusterroles --all
clusterrole.rbac.authorization.k8s.io "admin" deleted
clusterrole.rbac.authorization.k8s.io "aws-node" deleted
clusterrole.rbac.authorization.k8s.io "cluster-admin" deleted
clusterrole.rbac.authorization.k8s.io "edit" deleted
clusterrole.rbac.authorization.k8s.io "eks:addon-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:fargate-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:node-bootstrapper" deleted
clusterrole.rbac.authorization.k8s.io "eks:node-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:podsecuritypolicy:privileged" deleted
<OMITTED>
dev@pwnbox:~$ kubectl delete roles --all-namespaces --all
role.rbac.authorization.k8s.io "system:controller:bootstrap-signer" deleted
role.rbac.authorization.k8s.io "eks:addon-manager" deleted
role.rbac.authorization.k8s.io "eks:certificate-controller" deleted
role.rbac.authorization.k8s.io "eks:fargate-manager" deleted
role.rbac.authorization.k8s.io "eks:node-manager" deleted
<OMITTED>
dev@pwnbox:~$ kubectl auth can-i --list
Resources Non-Resource URLs Resource Names Verbs
*.* [] [] [*]
[*] [] [*]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]
dev@pwnbox:~$ kubectl get configmap --all-namespaces
NAMESPACE NAME DATA AGE
kube-system coredns 1 7h59m
kube-system cp-vpc-resource-controller 0 7h58m
kube-system eks-certificates-controller 0 7h59m
kube-system extension-apiserver-authentication 6 7h59m
kube-system kube-proxy 1 7h59m
kube-system kube-proxy-config 1 7h59mAmazon EKS uses webhook token authentication in order to integrate with IAM. We can see the configuration in the api server arguments.
FLAG: --authentication-token-webhook-cache-ttl="7m0s"
FLAG: --authentication-token-webhook-config-file="/etc/kubernetes/authenticator/apiserver-webhook-kubeconfig.yaml"
FLAG: --authentication-token-webhook-version="v1beta1"AWS IAM has a concept of unique identifiers which can be used in specific services to avoid name reuse misconfiguration. It is unclear from the Amazon EKS documentation whether the system:masters permission is bound to the unique identifier of the entity or its friendly name. In order to validate if removing the original entity will revoke all permissions from the cluster, we have deleted the temporary role and recreated it.
dev@pwnbox:~$ aws iam get-role --role-name ci-eks-test
{
"Role": {
"Path": "/",
"RoleName": "ci-eks-test",
"RoleId": "AROAYREY3WYOOSHLPO3W6",
"Arn": "arn:aws:iam::123456789012:role/ci-eks-test",
"CreateDate": "2021-06-20T21:09:01+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/ci"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
},
"MaxSessionDuration": 3600,
"RoleLastUsed": {}
}
}
dev@pwnbox:~$ kubectl auth can-i --list
Resources Non-Resource URLs Resource Names Verbs
*.* [] [] [*]
[*] [] [*]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]
podsecuritypolicies.policy [] [eks.privileged] [use]
dev@pwnbox:~$ aws iam get-role --role-name ci-eks-test
{
"Role": {
"Path": "/",
"RoleName": "ci-eks-test",
"RoleId": "AROAYREY3WYOAK6QMGSUT",
"Arn": "arn:aws:iam::123456789012:role/ci-eks-test",
"CreateDate": "2021-06-20T21:14:28+00:00",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:user/ci"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
},
"MaxSessionDuration": 3600,
"RoleLastUsed": {}
}
}
dev@pwnbox:~$ kubectl auth can-i --list
Resources Non-Resource URLs Resource Names Verbs
*.* [] [] [*]
[*] [] [*]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]
podsecuritypolicies.policy [] [eks.privileged] [use]As you can see we have a completely new role as identified by different RoleId. However, the Amazon EKS authentication service uses the friendly name of the role to grant us the same system:masters permissions.
The most effective way to mitigate this issue is to use a centralized deployment system, use a dedicated role for deployments, and avoid use of personal credentials at all cost. Tools such as aws-vault can help you securely manage switching between different roles.
Next let’s have a look at how nodes authenticate to the control plane.
Use multi-account separation for clusters
Amazon EKS requires that all nodes have an instance profile attached with the following two policies: AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryReadOnly.
dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy --version-id v1
{
"PolicyVersion": {
"Document": {
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeRouteTables",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVolumes",
"ec2:DescribeVolumesModifications",
"ec2:DescribeVpcs",
"eks:DescribeCluster"
],
"Resource": "*",
"Effect": "Allow"
}
]
},
"VersionId": "v1",
"IsDefaultVersion": true,
"CreateDate": "2018-05-27T21:09:01+00:00"
}
}
dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly --version-id v3
{
"PolicyVersion": {
"Document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"
],
"Resource": "*"
}
]
},
"VersionId": "v3",
"IsDefaultVersion": true,
"CreateDate": "2019-12-10T20:56:32+00:00"
}
}The policies by default gain privileges to all resources within the AWS account. This means that a compromised pod on this cluster will be able to enumerate information about any resource, vpc, subnet, and image in the account.
The Amazon EKS node actually requires access to one more managed policy named AmazonEKS_CNI_Policy. This policy allows the node to allocate IP addresses and perform any networking-related functionality.
dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy --version-id v4
{
"PolicyVersion": {
"Document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AssignPrivateIpAddresses",
"ec2:AttachNetworkInterface",
"ec2:CreateNetworkInterface",
"ec2:DeleteNetworkInterface",
"ec2:DescribeInstances",
"ec2:DescribeTags",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInstanceTypes",
"ec2:DetachNetworkInterface",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:UnassignPrivateIpAddresses"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:CreateTags"
],
"Resource": [
"arn:aws:ec2:*:*:network-interface/*"
]
}
]
},
"VersionId": "v4",
"IsDefaultVersion": true,
"CreateDate": "2020-04-20T20:52:01+00:00"
}
}This policy allows the caller to modify the networking configuration of any instance in the account. Amazon actually recommends this policy be attached directly to the aws-node Kubernetes service account. By default, the Terraform module attaches this policy to the IAM profile. We can change that behaviour by setting the attach_worker_cni_policy attribute to false.
Encrypt your Secrets
One of the control plane components managed by the Amazon EKS service is the etcd key value store. It is used by the Kubernetes API server to store all objects and configurations. By default Kubernetes does not encrypt data stored in this key value store. AWS is responsible for security of the control plane components however we can yet again apply a defence in depth strategy to further enhance the security of our data.
AWS KMS service is one of the encryption providers that can provide envelope encryption for secrets objects stored in our cluster. By setting the `cluster_encryption_config` option we can specify a KMS key which will be used to encrypt intermediate data encryption keys, which in turn will be used to encrypt specific objects. This Amazon EKS blog post from AWS provides a great overview of the underlying process. From this point onward anyone with access to an etcd cluster will not be able to read our secrets without also having access to our KMS key. The whole process is fully transparent to the end user.
Detect issues in the deployment pipeline
Infrastructure as code allows us to declaratively describe the desired state of the Amazon EKS cluster. With that we have the ability to statically discover some of these issues before anything is deployed.
In Terraform we can generate a plan of configuration that will be deployed.
dev@pwnbox:$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
<= read (data resources)
Terraform will perform the following actions:
# data.aws_eks_cluster.cluster will be read during apply
# (config refers to values not yet known)
<= data "aws_eks_cluster" "cluster" {
+ arn = (known after apply)
+ certificate_authority = (known after apply)
+ created_at = (known after apply)
+ enabled_cluster_log_types = (known after apply)
+ endpoint = (known after apply)
+ id = (known after apply)
+ identity = (known after apply)
+ kubernetes_network_config = (known after apply)
+ name = (known after apply)
+ platform_version = (known after apply)
+ role_arn = (known after apply)
+ status = (known after apply)
+ tags = (known after apply)
+ version = (known after apply)
+ vpc_config = (known after apply)
}
<OMITTED>
# module.vpc.aws_vpc.this[0] will be created
+ resource "aws_vpc" "this" {
+ arn = (known after apply)
+ assign_generated_ipv6_cidr_block = false
+ cidr_block = "10.0.0.0/16"
+ default_network_acl_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_group_id = (known after apply)
+ dhcp_options_id = (known after apply)
+ enable_classiclink = (known after apply)
+ enable_classiclink_dns_support = (known after apply)
+ enable_dns_hostnames = false
+ enable_dns_support = true
+ id = (known after apply)
+ instance_tenancy = "default"
+ ipv6_association_id = (known after apply)
+ ipv6_cidr_block = (known after apply)
+ main_route_table_id = (known after apply)
+ owner_id = (known after apply)
+ tags = {
+ "Name" = "eks-threat-modelling-ddddaaaa"
}
+ tags_all = {
+ "Name" = "eks-threat-modelling-ddddaaaa"
}
}
Plan: 44 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ cluster_endpoint = (known after apply)
+ cluster_name = (known after apply)
+ cluster_security_group_id = (known after apply)
+ config_map_aws_auth = []
+ kubectl_config = (known after apply)
+ region = "eu-west-1This plan can be converted into JSON format which allows us to perform static analysis on all the attributes.
dev@pwnbox:$ terraform show -json tfplan > tfplan.json
dev@pwnbox:$ jq '.resource_changes[] | select((.type == "aws_eks_cluster") and .mode == "managed").change.after' tfplan.json
{
"enabled_cluster_log_types": [
"audit",
"authenticator"
],
"encryption_config": [
{
"provider": [
{}
],
"resources": [
"secrets"
]
}
],
"kubernetes_network_config": [
{}
],
"name": "eks-threat-modelling-ddddaaaa",
"tags": null,
"timeouts": {
"create": "30m",
"delete": "15m",
"update": null
},
"version": "1.19",
"vpc_config": [
{
"endpoint_private_access": true,
"endpoint_public_access": false,
"public_access_cidrs": [
"0.0.0.0/0"
]
}
]
}Once we understand the structure of the Terraform plan, we can use the open policy agent and its declarative language to write simple tests.
package play
deny[msg]{
input.vpc_config[0].endpoint_public_access == true
msg := sprintf("The public endpoint is enabled on the cluster", [])
}You can play with this simple example in an online Rego Playground.
Writing these rules can become cumbersome. It requires analysis of Terraform output structures, parsing the files, library maintenance and results reported in a meaningful way. This is where products such as Snyk IaC can help. Here is an example of the security scan performed on our default deployment.
dev@pwnbox:$ snyk iac test tfplan.json
Testing tfplan.json...
Infrastructure as code issues:
✗ EKS cluster allows public access [High Severity] [SNYK-CC-TF-94] in EKS
introduced by aws_eks_cluster[this] > vpc_config
✗ Non-Encrypted root block device [Medium Severity] [SNYK-CC-TF-53] in EC2
introduced by aws_launch_configuration[workers] > root_block_device > encrypted
✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
introduced by aws_subnet[public_0] > map_public_ip_on_launch
✗ EKS control plane logging insufficient [Low Severity] [SNYK-CC-TF-131] in EKS
introduced by aws_eks_cluster[this] > enabled_cluster_log_types
✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
introduced by aws_subnet[public_2] > map_public_ip_on_launch
✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
introduced by aws_subnet[public_1] > map_public_ip_on_launch
Organization: p0tr3c
Type: Terraform
Target file: tfplan.json
Project name: hardened
Open source: no
Project path: tfplan.json
Tested tfplan.json for known issues, found 6 issues
dev@pwnbox:/$By using Snyk’s IaC scanning, you can detect issues with your Terraform configuration before deploying to production, ensuring your production environments remain secure. Sign up for a free account at https://app.snyk.io!
Harden your Amazon EKS security
Use Snyk IaC to secure your CI/CD pipelines for free.