We use cookies to ensure you get the best experience on our website.Read moreRead moreGot it

close
  • Products
    • Products
      • Snyk Code (SAST)
        Secure your code as it’s written
      • Snyk Open Source (SCA)
        Avoid vulnerable dependencies
      • Snyk Container
        Keep your base images secure
      • Snyk Infrastructure as Code
        Develop secure cloud infrastructure
      • Snyk Cloud
        Keep your cloud environment secure
    • Solutions
      • Application security
        Build secure, stay secure
      • Software supply chain security
        Mitigate supply chain risk
      • Cloud security
        Build and operate securely
    • Platform
      • What is Snyk?
        Developer-first security in action
      • Developer security platform
        Modern security in a single platform
      • Security intelligence
        Comprehensive vulnerability data
      • License compliance management
        Manage open source usage
      • Snyk Learn
        Self-service security education
  • Resources
    • Using Snyk
      • Documentation
      • Vulnerability intelligence
      • Product training
      • Support & services
      • Support portal & FAQ’s
      • User hub
    • learn & connect
      • Blog
      • Community
      • Events & webinars
      • DevSecOps hub
      • Developer & security resources
    • Listen to the Cloud Security Podcast, powered by Snyk
  • Company
    • About Snyk
    • Customers
    • Partners
    • Newsroom
    • Snyk Impact
    • Contact us
    • Jobs at Snyk We are hiring
  • Pricing
Log inBook a demoSign up
All articles
  • Application Security
  • Cloud Native Security
  • DevSecOps
  • Engineering
  • Partners
  • Snyk Team
  • Show more
    • Vulnerabilities
    • Product
    • Ecosystems
Amazon EKS security tips
Cloud Native SecurityDevSecOps

Four steps for hardening Amazon EKS security

Kamil PotrecJuly 21, 2021

In the first part of this blog series, we explored deploying Amazon EKS with Terraform, and looked at how to secure the initial RBAC implementation along with securing the Instance Metadata Service. In this second post, we’ll look at more best practices to harden Amazon EKS security, including the importance of dedicated continuous delivery IAM roles, multi-account architecture for Amazon EKS cluster isolation, and how to encrypt your secrets in the control plane. And finally, we will show how to incorporate static analysis tooling in your CD pipelines to catch these issues before they reach production environments.

Keep track of who deployed the cluster

The Amazon EKS service integrates with the Identity Access Management service to authenticate users to the cluster. The authorization decisions are performed by the native Kubernetes RBAC model. RBAC configuration is mapped to the IAM identities via aws-auth ConfigMap. However there is a special mapping that is not visible in any configuration file or setting. The IAM entity used to create the cluster is automatically mapped to the system:masters built-in Kubernetes group. This is well documented in the user documentation. However, it still has significant implications on auditability of the cluster permissions.

In order to showcase the impact of this configuration we will delete all Roles and ClusterRoles from our demo cluster.

dev@pwnbox:~$ kubectl delete clusterroles --all
clusterrole.rbac.authorization.k8s.io "admin" deleted
clusterrole.rbac.authorization.k8s.io "aws-node" deleted
clusterrole.rbac.authorization.k8s.io "cluster-admin" deleted
clusterrole.rbac.authorization.k8s.io "edit" deleted
clusterrole.rbac.authorization.k8s.io "eks:addon-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:fargate-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:node-bootstrapper" deleted
clusterrole.rbac.authorization.k8s.io "eks:node-manager" deleted
clusterrole.rbac.authorization.k8s.io "eks:podsecuritypolicy:privileged" deleted
<OMITTED>
dev@pwnbox:~$ kubectl delete roles --all-namespaces --all
role.rbac.authorization.k8s.io "system:controller:bootstrap-signer" deleted
role.rbac.authorization.k8s.io "eks:addon-manager" deleted
role.rbac.authorization.k8s.io "eks:certificate-controller" deleted
role.rbac.authorization.k8s.io "eks:fargate-manager" deleted
role.rbac.authorization.k8s.io "eks:node-manager" deleted
<OMITTED>
dev@pwnbox:~$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names   Verbs
*.*                                               []                  []               [*]
                                                [*]                 []               [*]
selfsubjectaccessreviews.authorization.k8s.io   []                  []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []               [create]
                                                [/api/*]            []               [get]
                                                [/api]              []               [get]
                                                [/apis/*]           []               [get]
                                                [/apis]             []               [get]
                                                [/healthz]          []               [get]
                                                [/healthz]          []               [get]
                                                [/livez]            []               [get]
                                                [/livez]            []               [get]
                                                [/openapi/*]        []               [get]
                                                [/openapi]          []               [get]
                                                [/readyz]           []               [get]
                                                [/readyz]           []               [get]
                                                [/version/]         []               [get]
                                                [/version/]         []               [get]
                                                [/version]          []               [get]
                                                [/version]          []               [get]
dev@pwnbox:~$ kubectl get configmap --all-namespaces
NAMESPACE     NAME                                 DATA   AGE
kube-system   coredns                              1      7h59m
kube-system   cp-vpc-resource-controller           0      7h58m
kube-system   eks-certificates-controller          0      7h59m
kube-system   extension-apiserver-authentication   6      7h59m
kube-system   kube-proxy                           1      7h59m
kube-system   kube-proxy-config                    1      7h59m

Amazon EKS uses webhook token authentication in order to integrate with IAM. We can see the configuration in the api server arguments.

FLAG: --authentication-token-webhook-cache-ttl="7m0s"
FLAG: --authentication-token-webhook-config-file="/etc/kubernetes/authenticator/apiserver-webhook-kubeconfig.yaml"
FLAG: --authentication-token-webhook-version="v1beta1"

AWS IAM has a concept of unique identifiers which can be used in specific services to avoid name reuse misconfiguration. It is unclear from the Amazon EKS documentation whether the system:masters permission is bound to the unique identifier of the entity or its friendly name. In order to validate if removing the original entity will revoke all permissions from the cluster, we have deleted the temporary role and recreated it.

dev@pwnbox:~$ aws iam get-role --role-name ci-eks-test
{
    "Role": {
        "Path": "/",
        "RoleName": "ci-eks-test",
        "RoleId": "AROAYREY3WYOOSHLPO3W6",
        "Arn": "arn:aws:iam::123456789012:role/ci-eks-test",
        "CreateDate": "2021-06-20T21:09:01+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::123456789012:user/ci"
                    },
                    "Action": "sts:AssumeRole",
                    "Condition": {}
                }
            ]
        },
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {}
    }
}
dev@pwnbox:~$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names     Verbs
*.*                                             []                  []                 [*]
                                                [*]                 []                 [*]
selfsubjectaccessreviews.authorization.k8s.io   []                  []                 [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []                 [create]
                                                [/api/*]            []                 [get]
                                                [/api]              []                 [get]
                                                [/apis/*]           []                 [get]
                                                [/apis]             []                 [get]
                                                [/healthz]          []                 [get]
                                                [/healthz]          []                 [get]
                                                [/livez]            []                 [get]
                                                [/livez]            []                 [get]
                                                [/openapi/*]        []                 [get]
                                                [/openapi]          []                 [get]
                                                [/readyz]           []                 [get]
                                                [/readyz]           []                 [get]
                                                [/version/]         []                 [get]
                                                [/version/]         []                 [get]
                                                [/version]          []                 [get]
                                                [/version]          []                 [get]
podsecuritypolicies.policy                      []                  [eks.privileged]   [use]


dev@pwnbox:~$ aws iam get-role --role-name ci-eks-test
{
    "Role": {
        "Path": "/",
        "RoleName": "ci-eks-test",
        "RoleId": "AROAYREY3WYOAK6QMGSUT",
        "Arn": "arn:aws:iam::123456789012:role/ci-eks-test",
        "CreateDate": "2021-06-20T21:14:28+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::123456789012:user/ci"
                    },
                    "Action": "sts:AssumeRole",
                    "Condition": {}
                }
            ]
        },
        "MaxSessionDuration": 3600,
        "RoleLastUsed": {}
    }
}
dev@pwnbox:~$ kubectl auth can-i --list
Resources                                       Non-Resource URLs   Resource Names     Verbs
*.*                                             []                  []                 [*]
                                                [*]                 []                 [*]
selfsubjectaccessreviews.authorization.k8s.io   []                  []                 [create]
selfsubjectrulesreviews.authorization.k8s.io    []                  []                 [create]
                                                [/api/*]            []                 [get]
                                                [/api]              []                 [get]
                                                [/apis/*]           []                 [get]
                                                [/apis]             []                 [get]
                                                [/healthz]          []                 [get]
                                                [/healthz]          []                 [get]
                                                [/livez]            []                 [get]
                                                [/livez]            []                 [get]
                                                [/openapi/*]        []                 [get]
                                                [/openapi]          []                 [get]
                                                [/readyz]           []                 [get]
                                                [/readyz]           []                 [get]
                                                [/version/]         []                 [get]
                                                [/version/]         []                 [get]
                                                [/version]          []                 [get]
                                                [/version]          []                 [get]
podsecuritypolicies.policy                      []                  [eks.privileged]   [use]

As you can see we have a completely new role as identified by different RoleId. However, the Amazon EKS authentication service uses the friendly name of the role to grant us the same system:masters permissions. 

The most effective way to mitigate this issue is to use a centralized deployment system, use a dedicated role for deployments, and avoid use of personal credentials at all cost. Tools such as aws-vault can help you securely manage switching between different roles. 

Next let’s have a look at how nodes authenticate to the control plane.

Use multi-account separation for clusters

Amazon EKS requires that all nodes have an instance profile attached with the following two policies: AmazonEKSWorkerNodePolicy and AmazonEC2ContainerRegistryReadOnly.

dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy --version-id v1
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "ec2:DescribeInstances",
                        "ec2:DescribeRouteTables",
                        "ec2:DescribeSecurityGroups",
                        "ec2:DescribeSubnets",
                        "ec2:DescribeVolumes",
                        "ec2:DescribeVolumesModifications",
                        "ec2:DescribeVpcs",
                        "eks:DescribeCluster"
                    ],
                    "Resource": "*",
                    "Effect": "Allow"
                }
            ]
        },
        "VersionId": "v1",
        "IsDefaultVersion": true,
        "CreateDate": "2018-05-27T21:09:01+00:00"
    }
}

dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly --version-id v3
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "ecr:GetAuthorizationToken",
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:GetDownloadUrlForLayer",
                        "ecr:GetRepositoryPolicy",
                        "ecr:DescribeRepositories",
                        "ecr:ListImages",
                        "ecr:DescribeImages",
                        "ecr:BatchGetImage",
                        "ecr:GetLifecyclePolicy",
                        "ecr:GetLifecyclePolicyPreview",
                        "ecr:ListTagsForResource",
                        "ecr:DescribeImageScanFindings"
                    ],
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v3",
        "IsDefaultVersion": true,
        "CreateDate": "2019-12-10T20:56:32+00:00"
    }
}

The policies by default gain privileges to all resources within the AWS account. This means that a compromised pod on this cluster will be able to enumerate information about any resource, vpc, subnet, and image in the account. 

The Amazon EKS node actually requires access to one more managed policy named AmazonEKS_CNI_Policy. This policy allows the node to allocate IP addresses and perform any networking-related functionality.

dev@pwnbox:$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy --version-id v4
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "ec2:AssignPrivateIpAddresses",
                        "ec2:AttachNetworkInterface",
                        "ec2:CreateNetworkInterface",
                        "ec2:DeleteNetworkInterface",
                        "ec2:DescribeInstances",
                        "ec2:DescribeTags",
                        "ec2:DescribeNetworkInterfaces",
                        "ec2:DescribeInstanceTypes",
                        "ec2:DetachNetworkInterface",
                        "ec2:ModifyNetworkInterfaceAttribute",
                        "ec2:UnassignPrivateIpAddresses"
                    ],
                    "Resource": "*"
                },
                {
                    "Effect": "Allow",
                    "Action": [
                        "ec2:CreateTags"
                    ],
                    "Resource": [
                        "arn:aws:ec2:*:*:network-interface/*"
                    ]
                }
            ]
        },
        "VersionId": "v4",
        "IsDefaultVersion": true,
        "CreateDate": "2020-04-20T20:52:01+00:00"
    }
}

This policy allows the caller to modify the networking configuration of any instance in the account. Amazon actually recommends this policy be attached directly to the aws-node Kubernetes service account. By default, the Terraform module attaches this policy to the IAM profile. We can change that behaviour by setting the attach_worker_cni_policy attribute to false. 

Encrypt your Secrets

One of the control plane components managed by the Amazon EKS service is the etcd key value store. It is used by the Kubernetes API server to store all objects and configurations. By default Kubernetes does not encrypt data stored in this key value store. AWS is responsible for security of the control plane components however we can yet again apply a defence in depth strategy to further enhance the security of our data. 

AWS KMS service is one of the encryption providers that can provide envelope encryption for secrets objects stored in our cluster. By setting the `cluster_encryption_config` option we can specify a KMS key which will be used to encrypt intermediate data encryption keys, which in turn will be used to encrypt specific objects. This Amazon EKS blog post from AWS provides a great overview of the underlying process. From this point onward anyone with access to an etcd cluster will not be able to read our secrets without also having access to our KMS key. The whole process is fully transparent to the end user. 

Detect issues in the deployment pipeline

Infrastructure as code allows us to declaratively describe the desired state of the Amazon EKS cluster. With that we have the ability to statically discover some of these issues before anything is deployed. 

In Terraform we can generate a plan of configuration that will be deployed.

dev@pwnbox:$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_eks_cluster.cluster will be read during apply
  # (config refers to values not yet known)
 <= data "aws_eks_cluster" "cluster"  {
      + arn                       = (known after apply)
      + certificate_authority     = (known after apply)
      + created_at                = (known after apply)
      + enabled_cluster_log_types = (known after apply)
      + endpoint                  = (known after apply)
      + id                        = (known after apply)
      + identity                  = (known after apply)
      + kubernetes_network_config = (known after apply)
      + name                      = (known after apply)
      + platform_version          = (known after apply)
      + role_arn                  = (known after apply)
      + status                    = (known after apply)
      + tags                      = (known after apply)
      + version                   = (known after apply)
      + vpc_config                = (known after apply)
    }

<OMITTED>

# module.vpc.aws_vpc.this[0] will be created
  + resource "aws_vpc" "this" {
      + arn                              = (known after apply)
      + assign_generated_ipv6_cidr_block = false
      + cidr_block                       = "10.0.0.0/16"
      + default_network_acl_id           = (known after apply)
      + default_route_table_id           = (known after apply)
      + default_security_group_id        = (known after apply)
      + dhcp_options_id                  = (known after apply)
      + enable_classiclink               = (known after apply)
      + enable_classiclink_dns_support   = (known after apply)
      + enable_dns_hostnames             = false
      + enable_dns_support               = true
      + id                               = (known after apply)
      + instance_tenancy                 = "default"
      + ipv6_association_id              = (known after apply)
      + ipv6_cidr_block                  = (known after apply)
      + main_route_table_id              = (known after apply)
      + owner_id                         = (known after apply)
      + tags                             = {
          + "Name" = "eks-threat-modelling-ddddaaaa"
        }
      + tags_all                         = {
          + "Name" = "eks-threat-modelling-ddddaaaa"
        }
    }

Plan: 44 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cluster_endpoint          = (known after apply)
  + cluster_name              = (known after apply)
  + cluster_security_group_id = (known after apply)
  + config_map_aws_auth       = []
  + kubectl_config            = (known after apply)
  + region                    = "eu-west-1

This plan can be converted into JSON format which allows us to perform static analysis on all the attributes.

dev@pwnbox:$ terraform show -json tfplan > tfplan.json
dev@pwnbox:$ jq '.resource_changes[] | select((.type == "aws_eks_cluster") and .mode == "managed").change.after' tfplan.json
{
  "enabled_cluster_log_types": [
    "audit",
    "authenticator"
  ],
  "encryption_config": [
    {
      "provider": [
        {}
      ],
      "resources": [
        "secrets"
      ]
    }
  ],
  "kubernetes_network_config": [
    {}
  ],
  "name": "eks-threat-modelling-ddddaaaa",
  "tags": null,
  "timeouts": {
    "create": "30m",
    "delete": "15m",
    "update": null
  },
  "version": "1.19",
  "vpc_config": [
    {
      "endpoint_private_access": true,
      "endpoint_public_access": false,
      "public_access_cidrs": [
        "0.0.0.0/0"
      ]
    }
  ]
}

Once we understand the structure of the Terraform plan, we can use the open policy agent and its declarative language to write simple tests.

package play

deny[msg]{
    input.vpc_config[0].endpoint_public_access == true
    msg := sprintf("The public endpoint is enabled on the cluster", [])
}

You can play with this simple example in an online Rego Playground. 

Writing these rules can become cumbersome. It requires analysis of Terraform output structures, parsing the files, library maintenance  and results reported in a meaningful way. This is where products such as Snyk IaC can help. Here is an example of the security scan performed on our default deployment.

dev@pwnbox:$ snyk iac test tfplan.json

Testing tfplan.json...


Infrastructure as code issues:
  ✗ EKS cluster allows public access [High Severity] [SNYK-CC-TF-94] in EKS
    introduced by aws_eks_cluster[this] > vpc_config

  ✗ Non-Encrypted root block device [Medium Severity] [SNYK-CC-TF-53] in EC2
    introduced by aws_launch_configuration[workers] > root_block_device > encrypted

  ✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
    introduced by aws_subnet[public_0] > map_public_ip_on_launch

  ✗ EKS control plane logging insufficient [Low Severity] [SNYK-CC-TF-131] in EKS
    introduced by aws_eks_cluster[this] > enabled_cluster_log_types

  ✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
    introduced by aws_subnet[public_2] > map_public_ip_on_launch

  ✗ Public IPs are automatically mapped to instances [Low Severity] [SNYK-CC-AWS-427] in VPC
    introduced by aws_subnet[public_1] > map_public_ip_on_launch


Organization:      p0tr3c
Type:              Terraform
Target file:       tfplan.json
Project name:      hardened
Open source:       no
Project path:      tfplan.json

Tested tfplan.json for known issues, found 6 issues

dev@pwnbox:/$

By using Snyk’s IaC scanning, you can detect issues with your Terraform configuration before deploying to production, ensuring your production environments remain secure. Sign up for a free account at https://app.snyk.io! Additionally, you can also try Snyk’s free code checker tool for a quick sense on the security of your code.

Harden your Amazon EKS security

Use Snyk IaC to secure your CI/CD pipelines for free.

Sign up for free

Discuss this blog on Discord

Join the DevSecOps Community on Discord to discuss this topic and more with other security-focused practitioners.

GO TO DISCORD
Footer Wave Top
Patch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo SegmentPatch Logo Segment
Develop Fast.
Stay Secure.
Snyk|Open Source Security Platform
Sign up for freeBook a demo

Product

  • Developers & DevOps
  • Vulnerability database
  • API status
  • Pricing
  • IDE plugins
  • What is Snyk?

Resources

  • Snyk Learn
  • Blog
  • Security fundamentals
  • Resources for security leaders
  • Documentation
  • Snyk API
  • Disclosed vulnerabilities
  • Open Source Advisor
  • FAQs
  • Website scanner
  • Code snippets
  • Japanese site
  • Audit services
  • Web stories

Company

  • About
  • Snyk Impact
  • Customers
  • Jobs at Snyk
  • Snyk for government
  • Legal terms
  • Privacy
  • Press kit
  • Events
  • Security and trust
  • Do not sell my personal information

Connect

  • Book a demo
  • Contact us
  • Support
  • Report a new vuln

Security

  • JavaScript Security
  • Container Security
  • Kubernetes Security
  • Application Security
  • Open Source Security
  • Cloud Security
  • Secure SDLC
  • Cloud Native Security
  • Secure coding
  • Python Code Examples
  • JavaScript Code Examples
  • Code Checker
  • Python Code Checker
  • JavaScript Code Checker
Snyk|Open Source Security Platform

Snyk is a developer security platform. Integrating directly into development tools, workflows, and automation pipelines, Snyk makes it easy for teams to find, prioritize, and fix security vulnerabilities in code, dependencies, containers, and infrastructure as code. Supported by industry-leading application and security intelligence, Snyk puts security expertise in any developer's toolkit.

Resources

  • Snyk Learn
  • Blog
  • Security fundamentals
  • Resources for security leaders
  • Documentation
  • Snyk API
  • Disclosed vulnerabilities
  • Open Source Advisor
  • FAQs
  • Website scanner
  • Code snippets
  • Japanese site
  • Audit services
  • Web stories

Track our development

© 2023 Snyk Limited
Registered in England and Wales
Company number: 09677925
Registered address: Highlands House, Basingstoke Road, Spencers Wood, Reading, Berkshire, RG7 1NT.
Footer Wave Bottom