Best practices for Kubernetes Secrets management

Kubernetes uses secret objects, called Secrets, to store OAuth tokens, secure shell (SSH) keys, passwords, and other secret data. Kubernetes Secrets allow us to keep confidential data separate from our application code by creating it separately from pods. This segregation, along with well-formed role-based access control (RBAC) configuration, reduces the chances of the Secret being exposed — and potentially exploited — when interacting with pods, thereby increasing security.

While Kubernetes Secrets are helpful for preventing accidental data exposure, they may not secure the cluster data against malicious cyberattacks. For example, an unauthorized user with permissions to access our etcd cluster can access all our Kubernetes Secrets, so we must carefully manage Kubernetes Secrets to ensure their security.

In this article, we’ll explore best practices for Kubernetes Secret management. Check out our Kubernetes security article for more security risks and best practices.

How Kubernetes Secrets work

Kubernetes Secrets allow users to store data in a secret object, which can then be accessed or modified by Kubernetes clients. A Secret is a key-value pair that includes a Secret ID and authenticates information. Kubelet uses the Secret ID to identify the credentials that must be provided to an application container when creating a pod.

Kubelet runs on each node in a cluster and manages pods and their containers. Secrets are only accessible by pods if they’re explicitly part of a mounted volume or at the moment kubelet pulls an image for use in a pod.

The Kubernetes API server stores Kubernetes Secrets. They’re only accessible to specific users with permission to access the Kubernetes API server. The Kubernetes API server is a single point of failure for our application, so we must keep our data safe and secure.

Kubernetes Secrets:

• Provide a secure way to share configuration data between a controller and its workers, such as between a kubelet and a pod

• Offer an alternative way to store sensitive data in the Kubernetes cluster, such as credentials to access external applications

• Provide a convenient way to create new resources on our Kubernetes cluster, such as new deployments or namespaces

Best practices for Kubernetes Secrets

Secrets, like keys, passwords, tokens, and other configuration values, must be stored correctly. If our Kubernetes cluster is compromised, the Secrets must remain secure. An attacker shouldn’t be able to exploit Secrets to compromise sensitive data, build a botnet, or command and control (C2) servers.

Here are some techniques to help us keep Kubernetes Secrets safe:

1. Enable encryption at rest

2. Configure RBAC rules

3. Encrypt etcd data

4. Use a centralized Secrets store for easy management

Enable encryption at rest

Kubernetes Secret data is encoded in the base64 format and stored as plain text in etcd. Etcd is a key-value store used as a backing store for Kubernetes cluster state and configuration data. Storing Secrets as plain text in etcd is risky, as they can be easily compromised by attackers and used to access systems.

The etcd database isn’t encrypted by default, so we need to encrypt Secret data at rest to protect the sensitive information contained in them from falling into the hands of an attacker. An attacker with access to the file system can read Secrets from the file.

Additionally, Kubernetes provides an encryption at rest feature that enables us to use the Kubernetes API to encrypt Secrets before storing them in etcd. We can also use encryption providers, such as KMSConfiguration, IdentityConfiguration, SecretboxConfiguration, and AESConfiguration to protect our Secrets at rest.

A Kubernetes cluster has multiple nodes, each with unique credentials for encryption and decryption operations. The EncryptionConfiguration object specifies the key material for encryption and decryption operations on a per-node basis. The configuration for these encryption providers is in the EncryptionConfiguration object, which allows the encryption of Secrets locally with a locally-managed key.

Configure RBAC rules

Encrypting Kubernetes Secrets at rest is only part of what’s required to keep Secrets safe. We should also control access to secrets using RBAC rules for Kubernetes. The security policy authorizing Secret access can be task-based, time-based, or role-based.

Kubernetes Secrets and RBAC rules work hand-in-hand, as one of the primary reasons Kubernetes Secret objects exist is to grant different RBAC access from what we would for ConfigMap.

We can use the ClusterRoles object to define the actions a user can perform within a cluster and a role to define the actions a user can perform within a namespace. RBAC restricts creating, deleting, and modifying Secrets to specific users only. For example, we can set a policy prohibiting developers from making Secrets for a given namespace and not others.

Kubernetes’ built-in RBAC framework allows us to enforce the principle of least privilege (PoLP) to restrict user or program access to only the resources or information required to perform assigned tasks or function properly. It helps us ensure that only privileged accounts and containers have access to Secrets. Should an attacker compromise one component, the PoLP works to prevent them from escalating their permissions and compromising other components.

Encrypt etcd data

Because we only need permission to access the etcd cluster to access Secrets, we need to be diligent when protecting sensitive data, such as automatic key rotation, insights into key age, and audit logs. The best way to do this is to make this data unavailable in etcd. The default storage driver for etcd is local, and the encryption keys are stored locally in the config file. This can make it vulnerable to malware or other malicious threats.

One way to harden the security of etcd data is to encrypt Secrets before storing them. We should use an encryption provider, such as a Key Management Service (KMS), to store our keys and Secrets. It’s good to note that most managed Kubernetes providers encrypt etcd Secrets storage by default when a cluster is created, which helps keep our etcd data safe.

Another security method is to allow access to etcd only by the API server. We should only permit the nodes that require access to use etcd clusters. We can grant access to etcd using the Kubernetes API server, and grant read and write permissions only for specific users or groups.

Use a centralized Secrets store for easy management

Kubernetes Secrets are critical to the operation of our application and we can store them in multiple locations. When we store secrets in many places, securing them becomes challenging — especially when we must manage Secrets in multiple clusters, or if we have a mix of private and public key pairs. Without a centralized Secrets management system, Secrets are sprawled across locations, increasing the attack surface for unauthorized users.

Managing Secrets centrally offers many benefits, especially when using the Secrets in multiple instances. A centralized management solution gives us a unified view of the Kubernetes security landscape. We can easily manage Secrets, access control, and audits, and a central audit trail gives insights into critical security events.

Third-party providers offer more secure and advanced features for Kubernetes Secrets management. Below are some popular third-party Secrets management services.

AWS Secrets Manager

AWS Secrets Manager provides a safe, easy way to store and rotate Secrets. It offers a unified management console for all our Amazon Web Services (AWS) resources. The AWS Secrets Manager tool is integrated with Kubernetes, allowing us to securely store our encrypted Secrets in Kubernetes. We can then use the Kubernetes API to access these secrets at runtime without worrying about exposing them in our code.

This Secrets management service provides the ability to retrieve, view, and manage AWS credentials and secrets in a single, secure location. We can use AWS Secrets Manager with other AWS services, such as AWS Identity and Access Management (IAM), AWS Key Management Service (AWS KMS), and AWS Simple Storage Service (S3).

Azure Key Vault and Azure Kubernetes Service

Microsoft Azure Key Vault is a Secrets manager that allows users to store and use cryptographic keys. We can back up and recover deleted vault objects, log and monitor Secrets, and use key vaults to authenticate users.

Additionally, we can use Azure Kubernetes Service (AKS) for Azure RBAC and Kubernetes RBAC authorization. We can also use Azure RBAC to control resource access through role assignments. It also authorizes groups and users, while the built-in Kubernetes RBAC allows Kubernetes service accounts.

HashiCorp Vault

HashiCorp Vault is an open source Secrets management tool for storing and managing Secrets and protecting sensitive data. This tool manages user access to sensitive data, allowing us to rotate sensitive data or revoke its access when there’s a security threat.

We can use HashiCorp Vault to employ various Kubernetes security measures, including data encryption, identity-based access, and Secrets management. HashiCorp Vault uses TSL and AES 256-bit encryption to secure data in transit and at rest, respectively.

Conclusion

Secrets authenticate users and services and control access to resources and other Secrets. Kubernetes Secrets allow users to store data in a secret object, which can then be accessed or modified by Kubernetes clients. Because Secrets contain sensitive data, such as tokens, keys, and passwords, it’s critical that we follow best practices to keep our Secrets secure.

There are several ways we can protect our Secrets. To start, we should enable encryption at rest. Encrypting secrets at rest improves the security of our Secrets. However, encrypting Kubernetes Secret objects at rest is only part of keeping Secrets safe. We should also control access to secrets using RBAC rules. Kubernetes’ built-in RBAC framework allows us to restrict user or program access, making available only the resources or information required. We can also use a third-party Secrets manager service, like those outlined earlier, to help us keep our Secrets secure.

To properly manage Secrets, both developers and Kubernetes cluster administrators need to closely monitor the security of our sensitive information. In addition to the best practices outlined here, be sure to review Kubnernetes’ guide for managing Secrets — written for both developers and cluster administrators — to keep your Secrets safe and secure.

Keeping credentials and other secrets secure is just one step in running more secure applications: you need to secure each step in the software development lifecycle. Identify and mitigate potential vulnerabilities in the code you write and the open source projects that you leverage with Snyk Code and Snyk Open Source, and leverage Snyk Container to start with a more secure base image and identify any additional vulnerabilities that you may introduce in the build process. Finally, verify that the code that you deploy with doesn't include security holes with Snyk IaC.

Related articles:

• Secret Management: Tools & Best Practice

• AWS Security

• Building Docker images with Kubernetes

• 10 Kubernetes security context settings you should understand

• Security implications of Kubernetes operators

• Kubernetes monitoring

• Container orchestration