Keeping Docker secrets secure (even if you’re not using Kubernetes)

Distributed containerized systems compose applications, resources, services, databases, and other artifacts. These components often need sensitive information such as user keys, passwords, API keys, and certificates to function properly.

Secrets management is critical for adequately handling sensitive information and Kubernetes deployments often utilize their built-in Secrets resource type and associated RBAC controls but what if you aren’t deploying on Kubernetes? Many organizations simply do not need the power or want to take on the complexity of running in Kubernetes but still must ensure they avoid the risk of data leakage or lost passwords. Fortunately, Docker’s built-in Swarm orchestrator provides a solution for this.

This article discusses managing secrets in Docker, exploring best practices and potential risks with their use. The article also presents a simple exercise on creating and listing Docker secrets as well as adding and removing them to a service. It is assumed the reader has a basic understanding of the Docker container runtime and its Swarm orchestrator.

Securing Docker secrets

Whenever sending secrets or sensitive data across your organization, there is a risk of data leakage or lost passwords. When working on a Docker project where you must handle sensitive information such as passwords, private keys, tokens, and API keys, you should define a safe way to store and transmit these secrets to containers.

One of the worst approaches is to leave sensitive data in the source code. In addition to the security concerns, hard-coding secretsin your code means that you have to rebuild the image when the secrets change.

Another bad practice is to take a “manual” approach to handling important information, such as passwords and SSH keys, leaving secret information on the local disk. You must remember that unauthorized or malicious programs might also read your files.

Developers often rely on environment variables to store sensitive data, which is okay for some scenarios but not recommended for Docker containers. Environment variables are even less secure than files. They are vulnerable in more ways, such as:

• Linked containers

• The docker inspect command

• Child processes

• Event log files

For example, an application may dump the context to a log as part of an exception handling code, and the log may now include the sensitive values of environment variables.

Using a secret manager is much better than the previous options, as it provides a central place to handle secrets in Docker Swarm. A secret manager helps you protect access to your applications, services, and IT resources by providing a convenient service for storing, managing, and retrieving secrets. These include API database credentials, keys, passwords, SSH private keys, SSL certificates, and other sensitive data needed by an application at runtime.

How Docker manages secrets

The diagram above shows that Docker uses Raft, a consensus algorithm for managing a replicated log in a distributed system, to ensure that your data remains consistent in the entire swarm and all the services running on it. The Raft algorithm consistently transmits a secret in an encrypted form to all manager nodes.

The diagram also shows the worker nodes, which are instances of Docker Engine dedicated to executing containers. The manager nodes distribute secrets to the containers that are part of the service with the key.

Docker secrets work like a vault where you can put sensitive data. You can only use it if you have the vault key, which you must get from the service nodes to which you assigned the key.

The Docker secrets management service is part of its container orchestration stack and provides an automated process for keeping this data secure.

In Docker Swarm services, secrets are in encrypted form while stored or in transit.

Docker secrets can be centrally managed and securely transmitted to the containers that need access. Each secret is only accessible to the services granted explicit access and only while the service runs.

When a service gains access to secrets, it decrypts and shares them with the container in an in-memory file system. When the container task stops, the decrypted secrets available are immediately detached and flushed from the node’s memory.

Using the Docker CLI to manage secrets

Docker’s built-in functionality enables users to input secrets and exercise fine-grained control over which services can access the secrets and under what circumstances they can use the secrets.

This simple exercise creates a MySQL service with a root password and adds the credentials as secrets.

Since Docker secrets are only available to swarm services, you must set up a Docker Swarm to demonstrate Docker secrets. But first, ensure you have installed the Docker command line (Docker CLI). If you have not already done so, initialize or join the swarm:

1docker swarm init
2
3Swarm initialized: current node (wv4c3hvdwqlssjpaxzvvhy3vj) is now a manager.
4
5To add a worker to this swarm, run the following command:
6
7docker swarm join --token SWMTKN-1-3qizt7vyw8cbpii3nosl8ws7zr1npikrepr0qqh9ad7esucwon-bfxct86tefphzo8xiz8ue0tx6 192.168.65.3:2377

Now add a secret to Docker.

Create a text file where you want to store your secret. The following command creates the password.txt file with the this-is-a-mysql-password string.

1$ echo "this-is-a-mysql-password" > $HOME/password.txt

Now use the docker secret create command to create a new secret using the file made in the previous step:

1$ docker secret create mysql-password-secret $HOME/password.txt
2
3l1m5jvgcox1l96i6bjbz4dvnt

Note that the l1m5jvgcox1l96i6bjbz4dvnt string above represents the secret ID, not the secret itself, so it will likely be different in your environment.

Now, let’s see another way of using the docker secret create command. Here, omit the file name by setting the last argument, which represents the file, to the minus symbol (-):

1$ echo "another-mysql-password" | docker secret create another-mysql-secret -

This command generates an output like the one below:

1q3bjo7wx4xqr45ma20lk5j2g7

This omits the file from which to read the secret, then the docker secret create command reads the standard input another-mysql-password.

Listing Docker secrets

To list the available secrets in the Docker Swarm, run the docker secret ls command:

1$ docker secret ls

This produces an output with the secrets you created in the Docker Swarm:

1ID                          NAME                    DRIVER    CREATED              UPDATED
2q3bjo7wx4xqr45ma20lk5j2g7   another-mysql-secret              46 seconds ago       46 seconds ago
3l1m5jvgcox1l96i6bjbz4dvnt   mysql-password-secret             About a minute ago   About a minute ago

Creating a service with secrets

You can create a service using one of the secrets created above.

Create the mysql-service service and use the --secret flag to give container access to the another-mysql-secret secret:

1$ docker service create --name mysql-service --secret another-mysql-secret mysql:5.7

Adding or removing secrets to and from an existing service

You can remove a secret from some existing service:

1$ docker service update --secret-rm another-mysql-secret mysql-service

Or you can add a secret to some service that is running:

1$ docker service update --secret-add mysql-password-secret mysql-service

When binding a secret to a service, Docker creates a text file with the name of your secret. The text file contains a string representing the secret and, by default, is in the /run/secrets directory mounted in a temporary file system (TMPFS) inside the container. In our case, the file is in the path /run/secrets/mysql-password-secret. Upon creation, the secret is available to all containers of that service.

External secret management alternatives

Docker secrets are an excellent solution but you should also consider offloading this task to external, third party secret storage solutions. This is especially true if your organization already has expertise and investment in a specific platform.

For example, if your company already is using Hashicorp Vault and your applications are integrated with it, there is little reason to switch to Docker secrets — or Kubernetes secrets for that matter — simply because they are provided in the platform. Leverage the existing processes, libraries and expertise around that solution in your services.

Some popular secrets management alternatives include:

• Hasicorp Vault

• Cyberark Conjour

• Bitwarden

• Cloud Provider solutions (AWS Secrets Manager, GCP Secret Manager, etc)

Keep your secrets secure

This article introduced the concept of secrets and secret management in the context of Docker Swarm and showed how Docker manages secrets. It demonstrated how using a secret manager is much better than hard-coding sensitive data, using environment variables, or taking a manual approach to managing secrets.

Data security is one of the pillars of effective corporate governance. With the help of CLI commands and an in-memory architecture that enables fine-grained deployment of secret data, Docker provides solid secret management tools that help you properly implement your organization’s information security.

Keeping secrets away from prying eyes is one step towards building secure containers. Snyk Code can help you find those hard coded secrets, while Snyk Container and Snyk Open Source point you toward the most secure base images and packages for your container. And once you're ready for production, Snyk IaC helps you reduce risk pre- and post-deployment. Start for free today, to find and automatically fix vulnerabilities in your code, open source dependencies, containers, and cloud infrastructure.