Supply chain security incident at CircleCI: Rotate your secrets

On January 4, CircleCI, an automated CI/CD pipeline setup tool, reported a security incident in their product by sharing an advisory.

Context around the CircleCI Incident

On December 27, security engineer Daniel Hückmann received an email notification about a potential intrusion in his CircleCI account thanks to an AWS CanaryToken placed by him. The decoy, in the form of an AWS key, was likely triggered by the attacker. Decoys, also known as honeytokens, can be placed in the systems to investigate and detect intrusions and activated by an attacker to alert you of potential security breaches.

On January 4, 2022, CircleCI advised customers to rotate their secrets in response to a security incident:

In the interest of full disclosure, Snyk is both a partner and customer of CircleCI.

CircleCI recommendations for users

CircleCI has recommended that all users immediately rotate any secrets stored in their platform, whether they be in project environment variables or contexts. In addition, they advise reviewing internal logs for any unauthorized access between December 21, 2020 and January 4, 2023 or after the secret rotation has been completed. CircleCI has invalidated Personal and Project API tokens, and rotated all OAuth tokens from GitHub and Bitbucket.

If you work in software development or infrastructure environments, you must be aware that rotating secrets can be difficult and can disrupt CI/CD workflows. What you can do is

• Create an inventory of the environment variables used in all of your projects and pipelines, including those stored at both the Context Level and Project level.

• Do not delete your secrets from CircleCI, instead revoke themto prevent any potential malicious actors from accessing them.

• Make sure to rotate the environment variables and SSH keys.

The impact of secret sprawling on the software supply chain

Storing secrets, such as passwords and API keys, in multiple locations is known as "secret sprawling" and can make it difficult to manage and secure them effectively. This can happen within an organization as well as in the software supply chain. When secrets are scattered in different locations, it becomes challenging to track and update them, which increases the risk of unauthorized access. Another issue is the use of long-standing privileges, also called static secrets, that are not rotated regularly.

Although secrets management platforms can effectively store and encrypt secrets, static secrets are still vulnerable. They may leak if included in plain-text form in application source code, recorded in application logs, or stored in configuration files. The more a secret is used, the greater the risk of it being compromised. Additionally, static secrets can grant persistent privileges across multiple systems, providing hackers with ongoing access to your infrastructure.

There are several issues that can arise with the use of static credentials, which are shared and have widespread usage. These include the lack of a clear owner, stale credentials that have not been cleaned up when services are decommissioned, a lack of regular rotation process, no expiration or time-to-live limit, and the unnecessary use of static credentials when more secure access control mechanisms, such as short-lived dynamically created tokens, are available.

In a supply chain attack, a hacker infiltrates a company's supply chain to gain access to their systems or data. This can be particularly dangerous because it allows the hacker to bypass security measures and access sensitive information. The attacker will often try to compromise the source code or build processes, initiate malicious activity, or allow further breaches down the supply chain.

It is important for organizations to recognise the risks of secret sprawling and put measures in place to prevent supply chain attacks. Software products often consist of various components sourced from different vendors and stored in different repositories. If hackers are able to gain access, they may embed malware and ensure that the sub-component is signed, allowing it to be trusted as it moves through the supply chain.

To address these issues, organizations can use a secrets management solution with an encrypted vault to secure credentials and keys. This solution should also offer developers streamlined workflows for accessing secrets as needed, governed by policies approved by the security team. To further protect keys and credentials, the solution should be able to easily rotate secrets.

Dynamic secrets are a type of credential that are generated on-demand and provide temporary access to a resource for a limited period of time with a limited set of permissions. They are not stored, making them less vulnerable to attack. Even if a dynamic secret is compromised, it becomes useless before a cybercriminal can use it. Dynamic secrets are often used in conjunction with zero standing privileges (ZSP), which means that clients have privileged access to a resource with only the minimum rights needed to accomplish a specific task for the minimum time required. This approach helps to reduce the risk of unauthorized access and ensures that privileges are granted only when needed.

How to rotate secrets

Rotating secrets is replacing an existing secret (such as a password, API key, or SSH key) with a new one in order to reduce the risk of them being compromised. This can be done manually or automatically, depending on the system being used.

There are a few different strategies you can use to rotate secrets:

• Regular intervals: You can set a schedule to rotate secrets at regular intervals. This can help to reduce the risk of a secret being compromised, as an attacker would only have a limited window of time to try to exploit it.

• On-demand: You can also rotate secrets whenever you suspect that a secret may have been compromised or if you want to reduce the risk of compromise proactively.

• Automation: You can use tools and scripts to automate rotating secrets. This helps ensure that secrets are rotated in consistently and promptly.

When rotating secrets, it is important to ensure that the new secret is adequately protected and properly implemented and configured. Communicating the change to any relevant parties, such as system administrators and users, is also necessary.

Rethinking the strategy of secrets management

While we may have previously believed that our secrets were secure, past instances of platform compromise have caused organizations to reevaluate their strategies for storing secrets and determine the most secure method. It's important to consider that once secrets are used and committed, they are no longer secrets. Therefore, what is the best way to manage secrets? Should they not be used at all or just used once and then revoked?

This post will be updated if any new information arises.