Lessons learned from the Argo CD zero-day vulnerability (CVE-2022-24348)
2022年2月10日
0 分で読めますVulnerability update
A new, high-severity vulnerability, CVE-2022-1025, has been announced. We’ve updated the recommended version numbers here to those recommended to resolve that issue as well.
On January 30, 2022, the Argo CD team was contacted by researchers at Apiiro regarding a vulnerability they had discovered in the popular continuous delivery platform that could allow bad actors to steal sensitive information from deployments. The Argo CD team was able to quickly develop fixes for all three of their currently supported releases and publish them to their users within 48 hours. In his blog article about the CVE, CodeFresh Co-Founder and Argo project maintainer Dan Garfield credits the swift response to the project having had a platform-wide focus on security over the last 18 months.
In this article, we’ll talk about the vulnerability, as well as the larger implications regarding securing yourself against the kind of supply chain attack this vulnerability could have caused.
What is the Argo CD vulnerability?
At its core, CVE-2022-24348 is a directory/path traversal vulnerability that allows an attacker to use a maliciously formed Helm chart to access data belonging to other applications. This can potentially lead toward a privilege escalation and expanded exploitation of the Kubernetes cluster on which Argo CD is running.
How do we know if we are vulnerable?
If you are running Argo CD 0.5.0 through 2.1.12, 2.2.7, or 2.3.1, you are vulnerable and should upgrade immediately to the latest versions. As of March 24, 2022, they are:
2.3.2
2.2.8
2.1.14
What is the risk?
This is another example of vectors in the ever-growing list of supply chain attacks where bad actors try to infiltrate software as it’s being constructed rather than storming the front gates after it’s released. Some recent examples include exploits of CodeCov, the iOS app store and, most famously, SolarWinds.In this latest incident, an attacker could craft a malicious Helm chart that exploits the vulnerability, allowing them to gain access to sensitive data belonging to other applications in the Argo CD reposerver.
The details of how this particular vulnerability works are in the original CVE disclosure, so we won’t rehash all of that here. Suffice to say, that by crafting a Helm chart that includes a URI with a specific, predictable, absolute file path, an attacker could exfiltrate sensitive filesystem data from that path in the Argo reposerver — including data from other applications/user’s scopes.
What’s the fix?
If you came here looking for the fix for CVE-2022-24348 in your clusters, that’s easy: stop reading now and upgrade your Argo CD deployments immediately! Then come back and continue reading for a look at the bigger picture.
Done? Good! Now let’s talk about the bigger challenges we face around software supply chain security.
Supply chain attack mitigation
How can we protect ourselves against the next zero-day like this? Since the nature of the vulnerability is specific to the Argo CD application, there’s not an easy or generic answer, but there are a few practices you can implement at a higher level to help keep maliciously-crafted files (like the Helm chart in this case) from getting into your systems in the first place:
1. Strive for small, easy to review commits
The agile practice of making small, iterative changes isn’t just a good idea for clean development practices, it also makes it easier to spot insecure or questionable changes when they are submitted. We’re not saying that anyone at your organization would copy and paste questionable code from StackOverflow, but it would be a lot easier to catch in a code review if it isn’t buried in a thousand-line commit.
2. Treat all of your SDLC systems like production
Too many of us are guilty of treating our build servers — especially DIY ones — as toys. You know, that Jenkins, GitLab or RunDeck server running on the extra desktop in the network closet with the well-known admin password and a couple hundred plugins using a random developer’s credentials to attach to external services? Yeah… that one! (We’re not picking on these projects specifically, as any tool that is managed poorly is low-hanging fruit for an attacker.)
Historically, build and deployment servers have been treated as low-priority and not particularly hardened against attacks. The SolarWinds and CodeCov attacks, for example, are proof that these systems need to be treated just as seriously as the production servers housing your users' data.
Don’t use shared credentials.
Don’t use bespoke plugins/actions/modules that have not been vetted.
Always ensure proper, centralized RBAC is utilized to limit access appropriately.
3. Know your supply chain
Knowing the origin of the artifacts your systems are building, and what’s in them, is critical. Implementing a secure supply chain is a topic that could fill volumes. In fact, entire conferences are being held just on such topics! Here are some key areas you should pay attention to:
Don’t blindly trust container images, OS packages, code libraries — or Helm charts for that matter — that originate from sources outside of your control. Private, managed repositories with vetted and signed artifacts are critical for helping to ensure that dependencies in your applications can be trusted. A common pattern we’ve seen successfully implemented for this scenario is to have a “quarantine” repository where teams can import newly acquired artifacts for vetting by security and functional testing. Then they can be promoted to a repository that developers and build systems have general access to. This vetting, of course, needs to be balanced with a team’s need to move quickly — so the process should be tailored to the business context — in the form of guardrails. Of course, teams should be scanning these artifacts for vulnerabilities. This includes IaC configuration files whenever possible. Regular re-scans are also important in case new vulnerabilities are found. Snyk can help you scan and fix vulnerabilities and misconfigurations.
Implement a software bill of materials (SBoM) for your applications. This is a fast developing space that is, thankfully, getting a lot of attention due to the mandates laid out in a 2021 US Presidential executive order.
Another topic of interest here: signed Git commits, which would allow you to detect any code changes from outside, unverified parties. To be honest, I’ve not seen these widely used and they aren’t the silver bullet they may seem to be. Dan Lorenc, an expert in this field, does a great job discussing their pros and cons in his blog from July 2021: Should You Sign Git Commits.
It’s all about DevSecOps
The tools and practices laid out here all point back to one important fact: to secure our supply chains, it is going to take the entire team from Developer to SRE and everyone in between. The more we empower and provide developers with tools and processes that include them, rather than simply restrict or govern them, the more successful we will be at protecting our applications. By integrating tools like Snyk into the SDLC, we can enable developers to easily find security vulnerabilities in their work—before they even ship a release! This is what shift-left and DevSecOps is all about; it brings me back to that statement from Dan about the Argo CD team’s focus on security. While I have no inside knowledge about how their team operates, it's evident that their fast turnaround and transparent discussion about this vulnerability had a direct impact on their speed in addressing it.
In our 2021 State of Cloud Native Application Security Report, we found that companies who highly automate testing in their SDLC are twice as likely to implement security testing, with over 72% of them reporting an average time to fix vulnerabilities of less than a week, and 36% of them averaging one day or less! According to the Argo project documentation, they clearly are following these practices. Their quick turnaround on this issue demonstrates that.
Further reading
Here are some additional resources that dive into the topics discussed: