How Atlassian used Snyk to solve Log4Shell
Snyk recently launched a multi-day live hack series with AWS, where experts demonstrated exploits in real-time and explained how to defend against those vulnerabilities. This series helped viewers discover new ways to improve security across the application stack for AWS workloads.
As part of the series, Micah Silverman (Director of Developer Relations, Snyk) and Chris Walz (Senior Security Engineer, Atlassian) discussed Log4Shell. For those who aren’t familiar, Log4Shell was a widespread zero-day vulnerability found within the ubiquitous Log4j Java library. We have an extensive Log4j resource library at Snyk if you want to learn more about the vulnerability.
In this post, we’ll recap Walz’s experience using Snyk to detect and remediate Log4Shell at Atlassian, as well as Silverman’s more in-depth talk about the impact of Log4Shell.
How Atlassian uses Snyk to scale security
Atlassian has thousands of repositories and services across the company, so scanning each one individually isn’t practical. That’s why the Atlassian security team built an automation that runs Snyk Open Source scans whenever there are code changes in any repository.
In addition, the automation takes Snyk’s scan results and files Jira tickets for any security issues. These tickets get assigned to the developer that committed the code change and include a link to the repository plus actionable advice for fixing the issue. Once the issue is remediated, the ticket is automatically closed.
We have one Jira project that’s used company-wide for all security vulnerabilities. That helps with visibility, building automation around it, [and] making sure SLOs are being met.
Chris Walz, Senior Security Engineer at Atlassian
Solving Log4Shell at Atlassian
In December 2021, the Apache Software Foundation announced the critical vulnerability within the Log4j library that was later dubbed Log4Shell. Atlassian immediately launched a security investigation to understand whether the company was using vulnerable Log4j versions, how severe the impact was for the products that were using Log4j, and what could be done to patch Log4Shell.
This was about as bad as it gets as far as library dependency issues go. We use a lot of Java at Atlassian, so this was a serious issue, and we immediately spun up a security investigation.
Chris Walz, Senior Security Engineer at Atlassian
Walz’s team knew that Snyk could help them identify which projects were using Log4j, but their automation only ran during code changes. They needed to be sure they had up-to-date information. Since the Log4Shell vulnerability was only just added to Snyk’s Vulnerability Database, Atlassian wrote a script to quickly rescan all of their projects to detect vulnerable Log4j instances.
Once the security team was sure that all the vulnerability data was updated, they manually kicked off the second part of their automation — the Jira sync. This automatically sent 1,426 Jira tickets for Log4j vulnerabilities to all the relevant developers so that they could quickly remediate them.
These Jira tickets were created automatically without the security team having to do anything. Within 12 hours of the original announcement, everybody that was using a vulnerability version of Log4j had a ticket telling them to update it.
Chris Walz, Senior Security Engineer at Atlassian
The impact of Log4Shell
Since Log4Shell was so highly prevalent in the Java ecosystem, the key to remediating it was actually identifying where vulnerability instances of Log4j were being used. However, this isn’t simply a matter of looking at project dependencies. Snyk has found that most vulnerabilities come from transitive dependencies.
Snyk’s timely and accurate security intelligence was able to help many companies detect Log4j in their code bases, even though over 60% of instances were indirect dependencies. In fact, Snyk quickly launched an update to the Snyk CLI that was purpose-built to detect Log4j in Java binaries or JAR files.
We had to question everything because of the nature of this issue. This is the double-edged sword of open source. The great thing is that we can respond to issues very quickly, but it often takes a few days to reach a point of stability.
Micah Silverman, Director of Developer Relations, at Snyk
After analyzing the industry-wide impact, Snyk discovered that over 17,000 Java packages were impacted, and there were over 800,000 attempted attacks in just the first 72 hours. However, a large number of attackers target companies that never remediate issues after that initial window. That’s why it’s important to continuously scan for vulnerabilities, like Log4Shell, long after a patch is discovered.
Sometimes we fall into the trap of the vulnerability cycle, where we breathe a sigh of relief after there’s a way to remediate the vulnerability. But that’s actually the middle of the timeline because some percentage of companies won’t remediate in a timely fashion, and that’s when attackers go to work.
Micah Silverman, Director of Developer Relations, at Snyk
Other sessions in the series included.
- Exploiting Your Weakest Lines of Code with StackHawk
- Exploiting Vulnerabilities in Your Container Workloads with AWS
- Hacking Your Misconfigured Kubernetes with HashiCorp
Watch the full AWS live hacking session with Snyk and Atlassian here.
