How LiveRamp used Snyk to remediate Log4Shell

Like any company that uses web apps or enterprise software built with Java, San Francisco-based LiveRamp was concerned that it had been infected by the Log4Shell zero-day vulnerability within Log4j — the popular open source logging library.

When Log4Shell hit in mid-December 2021, LiveRamp — a data connectivity company that integrates disparate customer data into a company’s own marketing platforms — had just completed its POC (proof of concept) trial with Snyk and was in the early stages of implementing Snyk Open Source and Snyk Container.

LiveRamp’s main motivation for deploying Snyk was to secure its CI/CD pipeline with automated vulnerability scanning. The LiveRamp development and security teams were not expecting Snyk to discover and help remediate a major zero-day exploit.

“We asked ourselves, ‘Do we want to use Snyk for this?’” explained LiveRamp Staff DevOps Engineer, John Jelinek. “Google had released its own tools to scan for exploits like Log4Shell, but we were doing the Snyk integration and the tool was designed for these situations so we decided to use the Snyk tool to locate instances of the Log4j library.”

Pinpointing Log4j instances with Snyk

As LiveRamp began to scan for Log4j using Snyk, Jelinek and his team were not sure how many repositories they needed to scan and who owned what repository. They had 1600 GitHub repositories in use. Jelinek used the Snyk API import tool to filter the GitHub repositories down to 400 Java projects that were most vulnerable to Log4Shell and needed to be imported into Snyk and scanned. He was able to import all the Java projects into Snyk within two hours.

Jelinek set up an internal Log4Shell task mitigation force and began getting Snyk reports flagging instances of Log4j in LiveRamp’s Java projects. He then gathered the Snyk reports and passed them on to the security team, who worked to remediate the vulnerability within every impacted instance of Log4j.

Jelinek and team discovered that 23% of the 400 Java projects contained the Log4Shell vulnerability and needed to be remediated.

Scanning containers for Log4j

The next step in the process was using Snyk to scan LiveRamp's containers in GCR for Log4j. This turned out to be a cumbersome process because each container had between 500-2,000 projects and contained Java binaries and multiple versions. “The number of container images with Log4j was inflated,” said Jelinek. “Many of those images needed to be archived because they’re no longer in our ecosystem,”.

While this exercise was labor-intensive, it was also valuable because it helped LiveRamp understand which containers should be prioritized and which should be deleted or archived.

The Log4Shell remediation effort

LiveRamp set up a pecking order for remediation. Front-facing applications that were exposed to clients were remediated first, followed by applications used by employees and only accessed internally.

With Snyk, LiveRamp was able to to remediate all instances of Log4Shell from its external and internal applications before the end of the year (Log4Shell was revealed on December 10). "Customer-facing apps took a week to remediate", said Jelinek, "and internal apps took maybe two weeks. Remediation for all vulnerable versions of Log4j (up to 2.17) took until the end of January."

Lessons learned for the next vulnerability

Using Snyk to scan for and remediate Log4Shell highlighted LiveRamp's need for a software bill of materials (SBOM). According to Jelinek, one challenge LiveRamp had when Log4Shell hit was knowing the libraries and dependencies each microservice was using. The lack of visibility into how LiveRamp's URLs, microservices, and repositories were connected made it harder to pinpoint where the Log4j library existed in their infrastructure and what services were vulnerable.

“The Log4Shell vulnerability made understanding our software footprint a to-do item,” said Jelinek. “There’s an ongoing effort now to get a better handle on that.” Now that LiveRamp has overcome the challenges of Log4Shell, Snyk continues to bring greater visibility into the security posture of the company’s open source dependencies and containers.