Improving developer experience with security tools at Pinterest

Using open source libraries securely is an ongoing priority at large organizations. One big challenge is integrating security tools into the developer workflow — and setting up a system that prioritizes vulnerability fixes — without overwhelming developers. But what does a successful approach look like?

Our very own Simon Maple (Field CTO, Snyk) talked with Kalpesh Dharwadkar (Product Security Engineer, Pinterest) to learn how Pinterest uses Snyk to build developer-friendly security practices.

Three big priorities: visibility, scanning, and triage

Pinterest uses a lot of open source software in their development stack, so a vulnerable open source library can definitely affect their executive profile. Before Kalpesh joined the company, they used an ad hoc system of building views into their open source libraries using NPM audit. But when Kalpesh joined, he wanted to set up a centralized system for getting visibility across all open source libraries in use. His team evaluated numerous solutions and chose Snyk for two main reasons: its developer-friendly features, and support for language-specific repos such as Bazel (their chosen build tool). Kalpesh described his top priorities for security early in the conversation:

For security of open source at Pinterest, there are three main things we focus on: getting visibility into the vulnerable libraries, tooling across the stack in addition to scanning for fixes, and finally, triaging the vulnerabilities.

Kalpesh explained that they use Jenkins for builds. They scan the build system using the Snyk CLI, then upload the results into the Snyk web UI. Their first goal is to gain visibility into all the dependencies in their code repositories, and prioritize fixing any vulnerabilities. So they looked into the Common Vulnerability Scoring System (CVSS), which provides visibility into vulnerabilities. If an exploit exists, they use information from Snyk to determine whether it’s a high-priority fix.

Adding security testing throughout the pipeline

Next, Simon asked about testing throughout the development pipeline, and what challenges come up when teaching developers to use Snyk. Kalpesh has a simple solution for this step. When he first started introducing Snyk into the developer pipeline, instead of exposing the Snyk console to his developers, he had the scan run as a step in the Jenkins build. He then started addressing some of the “low-hanging fruit” — the simple fixes — himself and had his work reviewed by tech leads or project owners. This got them interested in applying fixes with Snyk tools and made for more widespread buy-in.

Triaging efficiently, prioritizing fixes, and wrangling Log4Shell

The discussion turned to triaging vulnerabilities. Simon asked, “what kind of signals or red flags do you look for in your backlog to be able to tell developers ‘these are the top five vulnerabilities’ that they should be looking at?” Kalpesh’s solution is to consider the severity of the vulnerability, whether there’s an active exploit for it, and whether the vulnerability is in a service that’s exposed to the internet. These are the parameters he uses to set the priority of fixes. Then, a ticket is created for a developer or service owner to fix the vulnerability. Tracking information in the ticket helps his team understand if their assessment of the vulnerability is in line with what developers think is important.

The developer gets assigned to that ticket. He may come back to us saying, ‘hey, we don’t use this particular feature that is vulnerable to the dependency.’ In that case, we try to reduce the priority of the feature.

As the conversation moved to Log4Shell, Simon asked Kalpesh how Pinterest handles a big zero-day vulnerability. Kalpesh recalled that the morning after Log4Shell was announced, his team declared an incident to explore how many services were affected. A JVM flag (or workaround) was available, which they applied to their Java services, but they had to depend on the service owners to deploy all of the services — which takes time. Deploying the workaround took his team into the weekend, because they wanted to ensure that all of the services had deployed with the JVM flag. They established two streams of work: one to detect all of the Java services throughout Pinterest, and the other to detect any services that the JVM flag workaround wouldn’t cover. For services where the JVM flag wasn’t sufficient, an update was the only way to mitigate the threat. Kalpesh added that an important lesson from Log4Shell was to have a single dashboard showing everything running in production.

Automating scans to help development keep moving forward

Returning to how Pinterest uses Snyk, Simon asked how much Snyk has reduced Pinterest’s  manual efforts into developer self-service, and how much visibility they have into developer pipelines.

At Pinterest, we use language-specific mono repos. Once you add a mono repo to Snyk, any new project that gets created in that repo gets automatically added. So there’s no additional work that needs to be done when a new project is created in that repo. Once code is merged to the repository and the Snyk scan happens, that’s when we understand what dependencies were introduced. [When a developer creates a project in their mono repo, the scan is] transparent to the developer. They don’t even need to know that Snyk is running.

With this setup, any time a developer creates a project, that project is automatically scanned and pushed to the Snyk UI — allowing Kalpesh’s team to see everything from the top down.

Simon reflects that developers generally want to stay in their own workflow instead of going to other tools. He asked Kalpesh, “How do you have things set up so developers can stay in their pipelines?”

Kalpesh agreed that developers like to “stay in their tickets” while still knowing everything they need to know. He mentioned Snyk’s learning hub, which contains educational resources for developers. Kalpesh said having access to this curriculum has him considering exposing developers to more of the Snyk web console, or possibly adding a link to relevant information from Snyk Learn in the JIRA ticket, to give developers more context around vulnerabilities like XSS or SQL Injection. It’s a helpful way to encourage developers to build their security knowledge.

Keeping developers where they want to be

Overall, Kalpesh and his team at Pinterest value a developer-friendly workflow, with an emphasis on triage to protect developers from becoming overwhelmed. He notes that when they first integrated with Snyk, they saw a high number of vulnerabilities from their dependencies, making it important to surface only the ones that were high-priority for the business. Being able to automate scans and view results with Snyk, while letting developers stay within their tools of choice, helps Pinterest keep development running smoothly.

Developers want to know: what’s the problem, how to fix it, and what’s the priority. If you can specify those things in a developer’s ticket, you’re good to go.