Application security automation for GitHub repositories with Snyk

Snyk provides a wide array of integrations and a pretty comprehensive API to enable you to deploy Snyk across the SDLC and monitor all the code your organization is developing.

Of course - this is not always simple.

At scale, ensuring Snyk is monitoring all your repositories becomes more challenging. As you grow, more code is added in the shape of new repositories. Not only that, existing repositories keep on changing. Suddenly, security management operations like adding new projects into Snyk and updating existing projects take too long and are inefficient.

The Snyk AppSec team are also happy users of Snyk and so this is a hurdle we began to experience as well. We started to investigate alternative workflows, specifically around application security automation, and quickly solved our scaling challenges with some Snyk API-based tools designed by the Tech Services team.

Shall we take a closer look?

1. Scraping GitHub

The first tool I am going to cover is snyk-api-import. This tool is extremely powerful as can be used to automatically import new repositories as projects into Snyk. It uses the Snyk API to automatically import the repositories into Snyk at a controlled and stable pace with batch importing, error handling, and automatic retries.

The tool works by reading a list of targets and importing them into Snyk. It then uses the API to check the import status.

To use the tool, we first need to generate a list of targets to import. A target consist of three parameters:

• The organization identifier - the Snyk organization to which the targets are to be imported to

• The integration identifier - the integration type to use (GitHub/Bitbucket/Docker etc.)

• The target details -  in the case of GitHub, it will be the repository name, owner and the specific  branch to scan (for information on additional parameters, check out the docs)

So, we need to write some code that scrapes all the repositories we have in GitHub and generates a target for each repository. The result - a few lines of code:

1import {Octokit} from '@octokit/rest';
2import { paginateRest } from '@octokit/plugin-paginate-rest'
3import * as fs from 'fs/promises'
4const MyOctokit = Octokit.plugin(paginateRest);
5const octokit = new MyOctokit({
6 auth: process.env['GITHUB_TOKEN']
7}) as Octokit;
8
9(async () => {
10
11 let response = await octokit.paginate("GET /orgs/{org}/repos", {
12   Org: <your GitHub org>
13 }, res =>
14   res.data.filter(repo => !repo.fork && !repo.archived).map(repo => {
15     return {
16       orgId: <>,
17       integrationId: <>,
18       target: {
19         name: repo.name,
20         owner: <Your GitHUb Org>,
21         branch: repo.default_branch ?? "master"
22       }
23   }})
24 )
25
26 console.log(`found: ${response.length} potential targets`)
27
28 await fs.writeFile('targets.json', JSON.stringify({
29   targets: response
30 }))
31})().catch(e => {
32 console.error(e)
33})

The greatest power of this snippet is how customizable it is: You can control which org each repository should go (according to your group policies) , which repositories to ignore etc. And all can be achieved with just a few lines of code!

We can now invoke snyk-api-import. Depending on the number of repositories, this might take a while but when it completes, all your code will be covered by Snyk!

2. Keeping up with changes

So we did a one time import, but how do we continuously ensure that our repositories are monitored? One option is to run the same tool every day - but that will take too long and is pretty inefficient.

Luckily, snyk-api-import provides us with a solution - it can generate a list of all the imported projects and when run again, it will only import the diff - taking much less time (usually a few minutes, depending on how many repositories were added).

3. Refreshing existing projects

But changes are not limited only to new repositories. Changes are frequently applied to existing repositories. Manifest files are sometimes added, removed or simply moved around between repositories. These changes need to be updated in the relevant Snyk-managed projects as well.

And this is where another tool comes in handy: snyk-scm-refresh.

This tool was designed to ensure that all the projects in Snyk are in sync with the relevant repositories on GitHub. The tool can detect new manifest files, remove projects for manifests that no longer exist, update projects when a repo has been renamed, and detect deleted repos. Nice!

4. Automating the entire process end-to-end with CircleCI

To use these tools, we put together a very simple CircleCI job that runs nightly and does the following:

• Generate all targets from GitHub

• Run the import tools to generate the list of already imported projects

• Run the import tool to sync new repositories

• Run scm refresh tool to sync existing repositories

The next challenge we are planning to solve: container registry. A similar problem, but a bit more complicated!

The power of automation

A robust API is critical for enabling the extension of security tooling to fit the specific needs of your organization. In this case, Snyk’s API enabled us to add some automation that helped us overcome a pretty complex challenge we were facing - how to ensure all our code repositories were indeed covered by Snyk. Automating the process of importing and updating projects not only ensured security coverage but also did this in an efficient way, saving us time that was otherwise spent on more manual steps.

Perhaps more importantly though, the API helped us to extend Snyk to fit into our existing workflows. This flexibility is what helped us save valuable time that otherwise would have been spent on figuring out a more complicated - and resource-consuming - workaround.