Faster & improved tests for JavaScript lockfile based projects

For the past few months, we have been working hard to improve our lockfile support both in the CLI and the SCM integration. The new functionality already exists in the CLI and it’s currently being gradually released on the web and it will be soon enabled by default for all of the organizations.

Many Node.js projects rely on either yarn.lock or package-lock.json to help developers get repeatable installs and help synchronise environments for better collaborative development experience.

There is no doubt lockfiles are extremely useful – their usage has exploded. We have been working on improving our lockfile support for all users in order to bring even more accurate test results and much faster performance than ever before.

Why did we need better lockfile support in the CLI?

Up until recently, we have relied on traversing thenode_modules in order to pick up all dependencies that were installed on disk. This has proven to be rather slow and sometimes inaccurate asnode_modules often harboured long removed dependencies that are not actually used by the project any longer. While we all try our best to keep everything up to date, it is simply not practical to always delete the node_modules folder and install from scratch to ensure only the packages actually used in the project are present.

We dreamed of a world where the user doesn’t need to install their packages in order for us to run a test. After some planning, we created a shiny newnode-lockfile-parser lib that is able to traverse the lockfile itself and the package.json file instead of the entirenode_modules folder. As a result, we have a much better performance and accuracy available today.

If ayarn.lock orpackage-json.lock file is present in a project, we will auto detect these and process the project as lockfile based. For all projects with no lockfiles, the same old support is present as before.

Note:

• snyk patch andwizard will still require thenode_modules folder to be present, as we have to apply patches directly into the installed packages folders.

• Yarn projects on Node versions below 6 will also fall back tonode_module traversal. This is due to a Yarn lib, used to parse the lockfile, not supporting Node versions less than 6

How does this affect my web projects?

When importing a project via Github, GitLab or Bitbucket, we have previously relied on just thepackage.json file to determine what dependencies the project has. This meant that we had to make assumptions about the exact version of each package installed disk, and we always assumed the best (aka the latest version of each package that respects the semver range defined in thepackage.json file).

With the additional introduction of lockfile support in the mix, we are now able to give more accurate test results. A Snyk test today will build a tree based on the exact resolved versions of every package in the lockfile, and this means that we may pick up previously unreported vulnerabilities during the next Snyk test. Don’t worry though – if the autofix PR/MR setting is enabled, this will be upgraded as usual and you will receive notifications about other ways to handle these reported vulnerabilities.

Will Snyk re-generate the lockfiles for my project?

Not yet. We are working hard on providing this functionality for Yarn and npm projects. This is not an easy task due to massive differences in both Yarn and npm internal package resolution logic and lockfile differences.

Until then, all Pull Requests will require some manual intervention just like today, so any projects with yarn.lock and package-lock.json files will require a lockfile to be updated before merging the Pull Request if the package.json has changed.