npm Shrinkwrap reloaded: Locking npm Deps with Package-Lock and Yarn.Lock

In 2016, dependency management made headlines around the world when an unknown developer unpublished a tiny Node.js package called left-pad — which broke thousands of projects that depended on that package, and brought down some of the world’s biggest websites. npm, Inc. had to step in and restore the package to put an end to the chaos.

This story illustrates how significant dependencies can be. If they’re not there, or if they’re different than expected, big things can break.

Forget ShrinkWrap: Elegant dependency locking for Node

Locking or “pinning” dependencies is a widespread best practice in Ruby, Python, and other ecosystems. The idea is to freeze the version of a package and its dependencies, so that when you deploy a project, the same version of each dependency is always installed, making the install reliable and predictable.

In Node.js locking was much less widespread, until recently. npm Shrinkwrap was the solution, but suffered from several issues. One issue is that the shrinkwrap.json file, which contains sensitive information about your dependencies, may be included when publishing a package. Another is that Shrinkwrap makes it more difficult to add new dependencies. There is also a security risk inherent in shrinkwrap, which is vulnerable to remote code execution attack if HTTPS URLs are not used.

The npm community rejoiced recently over a newer and more elegant solution built into npm 5: package-lock.json. Those using Yarn already had access to yarn.lock which provides built-in dependency locking for Node.We’ll discuss these two solutions in a bit.

But first, should you lock?

Pros and cons of locking (pinning) your dependencies

Many practitioners, and even the US government’s guide to software manufacturers, “Before you ship”, advise that projects should pin all their dependencies. The motivation:

• Breaking changes - new versions of dependencies may introduce breaking changes that are not supported by your code, or are incompatible with your specific implementation.

• Bugs or issues - a new version might introduce problems that package developers missed. Because you can’t test every dependency in your code every time there is an upgrade, you’re better off defaulting to a version which has been proven to work in your environment before.

• Predictability - agile development and continuous delivery is founded on the ability to automatically deploy software in a predictable manner. Package upgrades which might cause inconsistent or unpredictable behavior fly in the face of this predictability.

There are also downsides to pinning packages:

• Security - Snyk provides a tool that scans open source vulnerability, so we are keenly aware of the dangers inherent in dependencies. Our data shows that at least in the Node.js community, 76% of projects are using packages with known vulnerabilities. By locking/pinning your dependencies, you are also locking in any vulnerabilities. If a security issue is discovered and the package author issues a fix, you’ll keep using the old, vulnerable version.

• Third-party modules - if you are the author of a third-party module which is itself a dependency for other projects, pinning your dependencies will force your users to stay with the same versions. The Python packaging docs, for example, says this is over-restrictive and not considered good practice. Thanks to Dustin Ingram for pointing this out.

While the security issue is significant, most developers won’t be able to resist the benefit of predictable installs and deployment. Our recommendation — lock dependencies, but add vulnerability scanning into your build process.

If you have security testing and monitoring in place, you’ll know when a dependency gets old and vulnerabilities are discovered, and can update it at that point. In an ideal world, you would immediately update all open source components in your project when new versions are released, for maximum protection against flaws and vulnerabilities. Upgrading at least when a known vulnerability is discovered seems an acceptable second best.

Locking dependencies with package-lock

package-lock.json is a new feature in npm 5 which describes the exact dependency tree that was generated in previous installs. This makes it possible to generate an identical dependency tree in subsequent installs, regardless of intermediate dependency updates.

npm 5 automatically creates the package-lock file, and it should be checked into source control. Although checking into source control can be a pain, there are two major benefits:

• Once you commit package-lock, you have an additional security layer. You can use SHA integrity values to confirm that what you install is exactly what you were expecting to install.

• Committing package-lock gives you a faster install process, because npm does not need to resolve the metadata for previously installed packages.

The package-lock.json format is identical to the format of npm-shrinkwrap.json, and if npm-shrinkwrap.json is present, package-lock.json is completely ignored.

Here are some of the variables in the package-lock.json file:

• name - package-lock.json defines a lock for a specific package, here you define the name of the package.

• version - the version you would like to lock in.

• lockfileVersion - versioning for the package-lock file, starting at 1.

• packageIntegrity - an integrity value created from package.json. Can be produced by modules like ssri.

• preserveSymlinks - indicates whether install was done with environment variable NODE_PRESERVE_SYMLINKS.

• dependencies - a mapping of the package name to a dependency objects. Dependency objects have the following properties:

  • version - a unique identifier for this package which can be used to fetch a new package of it. See the package-lock documentation for details on how to specify version for different package sources.

  • bundled - indicates if the dependency is bundled, and if so, whether it should be extracted from the parent module, rather than installed as a separate dependency.

  • dev - indicates if this dependency is a devdependency, or a transitive dependency of one.

  • Other properties are integrity, resolved, optional, and dependencies (sub-dependencies of this dependency).

Head over to the package-lock.json documentation. to learn more, and check out Jiří Pospíšil‘s post for some best practices on locking files in npm 5.

Locking dependencies with yarn.lock

Yarn comes with dependency locking built in, similar to Ruby. In order to get consistent installs across machines, Yarn needs information about exactly which versions of each dependency were installed.

Yarn provides an auto-generated yarn.lock file which lives in the root of your project. Like npm’s package-lock, yarn.lock is intended to be checked into source control.

The file looks like this (example taken from yarn.Lock documentation).

1       # THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. 
2       # yarn lockfile v1 
3       package-1@^1.0.0: 
4           version "1.0.3" 
5           resolved "https://registry.npmjs.org/package-1/-/package-1-1.0.3.tgz#a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0" 
6       package-2@^2.0.0: 
7           version "2.0.1" 
8           resolved "https://registry.npmjs.org/package-2/-/package-2-2.0.1.tgz#a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0" 
9           dependencies: 
10               package-4 "^4.0.0" 
11       ...

When you add, upgrade or remove dependencies, Yarn automatically updates the yarn.lock file. You should not edit it directly. Yarn only looks at the top-level yarn.lock file and ignores yarn.lock files within dependencies. The top-level file includes everything needed to lock versions of all packages in the dependency tree.

For more details see the yarn.lock documentation, or this extensive guide from Pluralsight explaining Yarn, including the yarn.lock feature.

Conclusion

Dependencies are a big deal. To avoid surprises, and because each time you install a package you might get different versions of its nested dependencies, most front end developers are locking or pinning their dependencies. npm users were left behind — but no longer, with npm 5’s new package-lock feature.

There are now at least two convenient ways to lock packages in npm — the package-lock.json file in npm5, which replaces shrinkwrap as a more elegant solution, and the built-in dependency locking mechanism in Yarn, known as yarn.lock.

Our advice — lock away, but don’t forget about your dependencies. Set up a process to consistently monitor vulnerabilities in your dependencies using a tool like Snyk. And from time to time, go back and revisit your dependency tree and actively update old dependencies, to afford maximum protection from flaws and future vulnerabilities.