Continuous dependency updates: Improving processes by front-loading pain

This is a story of bringing the pain forward, begging forgiveness, and continuous improvement. In the early days of Manifold — long before we joined Snyk — we were building an independent marketplace for developer services (like databases or transactional email senders). The structure of our code was typical: we had a React frontend app, and a collection of Go microservices talking to a database. A typical structure meant we had typical problems, too. One such problem was keeping our dependencies up to date. As a senior member of the team, and someone who had been with the company long enough to have caused many of our problems, I set out to fix our dependency issues, with the help of some willing supporters.

“Updating dependencies is a lot of work!”

Over the years, and across languages and frameworks, complaining about updating dependencies is a constant. Unless you need an update to build a new feature, or fix a new bug (and often still with those) the work is toilsome, and has little reward. How do you even know there’s an update? Are the changes useful for you, or will they break compilation, or introduce bugs? How many conflicts will your change have with other changes, as you update function signatures across countless files?

If the work is toilsome, and the reward is small, it’s natural to put it off, or ignore it entirely.

“It will take two sprints to update our React router.”

Eventually, you’ll find yourself in a children’s fable style situation. You’ve put off updating dependencies all summer, and now winter has arrived, and you need one small update to the grasshopper-food package, but its own tree of dependency updates conflicts so much with your code base, you’re not sure how long it will take.

At Manifold, the most pointed instance of this was upgrading versions for the router we used with React. We didn’t keep our dependencies up to date in our frontend code base. When we wanted to move to a new major version of a router library, the minor API signature changes were entangled with countless other conflicts and incompatibilities across dependencies. Other packages had updated multiple times since we started using them, and when we went to first do the upgrade. Untangling the other dependencies from the router upgrade was laborious, particularly when trying to find smaller incremental releases to upgrade to, in an effort to break down the work.

The upgrade looked to be too much work at first, so we put it off again. The longer we waited without changing our habits, the more the problem compounded as more dependencies updated. The longer we waited, the more time it would take to update the router version.

“Bring the pain forward.”

Dependency updates are painful, but they are an ideal candidate to “bring the pain forward” — updating dependencies more often and explicitly. Dependencies update themselves over time, so the more often you update them, the smaller each change is, reducing the overall effort (in a way, this is classic continuous integration). Smaller changes make the problem more definable; instead of saying “dependency updates are hard”, you can now say “updating package X to version Y needs changes in these 12 files”. Exposing yourself to the remaining toil will lead you to reducing the toil through automation or other means.

We turned on automated dependency updates across all of our repositories at Manifold. With all the tooling available (like Snyk!) implementing this change was not difficult, and something a single individual could do on their own, and make a meaningful impact. We could make this change without “asking permission” from the repository owners, but instead “begging forgiveness” — no consensus was needed, and no formal decision had to be made. The implementor weighted the cost of doing the change (low effort) against the positive impact it could have (high), and the difficulty to reverse (low, just toggle the switch).

The implementor made the changes in a number of our repositories, and let the other developers know about it, the reason for doing it, and the benefit it would have. They followed up by welcoming anyone to ask for help reverting the change, or do it themselves, if they didn’t want dependency updates.

Gradually, other developers opted-in the rest of our repositories for automated dependency updates, and daily, the updates started rolling in.

“We are spending too much time reviewing these automatic dependency pull requests.”

It feels strange at first, to have something that used to be infrequent and destabilizing and scary happen daily. Eventually, everyone adjusts to it, and becomes comfortable with the original problem (updating dependencies) no longer being a problem. When you hit this point, sentiment shifts, and the next most painful part of updating dependencies becomes the new Big Problem.

Doing dependency updates daily, even with an automated process to create pull requests, still results in a lot of toil. This became our new big problem at Manifold: developers felt they spent too much time reviewing and approving dependency updating pull requests. This led to three different behaviours:

• Ignore the pull requests, and let them rot in the review queue.

• Complain loudly about the situation, either asking to turn off the automated dependency updates, or have it fixed in whatever way possible.

• Continue (lightly) reviewing the pull requests, approving, and merging them.

The second behaviour (complaining loudly) was the fuel for continuous improvement of our dependency updating process.

Anyone who had just implemented an improvement to the process likely had some ideas of what to do next, and was equally likely to be happy with how things were. Other developers were detached from the change, and had a better chance of seeing the most immediate point of pain. The developers that spoke up about it rather than ignoring or accepting the issue kept us moving forward (even if it can sting to hear that your new process improvement isn’t good enough).

As we went along, we strove to keep the changes and improvements incremental, easy to reverse, and offered them with humility. When the voices complaining about dealing with a high load of dependency updating pull requests were loud enough, the developer who implemented the improvement reminded everyone that they were welcome to turn off the automated updates, but asked them for some time to implement relief for their pain instead. The owners of the code base were always in control, but they trusted that improvements would be made; they’d seen it happen before!

Rather than disabling or decreasing the frequency of automated dependency updates, we provided a mechanism to auto-approve and merge them, provided tests were passing.

“Automatic dependency changes are causing a backlog in our continuous deployment pipeline.”

With enough cycles of improvement, the pain you’re feeling looks nothing like the pain you first moved forward. You’ll move into having the “good kind of problem” — something that is a problem only due to your success. Problems of this kind may take more effort to solve, but often provide benefit to a bigger area.

Once we were auto-merging dependency update pull requests, the more frequently updated services at Manifold hit the next problem. We already had a GitOps pipeline that would move changes from PR to staging to production, batching changes to staging, and deploying the same set to production. As any deployment to production had to be approved by a service owner, and we didn’t modify the change set applied to staging after it landed there, developer changes would end up stuck behind a backlog of automated dependency pull requests.

Really, this was not only a problem with automated dependency changes. It was an early sign that a high volume of changes would clog our deployment pipeline, and it prompted developers to improve their build and test times, and prompted our platform team to decrease the time it took for the deployer to detect and apply changes.

For dependency pull requests, we implemented logic in our deployment pipeline to automatically approve their promotion to production, and to automatically merge and apply approved changes (having the nice side effect of letting developers do one less step of every other change). Moving to a process with no human intervention for some classes of change meant we had to put a lot of faith in our tests, either improving what was there or explicitly admitting they were as good as anything a human would do.

Continuous improvement works

What started with a desire to prevent dependency updates as major sprint line items gave us that and much more: faster deploy times, a fully automated path from pull request to production, and an explicit agreement on the efficacy of our tests.

The pain of dependency updates became mostly forgotten, replaced by our larger deployment pipeline. As good as that pipeline was, we weren’t satisfied with it, and we never would have been; that’s how continuous improvement works!