Engineering is somewhat like basketball

Today I want to focus on the engineering team goal of getting things shipped, and show what helps us achieve this at Snyk. There are several practices we observe in our development cycle that bind well and keep us shipping all the time. I’ll explain the philosophy behind our approach and show the Continuous Delivery practices we use.

Always be shipping

Shipping means making an impact on your users, and good engineering organisations are always shipping. You may call it deploying, releasing, pushing to production, but it all means the same thing. Cool technology, great processes and empowering teamwork help meet this goal.

Shipping a release once every few months is common in larger organisations. It makes me think of soccer matches: there are only a few goals in a soccer match, with a lot of back-and-forth movement across the field. A lot of energy is put into each maneuver, and only a few succeed.

A good analogy for Continuous Delivery is basketball: once the ball is in the hands of your team, you have 24 seconds to shoot and score. This repeats so often during the game that it becomes the norm. You are always shooting for the basket, planning only the current move that simply cannot take too long. The time constraint is critical here and contributes much to the nature of the game.

Ship all the things!

So how do you make shipping a habit? Would it really work if you came into the office today and proclaimed “Let’s ship everything within 2 hours!”? I doubt it.

The philosophy can be boiled down to just a few axioms:

1. Tackle obstacles - procrastinating the hard parts is never a good choice. It only provides incentives to ship later rather than sooner. Shift everything that is hard about releases to start as soon as possible in the process. In basketball terms, when you face a guard, don’t go back to deal with the block later.

2. Eyes on the basket - a feature is just the means to deliver more value to your users, not the target. Understand the requirement and come up with the best value-to-cost ratio for your feature. In basketball terms, when the ball is in your hands, look up to the basket and do the straightforward thing.

3. Fail fast - every feature has its risks: don’t seek a fool-proof plan. If something is not going to work, make sure you discover it soon and start again. In basketball terms, it is better to shoot for the basket and miss rather than having the referee blow the whistle after 24 seconds.

Good merges are no merges

What are those hard things in shipping that become harder when not tackled ASAP?

One such pain-point is merging your changes with others’. Merges can be hell. Everything we like to talk about when we say ‘joint code ownership’ goes out the window when I’m looking at 10 files with tens of conflicts each. Think about that basketball player - running down the field to the basket, she now needs to put the ball aside, grab a shovel and dig through a pile of… dirt. The energy of the game is drained, and the next time she gets the ball, she’ll look for the shovel instead the basket.

One way to avoid this is strict division of responsibility, so that during a sprint people don’t work on the same code as their peers. While this technique might work in a controlled environment, that’s not the environment we work in. Strict divisions like this go against joint ownership and contribute to a silo-ed approach to engineering. This does not empower the team, which is bad for personal growth. Best to avoid that.

Our approach is to take smaller bites. Recall the last time you had a huge merge full of conflicts to wrangle – how long were you coding before that merge? Weeks? Days? You shouldn’t be surprised that changes happened to that code during that time – people are working around you. When you are writing the first line of code on your next feature, think ahead. See the basket you are shooting for – when will you be pushing this into the feature branch? Better yet, to production? If the answer is more than half a day of work, rethink your plan. You don’t want to hear the referee’s whistle after 24 seconds.

Move faster with feature flags

At Snyk, we are working in personal branches that are opened, pushed, reviewed, merged and deployed in matter of hours. For bigger features, where progress is incremental across a few days, we use feature flags to push to production at the same pace, even if the feature is not fully baked. The underlying assumption here is that the code is the best documentation for itself, not some architectural design document. If you keep your plans to yourself in some private branch, not only are you not aiming for a short run, you are also not helping others around you to move faster. Remember the 24 seconds.

Good tests are early tests

The other pain point when releasing is testing. Tests are great, but their timing can make a huge difference. Discovering that something is not the way you thought it should be minutes before deployment is a far more daunting experience compared to the same discovery minutes after you wrote that less-than-perfect code. Think about all that context switching: looking for the cause of the bug, stopping that basketball game mid-way, etc. Not good.

While we are not strictly TDD oriented, having a comprehensive test suite that runs locally and for every pull request keeps us in shape.We focus on making tests easy to write and fast to run. This sounds simple, but requires a great deal of attention and most importantly – ownership. Kind of like tying those shoelaces before the game. If you understand it helps you score, you are going to put the effort.

Now for the practical steps: we add tests with new features, especially when fixing pesky bugs that made their way to production. Our GitHub repos are integrated with Travis CI, running the test suite on every pull request and after merges to develop and master. A successful test run on develop and master triggers an automated deployment to our dev and prod environments respectively, completing our Continuous Delivery cycle. We have a manual ‘opt out’ option not to deploy on a merge by using a secret string in the commit message, and I’m very glad it is an ‘opt out’ and not ‘opt in’ type of thing. I don’t remember the last time we used it.

Teamwork is key

What makes this all work is the agreement within the team to ship fast. Easy merges are a team effort – just like a supportive test suite. They take time, effort, and ownership. Talk about the pain points in your team’s release process and make incremental changes. Favour quick wins over complete overhauls.

Have a better analogy than my basketball one? Want to share tricks that make your team tick? Let us know on Twitter.