Releasing on Friday should not be a showstopper

It's a small change, I'll roll it out

It was 2021, Thursday evening. I've just finished some minor feature and felt there is no reason to hold back. Merged, release started, all was OK, went to bed. A couple of minutes later the unattended release failed and due to my naivety at that time I didn't have any monitoring for this. The project was down for the next 8 hours (until the morning when phones started to ring).

The Release Confidence

This never happened to me before (and after) but this failure resonated in my head for years. And it led to major improvements I had to implement within the release process. Nowadays we release on Fridays pretty often and it took us a while to get there.

The reason why we didn't want to make releases on Fridays wasn't lack of capacity or laziness. It was simply lack of release confidence. We wanted to keep an eye on the released result and expected bad things to happen. And they did. This was a clear sign of a bad release strategy, maybe even the development process.

But there are ways to solve it.

The Onion Approach

To gain our release confidence back, we had to turn lots of our processes upside down. And we iterated for a long time before we reached a state where the world would not burn on Friday afternoon.

We started with roles and release strategy definitions. I call this approach "The Onion". There are layers we can peel off with every new feature release. Here's how each layer works:

Developer

At the center is the developer. Someone who turns requirements (and coffee) into something we could call a product. Assuming all developers are well behaved and educated, adopted all best practices in software design and architecture, there is one very important detail we tend to skip. Testing.

A developer should test a lot. And all the time. And all over. As Sebastian Bergmann once said

At 100% coverage the fun begins

We should aim for the highest possible coverage not only in unit tests, but in general. Because only a well-tested product will gain your self-confidence. It can be your code, new design or a physical knob. But lack of testing can ruin your efforts.

So we write tests. A lot of them.

Staging

Our staging releases are focused on code formatting, unit testing and deploying the feature to a shared environment, where some other kids can play with your new toy.

Product Owner / Project Manager

It can be a Project Owner, Project Manager, Product Manager or any other combination you can think about. There has to be someone able to check what you have just done. That person probably knows the task and also should know what's expected to be delivered. This very same person should give you first feedback.

Controlled user group

In this stage the feature is already somehow available. Let real users from your controlled group (it can be the sales department in your company, customer support or anyone else) test it and provide feedback. And listen to them — they are much closer to end-users than you (developers) are. And often they are right, even if you don't like it. If the feedback is not good, go back to the first layer and repeat.

PR

Code review is the last human checkpoint before code hits the pipeline. There is a meme somewhere on Reddit

4 of 5 developers enjoy code review

and actually that's true. The PR is another layer you can peel off — we have adopted slightly extended PRs, where the description includes links (if any) to simply test the new functionality. This group of people will give you technical insights and will roast your architecture with smiling faces. But mostly it's the best way to produce something better than you'd figure on your own. We have adopted the "at least two reviewers" rule. Only then the feature can go out.

Release Pipeline

Your new baby is finally born — it's been waiting patiently in a Git branch, and now it will be rolled out. We're using pipelines with many steps (testing, docs generation, e2e testing etc.) depending on the project. An important piece of the puzzle is to be informed when something goes wrong. It can be a Slack notification, e-mail, or pigeon. And if the pipeline fails, you're still safe as the product has not affected production yet.

But still, what if we find an issue on production? Sure, it happens. Then you must have a robust rollback strategy — a way to simply and quickly jump back to the previous version. Hard to do right (especially in embedded or desktop software) but it should be there, and you have to think about it since the very beginning.

---

So, should you release on Friday?

It's Friday, let's release something. Or think about your release architecture — what's blocking the actual continuity in your CI/CD pipeline? Gaining your release confidence back is not that hard.