Hotfix Pipelines

Vinay Kanamarlapudi
6 min readMay 14, 2022

Ladies and gentlemen, on behalf of the crew I ask that you please direct your attention to the monitors above as we review the emergency procedures. There are six emergency exits on this aircraft. Should the cabin experience sudden pressure loss, stay calm and listen for instructions from the cabin crew. Oxygen masks will drop down from above your seat. Place the mask over your mouth and nose, like this. Pull the strap to tighten it. In the unlikely event of an emergency landing and evacuation, leave your carry-on items behind. Life rafts are located below your seats and emergency lighting will lead you to your closest exit and slide

Photo by ismail mohamed - SoviLe on Unsplash , content source

Does the above sound familiar?

Countless flights have safely delivered an uncountable number of people, but we continue to hear the above emergency protocols every time we board the plane. This is to alert passengers to the possibility of a catastrophic event and to provide instructions on how to preserve one’s life. The similar principle can be applied to Software in Production. Even if it has been running flawlessly for several days, it is possible that software in production could start to exhibit flaws in a specific workflow or that a fresh deployment will have a vital test fail but still make it to production. These production failures might be blocking customers to use the application or the application might be down and unavailable for usage.It’s critical to recognise that these scenarios are realistic and inevitable. When this happens, the software team gets together to deliver a quick fix termed as “hotfix” to resolve the issue and unblock customers.

A hotfix is a patch, bandaid, or temporary fix that stops the leak and restores normal operation of the production software. To avoid losing customer satisfaction and trust, the expected resolution time to deploy a hotfix to production is usually quite short.These are typically stressful situations, therefore it’s critical to figure out how to deliver a hotfix to production environment ahead of the time and build a Hotfix CI/CD pipeline in-addition to the main CI/CD pipeline

The main CI/CD pipeline normally includes a set of processes for building code from various development branches and moving it from lower environments to production environments. This also necessitates following all relevant procedures, such as doing regression tests, functional tests, and security tests, among others. Any development code follows this process.

Hotfix pipeline contains series of steps to build code from a hotfix branch and deploy to production environment. It is designed to be quick in execution skipping the normal steps of testing the code in lower environments etc. A sample hotfix pipeline will look like the below.

Sample hotfix pipeline

Defining a hotfix procedure depends on the current branching strategy and deployment model followed by team. Let’s take a closer look to understand how to design hotfix pipelines with some examples.

Model1:

Model1 CI/CD model

In this model, the team will initially start branching out their feature branches or bug-fix branches from develop branch. Whenever there is a checkin to develop branch, the CI/CD pipeline will build the code in the develop branch and deploy it to the stage environment. Once the sprint or iteration is over, the build in stage environment is promoted to the production environment. Subsequently, merge the develop branch to master, making the develop and master branches identically at that moment. The work for next sprint or iteration will kickoff and the team will keep check-in to develop branch as usual and so on.

In this branching strategy model, a hotfix pipeline can be designed as below

Hotfix pipeline for Model1
  • Create a branch, say hotfix_xxx from master(master branch has always code that can build identical to the latest production build)

Note: hotfix_xxx is not branched out from develop because there might be potentially new code already checked-in for ongoing features/iterations. Hence it is recommended to branch out from master in this model

  • Once a PR is raised against master, an ephemeral environment can be provisioned identically to production environment. An ephemeral environment means a transient/temporary environment which can be destroyed post hotfix deployment
  • Deploy the hotfix_xxx branch based build onto the ephemeral environment created in the previous step
  • Run the smoke test or if any pre-defined hotfix regression test
  • If all goes well , promote the build in ephemeral environment to production
  • Once hotfix is deployed to production, merge the hotfix code to master branch and delete the ephemeral environment
  • (Optional step) Before promoting to production, manual test the hotfix.

Post deployment procedure in the above strategy is to merge back hotfix code manually to develop branch to have things deployed as part of hotfix in the next deployment too.

Example2:

Model2 CI/CD model

In this model, the team will start branching out their feature branches or bug-fix branches from master branch. Whenever there is a check-in to master branch, the CI/CD pipeline will build the code in the master branch and deploy it to the stage environment. Once the sprint is over, the build in stage environment is promoted to the production environment. The work for the next sprint or iteration will kickoff and the team will keep check-in to master branch as usual and so on. In this model, the code in the master branch is tagged with release names to identify the commit that generated production releases.

In this branching strategy, a hotfix pipeline can be designed as below

Hotfix pipeline for Model2
  • Create a branch say hotfix_xxx from the latest production release tag in master. Eg:- git checkout -b newbranch v3.0 (v3.0 tag has always code that can build identical to the current production build)

Note: hotfix_xxx is not branched out from master because there might be potentially new code already written for on-going features/iterations. Hence it is recommended to branch out from master latest commit i.e. v3.0 in this model

  • Once a PR is raised against master, an ephemeral environment can be provisioned identical to production environment.
  • Deploy the hotfix_xxx branch based build onto the ephemeral environment created in the previous step
  • Run the smoke test or if any pre-defined hotfix regression test
  • If all goes well, promote the build in ephemeral environment to production
  • Once hotfix is deployed to production, delete the ephemeral environment
  • (Optional step) Before promoting to production, manual test the hotfix.

Post deployment procedure in the above strategy is to merge hotfix code manually to master branch(resolving any conflicts) to have things deployed as part of hotfix in the next deployment too.

Closing thoughts: The engineering maturity will improve with the implementation of hotfix pipelines. Hotfix pipeline should only be used in demanding situations, so defining what qualifies as a hotfix is beneficial. If the team often uses the hotfix pipeline, it’s essential to revisit the code review and testing process of application to identify and fill holes in order to limit usage.

If you would like to checkout my another article on a related CI/CD topic, visit this link

If you like my content give me a clap👏

You can also follow me on LinkedIn or Medium

--

--

Vinay Kanamarlapudi

Software Engineering Leader. I’m passionate about leadership, building products, platforms, teams, investing, learning new things.