It is not the best moment of the week, when after long hours of preparations and intense work during an implementation window, your change fails. Moreover, it breaks other things as well. Sometimes, it is evident right after implementation, but sometimes you find out hours or days later. If you are fortunate, you can apply an emergency change to fix a pretty apparent root cause, e.g. missing one of the items on the implementation checklist. However, in many cases it will not be possible and you will need to back out the change.
A key to a successful backout is to have a plan. Yes, a backout plan, the thing that is often overlooked. After all, you want your backout to be an honorable surrender, not a panic escape. In order to limit damage to the business, and your reputation, you need to stay in control of the situation. To do that, the team of engineers need to know what to do and the Service Desk needs to keep the business informed.
A backout plan is intended to keep you in control. It is your insurance policy against Murphy’s Law. Let’s be honest with ourselves: we do not insure everything. Do not prepare a formal backout plan for every change. Just make sure the team can verbally describe how to go back in case things get messy.
You do need a more formal plan for more complex changes, though. Creating such a plan is one of the least favorable activities of many technical people. That is why the Change Manager should be accountable for getting it done. It should be included with the rest of change documentation, ready to be used if necessary.
A good backout plan should include:
- low-level, technical instructions,
- specific communication instructions, with contact names.
The list of technical instructions is created by reversing the order of activities from your implementation plan and describing how to back out from each of the executed steps. It may be relatively straightforward if majority of the work could be achieved by restoring the most recent backup. Consider a sample backout plan for such a scenario:
- Notify the Service Desk about backout plan initiation. (Call them, send an email or raise a ticket – state it specifically.)
- Disable user access to the system. (How? List the actions.)
- Restore backup taken before the change implementation. (List the actions needed.)
- Conduct system health checks.(List them all.)
- Enable user access.
- Notify the Service Desk of successful backout.
Often the plan will be more complex than it would seem. There might be many more restoration steps, involving various databases, file systems and other areas of the IT infrastructure. The basic template still applies. It needs to be detailed and tailored to every organization and every change. Needless to say, every action should have an owner, so make sure it is clear who does what.
Communicating with the Service Desk is very important. Communication in general needs to be part of the plan to maintain control over the situation. Moreover, the business needs to know IT is in control. The Service Desk should take care of projecting the image of control towards the business. They can do it by issuing regular communication if business impact is severe enough. They will also take calls from dissatisfied users and inform them about the resolution status.
A backout plan is your insurance policy. It is up to you to have it or not. It is recommended to have it for every complex change, because business continuity and IT credibility are at stake. Start by preparing such a plan for the most complex change you have coming up in your pipeline. Then build on that and over time you will have it ready for all high-risk changes.
AUTOPOST by BEDEWY VISIT GAHZLY
