Traits of a Poor IT Change Management Process

We live in a world where the laws are getting so tight that management has changed to micro-management to quantum-management to paralysis.Jane Siberry

Executive Summary

A solid change management process is the cornerstone of a stable environment. When it’s good, it will hardly be noticed. When it’s bad, people won’t stop talking about how bad it is. This article lists some traits of poor change management processes where it hurts more than it helps. The original intentions behind each issue were good, such as starting tickets well outside business hours to lessen potential customer impact, or purchasing tooling off the shelf to ease ticket volume management. The problem is that the consequences of the intentions aren’t acknowledged. The process is never revised, resulting in pain that the entire organization constantly endures.

Change Management Processes’ Responsibilities

The change management process is responsible for managing risk by:

• Gathering a realtime list of tickets and tracking them through to successful deployment
  • Conflicts should be identified as early as possible to prioritize and reschedule tickets as needed.
• Enforcing business relevant black-out dates like end of month or holiday seasons.
• Understanding the high level dependencies between systems affected by the ticket
  • The process must understand that a change to Q would affect M and P which host the X1,Y2 and Z6 applications.
• Vetting the technical and process ticket requirements
  • A change is not acceptable just because the process requirements are met. The technical details must also be discussed in-depth as they are the most critical part of the ticket.
  • The ticket should be self describing such that a technical peer reading it for the first time would be able to implement the change.
• Approving tickets built on solid foundations and rejecting all others
  • The process should quickly reject bad tickets so that the missing information can be gathered.
  • The longer a ticket sits unvalidated for correctness, the slower the process becomes.
• Conducting post implementation reviews of failed and successful tickets.
  • Determine root cause of issues and tracking remediations
  • Long term causes should be identified
  • Gathering process feedback from successful tickets
• Iteratively improving the process based on usage patterns
  • The process must be a living document grounded in reality.

Now that we understand at a high level the goals of the change management process, we can talk the traits of a poor change management process.

Trait: Friction as a Feature

In some organizations, it appears that the goal of change management is to introduce a process so rigid and with so much red tape that the apparent preference is for every ticket to be rejected. The assumption is that slow and difficult implies methodical but this could not be further from the truth.

Slow and difficult introduces frustration and corner cutting. It causes more problems than it solves, such as:

• The effort needed for ticket approval is so great that small changes become deferred indefinitely. This is the enterprise version of death by a thousand pin pricks. Individually, an organization might be ok without the change but over time these little issues snowball into higher priority problems that eventually turn into service impacts.
• The process encourages dishonesty. If there are three priority levels for tickets (‘Low’, ‘Med’ and ‘High’) and the path of least resistance is ‘Low’ then magically, most tickets will be created with this priority. The desire to not have to fully engage the process is valued over being honest with priority categorization. This results in an inaccurate categorization system. Both ‘Low’ and ‘High’ will have similar failure rates because the false ‘low’ ticket bypassed process that could have found and addressed the issue.
• Living on the edges of the process is encouraged. If proper approval requires tickets to be created three weeks before the implementation date but a ticket can also be approved within an hour via escalation, then the plan will always be to escalate. These escalations are expensive as it drags multiple levels of management into discussions where they aren’t needed. It also defeats the ability of change management to proactively track planned implementation date conflicts as tickets are opened later than they should be.

Is it a healthy environment when your employees delay needed change, fudge with priorities and constantly push the boundaries of the process?

Trait: Arcane Tools

I’ve yet to see change management tooling that was simple to understand like a wizard asking a questions in a sane and conversational manner. Instead, the trend is to buy an all-in-one bloated ticket management solution that tries to be all things to all people and lands up being nothing to anyone.

The user experience in these tools is beyond terrible. The interface is always a mess of labels and buttons where useful information is haphazardly mixed in with unused fields.

These systems require reading pages and pages of documentation and detailed training classes. If you don’t have access to a poor soul who lived it previously, then you will never cut a successful ticket on your first try. A few days after missing a critical step something will reject the ticket. Your implementation dates are jeopardy due to this wasted time. Most of these solutions have teams dedicated to manually moving tickets between various states because the user interface doesn’t properly support the workflow.

These systems are built on top of the vendors generic model of change management with their own unique assumptions. These assumptions never fit an enterprise cleanly. When deviation occurs, it introduces manual workarounds where missing information is included as document templates uploaded to the ticket as attachments. The tooling is unaware of the content of this attached data and can’t provide any user interface guidance. Doesn’t that defeat the purpose of purchasing this solution in the first place?

Trait: Non-Technical Technical Approvers

Does a non-technical change management team vet the technical aspects of the ticket? If the answer is yes, then they have been given an impossible mandate. The sheer volume of technologies precludes anyone from being an expert on them all.

In this type of process ticket approval degrades into a game between the ticket owner and the change manager. The change manager will take the role of a lie detector by analyzing the owners response confidence. The ticket owner will make anything they say sound solid, regardless of the weak technical foundation it sits on. If the change manager detects confidence then they are forced to approve. Without a technical foundation, it’s the only way this process can work.

This is the illusion of technical change management and results in fundamentally weak tickets. Weak tickets turn into production deployment failures.

Trait: Disallowing Developers in Production

There is a trend to disallow developers, who have been assigned to the project from the beginning, from deploying into production. The fear is that a developer who has access to production can insert malicious code at will. The assumption is that this risk disappears when a production-only resource performs the deployment. This is false.

What happens in enterprises with this structure is that the production deployer for the technology is also a production deployer for many other technologies. The deployers are rarely trained them all, so they become button pushers guided by the developer. How can someone who lacks expertise on technical platforms possibly tell if what they are being asked to do is malicious?

If the intention is that these production-only deployment resources are the last man standing on defence, then they must to be trained on the technologies in question. But even then, due to the nature of the job, they’ll never be good enough to understand all the implications of the requested actions and somehow reject them.

Trait: Lack of Technical Approver Accountability

The technical approver must be held accountable for the quality of approved changes. They must deeply understand the ticket’s technical characteristics. If something breaks during production deployment, they should be one of the first people being paged.

When the technical approver has no skin in the game then the quality of tickets will be low. It will get so low that tickets will be automatically approved with little review. If this is allowed to happen, there is no other stage in the process where this technical deficiency can be caught. The enterprise will be pushing low quality tickets into production. This results in a low quality environment.

Trait: Shooting the Messenger when Tickets Fail

The goal of Change management is to ensure the stability of the technologies that run the enterprise. Rolling back when something unexpected occurs is a key tool to enable that goal. If the decision to back out forces the implementor to attend numerous meetings and defend themselves to hostile crowds then they will do anything to prevent rolling back. This includes pushing ahead by ignoring the newly discovered issue or even trying to fix the new problem by making changes outside scope of the ticket. Both decisions are worse than simply rolling back the change, addressing the concern and coming back another night to re-implement.

Aborting a change should never have a negative stigma attached. The change should go through an abbreviated re-approval process. Instead, I see processes that require extensive executive meetings, resource shaming and the requirement for the new ticket to ‘start from scratch’ at the beginning of a three week process.

Trait: Late Night Start Times

The intention is to implement changes when the fewest users are consuming the platform. This is generally chosen to be at midnight (or later). Starting this late is noble in theory but sacrifices quality in practice. The resources implementing the ticket usually work daily from 9am to 5pm. Asking them to be awake and alert at midnight to begin a technical change is difficult. On a normal night, they would already be asleep.

This leads to tired resources missing critical deployment steps. They’re not at 100% and they might not even notice that something has gone wrong. Focus turns from ensuring quality to ensuring the change is complete as soon as possible as they still need to work a full 8 hours the next day. This causes them to favour completion over correctness which results in a sloppy environment.

Conclusion

The change management process touches the entire IT organization. Its inefficiencies reverberate throughout the organization impacting duration, cost and quality. When an enterprise begins beating the drum of change and determines that they are moving ‘too slowly’, the cause can generally be found in an arcane change management process that has lost its way. Attempting to move faster without substantially rethinking and re-implementing the goals of change management will be impossible.

If the goal is to be quicker in production then the process has to be better. It needs to be nimble and responsive like a Ferrari and the existing Pinto process should be shipped off to the junkyard immediately.