A year ago, the ThoughtWorks Technical Advisory Board suggested we try categorising and prioritising our technical debt into a Technical Debt Catalogue, so that improvements can be focused on the items that are having the biggest detriment to the team. We’ll try to show you how start out with such a catalogue, how we like to extend it, how you can make it part of your everyday process and how to present it in a way that makes it easy to get funding for the improvements.

1. Pain Points Register

Firstly, we like to call the catalogue a Pain Points Register. Using the term Technical Debt unnecessarily limits the scope to problems that are technical in nature and requires an explanation to anyone who is not familiar with the term Technical Debt. A pain points register is fairly self-explanatory and doesn’t limit the scope.

2. Capture

The next step is to survey the team you are working with to find out what their pain points are. The survey doesn’t need to be exhaustive and the pain points don’t need to be captured in great detail. You want to capture enough information that other members of the team understand what the pain point is referring to. Ideally you want to try to avoid solutions to pain points as well. For instance:

Deployments are difficult and error-prone

is much better than:

Lack of automated deployments

Retrospectives can be an excellent input to this step, but if you’re working in a company that has been burnt by cowboys masquerading as agilists, you may want to stick to a simple pain points capture meeting.

3. Problem Management

Once we have a list of pain points, we like to borrow from ITIL and treat each pain point as a problem. Viewed this way, our goal for each pain point is to determine the current workaround and root cause. When you are documenting the current workaround, make sure you capture how the team currently deals with this issue, not how we think they should deal with it. For the root cause, 5 Whys is a great way to really get to the bottom of it.

This should give us something like the following:

Problem Workaround Root Cause
Deployments are difficult and error-prone
  • A run-sheet is produced for each major release.
  • Two people perform the deployment, the first one executing the steps in the run sheet and the other checking the first one’s work.
  • Mistakes are fixed as they are detected, either during or after the deployment.
  • The same deployment approach is used for SIT, UAT, Dry-Run and Production
Deployments require lots of manual activities, with too much risk of human error
Nobody notices when the build is broken
  • Developers manually check the build status periodically throughout the day
The build server is not configured to send out notifications on build failure

From an ITIL point of view, these then become Known Errors.

4. Impact Assessment

Once we have the workaround accurately captured, we can start to estimate how much the pain points are costing us. For instance, we might estimate the cost for the deployment problem above as follows:

Workaround Item Estimated Cost Formula Estimated Cost
A run-sheet is produced for each deployment 5 days of effort per production deployment x $1000 per FTE day $5000/production deployment
Two people perform the deployment, the first one executing the steps in the run sheet and the other checking the first one’s work. 42 (2 people plus–on average–40 people in other teams waiting for the deployment to complete) x 2 hrs (each deployment takes 2.5 hrs but the feeling is that this could be reduced to 30 min) per production deployment  x $1000 per FTE day $10,500/production deployment
Mistakes are fixed as they are detected, either during or after the deployment. 30 min (based on average time to resolve deployment issues) x 42 per production deployment x $1000 per FTE day + $10,000 (based on average business cost of the deployment mistake divided by average number of mistakes per production deployment) $12,625/production deployment
Same deployment approach is used for SIT, UAT, Dry-Run and Production 42 x 2.5 x 7 (based on there being on average 7 deployments across the other environments prior to each production deployment. e.g. 3 to SIT, 3 to UAT and 1 to Dry-run) x $1000 per FTE day $91,875/production deployment
Total (based on average of 4 production deployments per year) $120,000/quarter

Don’t spend too much time trying to get a really accurate workaround cost; at this stage you are just looking for a ball-park figure and a good guess is better than a very accurate value that takes months to figure out. It is important, however, to make sure you think outside of just your team and how the workaround may be costing other teams.

Add the workaround costs to your register and you should have something like:

Problem Workaround Root Cause Cost Estimate per Quarter
Deployments are difficult and error-prone
  • A run-sheet is produced for each major release.
  • Two people perform the deployment, the first one executing the steps in the run sheet and the other checking the first one’s work.
  • Mistakes are fixed as they are detected, either during or after the deployment.
  • The same deployment approach is used for SIT, UAT, Dry-Run and Production
Deployments require lots of manual activities, with too much risk of human error $120,000
Nobody notices when the build is broken
  • Developers manually check the build status periodically throughout the day
  • Developers occasionally have to wait for another developer to fix the build before they can commit their own changes.
Developers are not receiving notifications on build failure $7750

We like to normalise to cost per quarter, as it makes it easy to compare the pain points and our target return on investment (ROI) is about 1-2 quarters.

5. Prioritisation and Selection

Prioritise your register based on the cost estimate. We then normally focus on fixing two problems from the list, the first item is the one with the highest (or one of the highest) estimated costs. The second item is a quick win–something achievable in a week, like turning on build notifications.

6. Solution Candidates

The estimated workaround cost and a target return on investment  (ROI) give you a starting budget for fixing the problems. For instance, with a target ROI of 3-6 months, we have a budget of about $120K – $240K for our deployment problem above. This budget can then be used to help determine solution candidates. Similarly, for the broken build problem, we have a budget of $7750-$15,500.

When you are looking for solutions, don’t limit yourself to technical solutions; sometimes the best answer to a problem is a simple manual process. Also, don’t limit yourself to solutions that can only be delivered internally. As technologists, we are very keen to solve our own technology problems but quite often the biggest problems require solutions that are outside of our field of expertise. For instance, for the example above, automated deployments are an obvious solution. Quite often, however, teams don’t have any great experience in this area (at least for the teams we work with). Rather than struggling to try to deliver it internally, it may be more efficient to bring someone else in, like Windy Road. :)

Getting back to the deployment above, we might estimate that it will cost $200,000 to automate our deployments with Chef, which would allow us to abandon all of the above workarounds, giving an ROI of less than 6 months. This creates a nice little business case that we can then take for approval. If the number doesn’t play to your argument, check if there are more workaround costs that you haven’t factored in or if there is a cheaper solution to the problem. If the numbers still don’t stack up (e.g., ROI is 10 years), just move onto the next pain point. You can always come back to it later, when the workaround costs have changed or new solutions present themselves.

7. Measure the Change

When you implement your solutions, measure the change and record your wins and losses. If the solution was successful, then remove the pain point from the register. If it was partially successful, create a new pain point for what is left over. For instance, turning on build notifications may have cost $125, but only saved $7000 per quarter rather than the full $7750 we estimated, because we expected to save an hour a week where a developer was waiting for another developer to fix the build. It turned out that build notification only reduced this to an hour every two weeks. On the other hand, there may be additional benefits that the solution delivered, which you should try to capture as well. For instance, automating the deployments allows test managers to self-serve the deployments into their environments, reducing the wait time to test a new version by 2 days. Across a team of testers, this can result in a huge cost saving.

We like to write up the results into a cost savings register that advertises the current and projected cost savings. The cost savings needs to take into account the achieved cost saving reduction per quarter and the implementation cost. For example, if we implemented solutions to the above problems a year ago, we would get something like:

Date Current Cost Savings Projected Cost Savings (as of Next Year)
24/01/2012 $257,875 $765,875

 

Problem Cost Saving/quarter Solution Implementation Cost Implementation Date ROI Date
Deployments are difficult and error-prone $120,000 Automated Deployments $250,000 24/01/2011 29/07/2011
Nobody notices when the build is broken $7,000 Build Status Jabber notifications $125 24/01/2011 25/01/2011

This allows us to really sell the improvements that we have achieved to our stakeholders and helps us stay focused on solving problems that will actually save money. Having the solution and the actual implementation cost is also very helpful to other teams, who may be facing similar problems; it tells them that a solution is achievable and has real cost benefits.

8. Repeat

As new problems are recognised, add them to the pain points register. As problems are solved, move them to the cost savings register.

9. Take it Outside the Team

Once you have nailed down the process and have a few wins on the board, take the pain point register outside the team. Talk to the teams that work with you and find out their pain points in dealing with your team. Feed these directly into your pain points list and analyse and estimate them just as you do for your own. If you find overlaps between the internal and external pain points, then just add their workarounds and the estimated cost for the pain points they overlap with.

If you are in the right role, you can even think about taking it out of IT (i.e., what pain points does the business have) and outside of the company (what pain points do your customers have in dealing with your company), where even bigger benefits can be gained.

Conclusion

By estimating the cost of the workaround for each pain point and using that as a budget for proposing solutions to the root cause, we are effectively creating a mini business case for fixing each pain point. Having a tangible business case makes it almost too easy to get funding approval, even in the harshest of financial climates, as this is actually when everyone is under the greatest pressure to deliver quick-win cost savings.

Hopefully the steps outlined above will give you the confidence and guidance to start on your own journey towards value-focused continuous improvement. Of course, if you want to get the journey started faster, contact our services team. We’ve got great people who can help set up your pain points and cost savings registers, facilitate meetings to capture pain points and help you analyse them. We can even help you solve your pain points, so you can realise the cost savings sooner, with a lower implementation cost. We’re passionate about continuous improvement and we would love to share that passion with you.