The Financial Impact of Software Defects
Financial Impact from Software Defects can be avoided by focusing on defect prevention and using continuous testing throughout the software delivery. It’s the best way of avoiding high cost defects being found as part of your release cycle…
This approach requires investment (financial and non-financial) in quality up front and, quite often, a culture shift for your delivery teams too. It is difficult to quantify the investment required without a financial amount attributed to the risk. This is partly because it can be difficult to put an actual value on the cost of a software bug found after a release has gone Live. It is very dependent on impact, customers, organisations size, resources, ability to turn around fixes, and examples of how damaging it can be. The impact can be direct (time and cost) and indirect (reputation, brand damage and customer confidence) – which adds to the difficulty of quantifying the costs.
There has been a lot of research in estimating the overall impact of defects in Live, which can help as a guide.
Over the years there have been numerous estimates on how much defects cost. Global estimates of software failures in Live are significant:
– over $1trillion in costs
– hundreds of companies affected
– billions of customers impacted
– hundreds of hours of lost time
– business reputation impacted – including share price
In addition to this, the direct and indirect costs of defects are growing all the time as businesses and organisations become more and more dependent on software to function. Essentially, technology has become the backbone on which strong, customer-centric, businesses are built.
Our business partner, IBM, has done a significant amount of research into this, in which they found the cost to fix an issue found after a production release was significant, when compared to any other stage of the software delivery pipeline. These figures are compared to finding the issue after requirements analysis:
- 6 to 7 times higher when finding the issue in the when testing during the development phase
- 15 times greater in the QA phase of software release
- 100 times more expensive when finding the issue in Live
The IBM research also estimates the financial impact on finding issues later in the development process, an equivalent example would be:
- $100 to fix the issue in the requirements phase
- $700 to fix the issue development testing phase
- $1,500 to fix the issue in the QA phase
- $10,000 to fix the issue once in Production
Clearly, it is beneficial for anyone developing software solutions to fix issues sooner rather than later – when they can be fixed with the least detrimental impact on project time, cost and quality.
How can this be achieved? How can we mitigate more of the risk of more costly defects being found? The Answer?…
Continuous testing is testing right throughout the software delivery pipeline. For continuous delivery to be successfully, testing activity must be an embedded practice throughout the delivery pipeline rather than an additional time-boxed task. Continuous testing must use strategic combinations of manual, automation, functional and non-functional testing using the appropriate tools.
Testing must be something done throughout the delivery pipeline with the objective to complete all coding and testing activities by the end of the delivery pipeline so the software can be shipped.
This is a great way to fix issues when they have the least detrimental impact on the project.
Learning the Hard Way
Today, even when software often provides the very backbone to a business, there is a belief that a software bug will not cause significant financial distress or customer disruption. However, unfortunately, there are many recent examples where Live issues have caused a significant impact.
Software issues in Live can have many different root causes however some of the most detrimental bugs are due to minor mistakes that were not caught due to insufficient testing.
Some major companies have suffered significant impacts of defects in Live. We explore some examples below:
Amazon (AWS) (2017)
In 2017, AWS went down for approximately 4 hours and affected many sites. The financial cost of this instance is unclear, however, when Amazon’s site went down for 20 minutes in 2016, it is estimated the outage cost Amazon around $3.75 million. The more recent incident was much longer and involved more websites/services.
The IT failure occurred when migrating to a new system – which left some customers locked out their accounts, and some reporting they could access other customers details. In total the IT failure cost TSB £330 million, and an estimated 80,000 customers moved to a competitor.
Transport For London (TFL) – Barking to Gospel Oak Line (2019)
Due to numerous including significant delays due to software issues, the roll out of new electrified trains has meant passengers have been left overcrowded and delayed on fewer trains through the summer. The roll out was delayed by almost a year – causing additional expense in resources to deliver the project – in addition to which to compensate passengers have been offered a “thank you” of a months free travel on the line – for putting up months of disruption – costing an estimated £2million, to be picked up by the rolling stock provider. (Reference).
Transport For London (TFL) Crossrail (2018/19)
Shows an example of software testing timelines being squeezed and testing finding issues later in the day having a significant financial impact. In 2018, shortly before the opening of the main section of the line, a delay was announced due to issues found in software testing. This meant customers were having to put up with the overcrowded existing service, without the additional benefits of cross rail. At this time, the cost due of the delay was estimated to be “in the region of £20 million”. Moving forward into 2019, Crossrail has been hit with further delays across several areas, including software signalling systems. The most recent delays to Crossrail have been estimated at £30 million a week. (Reference).
Whilst it would be unreasonable to expect to test everything, many of the worst issues are simple mistakes which could be caught by a robust approach to quality. Continuous testing through the delivery pipeline significantly mitigates risks of issues occurring in a Live environment.
Test as early as you can, as often as you can. This helps de-risk the implementation of introducing software change. As you can see, if you don’t, it can cost you, your customers, hit profitability and cause irreparable brand damage.
Would you like more information?
If you’d like to know more about defect prevention and continuous testing and how to save time and money, please visit our Software Testing page or email us direct:
DeeperThanBlue Unify Ltd