Alternative Text Phil Facey | 12 September 2019

The Financial Impact of Software Defects

shallow focus photography of computer codes software testing

Financial Impact from Software Defects can be avoided by focusing on defect prevention and using continuous testing throughout the software delivery. It’s the best way of avoiding high cost defects being found as part of your release cycle…

 

This approach requires investment (financial and non-financial) in quality up front and, quite often, a culture shift for your delivery teams too. It is difficult to quantify the investment required without a financial amount attributed to the risk. This is partly because it can be difficult to put an actual value on the cost of a software bug found after a release has gone Live. It is very dependent on impact, customers, organisations size, resources, ability to turn around fixes, and examples of how damaging it can be. The impact can be direct (time and cost) and indirect (reputation, brand damage and customer confidence) – which adds to the difficulty of quantifying the costs.

There has been a lot of research in estimating the overall impact of defects in Live, which can help as a guide.

 

Over the years there have been numerous estimates on how much defects cost. Global estimates of software failures in Live are significant:

– over $1trillion in costs

– hundreds of companies affected

– billions of customers impacted

– hundreds of hours of lost time

– business reputation impacted – including share price

 

In addition to this, the direct and indirect costs of defects are growing all the time as businesses and organisations become more and more dependent on software to function. Essentially, technology has become the backbone on which strong, customer-centric, businesses are built.

Our business partner, IBM, has done a significant amount of research into this, in which they found the cost to fix an issue found after a production release was significant, when compared to any other stage of the software delivery pipeline. These figures are compared to finding the issue after requirements analysis:

    • 6 to 7 times higher when finding the issue in the when testing during the development phase
    • 15 times greater in the QA phase of software release
    • 100 times more expensive when finding the issue in Live

 

The IBM research also estimates the financial impact on finding issues later in the development process, an equivalent example would be:

    • $100 to fix the issue in the requirements phase
    • $700 to fix the issue development testing phase
    • $1,500 to fix the issue in the QA phase
    • $10,000 to fix the issue once in Production

 

Clearly, it is beneficial for anyone developing software solutions to fix issues sooner rather than later – when they can be fixed with the least detrimental impact on project time, cost and quality.

 

How can this be achieved? How can we mitigate more of the risk of more costly defects being found? The Answer?…

 

…Continuous Testing

Continuous testing is testing right throughout the software delivery pipeline. For continuous delivery to be successfully, testing activity must be an embedded practice throughout the delivery pipeline rather than an additional time-boxed task. Continuous testing must use strategic combinations of manual, automation, functional and non-functional testing using the appropriate tools.

More information on Continuous Testing can be found in this blog.

Testing must be something done throughout the delivery pipeline with the objective to complete all coding and testing activities by the end of the delivery pipeline so the software can be shipped.

This is a great way to fix issues when they have the least detrimental impact on the project.

 

Learning the Hard Way

Today, even when software often provides the very backbone to a business, there is a belief that a software bug will not cause significant financial distress or customer disruption. However, unfortunately, there are many recent examples where Live issues have caused a significant impact.

Software issues in Live can have many different root causes however some of the most detrimental bugs are due to minor mistakes that were not caught due to insufficient testing.

Some major companies have suffered significant impacts of defects in Live. We explore some examples below:

Amazon (AWS) (2017)

In 2017, AWS went down for approximately 4 hours and affected many sites. The financial cost of this instance is unclear, however, when Amazon’s site went down for 20 minutes in 2016, it is estimated the outage cost Amazon around $3.75 million. The more recent incident was much longer and involved more websites/services.

 

TSB (2018)

The IT failure occurred when migrating to a new system – which left some customers locked out their accounts, and some reporting they could access other customers details. In total the IT failure cost TSB £330 million, and an estimated 80,000 customers moved to a competitor.

 

Transport For London (TFL) – Barking to Gospel Oak Line (2019)

Due to numerous including significant delays due to software issues, the roll out of new electrified trains has meant passengers have been left overcrowded and delayed on fewer trains through the summer. The roll out was delayed by almost a year – causing additional expense in resources to deliver the project – in addition to which to compensate passengers have been offered a “thank you” of a months free travel on the line – for putting up months of disruption – costing an estimated £2million, to be picked up by the rolling stock provider. (Reference).

 

Transport For London (TFL) Crossrail  (2018/19)

Shows an example of software testing timelines being squeezed and testing finding issues later in the day having a significant financial impact. In 2018, shortly before the opening of the main section of the line, a delay was announced due to issues found in software testing. This meant customers were having to put up with the overcrowded existing service, without the additional benefits of cross rail. At this time, the cost due of the delay was estimated to be “in the region of £20 million”. Moving forward into 2019, Crossrail has been hit with further delays across several areas, including software signalling systems. The most recent delays to Crossrail have been estimated at £30 million a week. (Reference).

 

Whilst it would be unreasonable to expect to test everything, many of the worst issues are simple mistakes which could be caught by a robust approach to quality. Continuous testing through the delivery pipeline significantly mitigates risks of issues occurring in a Live environment.

 

In summary

Test as early as you can, as often as you can. This helps de-risk the implementation of introducing software change. As you can see, if you don’t, it can cost you, your customers, hit profitability and cause irreparable brand damage.

 

Would you like more information?

If you’d like to know more about defect prevention and continuous testing and how to save time and money, please visit our Software Testing page or email us direct:

[email protected]

[email protected]

DeeperThanBlue Unify Ltd

Related Articles

These might interest you

Blog - 10 September 2019

Continuous Testing – the oil for successful software delivery

The Problem Many businesses and software providers are still struggling to release quality code from software delivery cycles. Testing is Read More
Blog, News - 31 January 2018

A guide to Device testing for Digital Solutions

13 things to consider when testing across devices When introducing a new website for one of our clients, we face Read More