Alternative Text Phil Facey | 12 September 2019 |

Software Testing: The Financial Impact of Software Defects

shallow focus photography of computer codes software testing

The financial impact from Software Defects can be avoided by focusing on defect prevention and by using continuous testing throughout the software delivery cycle. It’s the best way of avoiding high cost defects being found as part of your release cycle…

Switching your approach to continuous software testing and defect prevention requires investment (both financial and non-financial) upfront to ensure world class testing processes and procedures are in place earlier and at all stages of the software delivery cycle. Such a change often requires a culture shift from your delivery teams too.

But it is difficult to quantify the investment required without a financial amount attributed to the risk. This is partly because it can be difficult to put an actual value on the cost of a software bug found after a release has gone Live. It is very dependent on impact, customers, organisation size, available resources, ability to turn around fixes, and illustrative examples of how damaging it can be.

The impact can be direct, such as with issues caused with time and cost, but it can also be indirect potentially resulting in damage to reputation, brand and customer confidence. This all adds to the difficulty of quantifying the costs.

 

The cost of software testing too late

There has however been a lot of research focused on estimating the overall impact of defects in Live, which can help as a guide.

Over the years there have been numerous estimates on how much defects cost. Global estimates of software failures in Live are significant:
– over £1trillion in costs
– hundreds of companies affected
– billions of customers impacted
– hundreds of hours of lost time
– business reputation impacted – including share price

 

In addition to this, the direct and indirect costs of defects are growing all the time as businesses and organisations become more and more dependent on software to function. Essentially, technology has become the backbone on which strong, customer-centric, businesses are built.

Our business partner, IBM, has undertaken a considerable amount of research into this. They’ve illustrated that the cost to fix an issue found after a production release was significant, when compared to any other stage of the software delivery pipeline.

The following figures are in comparison to finding the issue after requirements analysis:

    • 6 to 7 times higher when finding the issue in the when testing during the development phase
    • 15 times greater in the QA phase of software release
    • 100 times more expensive when finding the issue in Live

graphic to show relative costs to fix software defects

The IBM research also estimates the financial impact on finding issues later in the development process, an equivalent example would be:

    • £100 to fix the issue in the requirements phase
    • £700 to fix the issue development testing phase
    • £1,500 to fix the issue in the QA phase
    • £10,000 to fix the issue once in Production

 

Clearly, it is beneficial for anyone developing software solutions to fix issues sooner rather than later. Specifically when they can be fixed with the least detrimental impact on project time, cost and quality.

 

How can this be achieved? How can we mitigate more of the risk of more costly defects being found?

The Answer?…Continuous Testing

Continuous testing is the process of testing (whether performance testing, functional testing, or automation testing for example) right throughout the software delivery pipeline.

For continuous delivery to be successful, testing activity must be an embedded practice throughout the delivery pipeline rather than an additional time-boxed task. Continuous testing must use strategic combinations of manual, automation, functional and non-functional testing using the appropriate tools.

Learn more about Continuous Testing

Testing must be something done throughout the delivery pipeline with the objective to complete all coding and testing activities by the end of the pipeline so the software can be shipped.

This is a great way to fix issues when they have the least detrimental impact on the project.

 

Learning the Hard Way

Today, even when software provides the very backbone to a business, there is a belief that a software bug simply won’t cause significant financial distress or customer disruption. However, unfortunately, there are many recent examples where Live issues have caused a considerable impact.

Software issues in Live can have many different root causes. What we have seen however, is that some of the most detrimental bugs are down to minor mistakes that were not caught, due to insufficient testing. With continuous testing it would never have been an issue.

Some major companies have suffered significant impacts of defects in Live. These are some examples.

Amazon Web Services (AWS) (2017)

In 2017, AWS went down for approximately four hours which affected countless websites. Currently, the financial cost of this is unclear. What we do know however, is that when Amazon’s site went down for 20 minutes in 2016, it is estimated the outage cost the US tech giant around £3.75 million. The more recent incident was much longer and involved more websites/services.

 

TSB (2018)

In the TSB example, an IT failure occurred when migrating to a new system. This left some customers locked out their accounts, with some reporting that they could access other customers’ confidential details. In total the IT failure cost TSB £330 million, while an estimated 80,000 customers moved to a competitor.

 

Transport For London (TFL) – Barking to Gospel Oak Line (2019)

Due to numerous delays, including some significant software issues, the late roll out of new electrified trains has meant fewer services, while those that are running have been left overcrowded throughout the summer months.

The roll out was delayed by almost a year causing additional expense in resources to deliver the project. In addition to this passengers were compensated and provided with a months’ free travel on the line as a “thank you” for putting up with months of disruption. The cost of this is an estimated £2million, to be picked up by the rolling stock provider. (Reference).

 

Transport For London (TFL) Crossrail  (2018/19)

This is a key example of software testing timelines being squeezed, with testing finding issues later in the day that went on to cause a significant financial impact.

In 2018, shortly before the opening of the main section of the line, a delay was announced due to issues found in software testing. This meant customers had to put up with the overcrowded existing service, without the additional benefits of Crossrail. At this time, the cost of the delay was estimated to be “in the region of £20 million”. Moving forward into 2019, Crossrail was hit with further delays across several areas, including software signalling systems. The most recent delays to Crossrail have been estimated at £30 million a week. (Reference).

 

Whilst it would be unreasonable to expect to test everything, many of the worst issues are simple mistakes which could be caught by a robust approach to quality. Continuous testing through the delivery pipeline significantly mitigates risks of issues occurring in a Live environment.

 

In summary

When it comes to software testing, test as early as you can, and as often as you can. This helps to de-risk the implementation of introducing software change.

As you can see, if you don’t, it can cost your business money and time, it can impact your customers, hit your profitability hard, and cause irreparable brand damage.

 

Learn more about our approach to software testing, our testing methodologies and our software testing courses.

 

by Phil Facey – check out my LinkedIn page here and say hello 👋
Related Articles

These might interest you

Blog - 05 May 2020

DeeperThanBlue Unify recognised by The Rail Innovation Group’s ‘Recognised Innovation Scheme’

We are delighted to announce that DeeperThanBlue Unify has been recognised by The Rail Innovation Group under their Recognised Innovation Read More
Blog - 10 September 2019

Continuous Testing: the oil for successful software delivery

For many businesses and software providers, testing is very often seen as an afterthought, or at best, an additional task Read More
Blog, News - 31 January 2018

A guide to Device testing for Digital Solutions

13 things to consider when testing across devices When introducing a new website for one of our clients, we face Read More

If you want to deliver world class software testing and enhance the profitability of your business, get in touch with us today.

+44 (0)114 399 2820

[email protected]

Get in touch