Software Testing: The Financial Impact of Software Defects

The financial impact from Software Defects can be avoided by focusing on defect prevention and by using continuous testing throughout the software delivery cycle. It’s the best way of avoiding high cost defects being found as part of your release cycle…

Switching your approach to continuous software testing and defect prevention requires investment (both financial and non-financial) upfront to ensure world class testing processes and procedures are in place earlier and at all stages of the software delivery cycle. Such a change often requires a culture shift from your delivery teams too.

But it is difficult to quantify the investment required without a financial amount attributed to the risk. This is partly because it can be difficult to put an actual value on the cost of a software bug found after a release has gone Live. It is very dependent on impact, customers, organisation size, available resources, ability to turn around fixes, and illustrative examples of how damaging it can be.

The impact can be direct, such as with issues caused with time and cost, but it can also be indirect potentially resulting in damage to reputation, brand and customer confidence. This all adds to the difficulty of quantifying the costs.

Learn more about our approach to software testing

The cost of software testing too late

There has however been a lot of research focused on estimating the overall impact of defects in Live, which can help as a guide.

Over the years there have been numerous estimates on how much defects cost. Global estimates of software failures in Live are significant:

• Over £1trillion in costs

• Hundreds of companies affected

• Billions of customers impacted

• Hundreds of hours of lost time

• Business reputation impacted – including share price

In addition to this, the direct and indirect costs of defects are growing all the time as businesses and organisations become more and more dependent on software to function. Essentially, technology has become the backbone on which strong, customer-centric, businesses are built.

Our business partner, IBM, has undertaken a considerable amount of research into this. They’ve illustrated that the cost to fix an issue found after a production release was significant, when compared to any other stage of the software delivery pipeline.

The following figures are in comparison to finding the issue after requirements analysis:

• 6 to 7 times

  higher when finding the issue in the when testing during the development phase

• 15 times 

  greater in the QA phase of software release

• 100 times

  more expensive when finding the issue in Live

The IBM research also estimates the financial impact on finding issues later in the development process, an equivalent example would be:

• £100 to fix the issue in the requirements phase

• £700 to fix the issue development testing phase

• £1,500 to fix the issue in the QA phase

• £10,000 to fix the issue once in Production

Clearly, it is beneficial for anyone developing software solutions to fix issues sooner rather than later. Specifically when they can be fixed with the least detrimental impact on project time, cost and quality.

How can this be achieved? How can we mitigate more of the risk of more costly defects being found?

The Answer?…Continuous Testing

Continuous testing is the process of testing (whether performance testing, functional testing, or automation testing for example) right throughout the software delivery pipeline.

For continuous delivery to be successful, testing activity must be an embedded practice throughout the delivery pipeline rather than an additional time-boxed task. Continuous testing must use strategic combinations of manual, automation, functional and non-functional testing using the appropriate tools.

Testing must be something done throughout the delivery pipeline with the objective to complete all coding and testing activities by the end of the pipeline so the software can be shipped.

This is a great way to fix issues when they have the least detrimental impact on the project.

Learning the Hard Way

Today, even when software provides the very backbone to a business, there is a belief that a software bug simply won’t cause significant financial distress or customer disruption. However, unfortunately, there are many recent examples where Live issues have caused a considerable impact.

Software issues in Live can have many different root causes. What we have seen however, is that some of the most detrimental bugs are down to minor mistakes that were not caught, due to insufficient testing. With continuous testing it would never have been an issue.

Some major companies have suffered significant impacts of defects in Live. These are some examples.

Amazon Web Services (AWS) (2017)

In 2017, AWS went down for approximately four hours which affected countless websites. Currently, the financial cost of this is unclear. What we do know however, is that when Amazon’s site went down for 20 minutes in 2016, it is estimated the outage cost the US tech giant around £3.75 million. The more recent incident was much longer and involved more websites/services.

TSB (2018)

In the TSB example, an IT failure occurred when migrating to a new system. This left some customers locked out their accounts, with some reporting that they could access other customers’ confidential details. In total the IT failure cost TSB £330 million, while an estimated 80,000 customers moved to a competitor.

Transport For London (TFL) – Barking to Gospel Oak Line (2019)

Due to numerous delays, including some significant software issues, the late roll out of new electrified trains has meant fewer services, while those that are running have been left overcrowded throughout the summer months.

The roll out was delayed by almost a year causing additional expense in resources to deliver the project. In addition to this passengers were compensated and provided with a months’ free travel on the line as a “thank you” for putting up with months of disruption. The cost of this is an estimated £2million, to be picked up by the rolling stock provider. (Reference).

Transport For London (TFL) Crossrail  (2018/19)

This is a key example of software testing timelines being squeezed, with testing finding issues later in the day that went on to cause a significant financial impact.

In 2018, shortly before the opening of the main section of the line, a delay was announced due to issues found in software testing. This meant customers had to put up with the overcrowded existing service, without the additional benefits of Crossrail. At this time, the cost of the delay was estimated to be “in the region of £20 million”. Moving forward into 2019, Crossrail was hit with further delays across several areas, including software signalling systems. The most recent delays to Crossrail have been estimated at £30 million a week. (Reference).

Whilst it would be unreasonable to expect to test everything, many of the worst issues are simple mistakes which could be caught by a robust approach to quality. Continuous testing through the delivery pipeline significantly mitigates risks of issues occurring in a Live environment.

In summary

When it comes to software testing, test as early as you can, and as often as you can. This helps to de-risk the implementation of introducing software change.

As you can see, if you don’t, it can cost your business money and time, it can impact your customers, hit your profitability hard, and cause irreparable brand damage.

• Share