Alternative Text Dave Harris | 10 September 2019 |

Continuous Testing: the oil for successful software delivery

numeric test paper exam with pencil

For many businesses and software providers, testing is very often seen as an afterthought, or at best, an additional task that needs to be done. When this is the case it is often seen simply as an overhead or frustration.

The upshot of this is delays throughout the software delivery process, meaning that those businesses and software providers struggle to release quality code from their software delivery cycles.
So how can software delivery cycles release quality code on time and cost every time? Continuous testing is your answer.


How Continuous Testing can help

Continuous testing is when testing is performed throughout the software delivery pipeline. For this to be successful, continuous software testing must be an embedded practice throughout the delivery pipeline rather than an additional time boxed task.

Continuous testing must use strategic combinations of manual testing, automation testing, functional testing and non-functional testing using the appropriate tools.

The objective is to complete all coding and testing activities by the end of the delivery pipeline so the software can be shipped. The most efficient way of ensuring the quality of every change is to perform static testing of requirements, regression testing as soon as a change is made, and manual testing of the specific change to ensure that it is fit for purpose.


Essential steps to adopt Continuous Testing – “Clearly Identify Test Early, Test Often from the start”

Embedding the requirements of continuous testing can be a culture shift, for some, that involves moving from more traditional testing philosophies to a test early, test often approach.

This cultural shift must be supported throughout the organisation for it to be a success. This includes keeping testers involved in all delivery pipeline communications and feedback and embracing the testing activity which needs to be conducted, to gain confidence in the changes made.

There are many different tools and techniques to support test early, test often. Techniques include behaviour driven development (BDD) and Test Driven Development (TDD), while tools such as gherkin and cucumber allow everyone to have a clear view of the status of the deliverables. Business stakeholders, product managers, developers and testers alike are all aligned, ensuring that:

    • requirements are clearly documented and communicated
    • test cases clearly defined
    • test scripts created ahead of time to enable swift testing of code.


Opting to Shift Left or Shift Right

Following on from static test analysis, tests must be run as soon as possible in the delivery pipeline. This can be achieved using shift left – with developers and testers working together for unit testing. Automation testing can also be used strategically throughout the shift left approach. For example by automating unit tests – this may require further tooling but often there are plug ins for development kits.

The goal of this shift left approach is to focus on the business value being delivered to the customer. As previously mentioned, for shift left to truly work, it requires the correct tools and quite often a culture shift.

Although often overshadowed by the push towards Shift Left philosophies, Shift Right is an essential part of fully embracing DevOps and continuous testing.

Shift right enables businesses to quickly react to user trends in Live, as well as help to find any issues which may occur from faulty software. Data from the Live or Production environment will give the most realistic scenarios to optimise and fine tune test focus, as well as focus regression tests.

There are several shift right techniques including A/B testing and user monitoring. As an example, by shifting right, you can identify which features are being used in production and how they are being used. From this, you can ensure your regression packs cover these features and flows.

Another example is you can release a new feature to a small user group (internal or external), to understand how they use it and how it impacts production, to make adjustments should they be needed.


Implement Static Testing

Static testing is the testing of user stories/requirements before any code has been written. It is useful to provide an independent, analytical view, to identify any issues before they have been coded, so they can be addressed when they have the least detrimental impact on the project.

We will have an article on the costs and benefits of this approach in a future post.

The goal of static testing is defect prevention – so quality is built in, rather than testing defects out. In essence, it involves fixing defects before you have even encountered them.

Ultimately, it has the additional benefits of leading to clearer requirements and communication, and test planning and scoping.


Understanding When to Automate (and when not to)

Automation can often be seen as a silver bullet to improve test efficiency, and it can be – as long as it is used strategically.

A common trap is to fall into ‘over-automation’ for maximum coverage of every flow – such as an attempt to completely replace manual testing. In this case, the automation tests can be very heavyweight and difficult to maintain. This would lead to inconsistent test execution results, where it becomes difficult to interpret the validity of results (false positives). Overall it may not be clear about the benefit that test automation is adding vs the overhead of implementation, maintenance and quality of results.

The answer is to automate strategically, using Unit Testing and UI automation tools. This involves focused automation of critical flows, using well-structured functions, to automate alongside manual testing to get the most out of both test disciplines. Automating strategically in this way will give the best balance of investment in resources and clear results delivered. Building automated tests which don’t rely on data also helps. This is because when the data is changed, it won’t mean that you have to constantly change your tests.

Tools continually change and new innovations can help make this process easier. For this reason it’s important to stay in touch with what is happening. Consider inviting a consultancy to review what you’re doing and how. Their feedback may help you to make some positive change.


Ensuring the Right Data is Available, and that Test Environments are Correct 

Having the correct test environments is essential for continuous software testing. These should be ready on demand, with minimal wait times, otherwise this will block or slow the progress that can be made.

In addition to this, the test environments should have the relevant test data available, so test teams can perform comprehensive testing. Where appropriate, the test environments should also be connected to robust test interfaces, to provide reliable test results from external services Test stubs can be used when the full test interface is not available, for example if it is still in development. This comes with some risk though, as you’re not testing the actual interface.

Having the right data is essential to ensure that your tests produce valid results. However, getting accurate data can still cause significant delays in projects. If the test data lacks key aspects (for example fields, negative scenarios, specifications) the tests are less likely to find potential issues or defects where there are weak points.

Test data should mirror as closely as possible what the application is going to encounter in the live environment. We also know that understanding the different data models in order to extract the right data is a specialist skill in itself.

Ideally, production data is the most realistic, but data protection often restricts its use – although there are useful methods of anonymising sensitive data so it can be used for testing. In addition, if the system is not yet Live or an interface has not been developed, Live data may not be available. In these cases, test harnesses or test stubs can be used to generate test data.


Striving for Test relevancy and Coverage

Quite often, especially with regression testing, organisations default to running all of the tests, all of the time. However, this wastes a significant amount of resources and time, without actually ensuring adequate test coverage of code.

The most effective test cycles are targeted and test only what needs to be tested. Visual models allow a verity of paths to be explored and optimised, to minimise the number of tests to provide maximum coverage.

Test packs must be regularly reviewed to remove duplicates, as well as prioritising, and reviewing coverage at regular intervals. Regular reviews are also required to identify candidate scenarios for automation where appropriate to optimise test efficiency.

Don’t be afraid to approach others, that may be external to your business to get this feedback where it is necessary or relevant.


Continuously Measure and Improve

Continually reviewing, learning and improving is the key to the success of continuous testing.

Having the ability to adapt to new issues and challenges will ensure that your testing activity is always kept relevant to what is happening. This can be done with rapid feedback loops, cross-team collaboration, and by using meaningful data to make key decisions on how to move forward. Customer satisfaction and other key KPIs are essential for this process to work.

Using a test maturity measures index (TMMi) can also identify pain points and areas that need improvements, as well as the areas that are working well. From this a road map of quick wins and tactical changes can be created to build a strategic plan for improving quality.

To maintain clear communication, all teams must have access to this data in a meaningful way. Many places collect data in many different ways, but what is key with data is how it is reported. It is essential that there is a single version of the truth to enable everyone to identify the insights and actions to continually improve.


Get expert help!

There is no ‘one size fits all’ solution for testing and approach to quality. The ideal approach has to be tailor-made to fit the way you deliver software, based on what you’re delivering and how.

For example, if you’re a Java house and all of your developers know java, why adopt something which requires expert knowledge of C#?

It can also be beneficial to have a flexible resourcing model, so that you are only paying for the resources you need as and when you need them. This can also be useful if there are any peaks in demand, such as having to meet a regulatory requirement.


Learn more about our approach to software testing, performance testing, and our software testing courses or email us direct.

Related Articles

These might interest you

Company News, Press Releases - 05 May 2020

DeeperThanBlue Unify recognised by The Rail Innovation Group’s ‘Recognised Innovation Scheme’

We are delighted to announce that DeeperThanBlue Unify has been recognised by The Rail Innovation Group under their Recognised Innovation Read More
Software Testing - 12 September 2019

Software Testing: The Financial Impact of Software Defects

The financial impact from Software Defects can be avoided by focusing on defect prevention and by using continuous testing throughout Read More
Software Testing - 31 January 2018

13 things to consider when device testing for digital solutions

13 Factors to Consider when Testing across Devices When introducing a new website for one of our clients, we face Read More

If you want to deliver world class software testing and enhance the profitability of your business, get in touch with us today.

+44 (0)114 399 2820

[email protected]

Get in touch