Alternative Text Phil Facey | 10 September 2019

Continuous Testing – the oil for successful software delivery

numeric test paper exam with pencil

The Problem

Many businesses and software providers are still struggling to release quality code from software delivery cycles. Testing is very often still seen as an afterthought, or at best, an additional task that needs to be done – which is quite often seen as a frustration or an overhead, which can cause delays.


So how can software delivery cycles release quality code on time and cost every time? Continuous testing! 


How Continuous Testing can help

Continuous testing is performing testing right throughout the software delivery pipeline. For this to be successful, continuous testing must be an embedded practice throughout the delivery pipeline rather than an additional time boxed task. Continuous testing must use strategic combinations of manual, automation, functional and non-functional testing using the appropriate tools.

The objective is to complete all coding and testing activities by the end of the delivery pipeline so the software can be shipped. The most efficient way of ensuring the quality of every change is to perform static testing of requirements, regression testing as soon as a change is made, and manual testing of the specific change to ensure that it is fit for purpose.


Essential steps to adopting Continuous Testing – Clearly Identify Test Early, Test Often from the start”

It can be a culture shift, for some, to move from more traditional testing philosophies to a test early, test often approach. This cultural shift must be supported throughout the organisation for it to be a success. This includes keeping testers involved in all delivery pipeline communications and feedback, and embracing the testing activity which needs to be conducted, to gain confidence in the changes made.

There are many different tools and techniques to support test early, test often. Behaviour driven development (BDD) and Test Driven Development (TDD) and tools such as gherkin and cucumber. These tools allow everyone to have a clear view of the status of the deliverables – business stakeholders, product managers, developers and testers alike – to ensure that:-

    • requirements are clearly documented and communicated
    • test cases clearly defined
    • test scripts created ahead of time to enable swift testing of code.


Shift Left and Shift Right

Following on from static test analysis, tests must be run as soon as possible in the delivery pipeline. This can be achieved using shift left – with developers and testers working together for unit testing. Automation can also be used strategically throughout the shift left approach – for example automating unit tests. Automating Unit tests may require further tooling but often there are plug ins for development kits. The goal of this shift left approach is to focus on the business value being delivered to the customer. As previously mentioned, for shift left to truly work, it requires the correct tools and quite often a culture shift.

Although often overshadowed by the push towards Shift Left philosophies, Shift Right is an essential part of fully embracing devops and continuous testing. Shift right enables businesses to quickly react to user trends in Live, as well as help find any issues which may occur from faulty software. Data from the Live or Production environment will give the most realistic scenarios to optimise and fine tune test focus, as well as focus regression tests. There are several shift right techniques including A/B testing and user monitoring. As an example, by shifting right, you can identify which features are being used in production and how they are being used. From this, you can ensure your regression packs cover these features and flows. Another example is you can release a new feature to a small user group (internal or external), and understand how they use it, and how it impacts production, and make adjustments should they be needed. 


Implement Static Testing

This is the testing of user stories/requirements before any code has been written. This provides an independent, analytical view, to identify any issues before they have been coded, so they can be addressed when they have the least detrimental impact on the project. We will have an article on the costs and benefits of this approach in a future post. The goal of static testing is defect prevention – so quality is built in, rather than testing defects out – in essence, fixing defects before you have even encountered them. It has the additional benefits of leading to clearer requirements and communication, and test planning and scoping. 


When to Automate (and when not to)

Automation can often be seen as a silver bullet to improve test efficiency, and it can be – as long as it is used strategically. A common trap is to fall into ‘over-automation’ for maximum coverage of every flow – such as an attempt to completely replace manual testing. In this case, the automation tests can be very heavyweight and difficult to maintain. This would lead to inconsistent test execution results, and it becomes difficult to interpret the validity of results (false positives). Overall it may not be clear about the benefit that test automation is adding vs the overhead of implementation, maintenance and quality of results.

The answer is to automate strategically, using Unit Testing and UI automation tools. Focused automation of critical flows, using well-structured functions, to automate alongside manual testing to get the most out of both test disciplines. This will give the best balance of investment in resources and clear results delivered. Building automated tests which don’t rely on data helps, because when the data is changed, it won’t mean that you have to constantly change your tests.

Tools change and innovation can help make this process easier, so stay in touch with what is happening, or invite a consultancy to review what you’re doing and how. Their feedback may help you make some positive change.


Data and Environments

Having the correct test environments is essential for continuous testing. These should be ready on demand, with minimal wait times, otherwise this will block or slow the progress that can be made. In addition to which, the test environments should have the relevant test data available, so test teams can perform comprehensive testing. Where appropriate, the test environments should also be connected to robust test interfaces, to provide reliable test results from external services Test stubs can be used when the full test interface is not available e.g. still in development. This comes with some risk though, as you’re not testing the actual interface.

Having the right data is essential to tests producing valid results. However, still getting accurate data still can cause significant delays in projects. If the test data lacks key aspects (e.g. fields, negative scenarios, specifications) the tests are less likely to find potential issues or defects where there are weak points. Test data should mirror as closely as possible what the application is going to encounter in the live environment. Also, understanding the different data models in order to extract the right data is a specialist skill in itself. Ideally, production data is the most realistic, but data protection often restricts its use – although there are useful methods of anonymising sensitive data so it can be used for testing. In addition, if the system is not yet Live or an interface has not been developed, Live data may not be available. In these cases test harnesses or test stubs can be used to generate test data.


Test relevancy and Coverage

Quite often, especially with regression testing, organisations default to running all of the tests, all of the time. However, this wastes a significant amount of resources and time, without actually ensuring adequate test coverage of code. The most effective test cycles are targeted and test only what needs to be tested. Visual models allow a verity of paths to be explored and optimised, to minimise the number of tests to provide maximum coverage.

Test packs must be regularly reviewed to remove duplicates, prioritise, review coverage at regular intervals. Regular reviews are also required to identify candidate scenarios for automation where appropriate to optimise test efficiency.

Don’t be afraid to approach others, external to your business to get this feedback where necessary / relevant.


Continuously Measure and Improve

Continually reviewing, learning and improving is the key to the success of Continuous Testing. Being able to adapt will ensure you keep your testing activity relevant to what is happening. This can be done with rapid feedback loops, cross-team collaboration, and using meaningful data to make key decisions on how to move forward. Customer satisfaction and other key KPIs are essential for this process to work. Using a test maturity measures index (TMMi) can also identify areas pain points and areas that need improvements – as well as the areas that are working well. From this a road map of quick wins and strategy changes can be created to build a strategic plan for improving quality.

To maintain clear communication, all teams must have access to this data in a meaningful way. Many places collect data in many different ways, but what is key with data is how it is reported – it is essential there is a single version of the truth – to enable everyone to identify the insights and actions to continually improve.


Get expert help!

There is no ‘one size fits all’ solution for testing and approach to quality. The ideal solution has to be tailor-made to fit the way you deliver software, based on what you’re delivering and how. For example, if you’re a Java house and all of your developers know java, why adopt something which requires expert knowledge of C#?

It can also be beneficial to have a flexible resourcing model, so that you are only paying for the resources you need as and when you need them. This can also be useful if there are any peaks in demand, such as having to meet a regulatory requirement.


Would you like more information?

To learn more about continuous testing challenges and practices, please get in touch on the emails below or visit our Software Testing page:

[email protected]

[email protected]

DeeperThanBlue Unify Ltd

Related Articles

These might interest you

Blog, News - 31 January 2018

A guide to Device testing for Digital Solutions

13 things to consider when testing across devices When introducing a new website for one of our clients, we face Read More