DevOps is about Continuous Delivery, and the root of any Continuous activity is automation – not just the build creation, integration and deployment but the testing too. Automated test cycles optimally shorten the SDLC while increasing test quality. The increase in microservices require high-quality functional testing, which includes automated API testing.
Optimally means that that’s the plan, for DevOps to pay off without bottlenecks. Testing often remains as a big dark hole in DevOps, limited to developers inserting some simple unit testing instead of a more strategic approach that a formally trained QA person would attach. The developers are not at fault – they’re under tight deadlines and are often unable to devote time to complex testing strategies.
As testers, we can appreciate the potential for disaster when fuller coverage becomes an afterthought not an actual implementation. Feel free to pick your favorite example from personal experience where an under-tested release went out and failed, or remember a bug you caught that was based on exploratory testing instead of the test case scenario list.
To paraphrase Wikipedia throughout this paragraph, Continuous Testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Software has become a key business differentiator. Organizations expect to deliver more, and more innovative, software within shorter delivery cycles and have turned to lean approaches, such as Agile, DevOps, and Continuous Delivery, to try to speed up the SDLC. After accelerating other aspects of the delivery pipeline, teams typically find that their testing process is preventing them from achieving the expected benefits of their SDLC acceleration initiative.
This brings us to the question at hand: Is Continuous Testing achievable as conceived, or is it an impossible theoretical ideal to aim for that just looks nice to complete the graphic above? To answer this, we must check with testers, not developers, and don’t start worrying about the answer.
First, let’s look at the difference between automated testing and Continuous Testing. Thought they were the same? Then please look below:
There’s an obvious difference: one has human bottlenecks, and the other doesn’t. Continuous Testing doesn’t just manage risk by focusing on test efficiency, it also the effectiveness of our testing processes. If I test nothing, I can do so very efficiently (very fast!), but not very effectively (no coverage!).
Continuous Testing assesses business risk as its primary goal and establishes a safety net that helps the team protect the user experience. Continuous Testing also requires a stable test environment to be available on demand while integrating seamlessly into the software delivery pipeline (where it can deliver actionable feedback at each stage) and DevOps toolchain. But how do you get here? First of all, you need a CI/CD environment.
DevOps must be able to deliver software that can withstand actual production use. Testers hold the key to getting there. For testers, this is a new language and workflow with a potentially steep learning curve. Like the switch from manual testing to automated, some testers will not be able to make the leap. The ultimatum for testers is as follows:
Test suites need to be managed for maximum coverage, and test cases need to be optimized for quality and effectiveness. Advanced analytics may be needed to determine how code change impacts the requisite test scripts. Testers need to engage development immediately, so that bugs can be fixed immediately instead of lost in the bug database after being logged. These are all doable tasks for testing with obvious beneficial outcomes, but they may require a new way of doing business for many.
As mentioned earlier, there needs to be automated ways of doing everything. Easy-access production-like environments must be creatable on-the-fly by automated processes. Deployment, test data creation and service virtualization must also be generated automatically. QA is responsible not only for their own test processes (complete with feedback at every step), but also for ensuring that the automated processes outside of testing don’t break.
For development, many organizations are not ready to create containerized software, despite the availability and obvious payoff for using Docker and similar tools. Containers may not be the best way to build software into a known production environment, but that joint decision needs to be made by all stakeholders, as every organization has its quirks.
We know the importance of mitigating business risk, but how does it get done? The company holds the key; problems that adversely impact the customer are a top priority, as are essential elements of the business process. This forms the basis of the risks to be mapped. The test suite needs to be optimized to provide the greatest risk coverage with the fewest number of test cases, the report generated by testing must highlight the business risks, specifying technical, performance and compliance impact when consider priority.
How does one optimize test scope? Understand all integrations involved in critical user workflows including the application tech (web, mobile, message/API-layer etc.). Determine the most realistic data possible for each test, including synthetic data generation where needed. Select end-to-end tests that focus on the critical and high-risk app features. Implement service virtualization for integrations and components that are harder to access to run end-to-end test continuously. Include exploratory testing to find Ux and other bugs that evade automation testing.
How does one identify and mitigate bottlenecks? Historically, this is a tricky area for DevOps. Testers are often left out of critical work that could better educate them for determining acceptance criteria and SDLC velocity. Development may be too scheduled to address debug work in a timely manner (or Ops may prioritize development’s time to duplicate that effect, as well as lowering bug priority, creating technical debt). Regression suites may run too long or not test effectively (keep in mind that the pesticide paradox states that static regression suites degrade in effectiveness over time). Additional steps may still be automatable or be able to become more automated.
So, what’s the strategy for Continuous Improvement to Continuous Testing?
