Qualitest Builds a UAV Test Automation Solution with Reusable Scenarios and Building Blocks
To support a strategic move towards continuous deployment, our client has new tools and processes to shorten and automate their testing cycles.
Implement a continuous delivery program that allows to deploy software and system versions to internal and external clients.
Manual test processes were extending and complicating development cycles.
Qualitest deployed highly experienced engineering teams who were responsible for designing strategy, process and framework for automation.
We developed automatic tools and mechanisms to improve quality and speed, enable scalability and provide continuous visibility as well as achieve DO-178C certification.
By delivering flyable software versions more quickly, development delays have been reduced, saving hundreds of thousands of dollars at a time.
The Client was able to increase testing scope by 500% and to shorten regression cycle time by 150 working hours.
Our Client is an Israeli military systems and technology provider. Their Unmanned Aerial Vehicle (UAV) systems operate multiple missions that include tactical and military operations with a very high standard of control and precision.
Our Client releases regular software updates and patches to ensure optimal UAV performance for their clients. Because of the mission-critical nature of these updates, the client needs to accelerate their development and testing cycles.
Accelerating software development without compromising quality
The Client develops multiple projects with “System-of-Systems” engineering, which is tested as part of the development cycle. A complete acceptance test is conducted at the system level as each cycle completes.
To better serve the needs of their customers, the Client wanted to:
- Continuously deliver SW and systems versions to multiple internal and external (end) clients
- Shorten test cycle lengths while keeping coverage and quality
- Lower test effectiveness erosion with high repetition by hand
- Raise coverage on complex and under-documented parts of the system
Existing testing systems were too restrictive and reliant on manual processes that were slow, inefficient and completely unworkable under a continuous delivery model.
Re-engineering for reusability
Initially, Qualitest implemented a test automation tool alongside their testing services. This allowed the client to streamline their test cycles and increase the rate of approvals for each subsequent version deployment.
At the same time, Qualitest used their experience to take the lead on several key processes. Working alongside the Client’s development team we were able to design the strategy, process and framework that would be used to automate testing. We were also able to assist with code reviews on deliverables and to provide technical accountability, applying industry best practices to raise overall standards of testing and deliverables.
To help the Client understand the improvements being made, Qualitest distributed weekly reports and bi-weekly progress reviews, enabling continuous visibility of automation scripts and delivery status. These were accompanied by weekly progress reviews with the development team that highlighted main achievements, issues, needs or blockers, as well as outlining the key tasks planned for the next stage of the project.
“We developed automatic tools and mechanisms to improve quality and speed, enable scalability and provide continuous visibility as well as achieve DO-178C certification.”
Testing for two product lines
The Client operates two distinct Command, Control, Communications, Computers and Intelligence (C4I) product lines. Although very different, Qualitest analyzed both to identify technical similarities that could be used as process building blocks. These would then be used to integrate with the specific process or scenario on each product.
For the Client, this resulted in a reusable code infrastructure comprised of one main suite of ePF with tens of reusable scenarios. Each scenario could be applied on both major product lines (subject to minor changes) with a coverage of 80% of the Sanity processes and an increased percentage of regression and ATP coverage.
Further improvements were made by abstracting data input into individual scripts into flat .csv or .tsv files. This abstraction enables the Client’s automated solution to be data-driven so that different test runs can explore different combinations of test data without needing to alter the actual test scripts.
Testing the bigger picture
Automatically testing designated software features was just part of the Client’s challenge. Acceptance testing also needed to address the full aerial system and environment of the UAV.
Qualitest incorporated multi thread monitoring functions that runs in the background of each automatic test case and logs all abnormal events as defined by the Client, such as loss of control, crossing pitch and roll boundaries or there is a system failure in one of the flight critical sensors.
By introducing these monitors, Qualitest was able to reduce the manual oversight previously required for testing. The tool not only keeps track and record of the main tested component but also zooms out to validate the entire system and environmental state. The monitoring solution was incorporated across the Client’s software – GCS reports and display, mechanical line test station SW etc. – all crucial parts of the tested system.
Once processes and tools were aligned, the new automation solution underwent a Tool Qualification Process to confirm it meets DO-178C definitions. The Client can now prove to aerospace certification authorities (FAA, EASA and CAAI) that their commercial software-based aerospace systems adhere to industry standards.
The new scenarios built by Qualitest now run in software development labs and system labs, allowing for repeatable tests to be executed extremely quickly as part of the standard testing cycle. With the new tools and processes in place, the Client’s test effort has now shifted, from repetition of sanity and regression testing to testing of major changes and testing of complex and hard to retrace defect paths.
“The Client was able to increase testing scope by 500% and to shorten regression cycle time by 150 working hours.”
Throughout the project and beyond, the Client has realized some significant benefits, including:
- The Client had constant access to an expert-level test resource.
- Re-usable code and a data-driven infrastructure that allow easy assembly and maintain test cases on all systems.
- The Client received a new capability that allowed significant testing of non-formal builds and non-final versions, a capability once considered impossible due to limited resources. The Client can now lower the cost of changes before the main releases.
- Due to automated functionalities, the teams can shift-left multiple version related findings, such as defects, performance issues, conflicting system configurations etc.
- Regression testing scope, that plays a significant role in the formal version release, was scaled up without the time cost of the manual process; More confidence in less time – testing scope increased x5 while shortening the time of regression cycle by 150 working hours (over three calendar weeks).
- The automation solution provided is 0 Hours-driven – tests are performed at night, leaving the HW resources free during the work day.
- Enabling faster version release allowed the Client to shorten the waiting time between flyable versions on the field, thus preventing delays that at times accumulated to hundreds of thousands of dollars.