KDT is the next generation test automation approach that separates the task of automated test case implementation from the automation infrastructure.
Test Language is not a test automation approach. Test Language is a comprehensive test approach that hands over the responsibility of automation plan, design and execution to the functional testers by using KDT based solution.
Several years ago, I was a Test Manager for a large organization in the defense industry. The team was consisted of 10 test engineers, 2 of which were test automation experts and the others were functional testers. The test automation experts were focused on developing test automation scripts that covered some of the functional processes of the application.
It took me only few days to realize that things are not efficient.
The same tests that were executed automatically were also tested manually. The functional testers had low confidence in test automation scripts, since the test automation experts did not have the required knowledge about the application under test. The test automation development could start only after the application was ready for testing, and thus was relevant only for small part of the tests (mainly regression test).
I presented the problem to the team. After many discussions, we came up with the idea of letting the functional testers create test automation scripts.
The idea may be simple, but we faced a number of challenges in trying to implement it. The main obstacle was that functional testers did not have programming skills or test automation knowledge. To solve this, the test automation suggested that the functional testers would create their test cases using pre-defined words and they will “translate” this into automated scripts. This approach was favorably accepted by the entire team.
We proceeded to create a Keyword dictionary. The keywords were mutually defined by the automation experts and functional testers. During the first phase, the functional testers wrote their test cases using the keywords, and the automation experts translated them into automation scripts. At a later stage, we developed an automated application that took care of the translation. This enabled the functional testers to create and execute the automated tests and saved even more time.
The keyword dictionary combined with the automated translation application generated a fast return on investment. This allowed us to position test automation as a basis of our testing strategy rather than a luxury.
This was the beginning of Test Language.
Let’s review the way Test Automation is commonly approached.
The testing process includes a number of phases test planning, test definition, bug tracking and test closure activities. The actual test execution phase is often a recursive, repetitive and manual. This phase usually described as a tedious and boring and can be carried out automatically.
To automate the execution phase, you need to create test automation scripts that are usually implemented in commercial, test automation tools; such as HP’s QTP IBM’s Robot, AutomatedQA’s Test Complete, MS VS 2010 team system and freeware tools such as selenium etc..
Each test automation script represents a test case or complementary steps for manual test cases. The scripts are created by test automation experts after the completion of the manual test design.
Contrary what test-tool vendors would have you believe, Test automation is expensive. It does not replace the need for manual testing or enable you to “down-size” your testing department.
Automated testing is an addition to your testing process. According to Cem Kaner, “Improving the Maintainability of Automated Test Suites”, it can take between 3 to 10 times longer to develop, verify, and document an automated test case than to create and execute a manual test case. This is especially true if you elect to use the “record/playback” feature (contained in most test tools, as your primary automated testing methodology).
Sanity check: Measure the success of your Testing Automation project.
Step 1: Measure how many man hours were invested in test automation in the past 6 months (include Test Automation experts, functional testers, management overhead etc.). Mark this as A.
Step 2: Measure how many working hours would have needed in order to manually execute all tests that were executed automatically in the past 6 month. Mark this as B.
If A > B than most likely your Test Automation project is a failure candidate.
These are the common reasons for test automation failures:
Many organizations fail to plan or implement the required changes.
Obviously a different approach is required to make test automation work.
Test Language is a dictionary of keywords that helps testers to communicate with each other and with other Subject-Matter experts. The keywords replace the common English or as the basis and create an approach called keyword driven testing (KDT).
KDT can be used to achieve a number of goals:
The Test Language is based on a dictionary, which is comprised of words (keywords) and parameters.
Test Cases
A Test Case is a sequence of steps that tests the behavior of a given functionality/ feature in an application. Unlike traditional test approaches, Test Language uses pre-defined keywords to describe the steps and expected results (see example in Figure 2 below)
The Keywords
Keywords are the basic functional sub-procedures for the test cases of the application under test. A test case is comprised of at least one keyword.
Item Operation (Item) – an action that performs a specific operation on a given GUI component. For example set value “Larry Swartz” in “customer name” control, verify that the value “3” appears in “result” field. When performing an operation on a GUI item, the following parameters should be specified: Name of GUI item, what operation to perform and the values.
Utility Functions (Function) – a script that executes a certain functional operation that is hard non-effective to implement as a Sequence. For example: wait X seconds, retrieve data from DB etc.
Sequence – a set of keywords that produces a business process, such as “create customer”. We recommend collecting frequently used functional processes such as login, addition of new records to the system as a sequence instead of implementing them as items in test cases.
In most cases, parameters should be defined for the created keywords. The parameters are additional information required in order to generate the test conditions. For example: failed authentication by passing username with illegal password, number for mathematical calculation, etc.
Examples for sequence parameters:
When the user wants to create a new customer, the following syntax is used:
create_custmer (Bob,Dylan,1/1/2000,[email protected])
Using default parameters
Some of the keywords may contain dozens of parameters. In order to simplify the test creation, all parameters should contain default values. The tester should be able to change each one of the default parameters according to the context of the test. For example, if the tester would like to create new customer that is older the 100 years, only the birth date will be changed and all other parameters will remain the same. Obviously, a specific change will not affect the default parameters being used for other tests.
It is extremely important to plan the keywords well and to optimize the amount of keywords by creating multi-function keywords (for example, creating “Change_customer_status” keyword is a better approach than creating 2 special keywords for “activate_customer” and “deactivate_customer”)
KDT is the mechanism used in order to implement test language.
Keyword Driven Testing can be divided into two main layers:
Infrastructure Layer (KDT Engine), a combination of Item Operation, Utility Functions and User Defined Functions (also called Sequence), used as an “engine” that receives inputs (keywords) and performs operations on the application under test.
Logical Layer (KDT Test Case) – used by manual testers and other subject matter experts to build and execute test scripts by using a pre-defined keyword.
Simplicity of test case implementation and reviewing
Functional testers should create the automated test cases directly in a simple, non-code, elite language. This eliminates the need for the time consuming two steps process in which testers create manual test cases and then converting them to automated scripts by the test automation team.
Automation Infrastructure and Test Cases Coexistence
The test automation infrastructure should be separated from the logical layer (which is the test cases). This allows testers to create automated test cases at an early stage in the development life-cycle before the application is ready and even before the automation infrastructure is developed.
Plan and control
In order to maximize the benefit from KDT, a plan used to assess the on-going effectiveness of the automation program.
Test Language should focus on testing new features rather than on regression testing, which is the common practice in traditional test automation
In traditional test automation implementations automation scripts cannot be created before the application is ready for testing. As a result, sanity and regression tests are the main areas, which are automated. This approach usually leads to parallel executions (both manually and automatically) of the same tests.
In addition, tests, written for manual execution, are phrased in a way that does not take into account the strengths of automation scripting. The main gap between the two approaches is:
KDT allows functional testers to plan test automation before the application is ready. This enables organizations to broaden the automated testing spectrum.
We recommend that new tests rather than translation of existing tests should be the focus of automated testing using KDT. All new tests can be planned using the KDT approach. We believe that this approach ease the KDT implementation process and greatly improve automated testing ROI.
The organizational structure of KDT based organization is similar to traditional test automation based organizations. It is consisted of Core Test Automation teams and Functional Testers. However the responsibilities are much different:
| Activity | Automation Experts | Functional Testers | Notes |
| Define Keywords | + | Keywords should be defined by functional testers under the supervision and control of the Test Automation Experts | |
| Develop Utility Functions | + | ||
| Develop Sequences (User defined functions) | + | + | Sequences are developed by functional testers in small scales organizations and for private use (functions that will not be used by other testers). |
| Develop Test Cases | + | ||
| Execute Test Cases | + |
Figure 3: KDT Responsibilities Matrix
The key factor of fully use the benefit of Keyword Driven approach is by fully integrate it throughout the entire test process.
The following diagram shows the KDT process:
