Selecting Tests for Automation
One of the major criteria that has an effect on the schedule and effort of an automation project is Selecting Tests. There are a lot of things that need to be weighed-in while selecting tests for automation.
A test case is typically based on a end user workflow (a scenario). We simulate the actions performed by the user, look at the results and assign a pass or fail status to a test. Depending on the application under test you have to decide on what scenarios to automate. Still, there are criterions that can be used to make automate or manual decision.
Selecting Tests to Automate
Automate Basic Application Functions
There are some functionalities of the application that forms the critical path. Whenever you make a release, you will be checking these functionalities. These tests will be a good start for automation.
These tests tend to be routine tests and run often. Manually running these tests leads to fatigue and might lead to errors.
Smoke and Sanity Tests
SMOKE TESTING, also known as “Build Verification Testing”, is a type of software testing that comprises of a non-exhaustive set of tests that aim at ensuring that the most important functions work. The result of this testing is used to decide if a build is stable enough to proceed with further testing.
Automate Tests Targeted for Differing Configurations
Automate Tests that have Consistent Outcomes
One of the criteria for automating a test is that when the steps are performed, the outcome should always be the same. You can’t automate a test that requires a subjective verification to find whether it succeeded or failed.
That said most scenarios, even when the outcomes do not match at different times, can be automated by looking at the common patterns in the output. For example, a Google search might return different outputs for the same search term — but you can be sure that the search term appears in (at least few) of the search results. We can still automate such scenarios.
Capture the Low Hanging Fruits
Automate Tests that are Impossible to Run Manually
Some tests are not possible to run manually. For example, your test might be based on some timing or an action from external system. Automating these scenarios resolves most of the issues and helps in delivering a robust system.
Automate Tests that Follow the Same Steps, but Differ in Input and Expected Results
Most of our application does some logic processing at some time. These are tests that check for given input what is the output. Think of a calculator. These tests are best performed through automation as data driven tests.
Tests that should not be Automated
One off Tests
Avoid automating tests that you run only once (or very few times). Besides increasing the test run time, one off tests also require maintenance.
As always there is an exception to the rule. If you want to run the test against a huge data set or the test is a long running one – it may be still worth your while to automate it. However, you should either delete it once you get the result or not add into your regular test suite to save running time of your test suites.
Tests that do not have Definitive Outputs
Automation is most effective when the tests product definitive and predictable outputs that can be verified to generate pass or fail results. Tests without predictable outputs bloats your automated test suite and requires manual intervention each time the tests are executed.
You do not need to be disheartened by this rule. In quite a few cases in is possible to convert an indefinite result to a definite result by generalising the expected output.
User Interface and User Experience Tests
When you check user interface and user experience it requires manual intervention. Specifically, user experience tests require domain expertise.
Tests that do not provide Good ROI
Return on Investment (ROI) is a performance measure used to evaluate the efficiency of an investment or compare the efficiency of a number of different investments. Keep the metric when you decide to automate a test. Against the benefit of faster execution, less manual intervention you need to compare the cost of resources, time. Do not forget that automated tests are an investment that requires continuous maintenance.
We considered various measures that you can use to select tests to automate. Following these guidelines should allow you to achieve good return on investment. Note that each project is different and what works for a particular project might turn out to be a big mistake for another one. Consider these guidelines and evaluate them in the context of your project to get the best results.
Marathon Integrated Testing Environment – MarathonITE, is an affordable, easy-to-use, cross-platform test automation tool for Java/Swing™, Java/FX™ and Web applications. Using MarathonITE you can quickly automate your daily tests.