Selecting Tests for Automation

One of the major criteria that has an effect on the schedule and effort of an automation project is Selecting Tests. There are a lot of things that need to be weighed-in while selecting tests for automation.

Introduction

One of the major criteria that has an effect on the schedule and effort of an automation project is Selecting Tests. There are a lot of things that need to be weighed-in while selecting tests for automation.

Before You Start Automating a Test 

You need to weigh in what benefits accrue by automating the test after you consider the time, effort and resource invested in test automation.

A test case is typically based on a end user workflow (a scenario). We simulate the actions performed by the user, look at the results and assign a pass or fail status to a test. Depending on the application under test you have to decide on what scenarios to automate. Still, there are criterions that can be used to make automate or manual decision.

Selecting Tests to Automate

Automate Basic Application Functions

Keep it Simple, Stupid

There are some functionalities of the application that forms the critical path. Whenever you make a release, you will be checking these functionalities. These tests will be a good start for automation.

These tests tend to be routine tests and run often. Manually running these tests leads to fatigue and might lead to errors.

Smoke Tests

Smoke and Sanity Tests 

SMOKE TESTING, also known as “Build Verification Testing”, is a type of software testing that comprises of a non-exhaustive set of tests that aim at ensuring that the most important functions work. The result of this testing is used to decide if a build is stable enough to proceed with further testing.

Automate Tests Targeted for Differing Configurations

Operating Systems supported by Marathon and MarathonITE
Most of our applications are delivered for multiple platforms and operating systems. So automating tests that you run on different configurations (Operating Systems, Versions, Platforms) will save you lot of time and effort.

Automate Tests that have Consistent Outcomes

Consistent Outcome - Good Headphones

One of the criteria for automating a test is that when the steps are performed, the outcome should always be the same. You can’t automate a test that requires a subjective verification to find whether it succeeded or failed.

That said most scenarios, even when the outcomes do not match at different times, can be automated by looking at the common patterns in the output. For example, a Google search might return different outputs for the same search term — but you can be sure that the search term appears in (at least few) of the search results. We can still automate such scenarios.

Capture the Low Hanging Fruits

There are some tests that are simple to automate. For example, checking for successful and failure of login from a dialog is simple to automate. These kinds of tests when automated adds value to the test suite and improves ROI.

Automate Tests that are Impossible to Run Manually

Some tests are not possible to run manually. For example, your test might be based on some timing or an action from external system. Automating these scenarios resolves most of the issues and helps in delivering a robust system.

Automate Tests that Follow the Same Steps, but Differ in Input and Expected Results

Most of our application does some logic processing at some time. These are tests that check for given input what is the output. Think of a calculator. These tests are best performed through automation as data driven tests.

Tests that should not be Automated

One off Tests

Avoid automating tests that you run only once (or very few times). Besides increasing the test run time, one off tests also require maintenance.

As always there is an exception to the rule. If you want to run the test against a huge data set or the test is a long running one – it may be still worth your while to automate it. However, you should either delete it once you get the result or not add into your regular test suite to save running time of your test suites.

Tests that do not have Definitive Outputs

Automation is most effective when the tests product definitive and predictable outputs that can be verified to generate pass or fail results. Tests without predictable outputs bloats your automated test suite and requires manual intervention each time the tests are executed.

You do not need to be disheartened by this rule. In quite a few cases in is possible to convert an indefinite result to a definite result by generalising the expected output.

User Interface and User Experience Tests

When you check user interface and user experience it requires manual intervention. Specifically, user experience tests require domain expertise.

Still Want to Automate UI Tests? 

We know a team that tried to automate UI tests by looking at the layout of the components for standardisation. The final result was grief and abandoning the whole suite when a used 3rd party library changed its layout structure.

Tests that do not provide Good ROI

Return on Investment (ROI) is a performance measure used to evaluate the efficiency of an investment or compare the efficiency of a number of different investments. Keep the metric when you decide to automate a test. Against the benefit of faster execution, less manual intervention you need to compare the cost of resources, time. Do not forget that automated tests are an investment that requires continuous maintenance.

Conclusion

We considered various measures that you can use to select tests to automate. Following these guidelines should allow you to achieve good return on investment. Note that each project is different and what works for a particular project might turn out to be a big mistake for another one. Consider these guidelines and evaluate them in the context of your project to get the best results.

A Shout for MarathonITE

Automating Functional Tests with MarathonITE 

Marathon Integrated Testing Environment – MarathonITE, is an affordable, easy-to-use, cross-platform test automation tool for Java/Swing™, Java/FX™ and Web applications. Using MarathonITE you can quickly automate your daily tests.

Close Menu