Introduction to Exploratory Testing

Exploratory Testing #101

Introduction

Exploratory testing is an approach to software testing that is concisely described as simultaneous learning, test design and test execution.

Exploratory Testing with MarathonITE

Exploratory testing is not a new discipline. Skilled testers have always performed it. Unfortunately the same became synonymous with ad-hoc and sloppy work. A group of test methodologists calling themselves Context Driven school of testers brought Exploratory Testing out of the closet. They have been trying to change it into a teachable discipline. James Bach and Cem Kaner (who coined the term Exploratory Testing) have been at the forefront of emphasising the thought process underlying the unscripted testing that is the backbone of Exploratory Testing.

Cem Kaner 

Exploratory testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.

During an exploratory test session the tester generates new good tests to run. The tester uses his experience, heuristics and creativity to create new tests. Exploratory Testing is used primarily for blackbox testing.

Performing Exploratory Testing

Exploratory testing is a discipline that emphasises the learning aspect of the individual testing. There are no hard and fast rules regarding the steps in which Exploratory Testing is done. There are a set of steps that could be followed to perform an effective exploratory test session.

Create a Test Charter

Exploratory testing is not ad-hoc, unplanned testing. It is well documented through test charters. Use exploratory test charters for tracking the findings during an exploratory test session. A test charter should at the minimum record the time allotted for the test session, a recording of the findings and notes related to the test session.

Use the following sections to create your test charters. Note that not all sections make sense for a particular project. The list is supposed to be exhaustive rather than ideal.

  • Charter Name
    The name of the charter. If you are working on a known issue, use the name of the issue. If you are working on a feature, use the feature name. Should be a single line short description of what you are trying to analyse during this session.
  • Focus Areas
    Each session should focus on a specific aspect of the application. List out the focus areas for this session. We suggest each session should focus on a single area. However, there may be cases where you would like to use a single session for working on more than one related areas.
  • Tester Details
    Basically the name of the tester(s) who is involved in this session.
  • Start Date and Time
    Note the start date and time for the session.
  • Duration Time Box
    See the discussion below.
  • Reference Resources
    List out the reference data files (if any), issue tracker details (if any), any external documentation used for the test session .

An exploratory test session may have multiple runs of the application and may find multiple issues. For each of the run note the following into the charter.

  • Test Step Documentation
    Document all the test steps followed. These may include step-by-step instructions, screenshots, screen/video recordings. A path to a central location where these are stored is enough.
  • Issues and Bugs
    The final output of a test session. List out the issues found. The test step documentation should be able to reproduce the issue.
  • Test Notes
    Any observations from the test session that might be useful for the team. These may include suggestions to improve UX of the product, UI related issues or any other issue that is not a bug but still may be of interest.

Time Boxing a Test Session

You need to ensure that an Exploratory Test Session doesn't become a roving expedition. Each should be limited to a set duration. This practice is known as Time Boxing. The session should be for a fixed duration with focus on the specific areas.

Time Boxing helps you concentrate on the goals and scope and discourages you from unfocused exploration.

Value of Exploratory Testing

Exploratory Testing provides better value than other testing disciplines like scripted testing in multiple ways.

Wikipedia 

Most bugs are discovered by some sort of exploratory testing. This can be demonstrated logically by stating, "Programs that pass certain tests tend to continue to pass the same tests and are more likely to fail other tests or scenarios that are yet to be explored."

  • Less preparation time
    You need to spend less time preparing for an Exploratory test session than for a normal test session.
  • Faster discovery of defects
    An Exploratory test session helps in identifying major defects faster. Since the tests are performed with simultaneous learning, Exploratory Test sessions are effective in identifying user experience issues.
  • Discover hidden bugs
    Exploratory tests sessions that focus on new areas and areas that are not tested earlier helps in identifying hidden and out-of-critical-path errors early.
  • Tester Motivation
    Exploratory tests sessions are intellectually stimulating when compared to the standard scripted tests. This tends to highly motivate the test team.

Exploratory Testing - Drawbacks

Like with any approach, Exploratory testing has it’s own share of drawbacks.

  • Test cases can't be reviewed in advance
    When test case design happens while a test session, it is not possible to review the test cases in advance. This is a major hindrance if your test team consists of novices.
  • Not possible to definitely say what tests have been executed
    Though diligently following test charter helps reducing this risk, it is still not possible to have a definite record of test cases executed.
  • Can't execute the same tests again
    Unlike in scripted testing where the test steps are documented, Exploratory Testing is freestyle. It is not possible to execute exactly the same steps as in an earlier test session. Though it is helpful for identifying new tests, it is a hindrance when checking for regressions.
  • Focus on the Implementation
    Scripted test scripts are derived from the requirement and design documents and focus on what is desired. Since Exploratory Testing focuses on the product what is tested usually is the implementation rather than the desired functionality.
  • Restricted Focus Areas
    There is a possibility that the test team may focus on some areas and spend lot of time in these areas at the cost of other areas.
  • Need for Experienced Testers
    The effectiveness of Exploratory Testing is based on the efficacy of the individual tester. The approach is based on the heuristics and a-priori knowledge of the tester. You can somewhat mitigate this risk by pair-testing and ensure that novice testers can come up to speed.

What Next?

A famous phrase often used with software methodologies is “It depends”. Does Exploratory Testing begets benefits to your project? It depends

Exploratory Testing provides great number of benefits – less preparation time, finding new bugs etc. You need to weigh these benefits against the context of your project. If your project has a detailed requirement specifications and novice testers – it is better for your team to perform scripted testing with one or two senior testers helping others by creating test scenarios. On the other hand, if you have experienced testers on team it is beneficial to try Exploratory Testing regardless whether you have good requirements.

A Shout for MarathonITE

Exploratory Testing with MarathonITE

MarathonITE supports an exploratory mode. While performing exploratory test session with MarathonITE you can record script (steps), attach annotated screenshots and note observations using checklists.

Close Menu