Exploratory Testing

What Is Exploratory Testing?

When people think of software testing, they typically think of a well-defined process which includes setting up environments, preparing data, creating test cases that define exact steps and verification points, and then executing the test cases, whether manually or automated. Exploratory testing departs from this formal scripted testing process. It is the practice of verifying the functionality of a system without using a predefined script.

 

Exploratory Testing Versus Ad Hoc TestingExploratory Testing Map

Although exploratory testing does not follow a pre-defined script, it does not mean that there is no process at all, like in ad hoc testing. In ad hoc testing, there is no plan or structure around testing nor test cases to follow. Someone might simply log onto a system, test any functionality they happen to find, and then log off when they have either found something to fix or are satisfied that features seem to be working correctly.

In exploratory testing, test analysts use their knowledge and understanding of the purpose and functionality of the system to explore and test the system while staying within certain predefined testing parameters. For example, a team of test analysts might tackle exploratory testing for a banking system by assigning one test analyst to test wire transfers, another to test withdrawals and deposits, and still another to test balance checks and transaction history. Each of them would then log on and attempt both valid and invalid transactions to see how the system responds.

Also, unlike ad hoc testing, exploratory testing often has predefined limitations of time and scope. For example, teams may decide to limit testing to a particular new feature within a system or give all test analysts only one hour to try to find bugs. In any case, exploratory testing teams will carefully document their findings, which are every bit as important as the findings of teams performing scripted testing.

Exploratory testing is manual testing—by its very nature, it cannot be automated. As teams shift towards test automation to increase the speed and efficiency of testing, why should exploratory testing still have a place in the development cycle?

 

Why Should You Perform Exploratory Tests?

There are many benefits that come from exploratory testing. With exploratory testing, the typical organization can:

  • Increase test coverage. Since scripted tests follow a prescribed set of steps, executing these steps will always uncover the same defects, regardless of whether it’s manual or automated. The only difference is that automating the tests would find the defects much quicker.
    Because exploratory testing does not follow a prescribed script, it will generally uncover different issues with the system, thus increasing the test coverage. Suppose that the scope of all the functionalities in a system is represented by the box in the image below and the risk areas for the business are represented by the red hexagons.

Risk Analysis

The test coverage from scripted testing is then represented by the lines that cut through the different areas of the system. The test coverage from exploratory testing would be represented by the green hexagons.

  • Find functional defects quickly. In a perfect world, testing teams would be able to set up automated testing for new functionalities within minutes. In the real world—the world in which Agile development has taken root—there simply isn’t time to build automation around every test script. Teams must choose their spots for automation carefully.

    This does not change the fact that development teams often need to get quick feedback on the viability of a new piece of functionality. Rather than taking the time to write test cases—let alone automate them—and the fact that exploratory testing does not require intensive planning, setting up environments and test data, creating test cases, and even needing well-defined requirements before starting, a team can use exploratory testing to assess a system within a set time frame and perhaps identify areas for further scripted testing.
  • Build greater domain knowledge. Team members can learn something new about the system during exploratory testing. This can be helpful to build more team knowledge, especially with resources that are newer to the system.
  • Improve user experience. Although their primary intention is to find bugs, testers can provide feedback regarding the user experience that can improve the overall quality of the product.
  • Generate new ideas for testing. The creative nature of exploratory testing makes it an ideal way to uncover new areas and ideas to add to the formal scripted testing process.  
  • Offer testers more intellectual stimulation. Testers who are still performing manual testing based on scripts often welcome the opportunity to use their creativity and follow their instincts through exploratory testing.
  • Increase involvement and collaboration. Scripted testing is usually limited to members of the development and quality assurance teams. Exploratory testing should involve business, technical writers, and just about anyone else who has a stake in the system. Participants do not need any particular knowledge of development techniques, test case design, or test automation.

    The diversity of an exploratory testing team is one of its greatest strengths. People from different roles within an organization will naturally approach a system with a different mindset—and in so doing, they will expose different types of defects that likely would have escaped notice otherwise. The more perspectives a team provides on a system, the greater the number of problems it can identify within the time allotted for exploratory testing.

 

What Are the Drawbacks of Exploratory Testing?

Every form of testing has its limitations. Organizations should keep the limitations of exploratory testing in mind so that they can maintain realistic expectations.

Exploratory testing:

  • Requires test analysts to have domain knowledge of the system being tested. In theory, development teams could enlist the help of anyone in the organization to click through a system and try to “break” the functionality. But exploratory testing performed by novices will not yield results nearly as meaningful as testing performed by people who are already familiar with the system. Novices may actually slow down the testing cycle by generating trivial bugs based on their own lack of understanding of the correct behavior of the system. They may also miss out on testing key parts of the system.
  • Is tedious to document. One advantage of scripted testing is that the entire test plan is laid out in advance in the form of test cases. Documentation, then, is simply a matter of reporting the results of each of these cases. In exploratory testing, test analysts can choose their own path through a system. They must take the time to document each step they took as well as the results of these actions. For this reason, many testing teams use recording tools to ease the documentation burden.
  • Requires additional efforts to analyze issues found. A test analyst may spontaneously take many paths through a system and enter a wide range of values before encountering a bug. He or she then must remember exactly what they did to cause the bug so that they can demonstrate it to the development team. Recording tools can help alleviate the problem, however, recording tools records all the actions the test analyst performed. A defect may arise only after the test analyst had performed hundreds of steps. Developers and/or test analysts must review all the steps to identify the critical steps that caused the defect.
  • Makes it difficult to measure test coverage because it is not a systematic approach. One of the biggest advantages of exploratory testing—the freedom and creativity it affords the test analysts—is also a drawback. Even with the most thorough documentation, it is difficult to determine how much of the system test analysts covered through exploratory testing. This is why it is essential for teams to strike a balance between exploratory testing and the much more structured scripted testing.

 

How to Use Exploratory TestingExploratory Testing Binoculars

Although exploratory testing requires far less preparation than scripted testing, it still requires a structured process to yield the best results. Organizations should:

  • Create a charter. The charter should be a statement about what the exploratory testing session is designed to accomplish.
  • Decide who is involved. It is beneficial to include people outside of the typical testing team—such as developers, product owners, and business users.
  • Define the scope. Specify the features that will be covered and/or not covered in a session so that testing can be focused on a specific area of the system, e.g. a high-risk functionality or new features that require quick feedback.
  • Schedule exploratory testing sessions and limit them to 1-2 hours. Because exploratory testing emphasizes personal initiative and testing could theoretically be endless, it is wise to schedule sessions of no more than two hours. This approach will help teams to maintain better focus and make it easier to quantify the effectiveness of each exploratory testing session.
  • Specify test analyst’s scope and tactics. If more than one test analyst is part of the session, defining each test analyst’s scope can ensure that more of the system is tested and there is no redundancy. It can also be beneficial to specify a specific tactic that the test analyst should approach the system, e.g. he may approach interacting with the system with a menacing mindset—looking for loopholes in the system—and another test analyst approach the system with a new user mindset—someone unfamiliar with the system.
  • Use a tool to record the processes and results within each testing session. This way, there will be documentation on the steps required to reproduce any bug that the team encounters.
  • Review results and add them to learning as well as the scripted test suite. At the end of testing, teams should call all participants together and review their results. It may be helpful to appoint a moderator for this task.

 

When to Use Exploratory Testing

Exploratory testing fits well in agile development practices. During a sprint, teams can schedule time for a round of exploratory testing that focuses on new features or areas of major change. The learnings from the exploratory session can feed into the creation of new tests that can be automated and added to the regression suite.

Adding exploratory testing into a waterfall development practice can be more of a challenge. One possibility is to encourage development teams to release builds more often, even with partial functionalities so that an exploratory session can be used to provide quicker feedback on the user experience along with any issues found. Another possibility is to perform exploratory testing in parallel with the normal scripted testing to increase the test coverage.

Organizations should keep in mind that the value of exploratory testing lies not only in the bugs it finds, but also in its ability to improve the user experience and to generate ideas on additional areas of testing.