Test Automation

What Is Test Automation?

Test automation is the practice of using software to automate the verification of functionality of an application or system. It is designed to provide speedy notification to developers when any issues are introduced into the code of an application.

Because automated test case execution is performed by machines and does not require manual intervention, development teams can seamlessly integrate it into the testing cycle. Automation also significantly reduces test cycle time. For these and other reasons, test automation is at the core of Continuous Testing, and is the key enabler of a fully automated CI/CD pipeline.


Top Benefits of Test Automation

Benefits of Test Automation

Test automation can help teams:

  • Provide faster feedback. Issues can be found much more quickly and earlier in the testing cycle, enabling teams to fix them earlier in the development cycle.
  • Increase testing efficiency. Tests can run 24/7. The machines that run these tests will not tire out or make mistakes. 
  • Reduce time to market. Test automation shortens the testing cycle because it runs much more quickly and scales more easily through the addition of testing machines.
  • Achieve better test coverage. Automation can run different configurations, machines, data, and so on. This generally isn’t possible in manual testing due to time constraints.
  • Increase morale. As repetitive tasks get eliminated from the work to be done, teams have more time for innovation, and their job satisfaction increases.


Implementing Test Automation


Test automation isn’t free—it requires an investment of time and money to set up and maintain. Organizations should be prepared to create a roadmap and sell the idea by preparing a business case that includes an ROI calculation. Organizations should also set the reasonable expectations and understand that test automation will not eliminate the need for test analysts and it will require investment in time and resources. There are several high level steps that can greatly help an organization to implement test automation successfully.


Step 1. Identify what tests to automate

Remember that, due to time and budget constraints, it’s not possible to fully test any software and even more so, automate all tests. Prioritize automation based on:

  • Importance of the functionality. Create test automation first for the functionality that represents the greatest risk for the business, and then expand automation when time and budget allow. To identify the highest-risk or highest-impact areas of an application, use a risk-based testing approach.
  • Maturity of the functionality. If a certain piece of functionality is known to be changing in the near future (for example, a team is piloting a new feature that will only be available for one week), the automation that tests it will be irrelevant before it can generate any ROI.
  • Frequency of updates to the application. If an application only changes once per year or less, it will rarely need to be tested. Automating tests for this application, then, won’t save enough time to justify the work and cost involved in building the automation. But if an application is changing frequently, then automating the core functionalities can greatly reduce the testing cycle. Risk-based testing can also help teams identify candidates for test automation in a fast-changing environment.
  • Technical feasibility. If there are no tools available to automate an application, then it may be better to focus on applications that can be automated with tools rather than take the time to build an automation tool from scratch.
  • API over UI. Shift automation to testing API whenever possible. Although manual testing normally focuses on UI testing, UI automation tends to lead to more issues with resilience because of its dependency on UI components.


Step 2. Select the automation tool to use

At the core of test automation sit the test tools used for automation. Teams can choose to create their own custom-built automation tools, use open source tools, or purchase commercial tools. Each type of tool comes with its own pros and cons.

Custom-built automation tools naturally provide the most flexibility and customization since the functionality is defined and built by the team. However, custom-built tools are also the most expensive in terms of the resource cost to build and maintain. Unless there isn’t a solution in the market that can automate the application being tested, it is not recommended to create a custom-built automation tool.

Open source tools are, of course, free in terms of licensing. They may already deliver all the functionalities needed by the team, but even if not, teams have the flexibility to modify or add functionality to fit their particular needs. Open source tools are supported by the user community, which is also free and can be very responsive if the community is active.

On the flip side, if the community is not active, then support will be lacking and the team will have to resolve issues themselves. This may consume significant time and resources. In addition, extending the capabilities of a tool and integrating it with other tools often requires development skills comparable to those needed for building a custom-built tool. Finally, the argument that users are not locked into the tool falls short unless they plan to migrate the test cases to a tool that uses similar technologies. In the end, “free” sometimes isn’t really free.

Commercial tools often deliver many capabilities right out of the box and can get a team up and running much more quickly. They typically provide pre-built integrations with other tools, further reducing the need for development resources. Commercial tools may support a broad technology stack out of the box, which can facilitate the automation of most commonly used technologies. They also come with technical support and an SLA that can ensure issues will be resolved in a timely manner.

The disadvantage of commercial tools is that they can be expensive in terms of licensing fees. They are normally less flexible in terms of modifying or extending functionalities. Thus, users who want additional functionality are at the mercy of the vendor’s product roadmap.

Choosing the right tool should not just depend on the initial licensing fees. Teams should look at whether the tool delivers the functionality right out of the box, or whether they will have to devote development resources to building functionality. They should consider the skillsets of their resources—will their automation specialists need to learn how to code, or is there a low-code or no-code user interface that users can more easily learn? Teams should also look at the support structure—do they need certain SLAs, or do they intend their teams to figure things out themselves? In the end, organizations should build an ROI model to compare costs (not only license fees, but also the cost of the support services and resources needed to build and maintain the tool) and time to market (how quickly the team can get up and running).

Keep in mind that the efficiency of test automation in any organization won’t be determined by the type of testing tools it chooses—it will be determined by the processes and skills the organization applies to the testing process.


Step 3. Train or hire automation resources

As mentioned in the previous section, there is no magical automation tool currently in the market that can create test automation without the need for testers. Organizations will need to invest in training or hiring resources who have with test automation skills and train those resources in the best practices around using the automation tools selected. The roles that are required and the automation focus depend on whether organizations are automating at the team level, program level, or enterprise level as well as the testing cycle stage.


Step 4. Automate

With the plans in place, tools selected and resources trained, the next step is to create the test automation. At the beginning of the journey, avoid emphasisis on the quantity of automation created, but rather focus on the quality of the automation created. It is better to create a smaller number of good automated test cases rather than a large number that has to be thrown away or reworked later on. Use test design methodologies to reduce the redundancy of the test suite, which becomes critical while the test suite grows. Every piece of automation introduced should aim to increase test coverage.

Resilient test automation, repeatable and reliable test automation, is the largest challenge when creating automation. Use root cause analysis methods such as ASSESS to reduce the false positive rates of automation. ASSESS enables teams to figure out why a release failed testing and how to avoid the underlying mistakes in the future. False positives are normally caused by three factors: test environment, test data, and poorly written test cases.

  • Create a test environment strategy to ensure that test environments are available and stable for test execution.
  • Create a test data strategy to ensure that the required test data is stable and readily available before every test execution.
  • Train automation specialists, automation engineers, and test architects on how to use the automation tools as well as automation best practices. Establish a process to ensure that the best practices are being followed, and that assets will remain resilient and reusable. Have them establish a review process to ensure that quality tests are being created and maintained.

When building test cases, try to build reusable test assets. It’s not enough to enjoy the immediate benefits of automating isolated pieces of the testing cycle—if the test artifacts aren’t reusable, it will be nearly impossible to scale up automation cost-effectively. Building reusable test assets not only reduces work when teams are creating initial test cases, but also reduces the maintenance burden because they can make changes in one place.


Step 5. Execute the automation

Many times, organizations are so focus on creating test automation and forget that the ROI from test automation is only realized when they are executed. It is important to establish a test execution strategy to balance the speed of providing meaningful results against the optimal test coverage for every stage in the testing cycle. Teams should make sure they execute their automation as often as it makes sense. 


Test automation requires a significant amount of investment in resources and time. It is not as simple as buying an automation tool and having a few resources work on it. However, if implemented correctly, the returns will greatly eclipse the investments placed in it.