Test Automation Strategy: Best Practices and Examples

Software development is a path full of chaos. Here’s why!

Software projects start with excess enthusiasm. It starts with the design, then comes the architectural pattern, and everything looks fine. Even ahead of the schedule. But then things get weird. Rework happens and you’re now a bit behind schedule. 

In the end, your time-to-market is months behind. You cut some corners knowingly acknowledging the technical debt. Because, of course, you’ll take care of that later. Finally, you ship to production. First some bugs trickle in and then it streams in. Now, you’re in the loophole of breaking/fixing things in parallel with new development.

You think software testing automation can save your time and things can be back on track. You start doing things only to find that everything is haywire. Because you don’t know which tools to use or have any software test automation strategy in place. At the end, it turns out, you’re never going to pay down that technical debt. 

Why does implementing test automation seem to be a rescue option, rather than the rule of thumb, in software development? Why, after all these years, does software testing still inevitably seem to work out this way? 

Let’s help you out!

One of the missing parts in the software development industry is bug fixing, which leads to delays in projects. This is backed by the world quality report that suggests 61% respondents have difficulties in automating their QA, resulting in the biggest disregarded costs. Test automation breaks the conventional pattern of finding and fixing bugs and fixing them back followed by the longer regression tests.

“Test automation is making software work to test one.“

The figure shows the initial time & efforts it takes in learning the best practices of test automation. But we can see how it saves the overall effort and time spent.

It’s like, it takes time to learn a programming language. But then, we have today’s world. 

The critical scenario that comes up is at the time of software delivery. And testing turns out as the common “bottleneck.” We started our test automation journey back in 2010, which brought a major change in our processes. Now, why did we need that change? Not because we couldn’t cope with faster deliveries, but our process needed to be more productive.  

I didn’t have much experience with test automation back then, but it was more of a learning experience for all of us. We, at Simform, however, applied test automation for a pharmaceutical client (one of our then clients) where the UI was tested first followed by the known areas and eventually, we took off in the direction of fully automated test runs. 

At the time of manual testing, we used to allocate half of the time to testing.  Later, if things go well, we deliver the product/software. Otherwise, again we’d get back on phase one of development. Because developers could never come to know the faults until they complete the entire thing.

With test automation, we’ve learned a lot and have started strategizing it with the help of experienced agile coaches. Let’s understand the test automation strategies below.

A test automation strategy determines how you approach automation testing for a particular project. Fundamentally, it focuses on critical areas such as goals, testing levels, types of testing, tools, scope, and test environment. It also involves decisions like what to automate, what not to automate, prioritization of test cases, and which test cases are worth automation.

The test automation life cycle occurs parallel to the software development life cycle (SDLC). A typical test automation life cycle consists of a multi-stage process required to introduce an automation testing tool, develop test cases, develop test design, and build and handle test data. There are six major stages of a test automation life cycle:

Stage 1: Determine the scope of test automation

In the first stage of the test automation life cycle, teams identify the viability of the automation testing. By conducting a thorough feasibility study, you can know whether the scope you’re determining will work well and help in designing test scripts.

• At this stage, you need to consider:
• Application modules that are to be automated
• Test cases that need automation
• Approaches to automate those test cases
• Right tools for automation testing
• Budget analysis, implementation cost, available resources, and skills

After determining these things, you should perform two types of feasibility tests:

• Test Case Automation Feasibility
• Application Under Test (AUT) Automation Feasibility

Stage 2: Select the right automation testing tool

Selecting the right tool for automation testing is paramount as it determines whether you will accomplish your testing goal. Before you choose any tool, it’s essential to keep the budget and cost in mind. Also, you should know the available skillset and then analyze if the tools you select match your technical expertise in the team. Moreover, the tool should have a strong support team to answer your questions.

Here are some of the popular automation testing tools you can choose from:

• Selenium
• Katalan Studio
• TestComplete
• Appium
• Watir
• Perfecto
• LambdaTest
• Apache JMeter

Stage 3: Plan, strategize, and design test cases

Here, you need to decide the approach and procedure for automation testing. You also need to work out the automation testing framework at this point. Lastly, you need to understand the tool thoroughly and be aware of the technologies involved in your project.

As a testing team, you should define standards and procedures for test case creation. Also, consider the hardware, software, network requirements, and test data. Lastly, determine the testing schedule and tool for detecting errors.

Here are some of the best practices to follow for this test management phase:

• Collect all manual test cases from test management tools and identify the test cases that you can automate
• Study the pros and cons of testing frameworks before determining the best one for your project requirements
• Build a test suite for automation testing
• Understand the risk and dependencies between testing tools and your application
• Keep the clients and stakeholders informed about the test strategy

Stage 4: Set up a test environment

Here, you need to decide whether to use the remote or virtual machines. If you should consider cross-browser and cross-device compatibility or not.

Based on it, plan the testing environment in a way that covers maximum testing scenarios and ensures comprehensive test coverage. Testing teams should schedule and track the setup of testing environments.

They should also install the requisite software alongside network and hardware setups, refine the test database, and build automated test scripts for testing environments. Here are some of the things you should keep in mind while setting up a test environment:

• Feed the data into a test environment closely aligned with production data
• Set up a front-end running environment to conduct load testing
• Build a list of systems, modules, and applications you need to test
• Keep a distinct database server for staging environments
• Conduct tests on various browsers, operating systems, and devices
• Test application under low and high network conditions to understand rendering time
• Keep all the documentation related to testing in one centralized repository

Stage 5: Create and execute test scripts

After setting up the test environment, focus on creating and executing test scripts. Here’s what you should consider while building a test script:

• Test scripts should take their basis on actual project requirements
• Use a standard testing method throughout the project
• Test scripts should be simple, reusable, and easy to understand
• You should thoroughly review the code in the test script

After creating the test script, the next step is to execute them, and for that purpose:

• A test script should have all the functional aspects
• The test script should run on various environments and platforms
• Try to execute test scripts in a batch manner to save time and resources
• There should be a bug report in case of any errors

Stage 6: Analyze and report the test results

After executing the test scripts, you should document and analyze the results to identify the issues that occurred during a particular test case. Next, create an in-depth report about the behavior of various test cases under specific conditions.

These types of analysis and reporting will help the testing team know the application’s areas that still require improvement. Also, the testing team will know whether an application can work under unfavorable conditions. Lastly, you should share the test reports with all stakeholders to keep transparency.

Test automation improves processes, saves time and resources, and leads to higher-quality software. Here are the other reasons to adopt test automation strategy:

• Provides more comprehensive test coverage
• Promotes re-usability of critical components
• Reduces maintenance costs
• Creates a standardized testing procedure
• Improves test accuracy
• Increases test case efficiency
• Improves the speed of execution
• Minimizes the risk factor

As said in the tweet below, test strategies should be made specific to the project. Let’s walk through the steps that will help you define the strategy for your next project.

When I started my career, test automation wasn’t looked up with many aspirations. We have faced many test automation challenges like you might have too. If you ask me today, things have changed dramatically, mainly due to the availability of reliable tools. However, this doesn’t mean we automate everything we can. The first step towards a successful enterprise test automation strategy is to define our goals and figure out which tests to automate. 

The deciding factor for any test to automate is linked with how many times that particular test can be repeated. The best example of a test that can be automated is the one that runs often, is a mundane task, time-consuming, and requires a huge amount of data to perform the stipulated task. Here’s the list of potential test cases that can be automated:

1. Automate tasks that are highly repetitive, for example, log-ins.
2. Tasks that carry high risk or probability of failure or high
3. Tasks that require testing over multiple browsers/devices/OS/environment/hardware/configurations 
4. Tests with clear pass/fail results 
5. Automate tests that require operating through multiple data sets
6. Automate exercises that allow for multiple options, for example, form submission that accepts different combinations.
7. Tests that require a lot of time if done manually; for example, we had a test that required logging in every time we run a new task.
8. Lastly, you should definitely automate tests that require checking of stable features. 

Andrei Mikhailau, Software Testing Director at ScienceSoft, and his team applies test automation to reduce the meantime to validate new features or fixes, improve regression testing coverage, and eliminate human errors. They also apply automated tests when manual testing is inefficient or impossible, e.g., for testing performance.

However, their biggest challenges in test automation resides in surviving the fast and frequent UI layout and functionality changes. Even the smallest modification may require a rewrite of a substantial number of tests. One of the ways to minimize this inconvenience is to ensure maximum test code reuse. Creating standard high-level application-specific libraries is a suggested option here. 

Success in test automation requires you to plan things meticulously. Once you’re done defining your goals and scope of test automation, the next step is to look out for different test approaches.

My overall experience in testing has taught me one thing: the overall testing culture in any organization is highly influenced by the prevailing testing approach, apart from how they envision the test automation and how are they planning to collaborate beyond the development teams.

This means retrospecting the current process, finalizing and deciding the new test approach as well as determining the level of testing, roles, and responsibilities of the team members. 

But first, let’s decide on the testing method that can provide the maximum value when automated. Different testing methods can be employed for different testing levels. 

Unit Testing

Unit testing is the backbone of any test automation strategy in agile that provides the team with the highest ROI. This testing employs a small piece of code (either function or method) that developers can write, execute, and maintain.

For instance, if you’re constructing a building, a unit might refer to the cement. This cement is one functional piece of code that is irrelevant outside the context. You could test this grey powder separately to check if it hardens after mixing with the water.

Likewise, each unit should be tested individually, but this won’t provide you with a house. Unit testing will give you observability at a granular level which is important for the developer, however useful for the product owners. It is recommended to run these tests locally and in the build. 

Regression Testing

Imagine yourself updating your product build. Now, you return to your product to check the new functionality only to land in disappointment because the feature that you’ve been using since ages has stopped working.

Each time you introduce new functionality into your product, you need to check whether it breaks or makes the existing features. Since regression tests return investments after each build update, they should be your second most priority after unit tests. 

Component Testing

Let’s take our constructing a house metaphor a little further. You’ve tested bricks, cement, tiles, and everything that is needed to construct the house. Will all these ingredients result in a house after they’ve been assembled together? And does it look like a house at all?

Component testing doesn’t care about the quality and quantity of these entities. It only tests the overall business value.   

End-to-End Testing

Imagine a house. A house will have all sorts of parts– bathroom, porch, dining room, kitchen, etc. Are each of these areas easily accessible? For example, will a person looking for a bathroom result in him actually finding a way to it? The utmost concern of the end-to-end tests is to see if all the components of the product can interact with each other in the correct order.

As the name suggests, this type of testing is slow since it conducts the test of the entire business flow from beginning to end. It is surely an effective way to verify the system interaction, but only when you don’t have too many systems.

We always run these tests in our release pipeline once the system is deployed. You can also run it locally if your architectural pattern requires. 

As said earlier, end-to-end tests are complex by nature. So how do you tackle this? Well, the solution that has worked for us is to select the tools that are easy to use, maintain, and scale so as to ensure the long term success. 

Integration/API Testing

Integration and API testing allows you to test how well the system works with the external systems. In other words, how stable your system is when it comes to interacting with APIs.

Since modern software development relies on APIs & SDKs to a larger extent, validating the overall business logic and app’s behavior is critical. 

The best way to execute these tests is to run them before deploying the system in your release pipeline. We often automate API tests that are repeatedly executed like regression cases. The common test cases where we run automated API testing are the areas where the changes of the code directly affect the API functionality. 

Cross-browser Testing

Checking your app’s functionality over multiple browser/OS/devices/screen resolutions is no brainer. Conducting these tests manually is a time-consuming task. But what if you want to do it at scale with high speed? The answer, automating the cross-browser tests.

Automated cross-browser testing has tremendously increased the test coverage and reduced our testing efforts and time by 80%. Leverage the power of test automation by conducting tests for the most common combinations your target users employs. Another focus area should be testing high-risk features. 

Smoke Testing

We conduct smoke tests to verify and get instant feedback about the system builds. It ensures that all the critical functionalities (config, permissions, .net frameworks, etc.) are working correctly.

If these parameters  fail under smoke tests, the product updates are rolled back. However, a series of smoke tests is even more important since it gives us the answer to these three preliminary questions: 

• Is the application successfully initiating without an error?
• Is the login functionality working correctly?
• Are the primary buttons and menus working correctly?

We run these tests in our release pipeline once the system is deployed in production. To conduct smoke tests efficiently, the number of tests should be manageable so as to execute quickly.

The test automation framework is a set of detailed guidelines that one needs to follow while writing and running tests. For example, coding standards, procedures, test-data reporting, etc. Here is the list of six test automation framework that you can choose from: 

Linear Scripting- Record & Playback 

One of the handiest frameworks of all. Once set, you can play it back for the rest of the time. It allows testers to record the steps sequentially. For example, navigation, inputs, etc. 

Pros:

• Coding expertise is not required
• Generates test scripts faster
• Maintains the sequence and hence easy to understand for anyone 

Cons: 

• Impossible to re-run the test cases with multiple datasets 
• Not scalable for the extended project scope 
• Rework will require for the changes in the application

Library Architecture Testing Framework

Library architecture works on the principle of identifying and dividing. That means the framework easily identifies the common tasks and group them accordingly. The framework saves these similar functions into the library and uses it whenever needed by the test scripts. 

Pros:

• Maintains high-level of modularity
• Cost-effective and scalable 
• Easy to run multiple test scripts

Cons: 

• Requires changes to the scripts since data is hardcoded
• Technical expertise is required

Modular Based Testing Framework

As the name suggests, the framework divides an application into separate units and conducts isolated testing. A separate test script is created for each part and later combined as merged tests. 

Pros:

• Modular changes will not affect the entire application
• Test scripts are reusable
• Less efforts and time for creating test scripts

Cons:

• Multiple datasets can’t be used
• Technical expertise is required for setup

Data-driven Test Framework 

 It overcomes the glitch of the linear or modular-based framework. It doesn’t hardcode the data, but allows to store and access it from the external file such as Excel, CSV, etc. It allows testers to test the same feature with different sets of data. 

Pros:

• Tests with multiple datasets are possible
• Less number of scripts 
• Future changes in the modules will not affect the entire application

Cons:

• Framework setup is time-consuming
• Expert needed to format the data sources
• Data is hardcoded

Keyword-driven Test Framework 

Keywords are used to represent the actions performed on GUI. For example, phrases like “click the link to verify” or “click the login button”. It stores the keywords externally and is used for testing the GUI of an application. These keywords are separated from the test logic, and it makes seamless testing of actions by following a set of instructions. 

Pros:

• Independent test scripts of AUT is possible
• One keyword for multiple test scripts is used 
• Code is reusable 

Cons:

• Setup cost is high
• Time-consuming as the object repositories need to be setup 
• QA engineer with great automation skills is required

Hybrid Testing Framework

It is the combination of previous frameworks that overcomes the flaws and boosts the benefits. The hybrid framework favors the flexibility for different applications; hence, it is quite good at bringing accurate test results. 

Pros:

• Flexible, reusable and adaptable
• Wide community of support
• Can be integrated with CI pipeline
• Granular reporting 

Cons:

• Test automation expertise is needed

Automation testing largely depends on the selection of the right testing tools. The first step to selecting the right automation tool is to understand the technology the application is built on and test requirements of the Application Under Test (AUT). One of the main aspects of selecting the right automation tool is the compatibility with the tech stack of the AUT. 

There are few tools that are popular among testers; however, they might fail to address the testing demands of certain kinds of applications. One more thing, the tool must support the programming languages and testing environments your testers are most comfortable with. 

A specific example is to figure out whether your application needs both functional and performance testing. For mobile apps, if you’re targeting both iOS & Android, opt for a platform that supports both – Selenium, Appium – or you may also use separate testing suits– Robotium for Android, UI Automation for iOS. 

The second step is to check whether the tool has the right balance between the supported platform and the ease of use. A growing number of platforms impose the requirement of testing the app deployments on various platforms.

You must note that even in a single variant of the platform there are various versions that need to be supported. For example, if a desktop app claims to run on Windows, then it has to run on Windows 7, 10 (both 32 & 64 bit), etc. 

Similarly, a mobile application could be supported on different versions of Android and iOS. Having said that, it is mandatory for the automation tool to test the application on all the platforms with a minimal change to the configuration file once the test script is written.

A must have for test automation tool is to support cross-platform testing for all the required platforms with maximum use of the test scripts. 

The third aspect is to find a tool based on popularity. Popularity is proof that the tool has the availability of support, quality documentation, and technical forums. This helps you in finding the relevant testing engineers for executing and maintaining the testing tool.

There are higher chances in the way that some of the other testers might have  exposure of using the tool. For example, if we consider web app automation testing, Selenium is popular and widely used in comparison to Sahi. 

The fourth aspect to take into consideration is the licensing cost of the tool. However, this isn’t as simple as comparing the pricing of the shortlisted products. You need to select the tool which has the right balance of cost and fulfillment of your testing requirements.

Firstly, there are two kinds of tools that you can choose from– open-source and commercial. Open-source tools are an attractive option; however, commercial tools offer better support and learning resources. 

Yet, while choosing an open-source tool, the license agreement needs to be carefully assessed before purchasing since all come with their own caveats. There are two kinds of license associated with commercial tools: 

1. Development license which allows you to write scripts on the limited number of machines 
2. Runtime license which allows you to execute tests on a limited number of machines 

Commercial tool necessitates you to buy both the licenses while the open-source tool provides them for free. Lastly, the two important aspects that we consider at Simform are integration with project management software & CI suite, and the availability of training resources and customer support. 

In my experience, building a custom test automation framework over a free open-source tool has worked best so far. The greatest advantage is control. With the in-house automation path, what you will need is exactly what your team builds.

If you’re still not sure how to get started, you can choose both initially. Most commercial tools come with a free trial period. At the same time, you can start building the MVP of the in-house framework. 

After three weeks, it should be clear whether the commercial tool will work or it is within the budget or your in-house tool is a better option. Hope this helps.!

Once you’re done with creating your test automation strategy and selecting the right tools, it’s time to write and execute the scripts. From time and again, we’ve observed that making automation testing a part-time job of manual testers decreases team morale and productivity.

These are the two things we’d not suggest you to compromise with. Suggested option is to hire a dedicated automation testing team. If hiring isn’t a feasible option for you, outsourcing software testing services can be a good option. 

When you start writing test cases, it is advisable to follow best practices. Here are a few of them that we rigorously followed at Simform. 

• Writing test case templates in a way that they could be re-used in multiple projects. Before writing any new test cases, we make sure to check if there are similar test cases written already. This helps us in reducing the redundancy. 
• Often times it happens that the one who has designed the test case isn’t the one executing it. This has encouraged us to write test cases in a concise and understandable manner. 
• Over time, we’ve learned to prioritize each test case based on the feature or component involved. Practicing this helps us in ensuring that high priority cases are executed.

Maintaining test scripts involves scrutinizing the testing parameters. For instance, when the product is rich with features, implementing a regression test may take more time and under-exploit the importance of test automation.

For such cases, maintaining test cases plays a crucial role. The test cases should be optimized and categorized in such a way that  it evaluates the subset of test cases and clearly defines the purpose of test automation.

In my experience of working with agile coaches and QA professionals, I’ve always been advised to remain focussed on maintaining the effectiveness of regression tests.

This can be achieved by periodically cleaning older test cases and categorizing the current ones. You should remain clear on your requirements and whether old test cases are effective to keep.

Test scripts should be minimal with more outcomes. If you can make smaller test cases, automation becomes easier.

For example, if you are testing an eCommerce website, you have to check events such as placing an order, selecting an item, completing the transaction, and more. Placing an order requires clicking on the button and it can be separated from the navigation test case. 

Automation will not do everything automatically unless you trigger it. You will have to make it work for you by designing the test cases as per the need since it is directly related to your product delivery and ROI. Ask your QA analysts to work on subsets of test cases to save a lot of time. 

The more streamlined the test cases are the more rapidly a software can be ready for production. 

Don’t go for meandering objectives when writing a test case. Keep it concise. Keep narrowing the scope of a test case when it comes to automation. For example, when you write a script for checking whether a user can upload a profile picture or not, the script should not outperform the scope. 

It means it should not be checking whether the user can successfully register or not or should not be trying to change the profile picture in order to check the upload process. You need to automate the process of checking functionalities. In order to automate other functions as well, you can better split the test cases separately.

 Over time, while implementing test automation, we have realized that in order to be successful, automation needs to be everyone’s job. We worked hard on changing the way how business analysts and testers collaborate together along with creating and running tests.

By implementing the automation to our testing process, we are able to spend more time on planning, detect more defects faster, and better meet the project demands.

For Joe Wilson from Volare Systems, the end goal of automated testing is to have a decoupled software design and confidence in the quality of the software product being built. Automated testing doesn’t mean there is no manual testing. It means the important things the application used to do still work as they did before. However, there were two main challenges that they faced.

To tackle these challenges efficiently, they have used lots of tools over the years for testing. For example, for NET, they’ve used MSTest, NUnit, and XUnit, for JavaScript: Jasmine, Mocha, and Jest, for end-to-end testing: SpecFlow, Cucumber, WatiR, WatinN, Selenium, and Cypress.io.

In the next section, I am sharing the steps and considerations you can follow to create  the test automation strategy document.  

Software glitches cause huge disasters. Error could occur is possible while programming, but it shouldn’t be when you’re testing. To avoid these errors, we have test automation. However, to implement this, you will need a thoroughly prepared test automation strategy document.

Let’s discuss the building blocks of this automation test strategy example document.

#1. Test Automation Scope

Before getting started, it’s good to have a clear picture in mind. For each module of the project, there should be timelines and milestones defined from an automation perspective. For any particular project, define a scope of it. It should keep everyone on the same page (project stakeholders, consultants, project managers, tech leads, developers, testers, etc.) 

#2. Test automation Approach

This is a broader term to define; hence, it can be broken into three parts: process, roles, and technology. This includes what tests to automate and the decision of when the test cases should be developed during the sprints.

During this part, you should define the roles for designing and reviewing the test cases. A well-followed approach leads you to a successful test automation. 

#3. Risk analysis 

As the name suggests, what are the risks involved in running the automated test case? If yes, at what percentage it may happen and how to optimize it? This will keep you one step ahead of the unexpected disasters. You should consider conducting it before starting the test automation. 

#4. Test automation environment 

Make a concrete pipeline to go through before the software is ready for production. This is how the test automation environment is defined. You need to have a clear understanding of where the data will be stored and whether it can be reused or not.

There has to be a stable and predictable test environment. For instance, for a particular project to go under the test automation, you need to plan the test cycles, prepare the test cases, and predict the testing time required. 

Test automation infrastructure should make it easy for anyone to create, maintain, run and consume your tests and test results – Carlos Kidman, Founder of QA at the Point

#5. Execution plan

The execution plan is the backbone of “bug-free” testing. Run individual test cases before the regression test suite and ensure they are working great. Avoid false outcomes and test fails. It consumes a lot of additional time.

The execution plan should include day to day tasks and procedures. You can use a tool like Bamboo or Jenkins for scheduling the execution. Ask your QA Analyst to set up a test case in a fashion that brings faster feedback to the development team. 

#6. Review & analysis 

Filter the test cases that have consumed more time and analyze them for optimization. Review the entire test automation strategy and take feedback from everyone involved i.e., peers, DevOps engineers.

Ask DevOps engineers to plan out the things that hinder the automation environment. Ensure the QA team to acknowledge the bugs in test scripts and they are resolved. Remember to document everything i.e., analysis, results, and suggestions.