architecting your automation strategy - part 1

The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.

Bill Gates

True to Bill's quote, I have met only a handful of those who have made automation work. If you are one of them who cares about bringing efficiency to your Test Automation to achieve a better return on investment, I believe I can share from my experience. With the right set of processes and tools getting the most out of your Test Automation is certainly achievable.

Start with your team:

Is your team well suited to automate on their own?

Should your team consist of only Test Engineers, will your company be ready to invest in a Record and Playback automation tool that requires very little learning curve?

Do you want to bring expert or external help to start with the automation?

You may want to bring in automation experts in case you would want to hive off automation as a separate discipline or take external help to build a minimal framework that can later be included as a part of your automation ecosystem.

In my opinion, automation works best if your team does it. There are two reasons:

a. They will get to learn something new plus they will easily understand the importance of automation and how it could help save countless hours spent for regression testing.

b. They will also know which test cases are critical and must be automated

So it pays to up-skill your team by investing in an automation tool or offer training to do automation on their own. I have put together a number of links in this article that could help self-train. You may also engage a consultant during the up-skill phase to ramp-up this strategy.

Automate early

The biggest benefit of automating early is the return on your investment.

Just like how testing early helps you identify issues early, automation also helps in finding issues early if you wish to shift-left your automation test strategy.

Also, it helps if you run the test scripts frequently helping gain confidence in a release. This avoids the risk of not using the automation script later for want of stable scripts.

Objectives to consider while automating early

a. Forget end-to-end automation:

This Google Testing Blogpost discusses at length why end-to-end automation is not an optimal approach.

Why?

You will have to deal with a lot of flakiness if there are several integration points involved. A script failure could also take time to analyze based on the complexity of the test script.

The more end-to-end tests that you own the longer it takes to run. The last thing you would want is automation taking more time to run when a Test Engineer would have completed it quickly.

b. Automate APIs first:

If your product uses APIs, better to start with automating them first. Your UI layer is only as good as the data it consumes. Also, your team will mostly use a tool to test the APIs, so why not automate it instead?

Use a contract-driven approach (Not consumer-driven contracts) to start with API automation. A contract in integration testing parlance is simply “an API specification that contains end-points with requests and response along with the data type”.

The Developer and the Test Engineer can work out the contract during the design phase. The developer can publish the contract in a word document or WIKI or in a widely used knowledge base format. The Test Engineer can now start with API automation using the contract and run it whenever the Developer’s code is deployed.

Keep in mind to unit test individual APIs before you start with the integration testing.

c. Automate user stories next:

Whether it is the shift-left approach or automating in a regression cycle, when you automate UIs, always automate a user story.

For ex: In an e-commerce website the user would most likely search for a product, view the product description, add it to cart, make the payment and track the order.

Although it sounds like an end-to-end case it is a story that binds all your UIs in a single story. This will also make it easy to debug scripts by taking you to the exact step should a script fail. It will help you complete the integration tests from an API perspective as well.

Automate what is necessary

a. Catch up with complex stories:

More often than not you may ignore a few test scenarios or postpone it for the future.

Do catch up later to see if you can automate it. See if there is additional work required in the framework or get creative in solving the problem using a different approach.

Keep in mind to automate only such a story that the user would absolutely use. You may as well cover this in a regular regression cycle if there are too many interaction points.

For ex: Few workflows in an e-commerce application may require a third party integration, like OTP based login or fill out a payment form. You may work with the developer to develop a script or mock the interface to automate such workflows.

b. Automate for better coverage:

If you are starting with UI automation, then plan to start with a breadth of scenarios that will give adequate coverage to all modules in the system.

This will help visualize how automation will work and help plan coverage for new modules in the future. Automating a large set of scenarios for each UI module can have adverse side-effects like longer run times not fetching enough issues, script flakiness, etc…

In part 2, I will discuss making automation work.