Automated Testing – When and How to Implement Test Automation?

Did you know that the Amazon Web Services team was making 50 million releases a year back in 2015? This is more than one per second! How did they do that? Amazon explains its productivity by using DevOps principles and effective test automation.

In turn, QASymphony interviewed experts, including, for example, Angie Jones, lead testing engineer for Twitter. And they all agree that the market is making ever more stringent requirements for the quality of software products and the speed of their updates. Therefore, experts call the shift in emphasis from manual testing to automated testing one of the trends.

But in terms of the effectiveness and applicability of automated testing, experts are often divided in opinions about where, how, and how much to use it. Why is this happening? Let’s see.

Why Automation Is Not a Panacea: The Pros and Cons of Test Automation

Let’s start with the pros:

1. Relentless: automated tests work even when you sleep. They start automatically, remotely, and according to a schedule. They don’t get distracted, don’t forget, and they test things as many times as necessary.
2. Fast: the robot will pass the test faster than a manual tester in 99% of cases. In the remaining 1% of cases, do not forget that a person can get tired, but a robot cannot.
3. Multifunctional: automated tests can quickly test functionality in different environments and with different settings of the tested software.
4. Big scale: automation allows you to simulate the actions of a large number of users.
5. Energy-saving: test automation frees manual testers from routine. Test automation helps to check basic functionality and the tester focuses on testing new products.
6. Cost-saving: the main task of test automation in business is to reduce testing costs. And they do a great job here if implemented wisely and in the right place.

Now let’s move on to the cons:

1. Easy to break: automated tests break, sometimes even because of a minor code change. It takes time to update.
2. Narrow: Automated test only checks what it is programmed for. It will not notice the error that it was not instructed to look for.
3. Difficult to maintain: with an increase in the number of automated tests, the time for updating them and analyzing old ones exceeds the time for developing new ones.
4. Cannot be applied everywhere: there are areas of testing that cannot be automated. These are usability, layout, translation, installation testing, and other similar areas.
5. Costly: an automated test is like industrial equipment, you must first invest in it, and then look at the payback. And if the “machine” is constantly being repaired and upgraded, it may not pay off.

Some controversial points

1. Benefit of test automation. On the one hand, almost always the time for developing an automated test will be more than the time for passing tests by hand. It has to be a more qualified/highly paid specialist. On the other hand, if the automated test does not need constant updating, then it works almost free of charge. This is why the calculation of ROI of automation is very important. And has to be done regularly! We recommend analyzing the ROI of test automation even before the start of the project.
2. The complexity of the analysis of the results. On the one hand, the developer of an automated test can make the reports understandable only to them. On the other hand, if you correctly approach the strategy of logging the results, then even a new tester will be able to understand at what step the automated test failed. Be sure to check out our online automation QA course.

When Is Manual Testing Better, and When Does the Process Require Automation?

Manual testing is more suitable if you have:

• a young project with unstable functionality;
• manual tests are few and fast;
• you need to check the layout, translation, or usability;
• it is necessary to localize and describe errors;
• no time to develop automated tests.

The need for test automation arises when you have:

• a large number of manual tests and not enough time for conducting full regression regularly;
• a large percentage of missing errors due to the human factor;
• a long time gap between the reporting of an error, detecting and fixing it;
• preparation for testing (setting up the configuration, generating test data) is time-consuming;
• large teams and you need to be sure that the new code will not break the code of other developers;
• old software version support (new patches and service packs need to be tested).

How to Automate Testing

1. Determine what you want from test automation

Someone is looking for cost optimization, someone wants to unload manual testers and others want to increase test coverage.
Check if test automation is helping you achieve your goal, which also allows you to determine what exactly to automate and what to do first.

2. Calculate the ROI of automated tests regularly and collect metrics accordingly

This allows you to find out if you really need test automation and, if necessary, adjust the automation plan.

3. Analyze how many automated tests you need

And how often you will need to write new ones and update old ones. It might be more profitable for you to outsource automation and only pay for the performed work. The hired worker at the same time will sit around doing nothing as they finished their work at the start of the project.

4. Determine what you want to see in the reports

This will help to understand how useful the tool is and how much you trust it. It is important to find a balance between the minimum and maximum data. In this way, test automation will be useful and will not eat up the resources you have. The collected information should be useful. Experienced outsourcing automation engineers can advise you on what should be in the report.

5. Make sure that testers understand what exactly does the automated test does

Only in this way, after the crash, the script can be rechecked manually. Create clear step-by-step instructions. If you have outsourced this task to a company that has both manual testers and automation engineers, ask them for such instructions – just in case, you suddenly have to check it yourself.

6. Do not trust the initial stage of automation to the Junior programmer

The developer is vital for efficiency, correctly built architecture, and ease of updating.

7. Stick to the automated testing pyramid

Create a large number of low-level tests (for example, API) and fewer high-level UI tests.

Conclusion

It is obvious to many QA specialists that the question is “automate or test by hand?” sounds, at least, incorrect. You cannot choose one thing once and for all.

Automating all tests is unprofitable, and sometimes even impossible. Until the AIOps reach the required level, manual testing will be in demand on projects. Therefore, the qualifications of a project manager here are determined rather by the ability to find an exact balance between these two approaches. Determine where test automation will be most effective and implement it in this area.

As for the question of whether to outsource automation or do it yourself, everything needs to be calculated in relation to your business. To do automated testing yourself, many factors must be taken into account.

1. At least you must be sure that automation will pay off and be able to correctly calculate ROI.
2. Finding a qualified automation tester is difficult given their demand in the labor market.
3. You must also load it with work not only at the start of the project but also on an ongoing basis.
4. Finally, make sure that automated tests will work without that specialist. Otherwise, start all over again.