Automated testing has become essential to software development, helping teams improve their work’s speed and quality. It’s so critical that some professionals specialize in just automating tests (even though they are often also qualified to design and manually execute tests).

However, it doesn’t mean that all tests can or should be automated. In some cases, manual testing may be more effective or efficient; in other cases, the time and effort required to automate a test may not be justified. This blog post will explore some examples of tests that should not be automated and why.

When the software is just a Proof of Concept

The one situation I think people neglect the most is when the object under test is a PoC. There are different kinds of PoC. Engineers usually build PoCs to test technology, design approaches, assess how tools, frameworks, or libraries fit their needs, etc. Product teams sometimes request PoCs to validate assumptions with target audiences or refine their product vision.

In both cases, heavy rewriting is expected. Technical/engineering PoCs often offer nothing reusable except the knowledge gained. For product PoCs, it’s common to have to rewrite or redefine requirements completely.

In this situation, it’s likely that if there were any automated tests, they would all have to be rewritten. Remember that automated tests are code that needs maintenance subject to the same requirements as the software under test.

Maybe it’s wiser to wait until the software is not a PoC anymore and only automate the tests. This is something to be considered case-by-case.

When tests don’t need to be re-executed (not part of retests)

Automated testing can be a valuable tool for testing changes to critical features or functionalities of a system. However, there are situations where automated testing may not be necessary. If you will not re-execute it multiple times, why bother automating it?

It will likely take longer to automate it than to run it manually, and that’s not considering situations where manual execution is a prerequisite for automation (like recording actions to write a script).

Some examples of tests that don’t require re-execution that may not be suitable for automation include:

• Changes to edge cases, non-critical or low-impact features, or functionalities. The time and effort required to automate the testing process may not be justified.
• Changes to documentation or help content. These updates are typically straightforward, do not affect the system’s functionality, and are not dynamic. Just a visual inspection should be acceptable.
• Changes to the look-and-feel/appearance of UI components or modifications to strings in the UI that come from files without any processing. These tests are usually simple, often low-impact, and may not require retesting. Manual testing may be sufficient to ensure the accuracy and functionality of the system. Also, verifying visual changes is not the biggest strength of most automation tools and frameworks.

When tests must be re-executed (retests) but the automation cost is too high

As I’ve repeatedly stated, automated testing is a powerful tool that can improve the efficiency and effectiveness of the testing process. I’m not advocating against test automation. On the contrary, I firmly believe the vast majority of tests should be automated.

However, there are situations where manual testing may be more appropriate. Some examples of tests that require re-execution that may not be suitable for automation include:

• Systems that call other systems and depend on time-sensitive or real-life data, such as stock exchange data. Simulators or specialized infrastructure may be required, making automated testing too expensive. If the cost of running these manually is too high, maybe it justifies building this extra infrastructure to allow the automation of more tests. If this is the current situation, monitor the effort spent executing such tests, as they may increase over time until it may be worth the investment.

• Systems that require actions to be performed after some time has passed would require a “time machine” ability (a feature to change the time the system is capturing to make it act as if the time had passed in real life. These are situations where, because of environmental or regulation constraints, the results of an action can only be experienced some days later.

  Even if the dependencies can be mocked or future values can be manufactured, the software may be getting the time from the operating system. Refactoring of the software may be required to allow faking the current time, setting it in the future or even in the past.

  Sometimes this situation is known at the inception of the software, and it may be built with this capability from the beginning, allowing for greater testability.

• Systems that involve complex workflows and user interactions or require human judgment or decision-making. It may be challenging to simulate all possible scenarios, making manual testing more effective. This may be more frequent with end-to-end tests.

In conclusion, automated testing is a valuable tool for software development teams, but it’s not always the best solution in every situation. In some cases, manual testing may be more effective or efficient; in other cases, the time and effort required to automate a test may not be justified. Knowing when and where to use automated testing is essential for ensuring the quality and accuracy of the software application.

I hope these examples help teams make informed decisions about their testing strategy and optimize their resources for maximum impact.

Cheers!