There are several myths regarding the implementation, maintenance, costs and types of tests that can be automated. Discussions on test automation have kept coming up during multiple IT conferences and meetups.The number of tools, and thereby the possibilities of using them, is increasing; however vague knowledge of test automation does not help to popularize such tests and therefore to deliver products of the highest quality. In the article I shall examine common misunderstandings around automated tests to clear up doubts in this area.

Test automation – how does it work?

Before I discuss the test automation process further, let’s see how such tests are created.

Firstly, we specify areas requiring automated testing, focusing on relevant issues from a business perspective. We set up the goal and terms depending on the tested system, bearing in mind one of the principles of testing: “Testing is context-dependent”.

Once the goal is set up, we proceed to technical aspects. We choose a testing tool, and create the testing environment and testing scripts. Having run tests, we analyze the results and verify whether we have managed to obtain business value as assumed earlier on.

Deterministic automated tests work in a quite simple way. First of all, we put the application in the initial state to test a desired action, for example how the application acts while being used by, say, 1000 users. The aim of the verification is to compare the assumed results with the actual state.

Test automation – myths

1. Each test can be automated

According to one of the widespread myths, each test can be automated. Let’s think it over: is it really possible? As per the International Software Testing Qualifications Board – the software testing certification organization – the testing process is infinite, which is why we cannot forecast and test everything.

Test automation is still incomparable to efficiency tests, volume testing, and stress testing. Automated testing has its limitations, though. For example, it cannot be used for non-functional tests. These are related to our personal perception of esthetics, legibility and transparency. In the abovementioned area we are unable to define the initial requirements and the expected final results. Some of the aspects, for example the appearance and transparency of the application, are perceptible only to the human eye. The same thing applies to accessibility testing, which aims to determine whether users with disabilities can use the system with ease.

All in all, knowing that some of the tests cannot be automated, we understand that it is impossible to entirely replace manual testing with automated testing – as per the second common myth. There will always be areas requiring manual testing.

2. Automated tests can replace manual testing

This misunderstanding brings to mind a common myth from the past, that one day television would entirely replace radio. Nowadays, as we can see, radio, television and the Internet coexist and complement each other. It works similarly when it comes to automated and manual tests.

The first ones are advantageous due to their repeatability. With appropriately implemented tests we can be sure that every test will be run in the same, replicable manner. Additionally, contrary to human nature, automatons do not need coffee, the weekend or sleep, so they can work perpetually. On the other hand, humans are better when it comes to intelligence, intuition and imagination, which are essential to running an effective test of the application.

The majority of tests related to functional and non-functional requirements can be performed manually. There will always be a need for manual tests, calling for the engagement and knowledge of a tester – namely usability tests, accessibility tests and security tests. Test automation supports manual testing, but cannot replace exploratory testing or the technique of error guessing, wherein the tester’s knowledge in the context of the product and tools is essential.

3. Automated tests help to catch a lot of defects

Contrary to popular opinion, automated tests will not help to detect a lot of defects. One can ask: “How can that be? Isn’t that what we run them for?” Of course, it is one of the goals we are trying to attain; however, automated tests are quality-oriented, not quantity-oriented. To give an example, automated tests are useful in catching functionalities’ change-related defects in the event that software code has been amended. These are called regression tests and are used to ensure that changes implemented to either software, configuration or the environment will not impact the developed product itself. Automated regression tests can successfully replace manual testing carried out by entire development teams, which – depending on the software’s complexity – would normally take hours, days or even weeks. Our trust increases with every single automated test. Even though they do not catch a lot of defects, they allow us to save time and money, enabling us to focus on the quality of the product instead.

4. Each automatable case should be automated

Knowing that not all tests can be automated and that there are areas in which we will still need manual testing – is it worth automating every automatable testing case?

There are situations in which swift software testing is required and engaging additional resources, tools or funds (which is often related to test automation) is not an option. Let’s paraphrase a quote by Roger Needham, Microsoft’s Research Laboratory Director, to clear up doubts: automation is a process of replacing something that works with something that almost works, but is quicker and cheaper.

This reminds us of the two principal ways of defining whether we should automate tests or not:

  • automated tests should be faster than manual verification
  • the cost of programming tests cannot exceed the income gained thanks to test automation

From a business perspective, the cost of tools and time needed for the implementation of automated tests in comparison with manual testing should be the decisive factor. This should facilitate the decision-making process in terms of automating testing cases.


Despite many myths related to test automation, automated tests are gaining greater understanding and popularity. Myths related to automated tests keep coming up, which proves that a lot still needs to be done to clarify doubts. I hope that I have managed to clear up some of them and that the article will help to define further discussion on automated tests. This is priceless help for testers in the process of delivering a high quality product.

Daria Bajda-Czyżyńska, JCommerce

Technical leader and software tester, specializing in automated tests. Her adventure with testing began quite unexpectedly - on a production line in a Slovak car factory. Then she entered the world of embedded software testing and fell in love with the world of mobile and web applications.



Would you like to learn more about the possibilities of cooperation? Do you have a question? Write to us!

I hereby agree that JCommerce Sp. z o.o. shall process my personal data (hereinafter ‘personal data’), such as my name, surname, e-mail address, telephone number and company name, for commercial purposes.
I hereby agree that JCommerce Sp. z o.o. shall process my personal data (hereinafter ‘personal data’), such as my name, surname, e-mail address, telephone number and company name, for marketing purposes.
I hereby agree that JCommerce Sp. z o.o. shall process my personal data (hereinafter ‘personal data’), such as my name, surname, e-mail address, telephone number and company name, for recruitment purposes.
I hereby agree that JCommerce Sp. z o.o. shall process my personal data (hereinafter ‘personal data’), such as my name, surname, e-mail address, telephone number and company name, for future recruitment purposes.
I have been informed by JCommerce Sp. z o.o., 3 Ks. Piotra Sciegiennego St. 40-114 Katowice – the personal data controller – that: - The provision of the aforementioned personal data is voluntary but essential for commercial purposes if I have chosen a request for proposal, or recruitment purposes, if I have chosen the remaining options;
- I have the right to access the content of my data, including to receive copies of it and correct it, delete it and limit the processing of it, as well as the right to transfer it, the right to object to the processing of it, and the right to withdraw my consent at any time. However, the withdrawal of my consent shall not affect the lawfulness of processing carried out on the basis of the consent in question prior to its withdrawal;
- A statement of withdrawal of my consent to the processing of personal data should be submitted to the headquarters of JCommerce Sp. z o.o. or sent to the following e-mail address: The withdrawal of consent to the processing of personal data shall result in the inability to fulfil the aforementioned processing purposes;
- The personal data provided shall be shared by JCommerce Sp. z o.o. with the company’s authorised employees and individuals collaborating with JC under civil-law contracts, who are involved in the implementation of the purpose of the processing;
- The data provided shall be processed on the basis of the relevant provisions of Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation), ‘GDPR’;
- Should you have any questions regarding the protection of your personal data, please contact us by e-mail:;
- The personal data provided shall be processed for the purpose for which it was supplied, or until I express my objection in this regard. In the event of filing an objection, JCommerce Sp. z o.o. shall no longer process my personal data for the aforementioned purposes, unless it can demonstrate that there are valid and legally justified grounds overriding my interests, rights and freedoms or my data is necessary to establish, pursue or defend a claim, if any;
- I have the right to file a complaint to the supervisory authority if I consider that the processing of the aforementioned personal data violates the provisions of the General Data Protection Regulation of 27 April 2016.