In this article, I wanted to share my experience with choosing and comparing modern UI test automation tools.

It is important to note that I had no previous experience implementing test automation in a QA team. I had a lot of questions when, as a team, we initiated the process of setting up a test automation project from scratch.

I learned some things along the way. This post is a short summary and reflection on lessons learned.

I am not assuming the below process is the right way of tackling test automation in a new team. Instead, I am just sharing my thoughts and experiences. I also share some links and resources I created which you might find helpful.

1. Understand the context
2. Define goals and requirements
3. Review available options and shortlist 2-3 candidates
4. Complete proof-of-concept (POC)
5. Ask for advice. Reach out to the testing community
6. Be Aware About the Risks
7. Discuss results with your team
8. Make a decision

1. Understand Context

I think it is important to start by asking key questions related to your team’s and project context. Later, this helped us to understand why one option is better suited for us than the other. There are always multiple variables..We need to identify what is more important and why.

1. Q: Who will lead test automation project? A: We have a test automation engineer recently hired
2. What role will play other team members? A. first will be helping to define what to automate and review automated tests b. later, up-skill and help build tests
3. How much experience do other team members have in test automation and programming?
   1. some team members had limited test automation experience
   2. most of the team members have beginner programming level
   3. everyone is willing to learn
   4. time is critical; we are quite busy with other testing activities
4. What would be the most suitable programming language and why?
   1. in our case, JavaScript/Typescript looked like the most suitable options because Typescript is used to build the app.
5. How does our product work? What are we should consider when choosing
   1. Multiple iFrames
   2. Multiple domains
   3. Testing courses content
   4. Some tests can take a relatively long time due to waits (2+ mins)
   5. Preferably allow integration with Azure DevOps CI/CD
6. Will our team be able to adapt quickly to more technical test automation tool?
   1. probably not

2. Define Goals and Requirements.

Based on the answers I got, I tried to outline some general and specific requirements

General Requirements for Automation Tools

1. Intuitive and easy-to-use UI
2. Good documentation, community and easy-to-find answers
3. Support for multiple validation types
4. Exhaustive features
5. Advanced validation expression support
6. Ways to document changes, test cases, and test results

Specific Requirements for Automation Tools

1. At which levels do we need test automation?
   • UI (web application)
   • Why?
     • We want to cover areas of the app that cannot be covered by API testing
     • We want to ensure UI is not broken
     • UI automation might be suitable for team members without strong technical skills
2. Do we have skilled resources? a. Yes. We hired a skilled test automation engineer b. It is important that we should be able to upskill other testers and involve them in test automation later.
3. How much do we want to invest? Are we willing to allocate a budget for a non-open-source tool?
   • This should be discussed with management.
4. Is there anything special about our application that can affect our decision? Which factors we should take into account?
   1. See No. 5 in the previous chapter.

3. Review Available Options and Shortlist 2-3 Candidates

Basically, we spent some time googling recent articles comparing the most popular modern UI tools. This allowed us to shortlist the following options

1. TestComplete
2. Playwright
3. Cypress

Some of the team members had experience working with Selenium. We agreed that although it is an industry-standard, modern tools provide easier to create automated tests and have some important improvements when compared to Selenium (e.g. less code required, it is more readable, better speed, automatic waits, less time required to set up)

4. Complete Proof-of-concept (POC)

We defined 3-5 tests that we want to automate using each tool as quickly as possible. These tests should be simple and yet cover some important areas we identified earlier.

Once we finished that, we had some first-hand experience with each tool. We started to understand better their strengths and weaknesses.

5. Ask for Advice. Reach Out to Testing Community

Well, there are a lot of testers smarter than me and testers who have more experience. Why not ask for advice? That’s what I did. I posted my questions in 3 testing communities. The replies were really helpful.

1. Ministry of Testing website | MoT
2. Ministry of Testing Slack Invite | MoT
3. Test automation university Slack

6. Be Aware Bout the Risks

I asked myself, “What can go wrong? Why would our test automation undertakings fail? What are some example scenarios? “

There are some things we cannot control. However, we can make better decisions when we know about risks and internal factors which can negatively affect test automation project. For example, there is a risk of creating low-quality test automation scripts or a risk of not having a proper work environment for a team to upskill in test automation.

1. Hired automation engineer leaves the company and other team members do not have enough time and/or skills to drive results in test automation
2. Improper Staff Selection and Resource Planning. Examples:
   1. hired tester does not have/lack required skills to design, configure, and implement test automation, a specific skill set is required.
   2. hired testers not having deep technical knowledge.
   3. hired tester does not communicate well with team members and other teams.
3. License costs are too high (if we have chosen a non-open-source tool) and the company no longer wants to pay for it.
4. Test Automation did not deliver the expected results because the chosen tool is
   1. too complex
   2. not flexible enough
   3. test are too flaky
5. QA team members have insufficient skills
6. Low quality of test automation scripts. Examples: producing code with hard-coded waits. Code that is hard to maintain, reuse, read and understand. Code that generates tech-debt.
7. Unrealistic expectations. Example: The speed of authoring is lower than we expected
8. Not having independent tests.
9. Not scaling test automation. Example: we can run tests in QA environment, but cannot do it in other environments. We cannot execute tests in different production regions.
10. Test execution is too slow. Not executing tests in parallel.
11. Lack of test reports. Example: automation test run does not provide a clear answer “Do we have critical fails? How many tests failed in a particular area?”

7. Discuss Results with Your Team

In our team we actively discussed the results and next steps as we went through the process. For example, each team member made a short demo for a tool(s) that were shortlisted. As always, I tried to ask questions and learn what other team members think. I always struggled with making decision. That’s why I try to put my thoughts and researched information on a paper. In this case, I created a Word document where I listed Important factors and corresponding requirements that we should consider when choosing a test automation tool. Interesting links to online resources that I came across during research were included in that document too.

I also created a spreadsheet with side-by-side comparison with scoring and conditional formatting based on the importance of each item. Here is the link.

• Test Automation Tools Comparison – Google Spreadsheet download

Of course the scoring can be subjective. There are still some gaps and some cells have no score, but I think I did a decent research. Now I understand the capabilities, pros and cons of each tool much better. Feel free to comment/contribute. I will be happy to update that spreadsheet and improve it.

8. Make a Decision

Well, in our team we decided to go with TestComplete. To be honest, my personal preference was Playwright. However, I agree that for a team that does not have much test automation experience, the learning curve is an important factor. In addition, our test automation engineer had previous experience with TC.

Short summary (my opinion),

Playwright. It is more powerful and faster. Many teams were recently switching Cypress > Playwright. Open-source and 100% free.

Cypress. Code in Cypress is more concise and elegant in most cases. Also, Cypress has a nice desktop app making development experience and debugging easier. Open-source.

TestComplete. Paid tool by SmartBear. Does not support Mac. Can be a good choice for a team without solid technical skills. Mild learning curve. Allows test script recording. Has AI support, visual testing, allows testing desktop/web/mobile apps.