In this article, I wanted to share my experience with choosing and comparing modern UI test automation tools.
It is important to note that I had no previous experience implementing test automation in a QA team. I had a lot of questions when, as a team, we initiated the process of setting up a test automation project from scratch.
I learned some things along the way. This post is a short summary and reflection on lessons learned.
I am not assuming the below process is the right way of tackling test automation in a new team. Instead, I am just sharing my thought and experience. I also share some links and resources I created which you might find helpful.
- Understand the context
- Define goals and requirements
- Review available options and shortlist 2-3 candidates
- Complete proof-of-concept (POC)
- Ask for advice. Reach out to testing community
- Be Aware About the Risks
- Discuss results with your team
- Make a decision
1. Understand Context
I think it is important to start by asking key questions related to your team's and project context. Later, this helped us to understand why one option is better suited for us than the other. There are always multiple variables..We need to identify what is more important and why.
- Q:Who will lead test automation project? A: We have a test automation engineer recently hired
- What role will play other team members? A. first will be helping to define what to automate and review automated tests b. later, up-skill and help build tests
- How much experience do other team members have in test automation and programming?
- some team members had limited test automation experience
- most of the team members have beginner programming level
- everyone is willing to learn
- time is critical; we are quite busy with other testing activities
- What would be the most suitable programming language and why?
- How our product works? What are we should consider when choosing
- Multiple iFrames
- Multiple domains
- Testing courses content
- Some tests can take a relatively long time due to waits (2+ mins)
- Preferably allow integration with Azure DevOps CI/CD
- Will our team be able to adapt quickly to more technical test automation tool?
- probably not
2. Define Goals and Requirements.
Based on the answers I got, I tried to outline some general and specific requirements
General Requirements for Automation Tools
- Intuitive and easy-to-use UI
- Good documentation, community and easy-to-find answers
- Support for multiple validation types
- Exhaustive features
- Advanced validation expression support
- Ways to document changes, test cases, and test results
Specific Requirements for Automation Tools
- At which levels do we need test automation?
- UI (web application)
- We want to cover areas of the app which cannot be covered by API testing
- We want to ensure UI is no broken
- UI automation might be suitable for team members without strong technical skills
- Do we have skilled resources? a. Yes. We hired skilled test automation engineer b. It is important that we should be able to upskill other testers and involve them in test automation later.
- How much do we want to invest? Are we willing to allocate a budget for a non-open-source tool
- Should be discussed with management.
- Anything special about our application that can affect our decision. Which factors we should take into account?
- See No. 5 in the previous chapter.
3. Review Available Options and Shortlist 2-3 Candidates
Basically, we spend some time googling recent articles comparing the most popular modern UI tools. This allowed us to shortlist the following options
Some of the team members had experience working with Selenium. We agreed that although it is an industry standard, modern tools provide easier to create automated tests and have some important improvements when compared to Selenium (e.g. less code required, it is more readable, better speed, automatic waits, less time required to set up)
4. Complete Proof-of-concept (POC)
We defined 3-5 tests that we want to automate using each tool as quickly as possible. These test should be simple and yet cover some important areas we identified earlier.
Once we finished that, we had some first-hand experience with each tool. We started to understand better their strengths and weaknesses.
5. Ask for Advice. Reach Out to Testing Community
Well, there are a lot of testers smarter than me and testers who have more experience. Why not to ask for advice? That\'s what I did. I posted my questions in 3 testing communities. The replies were really helpful.
- Ministry of Testing website | MoT
- Ministry of Testing Slack Invite | MoT
- Test automation university Slack
6. Be Aware Bout the Risks
There are some things we cannot control. However, we can make better decisions when we know about risks and internal factors which can negatively affect test automation project. For example, there is a risk of creating low quality test automation scripts or risk of not having a proper work environment for a team to upskill in test automation.
- Hired automation engineer leaves company and other team members do not have enough time and/or skills to drive results in test automation
- Improper Staff Selection and Resource Planning. Examples:
- hired tester does not have/lack required skills to design, configure, and implement test automation, a specific skill set is required.
- hired testers not having deep technical knowledge.
- hired tester does not communicate well with team members and other teams.
- License costs are too high (if we have chosen a non-open-source tool) and company no longer wants to pay for it.
- Test Automation did not deliver the expected results because chosen tool is
- too complex
- not flexible enough
- test are too flaky
- QA team members have insufficient skills
- Low quality of test automation scripts. Examples: producing code with hard-coded waits. Code that is hard to maintain, reuse, read and understand. Code that generates tech-debt.
- Unrealistic expectations. Example: The speed of authoring is lower than we expected
- Not having independent tests.
- Not scaling test automation. Example: we can run tests in QA environment, but cannot do it in other environments. We cannot execute tests in different production regions.
- Test execution is too slow. Not executing tests in parallel.
- Lack of test reports. Example: automation test run does not provide clear answer "Do we have critical fails? How many tests failed in a particular area?"
7. Discuss Results with Your Team
In our team we actively discussed the results and next steps as we went through the process. For example, each team member made a short demo for a tool(s) that were shortlisted. As always, I tried to ask questions and learn what other team members think. I always struggled with making decision. That\'s why I try to put my thoughts and researched information on a paper. In this case I created Word document where I listed Important factors and corresponding requirements which we should consider when choosing a test automation tool. Interesting links to online resources which I came across during research were included in that document too.
I also created spreadsheet with side-by-side comparison with scoring and conditional formatting based on importance of each item. Here is the link.
Of course the scoring can be subjective. There are still some gaps and some cells have no score, but I think I did a decent research. Now I understand capabilities, pros and cons of each tool much better. Fee l free to comment/contribute. I will be happy to update that spreadsheet and improve it.
8. Make a Decision
Well, in our team we decided to go with TestComplete. To be honest, my personal preference was Playwright. However, I agreed that for a team that does not have much test automation experience, learning curve is an important factor. In addition, our test automation engineer had previous experience with TC.
Short summary (my opinion),
Playwright. It is more powerful and faster. Many teams were recently switching Cypress > Playwright. Open-source.
Cypress. Code in Cypress is more concise and elegant in most cases. Also, Cypress has nice desktop app making development experience and debugging easier. Open-source.
TestComplete. Paid tool by SmartBear. Does not support Mac. Can be a good choice for a team without solid technical skills. Mild learning curve. Allows test script recording. Has AI support, visual testing, allows testing desktop/web/mobile apps.