Selenium is rapidly growing as one of the most widely used test automation tools. In fact, Gartner reports that “open-source testing tools continue to make progress, and Selenium in particular has high adoption.”
We have been finding that many of our customers are making the switch to the open-source trend, and viewing Selenium as highly desirable. This made us wonder about other facets surrounding the adoption of Selenium, such as the challenges that people are facing, other tools that were considered, and usage in large versus small organizations.
To quell our speculations, we performed a survey that addressed these very questions. Some findings were very surprising, and others were more validations.
One such validating statistic confirmed that the market leans towards QTP, Rational, SoapUI, and Selenium, with Selenium being used nearly as widely as QTP.
So the big question comes down to how to really succeed with Selenium. With no support contract or formal training available, a lot of test engineers just want to focus on their main concerns in order to get their project up and running.
The survey showed that exactly 50% of organizations are developing their own testing tools in house, either as standalone tools or in conjunction with tools like Selenium. Amongst Selenium users, this is closer to 60%. The fact of the matter is that Selenium is great at what it does, but it’s not the one tool to automate them all.
So what are the primary concerns and challenges that more than half of the community develops solutions to address?
Missing Features:
• 31.3% noted a lack of a built in object repository
• 34.4% limited ability to automate flash, silverlight, other content types
• 18.8% lack of logging and reporting
Undoubtedly, the two most important things that test automation needs to be able to do at an absolute minimum are to interact with the application under test and to output results, which means we need to identify objects and generate logs. With that being said, it’s no surprise that roughly 85% of users are trying to address these missing features.
Without developing additional capability or utilizing another tool to address this, it is almost impossible to succeed with Selenium. Logging and reporting are more easily addressed by additional tools than object recognition. Many cloud platforms for Selenium will address this; testing frameworks like JUnit and TestNG also have a lot to bring to the table on these topics.
Script Maintenance:
• 31.3% Managing issues with dynamic pages
• 34.4% Automation is brittle to changes in app under test
There are two primary ways in which an application changes and automation sinks or swims as a result: Either the application is dynamic in nature and can change during or in between test runs or new releases change the application in such a way that the automation stops working. Whether you’re using Selenium or any other tool, reducing the cost of test automation maintenance should be a priority.
What many people are often surprised to learn is that most solutions to the problem are non-technical in nature.
The three most important factors in reducing automation maintenance are process, standards, and documentation. This ends up being true because there are only two ways to reduce the cost of maintenance: reduce the amount of maintenance required or reduce the time spent to perform maintenance. The reason non-technical solutions are more important here is that they have the advantage of addressing both facets of maintenance costs.
Process around the creation and implementation of automated test cases reduces the chance of human error, decreases the likelihood of missed edge cases, and improves test coverage.
Good standards lead to the creation of a consistent product which reduces turnaround time in developing new automation, reduces repeat errors since solutions are recycled rather than reinvented, and simplifies maintenance for obvious reasons.
Documentation of the application during development allows for better prediction of dynamic content by the author of the automation and the identification of reliable object identifiers that are less likely to change. Documentation of the test automation itself will make identifying maintenance related issues a simple matter.
When migrating to automation testing, you have to select the right tool based on the methodologies and additional support that will be required to successfully get your product tested. More and more, we are seeing our clients migrate to Selenium as their tool of choice. Although there are some challenges involved with Selenium, plenty of plug-ins and supplementary tools have emerged to leverage one of the most powerful and flexible automation testing tools on the current market.
QualiTest conducts survey to understand how Selenium is being used within organizations. Read more here! >>tweet<<
Test automation tool market leans toward Selenium, QTP, Rational, and SoapUI, with Selenium being used as widely as QTP. >>tweet<<
50% of organizations are developing testing tools in house, either as standalone tools or in conjunction with tools like Selenium. >>tweet<<
34.4% of survey respondents find the inability to automate flash & Silverlight as the most limiting aspect of Selenium. >>tweet<<
34% of survey respondents say the most difficult aspect of writing scripts with Selenium is that the tool is brittle to changes in the application under test. >>tweet<<
