In Increased productivity and effectiveness, reduced time-to-market, enhanced product quality, lower spending on testing, and faster test cycles are some among the many allurements offered to entice organizations to invest in test automation tools. Test automation, if executed judiciously, can streamline the testing process, and deliver high returns on investment (ROI). It can speed up testing to accelerate product releases, improve test coverage and reliability, ensure consistency, and offer significant financial savings.
Nevertheless, studies indicate that globally close to 50 percent of all test automation projects fail. The test automation outcomes either fail to deliver on financial expectations or do not satisfy stakeholder expectations. There have been several cases in the past where organizations have abandoned their expensive test automation tools and have resorted to manual testing to get the development project back on track. Reasons for failure are manylack of a clearly defined and appropriate automation framework for project execution and delivery being the most common. This generally stems from a skewed understanding of the subject. This paper will look at the common reasons for test automation failure, and measures that organization can take to ensure success.
PROBLEMS PLAGUING TEST AUTOMATION
Automating the testing process improves bug discovery, compresses testing cycle time, enhances productivity and testing effectiveness, reduces time-to-market, and delivers very effective QA. It directly impacts organizational top line as well as bottom line by providing organizations a first mover advantage and thereby a virtual price monopoly till the competition enters the market.
Yet, billions of dollars are lost each year due to failed or abandoned testing automation projects that invariably lead to product release delays and bugs seeping unnoticed into the post production environment. Why do so many test automation projects fail to achieve their potential? What are the common pitfalls that organizations regularly fall prey too? And how can organizations avoid these issues to extract the full benefits of automating the testing environment? Before we proceed to the best practices application developers can leverage to ensure testing automation success, listed below are a few of the most common reasons for failure.
UNREALISTIC EXPECTATIONS
Most organizations consider automation the panacea for all their testing ailments. The common belief is that automation will eliminate the need for all manual testing and is the perfect answer to time and resource crunch. This compels the need to automate all tests, which is not a practical solution. Automation, in itself, does not improve testing. It may seem more reliable than humans in detecting and reporting bugs. However, a test script does not have the intuitive ability to analyze the impact or usefulness of the final output. It can repeatedly run the same non critical tests, without providing any real value, as the basic premise of the script creation is flawed.
Another common reason is expecting immediate payback. This again is not practical as usually the payback occurs only after conducting several rounds of testing using the test scripts in question. Automation is also not the solution to cutting down testing costs. Setting up the facility involves significant startup costs. Also, it involves regular maintenance cost during the lifecycle of the project. In the short term, it can free manual testing from mundane and laborious testing work to focus on value-addition. However, immediate cost reduction is a fallacy. A few key reasons for failure are mentioned below:
UNCLEAR OBJECTIVES
Stakeholders might have differing expectations from the project. For the management the goal might be improving time-to-market for new releases by reducing test cycle time; for the IT department it might be cost reduction, and for the development team it might be improvement in quality while simultaneously reducing manual effort. This divergent set of goals often leads to disappointment. Also, many organizations fail to share the objective of the project with the automation team leading to lack of sync with the business objectives of the organization.
LACK OF FOCUS
Most organizations do not give testing its due credit and priority. Instead of engaging full time resources to automate tests most organizations involve their developers to work on test automation as a back burner project. Some organizations also look at it as a means of building employee skill sets.
NOT CHOOSING RIGHT TEST CASES
Test automation is not an alternative to manual testing. However, many organizations indulge in vacuous automation without distinguishing between tests that can benefit from the same and those that need to be conducted manually. It is necessary to clearly identify the tests that can most benefit from automation viz repeatability of the test case over releases, execution over multiple datasets, etc, and the return on investment that can be expected from the same.
WRONG CHOICE OF TOOLS
License fees for commercial automation tools tend to be quite expensive. There are a multitude of vendors offering myriad test automation tools with various different capabilities and features. Organizations need to exercise care in selecting the tools best suited to their requirements before planning and implementing automated tests. Not all tools are suited to the needs of all organizations. Many projects fail due to choosing inappropriate tools or not having the capabilities or skills to efficiently utilize the same.
AMBIGUITY IN COST STRUCTURE
Most organizations are left floundering when it comes to understanding the various cost components of a test automation project. Lack of clarity of the actual costs involved in automation often lead to sudden surprises and budget overshoots. Testing automation involves certain fixed costs on hardware, software licenses, software maintenance and support, tools, and training. License fees for automation tools are expensive and perpetual. An automation testing project involves script development, which needs to be planned, estimated, and managed like any other software development project. Also, creating test automation scripts require specialized skills, which come at a cost. Organizations need to look at all these cost components in this entirety to gain a clear understanding of the returns on investment.
Add new comment