Why do words like tedious and time consuming keep coming up in these Rants? Check out Lee’s latest verbal tirade on the “monumental importance” of the percentage of test cases automated.
Interested in knowing what your organization’s level of readiness is for implementing Test Automation? Click here to take Utopia’s Test Automation Assessment, our free online automation assessment will give each participant a personalized scorecard to gauge their test automation readiness and existing health.
Take our user-friendly functional test automation assessment to determine your automation potential. Whether you get top scores or have plenty of room to grow, Utopia Solutions will help you optimize the return from your investment in test automation.
-Check out more of Lee’s Rants-
Part 2 – Managing Expectations
Part 3 – Consultants and the Perfect Framework
How about measuring # of validation points? We are looking to show this metric in an attempt to show that a test case can cover a number of functional elements. For example, you could have one test case that explicitly shows validation points. For example, in rspec (our test runner of choice, for now) has this:
expect(message_invoice).to eq(invoice_number)
There are multiple ‘expects’ in our test cases. UI automation (we use Selenium/Ruby) is expensive to run so we take advantage of inspections and add them throughout our tests. When that happens, could have 1 test with 10+ validations. People get nervous when there aren’t “enough” test cases.
Myopic? Yes.
I am hoping to change that perception at my shop and show the actual value automation is providing.
We have the data we just need to advertise it better.
Yes, validation points (and other relevant automation metrics) are a great thing to track. The point of this rant was that I see so many organizations start on their automation journey by blindly setting a goal of automating X% of their manual test cases. Whether you’re tracking validation points, test cases, or % of testing activity automated I believe the key is to set the goal intentionally using data that can support it.
I like to track % of test activity (i.e. time spent on testing related tasks) as that gets people thinking about additional activities that can benefit from automation. What I don’t like specifically about % of test cases automated is that it often leads people to take their manual test cases and start automating them one-by-one verbatim. They completely skip the process of thinking about what should be automated, how to approach automation most efficiently, etc. and end up with a disaster they can’t maintain.
Finally, I agree 100% on advertising your automation data. Your ahead of the game in that you HAVE data!