Test Automation Assessment

Welcome

Welcome to the Utopia Solutions Test Automation Assessment. We have compiled a series of questions that will help us to provide an initial assessment of your organization’s readiness and capability to succeed with test automation.

Please answer the following questions with the answer that best represents your organization at this time. It is important to be candid and realistic when answering. We are not here to judge, but rather to provide as true and accurate picture of your organization’s capabilities as we can.

Upon completion of the Assessment click the “Submit” button. Utopia will use your confidential answers as input in the preparation of a complimentary assessment.

All material, including questions and answers, contained in this assessment are the property of Utopia Solutions, Inc.

Scope of Assessment

This assessment has two main sections:

Readiness, which assesses an organization’s readiness to implement and support test automation, and
Health, which assesses the health of an existing test automation implementation.

If you do NOT have test automation implemented you should Select “Readiness ONLY” below.

If your organization currently performs test automation to some degree, select “Both Readiness and Health”.

Select your assessment:
Readiness ONLYBoth Readiness and Health

TEST ASSESSMENT READINESS QUESTIONS

ORGANIZATIONAL STRUCTURE

1a. Which option best describes your software development process?
All projects and applications utilize a waterfall development processAll projects utilize an agile (or similar iterative) development processAll projects are utilizing DevOps principles in their development process

1b. Which option describes how your software testing function is organized?
Test team is put together for each project or release cycle and disbanded afterwardsEach application team has a dedicated full-time test teamOne dedicated full-time team performing a centralized testing function for all applicationsOther

Please describe #1b "other":

1c. Which option describes how your software testing function is organized?
Testers are separate from the development team(s)Testers are imbedded with the development team(s)A separate testing role is not defined in our software development processOther

Please describe #1c "other":

2. Which option best describes the level of support for test automation in your organization?
Test automation is not supported or funded by managementTest automation is informally supported by management, but does not have dedicated budgetTest automation is informally supported by management with dedicated budgetTest automation is formally supported by management with publicized goals / performance objectives and dedicated budget

3. Which statements best describe your organization’s understanding / expectations of test automation? (check all that apply)
Test automation success will be achieved if we use the right toolsTest automation will allow us to significantly decrease our testing headcount and/or budgetWe can automate 100% of our testing activityTest automation consists of importing our existing manual test cases into a test automation toolTest automation is an investment which will pay a return immediatelyTest automation is software development and must be treated as suchTest automation is the application of automation principles to testing related activitiesTest automation requires constant review and improvement to remain effectiveTest automation will allow us to achieve 100% test coverageTest automation requires specialized expertise and experienceOther

Please describe #3 "other":

PROCESS

4. Which type(s) of testing are you looking to automate? (check all that apply)
Unit testing (testing individual behavior of code modules/methods)Integration testing (testing the behavior of multiple modules or services that work together to perform a specific function)System testing (testing the behavior of an entire system in a specific environment)End-to-end testing (testing user and/or data flows that span one or more systems)Regression testing (testing that existing code/functionality/systems have not been impacted by newly introduced changes)Other

Please describe #4 "other":

5a. Have you identified other testing related activities (outside of test case execution) that can benefit from automation?
YesNo

5b. What other testing activities have you identified as automation targets? (check all that apply)
Test data creationTest case generationTest environment creation and/or configurationOther

Please describe #5b "other":

6. Which option best describes the time at which you assess the impact of application changes to your manual test cases?
At the time the requirements, user stories, acceptance criteria, etc. for the application change are createdAfter the coding for the changes have been completedImmediately before the manual test suite is executedDuring execution of the manual test suiteWe don’t have a manual test suite (i.e. test cases are not documented at any level)Other

Please describe #6 "other":

7. Which option best describes your understanding of the data requirements associated with your existing test cases?
We have a detailed understanding of our test data requirements including how data changes over time (i.e. time dependent data, data that can only be used once, etc.)We generally understand the types of data required by our test cases at a high levelWe rely on the knowledge of the tester to use appropriate data values at execution timeOther

Please describe #7 "other":

8. Which option best describes the desired execution frequency for the testing that is desired to be automated?
Multiple times per dayMultiple times per weekOnce per weekSemi-monthlyQuarterlyOther

Please describe #8 "other":

ENVIRONMENT AND DATA

9. Which option best describes the environment(s) targeted for automated test execution?
The environment is dedicated to automation only and created on-demandThe environment is dedicated to automated execution onlyThe environment is shared with other testing activitiesThe environment is shared with other testing activities as well as non-testing activities (e.g. support, training, etc.)Other

Please describe #9 "other":

10. Which option best describes how data is managed in the environment(s) targeted for automated test execution?
Data is restored to a known state prior to each executionData is restored to a known state periodicallyData is refreshed from production periodicallyData required for test execution is created prior to each runData is not controlled at allOther

Please describe #10 "other":

11a. Do you currently have resources targeted to perform test automation activities?
YesNo

11b. Are these resources dedicated to test automation activities full-time?
YesNo

11c. Do these resources have any previous experience implementing test automation?
YesNo

11d. Which option best describes their test automation experience?
UnknownResources have been exposed to test automation, but not directly involved in implementationResources have been involved in a successful test automation implementation, but did not design/lead the effortResources have designed/led a successful test automation implementationOther

Please describe #11d "other":

TEST ASSESSMENT HEALTH QUESTIONS

SCOPE DETERMINATION

12. Which option best describes how the high-level scope for test automation was determined?
All testing activities were investigated to determine if they could be made more efficient and/or effective by applying automation techniquesOnly test case execution was considered to be in scope for test automationOther

Please describe #12 "other":

13. Which option best describes which test cases were identified to be in scope for automation?
An arbitrary percentage of existing test cases were chosen for automationTest cases were chosen for automation based on their priority (or the priority of the functionality to which they are associated)Test cases were evaluated with specific criteria designed to identify ideal candidates for automationOther

Please describe #13 "other":

TEST CASE AUTOMATION APPROACH

14. Which option best describes the approach to automating existing test cases?
Existing test cases were automated exactly as they are documented for manual executionExisting test cases were first examined to identify test objectives. Those test objectives were then automated in a way that is most efficient for automated executionOther

Please describe #14 "other":

15. Which option best describes how the level (e.g. UI, API, etc.) of test automation was chosen?
Our application is an API/service layer only. We test 100% at the API levelOur application is UI only (we do not have an available API/service layer). We test 100% at the UI levelOur application has both a UI and API/service layer, but we test 100% at the UI levelOur application has both a UI and API/service layer, but we test 100% at the API levelOur application has both a UI and API/service layer. We test as close to the code being tested as possible.

MAINTENANCE EFFORT

16. Which option best describes the percentage of the overall effort spent on maintaining the test automation solution?
> 75%50 - 75%25 - 50%< 25%

17. Which option best describes the percentage of the overall effort spent on preparing data for each execution of the test automation solution?
> 75%50 - 75%25 - 50%< 25%

18. What is the most common cause of required maintenance to the test automation solution?
Changes to the application under test (i.e. new or changed features)Changes to the test casesChanges to the application dataChanges to the test environmentOther

Please describe #18 "other":

EXECUTION ISSUES

19. Which option best describes your automated test suite’s ability to complete a test run?
The test suite consistently encounters many issues that prevent a complete test runThe test suite consistently encounters a few issues that prevent a complete test runThe test suite occasionally encounters issues that prevent a complete test runThe test suite consistently completes a test runOther

Please describe #19 "other":

20. Which option best describes the type of issue encountered most frequently during execution of your automated test suite?
Application defectsTest environment configuration issuesTest data issues (expected data missing or in an unexpected state)Automated test or framework issuesOther

Please describe #20 "other":

RESULTS REPORTING/ANALYSIS CAPABILITIES

21. Which option best describes the reporting and results analysis effort associated with your automated test suite?
Results are clear and concise for both 'pass' and 'fail' results. Detected issues are supported with the appropriate information to quickly identify the cause.Results are clear and concise for 'pass' results only. Detected issues require significant effort to identify the cause.Results are often ambiguous and require significant analysis to determine accuracyOther

Please describe #21 "other":

PROCESS

22. Which option best describes the process used to proactively review and improve the automated estimate?
The test suite is not proactively reviewed/improvedThe test suite is only reviewed/improved when execution issues occur consistentlyThe test suite is proactively reviewed, but only for test scopeThe test suite is proactively reviewed and improved from both a scope and approach perspectiveOther

Please describe #22 "other":

23. Which metrics do you measure related to your automated test suite? (check all that apply)
Test coverageMaintenance effort/costExecution timeReturn on investment / effort savedOther

Please describe #23 "other":

CONTACT INFO

First Name*

Last Name*

Company Name*

Phone*

Email*

Confirm Email*

Test Automation Tools Currently in-house (select all that apply)*
NoneIBM Rational Functional TesterMicro Focus Silk TestMicro Focus Unified Functional TestingRanorex StudioSelenium-based Tool / FrameworkSmartBear TestCompleteTricentis ToscaWorksoft CertifyOther

Please describe "other":

Full Service Software Testing Solutions. Our 25+ Years of Experience is Unparalleled		Call us for more info: (630) 593-2074
Onsite – Onshore – Offshore		request a discussion