First Steps in Test Automation

Like most endeavors, there is no substitute for good planning when it comes to test automation.  This document explores a dozen topics for ensuring a successful first experience in automated testing.

Setting goals

Each application and each team will have its own parameters for determining how much automation can be accomplished within a given period of time. Rather than fumble with rules of thumb that may have no relevance to your project, it is best to select a pilot project and conduct a time study in your own environment. Select a project that you believe can be managed in less than a month. Keep track of the number of hours spent on developing the automated tests for the selected features. Compare these times to the number of hours that it would take to test the same features manually. These time studies provide two measures that can be used for setting future goals and predicting the percentage of coverage from test automation:

• Time to automate per feature

• Time to automate as compared to time to manually tests

By comparing the relative size of future projects to those already completed, it is possible to set reasonable goals and reliably predict the amount of coverage that can be expected from test automation. Keep in mind that subsequent projects will take less time because of experience and reuse of test components so it will be necessary to refine your measurements over time. 

Hiring and training staff

Experience and training are necessary to ensure a good ROI from test automation. Most automation efforts that fail do so because of lack of training not because of shortcomings of the testing tool. This is a place where it does not pay to skimp on the budget. Instead, prepare to send the test automation staff to the full set of courses offered by the test tool vendor. In addition, look for a course on test automation methodology and architecture. Be sure that it is based on practical experience. It should include topics like:

• Deciding What To Automate
• Test Planning for Automated Testing
• Designing Good Test Cases
• Preparing Test Case Data
• Using a Data-driven Test Engine
• Programming Standards and Conventions

It is more than helpful to have someone with prior experience on the automation team. If you are unable to hire someone already skilled in test automation, consider hiring a consultant to jump start your first project. It will put you months ahead of the curve and provide your staff with experience they might spend years obtaining. 

Evaluating test tools

You can develop poor automated tests with a good testing tool but you can’t write good automation with a poor one. It makes sense to spend a couple of days evaluating a testing tool against the application that it will be used to test. Be certain that the tool under consideration can drive the application under tests through the keyboard and mouse and that it can collect information about the state of the application for getting test results. It is especially important to verify the interaction of the test tool with any custom or third party objects. It is also important to ensure that the testing tool has specialized support for environments like ActiveX or Java if applicable. When evaluating ease of use, keep in mind that what appears easy upon initial usage might not be easy when it comes time to maintain tests in the future. It is a big mistake to judge ease of use based on the robustness of a test tool’s recorder. Most users outgrow the recorder after the first couple of months. Instead evaluate the testing tool based on features like:

• Object recognition
• Powerful scripting language
• Built-in support for your environment (e.g. Java, ActiveX)
• Built-in recovery system

Evaluating the application under test

During your test tool evaluation, make notes with regard to the friendliness of the application under test to automated testing tools. Answer the following questions:

• Does every window/page have a unique caption?
• Do controls behave in non-standard ways?
• Are third-party packages implemented that do not expose an API?
• Can an API be exposed for custom objects?

Organizations that rely upon test automation to provide better coverage and collapsed test cycles, evaluate the impact of design decisions on test automation and make changes to promote a better fit between the application and the test tool. For example if there is a choice between third-party grid packages, choose one with a public API so that the test tool can access its methods and properties.

Developing automation-ready test plans

Depending on your existing test planning practices, you may or may not have test plans that are optimized for test automation. In general, test plans which are scenario based are not automation-ready. Instead, it is recommended to adopt a format which is more logically based such as an outline. Test plans should be written such that similar test cases are grouped together, where the objective of an individual case is clearly distinguished. The following is an example an outline based test plan :

 Perform a Search

A.     Case Sensitive

1.      Partial word

                                                                                   i.      Down

                                                                                 ii.      Up

2.      Whole word

                                                                                   i.      Down

                                                                                 ii.      Up

B.     Case Insensitive

1.        Partial word

                                                                                   i.      Down

                                                                                 ii.      Up

2.      Whole word

                                                                                   i.      Down

                                                                                 ii.      Up

It is also important to add data or references to the input and expected results of each test so that the automation engineer has all the necessary information to build an automated test.

Deciding what to automate

Deciding what to automate might be the most important factor in the success or failure of your automation projects. There are aspects of any application that are relatively easy or relatively difficult to automate. Some factors include use of non-standard controls, non-persistence of data and changing application context over time. Here is a place where the experience is the best teacher. When evaluating what to automate, use the following criteria to help select good candidates:

• High priority / data intensive areas that follow a common processing path
• Core (critical) functionality
• New or problematic areas of code
• Labor intensive activities where there are a lot of setup steps
• Areas that are difficult to execute manually (where timing or duration are factors)
   

• stress

• concurrency

• performance

• memory leaks
   

• Easily automated features
• Broad objectives - only where common code can be reused across application

Building a strong foundation

A good set of automated tests that can be developed quickly and readily maintained over time requires a strong architecture and methodology. While the subject of architecture could be written about in volumes, any sound approach must support the following:

• Division of labor
  • Test plans and data provided by subject matter experts
  • Automation developed by automation engineers
  • Tests potentially run by other staff members
• Data driven tests
  • Relatively few test scripts to number of test cases
  • Permutations of tests can be created by adding data sets

Designing good test cases

Good test case design is important for both manual and automated testing. A test case satisfies a requirement in a test plan. It should have a single objective and result in only one of two dispositions: pass or fail. The disposition of a test should always be determined by comparing the actual results to the expected results. A well designed suite of tests can be run in any order and adhere to the following golden rules:

• Test cases are independent
• No test case relies on the successful completion of another test
• Test cases start and stop at a known ‘base’ state

Preparing test data

The ability to control test data in the test environment is important for both manual and automated testing in order to ensure the validity of expected results. When tests are automated, it is even more critical because valuable time is lost whenever tests fail solely because the data state of the environment has changed. To promote successful test runs and ensure validity it is important to be able to have the:

• ability to set up day 0 database
• ability to reset database prior to test execution
• ability to simulate data feeds

It is also important to prepare a separate set of data for each test case to ensure that the result of one test case does not impact the result of another. For example, it is a design error to allow one test case to create a piece of data that will be consumed by another test. Furthermore, a system of data ownership should be established in the test environment, such that a piece of data is associated with a specific test. It’s a good idea to set up data naming conventions so that other testers do not accidentally modify a piece of data that is reserved for an automated test. For example, you might name an account used for automated testing with a first name of “Automated” and a last name signifying the test case that owns it.

Preparing the test code environment

Test code should be treated with the same regard as application code. Aside from the cost of developing it, testing code contains knowledge about how the application is expected to perform. Automated tests will likely remain after the engineers that built them have moved on to other projects or organizations. Test code should be managed through a source code control system just like application code. It makes sense to use the same system and employ the same tagging mechanism in case a prior version of the application and tests need to be recreated at some time in the future. It is also important to develop and enforce a set coding standards and naming conventions to promote efficiency and orientation of new team members.

Setting up a lab

When preparing the budget for test automation, remember to include costs for a lab where automated tests will be run. To determine the initial number of machines required, consider the different OS/Browser configurations that your application will support. Remember that although it is possible to have multiple versions of Netscape on a given machine, only one version of MS Internet Explorer can be installed on a given machine without having to re-ghost between test runs. You should also consider the desired through-put. Will you want to run the entire regression test over night or can a full suite of tests be run over the course of several days? Answers to these questions will help to determine the initial number of machines required.

Scheduling tests to run

Automated tests provide the best ROI when they are run frequently. Ideally tests should be run after each new build of the application under test to provide immediate feedback. It is much easier to find and fix a bug when developers are notified right after it is introduced. When tests are run regularly it is also easier to maintain them as small changes are made to the application. Doing so also provides the QA engineer with feedback regarding what has changed in each build. To ensure that tests are run on a regular basis, consider using a job scheduler to run them automatically. Better yet, integrate automatic test runs with application builds.