FAQ

How do you run smoke or regression tests at a scheduled time?

Detail

The ability to run a smoke or regression test at a scheduled time can be a real asset to a Development or IT organization. Tests can be triggered to run after a daily build is complete or after new content is pushed to a web application each morning. With SilkTest’s test plan queries, its command line interface and a task scheduler such as NT’s “at” or the newer “Scheduled Tasks Wizard” , it’s easy to do. Let’s take a closer look at each of the components.

Testplan Attributes and Queries

SilkTest’s test planning module (a.k.a Silk Organizer) offers the ability to select tests based on an attribute query. Choose between one of the existing attributes (category, component, developer) or create your own. To define a new attribute, invoke the Define Attributes dialog by selecting the Main Menu->Testplan->Define Attributes menu item. In this example, a new attribute called Environment has been added. The values web1 thru web5 were also added as possible values for the attribute.

 

 

 

 

 

 

 

 

The next step is to label sub-plans, sections or individual tests in your test plan with your new attribute values. For example, you might have a sub-plan that contains your web environment smoke test. You can easily select which environments to run upon by labeling your sub-plans by environment and then creating a test plan query to mark them at runtime. The following is an example of how the test plan might be labeled to select a smoke test to run in the desired environment(s):

            optionset: myapp_web1.opt
            environment: web1
            include: mysmoketest.pln 

            optionset: myapp_web2.opt
            environment: web2
            include: mysmoketest.pln 

            optionset: myapp_web3.opt
            environment: web3
            include: mysmoketest.pln 

Using the test plan query feature, a query is created to select each web environment. To create a new query, invoke the Mark By Query dialog by selecting the MainMenu->Testplan->Mark By Query menu item. Select the attribute(s) and value(s) from the Test Attributes Tab and name the query for later use.

 

 

 

 

 

 

 

 

 

To mark the tests based on this query, invoke the Mark By Named Query dialog by selecting the MainMenu->Testplan->Mark By Named Query menu item. This selection provides a means of running only those tests associated with the attribute(s) specified by the query.

 

 

 

 

 

 


 

 

The power of Silk Organizer queries is in the manner in which they can be combined. Through unions and intersections of different queries, virtually any combination of tests could be selected for runtime. For example, additional queries can be created by combining the web1 thru web5 queries so that the smoke test could be run against any combination of environments. 

Understanding SilkTest Command Line Options

Now that we have a testplan labeled with attributes and queries to select upon them, the next step is to learn how to launch SilkTest from the command line. The SilkTest help provides a syntax guide with an explanation all of the configuration options. For our purposes, only the following configuration options are required: 

partner [[-q] [-query query name [-r] scr.t/suite.s/plan.pln]

The first term in the command line string, “partner”, is the name of the silktest executable. What follows are two switches and their arguments. The first switch, “-q” allows the user to name the query that will be used to mark tests for execution. The query name is defined by the user such as “web1” from our example above. The last switch “-r” indicates what to run. In our example case, a testplan is to be run. You can also specify the name of a script or suite file. An example of a command line for our example follows:           

partner.exe -query web1 –r smoketests.pln

This command line string can be improved by using environment variables to specify the location of the partner executable and testplan to be run:

"%SEGUE_HOME%\partner.exe" -query web1 -r "%SCRIPTDIR%\ smoketests.pln"

This command line string will run successfully on any machine regardless of the location of the SilkTest executable or the test plan to be run.

Scheduling Your Test Jobs

There are two ways to schedule your test jobs. If you are using Windows NT and have not installed Internet Explorer 5.x, your machine is probably using an executable called atsvc.exe as its task scheduler. This scheduler is also known as “at”. To use at, you’ll want to save your command line as a batch file. To do so, simply paste your command line string into a notepad file and save it with a .bat extension, e.g. mysilktestjob.bat. Then complete the following steps:

  1. Bring up a command prompt.
  2. Type: at time /interactive /every:days “%SCRIPTDIR%\ mysilktestjob.bat”

Where time = the time of day to schedule using a 24 clock and days = the days of the week to schedule the task using M,F,T,W,Th,F to represent the desired days. For example: 

at 06:00 /interactive /every:M,F,T,W,Th,F “%SCRIPTDIR%\mysilktestjob.bat”

  1. To check on the status of your scheduled jobs, simply type at without any arguments.

There are some security issues that you may encounter along the way because the at scheduler uses the system account to launch jobs. For example, if your test attempts to delete browser cookies, you will likely not have adequate permissions. To resolve this problem, change the location where the browser stores temporary internet files from the default of “all users” to a location where the current user has read and write permissions, e.g. “C:\WinNT\Profiles\youraccount\Temporary Internet Files”. Another limitation is that you will not be able to copy files to other machines on the network because the network does not recognize the system account on the local machine. Although Microsoft’s documentation suggests that you can configure the at scheduler to use an alternate account with adequate permissions, the facility appears to be disabled.

If you are using Windows2000 or Windows NT and have installed Internet Explorer 5.x  your machine is probably using an executable called mstask.exe as its task scheduler. This executable provides a mechanism for using an alternate account to resolve the permissions issues described above. It also provides a handy wizard for scheduling your tasks. To schedule your test job, simply complete the following steps:

  1. Doubleclick MyComputer on the desktop
  2. Doubleclick on Scheduled Tasks
  3. Doubleclick on Add Scheduled Task
  4. Follow thru the steps of the wizard:

  • Select SilkTest from the list of applications

  • Select Daily

  • Select Weekdays unless you want the task to run on the weekend as well

  • Select the desired start time of the task

  •  Enter the user id and password of the user to run the task as (typically your own)

  •  Check the CheckBox to invoke the advanced properties dialog

  •   Complete the Run field per this example: "%SEGUE_HOME%\partner.exe" -query   web1 -r “scriptdir location on your machine\smoketests.pln"

  •  Complete the Start in field as follows: % SEGUE_HOME %

The next time you get home and realize that you forgot to kick off the daily regression, consider this automatic scheduling alternative. It could save you a day on your schedule.