FAQ

How can you prevent cascading errors?
(getting back to the base state

Detail

The recovery system is one of Silk Test’s best kept secrets. And that’s by design, because the recovery system is invisible to the user … or nearly so. Built into the product since its inception in 1992, the recovery system attempts to get the Application Under Test (AUT) back to a known base state after a testcase has been executed. This feature protects a suite of tests from the “domino effect” where a failure in one testcase causes the entire run to fail. Without it, the user would be required to add recovery code to each testcase and then hope for the best. In short, the recovery system was designed to support unattended testing.

What does the Recovery System do?

By default the recovery system carries out four tasks for client-server applications:

  • Ensures that the application or browser is running
  • Restores the application if it is minimized
  • Sets the application active
  • Closes any application child windows or dialog boxes

In addition to these tasks, the recovery system does the following for browser-based applications:

  • Waits for the browser to become ready
  • Ensures that browser toolbars, location textfield and status bar are displayed

How does the Recovery System Work?

The recovery system is stored in a file called defaults.inc in the directory where SilkTest is installed. Most of the action happens in an appstate function called DefaultBaseState (). This is the appstate that testcases are based upon by default.

The recovery system gets invoked automatically when a script or testcase is run. The flow of control differs slightly if tests are run from a testplan vs. from a main function in a .t file. When running testcase through SilkOrganizer (via a testplan), control first passes to the testcase that is being run. Before the testcase executes, the DefaultTestCaseEnter function is called. It calls SetAppState which by default calls DefaultBaseState. Then the testcase begins to run. If the testcase runs to completion or if an error is encountered that is not trapped with a do .. except, control passes to DefaultTestCaseExit. It logs any exceptions raised in the testcase and calls SetBaseState which by default calls DefaultBaseState.

When running tests through a main function in a .t file, there are two other recovery system functions that are invoked in a similar fashion. DefaultScriptEnter is called before the first testcase is run. It is NULL by default. DefaultScriptExit is called after the last testcase completes. It logs any errors that may occur outside of a testcase.

How can the Recovery System be configured?

As noted above, the recovery system functions are defined in defaults.inc. But there are times when the default functions are not sufficient to return the application to its base state. In some cases there are windows that need to be closed that are not children of the AUT’s main window. In other cases, it may be necessary to leave child windows open as in the case of a floating tool palette. Rest assured, SilkTest provides an array of ways in which the user can configure the recovery system. Here is a summary of the primary options:

  • wStartup is a member constant WINDOW of the AUT’s MainWin declaration. It can be assigned the 4Test identifier of the first window to be displayed when the application is invoked. This configures the recovery system to expect that window instead of the AUT’s MainWin.
     
  • wLeaveOpen is a member constant LIST OF WINDOW of the AUT’s MainWin declaration that can be assigned a list of 4Test identifiers of the windows that should not be closed by the recovery system.
     
  • Close Methods can be added to the declaration of any window that requires custom steps to successfully close the window. Keep in mind that the signature for your Close methods must always match that of the default: VOID Close (BOOLEAN bConfirm optional, BOOLEAN bSetActive optional). If you do not include the optional BOOLEAN arguments, the recovery system will encounter an error when calling your methods because the argument list will not match. Unfortunately the error is trapped within a do .. except statement in DefaultBaseState and is not reported. If a Close method has not been defined for a window to be closed, the recovery system will call a default Close method that can be configured via the Close TAB in the Agent Options dialog.
     
  • BaseState is an optional method that you can include in the window declaration of the AUT’s MainWin. Application specific steps to be carried out by the recovery system beyond the default tasks should be included in this method. If a BaseState method is defined it will be called by DefaultBaseState after the default tasks are executed.
     
  • AppStates are not intended to recover the AUT to a base state but rather transform it from one state to another. They provide a convenient way to store application tasks needed to get the application from the base state to a test state so that the code can be easily reused by many testcases. Appstates are invoked simple by appending the keyword appstate to a testcase and providing the name of the appstate function, e.g.

testcase MyTestCase appstate MyAppState

To declare an application state function use the keyword appstate. It is also possible to base an appstate upon another appstate creating chains of setup steps. The following is an example:

appstate MyAppState () basedon MyBaseState

  • Overriding default recovery functions – Sometimes there are application specific steps that need to be carried out before or in lieu of the default recovery tasks. By defining functions named TestCaseEnter, TestCaseExit, ScriptEnter and/or ScriptExit, the user is able to override the associated default recovery system functions. When these functions are defined, they are called instead of their default counterparts. If the default functionality is still desired, the built-in functions can be called from within the user defined overrides. For example to call DefaultTestCaseExit from within your custom TestCaseExit, the code will look something like this:

[-]TestCaseExit (BOOLEAN bException)
      [-] if (bException)
             [ ] ExceptLog ()
             bException = FALSE //Set the switch back to false
                                             //since the error has been reported.
                                             //If you forget to do this, the error
                                             //will get reported again when
                                             //you call DefaultTestCaseExit

       //perform some steps that the default recovery system
       //is not able to do successfully

…

       //now call the default function to complete the recovery tasks
       DefaultTextCaseExit (bException)

Gotchas and Other Tips

  • The recovery system helps to enforce good testcase design by discouraging test engineers from beginning one test where the previous test left off. It is always important to make each testcase independent by starting each from a known base state.
     
  • A testcase is really a function with special behavior. A function prefixed with the keyword testcase will automatically invoke the recovery system.
     
  • If you write your own TestCaseExit function be sure that the first step reports any errors that may have been raised in a testcase. If you forget to do this, all of your testcase will pass because the errors will have been silenced. The following is an example:

[-]TestCaseExit (BOOLEAN bException)
       [-] if (bException)
               [ ] ExceptLog ()
//your application specific code starts here

  • There are several ways to turn off the recovery system. To disable it entirely, remove the const wMainWin from your window declarations file. Often, it is convenient to turn off the recovery system while you are experimenting so that the application does not get returned to the base state before the code you are testing is run. An easy way to do this is to append the keywords “appstate none” to your testcase.
     
  • Unfortunately once you’ve defined a BaseState function or written your own TestCaseExit function, attempting to turn off the recovery system with the keywords “appstate none” will not work. You have to comment the functions out to disable them. Make sure you uncomment them when you’re done experimenting or your team mates will grumble.