FAQ

How can you verify web pages with dynamic content?

Detail

Techniques for Automating Dynamic Web Applications

This FAQ address a problem typical of testing web applications – the need to verify web pages with dynamic content. Although SilkTest is very capable of detecting HtmlLinks, HtmlText, HtmlTables, etc., what it cannot do is anticipate how changes in content will relocate and rename the controls used to display dynamic data. For instance, you may know that the items which are beneath a specific header are the text items you need to verify, but if the content and indices are variable, it is nearly impossible to “find” them using a traditional window tagging scheme. Clearly a new approach is required for these highly dynamic pages.

A useful technique for grabbing information from a highly dynamic page such as this one is to select all content on the page, copy it to the clipboard, and parse it.

As an example, let’s look at the Yahoo! Finance page. The page looked like this (on the day I looked at it). Obviously the next day it would look very different!

Let’s look at the Market Summary section of the page. You may want to write automated scripts that collect the statistics of the Dow, Nasdaq, etc. and verify them against another source to verify that the data is being correctly presented.

When capturing the page with SilkTest, this task may seem formidable. In this particular situation (this content, this browser, this resolution, VO extension, etc.) SilkTest saw an HtmlTable where the first column would contain the links “Dow”, “Nasdaq”, etc., and column 2 contained the text “9910.45 +99.30 (+1.01%)”, “1988.86 +35.69 (+1.83%)”, etc. Although this would possible to work with, it would be easier if the data had been seen as a table with 4 columns: “Dow”, “9910.45”, “+99.30”, “(+1.01%)”, and so on.

Continuing on with this section, the items “NYSE Volume” and “Nasdaq Volume” were seen as links on the page, not in the table, and the associated volumes “1,054,876,000” and “1,549,540,000” as HtmlText elements on the page. These controls are very difficult to tag, as both their text and indices are subject to change.

What can make this problem even more difficult is that as the page layout changes from day to day with a different set of content, SilkTest might not group these controls into a table at all, or might see the table with a different number of columns.

Now, lets grab the info onto the clipboard. The easiest way to write the parser is to open a MS Notepad window and paste in the text so that you can examine it. Go to the web page, type “<Ctrl-A>” (Select All), “<Ctrl-C>” (Copy), and then paste it into the Notepad window. Locating the appropriate section, you will see that it looks like this:

Market Summary

Dow 9910.45 +99.30 (+1.01%)

Nasdaq 1988.86 +35.69 (+1.83%)

S&P 500 1135.98 +12.89 (+1.15%)

10-Yr Bond 5.274% +0.113

NYSE Volume 1,054,876,000

Nasdaq Volume 1,549,540,000

The following code can be used to parse the page for the market summary information:

VOID ParseMarketSummary ()

INTEGER iLoop

LIST OF STRING lsClipboard

// Capture everything on the page onto the clipboard

YahooFinance.TypeKeys ("<Ctrl-a>")

YahooFinance.TypeKeys ("<Ctrl-c>")

lsClipboard = Clipboard.GetText ()

for iLoop = 1 to ListCount (lsClipboard)

if Trim (lsClipboard[iLoop]) == "Market Summary"

// the Dow line is the second following line

Print ("The value of the Dow index is:", GetField (lsClipboard[iLoop + 2], " ", 2))

Print ("The change in the Dow index is:", GetField (lsClipboard[iLoop + 2], " ", 3))

Print ("The % change of the Dow index is:", GetField (lsClipboard[iLoop + 2], " ", 4))

// the Nasdaq is the third following line

Print ("The value of the Nasdaq index is:", GetField (lsClipboard[iLoop + 3], " ", 2))

Print ("The change in the Nasdaq index is:", GetField (lsClipboard[iLoop + 3], " ", 3))

Print ("The % change of the Nasdaq index is:", GetField (lsClipboard[iLoop + 3], " ", 4))

// the S&P 500 is the fourth following line

Print ("The value of the S&P 500 index is:", GetField (lsClipboard[iLoop + 4], " ", 3))

Print ("The change in the S&P 500 index is:", GetField (lsClipboard[iLoop + 4], " ", 4))

Print ("The % change of the S&P 500 index is:", GetField (lsClipboard[iLoop + 4], " ", 5))

// the 10-Yr Bond is the fifth following line

Print ("The 10-Yr Bond is paying:", GetField (lsClipboard[iLoop + 5], " ", 3))

Print ("The change in the 10-Yr Bond is:", GetField (lsClipboard[iLoop + 5], " ", 4))

// the NYSE Volume is the sixth following line

Print ("The NYSE Volume is:", GetField (lsClipboard[iLoop + 6], " ", 3))

// the Nasdaq Volume is the seventh following line

Print ("The Nasdaq Volume is:", GetField (lsClipboard[iLoop + 7], " ", 3))

break

return

Running this results in the following:

Testcase One - Passed

The value of the Dow index is: 9910.45

The change in the Dow index is: +99.30

The % change of the Dow index is: (+1.01%)

The value of the Nasdaq index is: 1988.86

The change in the Nasdaq index is: +35.69

The % change of the Nasdaq index is: (+1.83%)

The value of the S&P 500 index is: 1135.98

The change in the S&P 500 index is: +12.89

The % change of the S&P 500 index is: (+1.15%)

The 10-Yr Bond is paying: 5.274%

The change in the 10-Yr Bond is: +0.113

The NYSE Volume is: 1,054,876,000

The Nasdaq Volume is: 1,549,540,000

The data from the page could also be stored in variables if needed or used in Verify statements.

The parsing code can often be written in different ways. In this case, rather than keying only off of the string “Market Summary”, the strings “10-Yr Bond”, etc. could also be used. Also, if there were not always a single blank line between “Market Summary” and the other data, this would not be hard coded.

Note: When the objects on the page are all selected, SilkTest’s object recognition on the page may (and usually does) change. After you copy the page contents, you may want to click elsewhere on the page to clear the selection.

The dynamic data which is captured this way can be used to drive the application, as well as to verify it. To demonstrate this, let’s look at another section of the Yahoo! Finance page:

Let’s suppose that you are testing the Business News section, and that you need to automate following all of the links in this section of the page. These links will change daily and may not have the same index from day to day or browser to browser.

Looking at the clipboard text for that area of the page yields the following:

Business News

Stocks Up on Hopes of 'Santa Claus' Rally

Mon 3:22pm ET

Stocks kept a firm grasp on the session's gains on Monday, with investors cheering several large merger deals supporting the market, which heightened the possibility of a December rally.

Amgen to Buy Immunex, Gaining Enbrel

Mon 2:20pm ET

Amgen Inc. said on Monday it will buy rival Immunex Corp. for about $16 billion in cash and stock in the largest-ever biotechnology merger, giving it popular rheumatoid arthritis drug Enbrel.

Pfizer Goals in Line with Prior Estimates

Mon 3:08pm ET

Pfizer Inc., the world's largest drugmaker, on Monday set earnings expectations in line with previous forecasts, and said it plans to spend about $5.3 billion in research and development next year on experimental medicines.

Shoppers Keep Wallets Shut Ahead of Xmas

Mon 1:21pm ET

American shoppers again proved picky in their spending over the weekend, ignoring even some of the last-minute pre-Christmas specials on apparel and other luxury items, as worries over the faltering U.S. economy continued to weigh on sentiment.

Airline Traffic Down Sharply in November

Mon 3:22pm ET

Both domestic and international traffic on U.S. airlines fell sharply in November from the year-ago period as the effects of the Sept. 11 jetliner attacks lingered, a U.S. airline industry group reported on Monday.

more business news...

The links can be parsed out as follows:

VOID ParseBusinessNews ()

INTEGER iLoop, iLoop2

LIST OF STRING lsClipboard, lsNewsLinks

STRING sLine

// Capture everything on the page onto the clipboard

YahooFinance.TypeKeys ("<Ctrl-a>")

YahooFinance.TypeKeys ("<Ctrl-c>")

lsClipboard = Clipboard.GetText ()

[-] for iLoop = 1 to ListCount (lsClipboard)

[-] if Trim (lsClipboard[iLoop]) == "Business News"

// the next line is the first piece of Business News

lsNewsLinks = {Trim(lsClipboard[iLoop+1])}

// look for more links

[-] for iLoop2 = iLoop + 2 to ListCount (lsClipboard)

// when the line 'more business news' is found, section is done

[-] if Trim (lsClipboard[iLoop2]) == "more business news..."

break

[-] else

// the links are on the lines following blank lines

[-] if Trim (lsClipboard[iLoop2]) == ""

[-] if Trim (lsClipboard[iLoop2+1]) != "more business news..."

ListAppend (lsNewsLinks, Trim (lsClipboard[iLoop2+1]))

break

ResPrintList ("Business News Links:", lsNewsLinks)

return

The result of executing this function is:

Testcase Two - Passed

Business News Links:

Stocks Up on Hopes of 'Santa Claus' Rally

Amgen to Buy Immunex, Gaining Enbrel

Pfizer Goals in Line with Prior Estimates

Shoppers Keep Wallets Shut Ahead of Xmas

Airline Traffic Down Sharply in November

Once located, the string for the links can be used to identify the control using dynamic instantiation. The link can be followed using the statement “YahooFinance.HtmlLink(‘Amgen to Buy Immunex, Gaining Enbrel’).Click (). Furthermore, each link could be followed after parsing for them:

for each sLink in lsNewsLinks

YahooFinance.HtmlLink(sLink).Click ()

Browser.Back.Click ()

If you are testing a web application, experiment with these techniques. Capture the page and look at it with a Notepad window to locate patterns that can be easily parsed. This approach is usually browser-independent, robust, and quickly implemented.