Customer Profile

NeuVis, Inc., headquartered in Shelton, CT, offers an Internet Rapid Application Development (I-RAD) platform that provides customers with sustained e-business leadership. The NeuVis I-RAD platform is a shift from the traditional, time consuming, and expensive programming-based development model to an architected, engineered model that uses visual tools, automated construction, and an advanced software engineering methodology to rapidly create high quality, highly flexible e-business applications. I-RAD represents the necessary evolution of traditional Rapid Application Development approaches into the Internet era.

NeuVis' I-RAD platform allows creation of e-business applications that are:

• brought to market quickly;

• technology neutral;

• less costly to maintain;

• highly flexible and adaptable;

• protected from technology obsolescence.

The platform consists of NeuArchitect (NeuVis’ end-to-end development environment) and NeuFrameworks (including certain industry specific components along with NeuXchange, for buyer, seller and marketplace systems). NeuArchitect is technology neutral and will support the leading Internet deployment technologies, including HTML Web sites, Java applets, Microsoft and Java (EJB) servers, leading databases, PDAs and wireless devices, XML, EDI and other messaging standards.

NeuVis was founded by Arun Gupta, Chairman and CEO, who has twenty years experience in rapid application development (RAD) software. Gupta previously founded DataEase International, a provider of RAD applications, in 1981. The client/server division of DataEase was sold to Symantec in 1992. Gupta left Symantec in 1994 and, that same year, started a new company that became known as NeuVis in 1999. Today, one year after formally launching the company, there are 150 employees. NeuVis’ customers include: Lehman Brothers, SunAmerica Securities, Freshnex.com, Asian Vendors, and a variety of other Global 2000 organizations in the financial services, healthcare and insurance industries. Finally, the company has formal alliances with several partners, including: IBM Global Services, Computer Sciences Corporation (CSC), Cisco, AT&T, CyberSource and ScreamingMedia.

QA at NeuVis

Jim Mayhew is Manager, Quality Assurance. Jim has 22 years of experience in software development, with over 20 years spent in software quality assurance. He has held management positions at Allied Signal, Bunker Ramo, Olivetti, Oxford Health Plans, and Trecom Business Systems. Jim earned his BS in computer science from the University of New Haven, located in New Haven, Connecticut.

Ralph Pietrosimone is Lead QA Analyst, Automation and Tools. Ralph has over 12 years of experience in software quality assurance. He has held various QA positions at Notara, MECA Software, Bayer Pharmaceuticals, Software AG, Executone, and Oxford Health Plans. His areas of expertise are Internet technologies, QA automation, databases, and Windows NT administration and programming. Ralph earned his BS in computer science from Southern Connecticut University, also in New Haven, Connecticut.

NeuVis utilizes SilkTest and SilkRadar for their QA testing processes. [ed. note: SilkPerformer is also used in NeuVis’ deployment division.] QA is a dedicated function at NeuVis and Jim’s QA department currently contains ten engineers and two interns. "Typically we have one major release per quarter. Our next release will be version 3.0, due out around the end of the year," says Jim. "Within QA, we have three different testing strategies. We have a smoke test, regression test and acceptance test. We receive new builds daily from development so the first test we run is a smoke test against the new build. The smoke test verifies all controls, the state of the controls, navigation, and generates Web pages and databases. If the new build doesn’t pass the smoke test, the build goes right back to development. If it passes, we accept the build. Then, once testing commences, we will begin by testing all of the new features added to a release, to be sure they work as per their specification."

"During the release cycle we will run two separate regression tests. The regression test verifies that the existing functionality performs properly. The first regression test starts part way into the QA cycle and checks all of the existing functionality. Then, toward the end of the QA cycle, we run the regression test again. This second regression test is generally run once we have completed testing of the new features and we are satisfied that the defect level is within our guidelines for release," explains Jim.

Ralph adds, "We have automation suites that supplement each manual regression. We have a database suite that allows us to easily change databases, and examine what our product creates (tables, columns, indexes) in each one. Then the tests create Web pages that hit the chosen database and perform loads, inserts, updates and deletes, so that the database portion is tested end-to-end. Our GUI/feature suite grows as our manual test cases grow. NeuVis tracks features, test scenarios, test cases and test results. So, as manual test cases are written, they are taken by the automation team and automated. Then, the analyst who wrote the tests becomes the person who runs and verifies them using Silk."

"Finally, the controls that are used to create Web pages have been automated into a suite. Each control or series of controls are dropped onto a page and the different properties for the controls are generated against multiple technologies. These are checked via Silk on the generated Web pages. Because of the enormous matrix of control properties, this test saves a large amount of manual test hours," concludes Ralph.

QA has the final say on release readiness, according to Jim. "Before a release can become GA, and for any maintenance release, it must pass our third series of tests - our acceptance test. Our acceptance test is an automated test that covers about 25% of our application from beginning to end. Our acceptance test takes a cut through our application and verifies that what was previously working is still performing correctly."

The QA Test Tool Selection Process

NeuVis initially evaluated Segue and two other automated test solutions. All three solutions were brought in-house and a proof-of-concept was run with each vendor. "We had three problems we needed to solve," explains Ralph. "First, was simply a scarcity of resources. It is extremely difficult to find experienced staff in today’s job market. We needed to have a solution in place that would allow us to optimize testing productivity. Second, we need to support a large number of technology platforms. As you can see, the number of different configurations supported is immense. We did not want to have to create an entire new set of tests for every supported configuration. Finally, we are an Internet infrastructure company. We must keep pace with the latest technology – and, on the Web, new technologies are emerging all the time. We need to be able to change quickly. Our technologies, application functionality, and GUI all change constantly."

NeuVis ranked Segue the winner in the evaluation. Ralph explains that SilkTest provided a superior solution to NeuVis’ test automation issues, "In short, Segue’s underlying architecture was the factor in providing a solution to all three of our issues. SilkTest is based on a true object-oriented architecture and provides a powerful scripting language. With a record-and-playback paradigm, we would have a different set of scripts for every platform we support. With SilkTest, we have one set of scripts for all supported platforms. Furthermore, when something in our application changes – and, as I just mentioned, our GUI changes all the time – we don’t have to scrap the efforts already expended in developing our automated tests. SilkTest provides a high level of reusability – we easily maintain the existing tests that have already been created. Finally, SilkTest is fast! The tests run quickly, efficiently and unattended."

Automation Gains with Silk

SilkTest and SilkRadar were purchased very early on in the formation of the QA department. Prior to SilkTest, all testing was performed manually. SilkTest was installed in early 2000, and Jim explains the progress NeuVis has made with automation to date, "Our regression tests now take two and one-half weeks to run. This is down, significantly, since we implemented SilkTest. We closely track manual testing time so that, as the tests are automated, we have an instant accounting for our automation ROI. We have quantified the time saved by utilizing SilkTest for regression testing at 200 manual hours per regression test run. We run two sets of regression tests per release, so our Silk automation is saving us a minimum of 400 hours throughout each release cycle. Those kinds of savings are very impressive and translate directly to the bottom line. Additionally, we can kick-off the regressions to run, unattended – which also means we can be doing other things while they are running."

Jim continues, "In ten months of using SilkTest, we have automated 33% of our application. The productivity increases have already paid for the cost of the tool. Automation has really allowed us to leap forward in our ability to test and to cover more of our code. And, we are expanding our automation efforts all the time, so we will continue to realize substantial gains. This increase in productivity translates right to the bottom line. Furthermore, utilizing Silk, I have not only maximized the productivity of my staff, but I am able to meet the business demands of our very dynamic industry."

"By utilizing the queries and reports that we have in SilkRadar, we can track many different metrics. We can see how many defects were found by automation and we can also track how well our smoke, regression and acceptance tests run. As I indicated, we have taken substantial manual test time off the board," explains Jim.

The Implementation Process and Star Quality

Because of the shortage of QA personnel throughout the industry, NeuVis has utilized Star Quality, a Segue consulting partner, throughout their SilkTest and SilkRadar implementation project. Brian Le Suer, co-founder, Star Quality, has worked with NeuVis throughout NeuVis’ implementation. Jim elaborates on the value Star Quality has brought throughout the automation project, "We selected SilkTest and implemented the Star Quality methodology, because together they allow us to easily adapt our test scripts to changes in the GUI and/or any new features added by development – a key goal for us. Star Quality has been instrumental in the success of our project. Brian came in at the beginning of our automation project and was immediately productive. He really provided a lot of expertise and guidance to be sure we maximized our use of Segue’s solutions right from the start. Star Quality brought us a defined methodology; they are experts in the field of automation. Utilizing Star Quality has insured that there was no wheel-spinning – which can commonly occur any time you bring in a new tool."

Brian explains the Star Quality methodology, "Our methodology is both an approach and a technology integrated with SilkTest. Our philosophy with regards to automation productivity is simple: the less test script code to write and maintain, the better. Two of the functions provided in Star Quality’s toolkit have been used heavily at NeuVis to reduce the amount of test code needed to a minimum and ensure the return on investment that Jim and Ralph require. The first method inputs data into any form or screen. The second method verifies application responses and compares them to expected results. With little code to write and maintain, NeuVis has been able to quickly develop tests simply by creating data records, which describe the inputs and expected results. These methods have been key to NeuVis’ automation success. NeuVis’ application is a very large one, containing many screens. Because the GUI changes constantly, our test infrastructure allows us to modify our tests quickly, to validate each new build. The key reason we have been able to do this with SilkTest is 4Test - the robust, flexible scripting language offered within SilkTest. No other testing solution offers a scripting language with the power and flexibility of 4Test."

Other Gains Realized with Automation

Jim describes some of the other things NeuVis has been able to accomplish with SilkTest, "We have also tried to creatively apply the use of automation in other areas of our business. For example, using SilkTest, we have automated our training materials. When our instructors are teaching a class, they have a series of 29 Training Workshops, or exercises, that a student must go through, during training. We also use these training materials internally, to train our own employees on a new release, and externally, for our partners. Obviously, it is imperative to have the Training Workshops ready to go as soon as possible once we are ready to issue a new release – both NeuVis and our partners have to be able to support customers immediately on it. Using SilkTest, we run these workshops against the release to be sure they all still work accurately and reliably. The quality of our training has improved greatly since we automated the testing of our Training Workshops."

Brian elaborates on a specific technique used to test the Training Workshops, "We run the Training Workshops in unattended mode, with SilkTest. In a typical SilkTest automation project, SilkTest’s built-in recovery system is used to return the application under test to a stable base state, if it encounters an application error, and then continue on with the next test. This feature of SilkTest provides superior functionality for unattended testing. In this particular project, we actually disabled that feature for testing within each Workshop; here is why. Each of the 29 Workshops contains steps that actually rely on the successful completion of preceding steps. To ensure that subsequent steps would not execute in the event of an error, we tailored the recovery system to look for a positive return code from the preceding test to determine whether or not to continue. Because SilkTest provides such a high level of flexibility, we were able to tell SilkTest not to continue with that specific Workshop if it encountered an error. Instead, it returns to a stable base state and continues on with the next Workshop exercise. Again, no other regression tool would be able to handle this testing requirement as comprehensively as SilkTest."

Ralph has also created automated tests to examine NeuVis’ load balancing feature. Using a simulation application and Silk’s distributed testing capabilities, the test checks NeuArchitect’s load balancing feature by changing the registry settings and placing a load on the chosen machines. Silk reports back each machine’s load and the machine NeuArchitect chose to hit. This feature is extremely important, since Windows NT does not handle this like UNIX platforms.

In Summary

Jim describes the atmosphere within the QA group, "It is a great group. We have a team of very talented, hard working professionals and we like to do fun things together. For example, we all go out to lunch regularly. NeuVis, at a corporate level, is also a very enjoyable place to work. One example that I will share is our Crazy Shirt Fridays. Every Friday employees wear crazy, wacky shirts to work. We then have a judging contest and a first, second, and third place winner is chosen. They each receive prizes. Here in QA, we have had our share of winning shirts!"

Jim concludes with what he views as the greatest achievements the QA team has realized, "I am particularly proud of what we have accomplished to date in our SilkTest and SilkRadar automation. And, simultaneously with implementing automation, we have continued to meet the business demands we are tasked to meet. We have been able to test so much more than we were able to test - prior to automation - in much less time. The combination of Silk, Star Quality, and our own internal efforts has made this project a very rewarding one, for us, and a lucrative one for both NeuVis and our customers."