Choosing an Automated Testing Tool

Choosing an Automated Testing Tool
This guide will help you find the best testing tool for your needs.
by Karen D. Schwartz

Posted September 29, 2003

Companies are turning to automated testing tools more often than ever before to ensure that their applications are working properly prior to deployment. That's particularly important today, because more applications are written for use on the Web—the most public of venues. If a browser-based application crashes or performs improperly, it can cause more problems than a smaller, local application.

But for many IT and quality assurance managers, the decision of which testing tools to use can cause confusion. The first decision is which category of tool to use—one that tests specific units of code before the application is fully combined, one that tests how well the code is working as envisioned, or one that tests how well the application performs under stress. And once that decision is made, the team must wade through a variety of choices in each category to determine which tool best meets its needs.

""Evaluating your needs will narrow down your short list very rapidly," says Graham Titterington, principal analyst at Ovum, a UK-based consulting firm. "An Enterprise Resource Planning (ERP) package tends to need special protocols and hooks, for example, and some tools will have them and some won't."

This guide will help you sort through some of the many options. Choose from:

Developer-Oriented Tools
Tools in this category, also called component testing tools or unit testing tools, test individual software components or groups of related components, helping isolate and rectify problems as early in the development process as possible. Areas of testing include memory analysis, function analysis (to determine if the product has bottlenecks), and optimization of those areas. Key products include:

DevPartner Studio (Compuware Corp.) allows developers to test and debut their .NET and native Windows code prior to formal functional-testing. Through static source code analysis, developers test code against a set of more than 600 rules identifying errors, performance issues, poor software engineering practices, and adherence to group coding standards. Other features include error detection and code coverage analysis, .NET memory analysis, and performance analysis.
PurifyPlus (IBM Rational) focuses on white box testing (also called open testing), which involves getting into the code to determine where memory might be leaking, where bugs might be lurking, and where bottlenecks might be occurring. Another aspect is development of the component, or unit. Developers use Rational XDE to model, develop, and generate code for those components.
Optimizeit Enterprise Suite (Borland Software Corp.) . Developers can use this tool for Java-based applications to isolate J2EE performance issues during the development phase, with a combination of code-level performance tools and J2EE system component-level performance profiling, rather than locating potential problems during quality assurance.
C++Test, Jtest, and .TEST (Parasoft). C++Test allows developers to perform unit testing and static analysis on C or C++ code to determine if it is working immediately after being written. Jtest does the same for Java code. .TEST tests that each module of a Web service being written on the .NET platform is working correctly.

Functional-Testing Tools
Tools in this class help verify that applications will work as expected. At a functional level, they provide capture/replay capabilities, which allow developers to record an existing application and modify scripts to meet changes in an upcoming release. They also provide for regression testing on the new release using the test scripts developers have captured up until that point. In addition, these tools enable developers to capture scripts at the network protocol level so they can formulate load and performance tests on large user bases. Tools in this category include:

WinRunner (Mercury Interactive Corp.) provides a relatively simple way to design tests and build reusable scripts without extensive programming knowledge. In addition, WinRunner allows testers to compare outcome and identify potential problems in a variety of functions. According to Mercury Interactive, WinRunner supports more than 30 environments, including Web, Java, and Visual Basic. It also provides solutions for leading ERP and Customer Relationship Management (CRM) applications.
Astra QuickTest (Mercury Interactive). This functional-testing tool is built specifically to test Web-based applications. It helps ensure that objects, images, and text on Web pages function properly and can test multiple browsers. Astra QuickText provides record and playback support for every ActiveX control in a Web browser and uses checkpoints to verify specific information during a test run.
SilkTest (Segue Software Inc.) tests both mainframe and client/server applications. The tool includes single-test scripts that can run unmodified across multiple browser and single-recording testing for cross-platform Java testing. It also provides facilities for rapid test customization and automated infrastructure development. SilkTest works seamlessly with other Segue testing products to produce lifecycle testing capabilities.
Rational Suite TestStudio (IBM Rational). Rational Suite TestStudio is a full suite of testing tools. Its functional-testing component, called Rational Robot, uses automated regression as the first step in the functional-testing process. The tool records and replays test scripts that recognize objects through point-and-click processes. The tool also tracks, reports, and charts information about the developer's quality assurance testing process and can view and edit test scripts during the recording process. It also enables the developer to use the same script to test an application in multiple platforms without modifications.
QACenter (Compuware) is a full suite of testing tools. One functional tool considered part of QACenter is QARun, which automates the creation and execution of test scripts, verifies tests, and analyzes test results. Advanced data functions allow developers to abstract input data into an external file, allowing one script to work with various testing scenarios. A second functional tool under the QACenter umbrella is TestPartner, which uses visual scripting and automatic wizards to help test applications based on Microsoft, Java, and Web-based technologies. TestPartner offers fast record and playback of application test scripts, and provides facilities for testers without much programming experience to create and execute tests.
e-Tester (Empirix Inc.) . Part of the e-TEST suite, e-Tester was built solely to test Web-based applications. The tool helps create test scripts without programming and integrates with e-TEST's test management and defect tracking to create an end-to-end testing process.

Load-Testing Tools
This category, also called performance or stress testing tools, tests what happens to the code as the application scales with multiple users in a simulated environment. The tools test, among other things, whether performance degrades as the load is increased. Once bottlenecks are found, these tools can determine the source of the problem and begin fixing it. Key tools in this area include:

LoadRunner (Mercury Interactive). LoadRunner offers a host of capabilities, including WAN emulation support, and support for Java, Citrix MetaFrame, XML and other environments. A Web transaction breakdown monitor helps isolate performance problems, while the tool's data wizard helps developers create data-driven tests quickly. In addition, the new tuning module add-on provides component test libraries and a knowledge base that helps isolate and resolve performance bottlenecks, according to the company. Mercury Interactive's Astra LoadTest is a similar tool for the Web environment.
SilkPerformer (Segue). Developers can simulate virtual users working with different Web, middleware and database protocols. Root cause analysis tools and management reports help this load testing tool isolate problems, and verification under load helps ensure that functional errors are detected under load conditions.
Test Perspective, Load Test Edition (Keynote Systems Inc.) enables developers to test the capacity of their Web-based applications before going live. Developers can download a script recorder from Keynote's testing portal to record their own scripts. Once recorded, developers can then run tests immediately or schedule them for later.
QACenter, Performance Edition (Compuware). This automated testing tool combines load testing, server monitoring and data management tools to establish testing environments, load test applications and monitor the effect of load on underlying servers, applications and databases while pinpointing problems.
e-Load (Empirix). Part of the e-TEST suite, e-Load tests the performance of Web-based applications through reusable visual scripts, a scenario manager that allows developers to define custom load scripts, and application server support for a wide variety of development environments. In addition, it allows users to create data-driven load tests without programming and vary the number and type of users during a performance test.
Rational Suite TestStudio (IBM Rational). Users employ Rational Robot to capture and create both Virtual User and GUI tests, and use TestManager to schedule, execute and analyze the results of the tests. TestStudio allows testers to quickly build complex usage scenarios without programming to emulate user-type activities in sequential, random or synchronized patterns. By weighting the various user groups in the schedule, the tool produces realistic system loads. Once created, the load test can be scaled up to tens of thousands of users without having to create a new schedule for each user count.

Performance Monitoring and Management Tools
Unlike other tools described above, tools in this category are used after the application is already in production by examining how the application is working in a real-world environment. In essence, these tools monitor the production environment to ensure that all requirements and defined thresholds are continually being met. Key products in this category include:

Vantage (Compuware). This performance management solution for distributed applications monitors the application from the end user, through the network, to the server and database tiers. The tool helps determine the precise causes of performance bottlenecks by showing an application interacting with its supporting infrastructure. Using production metrics, Vantage also provides forecasting capabilities for efficient growth management.
VTune Analyzer (Intel Corp.). This tool, which is offered for Java, .NET, and Linux, focuses on performance tuning and optimization. It looks at different configurations for servers and databases, determining how they all work together and making sure different settings are correct to give the most optimal performance. The Sampling capability allows developers to see results on system-wide activity and drill down to source code, while the Counter Monitor allows developers to examine a detailed piece of code in relation to performance counters.
Optimizeit Enterprise Suite (Borland Software Corp.). This performance assurance solution for developers includes J2EE profiling capabilities in addition to code-level performance benefits. This allows developers to locate J2EE performance issues during the development phase rather than locating potential problems during quality assurance.
ProTune (Mercury Interactive). This set of tools includes ProTune Delivery Service for tuning and optimizing production systems across the application, database, and infrastructure, and ProTune Validation Service, offering full-scale capacity validation testing of Web sites and Web-based applications, helping identify bottlenecks and capacity constraints within a Web infrastructure.
OneSight (Empirix). OneSight is a dedicated production monitoring system is deployed inside a corporation's firewall. This product uses the same transaction engine as Empirix's e-TEST suite and is used for transaction monitoring of complex, dynamic Web applications and Web services.
Tivoli Monitoring for Transaction Performance (TMTP) (IBM Corp.). This tool is designed to extend application development testing from IBM Rational to the production environment. Features include automated transaction decomposition, dynamically discovered transaction flows based on URL or J2EE components, simulated transactions to identify application performance issues, and detailed analysis of actual and simulated transactions to identify specific causes of performance problems versus general infrastructure availability.

About the Author
Karen D. Schwartz has written about technology for nearly 20 years. Her work has appeared in a variety of publications, including CIO, InformationWeek, Business 2.0, Mobile Computing & Communications, NewsForge, ZDNet, PlanetIT, Consumer Goods Technology, and Government Executive. Reach her by e-mail at karen.schwartz@comcast.NET.