Testing and Performance FAQ

Here are some of the most common questions on testing and performance trends, strategies and tools, as provided by our authors for this special report.

Posted September 29, 2003

Q. What are the three main trends in testing and performance?
A. They are: an increased emphasis on testing for leaks in security and authentication; an increase in outsourcing activity; and the need to test software for use in grid computing systems.

Q. Who are the key product vendors in the testing and performance market?
A. According to the Gartner Group, Mercury Interactive has about 50 percent of the market, followed by Compuware and Parasoft. Other leaders are IBM/Rational, Empirix, Segue, and Keynote Systems. Key "niche" players to watch are Telelogic, Quest Software, Solstice Software, and RadView Software.

Q. What is grid computing?
A. Grid computing is one of the latest trends in enterprise IT systems. Ostensibly, the advantage is that companies can save money immediately by using existing computers together with inexpensive newer computers in a fail-safe network powered by a high-end central database and application servers. This opposes conventional thinking that companies have to continually purchase more-and more powerful-computers and software every few years.

Q. What is Enterprise Software Packaging?
A. Enterprise Software Packaging (ESP) is the preparation of standard, structured automated installations for deployment within a specific corporate environment. It plays a key role within the software lifecycle because it takes charge of all the activities that are related to software package preparation and maintenance. This preparation cycle includes five activities: Request; Integration; Product Testing; Quality Assurance; and Release Management.

Q. What are five areas of an application that you can test and/or tune to take advantage of the target server's capacity for concurrency?
A. The five areas that should be tested every time optimization occurs are: input/output; multi-threading and hyper-threading the scheduling apparatus; "critical sections"—places in a multi-threaded application where only one thread can have access at a time; data access and structures; memory allocation and management.

Q. What is the SPIN model checker?
A. SPIN's author, Gerard J. Holzmann, explains that SPIN is a software verification tool that can help users to find and diagnose subtle, concurrency-related bugs in multi-threaded and distributed software systems.

Tools
Q. What are some of the most common categories of testing tools?
A. Four common categories are developer-oriented tools; functional testing tools; load-testing tools; and performance monitoring and management tools.

Q. What's the difference between a functional tool and a load-testing tool?
A. They are used to test the application in different stages of the development process. Functional testing is usually done first, to test how well the application performs under a variety of circumstances. Once the application passes these tests, it's generally submitted to load testing (also called performance or stress testing). During this process, the application is stressed to its breaking point to see how well it performs when used by many users simultaneously.

Q. How do I decide which tool within a particular testing category to choose?
A. Base your decision on a variety of factors, including whether a tool is known for testing applications similar to yours, whether the tool works well with the languages and protocols you're using, and how technology-savvy your testers are. That's important, because some tools are easier for those with limited programming knowledge to use than others.

Q. Once I purchase the tool, how do I ensure that my team will use it effectively?
A. Develop a test plan that determines how and what you plan to test before you start using the tool. And make sure the vendor sticks around to provide training and answer questions—it helps ensure that tool makes it into wide use instead of being relegated to a shelf.

Q. What type of vendor works best with applications written in specific program languages, such as C++, Java or Visual Basic?
A. All modern testing products today can accommodate widely used programming languages like these, but if your applications have a large amount of code in disciplines like SOAP, consider a boutique software vendor. Parasoft has strong roots in C++, SOAP and Java, while Quest Software Inc.'s Sitraka division specializes in Java. Other specialty vendors may be useful in other situations. Sweden-based Telelogic, for example, offers the Tau product line for communications-related applications.

Q. Are there tools I can use to test my applications after they are live?
A. Yes. Testing tools for the production environment include performance monitoring and management tools, and test management tools. Tools in the first category run scripts to determine the type of performance the application is achieving at different points during the day. If performance lags below what is expected, the tool will alert the tester. Vendors offering tools in this category include Mercury Interactive, Compuware, Empirix, and IBM Rational. In the test management category, several vendors, including Compuware, IBM Rational, Segue, and Mercury Interactive, provide a historical documentation of which tests were run on which applications and what problems were found.