|
...testing effort often shortened in order to compensate for schedule slippages during earlier development activities. This results in insufficient testing of products before their release. Model-Based Test Generation (MBTG) has recently emerged as a possible approach to alleviating this problem by improving the effectiveness of the testing effort.
According to the consensus prevalent at the Workshop on Advances in Model-Based Software Testing (A-MOST 2005), (1) in a model-based testing technique the behavior of the application under test, as specified by the user, is used exclusively for generating a suite of tests for validating the application. The level of granularity of the input information provided by the user can vary widely. With this broad definition in mind, a number of studies on model-based testing are found in the recent literature.
Dalal et al. (2) highlight the advantages and challenges associated with the use of the Automatic Efficient Test Generator (AETG) technique on four industrial products. Pretschner et al. present a case study involving the use of AutoFocus (3) for testing an "infotainment" network. This case study highlights the model coverage, the implementation coverage, and the error-detection capabilities of AutoFocus. Veanes et al. (4) report on a case study that evaluates the effectiveness of the tool Asml/Spec# in an online testing setup. Another related study by Paradkar (5) presents a case study that seeks to identify fault models for better fault detection effectiveness using MBTG.
Although these studies provide insights into specific MBTG techniques, to our knowledge there are no publications that propose a measurement framework for evaluating MBTG tools. Consulting companies, such as Grove Consultants (6) or Ovum Inc., (7) often publish reports analyzing the capabilities of specific tools. Whereas these reports may serve well for a particular MBTG tool, they lack a framework for comparing tools.
When managers and other decision makers in the information technology (IT) industry need to select an MBTG tool for their software development project, they are often at a loss, primarily because there are no measurement frameworks to compare the capabilities of various MBTG tools. In this paper, we provide one such measurement framework that is based on the Goal Question Metric (GQM) methodology. (8) Applying GQM in MBTG helps formulate the metrics of interest: complexity, ease of learning, effectiveness, efficiency, and scalability. We demonstrate how to use our measurement framework by describing a rudimentary case study for comparing four MBTG tools. Although this case study is not sufficiently extensive for reliable comparison data, its purpose is to describe in detail the steps involved in comparing MBTG tools. The four tools chosen for this study (Archetest version 0.5, (9) ASMLT version 2.0, (10) TestMaster release 1.9.2, (11-13) and HOTTest release 0.1 (14)) differ in their modeling techniques, test-case specification techniques, and test generation algorithms.
The rest of the paper is organized as follows. In the next section, "Design of the measurement framework," we briefly describe the GQM approach, and we show how GQM-driven questions lead to the set of metrics of interest: complexity, ease of learning, effectiveness, efficiency, and scalability. In the section "Case study," we describe the four MBTG tools used, the three target applications that are tested, and the process we follow, and we discuss possible pitfalls when analyzing measurement data. In the section "Case study measurements," we describe and discuss the measurement data. The last section contains a summary and final comments.
DESIGN OF THE MEASUREMENT FRAMEWORK
The measurement framework was designed following the GQM methodology. (8) GQM defines a measurement model on three levels:
1. Conceptual (goal)--This defines the purpose of the study. The goal is to analyze some object of study (e.g., process, product) toward a specific purpose (e.g., characterize, evaluate, predict, motivate, improve) with respect to a focus (e.g., effectiveness) on behalf of (from the point of view of) a stakeholder (e.g., customer, organization).
2. Operational (question)--Questions are formulated to identify the information that is needed to achieve the goal. The set of questions is used to define models of the object of study and characterize the way a specific goal is achieved.
3. Quantitative (metric)--This consists of sets of metrics, where each set of metrics is based on the models and is associated with a question that has to be answered in a measurable way.
The purpose of the our framework is to compare selected MBTG tools for test generation abilities. We define the goal in accordance with the template prescribed by the GQM methodology as follows: "Analyze MBTG test generation tools to characterize them with respect to their test generation ability from the point of view of testers of IT systems."
In using our measurement framework for evaluating a set of MBTG tools, we do not consider specific requirements that a project may have, as we want the framework to be universally applicable, but the framework can be easily adapted, following the GQM methodology, to suit individual needs of projects.
From questions to metrics
At the operational level, we ask the following questions:
1. How complex is the tool to use?
2. How easy is it to learn the tool?
3. How effective is the tool?
4. How much effort is needed to test applications?
5. How does the test generation effort scale with application size?
It is understood that the higher the complexity of using the tool, the lower its usability. A higher complexity also means a higher investment in human resources due to the higher skill level required. Ease of learning is another vital issue that governs applicability. Lower ease of learning implies an increased effort in educating the testers on the test generation technique. Complexity and ease of learning determine directly the adoptability of the technique.
Questions 3 and 4 determine the return-on-investment from using the test generation technique. The effort needed for testing and the effectiveness of the test technique jointly determine the productivity of the tester. The gain in effectiveness should not be at the cost of a heavy increase in effort.
The staff responsible for selecting a tool needs to ascertain that the testing technique scales to large applications. In other words, they need to make sure that when the size of the application grows, the effort required stays manageable.
Metrics and measurement models
At the quantitative level, GQM deals with metrics. These metrics help answer the questions asked at the operational level. The metrics that are...
NOTE: All illustrations and photos
have been removed from this article.
More articles from IBM Systems Journal
Model driven development for business performance management., July 01, 2006
UML 2: a model-driven development tool., July 01, 2006
Feature-based survey of model transformation approaches., July 01, 2006
Using logical data models for understanding and transforming legacy bu..., July 01, 2006
Model traceability., July 01, 2006
Looking for additional articles?
Search our database of over 3 million articles.
Looking for more in-depth information on this industry?
Search our complete database of Industry & Market reports by text, subject, publication
name or publication date.
About Goliath
Whether you're looking for sales prospects, competitive information, company
analysis or best practices in managing your organization,
Goliath can help you meet your business needs.
Our extensive business information databases empower business
professionals with both the breadth and depth of credible,
authoritative information they need to support their business
goals. Whether it be strategic planning, sales prospecting,
company research or defining management best practices -
Goliath is your leading source for accurate information.
|
