|
...human limitations. Frederick Brooks explained over a quarter of a century ago, (1) there is a big difference between an isolated program created by a lone programmer and a programming systems product. A programming systems product "can be run, tested, repaired, and extended by anybody ... in many operating environments, for many sets of data" and forms a part of "a collection of interacting programs, coordinated in function and disciplined in format, so that the assemblage constitutes an entire facility for large tasks." Brooks asserted a nine-fold increase in cost to develop a programming system product from an isolated program (see Figure 1).
[FIGURE 1 OMITTED]
With the advent of the Internet and the World Wide Web, the problems that were recognized a quarter century ago as having "no silver bullet" for the solution (1) have been magnified. The challenges of designing and testing distributed computing systems, with distributed data and Web services, with the need for coexistence of heterogeneous platforms, unpredictable run-time environments, and so on, make the already difficult problem even harder.
A key ingredient that contributes to a reliable programming systems product is the assurance that the program will perform satisfactorily in terms of its functional and nonfunctional specifications within the expected deployment environments. In a typical commercial development organization, the cost of providing this assurance via appropriate debugging, testing, and verification activities can easily range from 50 to 75 percent of the total development cost. Thus we should consider what is involved in these activities that make them so challenging and so expensive.
Since one of the goals of this special issue of the IBM Systems Journal is to be accessible to the students of software engineering at large, we define relevant terminology and its implications (we include formal notation for this terminology, but it is not essential for the basic understanding of problem definition). Note that the terms "debugging," "testing," and "verification" are not mutually exclusive activities, especially in everyday practice. The definitions draw distinctions, but the boundaries may actually be fuzzy. We begin with a software program written in a programming language (let P be the program written in language L). The program is expected to satisfy a set of specifications, where those specifications are written in a specification language (call the set of specifications [PHI] = {[[phi].sub.1], [[phi].sub.2], [[phi].sub.3], ..., [[phi].sub.n]} and the specification language [LAMBDA]). In most real-world cases, the specification language ([LAMBDA]) is the natural language of the development team (i.e., English, Spanish, etc.).
Debugging: The process of debugging involves analyzing and possibly extending (with debugging statements) the given program that does not meet the specifications in order to find a new program that is close to the original and does satisfy the specifications (given specifications [PHI] and a program P, not satisfying some [[phi].sub.k] [member of] [PHI], find a program P' "close" to P that does satisfy [[phi].sub.k]). Thus it is the process of "diagnosing the precise nature of a known error and then correcting it." (2)
Verification: Given a program and a set of specifications, show that the program satisfies those specifications (given P and a set of specifications [PHI] = {[[phi].sub.1], [[phi].sub.2], [[phi].sub.3], ..., [[phi].sub.n]}, show that P satisfies [PHI]). Thus, verification is the process of proving or demonstrating that the program correctly satisfies the specifications. (2) Notice that we use the term verification in the sense of "functional correctness," which is different from the typical discussion of verification activities discussed in some software engineering literature, (3,4) where it applies to ensuring that "each step of the development process correctly echoes the intentions of the immediately preceding step."
Testing: Whereas verification proves conformance with a specification, testing finds cases where a program does not meet its specification (given specifications [PHI] and a program P, find as many of [[phi].sub.1], [[phi].sub.2], [[phi].sub.3], ..., [[phi].sub.p] [member of] [PHI] not satisfied by P). Based on this definition, any activity that exposes the program behavior violating a specification can be called testing. In this context, activities such as design reviews, code inspections, and static analysis of source code can all be called testing, even though code is not being executed in the process of finding the error or unexpected behavior. These activities are sometimes referred to as "static testing." (5) Of course, execution of code by invoking specific test cases targeting specific functionality (using, for example, regression test suites) is a major part of testing.
Validation: Validation is the process of evaluating software, at the end of the development process, to ensure compliance with requirements. Note that the verification community also uses the term validation to differentiate formal functional verification from extensive testing of a program against its specifications.
Defect: Each occurrence of the program design or the...
NOTE: All illustrations and photos
have been removed from this article.
More articles from IBM Systems Journal
Improving software testing via ODC: three case studies.(Orthogonal Def..., March 01, 2002
A metric for predicting the performance of an application under a grow..., March 01, 2002
Testing z/OS: the premier operating system for IBM's zSeries server., March 01, 2002
The STCL test tools architecture.(Software Test Community Leaders), March 01, 2002
Using a model-based test generator to test for standard conformance., March 01, 2002
Looking for additional articles?
Search our database of over 3 million articles.
Looking for more in-depth information on this industry?
Search our complete database of Industry & Market reports by text, subject, publication
name or publication date.
About Goliath
Whether you're looking for sales prospects, competitive information, company
analysis or best practices in managing your organization,
Goliath can help you meet your business needs.
Our extensive business information databases empower business
professionals with both the breadth and depth of credible,
authoritative information they need to support their business
goals. Whether it be strategic planning, sales prospecting,
company research or defining management best practices -
Goliath is your leading source for accurate information.
|
