Software Testability - Prince Sultan University

International Journal of Computer Applications (0975 8887)
Volume 113 - No. 7, March 2015
Software Testability: Recovering from 2 Decades of
Research
Mamdouh Alenezi
College of Computer and Information Sciences
Prince Sultan University, Riyadh 11586
Saudi Arabia
ABSTRACT
Software Testability investigation has been a research focus since
1990s and grows into more prevalent when entering 21st century.
Software Testability is an important internal quality characteristic
of any software system which measures how easy or difficult to
test a piece of software. In this paper, we study Software Testability from various aspects, such as definitions, its relation to quality
models, and what software metrics were used to assess Testability.
This paper brings an introductional review on Software Testability and provide the state-of-the-art of Software Testability studies.
General Terms:
Software Engineering, Software Testing
Keywords:
Software Quality, Testability, Metrics.
1.
INTRODUCTION
Quality assurance is extremely important in the software industry
in which practitioners and researchers are seeking to achieve quality with various techniques. Software testing and reuse are the two
promising techniques to improve the quality of the software systems. The focus of the Software development processes normally
is limiting the number of errors in software, finding and solving
software problems, and predicting software reliability after development. Delivering quality software is no longer an advantage but a
necessary factor. Software testing is a major aspect of the software
development life cycle. Software testing is almost the only ways
of promising excellence of software systems. Software testing is a
costly process with direct relationship to nearly all major technical
issues in Software Engineering.
When a software system grow in size it complexity increases and in
almost everyday activities, its quality becomes a necessity. Therefore, effective testing is needed to achieve an acceptable level of
quality. To maximize the benefit of testing, the testability of software systems should be optimal. Software testability is a software
attribute that assesses the effort needed for software testing. IEEE
[4] defines Testability as “the degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been
met?”. Testability is one of the maintainability characteristics of
the ISO/IEC SQuaRe quality standard [19]. According to this standard, testability is defined as “degree of effectiveness and efficiency
with which test criteria can be established for a system, product or
component and tests can be performed to determine whether those
criteria have been met” [19]. The most common definition of Testability is ease of performing testing [4] which has its roots in testing
and usually defined in terms of observability and controllability.
Testability is one of the important internal quality factors for the
success of the software. Testability is a software quality characteristic that expresses the effect of structural and semantic aspects of
software on the effectiveness of testing following certain criterion.
Testability measurement enables software engineers to estimate the
effort needed to test and validate a certain software module. Testability is not yet fully understood and what actually relates to testability is still not clearly defined [21]. Software engineering methods emphasizes the importance of unit testing to the degree that
you find some projects that have test cases exceed the amount of
the actual code. This is a clear demonstration of the importance of
testability as a quality characteristic [22].
The remainder of this paper is organized as follows. Section 2 discusses the motivation behind this work. Section 3 discusses testability is different software standards and quality models. Section 4
discusses a literature review about testability. Section 5 states the
metrics used in this study. Conclusions are presented in Section 6.
2.
MOTIVATION
What makes testability an important issue in software industry?
What are the limitations of existing research? These can be demonstrated by the following three aspects:
(1) There a lot of confusion of testability definitions. Testability
has many definitions in the literature. IEEE [4] defines Testability as “the degree to which a system or component facilitates
the establishment of test criteria and the performance of tests
to determine whether those criteria have been met?”. ISO defined Testability as “degree of effectiveness and efficiency with
which test criteria can be established for a system, product or
component and tests can be performed to determine whether
those criteria have been met” [19]. Several researchers came
up with their own definitions of Testability. Freedman [14] defined Testability as observability and controllability. Voas et. al
[30] defined Testability as predicting the failure tendency to be
observed throughout random black-box testing. Gao and Shih
[16] defined Testability as the effort needed to test the program
1
International Journal of Computer Applications (0975 8887)
Volume 113 - No. 7, March 2015
according to a specific measure. These mixed definitions reveal that Testability is its nature has different points of view,
but also introduce confusion of testability understanding.
(2) Is Testability an internal or external quality attribute? Fenton
and Pfleeger [12] explicitly specified Testability as an external
attribute. Other researchers take the same view such as [3] and
[29]. Other researchers implicitly specify Testability as an external attribute such as [14] and [30]. Testability by definion
relates to testing [11].
(3) The issue of compositionality which is one of the research
challenges encountered by software engineers [13]. This applies to Testability since there is no agreed upon or a systematic
way of deriving a software testability measure.
3.
TESTABILITY IN SOFTWARE STANDARDS
AND MODELS
defined Testability as the ease of validation, that the software meets
the requirements.
General Utility
As-is Utility
Reliability
Portability
Maintainability
Efficieincy
Human Engineering
Testability
Understandability
Modifiability
Device Independace
Self Containedness
Self Containedness
Accountability
Robustness /
Integrity
Accountability
Consistency
Structuredness
Accuracy
Device Efficiency
Acessibility
Communicativiness
Structuredness
Augmentability
Completeness
Acessibility
Communicativiness
Self Descriptiveness
Conciseness
Structuredness
Legibility
Robustness /
Integrity
Consistency
Fig. 2. Boehm’s Software Quality Characteristics Tree [9].
McCall [2] was the first to propose a software quality model. McCall model was an attempt to close the gap between developers
and users with emphasis on software quality. The McCall quality
model has three main quality levels for defining the quality of a
software product: product revision, product transition, and product operations. Product revision includes maintainability, flexibility, and testability (the ease of testing the program, to ensure that it
is error-free and meets its specification). In McCall’s model, Testability was under Product Review. McCall defined Testability as the
ability to validate the software requirements. McCall further decomposed Testability to three sub-characteristics, namely Simplicity, Instrumentation, and Modularity. The main contribution of the
McCall model was to establish relationships between quality characteristics and metrics.
Software Product Quality
The FURPS Model stands for Functionality, Usability, Reliability,
Performance, and Supportability. The FURPS Model requirements
are of two types: Functional Requirements (RF) and non-functional
(NF). The model was initially proposed by Robert Grady [17] and
extended by Rational Software [24]. The FURPS model places
Testability under Supportability.
Quality Characteristics
Functionality
Reliability
Efficiency
Usability
Portability
Accuracy
Fault tolerance
Resource Behaviour
Learnability
Adaptability
Analysability
Compliance
Maturity
Time Behaviour
Operability
Installability
Changeability
Interoperability
Recoverability
Understandability
Replaceability
Stability
Security
Product Operations
Product Revison
Maintainability
Testability
Product Transitions
Suitability
Correctness
Flexibility
Interoperability
Efficiency
Maintainability
Portability
Integrity
Testability
Reusability
Fig. 3. The FURPS Model [17].
Reliability
Usability
Fig. 1. The McCall quality model (a.k.a. McCall’s Triangle of Quality).
Boehm [9] proposed a large-scale model where he added several
factors at different levels as an improvement over the McCall’s
model. Boehm model addressed the contemporary drawbacks of
McCall’s model by measuring the quality in qualitatively manner
instead of quantitatively manners. Boehm classified the software
quality into three main properties, Portability, As in Utility, and
Maintainability. Testability was considered under Maintainability
and has four different dimensions, namely Structured-ness, Selfdescriptiveness, Accountability, and Communicativeness. Boehm
The ISO 9126 quality model [18] was revolved around both McCall
and Boehm Models. The model consists of two main parts: internal
and external quality attributes and quality in use attributes. Internal
attributes are considered static attributes since you need to execute
the software to evaluate them, whereas external attributes are assessed during the system execution. The quality in use attributes
relate to the productivity, security, and the effectiveness of the software product. Testability in this model resides under the Maintainability which resides under Internal and External attributes. Testability was defined as attributes of software that bear on the effort
needed for validating the modified software.
The ISO 25010 model [8] emerged in 2007 with some updates
to the ISO 9126 model. The new model has been divided into 8
key characteristics. Two new characteristics were introduced in this
model, security and compatibility. Testability in this model resides
under the Maintainability which is one of the key characteristics of
that model.
The SQO-OSS model [1] is a hierarchical model which assesses
the process and the source code of a system. The model allows
for automated collection of several metrics. The model is different from other quality models because of several reasons, (1) the
2
International Journal of Computer Applications (0975 8887)
Volume 113 - No. 7, March 2015
4.
Quality Characteristics
Functionality
Reliability
Usability
Efficiency
Maintainability
Portability
Suitability
Maturity
Understandability
Time Behaviour
Analysability
Adaptability
Accuracy
Fault tolerance
Learnability
Resource utilization
Changeability
Installlabilty
Interoperability
Recoverability
Operability
Efficiency
compliance
Stability
Co-existence
Security
Reliability
compliance
Attractiveness
Testability
Replaceability
Usability
compliance
Maintainability
compliance
Portability
compliance
Functionality
compliance
Fig. 4. The ISO/IEC 9126-1 Internal/External Quality Model [18].
Software Product Quality
Functional
Suitability
Performance
Efficiency
Reliability
Operability
Security
Compatibility
Maintainability
Transferability
Appropriateness
Availability
Time Behaviour
Appropriateness
recognisability
Analysability
Adaptability
Modularity
Portability
Accuracy
Fault tolerance
Resource
utilization
Learnability
Changeability
Installlabilty
Reuseability
Adaptability
Compliance
Recoverability
Compliance
Compliance
Ease of use
Stability
Co-existence
Analyzeability
Installability
Helpfulness
Testability
Replaceability
Changeability
Compliance
Attractiveness
Maintainability
compliance
Portability
compliance
Modification
Technical
accessibility
Stability
Compliance
Testability
Compliance
Fig. 5. The ISO 25010 model Software Product Quality Model [8].
main focus is automation, (2) It focuses on the source code, and (3)
considers only process related measures that can be automatically
calculated. Maintainability is classified under Product Code Quality and Testability in under Maintainability. According to the SQOOSS model, testability can be estimated by several metrics, namely
Number of exits of conditional structs, Cyclomatic number, Number of nested levels, Number of unconditional jumps, Response
for a class (RFC), Average cyclomatic complexity per method, and
Number of children (NOC).
SQO-OSS Quality
Characteristics
Product (Code) Quality
Maintainability
Reliability
Analyzability
Maturity
Changeability
Effectiveness
Community Quality
Security
Stability
Testability
Fig. 6. The SQO-OSS quality model [1].
Mailing list quality
Documentation
quality
Developer base
quality
LITERATURE REVIEW
Testability is one of the important internal software attributes. In
order to understand and comprehend the meaning of Testability,
many studies tried to analyze and measure Testability. These studies investigated Testability within different application domains. In
this Section, we discuses these studies in a historical order.
Voas and Miller [30, 31] attracted attention to software testability in their studies. They defined testability as the probability of a
failed test case, if the program has a fault. They explained how to
obtain fault sensitivity by multiplying several probabilities, namely
1) the fault location; 2) the fault alter the program’s state; and 3)
the altered state gets propagated to the output. Low fault sensitivity
points out low testability and vice versa. They introduced a testability metric that is based on inputs and outputs of a software test.
They also proposed an approach which is called PIE. PIE stands for
Propagation, Infection, and Execution with the aim of evaluating
software testability. However, the PIE technique was multifarious
and has high complexity.
Binder [7] made a huge improvement on our understanding of software testability. He highlighted the importance of improving testability in the software development life cycle. Binder proposed several existing metrics to evaluate testability, including: DIT (depth of
inheritance tree), LCOM (lack of cohesion in methods), DYN (percentage of dynamic calls), and OVR (percentage of non-overloaded
calls). However, Binder did show empirical evidence to support
their new metrics and did not show that there is a strong relationship between the suggested metrics and testability. He developed
a fish-bone model embodying that testability is an outcome of six
different factors: (1) characteristics of the representation; (2) characteristics of the implementation; (3) built-in test capabilities; (4)
the test suite (test cases and associated information); (5) the test
support environment; and (6) the software testing process. However, all these factors only evaluate testability at a higher level of
abstraction with no or little comprehensible association with object
oriented design and implementation.
McGregor et al [26] tried to measure object-oriented testability by
introducing the visibility component measure (VC). VC is sensitive to object oriented like inheritance, encapsulation, collaboration
and exceptions. The purpose of VC was the feasibility of using it
at early stages of a development process. However, Calculating VC
requires the availability of accurate and complete specification documents.
Bruce and Haifeng Shi [25] decked the designing of object oriented software with testability factors that affect testability. They
introduced a preliminary framework to evaluate software testability metric. They formulated several guidelines to improve software
testability while designing object-oriented software. However, their
framework has different limitations from implementation perspective.
Jungmayr [20] related software testability with static dependencies.
He proposed a new approach for estimating software testability
from integration testing point of view. He investigated measuring
testability based on static dependencies to identify test-critical dependencies. One of these test-critical dependencies, according to
Jungmayr, is dependency cycles. Jungmayr defined a new metric
based on dependencies to measure the impact of dependencies on
testability. However, the metric evaluates each dependency individually without considering their interaction effects and the combined
impact of sets of dependencies.
Briand et al. [10] proposed a new approach which uses class invariants, preconditions, and postconditions (instrumented contracts)
with the aim of improving testability. These instrumented contracts
3
International Journal of Computer Applications (0975 8887)
Volume 113 - No. 7, March 2015
are assumed to increase the probability that a fault be detected
when test cases are executed. Their study showed that contract assertions detected a good percentage of failures depending on the
precision of these contracts. These contract assertions reduce the
effort needed to locate faults when failures are detected.
Bruntink and van Deursen [11] investigated testability from two
different perspectives: metrics to explore the system testability of a
system and the effort needed to develop test cases. They found, using two case studies, a correlation between source code metrics like
Number of Methods and test metrics like the number of test cases
and the number of lines of code per test class. They did not find
any relationship between inheritance related metrics and testability
metrics. Their test suites were reused from open source development where there was no clear understanding of what test techniques are employed.
Baudry et al. [6] investigated the measurement of testability of
object-oriented designs by focusing on design patterns and how
these patterns can improve testability. They related the testing effort
to testability and emphasized class interactions in class diagrams
(class interaction is considered when there is a path in the class dependency graph between two classes). However, their hypothesis
was not empirically validated. Their model is based on the class
dependency graph topology and does not account for the semantics
of class relationships. Jerry and Ming [16] proposed a component
testability model to assess the software testability which was based
on pentagon shaped and analytical approach.
Mulo [27] incorporated the importance of testability measurement
throughout all stages of software development. He classified metrics to measure testability into three categories structural and behavioral metrics and, data flow metrics. The study suggested that
these metrics are better to guide code reviewers and developers instead of measuring the actual testability. He concluded that estimating testability is not an easy task that requires tremendous effort and
huge budget.
Jianping Fu & Minyan Lu [15] proposed a request-oriented method
of software testability measurement. Their self-contained method
collects several elements to complete all kinds of testability measurement. The applied their methods in a small case study to
demonstrate its effectiveness. However, their approach was not applied to different case studies to unveil its actual usefulness and
their approach has not been widely accepted.
Khatri et. al [23] proposed an approach to improve testability using
reliability growth models. They acknowledged that bug complexity
hurts the testability a lot. They discussed several theoretical approaches for improving and measuring the testability of software.
However, their model was hypothetical and the quantitative measure of enhancement of testability was not quantified.
Badri et. al [5] established a correlation between the effort required
to develop a unit test case and some object oriented metrics. They
conducted empirical study using three Java open source systems
which their JUnit test cases. They concluded their finding by emphasizing that size, coupling, cohesion, and complexity were found
significant predictors of the effort required to develop unit tests. In
their study, they investigated testability only from the perspective
of unit testing.
After this brief historical demonstration of testability studies, we
found that several methods are available in the literature for estimating and improving software testability. A tremendous effort
was focused at later stages of the development life cycle. However,
software engineering best practices advocate integrating software
testability at design phase which will help designers to improve the
quality of the software and significantly reduce the overall develop-
ment cost which produces high quality maintainable, testable and
reliable software.
Another criticism of the current literature is that they use metrics
that are specific to a programming language. Metrics that are used
to estimate the testability should be language independent in order to generalize the results to more systems that are developed in
different languages.
5.
METRICS USED FOR TESTABILITY
A very large number of object-oriented (OO) metrics have been
proposed in the literature with the aim of evaluating different software attributes. Automatic collection of software metrics makes
collecting them a low cost process as they are useful in predicting
software quality attributes. Software metrics were used extensively
in the literature to assess and evaluate software testability. These
software metrics can be categorized into two main types, namely
source code metrics and test suite metrics. Table 1 shows the metrics that have been used in the literature.
6.
CONCLUSION
Testability has always been an intangible notion and its capacity a
difficult exercise. Existing works on testability either constrain it on
a specific viewpoint or take a very general level. Although, several
approaches have been proposed for measuring and improving software testability, the state of the art is disseminated and lacks operational guidelines on how to proceed in a systematic and structured
manner. Testability is a quality attribute with the aim of figuring out
how much effort is needed for software testing. Reducing the effort
needed for testing and improving software testability is necessary
to deliver high quality software within time and budget.
7.
REFERENCES
[1] Adewole Adewumi, Sanjay Misra, and Nicholas Omoregbe.
A review of models for evaluating quality in open source software. IERI Procedia, 4:88–92, 2013.
[2] Rafa E Al-Qutaish. Quality models in software engineering
literature: an analytical and comparative study. Journal of
American Science, 6(3):166–175, 2010.
[3] Richard Bache and Monika M¨ullerburg. Measures of testability as a basis for quality assurance. Software Engineering
Journal, 5(2):86–92, 1990.
[4] Linda Badri, Mourad Badri, and Fadel Toure. An empirical
analysis of lack of cohesion metrics for predicting testability
of classes. International Journal of Software Engineering and
Its Applications, 5(2):69–85, 2011.
[5] Mourad Badri and Fadel Toure. Empirical analysis of objectoriented design metrics for predicting unit testing effort of
classes. Journal of Software Engineering & Applications,
5(7), 2012.
[6] Benoit Baudry and Yves Le Traon. Measuring design testability of a uml class diagram. Information and Software Technology, 47(13):859–879, 2005.
[7] Robert V Binder. Design for testability in object-oriented systems. Communications of the ACM, 37(9):87–101, 1994.
[8] Jørgen Bøegh. A new standard for quality requirements. IEEE
Software, 25(2):57–63, 2008.
[9] Barry W Boehm, John R Brown, and Myron Lipow. Quantitative evaluation of software quality. In Proceedings of the
4
International Journal of Computer Applications (0975 8887)
Volume 113 - No. 7, March 2015
Table 1. Metrics Used for Testability in the Literature
Source Code Metrics
Depth of Inheritance Tree [11, 5]
Fan Out [11]
Lack of Cohesion of Methods [11, 4, 5]
Lines of Code per Class [11]
Number Of Children [11, 5]
Number Of Fields [11]
Number Of Methods [11]
Response For Class [11, 5]
Weighted Methods Per Class [11, 5]
Coupling Between Objects [5]
LOC [5]
Percentage of Dynamic Calls [7]
Percentage of Non-overloaded Calls [7]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]
2nd international conference on Software Engineering, pages
592–605. IEEE Computer Society Press, 1976.
Lionel C Briand, Yvan Labiche, and Hong Sun. Investigating the use of analysis contracts to improve the testability
of object-oriented code. Software: Practice and Experience,
33(7):637–672, 2003.
Magiel Bruntink and Arie Van Deursen. Predicting class
testability using object-oriented metrics. In Fourth IEEE International Workshop on Source Code Analysis and Manipulation, 2004, pages 136–145. IEEE, 2004.
Norman E Fenton and Shari Lawrence Pfleeger. Software metrics: a rigorous and practical approach. PWS Publishing Co.,
1998.
Anthony Finkelsteiin and Jeff Kramer. Software engineering:
a roadmap. In Proceedings of the conference on The future of
Software Engineering, pages 3–22. ACM, 2000.
Roy S Freedman. Testability of software components. IEEE
Transactions on Software Engineering, 17(6):553–564, 1991.
Jianping Fu and Minyan Lu. Request-oriented method of software testability measurement. In International Conference on
Information Technology and Computer Science, 2009. ITCS
2009, volume 2, pages 77–80. IEEE, 2009.
Jerry Gao and M-C Shih. A component testability model for
verification and measurement. In 29th Annual International
Computer Software and Applications Conference, COMPSAC
2005, volume 2, pages 211–218. IEEE, 2005.
Robert B Grady. Practical software metrics for project management and process improvement. Prentice-Hall, Inc., 1992.
ISO/IEC. Iso/iec 9126-1 - software engineering - product
quality part 1: Quality model. ISO/IEC 9126-1, 2001.
ISO/IEC. Systems and software engineering - systems
and software quality requirements and evaluation (square).
ISO/IEC 25010 - System and software quality models, 2011.
Stefan Jungmayr. Testability measurement and software dependencies. In Proceedings of the 12th International Workshop on Software Measurement, pages 179–202, 2002.
Mazhar Khaliq, Riyaz A Khan, and MH Khan. Software quality measurement: A revisit. Oriental Journal of Computer Science & Technology, 3(1):05–11, 2010.
R. A. Khan and K. Mustafa. Metric based testability model
for object oriented design (mtmood). SIGSOFT Software Engineering Notes, 34(2):1–6, February 2009.
Test Suite Metrics
dLOCC (Lines Of Code for Class) [11]
dNOTC (Number of Test Cases) [11]
TNbLOC: the size of the test suite [4]
TNbOfAssert (Number of assert methods) [4]
TNOO (Number of methods) [28]
TINVOK (Number of method invocations) [28]
TDATA (Number of new Java objects) [28]
[23] Sujata Khatri, RS Chhillar, and VB Singh. Improving the
testability of object-oriented software during testing and debugging processes. International Journal of Computer Applications, 35, 2011.
[24] Philippe Kruchten. The rational unified process: an introduction. Addison-Wesley Professional, 2004.
[25] Bruce WN Lo and Haifeng Shi. A preliminary testability
model for object-oriented software. In Proceedings. International Conference on Software Engineering: Education &
Practice, 1998, pages 330–337. IEEE, 1998.
[26] John D McGregor and Satyaprasad Srinivas. A measure
of testing effort. In Proceedings of the 2nd conference
on USENIX Conference on Object-Oriented Technologies
(COOTS)-Volume 2, pages 10–10. USENIX Association,
1996.
[27] Emmanuel Mulo. Design for Testability in Software Systems.
Master’s thesis, Delft University of Technology, Delft, the
Netherlands, 2007.
[28] Fadel Toure, Mourad Badri, and Luc Lamontagne. A metrics
suite for junit test code: a multiple case study on open source
software. Journal of Software Engineering Research and Development, 2(1):14, 2014.
[29] Jeffrey M. Voas. Pie: A dynamic failure-based technique.
IEEE Transactions on Software Engineering, 18(8):717–727,
1992.
[30] Jeffrey M Voas and Keith W Miller. Semantic metrics for software testability. Journal of Systems and Software, 20(3):207–
216, 1993.
[31] Jeffrey M Voas and Keith W Miller. Software testability: The
new verification. IEEE software, 12(3):17–28, 1995.
5