Benchmarking software Quality

Transcription

Benchmarking Software Quality
– With Applied Cost of Quality
Cost of Quality has remained theoretical in many
treatments despite its powerful relevance in software
quality. A survey-based approach makes this model an
important quality driver today.
Cost of Quality (CoQ) as a software quality model has
remained an attractive yet difficult proposition. It is
attractive, because it promises to provide decidedly
insightful measurements, thus making cost reduction
a driver for improving quality. It is difficult, because it
demands granular measurement and analysis, making
assessment tedious. The good news is that indirect
methods employed to leverage the model have
helped increase the adoption of Cost of Software
Quality (CoSQ). Correlation with CoSQ improvement
and CMM® maturity has also been established
convincingly in some studies. This document explores
the facets of CoSQ and its growing importance in
software quality. It suggests a proven approach in
making it practically implementable in large software
settings.
Benchmarking Software Quality – With Applied Cost of Quality
About the Author
Srinivasa Phanindra Vedula
Srinivasa Phanindra Vedula is a senior Process and
Quality Consultant at Tata Consultancy Services. He
has extensive experience in the fields of software
quality, CMM, CMMI, TPI, and Six sigma. He is a certified
Six Sigma Black Belt, and has mentored and coached
execution of Black belt projects. He has led and
executed consulting engagements for clients in
Financial Services, Banking and insurance domain
using cost of quality baseline framework outlined in
this paper. He is a CSQA and a postgraduate from
Indian Institute of Technology, Chennai.
1
Table of Content
1. Introduction – Quality for Competitiveness
3
2. Cost of Quality – the Cost that Reduces Cost
3
3. CoSQ Correlates with Software Capability Maturity Levels
5
4. Kick-starting the Journey – Baselining CoSQ
6
5. Case Study: Cost of Software Quality Study for a
Financial Services Major in the US
7
6. Concluding Remarks
10
7. References
10
2
Introduction – Quality for Competitiveness
Software quality has come a long way. Having said that, I am probably provoking a section of software makers and
recipients who are still skeptical about whether true software quality exists. With quality frameworks and
methodologies like CMMI® and Six Sigma being used to the hilt, we somehow think we need to add another
perspective to quality, one that can provide a sole indicator for benchmarking and fact-based improvement.
Software quality has become more tangible with time and it is interesting to see new dimensions being added to it.
In every case though, the quest has been to make it relatable to business outcomes.
Cost of Quality is becoming increasingly important as a driver in business quality. It is also finding serious application
in software. This is more so because it provides a unified indicator as well as a driver of business health. Despite the
challenge of quantifying the measures involved, it is possible to baseline software quality and benchmark
improvements using survey based techniques. This paper demystifies Cost of Software Quality by suggesting a
practical approach to baselining CoSQ, which has been successfully used with our global clients. Despite quality
generally being an applied discipline, COQ has remained theoretical in many treatments. Therefore, we call this
approach “applied cost of quality”.
Cost of Quality – the Cost that Reduces Cost
Cost of quality started with the need to reduce cost of meeting client requirements, thereby increasing
competitiveness. Over time, it progressed as a philosophy for successful business. Phil Crosby laid the foundation of
the current form of COQ in his book Quality is Free
Cost of Conformance
(McGraw Hill 1979). He suggested that cost of quality
was the sum of two primary costs – cost of conformance
Cost of Prevention
Cost of Appraisal
and cost of non-conformance. Cost of conformance in
turn sums up the cost of appraisal and cost of
Cost of Quality
prevention. The non-conformance cost, on the other
Internal
External
hand, has two components – cost of internal failure, that
Failure
Failure
is, the cost of defects that occur before release to market
(for instance in development and production), and cost
Cost of Non-Conformance
of external failure, that is, the cost of defect that occurs
after release to market.
In the case of software, these components can be illustrated as hereunder:
Cost of Appraisal:
These are costs incurred to determine the degree of conformance to the requirements. The typical activities of
appraisal are
l
Review of code and other artifacts
l
Purchased software testing
l
First iteration of integration and system testing
l
User Acceptance testing.
3
Cost of Prevention:
These are costs incurred to prevent poor quality by keeping failure and appraisal costs to a minimum. The typical
prevention occurs through
l
Casual Analysis and defect prevention
l
Quality planning and software quality assurance
l
Quality training on tools and techniques
l
Process improvements
l
Code and component reuse
l
Training on design review
Cost of Internal Failure:
These are costs associated with defects found prior to delivery of software to the customer. The typical activities
under this head are
l
Pre-release defect management
l
Rework
l
Re-reviews
l
Retesting.
Cost of External Failure:
These are costs associated with defects found after the software is released to customers. Typical activities are :
Cost of pre-enhancement maintenance and support after release
l
Reworks and defect management of field errors
l
Warranty redressal
Cost of Execution
These are costs incurred to build the software right first time or the cost of construction, also expressed as percentage
of project cost.
Therefore,
Cost of Execution (%) = 100 – Cost of Quality (%)
Cost of quality provides a means to quantify quality cost in business terms. It therefore helps in identifying
opportunities for cost reduction. The underlying philosophy is that by cutting down on the cost of quality, one is able
to stay more competitive in the market and reduce opportunity cost of losing out to competition. Therefore, in the
highest performance state, the effective cost of quality to
business is minimal. It emerges as a powerful driver and, at The underlying philosophy is that by cutting down
the same time, carries the simplicity of providing one on your cost of quality, you are able to stay more
unified metric to judge the health of the line of business. By competitive in the market and reduce opportunity
providing distribution of quality cost across the four cost of losing out to competition. Therefore, in the
highest performance state, your effective cost of
components of it, it helps in understanding the direction of
quality to business is minimal. It emerges as a
improvement focus. For instance, if the cost of appraisal is powerful driver and, at the same time, carries the
high compared to that of prevention, one would focus on simplicity of providing one unified metric to judge
finding improvement opportunities in defect prevention the health of the line of business.
(such as in design, coding best practices).
4
CoSQ Correlates with Software Capability Maturity Levels
The Cost of Software Quality (CoSQ) has a good correlation with capability maturity of the organization, and can be
explained by Knox’s model for CoSQ (Modeling the Cost of Software Quality, Stephen T. Knox). Knox used the CoQ
model developed in the manufacturing environments and extended it across the SEI CMM® to produce a theoretical
CoSQ model. CMM® today is a well-accepted framework to assess maturity of software delivery capability. Thanks to
its wide adoption, we have extensive data available on overall industry maturity in terms of CMM® levels. Knox
predicted that CMM® Level 1 units have 60% CoSQ and he further hypothesized that CoSQ reduces by 67% at level 5.
Knox was able to map the four
Focus on Process Improvement
Optimizing (5)
Process Measured and Controlled
Quantitatively
Managed (4)
Process characterized for the
organization and its proactive
Defined (3)
Process characterized for projects
and is often reactive
Managed (2)
Process unpredictable poorly
controlled and reactive
Initial (1)
components of CoQ with the CMM® maturity levels. His finding suggested that the ratio between conformance and
non-conformance cost at level 3 would typically be 0.5. In the study, the overall CoSQ exhibits a downward trend and
an improvement of cost of execution. Thus, the high maturity organizations get a return of 85% on marginal cost of
quality (return on penny spent).
Cos t a s a Per cent of
D ev el o p ment
60
50
40
30
20
10
0
1
2
3
4
5
SEI CMM® Levels
CoSQ
Internal Nonconformance Costs
External Nonconformance Costs
Appraisal
Prevention
Knox Theoretical model showing relationship between CoSQ and SEI CMM® levels (1)
5
An instance of this is seen in a study by Raytheon Electronic Systems (RES) published in Cost of Software Quality, Herb
Krasner, SASPIN, September 1999. It successfully proved that the theoretical CoSQ model is a very good estimator of
cost elements and correlates well with CMM® maturity.
Raytheon Electronic Systems CoSQ Study
100%
CoQ D i s t r i bu t i on
80%
66
45
24
34
6
11
18
60%
44
40%
18
Cost of Nonconformance
23
12
Cost of Prevention
15
Cost of Appraisal
76
7
20%
Cost of Execution
66
15
55
Cost of Appraisal was included
in the Cost of Prevention in L3
and L4 study.
34
0%
L1
CoSQ
L2
L3
L4
CMM® Levels
Cost of Quality, proportionate to cost of Execution, reduces with higher levels of maturity
Kick-starting the Journey – Baselining CoSQ
Establishing CoSQ in an organization is a challenging task. This is because the cost elements that are required for
CoSQ are highly fine-grained and demand extensive data. It is not practical to track and classify the project cost data
at a granular level. If at all an organization attempts to do so, it would run a high risk of being entrapped in a
regimented and ritualistic environment impeding efficiency and ingenuity. Therefore, we suggest a survey-based
approach for base-lining CoSQ. Our experience with this approach shows that it can serve as an effective indicator of
the direction in which improvement efforts should be focused. The following case study describes a methodology for
implementing this approach.
6
Initial
Organization
Study

Identify equivalent
terminologies in the
organization

Identify set of cost
element

Map to CoSQ Model
Questionnaire
Design

Design question sets
for prevention,
appraisal and failure
in terms of getting
percentage data.

Questions are
granular (For
example, percentage
of effort in first time
integration testing)

Consolidation of
response survey

Elimination of data
with biased patterns

Moderation of cost
value based on mean
and median
percentage values
deployment channels
Deployment of
questionnaire, and
response collection
Technical support to
responders
(temporary Help desk)

Study a sample of the
population and find
adequate
stratification.
Generally, a sample
represents
stakeholders
(Program manager,
EQA) and Process
participants
Facilitation
Sessions

Carry out facilitation
session with sample
population to make
them understand the
questionnaire, the
purpose, objective,
and tools and
methods to be used.
Identify and
Present
Opportunities
for improvement
Survey Response
Grouping and
Analytics
Administer
Questionnaire
Identification of
Population
Sampling

Listing and
prioritization of
Opportunities for
Improvements (OFIs)

Presentation to
stakeholders
Case Study: Cost of Software Quality Study for a Financial Services
Major in the US
TCS IT Process consultants helped in establishing cost of quality baseline using a survey based approach. Under a
process reengineering initiative using CMMI® framework, a cost of quality model was designed to estimate the
potential savings. The survey inferences were further validated with the qualitative findings during gap analysis.
7
The following is a part of the questionnaire that was deployed to evaluate the cost of appraisal:
Q. If you are managing a single project, indicate data across the given categories. If you are managing multiple
projects, indicate the percentage distribution across the categories.
Category
Design
document
walk-through
Code
walk-through
Measure
% of
development
effort
% of
% of
development development
effort
effort
Specifications
walk-through
Test planning,
strategy, test
case and test bed
preparation
Training to
testers on tools
and defect
management
Functional
testing
Regression
testing
% of
development
effort
% of
development
effort
(First time
effort only)
% of
development
effort
% of
development
effort
User
acceptance
testing
% of
development
effort
(First time
effort only)
Small
projects
Medium
projects
Large
projects
Very
large
projects
Result:
Average Cost of Quality Distribution
100%
90%
11.2
19
Percentage Effort
80%
70%
22.5
43.3
32.1
23.5
32.4
60%
3
50%
40%
32.7
12.6
2.5
30%
20%
10%
24.3
36
24
51
Cost of Execution
Cost of Appraisal
Cost of Prevention
30
Cost of Failure
0%
Type1
Type2
Type3
Type4
Proje ct Type
The results showed disparate performance across project types. One of the project types (type 2) showed high cost of
prevention. The analysis suggested that this project type had high cost of appraisal to cost of prevention ratio (2.6:1).
The interpretation was that opportunities for improvement exist in design and development.
8
Phase analysis of CoSQ components indicated high injection of defects in the analysis and design phase of large
projects, leading to higher cost of prevention during building phase. The average spent on defect prevention during
the building phase was significantly higher than industry average.
Per cent a ge of Def ect s wi t h i n
P r oj ect Ty p e
Project Type Wise Distribution of Defects
40%
35%
30%
Very large
25%
Large
20%
Medium
15%
Small
10%
5%
Industry Average
0%
Company Average
Analysis
Design
Build
Testing
Rollout
Phases of Development
On further analysis of the defects in building phase, almost two-thirds of the defects were found to be requirement
related. This suggested potential opportunities for improvement in analysis and design.
Defect Distribution in Building Phase
27%
11%
62%
Requirement related defects
Configuration related defects
Others
Word of Caution
Since the methodology is based on a survey questionnaire, one needs to guard against measurement errors
associated with survey questionnaires. These errors can typically be controlled by paying careful attention to: •
Choice of appropriate representative sample
l
Customization of questionnaire for the organization
l
Facilitation sessions before administering the survey
9
Concluding Remarks
Cost of Quality has emerged as a tool for looking at quality from the perspective of enhancing competitiveness. By
cutting down the cost of quality, we are improving the overall health of the line of business, which in this case is
software. Its strength lies in its ability to provide a unified benchmarking metric that can be further analyzed to
identify focus areas for improvements, which otherwise would remain obscure. For instance, a high cost of appraisal
may reveal a reactive approach to quality where sufficient attention is not being paid to preventing defects to occur
in the first place. However, challenges lie in the implementation of theoretical models and mapping those with the
granular activities of the software development lifecycle to gather real data. The survey-based approach to establish
the CoSQ financial baseline in IT organizations is a viable and proven alternative in this respect.
References
1.
2.
3.
4.
5.
A Review Research on Cost of Quality Models and Best Practices, Andrea Schiffauerova, Vince Thomson, 2006
Modeling the Cost of Software Quality, Stephen T. Knox, Copyright Digital Equipment Corporation 1999
Software Process Improvement the Road to a Professional Software Company SASPIN Nov 2002
Cost of Software Quality Herb Krasner SASPIN Sep 1999
Phil Crosby - Quality is Free, McGraw Hill 1979.
Note: CMMI® and CMM® are registered trademarks of Software Engineering Institute, Carnegie Mellon University
10
About Tata Consultancy Services (TCS)
About TCS IT Process and Service
Management Consulting
Tata Consultancy Services Limited is an IT services, business
IT Process and Service Management Consulting partners with
solutions and outsourcing organization that delivers real results to
organizations to enable IT process improvement aligned to
global businesses, ensuring a level of certainty no other firm can
TCS' experience in successfully
match. TCS offers a consulting-led, integrated portfolio of IT and IT-
adopting multiple frameworks like CMMI®, ISO-9001, ITIL®,
enabled services delivered through its unique Global Network
business needs. Leveraging
P-CMM®, BS15000, BS7799 and Six Sigma, the consulting group
Delivery ModelTM, recognized as the benchmark of excellence in
provides services with a practitioner’s approach to assessing,
software development.
defining, improving and deploying IT processes.
A part of the Tata Group, India's largest industrial conglomerate,
Service areas include:
TCS has over 94,000 of the world's best trained IT consultants in 47
l CMMI® Appraisal Services and Training
countries. The company generated consolidated revenues of US
$4.3 billion for fiscal year ended 31 March 2007 and is listed on the
l ITIL deployment and assessment
National Stock Exchange and Bombay Stock Exchange in India. For
l Agile development for Distributed Environment
more information, visit us at www.tcs.com
l Six Sigma based improvements
l Quality Improvement Initiatives
l Digitized enablers, which reduce 'Time to Deploy' and sustain
productivity for process
global.consulting@tcs.com
All content / information present here is the exclusive property of Tata Consultancy Services Limited
(TCS). The content / information contained here is correct at the time of publishing.
No material from here may be copied, modified, reproduced, republished, uploaded, transmitted,
posted or distributed in any form without prior written permission from TCS. Unauthorized use of the
content / information appearing here may violate copyright, trademark and other applicable laws,
and could result in criminal or civil penalties.
Copyright © 2007 Tata Consultancy Services Limited
www.tcs.com