TickIT International

TickIT
International
3
13
15
Our feature article this edition is all about testing – not how to do it but how to prepare and manage it.
Testing is a crucial stage in the development and deployment of IT systems, but how seriously is this viewed
as a highly skilled profession? Do we train our testers sufficiently to equip them with the skills needed to
do the job properly? Is the profession recognized and does it have the kudos that other IT functions enjoy?
These are all issues that Richard Knight addresses in his article ‘Assurance and Test – People and Skills’.
TickIT Plus development is hotting up so Derek Irving is back to give an
update on progress.
Finally, one of the invaluable project management tracking processes is that
of tracking Defect Reports. These have to converge towards zero outstanding
to support a release or update of a system. But are we able accurately to�
���
predict just when this will occur so we can set our customer’s expectations
confidently? Well, Eduardo Miranda provides us with a tutorial on using Line
of Balance to achieve this.
Mike Forrester
3Q08
TickIT
International
The quarterly journal of the TickIT software quality certification scheme ISSN 1354-5884
This publication is a means of communication with all TickIT-registered organizations and TickITminded individuals throughout the world. It also acts as an information exchange and soundingboard for anyone committed to IT quality.
The editorial team always welcomes input and comment.
As editor I will invite comment and debate on specific issues within each edition so as to develop
thinking and promote excellence within our industry.
To make comment or provide input please to e-mail the team.
click here
All copy and letters to be sent to the editor, at the following address:
TickIT Office, BSI, 389 Chiswick High Road, London W4 4AL.
Tel +44 (0)20 8996 7424 / Fax +44 (0)20 8996 7429
email: tickit@bsi-global.com
Copy on disk or email please – Copy should NOT be sent to the publisher
Copy Deadlines:
December 23 for publication January 15
March 31 for publication April 15
June 30 for publication July 15
September 30 for publication October 15
For advertising sales contact
Tina Shorter, Firm Focus, 26 Meadowside Road, Pangbourne, Berks RG8 7NH.
Tel +44 (0)118 984 3949 Fax +44 (0)118 984 2493
email: tina@firmfocus.co.uk
fir m focus
Published by Firm Focus on behalf of BSI July 2008
www:firmfocus.co.uk
© BSI and contributors 2008. Opinions expressed in this publication are not necessarily those of BSI.
3Q08
Assurance and Test – People and Skills
Assurance and Test – People and Skills
by: Richard Knight
Purpose
The purpose of this paper is to present a framework for viewing the relationship between training and alignment with
business objectives linked to a level of professionalism in the IT industry, in particular the software testing and assurance
sector.
Business Objectives and Professionalism
Organizations have the objective of being able to implement, quickly and efficiently, systems that support
their business processes and corporate objectives. To do this, it is recognized that the IT staff involved must be
appropriately conversant with all phases and activities of the Software Development Life Cycle, and to be able
to exercise their skills such that systems are delivered to meet requirements in an effective and efficient manner.
In addition, organizations have also recognized that growing the internal capability of their staff should help
define and build the right career path and opportunities to ‘up skill’ their staff, increase organizational capacity
and, significantly, aid staff and knowledge retention.
To this end, a key ‘people’ strategy can be seen as having a ‘team of people within the organization’ who:
• fulfil the core-competencies of different IT roles,
• meet the needs of the processes and toolsets used to underpin the life cycle,
• understand and possess the skills that are pertinent to the life cycle stage they work in and the activities
that underpin it,
• are multi-skilled,
• add value to the business.
In the same way, the nature and scope of the IT industry has changed significantly, and continues to expand
for both embedded and IT systems:
• it is increasingly pervasive; year on year, the number of IT systems and embedded systems continues to
grow,
• the business complexity and technical complexity of the projects in which we are engaged is increasing,
• the customers for IT are a wider group – global society rather than ‘just’ business and government so the
impact of software failure is more public,
• the customers require higher level quality attributes to support the complexity and change in their lives,
so increased reliability, maintainability, usability and performance characteristics are required.
If any of these pertained, the demand for testing would increase. In fact all of them pertain; therefore we have
a multiple increase in demand for software quality. This had led to an increased demand for software testing
which may be dealt with in a number of ways:
• Increasing the number of software and system testers, which implies that either we can tempt people with
high aptitude from other industries/activities or we drop the aptitude level for entry to the industry. The
first requires the incentive of joining a profession with status and rewards. The second requires a differentiation between profession and trade levels to protect customers, by allowing entry to roles by skill and
experience.
• Increasing the efficiency and effectiveness of existing software and system testers, which implies education, training, coaching and mentoring, resulting in recognition and evidence of professional competency
and certification of trade/craft skills.
• Improving the methods and tools used in software and system testing, which implies continuous development of the industry, including research and development supported by professional bodies.
• Increasing the scope and maturity of software and system testing, which implies increasing the understanding of the technical and business aspects of quality, society’s needs, and business and delivery programmes; this would be supported by a professional understanding of the testers’ role and by earlier
engagement in projects to take preventative measures.
3Q08
Assurance and Test – People and Skills
We also need to increase trust in our abilities. Other IT professionals, our customers and the general public
need to be able to trust us and to be aware of what services we provide. We need to support this trust with
awareness through increased and assured levels of competence, a clear ethical commitment, and awareness of
the needs of wider society, together with improvement through research and development. To support this,
professional status is needed.
In contrast to this need for professionalism, anyone can
‘…Classically, there were only three
claim to be a tester and start to charge for their services.
professions: ministry, medicine, and law.
There are at present no rules, training or mandatory
These three professions each hold to a
certification and employers often have low expectations of
specific
code of ethics, and members are almost
the skills and knowledge required. To address this, various
universally required to swear some form of oath
software testing training and certification schemes have
to uphold those ethics, therefore ‘professing’
been started by different groups.
to a higher standard of accountability. Each of
These include the ISEB (Information Systems
these professions also provides and requires
Examination Board), ISTQB (International Software
extensive training in the meaning, value, and
Testing Qualification Board, IIST (International Institute
importance of its particular oath in the practice
for Software Testing), CSTP (Certified Software Testing
of that profession. …’
Professional), AST (Association for Software Testing)
from the Wikipedia definition of ‘Profession’
(open certification) and ASQ (American Society for
Quality) schemes, as well as courses which do not lead to
examination and certification. I must declare an interest here; I have contributed to certification scheme syllabi,
worked with the accreditation panel for a particular scheme, written, contributed to and presented courses for
certification courses, and I work for a training provider. My own view is that each of the current schemes has
advantages and disadvantages – but I do not have room to describe, compare and contrast them all in this paper!
Views of the efficacy and usefulness of the schemes vary enormously. Some people in the industry argue that
certification is not necessary; others argue that it is necessary but the current schemes are not adequate and some
champion a particular scheme. The purpose, benefit, content, style of examination, degree of independence of
the examiner from the course provider, amount of practical work, ability of the examination to demonstrate the
competence of the candidate – all these vary across the schemes and are hotly debated in the software testing
industry, just as they have been in nursing, teaching and other aspirant professions.
Nevertheless, this need for training has led to the development of a generic training route map (see Figure 1).
Figure 1 – Generic Training Route Map
Assessment
Assessment in the form of the availability of people with the relevant technical skills and business knowledge,
to support projects/programmes for all stages of the development life cycle, should be risk assessed when project
3Q08
Assurance and Test – People and Skills
plans and project test strategies are being written, and development and test teams are being formed. In fact, this
is a requirement in the IEEE 829 standard for software test plans. Low skill levels, poor communications, and a
de-motivated workforce should also be considered as additional risks.
Role-Based Road Maps
These describe the key skills and training for the typical roles on an IT project. A good benchmark is the BCS
Skills Framework for the Information Age (SFIA): www.bcs.org/sfiaplus.
Learning Content
The learning content should be driven by the specific business needs and IT development environment of
the organization. To be an effective tester demands a subtle blend of business and application knowledge and
technical awareness, dependent on what level of testing is being undertaken.
Multiple Delivery Options
For effective training to be achieved the trainer needs to be aware of the different learning styles and delivery
methods preferred by the target audience.
Ongoing Self-Enablement
One of the key concepts of ‘ongoing self-enablement’ is the practice of Continuing Professional Development
(CPD). CPD is a combination of approaches, ideas and techniques that help manage your own learning and
growth. The focus of CPD is firmly on results and the benefits that professional development can bring in the
real world. Perhaps the most important message is that one size doesn’t fit all. Wherever you are in your career
now, and whatever you want to achieve, your CPD should be exactly that: yours.
CPD isn’t a fixed process; it’s a question of setting yourself objectives for development and then charting
your progress towards achieving them. It’s about where you want to be, and how you plan to get there. The
approach should focus on outcomes and results, rather than ‘time spent’ or ‘things done’, or how many boxes
you tick on a form. CPD is about capturing useful experiences and assessing the practical benefits of what you
have learned.
There is one decisive question that you should ask yourself to evaluate every piece of learning: What can you
do now that you couldn’t do before?
Similarly, when you record your CPD, it’s the value of the activity that counts. It’s not what you did, but how
you can use what you learned.
Should You Keep a CPD Record?
As a professional, you have a responsibility to keep your skills and knowledge up to date. CPD helps you turn
that accountability into a positive opportunity to identify and achieve your own career objectives.
At least once a year, you should review your learning over the previous 12 months, and set your development
objectives for the coming year. Reflecting on the past and planning for the future makes your development
more methodical and easier to measure. This is a particularly useful exercise prior to your annual appraisal!
Some people find it helpful to write things down in detail, while others record ‘insights and learning points’
in their diaries as they go along. This helps them to assess their learning continuously. These records and logs are
useful tools for planning and reflection: it would be difficult to review your learning and learning needs yearly
without regularly recording in some way your experiences. It is also a requirement of many assessment schemes
(ISO, CMMI, TMM and TickIT) that training records are maintained and that this assessment is undertaken at
both an organizational and individual level.
Certification
The basic level of certification tends to the BCS ISEB Foundation Certificate in Software Testing (www.bcs.org/
server.php?show=nav.6942 ). In fact, the recent Code of Best Practice for Software Testing produced by Intellect
(recently featured in TickIT International) highlighted this as one of the major ‘commitments’:
Approach for a Business-aligned Framework
This section provides a proposed approach for organising, managing and developing the skills and careers for
professionals working in assurance and testing across an organization consistent with a typical testing policy and
test strategy as referenced in both the ISEB/ISTQB certification schemes as detailed at: www.uktb.org.uk
3Q08
Assurance and Test – People and Skills
Commitment 3 – Resourcing and Professionalism
When undertaking testing projects we will ensure that all staff assigned have appropriate qualifications and
experience.
‘We will ensure that our staff acquires relevant professional qualifications such as the BCS ISEB Foundation
Certificate. We will encourage them to develop and maintain individual competencies by regularly attending training
events or conferences. Staff assigned will have experience suitable to the nature of the work being undertaken.
Professionalism in testing will demonstrate objectivity and communication in ways that are clear, precise and jargonfree. Good testers are thorough and diligent, have initiative, understand IT and business, are goal-aware, have
commitment to timescales, and accept responsibility for their work. Testers must own all observations and faults they
raise and remain involved until a satisfactory outcome is reached.’
The main elements of the approach are:
• assurance and testing affects everyone on a project, testing professionals and non-testing professionals
alike,
• to support the objective of detecting faults as early as possible,
• the need to ensure that assurance and testing is regarded as an integral part of everyone’s role,
• skills and training,
• raise the profile, credibility, and attractiveness of assurance and testing as a profession,
• create a flexible and effective testing ‘resource pool’ through:
– effective resource utilization and management,
– adopting a ‘job rotation’ model, thereby broadening experience and raising skill levels.
Assurance and Testing Roles
Everyone on a project has a stake in assurance and testing. Different people from the testing discipline and
other disciplines (such as requirements analysis, development, architecture and the business representatives)
need to be involved, at different times, and performing different (but complementary) activities throughout the
assurance and testing life cycle. Each individual’s involvement and the activities he/she performs on a project are
defined in a ‘role’. Typical assurance and testing roles on projects include:
• review leader,
• test automation specialist,
• review SME,
• test data specialist,
• reviewer,
• environment manager,
• component/unit test analyst,
• security test manager,
• test manager,
• performance test specialist,
• test team leader,
• usability test specialist,
• test analyst.
Roles should be distinct from the individuals’ job titles. Some of these may be taken up by people from any
discipline (for example, inspection can be taken up by someone from the technical architecture discipline, the
requirements analysis discipline, or the testing discipline). There are others, however, that can only be taken up
by people from a particular discipline (for example, test manager can only be taken up by someone from the
testing discipline).
Full information on a Skills Framework for the Information Age (SFIA), the high level UK governmentbacked competency framework describing the roles within IT and the skills needed to fulfil them, is available
from the British Computer Society website www.bcs.org/sfiaplus.
Skills and Training
There should be two aspects of skills and training, they are:
•career-based – those required for career development (within a job family),
•project-based – those required specifically for a project.
Career-based skills and training should be specified in a ‘�������������������������������������������������
job family���������������������������������������
’ to which individuals belong, whereas
project-based skills and training should be specified by the individual project to which the individual is
assigned. In addition, everyone carrying out assurance and testing activities (including the business user) should
3Q08
Assurance and Test – People and Skills
be trained and skilled in the processes and tools required to support risk-based assurance and testing. Also,
people from the development discipline and the business support area should be trained in assurance and testing
techniques relevant to their level of involvement.
Testing Discipline
A Testing Community
By combining all those identified from a typical company organization chart as being involved in assurance and
testing a cohesive ‘testing community’ can be established for an organization. This should be a logical entity in
which testing professionals can:
• have a sense of belonging (to a larger group of like-minded people),
• share knowledge and experiences,
• encourage and develop ‘best practice’.
Core Skills, Training and Qualification of Testing Professionals
There are fundamental skills that are needed by all testing professionals, regardless of role and grade, which
include the following:
• general IT skills, for example, Microsoft Office, Internet/intranet,
• knowledge of the systems (and applications) development life cycle,
• knowledge of quality and assurance and the important part testing plays in both,
• knowledge of risk management and a risk-based test process,
• effective communications,
• effective presentations,
• effective teamwork,
• negotiation skills,
• low level test techniques.
The Information Systems Examinations Board (ISEB) of the BCS offers qualifications widely recognized by
the IT industry around the world which include Software Testing:
• ISTQB Foundation Certificate.
• ISEB Practitioner Certificate.
It should be noted that the current ISEB Practitioner is being replaced by similar qualifications from the
International Software Testing Qualifications Board (ISTQB) during 2008.
As a minimum, everyone in a ‘testing community’ should have, or plan to obtain, the foundation certificate,
whilst professionals in more senior roles should have or plan to obtain the practitioner certificate. The most
senior professionals should aim for the ISEB Diploma in Software Testing when it becomes available.
Relationship with Other Disciplines
The testing discipline on a project cannot function effectively and efficiently in isolation, it is essential that
it ‘works’ in collaboration with all of the other disciplines on projects, particularly project management,
requirements analysis, architecture, design and development. It is important that the testing discipline is
proactive in involving other teams at appropriate times to add value to the assurance and testing life cycle.
Similarly, the testing discipline should provide help and advice, including training in specific assurance and
testing skills, to other disciplines on a project. These decisions should be documented in the project test
strategies and included in the project test plans.
Other Disciplines
Professionals from other (than testing) disciplines should play a major part in the assurance and testing process,
particularly during the static testing stages. They include:
• project/programme managers,
• requirements analysts,
• technical architects,
• designers,
• developers,
3Q08
Assurance and Test – People and Skills
• business representatives,
• live service support analysts.
For these people, assurance and testing should not be a full-time role, but an important part of their role
nonetheless. The level and extent of participation will vary depending on the individual’s role in these activities.
Skills and Training for Other Professionals
People from other (than testing) disciplines will also need to be educated in the importance of assurance and
testing and the principles of an assurance and test strategy. In addition they will need training in a review
process and techniques. Additionally, for developers, they will need training in low-level component testing
techniques, including static analysis and dynamic testing.
Assurance and Testing Techniques
For the static testing stages to be effective, people from other disciplines will also need to be trained in the
techniques used for static testing, such as:
• inspections (Fagan),
• reviews,
• walkthroughs,
• static analysis.
Developers would also require training in the principles of (dynamic) testing and the low-level component
testing techniques in particular.
More information on assurance and testing techniques and recommended training courses can be found
from most testing websites, for example: www.testing-solutions.com
Updated Learning Route Map/Framework
The key is alignment:
• business initiatives,
• staff objectives,
• learning and development objectives.
This leads to real practical benefits to both the individual and the business, skilling staff and organizations to
bring about self-sufficiency using the framework shown in Figure 2.
It is provided by a mix of tutor-led and self-study training supported by active mentoring to:
• skill the organization,
• skill the people,
• meet corporate and individual objectives,
• measure the training effectiveness.
Training Needs Analysis (TNA)
To establish and realize all the benefits of a cohesive, flexible and agile testing resource pool, organizational
and process changes may be necessary. These changes should be designed to move towards a job-rotation/
assignment-based model, the main elements of which are the alignment of objectives:
• business objectives and initiatives,
• learning and development objectives,
• staff objectives.
To make individuals ‘flexible’, that is, their work could be assignment-based or include ‘tours of duty’ in
deferent teams/departments being able to work on different projects across testing teams, thus widening and
increasing the level of skill and experience of both the individual and the resource pool as a whole. This TNA
is normally documented by SMART objectives at the annual staff appraisals (which can also be included in the
‘assessment’ phase in the generic route map).
Certification
As already mentioned, one aspect of certification is the linkage to external schemes such as ISEB/ISTQB, but this is
just one element of an overall certification framework. Many leading organizations are now adopting the concept of
an internal ‘training academy’, where the training is strongly aligned to specific business needs and deployed with a
blended approach of ‘hands on’ practical experience and ongoing assessment and support in the workplace.
3Q08
Assurance and Test – People and Skills
Training
needs
analysis
Planning
Business
initiatives
Validate
Learning and
development
objectives
Modular career
development
portfolio
Skilled
practitioners
Staff
objectives
Certification
ISEB/ISTQB
syllabi
External scheme
Internal scheme
Accredited
tutors
Delivery & Project Management
Instructor
led
Online
Client-specific
customization
Mentoring
Figure 2 – Updated Learning Route Map/Framework
Mentoring and Coaching
Mentoring is a long-standing form of training, learning and development and an increasingly popular tool for
supporting personal development. In a recent learning and development survey in 2007, the Chartered Institute
of Personnel and Development (CIPD) grouped coaching and mentoring together. However, mentoring is a
distinct activity.
Mentoring has become a widespread development tool, and we all know of famous mentoring relationships.
Ian Botham, for example, was mentored by Brian Close, Kevin Keegan by the great Bill Shankly. There are
many business mentoring relationships, notably Chris Gent and Arun Sarin at Vodafone, even Alan Sugar and
his apprentices! – as well as many examples from politics and other fields.
There is some confusion about what exactly mentoring is and how it differs from coaching. Broadly
speaking, the CIPD defines coaching as: ‘developing a person’s skills and knowledge so that their job
performance improves, hopefully leading to the achievement of organizational objectives. It targets high
performance and improvement at work, although it may also have an impact on an individual’s private life. It
usually lasts for a short period and focuses on specific skills and goals.’
Traditionally, mentoring is the long-term passing on of support, guidance and advice. In the workplace it has
tended to describe a relationship in which a more experienced ‘skilled practitioner’ uses their greater knowledge
and understanding of the work or workplace to support the development of a more junior or inexperienced
member of staff. This comes from the Greek myth where Odysseus entrusts the education of his son to his
friend Mentor. It’s also a form of apprenticeship, whereby an inexperienced team member learns the tricks of the
trade from an experienced colleague, backed-up, as in modern apprenticeship, by off-site training.
Mentoring is used specifically and separately as a form of long-term tailored development for the individual
which brings benefits to the organization. The characteristics of mentoring are:
3Q08
10
Assurance and Test – People and Skills
• it is essentially a supportive form of development,
• it focuses on helping an individual manage their career and improve skills,
• personal issues can be discussed more productively unlike in coaching where the emphasis is on performance at work,
• mentoring activities have both organizational and individual goals.
Table 1, adapted from Alred et al3, highlights the differences between mentoring and coaching. It is separate
and distinct from coaching, but coaching and mentoring can often overlap.
Mentoring
Coaching
Ongoing relationship that can last for a long time Relationship generally has a short duration
Can be more informal and meetings can take ���������������������������������������
Generally more structured in nature and
place as and when the mentored individual needs �������������������������������������
meetings scheduled on a regular basis
some guidance and or support
More long-term and takes a broader view of �����������������������������������
Short-term (sometimes time bounded)
the person. Often known as the ‘mentee’ but ������������������������������������������������
and focused on specific development areas/issues
the term client or mentored person can be used
Mentor usually passes on experience and is �������������������������������������������
Not generally performed on basis that coach
normally more senior in organization�������������������������������������������
needs direct experience of client’s formal�
occupational role
The focus is on career and personal development Focus generally on development/issues at work
Agenda is set by the mentored person with the �����������������������������������������������
Agenda focused on achieving specific, immediate
mentor providing support and guidance to prepare �����
goals
them for future roles
Revolves more around developing the mentee ������������������������������������������������
Revolves more around specific development areas/
professionally
issues
Table 1 – The Differences Between Mentoring and Coaching
Developing a Mentoring Approach
In Everyone Needs a Mentor1 Clutterbuck describes how mentoring works and the business benefits of the
approach.
Benefits to the Organization Are:
• significant impact upon recruitment and retention (one study found that the loss of young graduates in
the first expensive post training year was cut by two thirds),
• effective succession planning,
• makes organizations adapt to change,
• increased productivity through better engagement and job satisfaction.
Benefits to the mentored person are:
• development outcomes which may include, knowledge, technical and behavioural improvements,
• better management of career goals,
• developing wider network of influence,
• increased confidence and self awareness which help build performance and contribution,
• mentors also benefit from the satisfaction of developing their colleagues and of passing on their knowledge, skills and expertise,
• line managers and HR also benefit from better employee focus and engagement.
3Q08
11
Assurance and Test – People and Skills
Measurement of Success
The measurement of success of training effectiveness can be seen at both the company and individual level.
The Development of Thinking on Evaluating Training
In 1959, Kirkpatrick2 first outlined four levels for training evaluation:
• reactions – ‘liking or feelings for a programme’ known as ‘happy sheets’,
• learning – ‘principles, facts and so on absorbed’,
• behaviour – ‘using learning on the job’,
• results – ‘increased production, reduced costs, and so on’.
For the next 45 years the evaluation of training moved on very patchily in terms of research and new ideas,
and poorly in terms of practical application. In 2007, however, the CIPD produced a new ‘partnership of
learning model’ which emphasizes the need for all those involved in learning interventions actively to play their
part. The model is shown in detail in The value of learning: a new model of value and evaluation3. This contains
four strands:
1. proving,
2. improving,
3. learning (reinforcing),
4. controlling.
And involves all the stakeholders, the employer, line manager, individual learner and the trainer/facilitator.
Company-wide KPIs
Some Practical Tips:
Once a possible shift from ‘happy sheets’ to follow-ups is accepted, the what and how of information gathering
about the value of learning (and improved event design) become much easier.
The ‘what’ must include management information about issues important to the organization. And the
initial question to put to hard-pressed senior managers is: What measurable results (both short-term and long-term)
from the learning function would you like to see for your area of responsibility? Some possible measures, related to
each of the four main purposes for evaluation, are:
Relatively ‘hard’ metrics Relatively ‘soft’ measures
Proving – relating training to
Improving – relating training to
• data about reduced production and process ��������������������������������������������
• indications of greater harnessing of other
costs (or times) ��������������������������������������������
available learning and development processes
•����������������������������������������������������������������������������������������������
increased sales, market share, numbers of ���������������������������������������������������
• more courses perceived to be effective, valuable,
new customers, and so on ���������������������������������������������������
truly tailored, organizationally-focused, and so on
•���������������������������������������
increased service quality, stakeholder
satisfaction�����������
, and so on
Controlling – relating training to
Learning (reinforcing)
•�������������������������������������������������������������������������������������
reduced problems – for example accidents �������������������������������������������
• continually-improving skills/competencies
following health and safety training, grievances �������������������������������
– for example, better analysis,
following employee relations training �����������������������������������
problem solving and decision making
•��������������������������������������������������������������������������������
shorter, ‘smarter’ courses ����������������������������������������������������
• evidence that people are deriving a multiplicative
•������������������������������������������������������������������������������
more comprehensive / equitable training – ����������������������������������
effect from combining courses with
for example, covering all staff, access to ������������������������������
learning on the job, coaching,
prestigious events for those truly in need/able �������������������������������������
personal development plans, and so on
to apply the learning������������
, and so on�
3Q08
12
Assurance and Test – People and Skills
The ‘how’ includes gathering feedback from two major stakeholder groups: learners and their managers. The
main ways are:
• questionnaires from the learning function,
• interviews and focus groups,
• feedback from performance reviews,
• one-to-one discussions between managers and their staff,
• self-reporting by learners.
Individual Assessment of Training
This is normally undertaken by the individual as part of their ongoing CPD using the concept of ‘reflective
learning’. Reflecting on your learning enables you to link your professional development to practical outcomes
and widens the definition of what counts as useful activity. Quite simply, you need to keep asking ‘what did I
get out of this?’
Good Evaluation Questions:
1. What was the most important or valuable learning for you? Why? How did you learn this? – reflecting on
the immediate past experience and looking for insightful, long-lasting ‘ah-ha’ moments that help people to
see that they can learn, and how they learn.
2. How can you apply what you learned to the needs of your job, your team/department, and the priorities of
the business? – thinking about the present and looking for wider learning application and its relationship to
the business and its chosen metrics.
3. How could you integrate what you learned into further learning opportunities available to you, and the
developing requirements of your job, your team/department, the organization as a whole? – pointing to the
future and looking for the multiplicative effects of the variety of ways people can learn.
As a reflective learner, you’ll think about how you’ll use new knowledge and skills in your future activities
– so learning is always linked to action, and theory to practice. It’s also useful to reflect on how you learn best.
This may be through private study, networking with peers, formal courses, mentoring, or a combination of
techniques (including informative professional magazines like TickIT International).
To Conclude
Why is it important for me to reflect on my learning?
• to accept responsibility for your own personal growth,
• to help you see a clear link between the effort you put into your development activity and the benefits you
get out of it,
• to help you see more value in each learning experience, by knowing why you’re doing it and what’s in it for
you,
• to help you ‘learn how to learn’ and add new skills over time in your quest to become a valued ‘professional’.
References
Clutterbuck D ( 2004) Everyone needs a mentor, published via the CIPD
Kirkpatrick, D.L.(1959) Techniques for evaluating training programs. Journal of the American Society of
Training and Development. Vol 33
3 CIPD The value of learning: a new model of value and evaluation, via the Chartered Institute of Personnel and
Development (2007) Learning and development annual survey.
1
2
3Q08
13
TickIT Plus Progress Report
Richard S Knight has over 45 years’ experience in the computer industry, and has performed a wide range of jobs from quality manager,
trials engineer and technical costs consultant with MoD and various government departments to Director of Risk Management at Ernst
and Young. He has worked in a broad range of industry sectors from space and defence at Logica to high integrity banking and insurance
systems, healthcare, automotive, energy through to broadcast and media. He has also experienced working across the UK, Europe,
Australia, Japan, Russia and the USA and so has practical cultural diversity and
communication skills.
As a qualified electrical/electronics engineer and ISO 9000, CMMI, TMM EFQM
and ex-TickIT assessor, he has a wide range of experience with different software
development models from V to Agile together with ‘hands on’ experience for both
people and process improvement. On the training and skill transfer front he has
devised and delivered a range of training courses covering project management, risk
management total quality and value management, customer relationship and account
management, Six Sigma for software and agile systems development and testing.
Currently with Testing Solutions Group as Head of Consultancy Services he
specializes in business/software and test process improvement, and is a regular trainer
and coach for ISTQB Foundation to ISEB Practitioner exams.
The author doing some “hands on”
Richard can be contacted at: rknight@testing-solutions.com
coaching! at Brands Hatch
TickIT Plus – the Future of TickIT:
Project Progress, June 2008
by: Derek Irving
Last quarter I prepared an article on the TickIT Plus project proposals that described the basic concepts
of the scheme and the proposed timescales. In this and future articles I hope to keep you up to date with
developments. The TickIT scheme and this project are run by the Joint Technical Industry Steering Committee,
(JTISC), administered by BSI and involving several major companies, users and industry bodies, all of whom
have an interest in software and IT quality.
Getting agreement on such a major change in a scheme like TickIT, with all the implications for users and
with the involvement of a wide range of interested parties, was always going to be a challenge. But I’m pleased
to say that a consensus has been reached and we now have agreement on the fundamental specification for the
scheme. This has been achieved after months of debate, inputs from many quarters and not a few revisions
along the way. It is hoped, very soon, to start using the TickIT website to disseminate much of the ongoing
development information, but in the meantime if any reader wants to find out more details, please contact me
– address below.
Overall Progress
Having the specification baselined now means we can start on the next phase of the project, namely the design
of the scheme requirements, the development of training and the planning of a trials programme. Other aspects
such as the development of a marketing strategy and business planning also need to be considered. To give a
little more detail:
• Design – this is principally the requirements for scoping, process modelling and assessment and then how
this will be presented in the revized documentation format. This is only the first stage of course, but an
important fundamental starting point.
• Training and Auditor qualification criteria – we now have an agreed structure for this, (see below), are in
contact with training providers, and IRCA and will soon have training criteria material which can be used to
develop courses.
• Trials – successful trials are critical for the scheme and we hope to encourage as many participants
as possible, both large and small companies. At the basic level, the development of process models
3Q08
14
TickIT Plus Progress Report
– fundamental to TickIT Plus – can be done in-house without the need to go any further at this stage,
so the effort sought will be quite small. At the other end of the scale, we will need to conduct at least one
full assessment, probably at the ‘Silver’ level to test the principals and tools. We already have a number of
companies who have expressed an interest in participation and hope to confirm this with an agreed trials
plan soon. The planned period of the trials is October 2008 until January 2009.
• Marketing – on this front we have the support of Intellect, the UK IT industry representative, who are
assisting in the development of a strategy and marketing plan.
Auditor and Practitioner Qualifications
How auditors would qualify under TickIT Plus was still under debate when the last article was prepared. This
has now been agreed and there will be four levels of TickIT Plus qualification to match the capability grades:
Bronze, Silver, Gold and Platinum. This is needed to address the various requirements of transfer, training,
existing auditor gradings and the requirements of the scheme. How the auditors get to these levels is detailed in
the specification documents, which again can be obtained from myself or downloaded in future. Essentially, we
have tried to make it as straightforward as possible for existing TickIT Auditors to transfer at the Bronze level
with minimal additional training. There will also be two levels of Practitioner grades – standard and advanced
– for those who need to use the scheme as quality managers, developers, consultants and so on, but will not be
conducting audits.
Project Schedule
As mentioned above, the planned period for the trials is between this October and January. We are looking at
a public launch of the scheme in February, the provision of the first courses in April, and accreditation of the
first certification body in June 2009. Still an ambitious plan, but achievable given the support available. Once
the scheme is officially launched, there will be a three year migration period for both auditors and certificated
organizations.
Summary
In future articles I hope to give, as well as a progress update, an in-depth
discussion on some of the more technical aspects of the scheme as this is
developed. As always, we welcome comments and suggestions from all quarters,
so please contact the project either via the TickIT website or myself on
dkirving@iee.org.
Derek Irving
TickIT Development Project Manager
Software
Measurement
Do your capabilities match
ISO 9001:2000 requirements?
Measurement programme
consultancy, support, set-up
Visit
www.qasoft.demon.co.uk
DIY WORKBOOKS
save time and money
ISO 9001 TickIT
ISO 9001 RAD
ISO 9001 SSADM
ISO 9001 PRINCE
ISO 9001 Service Delivery
Listed on the Internet
http://www.nvo.com/management_systems–author
E. Sutherland, IEE Ind Aff
Trog Associates Ltd
P.O. Box 243
UK – South Croydon/Surrey, CR2 6NZ.
Tel +44 (0) 208 786 7094
Fax +44 (0)208 686 3580
Email trog@dial.pipex.com
3Q08
ISO 9000/BS7799
Guaranteed Results!
• 100% clients registered first time
• Consultant/Trainer since 1990
• Lead TickIT Auditor since 1992
• TEC and Business Link funded projects
• Practical advice and training
Consultancy Training Pre-Assessments
Bal Matu BSc(Hons) C.Eng.MIIEE FIQA MAQMC
Lead TickIT Auditor
Tel/Fax: 44 (0)1928 723701
bal@bsmqa.demon.co.uk
You are the project manager of a large project and testing is uncovering faults, trouble reports are starting to pile up and the
release date is coming soon. Are they going to be fixed on time? What could you do to help? Are there any bottlenecks? Where
should you assign more resources? Does this scenario sound familiar? Have you been there? This article will explain how you
can answer these questions by using an old method called Line of Balance in a new way.
Using LOB to Track the Progress of Fixing TRs
f balance (LOB) was devised by by a given time to meet the project dead- of it, time stamping each TR as they tranmembers of a group headed by lines. In other words, although the chart sition between states. This last feature
E. Fouch during the 1940’s to will give the project manager a gut feeling would allow the organization to produce
production at the Goodyear Tire about the situation, it would not answer the lead-time information required by the
er Company [1]. It was also suc- the questions of where are we in relation LOB method.
In addition to the state and timing
applied to the production plan- to where we are suppose to be, or how
d scheduling of the huge Navy much better we should be doing to get information, the TR includes other data
such as the severity of the problem. This
by the
time we want.
tion program of World War II where we want to get
by:
Eduardo
Miranda
information could be used to filter the TR
ng the Korean hostilities. Today,
apply themagazine)
LOB method to a subLife
Cycle
plication has been further expand- The TR
(Note this article first appeared in data
STSCand
Crosstalk
ng it suitable for a whole spec- Typically, a TR will go through a number set of all the TRs reported and in the prithe projectofmanager
of astates
large since
projectit and
testing isuntil
uncovering
faults;of
trouble
reports
arefix
starting
which
TRs to
first. to pile up and
oritization
stages or
is reported
activities ranging You
fromareresearch
the
release
date
is
coming
soon.
Are
they
going
to
be
fixed
on
time?
What
could
you
do
to
help?
Are
there
any bottlenecks?
elopment through job-shop and it is closed (see Figure 2, page 24). Each of
The
LOB
Method
these more
statesresources?
corresponds
milestone
in familiar? Have you been there? This article will explain
Does to
thisa scenario
sound
low operations. Where should you assign
process
answering
TRusing
into an
which
e context of managing a software
Applied
to TRs
how youthecan
answerofthese
questionsa by
old method
called Line
of Balance in a new way.
the LOB technique offers two the organization or project manager wants The LOB method consists of the followprogress,
to have
vantages over the traditional
Open (LOB)
elements
[4]:
Line of Balance
wasvisibility
devizedtobyevaluate
the members
of ai.e.,
grouping
headed
by George
E Fouch during the 1940s
how many TRs have been reported, how • A number
eports (TRs) Chart [2]:
of control points and their
[1]
to monitor
Tire have
and Rubber
Company
It wastoalso
successfully
appliedinto the
ofthe
theGoodyear
reported TRs
been anaows project managers
to see,production
in many at
lead .times
closing
as illustrated
production
planning
and
scheduling
of
the
huge
Navy
mobilization
program
of
World
War
II
during the
Figure 3 (see page 24), atandwhich
middle of a project, whether they lyzed, how many of the analyzed were
Korean
hostilities.
Today,
LOB
application
has
been
further
expanded,
making
it
suitable
for
a
whole
spectrum
rejected
and
so
on.
Elemental
states
could
progress is to be monitored.
meet the schedule if they continue
be
grouped
into
super
sets
for
reporting
Objective
Chart
or
target
plan
dis•
An
ng as they have been.
of activities ranging from research and development through job-shop and process flow operations.
i.e.,a while
theproject,
project the
manager
playing
thetwo
cumulative
closing schedposes process bottlenecks,
In theallowcontext purposes,
of managing
software
LOB technique
offers
main advantages
over the
might
be
interested
in
how
many
have
ule
as
planned
by
the
project
manager
to
focus
on
he project manager
traditional Open Trouble Reports (TRs) Chart [2]:
to meet a set deadline (Figure 4).
been analyzed, assigned, implemented, or
points responsible for slippage.
• it allows projectintegrated
managersthe
to see,
in thegroup
middle
of a project,• whether
theyStatus
can meet
the(see
schedule
they
The TRs
Chart
Figureif 5,
steering
overseeing
they have
page 25), which shows the actual numthe as
project
mightbeen,
only been interested in
pen TRs Chart continue working
• it exposes
process
allowing
the project
manager tober
focus
responsible
fora slippage.
TRsthose
that points
have passed
through
of on
howbottlenecks,
many TRs were
reported,
how many
er some of the questions
raised at
given control point versus the number
nning of this article, project man- were closed, and how many were still
that should have been passed (the
pending.
ally resort to the
Open
TRs Chart
The
Open
TRs Chart
LOB) according to the plan.
Most defect tracking systems will
Figure 1 or a variation of it.
contained
in theto the
information
The project
this raised
modelator
variation
To answer
some ofimplement
the questions
thesome
beginning
of this article,
managers
usually resort
Open TRs Chart shows
the cumumber of TRs written
Openover
TRstime,
Chart shown
Figure
1 or aReports
variation
Figure in
1: Open
Trouble
Chartof it.
breakdown into open and closed
Trouble Reports Over Time
the project progresses, the closed
350
ld converge toward the total line
open line towards zero. A closed
300
is not converging fast enough
he total or an open line that does
250
oach zero signals to the project
the need to devote additional
200
Total
s to fix problems.
Open
150
Closed
tions of the chart include showre detailed breakdown of the TR
100
nd ratios between total and open
].
50
ite all its usefulness, the Open TR
cks predictive ability and fails to
0
ntage of past and present perforata and TRs closure targets; i.e.,
ny TRs should be in a given state
15
26
/0
02 7/2
/0 00
09 8/2 4
/0 00
16 8/2 4
/0 00
23 8/2 4
/0 00
30 8/2 4
/0 00
06 8/2 4
/0 00
13 9/2 4
/0 00
20 9/2 4
/0 00
27 9/2 4
/0 00
04 9/2 4
/1 00
11 0/2 4
/1 00
18 0/2 4
/1 00
25 0/2 4
/1 00
01 0/2 4
/1 00
08 1/2 4
/1 00
15 1/2 4
/1 00
22 1/2 4
/1 00
29 1/2 4
/1 00
06 1/2 4
/1 00
2/ 4
20
04
Trouble Reports
Using Line of Balance to Track the Progress
of Fixing Trouble Reports
Figure 1 Open TRs Chart
Figure 1: Open Trouble Reports Chart
www.stsc.hill.af.mil
23
The Open TRs Chart shows the cumulative number of TRs written over time, and its breakdown into open
and closed TRs. As the project progresses, the closed line should converge toward the total line and the open
line towards zero. A closed line that is not converging fast enough toward the total or an open line that does not
approach zero signals to the project manager the need to devote additional resources to fix problems.
Variations of the chart include showing a more detailed breakdown of the TR status, and ratios between total
and open TRs [2, 3].
3Q08
16
Using LOB to Track the Progress of Fixing TRs
Despite all its usefulness, the Open TR Chart lacks predictive ability and fails to take advantage of past and
present performance data and TRs closure targets; that is, how many TRs should be in a given state by a given
time to meet the project deadlines. In other words, although the chart will give the project manager a gut feeling
about the situation, it would not answer the questions of where we are
��������������������������������������������
����������������������������������������
in relation to where we are supposed to
be, or how much better we should be doing to get where we want to get by the time we want.
The TR Life Cycle
Typically, a TR will go through a number of stages or states since
Software Engineering Technology
it is reported until it is closed (see Figure 2). Each of these states
corresponds to a milestone in the process of answering a TR into
Open
which the organization or project manager wants to have visibility to
Reported
evaluate progress, that is, how many TRs have been reported, how
Pending
many of the reported TRs have been analyzed, how many of the
Analyzed
analyzed were rejected and so on. Elemental states could be grouped
into super sets for reporting purposes, that is, while the project
Assigned
manager might be interested in how many have been analyzed,
assigned, implemented or integrated, the steering group overseeing
Implemented
the project might only been interested in how many TRs were
reported, how many were closed, and how many were still pending.
Integrated
Most defect tracking systems will implement this model, or some
variation of it, time stamping each TR as they transition between
states. This last feature would allow the organization to produce the
Closed
Verified
Rejected
Duplicated
lead-time information required by the LOB method.
In addition to the state and timing information, the TR includes
Figure
2: Typical
Life Cycle
Figure 2 Typical
TR life cycle
2: Typical
TR LifeTR
Cycle
other data such as the severity of the problem. This information could Figure
be used to filter the TR data and apply the LOB method to a subset of
Reported
all the TRs reported and in the prioritization of which TRs to fix first.
Objective
time infor
many TRs
given time
Control P
In LOB te
milestone o
wants to m
ing TRs, th
TR life cyc
this is not n
ject manag
track TRs
state woul
point. The
calculated
LeadTime n=
LeadTimeq=
Analyzed
The LOB Method Applied to TRs
Assuming
spends in
Table 1,
yield the re
Lead Times
The LOB method consists of the following elements [4]:
Assigned
• a number of control points and their lead times to closing as illustrated in Figure 3, at which progress
is
30
Days
The Obje
to be monitored,
25
Implemented
The Objec
Days
• an Objective Chart or target plan displaying the cumulative closing schedule as planned
19 by the project
verified TRs
Days
manager to meet a set deadline (Figure 4),
achievemen
Integrated
11
• the TRs Status Chart (see Figure 5), which shows the actual number of TRs that Days
have passed through a chart migh
5
achievemen
Days according to the plan.
given control point versus the number that should have been passed (the LOB)
Verified
The O
The information contained in the Objective Chart, together with the lead-time information is used to
that the pr
calculate how many TRs should be in a given state at a given time.
Figure 3: The Process of Solving a TR and Its
close 50 T
Corresponding Lead Times
by the end
Figure 3 The process of solving a TR and its corresponding lead times
Control Points
beginning
In LOB terminology, a control point is a milestone or event that the project
manager wants to monitor.
In the
Control Point Time in State
Lead Time
also show
context of tracking TRs, the control points and states in the TR life cycle would
coincide, 5but
Reportedmost likely
5
+ 25this
= 30is
progress is
6 + 19and
= 25
not necessary. For example, the project manager might not find it useful toAnalyzed
track TRs in6 the rejected state
delivering
Assigned
8
8 + 11 = 19
so this state would not be considered a control point. The lead time for a control
point is
the 80 pro
Implemented
6 calculated using
6 + 5 =the
11
Integrated
5
5+0=5
following formula:
0
Verified
0
TR Statu
LeadTimen = 0
The TR St
Table 1: Lead-Time Calculations
LeadTimeq=n-1,n-2,...n-1 = TimeInStateq + LeadTimeq+1
Assuming that the median2 times a TR spends in a given state are those shown in Table 1, the lead-time
To Be Verified
calculations will yield the results illustrated in Figure 3.
160
3Q08
uble Reports
140
120
100
80
60
TR life cycle would most likely coincide, but point is the
number
of TRs behind
o
1, the
wants1 to monitor. In the context of track-Table
LOB.
The lead-time
differencecalcu
betw
this
is
not
necessary
.
For
example,
the
proahead
of
schedule.
thethe
results
F
Implemented
Assigned
ing TRs, the control points and states in theyieldand
top illustrated
of the barin fo
ject manager might not find it useful to
Notice that the shape of the LOB wil
TR life cycle would most likely coincide,
but point is the number of T
30
track TRs in the rejected state and1 so this Days
change daily
even
if there are
no new TR
Objective
Chart
Integrated
this is not necessary . For example,
the pro-Theahead
of schedule.
25
state would Implemented
not be considered a control
reported,
since
its
calculation
depends
on
Closed
The
Objective
Chart
shows
cum
ject manager might not Days
find it useful to
Notice that the
shape
o
Verified
Rejected
Duplicated
19
Thethe
lead
time forofaFixing
controlTRs
point
is the plannedverified
curveTRs
of on
thethe
Objective
Char
Using LOBpoint.
to Track
Progress
scale
vertical
track TRs in the rejected
state and so this change daily even if there a
Days
calculated using
the following formula:
and the status date.
Integrated
state would not11 be considered a controlachievement
reported,along
sincethe
its horizont
calculati
Closed
The TRchart
Status
Chartalso
shows
that ather
Days
might
include
Figure 2 Typical
TR life cycle
Figure
2: Typical
TR LifeVerified
Cycle
Rejected
Duplicated
point.
The
lead
time
for
a
control
point
is
the
planned
curve
of thedisO
LeadTime n=0
5
are
almost
180
TRs
reported
so
far, 30
achievements
so
far.
Days
calculated
usingq+LeadTime
the following
formula:
and the status date.
q+1
LeadTimeq=n-1,n-2,..,n-1
=TimeInState
Verified
more than what
were planned
to Fig
fix
Chart
in
TheThe
Objective
Reported
TR Status Chart
sh
Figure 2 Typical
TR life cycle
Figure
2: Typical
TR Life Cycle
according to
thethe
Objective
Chart. This
sig
that
project
manager
has
co
LeadTime n=0
are
almost
180
TRs
repo
2
3: The
Processtimes
of Solving
a TRnals
and Its
Assuming Figure
thatLeadTime
the
median
a TR
the q+1
need
to50update
plan. of
It also
close
TRs
bythe
the
Sep
q
q=n-1,n-2,..,n-1=TimeInState +LeadTime
more
than
whatendwere
Analyzed
Reported
Lead
Times
spends Figure
in aCorresponding
given
state
are
those
shown
in
tells
us
that
TR
implementation
is
on
track
by
the
end
of
November,
and
3 The process of solving a TR and its corresponding lead times
according to the Objective
Table 1, the lead-time calculations will 2 as the actual
column and
thefollowing
LOB lineyea
fo
the
Assuming
that
the
median
times
a TRbeginning
nals theofneed
to update th
Control
Point
Time
in
State
Lead
Tim
e
yield the results illustrated in Figure 3.
that
control
point
coincide,
but
that
we
ar
Analyzed
that
as
of
mid
also
shows
Assigned
spends in a given
state are 5those
shown in tells us that TR implementa
Reported
5
+ 25falling
= 30 behind
in
their
integration
and
ver
progress
is
slightly
behind
with
30
Table 1, the
lead-time calculations
will as the actual column and th
Analyzed
6 + 19 = 25
Days
The Objective
Chart 68
ification.
This
suggests
that 75
adding mor
delivering
around
TR
Assigned
= 19 3.
yield the results
illustrated 8in+ 11
Figure
that control
point fixed
coincide,
Assigned
25
Implemented
The
Objective
Chart
shows
cumulative,
to
be
people
to
implementation
activities
wil
the
80
promised.
Implemented
6
6 + 5 = 11
Days
falling behind in their integ
30 TRs on
19
verified
and dates of
the vertical scale
Integrated
5
5 + 0 not
= 5 help recoup the delay, but that addi
Days
Days
The Objective
Chart 0
ification. This suggests tha
Verified
0
tional resources
could Chart
be used in integra
TR people
Status
25achievement along the horizontal scale. The
Integrated
Implemented
11
The
Objective
Chart
shows
cumulative,
to
be
to implementation
Days
Days
chart
might
also
include
a
display
of
the
tion
and
verification
activities.
TR
Status
Chart the
provides
The
19
Table
1: Lead-Time
Calculations
verified
TRs on the
vertical scale and dates of not help recoup
delay,
5
Table
Days
achievements so
far. 1: Lead-Time CalculationsThe LOB for each control point is cal
Days
along
the
horizontal
scale.
The
achievement
tional
resources
could
be u
Integrated
Verified
11
The Objective Chart in Figure 4 shows culated as follows:
Days
chart might also include a display
of Verified
the tion and verification activit
To
Be
that the project manager has committed to
5
achievements so far.
The LOB for each cont
Figure
Process
of Solving
a TR and
Figure3:3:The
The
Process
ofDays
Solving
a Its
TR close 50 TRs by
the160
end of September, 80
a1 + b1t t1<_ t <t2
Verified
Chart
in
Figure
4
shows
The
Objective
culated
as follows:
Corresponding
Lead Times
and Its Corresponding
Lead Times
by the end of November, and 150 by the
a2 + b2t t2<_ t <t3
Figure 3 The process of solving a TR and its corresponding lead times
that 140
the project manager has committed to
following year. The chart
.
Figure 3: The Process of Solving a beginning
TR and Itsof the
LOBq=1,2,..,n80
close 50 TRs by the end of September,
a1 + b1t
Control Point Time in State
Lead Time
that
as of mid-December
also
shows
.
Corresponding
Lead
Times
120
by
the
end
of
November,
and
150
by
the
a
2 + b2t
Figure 35The process of solving5a +
TR25
and=its
corresponding lead times
Reported
30
The Objective Chart
progress is slightly behind with the project
.
beginning of the following year. The chart
Analyzed
.
6
6 + 19 = 25
LOB
delivering
fixed
TRs scale
instead ofdates of achievement
+ bmt tm<_ t <tm+1
aalong
100
The Objective
ChartPoint
shows8Time
cumulative,
toLead
be verified
TRs
on75
the
vertical
mq=1,2,..,n
Time around
Assigned
8Control
+ 11in= State
19
that as and
of mid-December
also
shows
.
promised.
Implemented
6
= 11
5 + 5might
+ 2580
= 30
the horizontal6Reported
scale. The chart
also5the
include
a display
of80theisachievements
sowith
far. the project
progress
slightly
behind
.
Integrated
5Analyzed
5+0=5
6 + 19 = 25
The Objective
Chart in068Figure 4 shows
that
the
project
manager
has
committed
to
close
50
TRs
by
the
end
delivering
around
75
fixed
TRs
instead
of
a
m + bmt
Verified
0Assigned
8TR
+ 11Status
= 19
y - yi
Chart 60
bi = alsoi+1shows
thebeginning
80 promised.
of September,Implemented
80 by the end6 of November,
150
by the
ofquantitative
the following year. The chart
6 +and
5 = 11
TR
Status
Chart
provides
The
ti+1 - ti
Table
Calculationsprogress
Integrated
5
5+0=5
that1:asLead-Time
of mid-December
is slightly
behind, with the40
project delivering around 75 fixed TRs instead of
0
Verified
0
yi+1- yi
TR Status Chart
i =
the 80 promised.
yi and yi+1 are the number bof
TRs to be
TR
Status
Chart
provides
quantitative
The
To
Be
Verified
ti+1 - ti
20
Table 1: Lead-Time Calculations
fixed by time ti and t1+1 respectively, as
Lead Times
Integrated
Lead Times
{ {
Trouble Reports
Lead Times
17
160
140
Trouble Reports
planned by the project manager and
yi and yi+1 are the number
captured in
the objective
chart.
1-Aug-04
20-Sep-04
9-Nov-04
fixed by29-Dec-04
time ti and17-Feb-0
t1+1 res
planned by the project ma
Figure 4: The Plan Proposed by the PM to Clear the TRaBacklog
i = y i - b it i
captured in the objective
t =TimeNow+LeadTime q
0
To Be Verified
12-Jun-04
160
120
140
100
120
24
Figure 4 The plan proposed by the PM to clear the TR backlog
CROSSTALK The Journal of Defense Software Engineering
a i = y i - b it i
Trouble Reports
The idea behind the procedure is simple
t =TimeNow+Lea
Actual
If it takes an average of 10 days for a TR
100
to goPlanned
from a given state to the completion
60
The idea behind the proce
state,Actual
today’s status for that state should
80
If it takes an average of 10
40
be equal to the number of TRs that would
to go from a given state to
60
have to be completed according to th
20
state, today’s status for th
plan 10 days from now. See Figure 6 for
40
be equal to the number of
graphical example.
0
have to be completed ac
20
In Figure 6, the chart on the left show
12-Jun-04
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
17-Feb-05
plan 10 days from now. See
the planned line from Figure 4, while th
graphical example.
0
Figure 4: The Plan Proposed by the PM to Clear the TR Backlog
chart on the right shows the scheduled
In Figure 6, the chart on
12-Jun-04
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
17-Feb-05
Figu
the
planned line from
Figure 4 The plan proposed by the PM to clear the TR backlog
24 CROSSTALK The
Journal of Defense Software Engineering
April 200
Figure
4:Figure
The Plan
by the
PM to Clear
Backlog
chart on the right shows
4:Proposed
The Plan
Proposed
by the
theTR
PM
to Clear the TR Backlog
80
24
Planned
Figure 4 The plan proposed by the PM to clear the TR backlog
CROSSTALK The Journal of Defense Software Engineering
TR Status Chart
The TR Status Chart provides quantitative information with regards to progress, and whether or not there is a
bottleneck on the process.
The chart portrays the actual number of TRs that have passed through each control point against the
number that should have been passed according to the plan. These last quantities are called the LOB. The
difference between the LOB and the top of the bar for each control point is the number of TRs behind or ahead
of schedule.
3Q08
Control Points
control point against the number that
In LOB terminology, a control point is a should have been passed according to the
Assigned
milestone or event that the project manager plan. These last quantities are called the
wants to monitor. In the context of track- LOB. The difference between the LOB
Implemented
ing TRs, the control points and states in the and the top of the bar for each control
TR lifeUsing
cycle LOB
wouldtomost
coincide, but
point
Tracklikely
the Progress
of Fixing
TRsis the number of TRs behind or
Integrated
this is not necessary1. Using
For example,
the
proahead
of schedule.
Line of Balance to Track the Progress of Fixing Trouble Reports
ject manager might not find it useful to
Notice that the shape of the LOB will
track TRs in the rejected state and so this change daily even if there are no new TRs
igure 5. We obtained Figure 5’s
Progress
Points as
12/12/2004
state would
not atbeStatus
considered
a of
control
reported, since its calculation depends on
Closed
200
ine by finding
the interception
Verified
Rejected
Duplicated
point. The lead time for a control point is the planned curve of the Objective Chart
e TimeNow line and the curve in
180
calculated using the following formula:
and the status date.
ve chart (the a1 +b2t,...,am + bmt
160
The TR Status Chart shows that there
Figure 2 Typical
TR life cycle
Figure
2:
Typical
TR
Life
Cycle
ove), which yields the value for
140LeadTime n=0
are almost 180 TRs reported so far, 30
d Control Point, that is the
120LeadTimeq=n-1,n-2,..,n-1=TimeInStateq+LeadTimeq+1
more than what were planned to fix
Reported
Actual
TRs
that should be on that
100
according toScheduled
the Objective
Chart. This sig(LOB)
Dec. 12, 2004. The interception
80Assuming that the median2 times a TR
nals the need to update the plan. It also
e curve
Analyzed and the line at TimeNow
60spends in a given state are those shown in tells us that TR implementation is on track
Implemented yields the LOB value for
40Table 1, the lead-time calculations will as the actual column and the LOB line for
mented Control Point.
that control point coincide, but that we are
20yield the results illustrated in Figure 3.
Assigned
falling behind in their integration and ver0
30
ry
Days
The Objective Chart
ification. This suggests that adding more
ng
a credible early warning
25
Implemented
The Objective Chart shows cumulative, to be people to implementation activities will
Days
enecks in the process of19 fixing
verified TRs on the vertical scale and dates of not help recoup the delay, but that addiDaysmanOB method helps project
along the horizontal scale. The tional resources could be used in integraachievement
Integrated
11
corrective
actionsDays
such as allochart might also include a display of the tion and verification activities.
Figure 5: Trouble Reports Status Chart
5 prioritizing the
e resources or
achievements
so far.5: Trouble Reports Status Chart
The LOB for each control point is calFigure
Days
there
is
still
time
to
do
it.
Verified
Chart
in Figure 4 shows culated as follows:
The Objective
TimeNow
+
LeadTime
Implemented
s of the data required toFigure
imple- 5 TRs status
chart
that
the
project
has committed
the
LOB
will manager
change daily
even if to
LOB 3:technique,
of that
Figure
The Processmost
of Notice
Solving
aitTRthe
andshape
Its of
close 50
TRs
the end ofdepends
September,
a1 + b1t t1<_ t <t2
there
no new TRs reported,
since
itsby
calculation
on 80
readily available
fromareyour
Corresponding
Lead Times
Be Verified
at Status Points as aof2 +
12/12/2004
by theToend
of November, and 150 by Progress
the
b2t t2<_ t <t3
3 The process of solving a TR and its corresponding lead times
thebeplanned
the status
king system or could
derivedcurve of the
160
160 Objective
beginningChart
of theand
following
year.date.
The
chart
.
LOBq=1,2,..,n
h aControl
few calculations
Status
shows
that there
180 TRs140
Point Time implementin The
StateTRLead
TimChart
e
thatareasalmost
of mid-December
also shows
.
140
or
any other spreadsheet.�
Reported
5 reported so5far,
+ 2530
= 30
more than
what iswere
planned
to with
fix according
progress
slightly
behind
the project
.
Analyzed
6
6 + 19 = 25 120
120
delivering
around
75
fixed
TRs
instead
of
tm<_ t <tm+1
a
to
the
Objective
Chart.
This
signals
the
need
to
update
the
plan.
m + bmt
Assigned
8
8 + 11 = 19
ledgements
the
80
promised.
Implemented
6 It also tells us
6 + that
5 = 11TR100
100
implementation is on track as the actual
Jeremy
and GaeIntegratedO’Sullivan
5
5+0=5
column
and
the
LOB
line
for that control point coincide, but80
bardi
from
Ericsson;
Alain
0
Verified
0
80
y -y
TR Status Chart
bi = i+1 i
Technologie
om École de that
we are falling behind in
their
integration
and
verification.
TR
Status
Chart
provides
quantitative
The
ti+1 - ti
60
60
Table
1: Lead-Time
Calculations
- Université
du Québec;
and that adding
This
suggests
more people to implementation
their comnez from RMyA for
activities
will not help 40recoup the delay, but that additional 40
yi and yi+1 are the number of TRs to be
To Be Verified
earlier versions of this article;
fixed by time ti and t1+1 respectively, as
resources
could
be
used
20
20 in integration and verification activities.
n Corcoran
from Ericsson for
planned by the project manager and
160
The
LOB
for
each
control
point
is
calculated
as
shown
on
the
istics.
0
0
captured in the objective chart.
right:
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
140
ces
The idea behind the procedure is simple. If it takes an average
a i = y i - b it i
, Noel 120
N. “Line of
of 10
Balance.”
days for a TR to go from a given state to the completion
t =TimeNow+LeadTime q
Enterprise,
7 June state,
2003 today’s
<www. status for that state should be equal to the number
100
TimeNow Line
m>.
Planned
behind
the procedure
is simple.
of TRs that would have to be completed according
to the plan
10 daysThe
fromidea
now.
See Figure
6 for a graphical
(12/12/2004)
80 Juan. “Quality ConLaborde,
Actual
If it takes an average of 10 days for a TR
example.
Project Control?” Second Soft- Figure 6: The LOB for the Control Point
to go from a given state to the completion
60
In Figure
4, while the chart on the right shows
ngineering
Process Group
La- 6, the chart on the left shows the planned line from Figure
state, today’s status for that state should
merica Conference.
Mexico,
2005 line
2. from
The median
preferred
to Figure
the arithcomplex
TRs
from
skewing
the of
value
ofbetween
the
scheduled
Figure 5.is We
obtained
5’s scheduled
line
by
finding
thenumber
interception
40
be equal
to the
TRs
that would
si.es/SEPGLA/index_eng.
html>. line and
metic
to preventchart
rare but
statistic
to
the
right.
the TimeNow
themean
curve(average)
in the objective
(the a1 + the
b2t,...a
+
b
t
function
above),
which
yields
m
m
have
to
be
completed
according
to the
the
William 20A. “Software
value Quality
for the Verified Control Point, that is the number of TRs that should
be days
on that
state
as of
12,a
plan 10
from
now.
SeeDecember
Figure 6 for
ement: A Framework for
the
Author
example.yields the LOB value for
graphicalImplemented
0
2004.
The interception between the curve andAbout
the line at
TimeNow
+ LeadTime
and
Defects.”
ng Problems
In
Figure
6, the chart on the left shows
12-Jun-04 the
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
17-Feb-05
Implemented
Control Point.
SEI-92-TR-22. Pittsburgh,
PA:
development
Eduardo Miranda is a papers in software
Figure 4, while the
the planned
line frommethode Engineering
Institute,
Figure
4: The Plan
Proposed1992.
by the PM to Clear the TR system
Backlog professional with ologies, estimation,
chart on and
the project
right shows
the scheduled
managea, Eduardo. Summary
Running The
ment
and
is
the
author
of
“Running
the
20
years
of
experience
in
4 The plan
proposed
by the PM to clear the TR backlog
Project
Office.
fulFigure
Hi-Tech
24
CROSSTALK The Journal of Defense Software Engineering
April 2006
Successful
Project
Office.”
the development
of soft-in the
By providing a credible early warning
about bottlenecks
process ofHi-Tech
fixing TRs,
the LOB
method helps
House, 2003.
Miranda
has or
a Master
of Engineering
products
and more
University.
e Acquisition project
managers take correctiveware-based
actions such
as allocating
resources
prioritizing
the work when there is
degree
from
the
University
of Ottawa
information
management
ing for ProgramstillManagers.
time to do it.
e Systems Management. College systems. Currently, he works in the devel- and a master’s degree in project manageIn terms of the data required to implement the LOB technique, most of it should be readily available from
Oct. 2001 <www.dau.mil/pubs/ opment of new estimation and planning ment from the University of Linkoping.
your
system orforcould
be derived
it with a few calculations implemented in Excel™ or any
scheduling_guide.asp>. defect tracking
approaches
research
and from
developother spreadsheet.
119 Harwood Gate
ment projects. Miranda is affiliated with
ed
Ve
rif
i
In
te
gr
at
ed
ed
pl
em
en
te
d
Im
An
al
yz
As
si
gn
ed
ep
or
te
d
R
Lead Times
Trouble Reports in State
18
ntrol points are likely to be a
of the TR states. To avoid condo not create additional control
ed
ifi
Ve
r
In
te
gr
at
ed
en
te
d
d
ze
pl
em
An
al
y
si
gn
ed
As
Im
Trouble Reports
R
ep
or
te
d
Trouble Requests
{
Beaconsfield, Quebec
the Université du Québec à Montréal as
Canada H9W 3A5
an industrial researcher, and is a member
Phone: (514) 697-0594
of the International Electronics and En3Q08
E-mail: emt.miranda@computer.org
gineers. He has published more than 10
www.stsc.hill.af.mil
25
40
20
0
Tr
plemented Control Point.
mary
19
s the value for
140
t,owledgements
that is the
120
on that
sldtobeJeremy
O’Sullivan and100
Gaehe
interception
Lombardi
from Ericsson; 80
Alain
École de Technologie
nefrom
at TimeNow
60
eure
Université
LOB- value
for du Québec; and
40
Martinez
oint. from RMyA for their com20
on earlier versions of this article;
John Corcoran from Ericsson0 for
statistics.
Trouble Requests
Trouble Reports in State
ed
Ve
rif
i
In
te
gr
at
ed
ed
pl
em
en
te
d
Im
An
al
yz
As
si
gn
ed
R
ep
or
te
d
oviding a credible early warning
bottlenecks in the process of fixing
he LOB method helps project manUsing LOB to Track the Progress of Fixing TRs
ake corrective actions such as alloFigure
5:
Trouble
Reports
Status Chart
more resources or prioritizing the
Using Line of Balance to Track the Progress of Fixing Trouble Reports
when there is still time to do it.
erms of the data required toFigure
imple- 5 TRs status chart TimeNow + LeadTimeImplemented
the
LOB
technique,
most
of it
ined Figure 5’s
Progress at Status Points as of 12/12/2004
readily available from200your
hebe
interception
To Be Verified
Progress at Status Points as of 12/12/2004
tracking
system
160
160
nd
the curve
in or could be derived
180
with a few calculations implement+b
2t,...,am + bmt
140
160
140
Excel or any other spreadsheet.�
120
120
100
100
80
80
60
60
40
40
20
20
Actual
Scheduled (LOB)
ed
ifi
Ve
r
In
te
gr
at
ed
en
te
d
d
ze
pl
em
Im
An
al
y
As
si
gn
ed
ed
ep
or
te
d
R
Ve
rif
i
In
te
gr
at
ed
ed
ifi
Ve
r
In
te
gr
at
ed
en
te
d
d
pl
em
ze
Im
An
al
y
si
gn
ed
As
R
ep
or
te
d
Trouble Requests
Im
pl
em
en
te
d
ed
An
al
yz
As
si
gn
ed
R
ep
or
te
d
0
0
early warning
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
ocess
of fixing
rences
s project
rroff,
NoelmanN. “Line of Balance.”
s such
as allo-7 June 2003 <www.
NH
Enterprise,
TimeNow Line
h.com>.
rioritizing the Figure 5: Trouble Reports Status Chart
(12/12/2004)
Juan. “Quality Conssacq
Laborde,
e to do
it.
TimeNow
+
LeadTime
=
Project
Control?”
Second
SoftImplemented
uired toFigure
imple- 5 TRs status chart
Figure 6: The LOB for the Control
Point
Engineering
Figure 6: The LOB for the Control Point
,e most
of itProcess Group Lao America Conference. Mexico, 2005 2. The median is preferred to the arithcomplex TRs from skewing the value of
le
from your
Verified
Progress
Points as
12/12/2004
ww.esi.es/SEPGLA/index_eng. html>. To Bemetic
mean (average) to prevent rare
but at Status
the statistic
toofthe
right.
uld
be
derived
160
rac, William A.Acknowledgements
“Software160Quality
ns
implementasurement:
A Framework
for
140
140 to Jeremy O’Sullivan and Gaetano About
the
Author
Lombardi
from
Ericsson; Alain Abran from École de Technologie
eadsheet.�
and Defects.”
unting
Problems Thanks
Supérieure
– Université du Québec;
and Raul
MU/SEI-92-TR-22. Pittsburgh,
PA:
120 Martinez from RMyA for their comments on earlier versions of
120
Eduardo Miranda
is a papers in software development methodtware Engineering Institute,
1992.and to John Corcoran from Ericsson for the TR statistics.
this article;
with ologies, estimation, and project managesystem professional
100
100
randa,
Running The
van
andEduardo.
Gae20 years of experience in ment and is the author of “Running the
ccessful
icsson; Hi-Tech
Alain Project80Office.
the development80 of soft- Successful Hi-Tech Project Office.”
References
ech
House,
2003.
Technologie
ware-based products
and Miranda has a Master of Engineering
fense Acquisition University.
60
60
Québec; and 1. Harroff,
Noel N. ‘Line of Balance.’
NNH
Enterprise,degree
7 Junefrom
2003thewww.
nnh.com.
University
of Ottawa
information
management
heduling for Program
Managers.
for their
com40College
degree
in project
manage- Process Group
and a master’s
Currently,
heControl
works in the
devel- Control?’
systems.
fence
Systems
Management.
2. Pussacq
Laborde,
Juan.
‘Quality
=40 Project
Second
Software
Engineering
of
this 2001
article;
ss, Oct.
<www.dau.mil/pubs/
ment from the University ofhtml.
Linkoping.
opment
of new estimation
and20planning
20
Latino
America
Conference.
Mexico,
2005
www.esi.es/SEPGLA/index_eng.
m
Ericsson for
bks/scheduling_guide.asp>.
approaches for research and develop3. Florac,
William
A. projects.
‘SoftwareMiranda
Quality
for Counting Problems and Defects.’
0
0
119 Harwood Gate
is Measurement:
affiliated
with A Framework
ment
1-Aug-04
20-Sep-04
9-Nov-04
29-Dec-04
s
CMU/SEI-92-TR-22.
Pittsburgh,
PA:
Software
Engineering
Institute,
1992.
Beaconsfield,
Quebec
the Université du Québec à Montréal as
e control points are4.likely
to be aEduardo.
3A5
Miranda,
Running
The Successful
Office. H9W
Artech
House, 2003.
an industrial
researcher,
and is a Hi-Tech
member ProjectCanada
set
the TR states. To avoid conofofBalance.”
Phone:
(514)
697-0594
of
the
International
Electronics
and
En5. Defense
Acquisition University. Scheduling for Program Managers. Defence Systems Management. College Press,
control
do not
create additional
eion,
2003
<www.
E-mail: emt.miranda@computer.org
gineers. He has published
more than 10
Oct. 2001 www.dau.mil/pubs/
gdbks/scheduling_guide.asp
nts.
TimeNow Line
(12/12/2004)
“Quality Conwww.stsc.hill.af.mil 25
” Second Soft-Notes
Figure 6: The LOB for the Control Point
ess Group LaThe median
control is
points
are likely
to be
a subset of
the TRTRs
states.
Toskewing
avoid confusion,
. Mexico, 2005 2.1. The
preferred
to the
arithcomplex
from
the value ofdo not create additional
control
points.
dex_eng. html>.
metic
mean
(average) to prevent rare but
the statistic to the right.
ftware Quality 2. The median is preferred to the arithmetic mean (average) to prevent rare but complex TRs from skewing the
amework for
value of the statistic to theAbout
right. the Author
nd Defects.”
ittsburgh, PA:
Eduardo Miranda
is a system
with
20 years ofdevelopment
experience in the
development of software-based products and
methodsoftware
papers in
Miranda
Eduardo
is a professional
information management systems. Currently, he works in the development of new estimation and planning approaches
stitute, 1992.
estimation,
manageologies,
with projects.
system
professional
for research
and development
Miranda
is affiliatedand
withproject
the Université
du Québec à Montréal as an industrial
Running The
ment
and
is
the
author
of
“Running
the
20
years
of
experience
in
researcher,
and
is
a
member
of
the
International
Electronics
and
Engineers.
He has
published more than 10 papers in
roject Office.
Project Office.”
thesoftware
development
of methodologies,
soft- Successful
development
estimation,Hi-Tech
and project management
and is the author of ‘Running the Successful
Hi-Tech Project
Office’�������������������������������������������������������������������������������������������������
.������������������������������������������������������������������������������������������������
MirandaMiranda
has a Masterhas
of Engineering
the University of Ottawa and a master’s degree
a Masterdegree
of from
Engineering
ware-based
products
and
University.
in
project
management
from
the
University
of
Linkoping.
information management degree from the University of Ottawa
m Managers.
E-mail: emt.miranda@computer.org
ement. College systems. Currently, he works in the devel- and a master’s degree in project managedau.mil/pubs/ opment of new estimation and planning ment from the University of Linkoping.
.asp>.
approaches for research and develop119 Harwood Gate
ment projects. Miranda is affiliated with
Beaconsfield, Quebec
the Université du Québec à Montréal as
likely to be a an industrial researcher, and is a member
Canada H9W 3A5
To avoid con3Q08
Phone: (514)
697-0594
of the International Electronics and Enitional control
E-mail: emt.miranda@computer.org
gineers. He has published more than 10
www.stsc.hill.af.mil
25