Appendix 2 United States Agency for International Development

Transcription

Appendix 2
United States Agency for International Development
Performance Monitoring and Evaluation TIPS
NUMBER 1
2011 Printing
PERFORMANCE MONITORING & EVALUATION
TIPS
CONDUCTING A PARTICIPATORY EVALUATION
ABOUT TIPS
These TIPS provide practical advice and suggestions to USAID managers on issues related to
performance monitoring and evaluation. This publication is a supplemental reference to the
Automated Directive Service (ADS) Chapter 203.
WHAT IS DIRECT
OBSERVATION ?
USAID is promoting
participation in all aspects of its development
work.
Participatory evaluation provides for active involvement in the evaluation process of those
with a stake in the program: providers, partners, customers (beneficiaries), and any other
interested parties. Participation typically takes
place throughout all phases of the evaluation:
planning and design; gathering and analyzing the
data; identifying the evaluation findings, conclusions, and recommendations; disseminating results; and preparing an action plan to improve
program performance.
This TIPS outlines how
to conduct a participatory
evaluation.
CHARACTERISTICS OF
PARTICIPATORY
EVALUATION
1
Participatory evaluations typically share several cally, rapid appraisal techniques are used to decharacteristics that set them apart from trad- termine what happened and why.
tional evaluation approaches. These include:
Use of facilitators. Participants actually conParticipant focus and ownership. Partici- duct the evaluation, not outside evaluators as is
patory evaluations are primarily oriented to traditional. However, one or more outside exthe information needs of program stakehold- perts usually serve as facilitator—that is, proers rather than of the donor agency. The donor vide supporting roles as mentor, trainer, group
agency simply helps the participants conduct processor, negotiator, and/or methodologist.
their own evaluations, thus building their ownership and commitment to the results and facilitating their follow-up action.
WHY CONDUCT A
PARTICIPATORY
Scope of participation. The range of particiEVALUATION?
pants included and the roles they play may vary.
For example, some evaluations may target only
program providers or beneficiaries, while othExperience has shown that participatory evaluers may include the full array of stakeholders.
ations improve program performance. Listening
Participant negotiations. Participating to and learning from program beneficiaries, field
groups meet to communicate and negotiate to staff, and other stakeholders who know why a
reach a consensus on evaluation findings, solve program is or is not working is critical to makproblems, and make plans to improve perfor- ing improvements. Also, the more these insiders are involved in identifying evaluation quesmance.
tions and in gathering and analyzing data, the
Diversity of views.Views of all participants are more likely they are to use the information to
sought and recognized. More powerful stake- improve performance. Participatory evaluation
holders allow participation of the less powerful. empowers program providers and beneficiaries
to act on the knowledge gained.
Learning process. The process is a learning experience for participants. Emphasis is on Advantages to participatory evaluations are
identifying lessons learned that will help partici- that they:
pants improve program implementation, as well
as on assessing whether targets were achieved. • Examine relevant issues by involving key
players in evaluation design
Flexible design. While some preliminary
planning for the evaluation may be necessary,
design issues are decided (as much as possible)
in the participatory process. Generally, evaluation questions and data collection and analysis
methods are determined by the participants,
not by outside evaluators.
• Promote participants’ learning about the
program and its performance and enhance
their understanding of other stakeholders’
points of view
• Improve participants’ evaluation skills
• Mobilize stakeholders, enhance teamwork,
and build shared commitment to act on evalua-
Empirical orientation. Good participatory
evaluations are based on empirical data. Typi2
ment exists among stakeholders that a collaborative approach is likely to fail.
tion recommendations
• Increase likelihood that evaluation informaStep 2: Decide on the degree of particition will be used to improve performance
pation. What groups will participate and what
roles will they play? Participation may be broad,
But there may be disadvantages. For example,
with a wide array of program staff, beneficiaries,
participatory evaluations may
partners, and others. It may, alternatively, tar• Be viewed as less objective because program get one or two of these groups. For example,
if the aim is to uncover what hinders program
staff, customers, and other stakeholders
implementation, field staff may need to be inwith possible vested interests participate
volved. If the issue is a program’s effect on local communities, beneficiaries may be the most
• Be less useful in addressing highly technical
appropriate participants. If the aim is to know
aspects
if all stakeholders understand a program’s goals
and view progress similarly, broad participation
• Require considerable time and resources to
identify and involve a wide array of stakehold- may be best. Roles may range from serving as
a resource or informant to participating fully in
ers
some or all phases of the evaluation.
• Take participating staff away from ongoing
Step 3: Prepare the evaluation scope of
activities
work. Consider the evaluation approach—the
• Be dominated and misused by some stake- basic methods, schedule, logistics, and funding.
Special attention should go to defining roles of
holders to further their own interests
the outside facilitator and participating stakeholders. As much as possible, decisions such as
STEPS IN CONDUCTING A the evaluation questions to be addressed and
the development of data collection instruments
PARTICIPATORY
and analysis plans should be left to the particiEVALUATION
patory process rather than be predetermined
in the scope of work.
Step 1: Decide if a participatory evaluation approach is appropriate. Participatory
evaluations are especially useful when there are
questions about implementation difficulties or
program effects on beneficiaries, or when information is wanted on stakeholders’ knowledge
of program goals or their views of progress.
Traditional evaluation approaches may be more
suitable when there is a need for independent
outside judgment, when specialized information
is needed that only technical experts can provide, when key stakeholders don’t have time to
participate, or when such serious lack of agree-
Step 4: Conduct the team planning meeting. Typically, the participatory evaluation process begins with a workshop of the facilitator
and participants. The purpose is to build consensus on the aim of the evaluation; refine the
scope of work and clarify roles and responsibilities of the participants and facilitator; review
the schedule, logistical arrangements, and agenda; and train participants in basic data collection and analysis. Assisted by the facilitator, participants identify the evaluation questions they
want answered. The approach taken to identify
questions may be open ended or may stipulate
3
broad areas of inquiry. Participants then select
appropriate methods and develop data-gathering instruments and analysis plans needed to
answer the questions.
and interpreting them help participants build a
common body of knowledge. Once the analysis
is complete, facilitators work with participants
to reach consensus on findings, conclusions, and
recommendations. Facilitators may need to negotiate among stakeholder groups if disagreements emerge. Developing a common understanding of the results, on the basis of empirical
evidence, becomes the cornerstone for group
commitment to a plan of action.
Step 5: Conduct the evaluation. Participatory evaluations seek to maximize stakeholders’ involvement in conducting the evaluation
in order to promote learning. Participants define the questions, consider the data collection
skills, methods, and commitment of time and labor required. Participatory evaluations usually
use rapid appraisal techniques, which are simpler, quicker, and less costly than conventional
sample surveys. They include methods such as
those in the box below. Typically, facilitators are
skilled in these methods, and they help train
and guide other participants in their use.
Step 7: Prepare an action plan. Facilitators
work with participants to prepare an action
plan to improve program performance. The
knowledge shared by participants about a program’s strengths and weaknesses is turned into
action. Empowered by knowledge, participants
become agents of change and apply the lessons
they have learned to improve performance.
Step 6: Analyze the data and build consensus on results. Once the data are gathered, participatory approaches to analyzing
WHAT’S DIFFERENT ABOUT PARTICIPATORY
EVALUATIONS?
Traditional Evaluation
Participatory Evaluation
• participant focus and ownership of
evaluation
• donor focus and ownership of evaluation
• broad range of stakeholders participate
• stakeholders often don’t participate
• focus is on accountability
• focus is on learning
• predetermined design
• flexible design
• formal methods
• rapid appraisal methods
• outsiders are evaluators
• outsiders are facilitators
4
may be selected through probability or
nonprobability sampling techniques, or
through “convenience” sampling (interviewing stakeholders at locations where
they’re likely to be, such as a clinic for
a survey on health care programs). The
major advantage of minisurveys is that
the datacan be collected and analyzed
within a few days. It is the only rapid appraisal method that generates quantitative data.
Rapid Appraisal Methods
Key informant interviews. This involves interviewing 15 to 35 individuals
selected for their knowledge and experience in a topic of interest. Interviews are
qualitative, in-depth, and semistructured.
They rely on interview guides that list
topics or open-ended questions. The interviewer subtly probes the informant to
elicit information, opinions, and experiences.
Case studies. Case studies record
anedotes that illustrate a program’s
shortcomings or accomplishments. They
tell about incidents or concrete events,
often from one person’s experience.
Focus group interviews. In these,
8 to 12 carefully selected participants
freely discuss issues, ideas, and experiences among themselves. A moderator introduces the subject, keeps the
discussion going, and tries to prevent
domination of the discussion by a few
participants. Focus groups should be
homogeneous, with participants of similar backgrounds as much as possible.
Village imaging. This involves
groups of villagers drawing maps or diagrams to identify and visualize problems
and solutions.
Selected Further Reading
Community group interviews.
Aaker, Jerry and Jennifer Shumaker. 1994.
Looking Back and Looking Forward: A Participatory Approach to Evaluation. Heifer Project
International. P.O. Box 808, Little Rock, AK
72203.
These take place at public meetings
open to all community members. The primary interaction is between the participants and the interviewer, who presides
over the meeting and asks questions,
following a carefully prepared questionnaire.
Aubel, Judi. 1994. Participatory Program Evaluation: A Manual for Involving Program Stakeholders in the Evaluation Process. Catholic
Relief Services. USCC, 1011 First Avenue, New
York, NY 10022.
Direct observation. Using a detailed
observation form, observers record what
they see and hear at a program site. The
information may be about physical surroundings or about ongoing activities,
processes, or discussions.
Minisurveys. These are usually
Freeman, Jim. Participatory Evaluations: Making
Projects Work, 1994. Dialogue on Development Technical Paper No. TP94/2. International
Centre, The University of Calgary.
based on a structured questionnaire with
a limited number of mostly closeended
questions. They are usually administered to 25 to 50 people. Respondents
Feurstein, Marie-Therese. 1991. Partners inEvaluation: Evaluating Development and Community Programmes with Participants. TALC,
5
Box 49, St. Albans, Herts AL1 4AX, United
Kingdom.
Guba, Egon and Yvonna Lincoln. 1989. Fourth
Generation Evaluation. Sage Publications.
Pfohl, Jake. 1986. Participatory Evaluation: A
User’s Guide. PACT Publications. 777 United
Nations Plaza, New York, NY 10017.
Rugh, Jim. 1986. Self-Evaluation: Ideas for
Participatory Evaluation of Rural Community
Development Projects. World Neighbors Publication.
6
1996, Number 2
Performance Monitoring and Evaluation
TIPS
USAID C enter for D evelopment I nformation and E valuation
CONDUCTING KEY INFORMANT INTERVIEWS
What Are Key Informant Interviews?
USAID reengineering
emphasizes listening
to and consulting
with customers, partners and other stakeholders as we undertake development
activities.
Rapid appraisal techniques offer systematic ways of getting
such information
quickly and at low
cost. This Tips advises how to conduct
one such method—
key informant interviews.
They are qualitative, in-depth interviews of 15 to 35 people selected
for their first-hand knowledge about a topic of interst. The interviews are loosely structured, relying on a list of issues to be discussed. Key informant interviews resemble a conversation among
acquaintances, allowing a free flow of ideas and information. Interviewers frame questions spontaneously, probe for information and
takes notes, which are elaborated on later.
When Are Key Informant Interviews Appropriate?
This method is useful in all phases of development activities—
identification, planning, implementation, and evaluation. For example, it can provide information on the setting for a planned activity that might influence project design. Or, it could reveal why
intended beneficiaries aren’t using services offered by a project.
Specifically, it is useful in the following situations:
1. When qualitative, descriptive information is sufficient for decision-making.
2. When there is a need to understand motivation, behavior, and
perspectives of our customers and partners. In-depth interviews
of program planners and managers, service providers, host
government officials, and beneficiaries concerning their attitudes
and behaviors about a USAID activity can help explain its
successes and shortcomings.
3. When a main purpose is to generate recommendations. Key
informants can help formulate recommendations that can improve a program’s performance.
4. When quantitative data collected through other methods need to
be interpreted. Key informant interviews can provide the how
and why of what happened. If, for example, a sample survey
showed farmers were failing to make loan repayments, key
informant interviews could uncover the reasons.
PN-ABS-541
2
5. When preliminary information is needed to
design a comprehensive quantitative study.
Key informant interviews can help frame the
issues before the survey is undertaken.
Advantages and Limitations
Advantages of key informant interviews include:
•
they provide information directly from
knowledgeable people
•
they provide flexibility to explore new ideas
and issues not anticipated during planning
•
they are inexpensive and simple to conduct
Some disadvantages:
•
they are not appropriate if quantitative data are
needed
•
they may be biased if informants are not
carefully selected
•
•
they are susceptible to interviewer biases
Step 3. Select key informants.
The number should not normally exceed 35. It is
preferable to start with fewer (say, 25), since often
more people end up being interviewed than is
initially planned.
Key informants should be selected for their specialized knowledge and unique perspectives on a
topic. Planners should take care to select informants with various points of view.
Selection consists of two tasks: First, identify the
groups and organizations from which key informants should be drawn—for example, host government agencies, project implementing agencies,
contractors, beneficiaries. It is best to include all
major stakeholders so that divergent interests and
perceptions can be captured.
Second, select a few people from each category
after consulting with people familiar with the
groups under consideration. In addition, each
informant may be asked to suggest other people
who may be interviewed.
Step 4. Conduct interviews.
it may be difficult to prove validity of
findings
Once the decision has been made to conduct key
informant interviews, following the step-by-step
advice outlined below will help ensure highquality information.
Establish rapport. Begin with an explanation of
the purpose of the interview, the intended uses of
the information and assurances of confidentiality.
Often informants will want assurances that the
interview has been approved by relevant officials.
Except when interviewing technical experts,
questioners should avoid jargon.
Steps in Conducting the Interviews
Step 1. Formulate study questions.
These relate to specific concerns of the study.
Study questions generally should be limited to five
or fewer.
Step 2. Prepare a short interview guide.
Key informant interviews do not use rigid questionnaires, which inhibit free discussion. However,
interviewers must have an idea of what questions
to ask. The guide should list major topics and
issues to be covered under each study question.
Because the purpose is to explore a few issues in
depth, guides are usually limited to 12 items.
Different guides may be necessary for interviewing different groups of informants.
Sequence questions. Start with factual questions.
Questions requiring opinions and judgments
should follow. In general, begin with the present
and move to questions about the past or future.
Phrase questions carefully to elicit detailed information. Avoid questions that can be answered by a
simple yes or no. For example, questions such as
“Please tell me about the vaccination campaign?”
are better than “Do you know about the vaccination campaign?”
Use probing techniques. Encourage informants to
detail the basis for their conclusions and recommendations. For example, an informant’s comment, such as “The water program has really
changed things around here,” can be probed for
more details, such as “What changes have you
noticed?” “Who seems to have benefitted most?”
“Can you give me some specific examples?”
3
Maintain a neutral attitude. Interviewers should be
sympathetic listeners and avoid giving the impression of having strong views on the subject under
discussion. Neutrality is essential because some
informants, trying to be polite, will say what they
think the interviewer wants to hear.
Minimize translation difficulties. Sometimes it is
necessary to use a translator, which can change the
dynamics and add difficulties. For example,
differences in status between the translator and
informant may inhibit the conversation. Often
information is lost during translation. Difficulties
can be minimized by using translators who are not
known to the informants, briefing translators on
the purposes of the study to reduce misunderstandings, and having translators repeat the informant’s
comments verbatim.
Step 5. Take adequate notes.
Interviewers should take notes and develop them
in detail immediately after each interview to
ensure accuracy. Use a set of common subheadings
for interview texts, selected with an eye to the
major issues being explored. Common subheadings ease data analysis.
Step 6. Analyze interview data.
Interview summary sheets. At the end of each
interview, prepare a 1-2 page interview summary
sheet reducing information into manageable
themes, issues, and recommendations. Each
summary should provide information about the
key informant’s position, reason for inclusion in
the list of informants, main points made, implications of these observations, and any insights or
ideas the interviewer had during the interview.
Descriptive codes. Coding involves a systematic
recording of data. While numeric codes are not
appropriate, descriptive codes can help organize
responses. These codes may cover key themes,
concepts, questions, or ideas, such as
sustainability, impact on income, and participation
of women. A usual practice is to note the codes or
categories on the left-hand margins of the interview text. Then a summary lists the page numbers
where each item (code) appears. For example,
women’s participation might be given the code
“wom–par,” and the summary sheet might indicate
it is discussed on pages 7, 13, 21, 46, and 67 of the
interview text.
Categories and subcategories for coding (based on
key study questions, hypotheses, or conceptual
frameworks) can be developed before interviews
begin, or after the interviews are completed.
Precoding saves time, but the categories may not
be appropriate. Postcoding helps ensure empirically relevant categories, but is time consuming. A
compromise is to begin developing coding categories after 8 to 10 interviews, as it becomes apparent which categories are relevant.
Storage and retrieval. The next step is to develop a
simple storage and retrieval system. Access to a
computer program that sorts text is very helpful.
Relevant parts of interview text can then be organized according to the codes. The same effect can
be accomplished without computers by preparing
folders for each category, cutting relevant comments from the interview and pasting them onto
index cards according to the coding scheme, then
filing them in the appropriate folder. Each index
card should have an identification mark so the
comment can be attributed to its source.
Presentation of data. Visual displays such as
tables, boxes, and figures can condense information, present it in a clear format, and highlight
underlying relationships and trends. This helps
communicate findings to decision-makers more
clearly, quickly, and easily. Three examples below
and on page 4 illustrate how data from key informant interviews might be displayed.
Table 1. Problems Encountered in
Obtaining Credit
Male Farmers
1. Collateral
requirements
2. Burdensome
paperwork
Female Farmers
1. Collateral
requirements
2. Burdensome
paperwork
3. Long delays in 3. Long delays in
getting loans
getting loans
4. Land registered under
male's name
5. Difficulty getting to
bank location
4
Table 2. Impacts on Income of a
Microenterprise Activity
“In a survey I did of the participants last year, I
found that a majority felt their living conditions have improved.”
—university professor
Assess reliability of key informants. Assess informants’ knowledgeability, credibility, impartiality,
willingness to respond, and presence of outsiders
who may have inhibited their responses. Greater
weight can be given to information provided by
more reliable informants.
“I have doubled my crop and profits this year
as a result of the loan I got.”
—participant
Check interviewer or investigator bias. One’s own
biases as an investigator should be examined,
including tendencies to concentrate on information
that confirms preconceived notions and hypotheses, seek consistency too early and overlook
evidence inconsistent with earlier findings, and be
partial to the opinions of elite key informants.
“I believe that women have not benefitted as
much as men because it is more difficult for us
to get loans.”
—female participant
Check for negative evidence. Make a conscious
effort to look for evidence that questions preliminary findings. This brings out issues that may have
been overlooked.
Table 3. Recommendations for
Improving Training
Recommendation
Number of
Informants
Develop need-based training
courses
39
Develop more objective selection
procedures
20
Plan job placement after training
11
Get feedback from informants. Ask the key informants for feedback on major findings. A summary
report of the findings might be shared with them,
along with a request for written comments. Often a
more practical approach is to invite them to a
meeting where key findings are presented and ask
for their feedback.
These tips are drawn from Conducting Key Informant Interviews in Developing Countries, by
Krishna Kumar (AID Program Design and Evaluation Methodology Report No. 13. December 1986.
PN-AAX-226).
Step 7. Check for reliability and validity.
Key informant interviews are susceptible to error,
bias, and misinterpretation, which can lead to
flawed findings and recommendations.
Check representativeness of key informants. Take
a second look at the key informant list to ensure no
significant groups were overlooked.
U.S. Agency for International Development
For further information on this topic, contact Annette
Binnendijk, CDIE Senior Evaluation Advisor, via
phone (703) 875-4235), fax (703) 875-4866), or e-mail.
Copies of TIPS can be ordered from the Development
Information Services Clearinghouse by calling (703)
351-4006 or by faxing (703) 351-4039. Please refer to
the PN number. To order via the Internet, address a
request to docorder@disc.mhs.compuserve.com
Washington, D.C. 20523
2
ND
NUMBER 3
EDITION, 2010
TIPS
PREPARING AN EVALUATION STATEMENT OF WORK
ABOUT TIPS
performance management and evaluation. This publication is a supplemental reference to the
Automated Directive System (ADS) Chapter 203.
PARTICIPATION IS KEY
Use a participatory process to ensure
resulting information will be relevant
and useful. Include a range of staff
and partners that have an interest in
the evaluation to:
 Participate in planning meetings
and review the SOW;
 Elicit input on potential evaluation
questions; and
 Prioritize and narrow the list of
questions as a group.
WHAT IS AN
EVALUATION
STATEMENT OF
WORK (SOW)?
The statement of work (SOW) is
viewed as the single most critical
document in the development of
a good evaluation. The SOW
states (1) the purpose of an
evaluation, (2) the questions that
must be answered, (3) the
expected quality of the evaluation
results, (4) the expertise needed
to do the job and (5) the time
frame and budget available to
support the task.
WHY IS THE SOW IMPORTANT?
The SOW is important because it
is a basic road map of all the
elements of a well-crafted
evaluation. It is the substance of
a
contract
with
external
evaluators, as well as the
framework for guiding an internal
evaluation team. It contains the
information that anyone who
implements the evaluation needs
to know about the purpose of the
1
evaluation, the background and
history of the program being
evaluated,
and
the
issues/questions that must be
addressed. Writing a SOW is
about managing the first phase of
the evaluation process. Ideally,
the writer of the SOW will also
exercise management oversight
of the evaluation process.
PREPARATION – KEY
ISSUES
BALANCING FOUR
DIMENSIONS
A well drafted SOW is a critical
first step in ensuring the
credibility and utility of the final
evaluation report.
Four key
dimensions of the SOW are
interrelated and should be
balanced against one another
(see Figure 1):
 The number and complexity of
the evaluation questions that
need to be addressed;
 Adequacy of the time allotted
to obtain the answers;
 Availability of funding (budget)
to support the level of
evaluation design and rigor
required; and
 Availability
of the expertise
needed to complete the job.
The development of the SOW is
an iterative process in which the
writer has to revisit, and
sometimes adjust, each of these
dimensions.
Finding
the
appropriate balance is the main
challenge faced in developing any
SOW.
essential
that
evaluation
planning form an integral part
of the initial program or project
design. This includes factoring
in baseline data collection,
possible comparison or „control‟
site
selection,
and
the
preliminary design of data
collection
protocols
and
instruments. Decisions about
evaluation design must be
reflected in implementation
planning and in the budget.
 There
will always be unanticipated
problems
and
opportunities
that
emerge
during an evaluation.
It is
helpful to build-in ways to
accommodate
necessary
changes.
 The writer of the SOW is, in
essence, the architect of the
evaluation. It is important to
commit adequate time and
energy to the task.
 Adequate time is required to
ADVANCE PLANNING
It is a truism that good planning
is a necessary – but not the only –
condition for success in any
enterprise. The SOW preparation
process is itself an exercise in
careful and thorough planning.
The writer must consider several
principles when beginning the
process.
gather information and to build
productive relationships with
stakeholders (such as program
sponsors,
participants,
or
partners) as well as the
evaluation team, once selected.
 The sooner that information can
be made available to the
evaluation team, the more
efficient they can be in
providing credible answers to
the
important
questions
outlined in the SOW.
 The quality of the evaluation is
dependent on providing quality
guidance in the SOW.
 As USAID and other donors
place more emphasis on
rigorous impact evaluation, it is
WHO SHOULD BE INVOLVED?
Participation in all or some part of
the evaluation is an important
decision for the development of
the SOW. USAID and evaluation
experts strongly recommend that
evaluations maximize stakeholder
participation, especially in the
initial
planning
process.
Stakeholders may encompass a
wide array of persons and
institutions,
including
policy
makers,
program
managers,
implementing
partners,
host
country
organizations,
and
beneficiaries.
In some cases,
stakeholders
may
also
be
involved
throughout
the
evaluation
and
with
the
dissemination of results. The
benefits
of
stakeholder
participation
include
the
following:
 Learning
across a broader
group of decision-makers, thus
increasing the likelihood that
the evaluation findings will be
used to improve development
effectiveness;
 Acceptance of the purpose and
process of evaluation by those
concerned;
 A more inclusive and better
focused list of questions to be
answered;
 Increased
acceptance
and
ownership of the process,
findings and conclusions; and
 Increased possibility that the
evaluation will be used by
decision makers and other
stakeholders.
USAID operates in an increasingly
complex implementation world
2
FIGURE 2. ELEMENTS OF A
GOOD EVALUATION SOW
1. Describe the activity, program, or
process to be evaluated
2. Provide a brief background on the
development hypothesis and its
implementation
3. State the purpose and use of the
evaluation
4. Clarify the evaluation questions
5. Identify the evaluation method(s)
6. Identify existing performance
information sources, with special
attention to monitoring data
7. Specify the deliverables(s) and the
timeline
8. Identify the composition of the
evaluation team (one team
member should be an evaluation
specialist) and participation of
customers and partners
9. Address schedule and logistics
10. Clarify requirements for reporting
and dissemination
11. Include a budget
with many players, including
other USG agencies such as the
Departments of State, Defense,
Justice and others. If the activity
engages other players, it is
important to include them in the
process.
Within USAID, there are useful
synergies that can emerge when
the SOW development process is
inclusive. For example, a SOW
that focuses on civil society
advocacy might benefit from
input by those who are experts in
rule of law.
Participation by host government
and local organizational leaders
and beneficiaries is less common
among
USAID
supported
evaluations. It requires sensitivity
and
careful
management;
however,
the
benefits
to
development practitioners can be
substantial.
Participation of USAID managers
in evaluations is an increasingly
common practice and produces
many benefits. To ensure against
bias or conflict of interest, the
USAID manager‟s role can be
limited to participating in the fact
finding phase and contributing to
the analysis. However, the final
responsibility
for
analysis,
conclusions
and
recommendations will rest with
the independent members and
team leader.
THE ELEMENTS OF A
GOOD EVALUATION
SOW
1. DESCRIBE THE ACTIVITY,
PROGRAM, OR PROCESS TO BE
EVALUATED
Be as specific and complete as
possible in describing what is to
be evaluated.
The more
information provided at the
outset, the more time the
evaluation team will have to
develop the data needed to
answer the SOW questions.
If the USAID manager does not
have the time and resources to
bring together all the relevant
information needed to inform the
evaluation in advance, the SOW
might require the evaluation
team to submit a document
review as a first deliverable. This
will, of course, add to the amount
of time and budget needed in the
evaluation contract.
3
2. PROVIDE A BRIEF
BACKGROUND
Give a brief description of the
context, history and current status
of the activities or programs,
names of implementing agencies
and organizations involved, and
other information to help the
evaluation
team
understand
background and context.
In
addition, this section should state
the development hypothesis(es)
and clearly describe the program
(or project) theory that underlies
the program‟s design.
USAID
activities,
programs
and
strategies, as well as most
policies, are based on a set of “ifthen” propositions that predict
how a set of interventions will
produce intended results.
A
development
hypothesis
is
generally represented in a results
framework (or sometimes a
logical framework at the project
level) and identifies the causal
relationships
among
various
objectives sought by the program
(see TIPS 13: Building a Results
Framework). That is, if one or
more objectives are achieved,
then the next higher order
objective will be achieved.
Whether
the
development
hypothesis is the correct one, or
whether it remains valid at the
time of the evaluation, is an
important question for most
evaluation SOWs to consider.
3. STATE THE PURPOSE AND
USE OF THE EVALUATION
Why is an evaluation needed?
The clearer the purpose, the more
likely it is that the evaluation will
produce credible and useful
findings,
conclusions
and
recommendations.
In defining
the purpose, several questions
should be considered.
 Who wants the information?
Will higher level decision
makers be part of the intended
audience?
 What do they want to know?
 For
what purpose will
information be used?
the
 When will it be needed?
 How accurate must it be?
ADS 203.3.6.1 identifies a number
of triggers that may inform the
purpose and use of an evaluation,
as follows:
 A key management decision is
required for which there is
inadequate information;
 Performance
information
indicates an unexpected result
(positive or negative) that
should be explained (such as
gender differential results);
 Customer,
partner, or other
informed feedback suggests
that there are implementation
problems, unmet needs, or
unintended consequences or
impacts;
 Issues of impact, sustainability,
cost-effectiveness, or relevance
arise;
 The validity of the development
hypotheses
or
critical
assumptions is questioned, for
example, due to unanticipated
changes in the host country
environment; and
 Periodic portfolio reviews have
identified key questions that
need to be answered or require
consensus.
4. CLARIFY THE EVALUATION
QUESTIONS
The core element of an
evaluation SOW is the list of
questions
posed
for
the
evaluation.
One of the most
common
problems
with
evaluation SOWs is that they
contain a long list of poorly
defined or “difficult to answer”
questions given the time, budget
and resources provided. While a
participatory process ensures
wide ranging input into the initial
list of questions, it is equally
important to reduce this list to a
manageable number of key
questions. Keeping in mind the
relationship between budget,
time, and expertise needed, every
potential question should be
thoughtfully examined by asking
a number of questions.
 Is this question of essential
importance to the purpose and
the users of the evaluation?
 Is this question clear, precise
and „researchable‟?
 What level of reliability and
validity is expected in answering
the question?
 Does determining an answer to
the question require a certain
kind
of
experience
and
expertise?
 Are we prepared to provide the
management
commitment,
time and budget to secure a
credible
answer
to
this
question?
4
If these questions can be
answered yes, then the team
probably has a good list of
questions that will inform the
evaluation team and drive the
evaluation process to a successful
result.
5. IDENTIFY EVALUATION
METHODS
The SOW manager has to decide
whether the evaluation design
and methodology should be
specified in the SOW. 1
This
depends on whether the writer
has expertise, or has internal
access to evaluation research
knowledge and experience. If so,
and the writer is confident of the
„on the ground‟ conditions that
will allow for different evaluation
designs, then it is appropriate to
include specific requirements in
the SOW.
If the USAID SOW manager does
not have the kind of evaluation
experience needed, especially for
more formal and rigorous
evaluations, it is good practice to:
1) require that the team (or
bidders, if it is contracted out)
include a description of (or
approach for developing) the
proposed research design and
methodology, or 2) require a
detailed design and evaluation
plan to be submitted as a first
deliverable. In this way, the SOW
manager benefits from external
evaluation expertise. In either
case,
the
design
and
methodology should not be
finalized until the team has an
opportunity to gather detailed
1
See USAID ADS 203.3.6.4 on
Evaluation Methodologies;
information and discuss
issues with USAID.
final
The selection of the design and
data collection methods must be
a function of the type of
evaluation and the level of
statistical and quantitative data
confidence needed. If the project
is selected for a rigorous impact
evaluation, then the design and
methods used will be more
sophisticated and technically
complex. If external assistance is
necessary, the evaluation SOW
will be issued as part of the initial
RFP/RFA (Request for Proposal or
Request
for
Application)
solicitation process. All methods
and evaluation designs should be
as
rigorous
as
reasonably
possible. In some cases, a rapid
appraisal
is
sufficient
and
appropriate (see TIPS 5: Using
Rapid Appraisal Methods). At the
other extreme, planning for a
sophisticated
and
complex
evaluation
process
requires
greater up-front investment in
baselines, outcome monitoring
processes,
and
carefully
constructed
experimental
or
quasi-experimental designs.
6. IDENTIFY EXISTING
PERFORMANCE INFORMATION
Identify
the
existence
and
availability
of
relevant
performance information sources,
such as performance monitoring
systems
and/or
previous
evaluation reports. Including a
summary of the types of data
available, the timeframe, and an
indication of their quality and
reliability will help the evaluation
team to build on what is already
available.
7. SPECIFY DELIVERABLES
AND TIMELINE
The SOW must specify the
products, the time frame, and the
content of each deliverable that is
required
to
complete
the
evaluation contract. Some SOWs
simply require delivery of a draft
evaluation report by a certain
date. In other cases, a contract
may require several deliverables,
such as a detailed evaluation
design, a work plan, a document
review, and the evaluation report.
The most important deliverable is
the final evaluation report. TIPS
17: Constructing an Evaluation
Report provides a suggested
outline of an evaluation report
that may be adapted and
incorporated directly into this
section.
The evaluation report should
differentiate between findings,
conclusions,
and
recommendations, as outlined in
Figure 3. As evaluators move
beyond
the
facts,
greater
interpretation is required.
By
ensuring that the final report is
organized
in
this
manner,
decision makers can clearly
understand the facts on which the
evaluation is based. In addition,
it
facilitates
greater
understanding of where there
might
be
disagreements
concerning the interpretation of
those facts.
While individuals
may
disagree
on
recommendations, they should
not disagree on the basic facts.
5
Another consideration is whether
a section on “lessons learned”
should be included in the final
report. A good evaluation will
produce knowledge about best
practices, point out what works,
what does not, and contribute to
the more general fund of tested
experience on which other
program
designers
and
implementers can draw.
Because unforeseen obstacles
may emerge, it is helpful to be as
realistic as possible about what
can be accomplished within a
given time frame. Also, include
some wording that allows USAID
and the evaluation team to adjust
schedules in consultation with the
USAID manager should this be
necessary.
8. DISCUSS THE COMPOSITION
OF THE EVALUATION TEAM
USAID evaluation guidance for
team
selection
strongly
recommends that at least one
team member have credentials
and experience in evaluation
design and methods. The team
leader must have strong team
management skills, and sufficient
experience
with
evaluation
standards and practices to ensure
a credible product.
The
appropriate team leader is a
person with whom the SOW
manager can develop a working
partnership as the team moves
through the evaluation research
design and planning process.
He/she must also be a person
who can deal effectively with
senior U.S. and host country
officials and other leaders.
Experience with USAID is often an
important factor, particularly for
management
focused
evaluations, and in formative
evaluations designed to establish
the basis for a future USAID
program or the redesign of an
existing program.
If the
evaluation entails a high level of
complexity, survey research and
other sophisticated methods, it
may be useful to add a data
collection and analysis expert to
the team.
Generally, evaluation skills will be
supplemented with additional
subject matter experts. As the
level of research competence
increases in many countries
where USAID has programs, it
makes good sense to include
local
collaborators,
whether
survey
research
firms
or
independents, to be full members
of the evaluation team.
9. ADDRESS SCHEDULING,
LOGISTICS AND OTHER
SUPPORT
Good scheduling and effective
local support contributes greatly
to the efficiency of the evaluation
team. This section defines the
time frame and the support
structure needed to answer the
evaluation questions at the
required level of validity. For
evaluations involving complex
designs and sophisticated survey
research data collection methods,
the schedule must allow enough
time, for example, to develop
sample frames, prepare and
pretest
survey
instruments,
training interviewers, and analyze
data. New data collection and
analysis
technologies
can
accelerate this process, but need
to be provided for in the budget.
In some cases, an advance trip to
the field by the team leader
and/or methodology expert may
be justified where extensive
pretesting
and
revision
of
instruments is required or when
preparing for an evaluation in
difficult or complex operational
environments.
Adequate
logistical
and
administrative support is also
essential. USAID often works in
countries with poor infrastructure,
frequently in conflict/post-conflict
environments where security is an
issue. If the SOW requires the
team to make site visits to distant
or
difficult
locations,
such
planning must be incorporated
into the SOW.
6
Particularly overseas, teams often
rely on local sources for
administrative support, including
scheduling of appointments,
finding
translators
and
interpreters,
and
arranging
transportation. In many countries
where foreign assistance experts
have been active, local consulting
firms have developed this kind of
expertise. Good interpreters are
in high demand, and are essential
to any evaluation team‟s success,
especially when using qualitative
data collection methods.
10. CLARIFY REQUIREMENTS
FOR REPORTING AND
DISSEMINATION
Most evaluations involve several
phases of work, especially for
more complex designs.
The
SOW can set up the relationship
between the evaluation team, the
USAID manager and other
stakeholders. If a working group
was established to help define
the SOW questions, continue to
use the group as a forum for
interim reports and briefings
provided by the evaluation team.
The SOW should specify the
timing and details for each
briefing session.
Examples of
what might be specified include:
 Due dates for draft and final
reports;
 Dates for oral briefings (such as
a mid-term and final briefing);
 Number of copies needed;
 Language requirements, where
applicable;
 Formats and page limits;
 Requirements for datasets, if
primary
data
collected;
has
been
 A requirement to submit all
evaluations to the Development
Experience Clearing house for
archiving
this
is
the
responsibility of the evaluation
contractor; and
 Other
needs
communicating, marketing
disseminating results that
the
responsibility
of
evaluation team.
for
and
are
the
The SOW should specify when
working drafts are to be
submitted for review, the time
frame allowed for USAID review
and comment, and the time
frame to revise and submit the
final report.
11. INCLUDE A BUDGET
With the budget section, the
SOW comes full circle. As stated,
budget considerations have to be
part of the decision making
process from the beginning.
The budget is a product of the
questions
asked,
human
resources needed, logistical and
administrative support required,
and the time needed to produce
a high quality, rigorous and
useful evaluation report in the
most efficient and timely manner.
It is essential for contractors to
understand the quality, validity
and rigor required so they can
develop a responsive budget that
will meet the standards set forth
in the SOW.
For more information:
TIPS publications are available online at [insert website].
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including USAID‟s
Office of Management Policy, Budget and Performance (MPBP). This publication was written by Richard
Blue, Ph.D. of Management Systems International.
Comments regarding this publication can be directed to:
Gerald Britan, Ph.D.
Tel: (202) 712-1158
gbritan@usaid.gov
Contracted under RAN-M-00-04-00049-A-FY0S-84
Integrated Managing for Results II
7
1996, Number 4
TIPS
USAID Center for Development Information and Evaluation
USING DIRECT OBSERVATION TECHNIQUES
What is Direct Observation?
Most evaluation teams conduct some fieldwork, observing what's actually going on at
assistance activity sites. Often, this is done informally, without much thought to the
quality of data collection. Direct observation techniques allow for a more systematic,
structured process, using well-designed observation record forms.
Advantages and Limitations
USAID's
reengineering guidance encourages
the use of rapid, low
cost methods for
collecting information on the performance of our development activities.
The main advantage of direct observation is that an event, institution, facility, or
process can be studied in its natural setting, thereby providing a richer understanding
of the subject.
For example, an evaluation team that visits microenterprises is likely to better
understand their nature, problems, and successes after directly observing their
products, technologies, employees, and processes, than by relying solely on
documents or key informant interviews. Another advantage is that it may reveal
conditions, problems, or patterns many informants may be unaware of or unable to
describe adequately.
On the negative side, direct observation is susceptible to observer bias. The very act
of observation also can affect the behavior being studied.
When Is Direct Observation Useful?
Direct observation,
the subject of this
Tips, is one such
method.
Direct observation may be useful:
When performance monitoring data indicate results are not being
accomplished as planned, and when implementation problems are suspected,
but not understood. Direct observation can help identify whether the process
is poorly implemented or required inputs are absent.
When details of an activity's process need to be assessed, such as whether
tasks are being implementing according to standards required for
effectiveness.
When an inventory of physical facilities and inputs is needed and not
available from existing sources.
PN-ABY-208
2
When interview methods are unlikely to elicit
needed information accurately or reliably, either
because the respondents don't know or may be
reluctant to say.
When preparing direct observation forms, consider the
following:
1. Identify in advance the possible response categories for
each item, so that the observer can answer with a simple
Steps in Using Direct Observation
The quality of direct observation can be improved by
following these steps.
Step 1. Determine the focus
Because of typical time and resource constraints, direct
observation has to be selective, looking at a few activities,
events, or phenomena that are central to the evaluation
questions.
For example, suppose an evaluation team intends to study a
few health clinics providing immunization services for
children. Obviously, the team can assess a variety of
areas—physical facilities and surroundings, immunization
activities of health workers, recordkeeping and managerial
services, and community interactions. The team should
narrow its focus to one or two areas likely to generate the
most useful information and insights.
Next, break down each activity, event, or phenomena into
subcomponents. For example, if the team decides to look at
immunization activities of health workers, prepare a list of
the tasks to observe, such as preparation of vaccine,
consultation with mothers, and vaccine administration.
Each task may be further divided into subtasks; for
example, administering vaccine likely includes preparing
the recommended doses, using the correct administration
technique, using sterile syringes, and protecting vaccine
from heat and light during use.
If the team also wants to assess physical facilities and
surroundings, it will prepare an inventory of items to be
observed.
OBSERVATION OF GROWTH
MONITORING SESSION
Name of the Observer
Date
Time
Place
Was the scale set to 0 at the beginning of the growth
session?
Yes______ No ______
How was age determined?
By asking______
From growth chart_______
Other_______
When the child was weighed, was it stripped to
practical limit?
Yes______ No______
Was the weight read correctly?
Yes______No______
Process by which weight and age transferred to record
Health Worker wrote it_____
Someone else wrote it______
Other______
Did Health Worker interpret results for the mother?
Yes_______No_______
Step 2. Develop direct observation forms
The observation record form should list the items to be
observed and provide spaces to record observations. These
forms are similar to survey questionnaires, but
investigators record their own observations, not
respondents' answers.
Observation record forms help standardize the observation
process and ensure that all important items are covered.
They also facilitate better aggregation of data gathered
from various sites or by various investigators. An excerpt
from a direct observation form used in a study of primary
health care in the Philippines provides an illustration below.
yes or no, or by checking the appropriate answer. Closed
response categories help minimize observer variation, and
therefore improve the quality of data.
2. Limit the number of items in a form. Forms should
normally not exceed 40–50 items. If nessary, it is better to
use two or more smaller forms than a single large one that
runs several pages.
3
3. Provide adequate space to record additional observations
for which response categories were not determined.
4. Use of computer software designed to create forms can
be very helpful. It facilitates a neat, unconfusing form that
can be easily completed.
People and organizations follow daily routines associated
with set times. For example, credit institutions may accept
loan applications in the morning; farmers in tropical
climates may go to their fields early in the morning and
return home by noon. Observation periods should reflect
work rhythms.
Step 3. Select the sites
Step 5. Conduct the field observation
Once the forms are ready, the next step is to decide where
the observations will be carried out and whether it will be
based on one or more sites.
Establish rapport. Before embarking on direct observation,
a certain level of rapport should be established with the
people, community, or organization to be studied. The
presence of outside observers, especially if officials or
experts, may generate some anxiety among those being
observed. Often informal, friendly conversations can
reduce anxiety levels.
A single site observation may be justified if a site can be
treated as a typical case or if it is unique. Consider a
situation in which all five agricultural extension centers
established by an assistance activity have not been
performing well. Here, observation at a single site may be
justified as a typical case. A single site observation may
also be justified when the case is unique; for example, if
only one of five centers had been having major problems,
and the purpose of the evaluation is trying to discover why.
However, single site observations should be avoided
generally, because cases the team assumes to be typical or
unique may not be. As a rule, several sites are necessary to
obtain a reasonable understanding of a situation.
In most cases, teams select sites based on experts' advice.
The investigator develops criteria for selecting sites, then
relies on the judgment of knowledgeable people. For
example, if a team evaluating a family planning project
decides to observe three clinics—one highly successful,
one moderately successful, and one struggling clinic—it
may request USAID staff, local experts, or other
informants to suggest a few clinics for each category. The
team will then choose three after examining their
recommendations. Using more than one expert reduces
individual bias in selection.
Alternatively, sites can be selected based on data from
performance monitoring. For example, activity sites
(clinics, schools, credit institutions) can be ranked from
best to worst based on performance measures, and then a
sample drawn from them.
Step 4. Decide on the best timing
Timing is critical in direct observation, especially when
events are to be observed as they occur. Wrong timing can
distort findings. For example, rural credit
Also, let them know the purpose of the observation is not to
report on individuals' performance, but to find out what
kind of problems in general are being encountered.
Allow sufficient time for direct observation. Brief visits can
be deceptive partly because people tend to behave
differently in the presence of observers. It is not
uncommon, for example, for health workers to become
more caring or for extension workers to be more
persuasive when being watched. However, if observers
stay for relatively longer periods, people become less selfconscious and gradually start behaving naturally. It is
essential to stay at least two or three days on a site to
gather valid, reliable data.
Use a team approach. If possible, two observers should
observe together. A team can develop more
comprehensive, higher quality data, and avoid individual
bias.
Train observers. If many sites are to be observed,
nonexperts can be trained as observers, especially if
observation forms are clear, straightforward, and mostly
closed-ended.
Step 6. Complete forms
Take notes as inconspicuously as possible. The best time
for recording is during observation. However, this is not
always feasible because it may make some people selfconscious or disturb the situation. In these cases, recording
should take place as soon as possible after observation.
Step 7. Analyze the data
organizations receive most loan applications during the
planting season, when farmers wish to purchase
agricultural inputs. If credit institutions are observed during
the nonplanting season, an inaccurate picture of loan
processing may result.
Data from close-ended questions from the observation
form can be analyzed using basic procedures such as
frequency counts and cross-tabulations. Statistical software
packages such as SAS or SPSS facilitate such statistical
analysis and data display.
4
Analysis of any open-ended interview questions can also
provide extra richness of understanding and insights. Here,
use of database management software with text storage
capabilities, such as dBase, can be useful.
Step 8. Check for reliability and validity.
Direct observation techniques are susceptible to error and
bias that can affect reliability and validity. These can be
minimized by following some of the procedures suggested,
such as checking the representativeness of the sample of
Direct Observation of Primary
Health Care Services in the Philippines
An example of structured direct observation was an
effort to identify deficiencies in the primary health
care system in the Philippines. It was part of a
larger, multicountry research project, the Primary
Health Care Operations Research Project (PRICOR).
The evaluators prepared direct observation forms
covering the activities, tasks, and subtasks health
workers must carry out in health clinics to
accomplish clinical objectives. These forms were
closed-ended and in most cases observations could
simply be checked to save time. The team looked at
18 health units from a "typical" province, including
samples of units that were high, medium and low
performers in terms of key child survival outcome
indicators.
The evaluation team identified and quantified many
problems that required immediate government
attention. For example, in 40 percent of the cases
where followup treatment was required at home,
health workers failed to tell mothers the timing and
amount of medication required. In 90 percent of
cases, health workers failed to explain to mothers the
results of child weighing and growth plotting, thus
missing the opportunity to involve mothers in the
nutritional care of their child. Moreover, numerous
errors were made in weighing and plotting.
This case illustrates that use of closed-ended
observation instruments promotes the reliability and
consistency of data. The findings are thus more
credible and likely to influence program managers to
make needed improvements.
sites selected; using closed-ended, unambiguous response
categories on the observation forms, recording observations
promptly, and using teams of observers at each site.
Information in this Tips is based on "Rapid Data Collection
Methods for Field Assessments" by Krishna Kumar, in
Team Planning Notebook for Field-Based Program
Assessments (USAID PPC/CDIE, 1991).
For more on direct observation techniques applied to the
Philippines health care system, see Stewart N. Blumenfeld,
Manuel Roxas, and Maricor de los Santos, "Systematic
Observation in the Analysis of Primary Health Care
Services," in Rapid Appraisal Methods, edited by Krishna
Kumar (The World Bank:1993)
CDIE's Tips series provide advice and suggestions to
USAID managers on how to plan and conduct
performance monitoring and evaluation activities.
They are supplemental references to the reengineering
automated directives system (ADS), chapter 203. For
further information, contact Annette Binnendijk, CDIE
Senior Evaluation Advisor, phone (703) 875–4235, fax
(703) 875–4866, or e-mail. Tips can be ordered from
the Development Information Services Clearinghouse
by calling (703) 351-4006 or by faxing (703) 351–4039.
Please refer to the PN number. To order via Internet,
address requests to
docorder@disc.mhs.compuserve.com
NUMBER 5
2ND EDITION, 2010
TIPS
USING RAPID APPRAISAL METHODS
ABOUT TIPS
These TIPS provide practical advice and suggestions to USAID managers on issues related to performance
monitoring and evaluation. This publication is a supplemental reference to the Automated Directive
System (ADS) Chapter 203.
WHAT IS RAPID
APPRAISAL?
Rapid Appraisal (RA) is an approach
that draws on multiple evaluation
methods and techniques to quickly,
yet systematically, collect data when
time in the field is limited. RA
practices are also useful when there
are budget constraints or limited
availability of reliable secondary
data. For example, time and budget
limitations may preclude the option
of using representative sample
surveys.
BENEFITS – WHEN TO USE
RAPID APPRAISAL
METHODS
Rapid appraisals are quick and can
be done at relatively low cost.
Rapid appraisal methods can help
gather, analyze, and report relevant
information for decision-makers
within days or weeks. This is not
possible with sample surveys. RAs
can be used in the following cases:
• for formative evaluations, to make
mid-course corrections in project
design or implementation when
customer or partner feedback
indicates a problem (See ADS
203.3.6.1);
• when a key management decision
is required and there is inadequate
information;
• for performance monitoring, when
data are collected and the
techniques are repeated over time
for measurement purposes;
• to better understand the issues
behind performance monitoring
data; and
• for project pre-design assessment.
LIMITATIONS – WHEN
RAPID APPRAISALS ARE
NOT APPROPRIATE
Findings from rapid appraisals may
have limited reliability and validity,
and cannot be generalized to the
larger population. Accordingly,
1
rapid appraisal should not be the
sole basis for summative or impact
evaluations. Data can be biased and
inaccurate unless multiple methods
are used to strengthen the validity
of findings and careful preparation is
undertaken prior to beginning field
work.
WHEN ARE RAPID
APPRAISAL
METHODS
APPROPRIATE?
Choosing between rapid appraisal
methods for an assessment or more
time-consuming methods, such as
sample surveys, should depend on
balancing several factors, listed
below.
• Purpose of the study. The
importance and nature of the
decision depending on it.
• Confidence in results. The
accuracy, reliability, and validity of
findings needed for management
decisions.
• Time frame. When a decision
must be made.
• Resource constraints (budget).
• Evaluation questions to be
answered. (see TIPS 3: Preparing
an Evaluation Statement of Work)
USE IN TYPES OF
EVALUATION
Rapid appraisal methods are often
used in formative evaluations.
Findings are strengthened when
evaluators
use
triangulation
(employing more than one data
collection method) as a check on
the validity of findings from any one
method.
Rapid appraisal methods are also
used in the context of summative
evaluations. The data from rapid
appraisal methods and techniques
complement the use of quantitative
methods such as surveys based on
representative
sampling.
For
example, a randomized survey of
small holder farmers may tell you
that farmers have a difficult time
selling their goods at market, but
may not have provide you with the
details of why this is occurring. A
researcher
could
then
use
interviews
with
farmers
to
determine the details necessary to
construct a more complete theory
of why it is difficult for small holder
farmers to sell their goods.
KEY PRINCIPLES
FOR ENSURING
USEFUL RAPID
APPRAISAL DATA
COLLECTION
No set of rules dictates which
methods and techniques should be
used in a given field situation;
however, a number of key principles
can be followed to ensure the
collection of useful data in a rapid
appraisal.
• Preparation is key. As in any
evaluation, the evaluation design
and selection of methods must
begin
with
a
thorough
understanding of the evaluation
questions and the client’s needs
for evaluative information. The
client’s intended uses of data must
guide the evaluation design and
the types of methods that are
used.
• Triangulation increases the validity
of findings. To lessen bias and
strengthen the validity of findings
from rapid appraisal methods and
techniques, it is imperative to use
multiple methods. In this way,
data collected using one method
can be compared to that collected
using other methods, thus giving a
researcher the ability to generate
valid and reliable findings. If, for
example, data collected using Key
Informant Interviews reveal the
same findings as data collected
from Direct Observation and
Focus Group Interviews, there is
less chance that the findings from
the first method were due to
researcher bias or due to the
findings being outliers. Table 1
summarizes
common
rapid
appraisal methods and suggests
how findings from any one
method can be strengthened by
the use of other methods.
COMMON RAPID
APPRAISAL
METHODS
INTERVIEWS
This method involves one-on-one
interviews with individuals or key
informants selected for their
knowledge or diverse views.
Interviews are qualitative, in-depth
and semi-structured.
Interview
guides are usually used and
2
EVALUATION METHODS
COMMONLY USED IN RAPID
APPRAISAL
• Interviews
• Community Discussions
• Exit Polling
• Transect Walks (see p. 3)
• Focus Groups
• Minisurveys
• Community Mapping
• Secondary Data Collection
• Group Discussions
• Customer Service Surveys
• Direct Observation
questions may be further framed
during the interview, using subtle
probing techniques.
Individual
interviews may be used to gain
information on a general topic but
cannot provide the in-depth inside
knowledge on evaluation topics that
key informants may provide.
MINISURVEYS
A minisurvey consists of interviews
with between five to fifty individuals,
usually
selected
using
nonprobability sampling (sampling in
which respondents are chosen based
on their understanding of issues
related to a purpose or specific
questions, usually used when sample
sizes are small and time or access to
areas is limited).
Structured
questionnaires are used with a
limited number of close-ended
questions.
Minisurveys generate
quantitative data that can often be
collected and analyzed quickly.
FOCUS GROUPS
The focus group is a gathering of a
homogeneous body of five to twelve
participants to discuss issues and
experiences among themselves.
These are used to test an idea or to
get a reaction on specific topics. A
moderator introduces the topic,
stimulates
and
focuses
the
discussion, and prevents domination
of discussion by a few, while another
documents
the
evaluator
conversation.
THE ROLE OF TECHNOLOGY
IN RAPID APPRAISAL
Certain equipment and technologies
can aid the rapid collection of data
and help to decrease the incidence of
errors. These include, for example,
hand held computers or personal
digital assistants (PDAs) for data
input,
cellular
phones,
digital
recording devices for interviews,
videotaping and photography, and the
use of geographic information systems
(GIS) data and aerial photographs.
GROUP DISCUSSIONS
This method involves the selection
of approximately five participants
who are knowledgeable about a
given topic and are comfortable
enough with one another to freely
discuss the issue as a group. The
moderator introduces the topic and
keeps the discussion going while
another evaluator records the
discussion. Participants talk among
each other rather than respond
directly to the moderator.
COMMUNITY DISCUSSIONS
This method takes place at a public
meeting that is open to all
community members; it can be
successfully moderated with as
many as 100 or more people. The
primary interaction is between the
participants while the moderator
leads the discussion and asks
questions following a carefully
prepared interview guide.
DIRECT OBSERVATION
Teams of observers record what
they hear and see at a program site
using a detailed observation form.
Observation may be of the physical
surrounding or of ongoing activities,
processes, or interactions.
COLLECTING SECONDARY
DATA
This method involves the on-site
collection of existing secondary
data, such as export sales, loan
information, health service statistics,
etc. These data are an important
augmentation
to
information
collected using qualitative methods
such as interviews, focus groups, and
community discussions.
The
3
evaluator must be able to quickly
determine the validity and reliability
of the data. (see TIPS 12: Indicator
and Data Quality)
TRANSECT WALKS
The transect walk is a participatory
approach in which the evaluator
asks a selected community member
to walk with him or her, for
example, through the center of
town, from one end of a village to
the other, or through a market.
The evaluator asks the individual,
usually a key informant, to point out
and
discuss
important
sites,
neighborhoods, businesses, etc., and
to discuss related issues.
COMMUNITY MAPPING
Community mapping is a technique
that requires the participation of
residents on a program site. It can
be used to help locate natural
resources, routes, service delivery
points, regional markets, trouble
spots, etc., on a map of the area, or
to use residents’ feedback to drive
the development of a map that
includes such information.
COMMON RAPID APPRAISAL METHODS
Table 1
Method
Useful for
Providing
Example
Advantages
Limitations
Further
References
INDIVIDUAL INTERVIEWS
Interviews
− A general overview of
the topic from
someone who has a
broad knowledge and
in-depth experience
and understanding
(key informant) or indepth information on
a very specific topic or
subtopic (individual)
− Suggestions and
recommendations to
improve key aspects
of a program
Minisurveys
− Quantitative data on
narrowly focused
questions, for a
relatively
homogeneous
population, when
representative
sampling is not
possible or required
Key informant:
Interview with
program
implementation
director
− Provides in-depth, − Susceptible to
inside information
interviewer and
on specific issues
selection biases
from the
− Individual
individuals
interviews lack the
perspective and
broader
experience
understanding and
− Flexibility permits
insight that a key
exploring
informant can
unanticipated
provide
topics
Interview with
director of a regional
trade association
Individual:
Interview with an
activity manager within
an overall
− Easy to administer
development program
− Low cost
Interview with a local
entrepreneur trying to
enter export trade
− A customer service
assessment
− Quantitative data
from multiple
respondents
− Rapid exit interviews
after voting
− Low cost
TIPS No. 2,
Conducting Key
Informant Interviews
K. Kumar, Conducting
Key Informant Surveys
in Developing
Countries, 1986
Bamberger, Rugh, and
Mabry, Real World
Evaluation, 2006
UNICEF Website: M&E
Training Modules:
Overview of RAP
Techniques
− Findings are less
generalizable than
those from sample
surveys unless the
universe of the
population is
surveyed
TIPS No. 9,
Conducting Customer
Service Assessments
Mini Surveys in
Developing Countries,
1990
Mabry, RealWorld
Evaluation, 2006 on
purposeful sampling
− Quick data on
attitudes, beliefs,
behaviors of
beneficiaries or
partners
GROUP INTERVIEWS
Focus Groups
− Customer views on
services, products,
benefits
− Information on
implementation
problems
− Suggestions and
recommendations for
improving specific
activities
− Discussion on
− Group discussion
experience related
may reduce
to a specific program
inhibitions,
intervention
allowing free
exchange of ideas
− Effects of a new
business regulation
− Low cost
or proposed price
changes
4
− Discussion may be
dominated by a
few individuals
unless the process
is facilitated/
managed well
TIPS No. 10,
Conducting Focus
Group Interviews
Group Interviews in
Developing Countries,
1987
T. Greenbaum,
Moderating Focus
Groups: A Practical
Guide for Group
Facilitation, 2000
Group
Discussions
Community
Discussions
− Understanding of
issues from different
perspectives and
experiences of
participants from a
specific subpopulation
− Discussion with
young women on
access to prenatal
and infant care
− Small group size
allows full
participation
− Discussion with
entrepreneurs about
export regulations
− Understanding of an
− A Town Hall
issue or topic from a
meeting
wide range of
participants from key
evaluation sites within
a village, town, city, or
city neighborhood
− Findings cannot be Bamberger, Rugh, and
Mabry, RealWorld
generalized to a
Evaluation, 2006
larger population
UNICEF Website: M&E
Training Modules:
Community Meetings
− Allows good
understanding of
specific topics
− Low cost
− Yields a wide
range of opinions
on issues
important to
participants
− Findings cannot be
generalized to
larger population
or to
subpopulations of
concern
Mabry, RealWorld
Evaluation, 2006
− Observer bias
unless two to
three evaluators
observe same
place or activity
TIPS No. 4, Using
Direct Observation
Techniques
− Must be able to
determine
reliability and
validity of data
TIPS No. 12,
Guidelines for
Indicator and Data
Quality
− A great deal of
information can be − Larger groups
obtained at one
difficult to
point of time
moderate
UNICEF Website: M&E
Training Modules:
Community Meetings
ADDITIONAL COMMONLY USED TECHNIQUES
Direct
Observation
− Visual data on physical − Market place to
− Confirms data
infrastructure,
observe goods being
from interviews
supplies, conditions
bought and sold,
− Low cost
who is involved,
− Information about an
sales interactions
agency’s or business’s
delivery systems,
services
WFP Website:
Monitoring & Evaluation
Guidelines: What Is
Direct Observation and
When Should It Be Used?
− Insights into behaviors
or events
Collecting
Secondary
Data
− Validity to findings
gathered from
interviews and group
discussions
− Microenterprise
bank loan info.
− Value and volume of
exports
− Quick, low cost
way of obtaining
important
quantitative data
− Number of people
served by a health
clinic, social service
provider
PARTICIPATORY TECHNIQUES
Transect
Walks
Community
Mapping
− Important visual and
locational information
and a deeper
understanding of
situations and issues
− Walk with key
informant from one
end of a village or
urban neighborhood
to another, through
a market place, etc.
− Info. on locations
important for data
collection that could
be difficult to find
− Insiders viewpoint − Susceptible to
interviewer and
− Quick way to find
selection biases
out location of
places of interest
to the evaluator
− Low cost
− Map of village and
− Important
− Rough locational
surrounding area
locational data
information
with locations of
when there are no
markets, water and
detailed maps of
fuel sources, conflict
the program site
− Quick comprehension
areas, etc.
on spatial location of
services/resources in a
region which can give
insight to access issues
5
Mabry, Real World
Evaluation, 2006
UNICEF Website: M&E
Training Modules:
Overview of RAP
Techniques
Mabry, Real World
Evaluation, 2006
UNICEF Website: M&E
Training Modules:
Overview of RAP
Techniques
References Cited
M. Bamberger, J. Rugh, and L. Mabry, Real World Evaluation. Working Under Budget, Time, Data, and Political
Constraints. Sage Publications, Thousand Oaks, CA, 2006.
T. Greenbaum, Moderating Focus Groups: A Practical Guide for Group Facilitation. Sage Publications, Thousand Oaks,
CA, 2000.
K. Kumar, “Conducting Mini Surveys in Developing Countries,” USAID Program Design and Evaluation Methodology
Report No. 15, 1990 (revised 2006).
K. Kumar, “Conducting Group Interviews in Developing Countries,” USAID Program Design and Evaluation
Methodology Report No. 8, 1987.
K. Kumar, “Conducting Key Informant Interviews in Developing Countries,” USAID Program Design and Evaluation
Methodology Report No. 13, 1989.
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including USAID’s Office of
Management Policy, Budget and Performance (MPBP). This publication was authored by Patricia Vondal, PhD., of
Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
6
NUMBER 6
2ND EDITION, 2010
TIPS
SELECTING PERFORMANCE INDICATORS
ABOUT TIPS
WHAT ARE
PERFORMANCE
INDICATORS?
Performance indicators define a
measure of change for the
results identified in a Results
Framework (RF). When wellchosen, they convey whether
key objectives are achieved in a
meaningful
way
for
performance
management.
While a result (such as an
Assistance Objective or an
Intermediate Result) identifies
what we hope to accomplish,
indicators tell us by what
standard that result will be
measured.
Targets define
whether there will be an
expected increase or decrease,
and by what magnitude.1
Indicators may be quantitative
or
qualitative
in
nature.
Quantitative
indicators
are
numerical: an example is a
person’s height or weight. On
the other hand, qualitative
indicators require subjective
evaluation. Qualitative data are
sometimes
reported
in
numerical form, but those
numbers do not have arithmetic
meaning on their own. Some
examples are a score on an
institutional capacity index or
progress along a milestone
scale.
When developing
quantitative
or
qualitative
indicators, the important point
is that the indicator be
1
For further information, see TIPS 13:
Building a Results Framework and TIPS
8: Baselines and Targets.
1
Selecting an optimal set of indicators
to track progress against key results
lies at the heart of an effective
performance management system.
This TIPS provides guidance on how to
select effective performance
indicators.
constructed in a way that
permits
consistent
measurement over time.
USAID has developed many
performance indicators over the
years. Some examples include
the dollar value of nontraditional
exports,
private
investment as a percentage of
gross
domestic
product,
contraceptive prevalence rates,
child mortality rates, and
progress on a legislative reform
index.
WHY ARE
PERFORMANCE
INDICATORS
IMPORTANT?
FOR WHAT RESULTS
ARE PERFORMANCE
INDICATORS
REQUIRED?
Performance indicators provide
objective evidence that an
intended change is occurring.
Performance indicators lie at
the heart of developing an
effective
performance
management system – they
define the data to be collected
and enable actual results
achieved to be compared with
planned results over time.
Hence,
they
are
an
indispensable management tool
for making evidence-based
decisions
about
program
strategies
and
activities.
Performance indicators can also
be used:
THE PROGRAM LEVEL
USAID’s ADS requires that at
least one indicator be chosen
for each result in the Results
Framework in order to measure
progress (see ADS 203.3.3.1)2.
This includes the Assistance
Objective (the highest-level
objective
in
the
Results
Framework)
as
well
as
supporting Intermediate Results
(IRs)3. These indicators should
be included in the Mission or
Office
Performance
Management Plan (PMP) (see
TIPS 8: Preparing a PMP).





To assist managers in
focusing
on
the
achievement
of
development results.
To
provide
objective
evidence that results are
being achieved.
To orient and motivate staff
and
partners
toward
achieving results.
To communicate USAID
achievements
to
host
country counterparts, other
partners, and customers.
To more effectively report
results achieved to USAID's
stakeholders, including the
U.S. Congress, Office of
Management and Budget,
and citizens.
PROJECT LEVEL
AO teams are required to
collect data regularly for
projects and activities, including
inputs, outputs, and processes,
to ensure they are progressing
as
expected
and
are
contributing to relevant IRs and
AOs. These indicators should
be included in a project-level
monitoring
and
evaluation
2
For further discussion of AOs and IRs
(which are also termed impact and
outcomes respectively in other
systems) refer to TIPS 13: Building a
Results Framework.
3
Note that some results frameworks
incorporate IRs from other partners if
those results are important for USAID
to achieve the AO. This is discussed in
further detail in TIPS 13: Building a
Results Framework. If these IRs are
included, then it is recommended that
they be monitored, although less
rigorous standards apply.
2
(M&E) plan. The M&E plan
should be integrated in project
management and reporting
systems (e.g., quarterly, semiannual, or annual reports).
TYPES OF
INDICATORS IN
USAID SYSTEMS
Several different types of
indicators are used in USAID
systems.
It is important to
understand the different roles
and
functions
of
these
indicators so that managers can
construct
a
performance
management
system
that
effectively
meets
internal
management
and
Agency
reporting needs.
CUSTOM INDICATORS
Custom
Indicators
are
performance indicators that
reflect progress within each
unique country or program
context. While they are useful
for managers on the ground,
they
often
cannot
be
aggregated across a number of
programs
like
standard
indicators.
Example:
Progress on a
milestone scale reflecting
legal
reform
and
implementation to ensure
credible elections, as follows:
 Draft law is developed in
consultation with nongovernmental
organizations (NGOs) and
political parties.
 Public input is elicited.
PARTICIPATION IS ESSENTIAL
Experience suggests that
participatory approaches are an
essential aspect of developing and
maintaining effective performance
management systems. Collaboration
with development partners
(including host country institutions,
civil society organizations (CSOs),
and implementing partners) as well
as customers has important benefits.
It allows you to draw on the
experience of others, obtains buy-in
to achieving results and meeting
targets, and provides an opportunity
to ensure that systems are as
streamlined and practical as possible.
 Draft law is modified based
on feedback.
 The secretariat presents
the draft to the Assembly.
 The law is passed by the
Assembly.
 The
appropriate
government
body
completes internal policies
or
regulations
to
implement the law.
The example above would differ
for each country depending on
its unique process for legal
reform.
STANDARD INDICATORS
Standard indicators are used
primarily for Agency reporting
purposes. Standard indicators
produce data that can be
aggregated
across
many
programs. Optimally, standard
indicators meet both Agency
reporting and on-the-ground
management needs. However,
in
many
cases,
standard
indicators do not substitute for
performance
(or
custom
indicators) because they are
designed to meet different
needs. There is often a tension
between measuring a standard
across many programs and
selecting indicators that best
reflect true program results and
that can be used for internal
management purposes.
Example: Number of Laws or
Amendments
to
Ensure
Credible Elections Adopted
with
USG
Technical
Assistance.
In comparing the standard
indicator above with the
previous example of a custom
indicator, it becomes clear that
the custom indictor is more
likely to be useful as a
management tool, because it
provides greater specificity and
is more sensitive to change.
Standard indicators also tend to
measure change at the output
level, because they are precisely
the types of measures that are,
at face value, more easily
aggregated
across
many
programs, as the following
example demonstrates.
Example:
The number of
people trained in policy and
regulatory practices.
CONTEXTUAL INDICATORS
Contextual indicators are used
to understand the broader
environment
in
which
a
program operates, to track
assumptions, or to examine
externalities that may affect
success, failure, or progress.
3
INDICATORS AND DATA—SO
WHAT’S THE DIFFERENCE?
Indicators define the particular
characteristic or dimension that will
be used to measure change. Height
is an example of an indicator.
The data are the actual
measurements or factual information
that result from the indicator. Five
feet seven inches is an example of
data.
They do not represent program
performance,
because
the
indicator measures very highlevel change.
Example:
Score on the
Freedom House Index or
Gross
Domestic
Product
(GDP).
This sort of indicator may be
important
to
track
to
understand the context for
USAID programming (e.g. a
severe drop in GDP is likely to
affect
economic
growth
programming), but represents a
level of change that is outside
the manageable interest of
program managers. In most
cases, it would be difficult to
say that USAID programming
has affected the overall level of
freedom within a country or
GDP (given the size of most
USAID programs in comparison
to the host country economy,
for example).
WHAT ARE USAID’S
CRITERIA FOR
SELECTING
INDICATORS?
USAID policies (ADS 203.3.4.2)
identify seven key criteria to
guide
the
selection
of
performance indicators:







Direct
Objective
Useful for Management
Attributable
Practical
Adequate
Disaggregated, as necessary
These criteria are designed to
assist managers in selecting
optimal indicators. The extent
to
which
performance
indicators meet each of the
criteria must be consistent with
the requirements of good
management.
As managers
consider these criteria, they
should use a healthy measure
of
common
sense
and
reasonableness.
While we
always
want
the
―best‖
indicators, there are inevitably
trade-offs
among
various
criteria. For example, data for
the most direct or objective
indicators of a given result
might be very expensive to
collect or might be available
too infrequently. Table 1
includes a summary checklist
that can be used during the
selection process to assess
these trade-offs.
Two
overarching
factors
determine the extent to which
performance indicators function
as useful tools for managers
and decision-makers:


The degree to which
performance
indicators
accurately
reflect
the
process or phenomenon
they are being used to
measure.
The level of comparability of
performance indicators over
time: that is, can we
measure
results
in
a
consistent and comparable
manner over time?
1. DIRECT
An indicator is direct to the
extent that it clearly measures
the intended result.
This
criterion is, in many ways, the
most important. While this may
appear to be a simple concept,
it is one of the more common
problems
with
indicators.
Indicators should either be
widely accepted for use by
specialists in a subject area,
exhibit readily understandable
face validity (i.e., be intuitively
understandable),
or
be
supported
by
research.
Managers should place greater
confidence in indicators that are
direct. Consider the following
example:
Result:
Increased
Transparency of Key Public
Sector Institutions
4
Indirect Indicator: Passage
of
the
Freedom
of
Information Act (FOIA)
Direct Indicator: Progress
on
a
milestone
scale
demonstrating
enactment
and enforcement of policies
that require open hearings
The passage of FOIA, while an
important step, does not
actually measure whether a
target institution is more
transparent.
The
better
example outlined above is a
more direct measure.
Level
Another dimension of whether
an indicator is direct relates to
whether it measures the right
level of the objective. A
common problem is that there
is often a mismatch between
the stated result and the
indicator. The indicator should
not measure a higher or lower
level than the result.
For example, if a program
measures
improved
management practices through
the real value of agricultural
production, the indicator is
measuring a higher-level effect
than is stated (see Figure 1).
Understanding levels is rooted
in
understanding
the
development
hypothesis
inherent
in
the
Results
Framework (see TIPS 13:
Building a Results Framework).
Tracking indicators at each level
facilitates better understanding
and analysis of whether the
Figure 1. Levels
RESULT
INDICATOR
Increased
Production
Real value of
agricultural
production.
Improved
Management
Practices
Number and
percent of
farmers using a
new technology.
Improved
Knowledge
and
Awareness
Number and
percent of
farmers who can
identify five out
of eight steps
for
implementing a
new technology.
development hypothesis is
working.
For example, if
farmers are aware of how to
implement a new technology,
but the number or percent that
actually use the technology is
not increasing, there may be
other issues that need to be
addressed.
Perhaps the
technology is not readily
available in the community, or
there is not enough access to
credit. This flags the issue for
managers and provides an
opportunity
to
make
programmatic adjustments.
Proxy Indicators
Proxy indicators are linked to
the result by one or more
assumptions. They are often
used when the most direct
indicator is not practical (e.g.,
data collection is too costly or
the
program
is
being
implemented in a conflict zone).
When proxies are used, the
relationship
between
the
indicator and the result should
be well-understood and clearly
articulated.
The
more
assumptions the indicator is
based upon, the weaker the
indicator.
Consider
the
following examples:
Result: Increased Household
Income
Proxy Indicator:
Dollar
value
of
household
expenditures
The proxy indicator above
makes the assumption that an
increase in income will result in
increased
household
expenditures; this assumption is
well-grounded in research.
Result: Increased Access to
Justice
Proxy Indicator: Number of
new courts opened
The indicator above is based on
the assumption that physical
access to new courts is the
fundamental
development
problem—as
opposed
to
corruption, the costs associated
with using the court system, or
lack of knowledge of how to
obtain legal assistance and/or
use court systems. Proxies can
be used when assumptions are
clear and when there is research
to support that assumption.
2. OBJECTIVE
An indicator is objective if it is
unambiguous about 1) what is
being measured and 2) what
5
data are being collected. In
other words, two people should
be able to collect performance
information for the same
indicator and come to the same
conclusion.
Objectivity is
critical to collecting comparable
data over time, yet it is one of
the most common problems
noted in audits.
As a result,
pay particular attention to the
definition of the indicator to
ensure that each term is clearly
defined, as the following
examples demonstrate:
Poor Indicator: Number of
successful firms
Objective Indicator:
Number of firms with an
annual increase in revenues
of at least 5%
The better example outlines the
exact
criteria
for
how
―successful‖ is defined and
ensures that changes in the
data are not attributable to
differences in what is being
counted.
Objectivity can be particularly
challenging when constructing
qualitative indicators.
Good
qualitative indicators permit
regular, systematic judgment
about progress and reduce
subjectivity (to the extent
possible).
This means that
there must be clear criteria or
protocols for data collection.
3. USEFUL FOR
MANAGEMENT
An indicator is useful to the
extent that it provides a
meaningful measure of change
over time for management
decision-making. One aspect of
usefulness is to ensure that the
indicator is measuring the ―right
change‖ in order to achieve
development results.
For
example, the number of
meetings between Civil Society
Organizations
(CSOs)
and
government is something that
can be counted but does not
necessarily reflect meaningful
change. By selecting indicators,
managers are defining program
success in concrete ways.
Managers
will
focus
on
achieving targets for those
indicators, so it is important to
consider the intended and
unintended incentives
that
performance indicators create.
As a result, the system may
need to be fine-tuned to ensure
that incentives are focused on
achieving true results.
A second dimension is whether
the indictor measures a rate of
change that is useful for
management purposes.
This
means that the indicator is
constructed so that change can
be monitored at a rate that
facilitates management actions
(such as corrections and
improvements).
Consider the
following examples:
Result:
Targeted legal
reform
to
promote
investment
Less Useful for
Management: Number of
laws passed to promote
direct investment.
More Useful for
Management: Progress
toward targeted legal reform
based on the following
stages:
Stage 1. Interested groups
propose that legislation is
needed on issue.
Stage 2. Issue is introduced
in the relevant legislative
committee/executive
ministry.
Stage 3.
Legislation is
drafted
by
relevant
committee
or
executive
ministry.
Stage 4.
Legislation is
debated by the legislature.
Stage 5.
Legislation is
passed by full approval
process needed in legislature.
Stage 6.
Legislation is
approved by the executive
branch (where necessary).
Stage 7.
Implementing
actions are taken.
Stage 8. No immediate need
identified for amendments to
the law.
The less useful example may be
useful for reporting; however, it
is so general that it does not
provide a good way to track
progress
for
performance
management. The process of
passing or implementing laws is
a long-term one, so that over
the course of a year or two the
AO team may only be able to
report that one or two such
laws have passed when, in
reality, a high degree of effort is
6
invested in the process. In this
case, the more useful example
better articulates the important
steps that must occur for a law
to be passed and implemented
and facilitates management
decision-making. If there is a
problem in meeting interim
milestones, then corrections
can be made along the way.
4. ATTRIBUTABLE
An indicator is attributable if it
can be plausibly associated with
USAID interventions.
The
concept
of
―plausible
association‖ has been used in
USAID for some time. It does
not mean that X input equals Y
output. Rather, it is based on
the idea that a case can be
made to other development
practitioners that the program
has
materially
affected
identified change.
It is
important to consider the logic
behind what is proposed to
ensure attribution. If a Mission
is piloting a project in three
schools, but claims national
level
impact
in
school
completion, this would not pass
the
common
sense
test.
Consider
the
following
examples:
Result: Improved Budgeting
Capacity
Less Attributable: Budget
allocation for the Ministry of
Justice (MOJ)
More Attributable:
The
extent to which the budget
produced by the MOJ meets
established criteria for good
budgeting
If the program works with the
Ministry of Justice to improve
budgeting
capacity
(by
providing technical assistance
on budget analysis), the quality
of the budget submitted by the
MOJ may improve. However, it
is often difficult to attribute
changes in the overall budget
allocation
to
USAID
interventions, because there are
a number of externalities that
affect a country’s final budget –
much like in the U.S.
For
example, in tough economic
times, the budget for all
government institutions may
decrease. A crisis may emerge
that requires the host country
to reallocate resources. The
better example above is more
attributable (and directly linked)
to USAID’s intervention.
5. PRACTICAL
A practical indicator is one for
which data can be collected on a
timely basis and at a reasonable
cost. There are two dimensions
that determine whether an
indicator is practical. The first is
time and the second is cost.
Time
Consider whether resulting data
are available with enough
frequency for management
purposes (i.e., timely enough to
correspond
to
USAID
performance management and
reporting purposes). Second,
examine whether data are
current when available.
If
reliable data are available each
year, but the data are a year
old, then it may be problematic.
Cost
Performance indicators should
provide data to managers at a
cost that is reasonable and
appropriate as compared with
the management utility of the
data. As a very general rule of
thumb, it is suggested that
between 5% and 10% of
program or project resources
be allocated for monitoring and
evaluation (M&E) purposes.
However, it is also important to
consider priorities and program
context.
A program would
likely be willing to invest more
resources in measuring changes
that are central to decisionmaking and less resources in
measuring more tangential
results.
A more mature
program may have to invest
more in demonstrating higherlevel changes or impacts as
compared to a new program.
6. ADEQUATE
Taken as a group, the indicator
(or set of indicators) should be
sufficient to measure the stated
result. In other words, they
should be the minimum
number necessary and costeffective
for
performance
management. The number of
indicators
required
to
adequately measure a result
depends on 1) the complexity
of the result being measured, 2)
the amount of information
needed to make reasonably
confident decisions, and 3) the
7
level of resources available.
Too many indicators create
information
overload
and
become overly burdensome to
maintain. Too few indicators
are also problematic, because
the data may only provide a
partial or misleading picture of
performance. The following
demonstrates
how
one
indicator can be adequate to
measure the stated objective:
Result: Increased Traditional
Exports in Targeted Sectors
Adequate Indicator: Value
of traditional exports in
targeted sectors
In
contrast,
an
objective
focusing on improved maternal
health may require two or three
indicators to be adequate. A
general rule of thumb is to
select between two and three
performance indicators per
result. If many more indicators
are needed to adequately cover
the result, then it may signify
that the objective is not
properly focused.
7. DISAGGREGATED, AS
NECESSARY
The disaggregation of data by
gender, age, location, or some
other dimension is often
important
from
both
a
management and reporting
point of view. Development
programs
often
affect
population
cohorts
or
institutions in different ways.
For example, it might be
important to know to what
extent youth (up to age 25) or
adults (25 and older) are
participating
in
vocational
training, or in which districts
schools
have
improved.
Disaggregated data help track
whether or not specific groups
participate in and benefit from
activities intended to include
them.
In particular, USAID policies
(ADS 203.3.4.3) require that
performance
management
systems and evaluations at the
AO and project or activity levels
include
gender-sensitive
indicators
and
sexdisaggregated data if the
activities or their anticipated
results involve or affect women
and men differently. If so, this
difference
would
be
an
important factor in managing
for sustainable program impact.
Consider the following example:
Result: Increased Access to
Credit
Gender-Sensitive Indicator:
Value of loans disbursed,
disaggregated
by
male/female.
WHAT IS THE
PROCESS FOR
SELECTING
PERFORMANCE
INDICATORS?
Selecting
appropriate
and
useful performance indicators
requires
careful
thought,
iterative refining, collaboration,
and consensus-building. The
following describes a series of
steps
to
select
optimal
performance
indicators4.
Although presented as discrete
steps, in practice some of these
can be effectively undertaken
simultaneously or in a more
iterative manner. These steps
may be applied as a part of a
larger process to develop a new
PMP, or in part, when teams
have to modify individual
indicators.

STEP 1. DEVELOP A
PARTICIPATORY PROCESS
FOR IDENTIFYING
PERFORMANCE INDICATORS
The most effective way to
identify indicators is to set up a
process
that
elicits
the
participation and feedback of a
number
of
partners
and
stakeholders.
This allows
managers to:
A common way to begin the
process is to hold working
sessions. Start by reviewing the
Results Framework.
Next,
identify indicators for the
Assistance
Objective,
then
move down to the Intermediate
Results. In some cases, the AO
team establishes the first round
of indicators and then provides
them to other partners for
input.
In other cases, key
partners may be included in the
working sessions.



Draw on different areas of
expertise.
Ensure
that
indicators
measure the right changes
and represent part of a
larger approach to achieve
development impact.
Build
commitment
and
understanding
of
the
linkage between indicators
and
results.
This
will
increase the utility of the
performance management
system
among
key
stakeholders.
This process focuses on presenting
greater detail related specifically to
indicator selection. Refer to TIPS 7:
Preparing a PMP for a broader set of
steps on how to develop a full PMP.
4
8

Build
capacity
for
performance management
among partners, such as
NGOs and partner country
institutions.
Ensure that systems are as
practical and streamlined as
possible.
Often
development partners can
provide excellent insight on
the
practical
issues
associated with indicators
and data collection.
It is important to task the group
with identifying the set of
minimal indicators necessary
and sufficient to manage the
program effectively. That is, the
group must go through a
process of prioritization in order
to narrow down the list. While
participatory processes may
take more time at the front end,
they almost always result in
more coherent and effective
system.
STEP 2. CLARIFY THE RESULT
Carefully define the result
desired. Good performance
indicators are based on clearly
articulated
and
focused
objectives. Review the precise
wording and intention of the
objective.
Determine what
exactly is meant by the result.
For example, if the result is
―improved
business
environment,‖ what does that
mean? What specific aspects of
the business environment will
be improved? Optimally, the
result should be stated with as
much specificity as possible. If
the result is broad (and the
team doesn’t have the latitude
to change the objective), then
the team might further define
its meaning.
Example:
One AO team
further defined their IR,
―Improved
Business
Environment,‖ as follows:
Making it easier to do
business in terms of resolving
disputes, obtaining licenses
from the government, and
promoting investment.

An identified set of key
policies are in place to
support investment.
Key
policies
include
laws,
regulations,
and
policies
related to the simplification of
investment
procedures,
bankruptcy, and starting a
business.

As the team gains greater
clarity and consensus on what
results are sought, ideas for
potential indicators begin to
emerge.
Be clear about what type of
change is implied. What is
expected
to
change—a
situation, a condition, the level
of knowledge, an attitude, or a
behavior?
For
example,
changing a country's voting
law(s) is very different from
changing citizens' awareness of
their right to vote (which is
different from voting). Each
type of change is measured by
different types of performance
indicators.
Identify more precisely the
specific targets for change. Who
or what are the specific targets
for the change? For example, if
individuals, which individuals?
For an economic growth
program designed to increase
exports, does the program
target all exporters or only
exporters of non-traditional
agricultural products? This is
known as identifying the ―unit
of analysis‖ for the performance
indicator.
STEP 3: IDENTIFY POSSIBLE
INDICATORS
Usually there are many possible
indicators for a particular result,
but some are more appropriate
and useful than others.
In
selecting indicators, don’t settle
too quickly on the first ideas
that come most conveniently or
obviously to mind. Create an
initial list of possible indicators,
using the following approaches:

Conduct a brainstorming
session with colleagues to
draw upon the expertise of
9
the full Assistance Objective
Team. Ask, ―how will we
know if the result is
achieved?‖

Consider other resources.
Many organizations have
databases or indicator lists
for various sectors available
on the internet.

Consult
experts.

Review the PMPs and
indicators
of
previous
programs
or
similar
programs in other Missions.
with
technical
STEP 4. ASSESS THE BEST
CANDIDATE INDICATORS,
USING THE INDICATOR
CRITERIA
Next, from the initial list, select
the
best
candidates
as
indicators.
The seven basic
criteria that can be used to
judge
an
indicator’s
appropriateness and utility
described in the previous
section are summarized in
Table 1. When assessing and
comparing possible indicators,
it is helpful to use this type of
checklist
to
guide
the
assessment
process.
Remember that there will be
trade-offs between the criteria.
For example, the optimal
indicator may not be the most
cost-effective to select.
STEP 5. SELECT THE “BEST”
PERFORMANCE INDICATORS
Select the best indicators to
incorporate in the performance
management system.
They
should be the optimum set of
measures that are useful to
management and can be
obtained at reasonable cost.
Be Strategic and Streamline
Where Possible. In recent years,
there has been a substantial
increase in the number of
indicators used to monitor and
track programs. It is important
to remember that there are
costs, in terms of time and
money, to collect data for each
indicator. AO teams should:

Select indicators based on
strategic thinking about
what must truly be achieved
for program success.

Review
indicators
to
determine whether any final
narrowing can be done. Are
some indicators not useful?
If so, discard them.

Use
participatory
approaches in order to
discuss
and
establish
priorities
that
help
managers focus on key
indicators that are necessary
and sufficient.
Ensure that the rationale for
indicator selection is recorded in
the PMP.
There are rarely
perfect
indicators
in
the
development environment—it
is more often a case of
weighing different criteria and
making the optimal choices for
a particular program.
It is
important to ensure that the
rationale behind these choices
is recorded in the PMP so that
new staff, implementers, or
auditors understand why each
indicator was selected.
STEP 6. FINE TUNE WHEN
NECESSARY
Indicators are part of a larger
system that is ultimately
designed to assist managers in
achieving development impact.
On the one hand, indicators
must remain comparable over
time but, on the other hand,
some refinements will invariably
be needed to ensure the system
is as effective as possible. (Of
course, there is no value in
continuing to collect bad data,
for example.) As a result, these
two issues need to be balanced.
Remember that indicator issues
are often flags for other
10
underlying problems. If a large
number of indicators are
frequently changed, this may
signify a problem with program
management or focus. At the
other end of the continuum, if
no indicators were to change
over a long period of time, it is
possible that a program is not
adapting and evolving as
necessary. In our experience,
some refinements are inevitable
as data are collected and
lessons learned. After some
rounds of data collection are
completed, it is often useful to
discuss indicator issues and
refinements among AO team
members and/or with partners
and
implementers.
In
particular, the period following
portfolio reviews is a good time
to refine PMPs if necessary.
TABLE 1. INDICATOR SELECTION CRITERIA CHECKLIST
Criteria
1. Direct
Definition
Checklist
Direct. The indicator clearly represents the
intended result. An outsider or an expert
in the field would agree that the indicator
is a logical measure for the stated result.
 Level. The indicator reflects the right
level; that is, it does not measure a
higher or lower level than the stated
result.
 Proxies. The indicator is a proxy
measure. If the indicator is a proxy, note
what assumptions the proxy is based
upon.
2. Objective
The indicator is clear and unambiguous
about what is being measured.
3. Useful for
Management
The indicator is useful for management
decision-making.
4. Attributable
The indicator can be plausibly associated
with USAID interventions.
5. Practical
Time. Data are produced with enough
frequency for management purposes (i.e.
timely enough to correspond to USAID
performance management and reporting
purposes). Data are current when
available.
Cost. Data are worth the cost to USAID
managers.
6. Adequate
The indicators, taken as a group, are
sufficient to measure the stated result. All
major aspects of the result are measured.
7. Disaggregated,
as necessary
The indicators are appropriately
disaggregated by gender, age, location, or
some other dimension that is important for
programming. In particular, gender
disaggregation has been considered as
required (see ADS 203.3.4.3).
11
Comments
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication, including Gerry Britan
and Subhi Mehdi of USAID’s Office of Management Policy, Budget and Performance (MPBP). This
publication was updated by Michelle Adams-Matson of Management Systems International.
Comments can be directed to:
Tel: (202) 712-1158
gbritan@usaid.gov
12
1996, Number 7
TIPS
USAID Center for Development Information and Evaluation
PREPARING A PERFORMANCE MONITORING PLAN
What Is a Performance Monitoring Plan?
A performance monitoring plan (PMP) is a tool USAID operating units use to
plan and manage the collection of performance data. Sometimes the plan also
includes plans for data analysis, reporting, and use.
USAID's
reengineering
guidance
requires
operating units
to prepare a
Performance
Monitoring Plan
for the
systematic and
timely collection
of performance
data.
This Tips offers
advice for
preparing such a
plan.
Reengineering guidance requires operating units to prepare PMPs once their
strategic plans are approved. At a minimum, PMPs should include:
a detailed definition of each performance indicator
the source, method, frequency and schedule of data collection, and
the office, team, or individual responsible for ensuring data are
available on schedule
As part of the PMP process, it is also advisable (but not mandated) for
operating units to plan for:
how the performance data will be analyzed, and
how it will be reported, reviewed, and used to inform decisions
While PMPs are required, they are for the operating unit's own use. Review by
central or regional bureaus is not mandated, although some bureaus encourage
sharing PMPs. PMPs should be updated as needed to ensure plans, schedules,
and assignments remain current.
Why Are PMPs Important?
A performance monitoring plan is a critical tool for planning, managing, and
documenting data collection. It contributes to the effectiveness of the
performance monitoring system by assuring that comparable data will be
collected on a regular and timely basis. These are essential to the operation of a
credible and useful performance-based management approach.
PMPs promote the collection of comparable data by sufficiently documenting
indicator definitions, sources, and methods of data collection. This enables
operating units to collect comparable data over time even when key personnel
change.
PMPs support timely collection of data by documenting the frequency and
schedule of data collection as well as by assigning responsibilities. Operating
units should also consider developing plans for data analysis, reporting, and
review efforts as part of the PMP process. It makes sense to
PN-ABY-215
2
Use a Participatory Approach
The Agency's reengineering directives require that operating units involve USAID's partners, customers, and
stakeholders in planning approaches to monitoring performance. Experience indicates the value of collaborating
with relevant host government officials, implementing agency staff, contractors and grantees, other donors, and
customer groups, when preparing PMPs. They typically have the most familiarity with the quality, availability,
think through data collection, analysis, reporting, and
review as an integrated process. This will help keep the
performance monitoring system on track and ensure
performance data informs decision-making. While there
are strong arguments for including such integrated plans
in the PMP document, this is not mandated in the
reengineering guidance. Some operating units may wish
to prepare these plans separately.
Elements of a PMP
The following elements should be considered for
inclusion in a performance monitoring plan. Elements
1- 5 are required in the reengineering guidance, whereas
6 -9 are suggested as useful practices.
I. Plans for Data Collection (Required)
In its strategic plan, an operating unit will have identified
a few preliminary performance indicators for each of its
strategic objectives, strategic support objectives, and
special objectives (referred to below simply as SOs), and
USAID-supported intermediate results (IRs). In most
cases, preliminary baselines and targets will also have
been provided in the strategic plan. The PMP builds on
this initial information, verifying or modifying the
performance indicators, baselines and targets, and
documenting decisions.
PMPs are required to include information outlined below
(elements 1-5) on each performance indicator that has
been identified in the Strategic Plan for SOs and IRs.
Plans should also address how critical assumptions and
results supported by partners (such as the host
government, other donors, NGOs) will be monitored,
although the same standards and requirements for
developing indicators and collecting data do not apply.
Furthermore, it is useful to include in the PMP lowerlevel indicators of inputs, outputs, and processes at the
activity level, and how they will be monitored and
linked to IRs and SOs.
1. Performance Indicators and Their Definitions
Each performance indicator needs a detailed definition.
Be precise about all technical elements of the indicator
statement. As an illustration, consider the indicator,
number of small enterprises receiving loans from the
private banking system. How are small enterprises
defined -- all enterprises with 20 or fewer employees, or
50 or 100? What types of institutions are considered part
of the private banking sector -- credit unions,
government-private sector joint-venture financial
institutions?
Include in the definition the unit of measurement. For
example, an indicator on the value of exports might be
otherwise well defined, but it is also important to know
whether the value will be measured in current or constant
terms and in U.S. dollars or local currency.
The definition should be detailed enough to ensure that
different people at different times, given the task of
collecting data for a given indicator, would collect
identical types of data.
2. Data Source
Identify the data source for each performance indicator.
The source is the entity from which the data are obtained,
usually the organization that conducts the data collection
effort. Data sources may include government
departments, international organizations, other donors,
NGOs, private firms, USAID offices, contractors, or
activity implementing agencies.
Be as specific about the source as possible, so the same
source can be used routinely. Switching data sources for
the same indicator over time can lead to inconsistencies
and misinterpretations and should be avoided. For
example, switching from estimates of infant mortality
rates based on national sample surveys to estimates based
on hospital registration statistics can lead to false
impressions of change.
3
Plans may refer to needs and means for strengthening the
capacity of a particular data source to collect needed data
on a regular basis, or for building special data collection
efforts into USAID activities.
3. Method of Data Collection
Specify the method or approach to data collection for
each indicator. Note whether it is primary data collection
or is based on existing secondary data.
For primary data collection, consider:
the unit of analysis (individuals, families,
communities, clinics, wells)
data disaggregation needs (by gender, age, ethnic
groups, location)
sampling techniques for selecting cases (random
sampling, purposive sampling); and
techniques or instruments for acquiring data on
these selected cases (structured questionnaires,
direct observation forms, scales to weigh infants)
For indicators based on secondary data, give the method
of calculating the specific indicator data point and the
sources of data.
Note issues of data quality and reliability. For example,
using secondary data from existing sources cuts costs and
efforts, but its quality may not be as reliable.
Provide sufficient detail on the data collection or
calculation method to enable it to be replicated.
4. Frequency and Schedule of Data Collection
Performance monitoring systems must gather
comparable data periodically to measure progress. But
depending on the performance indicator, it may make
sense to collect data on a quarterly, annual, or less
frequent basis. For example, because of the expense and
because changes are slow, fertility rate data from sample
surveys may only be collected every few years whereas
data on contraceptive distributions and sales from clinics'
record systems may be gathered quarterly. PMPs can
also usefully provide the schedules (dates) for data
collection efforts.
When planning the frequency and scheduling of data
collection, an important factor to consider is
management's needs for timely information for decisionmaking.
5. Responsibilities for Acquiring Data
For each performance indicator, the responsibility the
operating unit for the timely acquisition of data from
their source should be clearly assigned to a particular
office, team, or individual.
II. Plans for Data Analysis, Reporting,
Review, and Use
An effective performance monitoring system needs to
plan not only for the collection of data, but also for data
analysis, reporting, review, and use. It may not be
possible to include everything in one document at one
time, but units should take the time early on for careful
planning of all these aspects in an integrated fashion.
6. Data Analysis Plans
To the extent possible, plan in advance how performance
data for individual indicators or groups of related
indicators will be analyzed. Identify data analysis
techniques and data presentation formats to be used.
Consider if and how the following aspects of data
analysis will be undertaken:
Comparing disaggregated data. For indicators with
disaggregated data, plan how it will be compared,
displayed, and analyzed.
Comparing current performance against multiple
criteria. For each indicator, plan how actual performance
data will be compared with a) past performance, b)
planned or targeted performance or
c) other relevant benchmarks.
Analyzing relationships among performance indicators.
Plan how internal analyses of the performance data will
examine interrelationships. For example
How will a set of indicators (if there are more
than one) for a particular SO or IR be analyzed
to reveal progress? What if only some of the
indicators reveal progress?
How will cause-effect relationships among SOs
and IRs within a results framework be analyzed?
How will USAID activities be linked to
achieving IRs and SOs?
Analyzing cost-effectiveness. When practical and
feasible, plan for using performance data to compare
systematically alternative program approaches in terms
of costs as well as results. The Government Performance
and Results Act (GPRA) encourages this.
4
7. Plans for Complementary Evaluations
Reengineering stresses that evaluations should be
conducted only if there is a clear management need. It
may not always be possible or desirable to predict years
in advance when or why they will be needed.
Nevertheless, operating units may find it useful to plan
on a regular basis what evaluation efforts are needed to
complement information from the performance
monitoring system. The operating unit's internal
performance reviews, to be held periodically during the
year, may be a good time for such evaluation planning.
For example, if the reviews reveal that certain
performance targets are not being met, and if the reasons
why are unclear, then planning evaluations to investigate
why would be in order.
8. Plans for Communicating and Using Performance
Information
Planning how performance information will be reported,
reviewed, and used is critical for effective managing for
results. For example, plan, schedule, and assign
responsibilities for internal and external reviews,
briefings, and reports. Clarify what, how and when
management decisions will consider performance
information. Specifically, plan for the following:
Operating unit performance reviews. Reengineering
guidance requires operating units to conduct internal
reviews of performance information at regular intervals
during the year to assess progress toward achieving SOs
and IRs. In addition, activity-level reviews should be
planned regularly by SO teams to assess if activities'
inputs, outputs, and processes are supporting
achievement of IRs and SOs.
USAID/Washington reviews and the R4 Report.
Reengineering requires operating units to prepare and
submit to USAID/Washington an annual Results Review
and Resource Request (R4) report, which is the basis for
a joint review with USAID/W of performance and
resource requirements. Help plan R4 preparation by
scheduling tasks and making assignments.
External reviews, reports, and briefings. Plan for
reporting and disseminating performance information to
key external audiences, such as host government
counterparts, collaborating NGOs, other partners, donors,
customer groups, and stakeholders. Communication
techniques may include reports, oral briefings,
videotapes, memos, newspaper articles.
Influencing management decisions. The ultimate aim of
performance monitoring systems is to promote
performance-based decision-making. To the extent
possible, plan in advance what management decisionmaking processes should be influenced by performance
information. For example, budget discussions,
programming decisions, evaluation designs/scopes of
work, office retreats, management contracts, and
personnel appraisals often benefit from the consideration
of performance information.
9. Budget
Estimate roughly the costs to the operating unit of
collecting, analyzing, and reporting performance data for
a specific indicator (or set of related indicators). Identify
the source of funds.
If adequate data are already available from secondary
sources, costs may be minimal. If primary data must be
collected at the operating unit's expense, costs can vary
depending on scope, method, and frequency of data
collection. Sample surveys may cost more than
$100,000, whereas rapid appraisal methods can be
conducted for much less. However, often these low-cost
methods do not provide quantitative data that are
sufficiently reliable or representative.
Reengineering guidance gives a range of 3 to 10 percent
of the total budget for an SO as a reasonable level to
spend on performance monitoring and evaluation.
CDIE's Tips series provides advice and
suggestions to USAID managers on how to
plan and conduct performance monitoring
and evaluation activities effectively. They
are supplemental references to the
reengineering automated directives system
(ADS), chapter 203. For further information, contact Annette Binnendijk, CDIE
Senior Evaluation Advisor, via phone
(703) 875-4235, fax (703) 875-4866, or email. Copies of TIPS can be ordered from
the Development Information Services
Clearinghouse by calling (703) 351-4006 or
by faxing (703) 351-4039. Please refer to
the PN number. To order via Internet,
address requests to
docorder@disc.mhs.compuserve.com
NUMBER 8
2ND EDITION, 2010
TIPS
BASELINES AND TARGETS
ABOUT TIPS
INTRODUCTION
The achievement of planned results
is at the heart of USAID’s
performance management system. In
order to understand where we, as
project managers, are going, we
need to understand where we have
been. Establishing quality baselines
and setting ambitious, yet achievable,
targets are essential for the
successful management of foreign
assistance programs.
WHAT ARE
BASELINES AND
TARGETS?
A baseline is the value of a
performance indicator before the
implementation of projects or
activities, while a target is the
specific, planned level of result to be
achieved
within
an
explicit
timeframe (see ADS 203.3.4.5).
Targets are set for indicators at the
Assistance
Objective
(AO),
Intermediate Result (IR), and output
levels.
WHY ARE
BASELINES
IMPORTANT?
Baselines help managers determine
progress in achieving outputs and
outcomes. They also help identify
the extent to which change has
happened at each level of result.
USAID ADS 203.3.3 requires a PMP
for each AO. Program managers
should provide baseline and target
values for every indicator in the
PMP.
Lack of baseline data not only
presents challenges for management
decision-making purposes, but also
hinders evaluation efforts.
For
example, it is generally not possible
to conduct a rigorous impact
1
evaluation without solid baseline
data (see TIPS 19: Rigorous Impact
Evaluation).
ESTABLISHING THE
BASELINE
Four common scenarios provide the
context for establishing baseline
data:
1. BASELINE IS
ESTABLISHED
If baseline data exist prior to the
start of a project or activity,
additional data collected over the
life of the project must be collected
in a consistent manner in order to
facilitate comparisons. For example,
consider the drop-out rate for girls 16
and under. If baseline data are
obtained from the Ministry of
Education, the project should
continue to collect these data from
this same source, ensuring that the
data
collection
remains the same.
methodology
Data may also be obtained from a
prior
implementing
partner’s
project, provided that the data
collection protocols, instruments,
and scoring procedures can be
replicated. For example, a policy
index might be used to measure
progress of legislation (see TIPS 14:
Monitoring the Policy Reform
Process). If these activities become
a part of a new project, program
managers should consider the
benefit
of using
the
same
instrument.
In cases where baseline data exist
from primary or secondary sources,
it is important that the data meet
USAID’s data quality standards for
validity,
reliability,
precision,
integrity, and timeliness (see TIPS
12: Data Quality Standards).
2. BASELINES MUST BE
COLLECTED
In cases where there are no existing
data with which to establish a
baseline,
USAID
and/or
its
implementing partners will have to
collect it if the required data are not
already being collected by, for
example,
a
host-country
government,
an
international
organization, or another donor.
Primary data collection can be
expensive, particularly if data are
collected through a formal survey or
Participation of key stakeholders in
setting targets helps establish a
common understanding about what
the project will accomplish and
when. USAID staff, implementing
partners, host country governments,
other donors, and civil society
partners, among others, should
attend working sessions at the outset
of program implementation to
review baseline data and other
information to set interim and final
targets.
a new index. Program managers
should consider this cost and
incorporate it into program or
project planning.
Ideally, data should be collected
prior to the initiation of the
program. If this is not feasible,
baselines should be collected as
soon as possible. For example, an
implementing partner may collect
perception data on the level of
corruption in targeted municipalities
for USAID’s PMP sixty days after
approval of a project’s work plan; in
another case, a score on an
advocacy capacity index may not be
collected until Community Service
Organizations (CSOs) are awarded
grants. If baseline data cannot be
collected until later in the course of
implementing an activity, the AO
Team should document when and
how the baseline data will be
collected (ADS 203.3.4.5).
3. BASELINES ARE
ESTABLISHED ON A
ROLLING BASIS
In some cases, it is possible to
collect baseline data on a rolling
basis as implementation proceeds.
For example, imagine that a health
project is being rolled out
sequentially across three provinces
over a three-year period. Data
collected in the first province will
serve as baseline for Year One; data
collected in the second province will
serve as baseline for the second
province in Year Two; and data
collected in the third province will
serve as baseline for that province in
Year Three.
The achievement of results requires
the joint action of many
stakeholders. Manageable interest
means we, as program managers,
have sufficient reason to believe that
the achievement of our planned
results can be significantly influenced
by interventions of USAID’s
program and staff resources. When
setting targets, take into account the
achievement of how other actors
will affect outcomes and what it
means for USAID to achieve
success.
program is the number of grants
awarded, the baseline is zero.
WHY ARE TARGETS
IMPORTANT?
Beyond
meeting
USAID
requirements, performance targets
are important for several reasons.
They help justify a program by
describing in concrete terms what
USAID’s investment will produce.
Targets orient stakeholders to the
tasks to be accomplished and
motivate individuals involved in a
program to do their best to ensure
the targets are met. Targets also
help to establish clear expectations
for USAID staff, implementing
partners, and key stakeholders.
Once a program is underway, they
serve as the guideposts for
monitoring whether progress is
being made on schedule and at the
levels originally envisioned. Lastly,
targets promote transparency and
accountability by making available
information on whether results have
been achieved or not over time.
4. BASELINE IS ZERO
For some indicators, baselines will
be zero. For example, if a new
program focuses on building the
teaching skills of teachers, the
baseline for the indicator “the
number of teachers trained” is zero.
Similarly, if an output of a new
2
A natural tension exists between the
need to set realistic targets and the
value,
from
a
motivational
perspective, of setting targets
ambitious enough to ensure that
staff and stakeholders will stretch to
meet them; when motivated, people
can often achieve more than they
imagine. Targets that are easily
achievable are not useful for
management and reporting purposes
since they are, in essence, pro forma.
AO Teams should plan ahead for
the analysis and interpretation of
actual
data
against
their
performance
targets
(ADS
203.3.4.5).
USING TARGETS
FOR
PERFORMANCE
MANAGEMENT IN A
LEARNING
ORGANIZATION
Targets can be important tools for
effective program management.
However, the extent to which
targets are or are not met should
not be the only criterion for judging
the success or failure of a program.
Targets are essentially flags for
managers; if the targets are wildly
exceeded
or
well-below
expectations, the program manager
should ask, “Why?”
Consider an economic growth
project. If a country experiences an
unanticipated downturn in its
economy,
the
underlying
FIGURE 1. PORTFOLIO
REVIEWS AND
PERFORMANCE TARGETS
To prepare for Portfolio Reviews,
AO Teams should conduct analysis of
program data, including achievement
of planned targets. ADS 203.3.7.2
provides illustrative questions for
these reviews:
• Are the desired results being
achieved?
• Are the results within USAID’s
manageable interest?
• Will planned targets be met?
• Is the performance management
system currently in place adequate
to capture data on the achievement
of results?
assumptions upon which that
project was designed may be
affected. If the project does not
meet targets, then it is important for
managers to focus on understanding
1) why targets were not met, and 2)
whether the project can be adjusted
to allow for an effective response to
changed circumstances.
In this
scenario, program managers may
need to reexamine the focus or
priorities of the project and make
related adjustments in indicators
and/or targets.
Senior
managers,
staff,
and
implementing
partners
should
review performance information and
targets as part of on-going project
management responsibilities and in
Portfolio Reviews (see Figure 1.)
TYPES OF TARGETS
FINAL AND INTERIM
TARGETS
A final target is the planned value of
a performance indicator at the end
of the AO or project. For AOs, the
final targets are often set three to
five years away, while for IRs they
are often set one to three years
away. Interim targets should be set
for the key points of time in
between the baseline and final target
in cases where change is expected
and data can be collected.
QUANTITATIVE AND
QUALITATIVE TARGETS
Targets may be either quantitative
or qualitative, depending on the
nature of the associated indicator.
Targets for quantitative indicators are
numerical, whereas targets and for
qualitative indicators are descriptive.
To
facilitate
comparison
of
baselines, targets, and performance
data for descriptive data, and to
maintain
data
quality,
some
indicators convert qualitative data
into a quantitative measure (see
Figure 2).
Nonetheless, baseline
and target data for quantitative and
3
FIGURE 2. TARGET
SETTING FOR
QUANTITATIVE AND
QUALITATIVE INDICATORS
- WHAT’S THE
DIFFERENCE?
Quantitative indicators and targets
are numerical. Examples include
the dropout rate, the value of
revenues, or number of children
vaccinated.
Qualitative indicators and targets
are descriptive. However,
descriptions must be based on a set
of pre-determined criteria. It is
much easier to establish baselines
and set targets when qualitative
data are converted into a
quantitative measure. For example,
the Advocacy Index is used to
measure the capacity of a target
organization, based on agreed-upon
standards that are rated and
scored. Other examples include
scales, indexes, and scorecards (see
Figure 3).
qualitative indicators must be
collected using the same instrument
so that change can be captured and
progress towards results measured
accurately (see TIPS 6: Selecting
Performance Indicators).
EXPRESSING
TARGETS
As with performance indicators,
targets can be expressed differently.
There are several possible ways to
structure
targets
to
answer
questions about the quantity of
expected change:
• Absolute level of achievement –
e.g., 75% of all trainees obtained
jobs by the end of the program or
7,000 people were employed by
the end of the program.
• Change in level of achievement –
e.g., math test scores for students
in grade nine increased by 10% in
Year One, or math test scores for
students in grade nine increased
FIGURE 3. SETTING TARGETS FOR QUALITATIVE MEASURES
For the IR Improvements in the Quality of Maternal and Child Health Services, a service delivery scale was used as the
indicator to measure progress. The scale, as shown below, transforms qualitative information about services into a rating
system against which targets can be set:
0 points = Service not offered
1 point = Offers routine antenatal care
1 point = Offers recognition and appropriate management of high risk pregnancies
1 point = Offers routine deliveries
1 point = Offers appropriate management of complicated deliveries
1 point = Offers post-partum care
1 point = Offers neonatal care
Score
= Total number of service delivery points
Illustrative Target: Increase average score to 5 by the end of year.
by three points in Year One.
Yields
per
hectare
under
improved management practices
increased by 25% or yields per
hectare increased by 100 bushels
from 2010 to 2013.
• Change in relation to the scale of
the problem – e.g., 35% of total
births in target area attended by
skilled health personnel by the end
of year two, or the proportion of
households with access to reliable
potable water increased by 50% by
2013.
• Creation
or
provision
of
something new – e.g., 4,000 doses
of tetanus vaccine distributed in
Year One, or a law permitting
non-government organizations to
generate income is passed by
2012.
Other targets may be concerned
with the quality of expected results.
Such targets can relate to indicators
measuring customer satisfaction,
public opinion, responsiveness rates,
enrollment rates, complaints, or
failure rates.
For example, the
average customer satisfaction score
for registration of a business license
(based on a seven-point scale)
increases to six by the end of the
program, or the percentage of
mothers who return six months
after delivery for postnatal care
increases to 20% by 2011.
Targets relating to cost efficiency or
producing outcomes at the least
expense are typically measured in
terms of unit costs. Examples of
such targets might include: cost of
providing
a
couple-year-ofprotection is reduced to $10 by
1999 or per-student costs of a
training program are reduced by
20% between 2010 and 2013.
demonstrate that:
DISAGGREGATING
TARGETS
A gender-sensitive indicator can be
defined as an indicator that captures
gender-related changes in society
over time. For example, a program
may focus on increasing enrollment
of children in secondary education.
Program managers may not only
want to look at increasing
enrollment rates, but also at the gap
between girls and boys. One way to
measure performance would be to
When a program’s progress is
measured in terms of its effects on
different segments of the population,
disaggregated targets can provide
USAID with nuanced information
that may not be obvious in the
aggregate. For example, a program
may seek to increase the number of
micro-enterprise loans received by
businesses in select rural provinces.
By disaggregating targets, program
inputs can be directed to reach a
particular target group.
Targets can be disaggregated along a
number of dimensions including
gender, location, income level,
occupation, administration level
(e.g., national vs. local), and social
groups.
For USAID programs, performance
management systems must include
gender-sensitive indicators and sexdisaggregated data when the
technical analyses supporting the
AO or project to be undertaken
4
• The different roles and status of
women and men affect the
activities differently; and
• The anticipated results of the
work would affect women and
men differently.
FIGURE 4. AN EXAMPLE OF
DISAGGREGATED TARGETS
FOR GENDER SENSITIVE
INDICATORS
Indicator: Number of children
graduating from secondary school;
percent gap between boys and girls.
B=boys; G=girls
Year
2010
(baseline)
Planned
2011
175
120B; 55G
50.0%
200
120B; 80G
25.0%
200
115B; 92G
2012
2013
Actual
145
115B; 30G
58.6%
160
120 B; 40G
56.3%
200
130 B; 70G
30.0%
205
110B; 95G
disaggregate the total number of
girls and boys attending school at
the beginning and at the end of the
school year (see Figure 4). Another
indicator might look at the quality of
the participation levels of girls vs.
boys with a target of increasing the
amount of time girls engage in
classroom discussions by two hours
per week.
Gender-sensitive indicators can use
qualitative
or
quantitative
methodologies to assess impact
directly on beneficiaries. They can
also be used to assess the
differential impacts of policies,
programs, or practices supported by
USAID on women and men (ADS
201.3.4.3).
Program managers should think
carefully about disaggregates prior
to collecting baseline data and
setting targets.
Expanding the
number of disaggregates can
increase the time and costs
associated with data collection and
analysis.
SETTING TARGETS
Targets
should
be
realistic,
evidence-based,
and
ambitious.
Setting meaningful targets provides
staff, implementing partners, and
stakeholders with benchmarks to
document
progress
toward
achieving results. Targets need to
take
into
account
program
resources,
the
implementation
period, and the development
hypothesis implicit in the results
framework.
PROGRAM RESOURCES
The level of funding, human
resources, material goods, and
institutional capacity contribute to
determining project outputs and
affecting change at different levels of
results and the AO. Increases or
decreases in planned program
resources should be considered
when setting targets.
ASSISTANCE OBJECTIVES
AND RESULTS
FRAMEWORKS
Performance
targets
represent
commitments that USAID AO
Teams make about the level and
timing of results to be achieved by a
program. Determining targets is
easier
when
objectives
and
indicators are within USAID’s
manageable interest.
Where a
result sits in the causal chain, critical
assumptions, and other contributors
to achievement of the AO will affect
targets.
Other key considerations include:
1. Historical Trends: Perhaps
even
more
important
than
examining a single baseline value, is
understanding
the
underlying
historical trend in the indicator
value over time. What pattern of
change has been evident in the past
five to ten years on the performance
indicator? Is there a trend, upward
FIGURE 5. PROGRESS IS NOT ALWAYS A STRAIGHT LINE
While it is easy to establish annual targets by picking an acceptable final
performance level and dividing expected progress evenly in the years between,
such straight-line thinking about progress is often inconsistent with the way
development programs really work. More often than not, no real progress – in
terms of measureable impacts or results – is evident during the start-up period.
Then, in the first stage of implementation, which may take the form of a pilot
test, some but not much progress is made, while the program team adjusts its
approaches. During the final two or three years of the program, all of this early
work comes to fruition. Progress leaps upward, and then rides a steady path at
the end of the program period. If plotted on a graph, it would look like “stair
steps,” not a straight line
5
or downward, that can be drawn
from existing reports, records, or
statistics? Trends are not always a
straight line; there may be a period
during which a program plateaus
before improvements are seen (see
Figure 5).
2. Expert Judgments: Another
option is to solicit expert opinions
as to what is possible or feasible
with respect to a particular indicator
and country setting. Experts should
be knowledgeable about the
program area as well as local
conditions. Experts will be familiar
with what is and what is not possible
from a technical and practical
standpoint – an important input for
any target-setting exercise.
3. Research Findings: Similarly,
reviewing development literature,
especially research and evaluation
findings, may help in choosing
realistic targets. In some program
areas, such as population and health,
extensive research findings on
development trends are already
widely available and what is possible
to achieve may be well-known. In
other areas, such as democracy,
research on performance indicators
and trends may be scarce.
4. Stakeholder Expectations:
While targets should be defined on
the basis of an objective assessment
of what can be accomplished given
certain conditions and resources, it
is useful to get input from
stakeholders regarding what they
want, need, and expect from USAID
activities.
What
are
the
expectations of progress? Soliciting
expectations may involve formal
interviews, rapid appraisals, or
informal conversations. Not only
end users should be surveyed;
intermediate
actors
(e.g.,
implementing agency staff) can be
especially useful in developing
realistic targets.
5. Achievement of Similar
Programs: Benchmarking is the
FIGURE 6. BENCHMARKING
One increasingly popular way of
setting targets and comparing
performance is to look at the
achievement of another program or
process by one or a collection of
high-performing organizations.
USAID is contributing to the
development of benchmarks for
programs such as water governance
(http://www.rewab.net), financial
management (www.fdirisk.com) and
health care systems
(www.healthsystems2020.org) Targets
may be set to reflect this “best in the
business” experience, provided of
course that consideration is given to
the comparability of country
conditions, resource availability, and
other factors likely to influence the
performance levels that can be
achieved.
process of comparing or checking
the progress of other similar
programs. It may be useful to
analyze progress of other USAID
Missions or offices, or other
development agencies and partners,
to understand the rate of change
that can be expected in similar
circumstances.
APPROACHES FOR
TARGET SETTING
There is no single best approach to
use when setting targets; the
process is an art and a science.
Although much depends on available
information, the experience and
knowledge of AO Team members
will add to the thinking behind
performance target.
Alternative
approaches include the following:
1. Projecting a future trend, then
adding the “valued added” by USAID
activities.
Probably the most
rigorous and credible approach, this
involves estimating the future trend
without USAID’s program, and then
adding whatever gains can be
expected as a result of USAID’s
efforts. This is no simple task, as
projecting the future can be very
tricky. The task is made somewhat
easier if historical data are available
and can be used to establish a trend
line.
2. Establishing a final performance
target for the end of the planning
period, and then planning the
progress from the baseline level.
This approach involves deciding on
the program’s performance target
for the final year, and then defining a
path of progress for the years in
between.
Final targets may be
judged on benchmarking techniques
or on judgments of experts,
program staff, customers, or
partners about the expectations of
what can be reasonably achieved
within the planning period. When
setting interim targets, remember
that progress is not always a straight
line. All targets, both final and
interim, should be based on a careful
analysis of what is realistic to
achieve, given the stage of program
implementation,
resource
availability,
country
conditions,
technical constraints, etc.
6
3. Setting annual performance
targets. Similar to the previous
approach, judgments are made
about what can be achieved each
year, instead of starting with a final
performance level and working
backwards. In both cases, consider
variations in performance, e.g.,
seasons and timing of activities and
expected results.
DOCUMENT AND
FILE
Typically, USAID project, baselines,
targets, and actual data are kept in a
data table for analysis either in the
PMP, as a separate document, or
electronically.
Furthermore, it is important to
document in the PMP how targets
were selected and why target values
were chosen.
Documentation
serves as a future reference for:
• Explaining
a
methodology.
target-setting
• Analyzing actual performance data.
• Setting targets in later years.
Responding to inquiries or audits
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication, including Gerry Britan and
Subhi Mehdi of USAID’s Office of Management Policy, Budget and Performance (MPBP). This publication was
updated by Jill Tirnauer of Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
7
NUMBER 9
2011 Printing
TIPS
CONDUCTING CUSTOMER SERVICE ASSESSMENTS
ABOUT TIPS
WHAT IS A CUSTOMER
SERVICE ASSESSMENT?
Under USAID’s new operations system, Agency operating units are required to
routinely and systematically
assess customer needs for,
perceptions of, and reactions
to USAID programs.
A customer service assessment is a management tool for understanding USAID’s programs
from the customer’s perspective. Most often
these assessments seek feedback from customers about a program’s service delivery performance. The Agency seeks views from both
ultimate customers (the end-users, or beneficiaries, of USAID activities—usually disadvantaged
groups) and intermediate customers (persons
or organizations using USAID resources, services, or products to serve the needs of the
ultimate customers).
This TIPS gives practical advice about customer service
assessments— for example,
when they should be conducted, what methods may
be used, and what information can be usefully included.
Customer service assessments may also be used
to elicit opinions from customers or potential
customers about USAID’s strategic plans, development objectives, or other planning issues.
1
For example, the operating unit may seek their
views on development needs and priorities to
help identify new, relevant activities.
WHO DOES CUSTOMER
SERVICE ASSESSMENTS?
WHY CONDUCT
CUSTOMER SERVICE
ASSESSMENTS?
USAID guidance specifies that all operating
units should develop a customer service plan.
The plan should include information about customers’ needs, preferences, and reactions as an
element in a unit’s planning, achieving, perfor-
USAID’s reengineered operating system calls
for regularly conducting customer service assessments for all program activities. Experience indicates that effective customer feedback
on service delivery improves performance,
achieves better results, and creates a more participatory working environment for programs,
and thus increases sustainability.
Box 1.The Customer Service Plan
The customer service plan presents the
operating unit’s vision for including customers and partners to achieve its objectives.
It explains how customer feedback will be
incorporated to determine customer needs
and perceptions
of services provided, and how this feedback
will be regularly incorporated into the unit’s
operations. The customer service plan is a
management tool for the operating unit and
does not require USAID/W approval.
Specifically, the plan
These assessments provide USAID staff with
the information they need for making constructive changes in the design and execution of development programs. This information may also
be shared with partners and customers as an
element in a collaborative, ongoing relationship.
In addition, customer service assessments provide input for reporting on results, allocating
resources, and presenting the operating unit’s
development programs to external audiences.
• Identifies the ultimate and intermediate customers for service delivery and
segments customer groups for different
programs, products, or services
Customer service assessments are relevant not
only to program-funded activities directed to
customers external to USAID. They can also
be very useful in assessing services provided to
internal USAID customers.
• Describes and regularly schedules appropriate means for assessing service
delivery, performance, and customer
satisfaction
Moreover, customer service assessments are
federally mandated. The Government Performance and Results Act of 1993 and Executive
Order 12862 of 1993 direct federal agencies to
reorient their programs toward achievement
of measurable results that reflect customers’
needs and to systematically assess those needs.
Agencies must report annually to the Administration on customer service performance.
• Establishes service principles and specifies measurable service performance
standards indicates staff responsibilities
for managing customer service activities—including assessments
• Specifies the resources required for customer service activities and assessments.
2
formance in delivering the program’s products
and services.
mance monitoring and evaluation functions (see
Box 1). Depending on the scope of its program
operations, an operating unit may find it needs
to plan several customer service assessments.
The various assessments might be tailored to
different strategic objectives, program activities
and services, or customer groups (differentiated, for example, by gender, ethnicity, or income).
Responsibility for designing and managing these
assessments typically is assigned to the relevant
development objective.
Unless the service or product delivery is satisfactory (i.e., timely, relevant, accessible, good
quality) from the perspective of the customers,
it is unlikely that the program will achieve its
substantive development results, which, after all,
ultimately depend on customers’ participation
and use of the service or product. For example,
a family planning program is unlikely to achieve
reduced fertility rates unless customers are satisfied with the contraceptive products
it offers and the delivery mechanism it uses to
provide them. If not sufficiently satisfied, customers will simply not use them.
HOW DO CUSTOMER
SERVICE ASSESSMENTS
COMPLEMENT PERFORMANCE MONITORING
AND EVALUATION?
Customer service assessments thus complement broader performance monitoring and
evaluation systems by monitoring a specific
type of result: service delivery performance
Performance monitoring and evaluation broad- from the customer’s perspective. By providing
ly addresses the results or outcomes of a pro- managers with information on whether cusgram.These results reflect objectives chosen by tomers are satisfied with and using a program’s
the operating unit (in consultation with part- products and services, these assessments are
ners and customer representatives) and may especially useful for giving early indications of
encompass several types of results.
whether longer term substantive development
results are likely to be met.
Often they are medium- to longer-term developmental changes or impacts. Examples: reduc- Both customer service assessments and perfortions in fertility rates, increases in income, im- mance monitoring and evaluation use the same
provements in agricultural yields, reductions in array of standard social science investigation
forest land destroyed.
techniques—surveys, rapid and participatory
appraisal, document reviews, and the like. In
some cases, the same survey or rapid appraisal
may even be used to gather both types of information. For example, a survey of customers of
an irrigation program might ask questions about
service delivery aspects (e.g., access, timeliness,
quality, use of irrigation water) and questions
concerning longer term development results
(e.g., yields, income).
Another type of result often included in performance monitoring and evaluation involves customer perceptions and responses to goods or
services delivered by a program— for example,
the percentage of women satisfied with the maternity care they receive, or the proportion of
farmers who have tried a new seed variety and
intend to use it again. Customer service assessments look at this type of result—customer
satisfaction, perceptions, preferences, and related opinions about the operating unit’s per3
planning the assessment should 1) identify the
purpose and intended uses of the information,
2) clarify the program products or services being assessed, 3) identify the customer groups
involved, and 4) define the issues the study will
address. Moreover, the scope of work typically discusses data collection methods, analysis
techniques, reporting and dissemination plans,
and a budget and time schedule.
STEPS IN CONDUCTING A
CUSTOMER SERVICE
ASSESSMENT
Step 1. Decide when the assessment
should be done.
Customer service assessments should be conducted whenever the operating unit requires
customer information for its management purposes. The general timing and frequency of customer service assessments is typically outlined
in the unit’s customer service plan.
Specific issues to be assessed will vary with the
development objective, program activities under way, socioeconomic conditions, and other
factors. However, customer service assessments generally aim at understanding
• Customer views regarding the importance
Customer service assessments are likely to
of various USAID-provided services (e.g.,
be most effective if they are planned to coortraining, information, commodities, technidinate with critical points in cycles associated
cal assistance) to their own needs and priwith the program being assessed (crop cycles,
orities
local school year cycles, host country fiscal year
cycles, etc.) as well as with the Agency’s own • Customer judgments, based on measurable
annual reporting and funding cycles.
service standards, on how well USAID is
performing service delivery
Customer service assessments will be most
valuable as management and reporting tools if • Customer comparisons of USAID service
they are carried out some months in advance of
delivery with that of other providers.
the operating unit’s annual planning and reporting process. For example, if a unit’s results re- Open-ended inquiry is especially well suited for
view and resources request (R4) report is to be addressing the first issue.The other two may be
completed by February, the customer service measured and analyzed quantitatively or qualiassessment might be conducted in November. tatively by consulting with ultimate or intermediate customers with respect to a number of
However, the precise scheduling and execution service delivery attributes or criteria important
of assessments is a task appropriate for those
responsible for results in a program sector—
members of the strategic objective or results
Box 2.
package team.
Illustrative Criteria For Assessing
Service Delivery
Step 2. Design the assessment.
Convenience. Ease of working with the
operating unit, simple processes, minimal
red tape, easy physical access to contacts
Depending on the scale of the effort, an operating unit may wish to develop a scope of work for
a customer service assessment. At a minimum,
4
Responsiveness. Follow up promptly, meet
changing needs, solve problems, answer questions, return calls
and program activity.
Reliability. On-time delivery that is thorough, accurate, complete
With its objective clearly in mind, and the information to be collected carefully specified, the
operating unit may decide in-house resources,
external assistance consultants, or a combination of the two, to conduct the assessment.
Step 3. Conduct the assessment.
Quality of products and services. Perform as intended; flexible in meeting local
needs; professionally qualified personnel
Contact personnel. Professional, knowledgable, understand local culture, language
skills
Select from a broad range of methods. A customer service assessment is not just a survey. It may
use a broad repertory of inquiry tools designed
to elicit information about the needs, preferences, or reactions of customers regarding a
USAID activity, product or service. Methods
may include the following:
to customer satisfaction (see Box 2).
• Formal customer surveys
In more formal surveys, for example, customers
may be asked to rate services and products on,
say, a 1-to-5 scale indicating their level of satisfaction with specific service characteristics or
attributes they consider important (e.g., quality,
reliability, responsiveness). In addition to rating
the actual services, customers may be asked
what they would consider “excellent” service,
referring to the same service attributes and using the same 5-point scale. Analysis of the gap
between what customers expect as an ideal
standard and what they perceive they actually
receive indicates the areas of service delivery
needing improvement.
• rapid appraisal methods (e.g., focus groups,
town meetings, interviews with key informants)
Breadth of choice. Sufficient choices to
meet customer needs and preferences
• Participatory appraisal techniques, in which
customers plan analyze, self-monitor, evaluate or set priorities for activities
• Document reviews, including systematic
use of social science research conducted by
others.
Use systematic research methods. A hastily prepared and executed effort does not provide
quality customer service assessment informaIn more qualitative approaches, such as focus tion. Sound social science methods are essengroups, customers discuss these issues among tial.
themselves while researchers listen carefully to
their perspectives. Operating units and teams Practice triangulation. To the extent resources
should design their customer assessments to and time permit, it is preferable to gather incollect customer feedback on service delivery formation from several sources and methods,
issues and attributes they believe are most im- rather than relying on just one. Such triangulaportant to achieving sustainable results toward tion will build confidence in findings and proa clearly defined strategic objective. These is- vide adequate depth of information for good
sues will vary with the nature of the objective decision-making and program management. In
5
particular, quantitative surveys and qualitative
studies often complement each other. Whereas
a quantitative survey can produce statistical
measurements of customer satisfaction (e.g.,
with quality, timeliness, or other aspects of a
program operation) that can be generalized
to a whole population, qualitative studies can
provide an in-depth understanding and insight
into customer perceptions and expectations on
these issues.
and encourage closer rapport with customers and partners. Moreover, they encourage a
more collaborative, participatory, and effective
approach to achievement of objectives.
Conduct assessments routinely. Customer service
assessments are designed to be consciously
iterative. In other words, they are undertaken
periodically to enable the operating unit to
build a foundation of findings over time to inform management of changing customer needs
and perceptions. Maintaining an outreach orientation will help the program adapt to changing
circumstances as reflected in customer views.
H. S. Plunkett and Elizabeth Baltimore, Customer
Focus Cookbook, USAID/M/ROR, August 1996.
Resource Manual for Customer Surveys. Statistical
Policy Office, Office of Management and Budget. October 1993.
Zeithaml, Valarie A; A. Parasuraman; and Leonard L.Berry. Delivering Quality Service. New York:
Free Press
Step 4. Broadly disseminate and use
assessment findings to improve performance.
Customer service assessments gain value when
broadly disseminated within the operating unit,
to other operating units active in similar program sectors, to partners, and more widely
within USAID. Sharing this information is also
important to maintaining open, transparent relations with customers themselves.
Assessment findings provide operating unit
managers with insight on what is important to
customers and how well the unit is delivering
its programs. They also can help identify operations that need quality improvement, provide
early detection of problems, and direct attention to areas where remedial action may be
taken to improve delivery of services.
Customer assessments form the basis for review of and recommitment to service principles. They enable measurement of service
delivery performance against service standards
6
NUMBER 10
2011 Printing
TIPS
CONDUCTING FOCUS GROUP INTERVIEWS
ABOUT TIPS
peroformance monitoring and evaluation. This publication is a supplemental reference to the
WHAT IS A FOCUS GROUP
INTERVIEW?
USAID’s guidelines encourage use of rapid, lowcost methods to collect
information on the
performance of
development assistance
activities.
A focus group interview is an inexpensive, rapid
appraisal technique that can provide managers with a wealth of qualitative information on
performance of development activities, services, and products, or other issues. A facilitator
guides 7 to 11 people in a discussion of their
experiences, feelings, and preferences about a
topic. The facilitator raises issues identified in
a discussion guide and uses probing techniques
to solicit views, ideas, and other information.
Sessions typically last one to two hours.
Focus group interviews,
the subject of this TIPS,
is one such method.
ADVANTAGES AND
LIMITATIONS
1
cannot be explained recommendations and
This technique has several advantages. It is low
suggestions are needed from customers,
cost and provides speedy results. Its flexible forpartners, experts, or other stakeholders
mat allows the facilitator to explore unanticipated issues and encourages interaction among
participants. In a group setting participants pro- For example, focus groups were used to unvide checks and balances, thus minimizing false cover problems in a Nepal family planning program where facilities were underutilized, and
or extreme views.
to obtain suggestions for improvements from
Focus groups have some limitations, however. customers.The focus groups revealed that rural
The flexible format makes it susceptible to fa- women considered family planning important.
cilitator bias, which can undermine the validity However, they did not use the clinics because of
and reliability of findings. Discussions can be caste system barriers and the demeaning mansidetracked or dominated by a few vocal individ- ner of clinic staff. Focus group participants suguals. Focus group interviews generate relevant gested appointing staff of the same social status
qualitative information, but no quantitative data to ensure that rural women were treated with
from which generalizations can be made for a respect. They also suggested that rural women
whole population. Moreover, the information disseminate information to their neighbors
can be difficult to analyze; comments should be about the health clinic.
interpreted in the context of the group setting.
Before deciding whether to use focus group interviews as a source of information, the study
WHEN ARE FOCUS GROUP purpose needs to be clarified. This requires
identifying who will use the information, deterINTERVIEWS USEFUL?
mining what information is needed, and understanding why the information is needed. Once
this is done, an appropriate methodology can
Focus group interviews can be useful in all phasbe selected. (See Tips 5 Using Rapid Appraisal
es of development activities— planning, impleMethods for additional information on selecting
mentation, monitoring, and evaluation. They can
appraisal techniques.)
be used to solicit views, insights, and recommendations of program staff, customers, stakeholders, technical experts, or other groups.
STEPS IN CONDUCTING
FOCUS GROUP
INTERVIEWS
They are especially appropriate when:
• program activities are being planned and it
is important for managers to understand
customers’ and other stakeholders’ attitudes, preferences or needs
Follow this step-by-step advice to help ensure
high-quality results.
• specific services or outreach approaches Step 1. Select the team
have to take into account customers’ prefConducting a focus group interview requires a
erences
small team, with at least a facilitator to guide
• major program implementation problems the discussion and a rapporteur to record it.
The facilitator should be a native speaker who
2
can put people at ease. The team should have Step 3. Decide on timing and location
substantive knowledge of the topic under discussion.
Discussions last one to two hours and should
be conducted in a convenient location with
Skills and experience in conducting focus some degree of privacy. Focus groups in a small
groups are also important. If the interviews village arouse curiosity and can result in uninare to be conducted by members of a broader vited participants. Open places are not good
evaluation team without previous experience spots for discussions.
in focus group techniques, training is suggested.
This training can take the form of role playing, Step 4. Prepare the discussion guide
formalized instruction on topic sequencing and
probing for generating and managing group dis- The discussion guide is an outline, prepared in
cussions, as well as pre-testing discussion guides advance, that covers the topics and issues to be
in pilot groups.
discussed. It should contain few items, allowing
some time and flexibility to pursue unanticipatStep 2. Select the participants
ed but relevant issues.
First, identify the types of groups and institutions that should be represented (such as program managers, customers, partners, technical experts, government officials) in the focus
groups. This will be determined by the informtion needs of the study. Often separate focus
groups are held for each type of group. Second,
identify the most suitable people in each group.
One of the best approaches is to consult key
informants who know about local conditions.
It is prudent to consult several informants to
minimize the biases of individual preferences.
Excerpt from a Discussion
Guide on Curative
Health Services
(20-30 minutes)
Q. Who treats/cures your children
when they get sick? Why?
Note: Look for opinions about
Each focus group should be 7 to 11 people to
allow the smooth flow of conversation.
• outcomes and results
• provider-user relations
• costs (consultations, transportation, medicine)
• waiting time
• physical aspects (privacy, cleanliness)
• availability of drugs, lab services
• access (distance, availability of
transportation)
• follow-up at home
Participants should be homogenous, from similar socioeconomic and cultural backgrounds.
They should share common traits related to the
discussion topic. For example, in a discussion
on contraceptive use, older and younger women should participate in separate focus groups.
Younger women may be reluctant to discuss
sexual behavior among their elders, especially if
it deviates from tradition. Ideally, people should
not know each other. Anonymity lowers inhibition and prevents formation of cliques.
3
The guide provides the framework for the facilitator to explore, probe, and ask questions.
Initiating each topic with a carefully crafted
question will help keep the discussion focused.
Using a guide also increases the comprehensiveness of the data and makes data collection
more efficient. Its flexibility, however can mean
that different focus groups are asked different
questions, reducing the credibility of the findings. An excerpt from a discussion guide used
in Bolivia to assess child survival services provides an illustration. (See box on page 3)
• What do you think about corruption in the
criminal justice system?
• How do you feel about the three parties
running in upcoming national elections?
Use probing techniques. When participants give
incomplete or irrelevant answers, the facilitator
can probe for fuller, clearer responses. A few
suggested techniques:
Repeat the question—repetition gives more time
to think
Step 5. Conduct the interview
Adopt sophisticated naivete” posture—convey
limited understanding of the issue and ask for
specific details
Establish rapport. Often participants do not
know what to expect from focus group discussions. It is helpful for the facilitator to outline
the purpose and format of the discussion at the
beginning of the session, and set the group at
ease. Participants should be told that the discussion is informal, everyone is expected to
participate, and divergent views are welcome.
Pause for the answer—a thoughtful nod or expectant look can convey that you want a fuller
answer
Repeat the reply—hearing it again sometimes
Phrase questions carefully. Certain types of ques- stimulates conversation. Ask when, what,
tions impede group discussions. For example, where, which, and how questions—they proyes-or-no questions are one dimensional and voke more detailed information
do not stimulate discussion. “Why” questions
put people on the defensive and cause them to Use neutral comments— Anything else?” Why do
take “politically correct” sides on controversial you feel this way?”
issues.
Control the discussion. In most groups a few indiOpen-ended questions are more useful be- viduals dominate the discussion. To balance out
cause they allow participants to tell their story participation:
in their own words and add details that can re• Address questions to individuals who are
sult in unanticipated findings. For example:
reluctant to talk
• What do you think about the criminal justice system?
• Give nonverbal cues (look in another direction or stop taking notes when an individual
• How do you feel about the upcoming natalks for an extended period)
tional elections?
• Intervene, politely summarize the point,
If the discussion is too broad the facilitator can
then refocus the discussion
narrow responses by asking such questions as:
4
• Take advantage of a pause and say, “Thank
you for that interesting idea, perhaps we can
discuss it in a separate session. Meanwhile
with your consent, I would like to move on
to another item.”
trends andpatterns, strongly held or frequently
aired opinions.
Read each transcript. Highlight sections that correspond to the discussion guide questions and
mark comments that could be used in the final
report.
Minimize group pressure. When an idea is being
adopted without any general discussion or disagreement, more than likely group pressure is
occurring. To minimize group pressure the facilitator can probe for alternate views. For example, the facilitator can raise another issue, or
say, “We had an interesting discussion but let’s
explore other alter natives.”
Analyze each question separately. After reviewing
all the responses to a question or topic, write a
summary statement that describes the discussion. In analyzing the results, the team should
consider:
• Words. Weigh the meaning of words participants used. Can a variety of words and
phrases categorize similar responses?
Step 6. Record the discussion
A rapporteur should perform this function.
Tape recordings in conjunction with written • Framework. Consider the circumstances in
notes are useful. Notes should be extensive
which a comment was made (context of
and reflect the content of the discussion as well
previous discussions, tone and intensity of
as nonverbal behavior (facial expressions, hand
the comment).
movements).
• Internal agreement. Figure out whether shifts
Shortly after each group interview, the team
in opinions during the discussion were
should summarize the information, the team’s
caused by group pressure.
impressions, and implications of the information for the study.
• Precision of responses. Decide which responses were based on personal experience and
Discussion should be reported in participants’
give them greater weight than those based
language, retaining their phrases and grammation vague impersonal impressions.
cal use. Summarizing or paraphrasing responses
can be misleading. For instance, a verbatim reply • The big picture. Pinpoint major ideas. Allo“Yes, indeed! I am positive,” loses its intensity
cate time to step back and reflect on major
when recorded as “Yes.”
findings.
Step 7. Analyze results
• Purpose of the report. Consider the objectives of the study and the information
needed for decisionmaking. The type and
After each session, the team should assemble
scope of reporting will guide the analytical
the interview notes (transcripts of each focus
process. For example, focus group reports
group interview), the summaries, and any other
typically are: (1) brief oral reports that highrelevant data to analyze trends and patterns.
light key findings; (2) descriptive reports
The following method can be used.
that summarize the discussion; and (3) analytical reports that provide trends, patterns,
Read summaries all at one time. Note potential
5
or findings and include selected comments.
Focus Group Interviews of Navarongo Community
Health and Family Planning Project in Ghana
The Ghanaian Ministry of Health launched a small pilot project in three villages
in 1994 to assess community reaction to family planning and elicit community
advice on program design and management. A new model of service deliverywas introduced: community health nurses were retrained as community health
officers living in the communities and providing village-based clinical services.
Focus group discussions were used to identify constraints to introducing family planning services and clarify ways to design operations that villagers value.
Discussions revealed that many women want more control over their ability
to reproduce, but believe their preferences are irrelevant to decisions made
in the male dominated lineage system. This indicated that outreach programs
aimed primarily at women are insufficient. Social groups must be included to
legitimize and support individuals’ family-planning decisions. Focus group discussions also revealed women’s concerns about the confidentiality of information and services. These findings preclude development of a conventional communitybased distribution program, since villagers clearly prefer outside service
delivery workers to those who are community members.
Krishna Kumar, Conducting Group Interviews in
Developing Countries, A.I.D. Program Design and
Evaluation Methodology Report No. 8, 1987
(PN-AAL-088)
Richard A. Krueger, Focus Groups: A Practical
Guide for Applied Research, Sage Publications,
1988
6
2009, NUMBER 12
2ND EDITION
TIPS
DATA QUALITY STANDARDS
ABOUT TIPS
WHY IS DATA
QUALITY
IMPORTANT?
Results-focused development
programming
requires
managers to design and
implement programs based
on evidence. Since data play a
central role in establishing
effective
performance
management systems, it is
essential to ensure good data
quality
(see
Figure
1).
Without this, decision makers
do not know whether to have
confidence in the data, or
worse, could make decisions
based on misleading data.
Attention to data quality
assists in:
Figure 1. Data Quality Plays a Central Role in Developing
Effective Performance Management Systems
Cycle:
Plan: Identify or Refine Key Program Objectives
Design: Develop or Refine the Performance Management Plan
Analyze Data
Use Data: Use Findings from Data Analysis to Improve Program Effectiveness
Data
Quality
 Ensuring
that
limited
development resources are
used as effectively as
possible
1
 Ensuring
that
Agency
program
and
budget
decisions in Washington
and the field are as well
informed
possible
as
practically
 Meeting the requirements
of
the
Government
Performance and Results
Act (GPRA)
 Reporting the impact of
USAID programs to external
stakeholders,
including
senior management, OMB,
the Congress, and the
public with confidence
DATA QUALITY
STANDARDS
Data quality is one element of
a
larger
interrelated
performance
management
system. Data quality flows
from a well designed and
logical strategic plan where
Assistance Objectives (AOs)
and Intermediate Results (IRs)
are clearly identified.
If a
result is poorly defined, it is
difficult to identify quality
indicators,
and
further,
without quality indicators, the
resulting data will often have
data quality problems.
One key challenge is to
determine what level of data
quality is acceptable (or “good
enough”) for management
purposes. It is important to
understand that we rarely
require the same degree of
rigor as needed in research or
for laboratory experiments.
Standards for data quality
must be keyed to our
intended use of the data. That
is, the level of accuracy,
currency,
precision,
and
reliability
of
performance
The Five Data Quality
Standards
1. Validity
2. Reliability
3. Precision
4. Integrity
5. Timeliness
information
should
be
consistent
with
the
requirements
of
good
management.
Determining
appropriate
or
adequate
thresholds of indicator and
data quality is not an exact
science.
This task is made
even more difficult by the
complicated and often datapoor development settings in
which USAID operates.
As
with
performance
indicators, we sometimes have
to consider trade-offs, or
make informed judgments,
when applying the standards
for data quality. This is
especially true if, as is often
the case, USAID relies on
others to provide data for
indicators. For example, if our
only existing source of data
for a critical economic growth
indicator is the Ministry of
Finance, and we know that the
Ministry’s
data
collection
methods are less than perfect,
we may have to weigh the
alternatives of relying on lessthan-ideal data, having no
data at all, or conducting a
potentially
costly
USAIDfunded
primary
data
collection effort. In this case,
2
a decision must be made as to
whether the Ministry’s data
would allow the Assistance
Objective team to have
confidence when assessing
program
performance
or
whether they are so flawed as
to be useless, or perhaps
misleading, in reporting and
managing for results.
The
main point is that managers
should not let the ideal drive
out the good.
1. VALIDITY
Validity refers to the extent to
which a measure actually
represents what we intend to
measure.1
Though simple in principle,
validity can be difficult to
assess in practice, particularly
when
measuring
social
phenomena.
For example,
how can we measure political
power or sustainability? Is the
poverty gap a good measure
of the extent of a country’s
poverty? However, even valid
indicators have little value, if
the data collected do not
correctly measure the variable
or characteristic encompassed
by the indicator. It is quite
possible, in other words, to
identify valid indicators but to
then
collect
inaccurate,
unrepresentative,
or
incomplete data.
In such
cases, the quality of the
indicator is moot. It would be
equally undesirable to collect
1
This criterion is closely related
to “directness” criteria for
indicators.
good data
indicator.
for
an
invalid
There are a number of ways to
organize or present concepts
related to data validity. In the
USAID context, we focus on
three key dimensions of
validity that are most often
relevant
to
development
programming, including: face
validity,
attribution,
and
measurement error.
FACE VALIDITY
Face validity means that an
outsider or an expert in the
field would agree that the
data is a true measure of the
result. For data to have high
face validity, the data must be
true representations of the
indicator, and the indicator
must be a valid measure of
the result. For example:
Result:
Increased
household income in a
target district
Indicator:
Value
of
median household income
in the target district
In this case, the indicator has a
high degree of face validity
when compared to the result.
That is, an external observer is
likely to agree that the data
measure
the
intended
objective. On the other hand,
consider
the
following
example:
Result:
Increased
target district
Indicator:
Number of
houses in the target
community with tin roofs
This example does not appear
to have a high degree of face
validity as a measure of
increased income, because it
is not immediately clear how
tin roofs are related to
increased income.
The
indicator above is a proxy
indicator
for
increased
income.
Proxy indicators
measure results indirectly, and
their validity hinges on the
assumptions made to relate
the indicator to the result. If
we assume that 1) household
income data are too costly to
obtain and 2) research shows
that when the poor have
increased income, they are
likely to spend it on tin roofs,
then this indicator could be an
appropriate
proxy
for
increased income.
ATTRIBUTION
Attribution focuses on the
extent to which a change in
the data is related to USAID
interventions. The concept of
attribution is discussed in
detail as a criterion for
indicator
selection,
but
reemerges when assessing
validity.
Attribution means
that changes in the data can
be plausibly associated with
USAID interventions.
For
example, an indicator that
measures changes at the
national level is not usually
appropriate for a program
targeting a few areas or a
particular segment of the
3
population.
following:
Consider the
Result:
revenues
in
municipalities.
Increased
targeted
Indicator: Number of
municipalities where tax
revenues have increased
by 5%.
In this case, assume that
increased
revenues
are
measured
among
all
municipalities
nationwide,
while the program only
focuses on a targeted group
of municipalities. This means
that the data would not be a
valid measure of performance
because the overall result is
not reasonably attributable to
program activities.
MEASUREMENT ERROR
Measurement error results
primarily from the poor
design or management of
data collection processes.
Examples include leading
questions,
unrepresentative
sampling,
or
inadequate
training of data collectors.
Even if data have high face
validity, they still might be an
inaccurate measure of our
result due to bias or error in
the measurement process.
Judgments about acceptable
measurement error should
reflect technical assessments
about what level of reductions
in measurement error are
possible and practical. This
can be assessed on the basis
of cost as well as management
judgments about what level of
accuracy
decisions.
is
needed
for
Some degree of measurement
error is inevitable, particularly
when dealing with social and
economic changes, but the
level of measurement error
associated
with
all
performance data collected or
used by operating units
should not be so large as to 1)
call into question either the
direction or degree of change
reflected by the data or 2)
overwhelm the amount of
anticipated change in an
indicator
(making
it
impossible for managers to
determine whether progress.
reflected in the data is a result
of actual change or of
measurement error). The two
main sources of measurement
error are sampling and nonsampling error.
Sampling Error (or
representativeness)
Data
are
said
to
be
representative
if
they
accurately
reflect
the
population they are intended
to
describe.
The
representativeness of data is a
function of the process used
to select a sample of the
population from which data
will be collected.
It is often not possible, or
even desirable, to collect data
from
every
individual,
household, or community
involved in a program due to
resource
or
practical
constraints. In these cases,
data are collected from a
sample to infer the status of
the population as a whole. If
we are interested in describing
the characteristics of a
country’s primary schools, for
example, we would not need
to examine every school in the
country. Depending on our
focus, a sample of a hundred
schools might be enough.
However, when the sample
used to collect data are not
representative
of
the
population as a whole,
significant
bias
can
be
introduced into the data. For
example, if we only use data
from 100 schools in the capital
area of the country, our data
will
not
likely
be
representative of all primary
schools in the country.
Drawing a sample that will
allow managers to confidently
generalize data/findings to
the population requires that
two basic criteria are met: 1)
that all units of a population
(e.g., households, schools,
enterprises) have an equal
chance of being selected for
the sample and 2) that the
sample is of adequate size.
The sample size necessary to
ensure that resulting data are
representative to any specified
degree can vary substantially,
depending on the unit of
analysis, the size of the
population, the variance of the
characteristics being tracked,
and
the
number
of
characteristics that we need to
analyze. Moreover, during
data collection it is rarely
possible to obtain data for
every member of an initially
4
chosen sample. Rather, there
are established techniques for
determining acceptable levels
of non-response or for
substituting new respondents.
If a sample is necessary, it is
important for managers to
consider the sample size and
method relative to the data
needs.
While data validity
should always be a concern,
there may be situations where
accuracy is a particular
priority. In these cases, it may
be useful to consult a
sampling expert to ensure the
data are representative.
Non-Sampling Error
Non-sampling error includes
poor design of the data
collection instrument, poorly
trained
or
partisan
enumerators, or the use of
questions (often related to
sensitive subjects) that elicit
incomplete
or
untruthful
answers from respondents.
Consider the earlier example:
Result: Increased
target district
Indicator: Value of
median household
income in the target
district
While these data appear to
have high face validity, there is
the potential for significant
measurement error through
reporting bias. If households
are asked about their income,
they might be tempted to
income
to
under-report
demonstrate the need for
additional assistance (or overreport
to
demonstrate
success). A similar type of
reporting bias may occur
when data is collected in
groups or with observers, as
respondents may modify their
responses to match group or
observer norms. This can be a
particular source of bias when
collecting data on vulnerable
groups. Likewise, survey or
interview
questions
and
sequencing
should
be
developed in a way that
minimizes the potential for
the leading of respondents to
predetermined responses. In
order to minimize nonsampling measurement error,
managers should carefully
plan and vet the data
collection process with a
careful eye towards potential
sources of bias.
Minimizing Measurement
Error
Keep in mind that USAID is
primarily
concerned
with
learning,
with
reasonable
confidence, that anticipated
improvements have occurred,
not with reducing error below
some arbitrary level. 2 Since it
is impossible to completely
eliminate measurement error,
and reducing error tends to
become
increasingly
expensive or difficult, it is
important to consider what an
2
For additional information, refer
to Common Problems/Issues with
Using Secondary Data in the CDIE
Resource Book on Strategic
Planning and Performance
Monitoring, April 1997.
acceptable level of error
would be.
Unfortunately,
there is no simple standard
that can be applied across all
of the data collected for
USAID’s varied programs and
results.
As
performance
management plans (PMPs) are
developed, teams should:
 Identify the existing or
potential sources of error
for each indicator and
document this in the PMP.
 Assess how this error
compares
with
the
magnitude of expected
change. If the anticipated
change is less than the
measurement error, then
the data are not valid.
 Decide whether alternative
data sources (or indicators)
need to be explored as
better alternatives or to
complement the data to
improve data validity.
2. RELIABILITY
Data should reflect stable and
consistent
data
collection
processes
and
analysis
methods over time.
Reliability is important so that
changes in data can be
recognized as true changes
rather than reflections of poor
or changed data collection
methods. For example, if we
use
a
thermometer
to
measure a child’s temperature
repeatedly and the results
vary from 95 to 105 degrees,
even though we know the
child’s temperature hasn’t
changed, the thermometer is
5
not a reliable instrument for
measuring fever.
In other
words, if a data collection
process is unreliable due to
changes in the data collection
instrument,
different
implementation across data
collectors, or poor question
choice, it will be difficult for
managers to determine if
changes in data over the life
of the project reflect true
changes or random error in
the data collection process.
Consider
the
following
examples:
Indicator: Percent
increase in income
among target
beneficiaries.
The first year, the project
reports
increased
total
income, including income as a
result of off-farm resources.
The second year a new
manager is responsible for
data collection, and only farm
based income is reported.
The third year, questions arise
as to how “farm based
income” is defined. In this
case, the reliability of the data
comes into question because
managers are not sure
whether changes in the data
are due to real change or
changes in definitions. The
following is another example:
Indicator: Increased
volume of agricultural
commodities sold by
farmers.
A scale is used to measure
volume
of
agricultural
commodities sold in the
What’s the Difference
Between Validity and
Reliability?
Validity refers to the
extent to which a
measure actually
represents what we
intend to measure.
Reliability refers to the
stability of the
measurement process.
That is, assuming there is
no real change in the
variable being measured,
would the same
measurement process
provide the same result if
the process were
repeated over and over?
market. The scale is jostled
around in the back of the
truck. As a result, it is no
longer properly calibrated at
each stop. Because of this,
the scale yields unreliable
data, and it is difficult for
managers
to
determine
whether changes in the data
truly reflect changes in
volume sold.
3. PRECISION
Precise data have a sufficient
level of detail to present a fair
picture of performance and
enable management decisionmaking.
The level of precision or detail
reflected in the data should be
smaller (or finer) than the
margin of error, or the tool of
measurement is considered
too imprecise. For some
indicators, for which the
magnitude
of
expected
change is large, even relatively
large measurement errors may
be perfectly tolerable; for
other
indicators,
small
amounts of change will be
important and even moderate
levels of measurement error
will be unacceptable.
Example: The number of
politically
active
nongovernmental organizations
(NGOs) is 900. Preliminary
data shows that after a few
years this had grown to
30,000 NGOs. In this case, a
10 percent measurement error
(+/- 3,000 NGOs) would be
essentially
irrelevant.
Similarly, it is not important to
know precisely whether there
are 29,999 or 30,001 NGOs. A
less precise level of detail is
still sufficient to be confident
in the magnitude of change.
Consider
an
alternative
scenario. If the second data
point is 1,000, a 10 percent
measurement error (+/- 100)
would
be
completely
unacceptable
because
it
would represent all of the
apparent change in the data.
4. INTEGRITY
Integrity focuses on whether
there is improper manipulation
of data.
Data that are collected,
analyzed and reported should
have established mechanisms
in
place
to
reduce
manipulation.
There are
generally two types of issues
that affect data integrity. The
first is transcription error. The
second, and somewhat more
complex issue, is whether
there is any incentive on the
6
part of the data source to
manipulate the data for
political or personal reasons.
Transcription Error
Transcription error refers to
simple data entry errors made
when transcribing data from
one document (electronic or
paper) or database to another.
Transcription
error
is
avoidable,
and
Missions
should seek to eliminate any
such error when producing
internal or external reports
and other documents. When
the data presented in a
document produced by an
operating unit are different
from the data (for the same
indicator and time frame)
presented in the original
source simply because of data
entry or copying mistakes, a
transcription
error
has
occurred.
Such differences
(unless due to rounding) can
be easily avoided by careful
cross-checking of data against
the original source. Rounding
may result in a slight
difference from the source
data but may be readily
justified when the underlying
data do not support such
specificity, or when the use of
the data does not benefit
materially from the originally
reported level of detail. (For
example, when making cost or
budget
projections,
we
typically
round
numbers.
When we make payments to
vendors, we do not round the
amount
paid
in
the
accounting ledger. Different
purposes can accept different
levels of specificity.)
Technology can help to
reduce transcription error.
Systems can be designed so
that the data source can enter
data directly into a database—
reducing the need to send in a
paper report that is then
entered into the system.
However, this requires access
to computers and reliable
internet services. Additionally,
databases can be developed
with internal consistency or
range checks to minimize
transcription errors.
The use of preliminary or
partial data should not be
confused with transcription
error. There are times, where
it makes sense to use partial
data (clearly identified as
preliminary or partial) to
inform management decisions
or to report on performance
because these are the best
data currently available. When
preliminary or partial data are
updated by the original
source, USAID should quickly
follow suit, and note that it
has done so. Any discrepancy
between preliminary data
included in a dated USAID
document and data that were
subsequently updated in an
original source does not
constitute transcription error.
Manipulation
A somewhat more complex
issue is whether data is
manipulated.
Manipulation
should be considered 1) if
there may be incentive on the
part of those that report data
to skew the data to benefit
the project or program and
managers suspect that this
may be a problem, 2) if
managers
believe
that
numbers
appear
to
be
unusually favorable, or 3) if
the data are of high value and
managers want to ensure the
integrity of the data.
There are a number of ways in
which managers can address
manipulation.
First, simply
understand the data collection
process. A well organized and
structured process is less likely
to be subject to manipulation
because each step in the
process is clearly documented
and handled in a standard
way. Second, be aware of
potential issues. If managers
have reason to believe that
data are manipulated, then
they should further explore
the issues. Managers can do
this by periodically spot
checking or verifying the data.
This establishes a principle
that the quality of the data is
important and helps to
determine
whether
manipulation is indeed a
problem.
If there is
substantial concern about this
issue,
managers
might
conduct a Data Quality
Assessment (DQA) for the AO,
IR, or specific data in question.
Example: A project assists
the Ministry of Water to
reduce
water
loss
for
agricultural use. The Ministry
reports key statistics on water
loss to the project. These
statistics are critical for the
Ministry, the project and
USAID to understand program
performance. Because of the
7
importance of the data, a
study is commissioned to
examine data quality and
more specifically whether
there is any tendency for the
data to be inflated. The study
finds that there is a very slight
tendency to inflate the data,
but it is within an acceptable
range.
5. TIMELINESS
Data should be available and
up to date enough to meet
management needs.
There are two key aspects of
timeliness. First, data must be
available frequently enough
to influence management
decision
making.
For
performance indicators for
which annual data collection is
not practical, operating units
will collect data regularly, but
at longer time intervals.
Second, data should be
current or, in other words,
sufficiently up to date to be
useful in decision-making. As
a general guideline, data
should lag no more than three
years.
Certainly, decisionmaking should be informed
by the most current data that
are
practically
available.
Frequently,
though,
data
obtained from a secondary
source, and at times even
USAID-funded primary data
collection,
will
reflect
substantial time lags between
initial data collection and final
analysis and publication. Many
of these time lags are
unavoidable,
even
if
considerable
additional
resources
were
to
be
expended.
Sometimes
preliminary estimates may be
obtainable, but they should be
clearly flagged as such and
replaced as soon as possible
as the final data become
available from the source.
The
following
example
demonstrates issues related to
timeliness:
Result: Primary school
attrition in a targeted
region reduced.
Indicator: Rate of
student attrition at
targeted schools.
In August 2009, the Ministry
of Education published full
enrollment analysis for the
2007 school year.
In this case, currency is a
problem because there is a 2
year time lag for these data.
While it is optimal to collect
and report data based on the
U.S. Government fiscal year,
there are often a number of
practical challenges in doing
so. We recognize that data
may come from preceding
calendar or fiscal years.
Moreover, data often measure
results for the specific point in
time that the data were
collected, not from September
to September, or December to
December.
Often the realities of the
recipient country context will
dictate the appropriate timing
of the data collection effort,
rather than the U.S. fiscal year.
For example, if agricultural
yields are at their peak in July,
then data collection efforts to
measure yields should be
conducted in July of each
year. Moreover, to the extent
that
USAID
relies
on
secondary data sources and
partners for data collection,
we may not be able to dictate
exact timing
ASSESSING DATA
QUALITY
Approaches and steps for how
to assess data quality are
discussed in more detail in
TIPS 18: Conducting Data
Quality Assessments. USAID
policy requires managers to
understand the strengths and
weaknesses of the data they
use on an on-going basis. In
addition, a Data Quality
Assessment (DQA) must be
conducted at least once every
3 years for those data
reported to Washington (ADS
203.3.5.2).
TIPS publications are available online at [insert website]
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including Gerry Britan
publication was updated by Michelle Adams-Matson of Management Systems International (MSI).
Tel: (202) 712-1158
gbritan@usaid.gov
8
NUMBER 13
2ND EDITION, 2010 DRAFT
TIPS
BUILDING A RESULTS FRAMEWORK
ABOUT TIPS
WHAT IS A RESULTS
FRAMEWORK?
The Results Framework (RF) is a
graphic representation of a
strategy to achieve a specific
objective that is grounded in
cause-and-effect logic.
The RF
includes the Assistance Objective
(AO) and Intermediate Results
(IRs), whether funded by USAID
or partners, necessary to achieve
the objective (see Figure 1 for an
example). The RF also includes
the critical assumptions that must
hold true for the strategy to
remain valid.
The
Results
represents
Framework
a development hypothesis or a
theory about how intended
change will occur. The RF shows
how the achievement of lower
level objectives (IRs) leads to the
achievement of the next higher
order of objectives, ultimately
resulting in the AO.
In short, a person looking at a
Results Framework should be
able to understand the basic
theory for how key program
objectives will be achieved. The
Results
Framework
is
an
important tool because it helps
managers identify and focus on
key objectives within a complex
development environment.
1
A RESULTS FRAMEWORK
INCLUDES:

An Assistance Objective (AO)

Intermediate Results (IR)

Hypothesized cause and
effect linkages

Critical Assumptions
WHY IS THE RESULTS
FRAMEWORK
IMPORTANT?
The development of a Results
Framework
represents
an
important first step in forming
the actual strategy. It facilitates
analytic thinking and helps
What’s the Difference
Between a Results Framework
and the Foreign Assistance
Framework (FAF)?
In one word, accountability. The
results framework identifies an
objective that a Mission or Office
will be held accountable for
achieving in a specific country or
program environment. The
Foreign Assistance Framework
outlines broad goals and
objectives (e.g. Peace and
Security) or, in other words,
programming categories.
Achievement of Mission or
Office AOs should contribute to
those broader FAF objectives.
program managers gain clarity
around
key
objectives.
Ultimately, it sets the foundation
not only for the strategy, but also
for numerous other management
and
planning
functions
downstream, including project
design, monitoring, evaluation,
and program management. To
summarize,
the
Results
Framework:
 Provides an opportunity to
build consensus and ownership
around shared objectives not
only among AO team members
but also, more broadly, with
host-country representatives,
partners, and stakeholders.
 Facilitates agreement with
other
actors
(such
as
USAID/Washington, other USG
entities, the host country, and
other donors) on the expected
results and resources necessary
to achieve those results. The
AO is the focal point of the
agreement
between
USAID/Washington and the
Mission. It is also the basis for
Assistance
Agreements
(formerly
called
Strategic
Objective
Assistance
Agreements).
 Functions as an effective
communication tool because it
succinctly captures the key
elements of a program’s intent
and content.
 Establishes the foundation to
design
monitoring
and
evaluation
systems.
Information from performance
monitoring and evaluation
systems should also inform the
development of new RFs.
 Identifies the objectives that
drive project design.
In order to be an effective tool, a
Results Framework should be
current. RFs should be revised
when 1) results are not achieved
or completed sooner than
expected, 2) critical assumptions
are no longer valid, 3) the
underlying development theory
must be modified, or 4) critical
problems with policy, operations,
or resources were not adequately
recognized.
KEY CONCEPTS
THE RESULTS FRAMEWORK
IS PART OF A BROADER
STRATEGY
While the Results Framework is
one of the core elements of a
strategy, it alone does not
constitute a complete strategy.
Typically it is complimented by
narrative that further describes
the thinking behind the RF, the
relationships
between
the
objectives, and the identification
of synergies. As a team develops
the RF, broader strategic issues
2
should be considered, including
the following:
 What has led the team to
propose
the
Results
Framework?
 What is strategic about what is
being proposed (that is, does it
reflect
a
comparative
advantage or a specific niche)?
 What are the main strategic
issues?
 What is different in the new
strategy when compared to the
old?
 What synergies emerge? How
are
cross-cutting
issues
addressed?
How can these
issues be tackled in project
level
planning
and
implementation?
THE UNDERPINNING OF THE
RESULTS FRAMEWORK
A good Results Framework is not
only based on logic. It draws on
analysis, standard theories in a
technical
sector,
and
the
expertise
of
on-the-ground
managers.
Supporting Analysis
Before developing a Results
Framework, the team should
determine what analysis exists
and what analysis must yet be
completed
to
construct
a
development hypothesis with a
reasonable level of confidence.
Evaluations
constitute
an
important source of analysis,
identify important lessons from
past programs, and may explore
the validity of causal linkages that
can be used to influence future
programming.
Analysis of past
performance monitoring data is
also an important source of
information.
FIGURE 2. SETTING THE CONTEXT
FOR PARTICIPATION
External Forces
(Host Country
Strategy)
Standard Sector Theories
Sectors, particularly those that
USAID has worked in for some
time, often identify a set of
common elements that constitute
theories for how to accomplish
certain
objectives.
These
common elements form a basic
―template‖ of sorts to consider in
developing an RF. For example,
democracy
and
governance
experts often refer to addressing
supply and demand.
Supply
represents
the
ability
of
government to play its role
effectively or provide effective
services. Demand represents the
ability of civil society to demand
or
advocate
for
change.
Education
generally
requires
improved quality in teaching and
curriculum,
community
engagement,
and
adequate
facilities. Health often requires
improved quality of services, as
well as access to -- and greater
awareness of – those services.
An understanding of these
common strategic elements is
useful because they lay out a
standard set of components that
a team must consider in
developing a good RF. Although,
not all of these elements will
apply to all countries in the same
way, they form a starting point to
inform the team’s thinking. As
the team makes decisions about
what (or what not) to address,
this becomes a part of the logic
The
―Fit‖
USAID Mission/
Vision
that is presented in the narrative.
Technical experts can assist teams
in understanding standard sector
theories. In addition, a number
of USAID publications outline
broader sector strategies or
provide guidance on how to
develop strategies in particular
technical areas1.
On-the-Ground Knowledge
and Experience
Program managers are an
important source of knowledge
on the unique program or incountry factors that should be
considered in the development of
the Results Framework. They are
best able to examine different
types of information, including
1
Examples include: Hansen,
Gary. 1996. Constituencies for
Reform: Strategic Approaches for
Donor-Supported Civic Advocacy
Groups or USAID. 2008. Securing
the Future: A Strategy for
Economic Growth.
3
Internal
Capacity
analyses and standard sector
theories, and tailor a strategy for
a specific country or program
environment.
PARTICIPATION AND
OWNERSHIP
Development
of
a
Results
Framework presents an important
opportunity for USAID to engage
its own teams, the host country,
civil society, other donors, and
other
partners
in
defining
program objectives. Experience
has shown that a Results
Framework built out of a
participatory process results in a
more effective strategy.
Recent donor commitments to
the Paris Declaration and the
Accra Agenda for Action reinforce
these points. USAID has agreed
to increase ownership, align
systems
with
country-led
strategies, use partner systems,
harmonize aid efforts, manage for
development
results,
and
establish mutual accountability.
Common questions include,
―how
do
we
manage
participation?‖ or ―how do we
avoid raising expectations that
we cannot meet?‖
One
approach for setting the context
for effective participation is to
simply set expectations with
participants before engaging in
strategic discussions. In essence,
USAID is looking for the
―strategic fit‖ (see Figure 2). That
is, USAID seeks the intersection
between what the host country
wants, what USAID is capable of
delivering, and the vision for the
program.
WHOLE-OF- GOVERNMENT
APPROACHES
Efforts are underway to institute
planning processes that take into
account the U.S. Government’s
overall approach in a particular
country.
A
whole-ofapproach
may
government
identify larger goals or objectives
to which many USG entities
contribute.
Essentially, those
objectives would be at a higher
level or above the level of
accountability of any one USG
agency alone. USAID Assistance
Objectives
should
clearly
contribute to those larger goals,
but also reflect what the USAID
Mission can be held accountable
for within a specified timeframe
and within budget parameters.
The
whole-of-government
approach may be reflected at a
lower level in the Results
Framework as well.
The RF
provides flexibility to include the
objectives of other
GUIDELINES FOR CONSTRUCTING AOs AND IRs
AOs and IRs should be:

Results Statements. AOs and IRs should express an outcome. In other words,
the results of actions, not the actions or processes themselves. For example,
the statement ―increased economic growth in targets sectors‖ is a result, while
the statement ―increased promotion of market-oriented policies‖ is more
process oriented.

Clear and Measurable. AOs and IRs should be stated clearly and precisely, and
in a way that can be objectively measured. For example, the statement
―increased ability of entrepreneurs to respond to an improved policy, legal,
and regulatory environment‖ is both ambiguous and subjective. How one
defines or measures ―ability to respond‖ to a changing policy environment is
unclear and open to different interpretations. A more precise and measurable
results statement in this case is ―increased level of investment.‖ It is true that
USAID often seeks results that are not easily quantified. In these cases, it is
critical to define what exactly is meant by key terms. For example, what is
meant by ―improved business environment‖? As this is discussed, appropriate
measures begin to emerge.

Unidimensional. AOs or IRs ideally consist of one clear overarching objective.
The Results Framework is intended to represent a discrete hypothesis with
cause-and-effect linkages. When too many dimensions are included, that
function is lost because lower level results do not really ―add up‖ to higher
level results. Unidimensional objectives permit a more straightforward
assessment of performance. For example, the statement ―healthier, better
educated, higher-income families‖ is an unacceptable multidimensional result
because it includes diverse components that may not be well-defined and
may be difficult to manage and measure. There are limited exceptions. It may
be appropriate for a result to contain more than one dimension when the
result is 1) achievable by a common set of mutually-reinforcing Intermediate
Results or 2) implemented in an integrated manner (ADS 201.3.8).
actors (whether other USG
entities, donors, the host country,
or other partners) where the
achievement of those objectives
are essential for USAID to achieve
its AO.
For example, if a
program achieves a specific
objective that contributes to
USAID’s AO, it should be
reflected as an IR.
This can
facilitate greater coordination of
efforts.
THE LINKAGE TO PROJECTS
The RF should form the
foundation for project planning.
4
Project teams may continue to
flesh out the Results Framework
in further detail or may use the
Logical Framework2. Either way,
all projects and activities should
be designed to accomplish the
AO and some combination of one
or more IRs.
2
The Logical Framework (or
logframe for short) is a project
design tool that complements the
Results Framework. It is also
based on cause-and-effect
linkages. For further information
reference ADS 201.3.11.8.
THE PROCESS FOR
DEVELOPING A
RESULTS
FRAMEWORK
SETTING UP THE PROCESS
Missions may use a variety of
approaches to develop their
respective results frameworks. In
setting up the process, consider
the following three questions.
When
should
the
results
frameworks be developed? It is
often helpful to think about a
point in time at which the team
will have enough analysis and
information
to
confidently
construct a results framework.
Who is going to participate
(and at what points in the
process)?
It is important to
develop a schedule and plan out
the process for engaging partners
and stakeholders. There are a
number of options (or a
combination) that might be
considered:
 Invite
key
partners
or
stakeholders
to
results
framework
development
sessions. If this is done, it may
be useful to incorporate some
training
on
the
results
framework methodology in
advance. Figure 3 outlines the
basic building blocks and
defines terms used in strategic
planning
across
different
organizations.
 The AO team may develop a
preliminary results framework
and hold sessions with key
counterparts to present the
draft strategy and obtain
feedback.
 Conduct a strategy workshop
for AO teams to present out
RFs and discuss strategic issues.
Although these options require
some time and effort, the results
framework will be more complete
and representative.
What process and approach
will be used to develop the
results frameworks?
We
strongly recommend that the AO
team hold group sessions to
construct the results framework.
It is often helpful to have one
person
(preferably
with
experience in strategic planning
and facilitation) to lead these
sessions.
This person should
focus on drawing out the ideas of
the group and translating them
into the results framework.
STEP 1. IDENTIFY THE
ASSISTANCE OBJECTIVE
The Assistance Objective (AO) is
the center point for any results
framework and is defined as:
The most ambitious result
(intended measurable change)
that a USAID Mission/Office,
along with its partners, can
materially affect, and for which
it is willing to be held
accountable (ADS 201.3.8).
Defining an AO at an appropriate
level of impact is one of the most
critical and difficult tasks a team
faces.
The AO forms the
5
―It is critical to stress the importance
of not rushing to finalize a results
framework. It is necessary to take
time for the process to mature and to
be truly participative.‖
—USAID staff member in Africa
standard by which the Mission or
Office is willing to be judged in
terms of its performance.
The
concept of ―managing for results‖
(a USAID value also reflected in
the Paris Declaration) is premised
on this idea.
The task can be challenging,
because an AO should reflect a
balance of two conflicting
and
considerations—ambition
accountability. On the one hand,
every team wants to deliver
significant impact for a given
investment. On the other hand,
there are a number of factors
outside the control of the team.
In fact, as one moves up the
Results Framework toward the
AO, USAID is more dependent on
other development partners to
achieve the result.
Identifying an appropriate level
of ambition for an AO depends
on a number of factors and will
be different for each country
context. For example, in one
country it may be appropriate for
the AO to be ―increased use of
family planning methods‖ while
in another, ―decreased total
fertility‖ (a higher level objective)
would be more suitable. Where
to set the objective is influenced
by the following factors:
 Programming
history.
There
are
different
expectations for more
mature programs, where
higher level impacts and
greater sustainability are
expected.
 The magnitude of the
development problem.
 The timeframe
strategy.
for
Figure 3. Results Framework Logic
So What?
Necessary
and
Sufficient
the
 The
range
of
resources
available or expected.
The AO should represent the
team’s best assessment of what
can realistically be achieved. In
other words, the AO team should
be able to make a plausible case
that the appropriate analysis has
been done and the likelihood of
success is great enough to
warrant investing resources in the
AO.
STEP 2. IDENTIFY
INTERMEDIATE RESULTS
After agreeing on the AO, the
team must identify the set of
―lower level‖ Intermediate Results
necessary to achieve the AO. An
Intermediate Result is defined as:
An important result that is
seen as an essential step to
achieving a final result or
outcome.
IRs
are
How?
measurable results that may
capture a number of
discrete and more specific
results (ADS 201.3.8.4).
As the team moves down from
the AO to IRs, it is useful to ask
―how‖ can the AO be achieved?
By answering this question, the
team begins to formulate the IRs
(see Figure 3). The team should
assess relevant country and
sector conditions and draw on
development experience in other
countries to better understand
the changes that must occur if
the AO is to be attained.
The
Results
Framework
methodology
is
sufficiently
flexible to allow the AO team to
include Intermediate Results that
are supported by other actors
when they are relevant and
critical to achieving the AO. For
example, if another donor is
6
building schools that are
essential for USAID to
accomplish an education AO
(e.g.
increased
primary
school completion), then
that should be reflected as
an IR because it is a
necessary ingredient for
success.
Initially, the AO team might
identify a large number of
possible results relevant to
the AO.
However, it is
important to eventually settle on
the critical set of Intermediate
Results. There is no set number
for how many IRs (or levels of IRs)
are appropriate. The number of
Intermediate Results will vary
with the scope and complexity of
the AO. Eventually, the team
should arrive at a final set of IRs
that
members
believe
are
reasonable. It is customary for
USAID Missions to submit a
Results Framework with one or
two
levels
of
IRs
to
USAID/Washington for review.
The key point is that there should
be enough information to
adequately
convey
the
development hypothesis.
So What is Causal Logic Anyway?
Causal logic is based on the concept of cause-and-effect. That is, the accomplishment of lower-level
objectives ―cause‖ the next higher-level objective (or the effect) to occur. In the following example, the
hypothesis is that if IR 1, 2, and 3 occur, it will lead to the AO.
AO: Increased
Completion of
Primary School
IR 1: Improved
Quality of
Teaching
STEP 3. CLARIFY THE
RESULTS FRAMEWORK
LOGIC
Through
the
process
of
identifying Intermediate Results,
the team begins to construct the
cause-and-effect logic that is
central to the Results Framework.
Once the team has identified the
Intermediate Results that support
an objective, it must review and
confirm this logic.
The accomplishment of lower
level results, taken as a group,
should result in the achievement
of the next higher objective. As
the team moves up the Results
Framework, they should ask, ―so
what?‖ If we accomplish these
lower
level
objectives,
is
something
of
significance
achieved at the next higher level?
The
higher-order
result
establishes the ―lens‖ through
which lower-level results are
viewed. For example, if one IR is
―Increased Opportunities for Outof-School Youth to Acquire Life
Skills,‖ then, by definition, all
lower level IRs would focus on
IR 2: Improved
Curriculum
IR 3: Increased
Parental
Commitment to
Education
the target population established
(out-of-school youth).
As the team looks across the
Results Framework, it should ask
whether the Intermediate Results
are necessary and sufficient to
achieve the AO.
Results Framework logic is not
always linear.
There may be
relationships across results or
even with other AOs. This can
sometimes be demonstrated on
the graphic (e.g., through the use
of arrows or dotted boxes with
some explanation) or simply in
the narrative. In some cases,
teams find a number of causal
connections in an RF. However,
teams have to find a balance
between the two extremes- on
the one hand, where logic is too
simple and linear and, on the
other, a situation where all
objectives are related to all
others.
STEP 4. IDENTIFY CRITICAL
ASSUMPTIONS
The next step is to identify the set
of critical assumptions that are
relevant to the achievement of
7
the AO. A critical assumption is
defined as:
―….a general condition under
which
the
development
hypothesis will hold true.
Critical
assumptions
are
outside
the
control
or
influence of USAID and its
partners (in other words, they
are not results), but they
reflect conditions that are
likely to affect the achievement
of results in the Results
Framework.
Critical
assumptions may also be
expressed
as
risks
or
vulnerabilities…‖
(ADS
201.3.8.3)
Identifying critical assumptions,
assessing associated risks, and
determining how they should be
addressed is a part of the
strategic
planning
process.
Assessing risk is a matter of
balancing the likelihood that the
critical assumption will hold true
with the ability of the team to
address the issue. For example,
consider the critical assumption
―adequate rainfall.‖
If this
assumption has held true for the
What is NOT Causal Logic?
Categorical Logic. Lower level results are simply sub-categories rather than cause and effect, as
demonstrated in the example below.
AO: Increased
Completion of
Primary School
IR 1: Improved
Pre-Primary
School
IR 2: Improved
Primary
Education
IR 3: Improved
Secondary
Education
Definitional Logic. Lower-level results are a restatement (or further definition) of a higher-level objective.
The use of definitional logic results in a problem later when identifying performance indicators because it is
difficult to differentiate indicators at each level.
IR: Strengthened
Institution
IR: Institutional
Capacity to Deliver
Goods & Services
target region only two of the past
six years, the risk associated with
this assumption is so great that it
poses a risk to the strategy.
the years when a drought may
occur.
In cases like this, the AO team
should attempt to identify ways
to actively address the problem.
For example, the team might
include efforts to improve water
storage or irrigation methods, or
increase use of drought-resistant
seeds or farming techniques.
This would then become an IR (a
specific
objective
to
be
accomplished by the program)
rather than a critical assumption.
Another option for the team is to
develop contingency plans for
As a final step, the AO team
should step back from the Results
Framework and review it as a
whole. The RF should be
straightforward
and
understandable. Check that the
results contained in the RF are
measurable and feasible with
anticipated USAID and partner
resource levels. This is also a
good point at which to identify
synergies between objectives and
across AOs.
STEP 5. COMPLETE THE
RESULTS FRAMEWORK
8
STEP 6. IDENTIFY
PRELIMINARY
PERFORMANCE MEASURES
Agency policies (ADS 201.3.8.6)
require that the AO team present
proposed indicators for the AO
with baseline data and targets.
The AO, along with indicators and
targets, represents the specific
results that will be achieved vis-avis the investment. To the extent
possible, indicators for IRs with
baseline and targets should be
included as well.
1.
Figure 1. Illustrative Results Framework
AO:
Increased
Production by
Farmers in the
Upper River Zone
IR:
Farmers’ Access to
Commercial
Capital Increased
IR: Farmers’
Capacity to
Develop Bank
Loan Applications
Increased
(4 years)
IR: Banks’ Loan
Policies Become
More Favorable
for the Rural
Sector
(3 years)
2.
3.
4.
Critical Assumptions
Market prices for farmers’ products remain stable
or increase.
Prices of agricultural inputs remain stable or
decrease.
Roads needed to get produce to market are
maintained.
Rainfall and other critical weather conditions
remain stable.
IR:
Farmers’ Transport
Costs Decreased
IR: Additional
Local Wholesale
Market Facilities
Constructed (with
the World Bank)
IR:
Farmers’
Knowledge About
Effective
Production
Methods
Increased
IR: Village
Associations
Capacity to
Negotiate
Contracts
Increased (4 years)
IR: New
Technologies
Available
(World Bank)
(
Key
USAID
Responsible
Partner(s)(4
Responsible
9
USAID +
Partner(s)
Responsible
IR: Farmers’
Exposure to OnFarm Experiences
of Peers Increased
Figure 3. The Fundamental Building Blocks for Planning
ASSISTANCE OBJECTIVE (AO)
AO
The highest level objective for which USAID is
willing to be held accountable. AOs may also
be referred to as outcomes, impacts, or results.
Increased Primary School Completion
INTERMEDIATE RESULTS (IRs)
IR
Interim events, occurrences, or conditions that
are essential for achieving the AO. IRs may
also be referred to as outcomes or results.
Teaching Skills Improved
OUTPUT
OUTPUT
Products or services produced as a result of
internal activity.
Number of teachers trained
INPUT
INPUT
Resources used to produce an output.
Funding or person days of training
10
Figure 4. Sample Results Framework and Crosswalk of FAF Program Hierarchy and a
Results Framework
F Program
Hierarchy for
Budgeting and
Reporting
Illustrative Results Framework for
Program Planning
Assistance Objective: Economic Competitiveness of
Private Enterprises Improved
IR 1: Enabling Environment for
Enterprises Improved
IR 1.1 Licensing
and registration
requirements for
enterprises
streamlined
The Illustrative Results Framework
links to the FAF Program
Hierarchy as follows:
• Objective 4 Economic Growth
• Program Areas 4.6 (Private Sector
•
•
•
•
•
•
•
Competitiveness) and 4.7
(Economic Opportunity
Program Elements 4.6.1, 4.6.2, 4.7
Sub-Elements 4.6.12 and 4.7.2.1
Sub-Element 4.6.1.3
Sub-Element 4.7.2.2
Sub-Element 4.6.2.1
Sub-Element 4.7.3
Sub-Element 4.6.2.4
IR 1.3
Regulatory
environment for
micro and small
enterprises
improved
IR 1.2
Commercial laws
that support
market-oriented
transactions
promoted
IR 2: Private Sector
Capacity Strengthened
IR 2.1
Competitiveness
of targeted
enterprises
improved
IR 2.2
Productivity of
microenterprises in
targeted
geographic
regions
increased
Critical Assumptions:
•
•
Key political leaders, including the President and the
Minister of Trade and Labor, will continue to support
policy reforms that advance private enterprise-led
growth.
Government will sign the Libonia Free Trade
Agreement, which will open up opportunities for
enterprises targeted under IR 2.1.
Note: The arrows demonstrate the linkage of AO1, IR 1, and IR 1.1 to the FAF. As an example, IR1 links to the program element 4.6.1
“Business Enabling Environment”. IR 1.1 links to 4.7.2.1 “Reduce Barriers to Registering Micro and Small Business”.
11
IR 2.3
Information
Exchange
Improved
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including Gerry Britan
publication was updated by Michelle Adams-Matson, of Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
12
NUMBER 15
2011 Printing
TIPS
MEASURING INSTITUTIONAL CAPACITY
ABOUT TIPS
peroformance monitoring and evaluation. This publication is a supplemental reference to the
What are the strengths and limitations of each
approach with regard to internal bias, quantification, or comparability over time or across
organizations?
INTRODUCTION
This PME Tips gives USAID managers information on measuring institutional capacity,* including some tools that measure the capacity of an
entire organization as well as others that look
at individual components or functions of an organization. The discussion concentrates on the
internal capacities of individual organizations,
rather than on the entire institutional context
in which organizations function. This Tips is not
about how to actually strengthen an institution, nor is it about how to assess the eventual
impact of an organization’s work. Rather, it is
limited to a specific topic: how to measure an
institution’s capacities.
How will the data be collected and how participatory can and should the measurement process be?
Measuring institutional capacity might be one
important aspect of a broader program in institutional strengthening; it may help managers
make strategic, operational, or funding decisions;
or it may help explain institutional strengthening activities and related performance.
It addresses the following questions:
Whatever the reason for assessing institutional
capacity, this Tips presents managers with several tools for identifying institutional strengths
and weaknesses.
Which measurement approaches are most
useful for particular types of capacity building?
The paper will define and discuss capacity assessment in general and present several ap1
able to accomplish their mission and provide
for their own needs in the long run.

USAID operating units build capacity with a broad
spectrum of partner and customer organizations.
These include but are not limited to:
proaches for measuring institutional capacity.
We assess the measurement features of each
approach to help USAID managers select the
tool that best fits their diverse management
and reporting needs. The paper is organized as
follows:
1. Background: Institutional Capacity • American private voluntary organizations
Building and USAID
(PVOs)
2. How to Measure Institutional Capacity
• Local and international nongovernmental
organizations (NGOs) and other civil society organizations (CSOs)
3. Measurement Issues
4. Institutional Assessment Tools
• Community-based membership cooperatives, such as a water users group
5. Measuring Individual Organizational
Components
• Networks and associations of organizations
6. Developing Indicators
• Political parties
7. Practical Tips for a Busy USAID Manager
• Government entities (ministries, departments, agencies, subunits, policy analysis
units, health clinics, schools)
BACKGROUND:
INSTITUTIONAL CAPACITY
BUILDING AND USAID
• Private sector organizations (financial institutions, companies, small businesses and
other forprofit organizations)
• Regional institutions
USAID operating units must work closely with
partner and customer organizations to meet
program objectives across all Agency goal areas, among them Peace and Security, Governing
Justly and Democratically, Economic Growth,
Investing in People, and Humanitarian Assistance. In the course of planning, implementing,
and measuring their programs, USAID managers often find that a partner or customer organization’s lack of capacity stands in the way
of achieving results. Increasing the capacity of
partner and customer organizations helps them
carry out their mandate effectively and function
more efficiently. Strong organizations are more
The Agency uses a variety of techniques to
build organizational capacity. The most common involve providing technical assistance, advisory services, and long-term consultants to
organizations, to help them build the skills and
experience necessary to contribute successfully to sustainable development. Other techniques include providing direct inputs, such as
financial, human, and technological resources.
Finally, USAID helps establish mentoring relationships; provides opportunities for formal
study in-country, in the United States or in
third countries; and it sets up internships or
2
apprenticeships with other organizations. The
goal of strengthening an institution is usually to
improve the organization’s overall performance
and viability by improving administrative and
management functions, increasing the effectiveness of service provision, enhancing the organization’s structure and culture, and furthering its sustainability. Institutional strengthening
programs may address one or more of these
components.
the capacity of an organization to help make
decisions about awarding grants or holding
grantees accountable for results. In this case,
the assessment is more of an external oversight/audit of an organization hired to carry out
Agency programs. Or, the manager may have a
programmatic commitment to strengthen the
abilities of customer and partner organizations.
Different tools and methods are available for
both situations. This paper deals primarily with
programs that fit the latter description.
In most cases, USAID managers are concerned
with institutional strengthening because they
are interested in the eventual program-level results (and the sustainability of these results) that
these stronger organizations can help achieve.
While recognizing the need to address eventual results, this Tips looks primarily at ways to
measure institutional capacity. Understanding
and measuring institutional capacity are critical
and often more complex than measuring the
services and products an organization delivers.
Within USAID, the former Office of Private and
Voluntary Cooperation (PVC) took the lead on
building the capacity of nongovernmental organization (NGO) and private voluntary organization (PVO) partners. PVC has defined development objectives and intermediate results aimed
specifically at improving the internal capacity
of U.S. PVOs. PVC has studied different approaches to institutional capacity building and
has begun to develop a comprehensive capacity assessment tool called discussion-oriented
Measuring organizational capacity is important organizational self-assessment, described in exbecause it both guides USAID interventions ample 1 in this paper. In addition to DOSA, PVC
and allows managers to demonstrate and re- has developed several indicators for measuring
port on progress. The data that emerge from institutional capacity development.
measuring institutional capacity are commonly
used in a number of valuable ways. These data PVC specifically targets NGOs and PVOs
establish baselines and provide the basis for and is particularly concerned with enhancsetting targets for improvements. They help ex- ing partnerships. USAID missions, by contrast,
plain where or why something is going wrong; work with a broader range of organizations
they identify changes to specific program in- on activities aimed at increasing institutional
terventions and activities that address areas of capacity. Such programs usually view instipoor performance; they inform managers of the tutional capacity as a means to achieve highimpact of an intervention or the effectiveness er level program results, rather than as an
of an intervention strategy; and they identify end in itself.
lessons learned.They are also useful for reporting to Washington and to partners.
HOW TO MEASURE
INSTITUTIONAL CAPACITY
It is important to note the difference between
assessing capacity for contracting and grantmaking decisions versus for a “capacity build- An organization can be thought of as a system
ing” relationship with partner/customer organi- of related components that work together to
zations. A USAID manager may want to assess achieve an agreed-upon mission. The follow3
ing list of organizational components is not
all-inclusive, nor does it apply universally to
all organizations. Rather, the components are
representative of most organizations involved
in development work and will vary according
to the type of organization and the context in
which it functions.
• External relations
Administrative and Support Functions
• Other
Resources
• Human
• Financial
• Administrative procedures and management systems
MANAGEMENT ISSUES
• Financial management (budgeting, accounting, fundraising, sustainability)
This TIPS presents capacity-assessment tools
and other measurement approaches that, while
similar in some ways, vary in both their emphasis and their method for evaluating an organization’s capacity. Some use scoring systems and
others don’t; some use questionnaires while
others employ focus groups; some use external evaluators , and others use selfassessments;
some emphasize problem solving, while others concentrate on appreciating organzational
strengths. Some tools can be used to measure
the same standard across many organizations,
while others are organization specific. Many of
the tools are designed so that the measurement
process is just as important as, if not more important than, the resulting information. They
may involve group discussions, workshops, or
exercises, and may explicitly attempt to be participatory. Such tools try to create a learning
opportunity for the organization’s members, so
that the assessment itself becomes an integral
part of the capacity-building effort.
• Human resource management (staff recruitment, placement, support)
• Management of other resources (information, equipment, infrastructure)
Technical/Program Functions
• Service delivery system
• Program planning
• Program monitoring and evaluation
• Use and management of technical knowledge and skills
Structure and Culture
• Organizational identity and culture
• Vision and purpose
Because of each user’s different needs, it would
be difficult to use this TIPS as a screen to predetermine the best capacity-assessment tool for
each situation. Rather, managers are encouraged to adopt the approaches most appropriate
to their program and to adapt the tools best
suited for local needs. To assist managers in
identifying the most useful tools and approach-
• Leadership capacity and style
• Organizational values
• Governance approach
4
es, we consider the following issues for each of
the tools presented:
methods are hands-on and highly participatory, involving a wide range of customers,
partners, and stakeholders, while others
are more exclusive, relying on the opinion
of one or two specialists. In most cases, it
is best to use more than one data collection method.
• Type of organization measured. Many
of the instruments developed to measure
institutional capacity are designed specifically for measuring NGOs and PVOs. Most
of these can be adapted easily for use with
other types of organizations, including government entities.
• Objectivity. By their nature, measures of
institutional capacity are subjective. They
rely heavily on individual perception, judgment, and interpretation. Some tools are
better than others at limiting this subjectivity. For instance, they balance perceptions with more empirical observations, or
they clearly define the capacity area being
measured and the criteria against which
it is being judged. Nevertheless, users of
these tools should be aware of the limitations to the findings.
• Comparability across organizations.
To measure multiple organizations, to compare them with each other, or to aggregate
the results of activities aimed at strengthening more than one organization, the tool
used should measure the same capacity
areas for all the organizations and use the
same scoring criteria and measurement
processes. Note, however, that a standard
tool, applied to diverse organizations, is
less able to respond to specific organizational or environmental circumstances. This
is less of a problem if a group of organizations, using the same standard tool, has
designed its diagnostic instrument together
(see the following discussion of PROSE).
• Quantification. Using numbers to represent capacity can be helpful when they
are recognized as relative and not absolute
measures. Many tools for measuring institutional capacity rely on ordinal scales.
Ordinal scales are scales in which values
can be ranked from high to low or more
to less in relation to each other. They are
useful in ordering by rank along a continuum, but they can also be misleading.
Despite the use of scoring criteria and
guidelines, one person’s “3” may be someone else’s “4.” In addition, ordinal scales
do not indicate how far apart one score is
from another. (For example, is the distance
between “agree” and “strongly agree” the
same as the distance between “disagree”
and “strongly disagree”?) Qualitative
descriptions of an organization’s capacity
level are a good complement to ordinal
scales.
• Comparability over time. In many cases, the value of measuring institutional capacity lies in the ability to track changes in
one organization over time. That requires
consistency in method and approach. A
measurement instrument, once selected
and adapted to the needs of a particular
organization, must be applied the same way
each time it is used. Otherwise, any shifts
that are noted may reflect a change in the
measurement technique rather than an
actual change in the organization.
• Data collection. Data can be collected
in a variety of ways: questionnaires, focus
groups, interviews, document searches, and
observation, to name only some. Some
• Internal versus external assessments.
Some tools require the use of external
5
facilitators or assessors; others offer a
process that the organization itself can
follow. Both methods can produce useful
data, and neither is automatically better
than the other. Internal assessments can
facilitate increased management use and
better understanding of an assessment’s
findings, since the members of the organization themselves are carrying out the
assessment. By contrast, the risk of bias
and subjectivity is higher in internal assessments. External assessments may be more
objective. They are less likely to introduce
internal bias and can make use of external
expertise. The downside is that external
assessors may be less likely to u cover
what is really going on inside an organization.
same format.
• Practicality. The best measurement
systems are designed to be as simple as
possible-- not too time consuming, not unreasonably costly, yet able to provide managers with good information often enough
to meet their management needs. Managers should take practicality into account
when selecting a measurement tool. They
should consider the level of effort and
resources required to develop the instrument and collect and analyze the data, and
think about how often and at what point
during the management cycle the data will
be available to managers.
Background
• Background of the methodology/tool
• Process (how the methodology/tool is used
in the field)
• Product (the types of outputs expected)
• Assessment (a discussion of the uses and
relative strengths of each methodology/
tool)
• An example of what the methodology/tool
looks like
PARTICIPATORY, RESULTS-ORIENTED
SELF-EVALUATION
The participatory, results-oriented self-evaluation (PROSE) method was developed by Evan
Bloom of Pact and Beryl Levinger of the Education Development Center. It has the dual
purpose of both assessing and enhancing organizational capacities. The PROSE method produces an assessment tool customized to the
organizations being measured. It is designed to
compare capacities across a set of peer organizations, called a cohort group, which allows
for benchmarking and networking among the
organizations. PROSE tools measure and profile
organizational capacities and assess, over time,
how strengthening activities affect organizational capacity. In addition, through a facilitated
workshop, PROSE tools are designed to allow
organizations to build staff capacity; create consensus around future organizational capacitybuilding activities; and select, implement, and
track organizational change and development
strategies.
INSTITUTIONAL
ASSESMENT TOOLS
This section describes capacity measurement
tools that USAID and other development organizations use. You can find complete references
and Web sites in the resources section at the One example of an instrument developed using
end of the paper. For each tool, we follow the the PROSE method is the discussion-oriented
6
organizational self-assessment. DOSA was developed in 1997 for the Office of Private and
Voluntary Cooperation and was designed specifically for a cohort of USAID PVO grantees.
Participatory, Results-Oriented
Self-Evaluation
Type of Organization Measured
Process
NGOs/PVOs; adaptable to other types of organizations
Developers of the PROSE method recommend
that organizations participate in DOSA or develop a customized DOSA-like tool to better fit
their organization’s specific circumstances. The
general PROSE process for developing such a
tool is as follows: After a cohort group of organizations is defined, the organizations meet in
a workshop setting to design the assessment
tool. With the help of a facilitator, they begin by
pointing to the critical organizational capacities
they want to measure and enhance. The cohort
group then develops two sets of questions: discussion questions and individual questionnaire
items. The discussion questions are designed to
get the group thinking about key issues. Further,
these structured discussion questions minimize
bias by pointing assessment team members toward a common set of events, policies, or conditions. The questionnaire items then capture
group members’ assessments of those issues
on an ordinal scale. During the workshop, both
sets of questions are revised until the cohort
group is satisfied. Near the end of the process,
tools or standards from similar organizations
can be introduced to check the cohort group’s
work against an external example. If the tool
is expected to compare several organizations
within the same cohort group, the tool must be
implemented by facilitators trained to administer it effectively and consistently across the
organizations.
Features
• Cross-organizational comparisons can be
made
• Measures change in one organization or a
cohort of organizations over time
• Measures well-defined capacity areas
against well-defined criteria
• Assessment based primarily upon perceived capacities
• Produces numeric score on capacity areas
• Assessment should be done with the help
of an outside facilitator or trained insider
• Data collected through group discussion
and individual questionnaires given to a
cross-section of the organization’s staff
team meets for four to six hours and should
represent a cross-functional, crosshierarchical
sample from the organization. Participants respond anonymously to a questionnaire, selecting the best response to statements about the
organization’s practices (1=strongly disagree,
2=disagree, 3=neutral, 4=agree, 5=strongly
Once the instrument is designed, it is applied
agree) in six capacity areas:
to each of the organizations in the cohort. In
the case of DOSA, the facilitator leads a team
• External Relations
of the organization’s members through a series
(constituency development, fund-raising
of group discussions interspersed with individand communications)
ual responses to 100 questionnaire items. The
7
Example 1. Excerpt From DOSA, a PROSE Tool
The DOSA questionnaire can be found in annex 1a
The following is a brief example drawn from the Human Resource Management section of the
DOSA questionnaire:
Discussion Questions
a. When was our most recent staff training?
b. How often over the last 12 months have we held staff training events?
Questionnaire items for individual response
Strongly
Disagress
Neutral
Agree
Strongly
Disagree
Agree
1. We routinely offer
1
2
3
4
5
staff training.
Discussion Questions
a. What are three primary, ongoing functions (e.g., monitoring and evaluation, proposal writing, resource mobilization) that we carry out to achieve our mission?
b. To what extent does staff, as a group, have the requisite skills to carry out these functions?
c. To what extent is the number of employees carrying out these functions commensurate
with work demands?
Questionnaire items for individual response
Strongly
Disagress
Neutral
Disagree
2. We have the ap1
2
3
propriate staff skills
to achieve our mission
3. We have the ap1
2
3
propriate staff numbers to achieve our
mission
Agree
4
4
Strongly
Agree
5
5
*The annexes for this paper are available separately and can be obtained through the USAID
Development Experience Clearinghouse at http://dec.usaid.gov/index.cfm
• Financial Resource Management
(budgeting, forecasting, and cash management)
• Human Resource Management
(staff training, supervision, and personnel
practices)
8
• Organizational Learning
(teamwork and information sharing)
check on the perceived capacities reported by
individual organizational members. It also helps
identify capacity areas that all members agree
• Strategic Management
need immediate attention.
(planning, governance, mission, and partnering) Because the cohort organizations develop the
specifics of the instrument together and share
• Service Delivery
a common understanding and application of the
(field-based program practices and sustainabil- approach, PROSE is relatively good at comparity issues)
ing organizations with each other or rolling up
results to report on a group of organizations
Although the analysis is statistically complex, together. However, the discussions could influquestionnaires can be scored and graphics pro- ence the scoring if facilitators are not consisduced using instructions provided on the DOSA tent in their administration of the tool.
Web site. In the case of DOSA, the DOSA team
in Washington processes the results and posts INSTITUTIONAL DEVELOPMENT FRAMEthem on the Internet. The assessment tool can WORK
be readministered annually to monitor organizational changes.
Background
The institutional development framework (IDF)
is a tool kit developed by Mark Renzi of Management Systems International. It has been used
in USAID/Namibia’s Living in a Finite Environment project as well as several other USAID
programs. Designed specifically to help nonprofit organizations improve efficiency and become more effective, the IDF is best suited for
the assessment of a single organization, rather
than a cohort group (as opposed to PROSE).
The kit contains three tools (Institutional Development Framework, Institutional Development Profile, and Institutional Development
Calculation Sheet), which help an organization
determine where it stands on a variety of organizational components, identify priority areas
of improvement, set targets, and measure progress over time. While it can be adapted for any
organization, the IDF was originally formulated
for environmental NGOs.
Product
PROSE instruments produce two types of
scores and accompanying graphics. The first is a
capacity score, which indicates how an organization perceives its strengths and weaknesses in
each of the capacity and subcapacity areas. The
second is a consensus score, which shows the
degree to which the assessment team members
agree on their evaluation of the organization’s
capacity.
Assessment
Unless the existing DOSA questions are used,
developing a PROSE instrument from scratch
can be time consuming and generally requires
facilitators to guide the process of developing and using the instrument. PROSE, like most
other such instruments, is based on perceived
capacities and does not currently include a
method for measuring externally observable
performance in various capacity areas (although
this is under consideration). It is unique among
the instruments in this paper in its use of a
consensus score. The consensus score acts as a
Process
An organization can use the IDF tools either
with or without the help of a facilitator. The
IDF identifies five organizational capacity areas,
9
Institutional Development Framework
Features
• Can be used, with limitations, to compare across organizations
• Measures change in the same organization over time
• Measures well-defined capacity areas against well-defined criteria
• Produces numeric score on capacity areas
• Produces qualitative description of an organization’s capacity in terms of developmental stages
• Assessment can be done internally or with help of an outside facilitator
•
• Data collected through group discussion with as many staff as feasible
called resource characteristics. Each capacity (public relations, ability to work with local
area is further broken down into key compo- communities, ability to work with government
nents, including:
bodies, ability to work with other NGOs)
Each key component within a capacity area is
• Oversight/Vision
rated at one of four stages along an organiza(board, mission, autonomy)
tional development continuum (1= start up, 2=
development, 3= expansion/consolidation, and
• Management Resources
4= sustainability). IDF offers criteria describing
(leadership style, participatory managment,
each stage of development for each of the key
management systems, planning, community
components (see example 2 below).
participation, monitoring, evaluation)
Different processes can be used depending on
• Human Resources
the organization’s size and the desired out(staff skills, staff development, organizational
come. Small organizations usually involve as
diversity)
many staff as possible; larger organizations may
work in small groups or use a few key infor• Financial Resources
mants. Members of the organization can modify
(financial management, financial vulnerability,
the Institutional Development Framework to fit
financial solvency)
their organization. Nonapplicable areas can be
ignored and new areas can be added, although
• External Resources
the creator of the tool warns against complete10
ly rewriting the criteria. Through discussion, the
participating members then use the criteria to
determine where along the development continuum their organization is situated for each
component. The resulting graphic, the Institutional Development Profile (IDP), uses bars or
“x”s to show where the organization ranks on
each key component.Through a facilitated meeting or group discussion, organization members
then determine which areas of organizational
capacity are most important to the organization
and which need priority attention for improvement. Using the IDP, they can visually mark their
targets for the future.
The IDF also provides numeric ratings. Each key
component can be rated on a scale of 1 to 4,
and all components can be averaged together
to provide a summary score for each capacity area. This allows numeric targets to be set
and monitored. The Institutional Development
Calculation Sheet is a simple table that permits
the organization to track progress over time by
recording the score of each component along
the development continuum.
Example 2. Excerpt From the IDF Tool
The following is an excerpt from the Financial Management section of the Institutional Development Framework. The entire framework appears in annex 2.
Resource
Key
Characteristic Component
Financial
Management
Budget as
Management
Tools
Cash
Controls
Financial
Security
Criteria for Each Progressive Stage
(the Development Continuum)
Start Up
Development
Expansion and
Consolidation
1
2
3
Total expendiBudgets are
Budgets are
not used as
developed for ture is usually
within 20% of
management
project actools.
tivities, but are budget, but
often over- or actual activity
underspent
often diverge
by more than from budget
20%.
predictions.
Improved
No clear
Financial
procedures ex- controls exist financial control
ist for handling but lack a sys- systems exist.
payables and
tematic office
receivables.
procedure.
Financing
comes from
only one
source.
Financing
comes from
multiple
sources, but
90% or more
from one
source.
11
No single
source of funding provides
more than 60%
of funding.
Sustainability
4
Budgets are
integral part of
project management
and are adjusted as project
implementation
warrants.
Excellent cash
controls for
payables and
receivables and
established
budget procedures.
No single source
provides more
than 40% of
funding.
Product
Process
The IDF produces a graphic that shows the
component parts of an organization and the organization’s ratings for each component at different points in time. It also provides a numeric
score/rating of capacity in each key component
and capacity area.
The OCAT is intended to be a participatory
self-assessment but may be modified to be an
external evaluation. An assessment team, composed of organizational members (representing different functions of the organization) plus
some external helpers, modifies the OCAT assessment sheet to meet its needs (annex 3).The
Assessment
assessment sheet consists of a series of statements under seven capacity areas (with subThe IDF is an example of a tool that not only elements). The assessment team then identifies
helps assess and measure an organization’s ca- sources of information, assigns tasks, and uses a
pacity but also sets priorities for future change variety of techniques (individual interviews, foand improvements. Compared with some of the cus groups, among others) to collect the inforother tools, IDF is relatively good at tracking mation they will later record on the assessment
one organization’s change over time because of sheet. The assessment team assigns a score to
the consistent criteria used for each progres- each capacity area statement (1=needs urgent
sive stage of development. It is probably not attention and improvement; 2=needs attention;
as well suited for making cross-organizational 3=needs improvement; 4=needs improvement
comparisons, because it allows for adjustment in limited aspects; but not major or urgent;
to fit the needs of each individual organization. 5=room for some improvement; 6=no need for
immediate improvement).The assessment team
ORGANIZATIONAL CAPACITY ASSESMENT would have to develop precise criteria for what
TOOL
rates as a “1” or a “2,” etc.
Background
The capacity areas and sub-elements are:
Pact developed the organizational capacity assessment tool (OCAT) in response to a need
to examine the impact of NGO capacity-building activities. Like the Institutional Development Framework, OCAT is better suited for
measuring one organization over time. The
OCAT differs substantially from the IDF in
its data collection technique. It is designed to
identify an organization’s relative strengths and
weaknesses and provides the baseline information needed to develop strengthening interventions. It can also be used to monitor progress.
The OCAT is well known; other development
organizations have widely adapted it. Designed
to be modified for each measurement situation,
the OCAT can also be standardized and used
across organizations.
• Governance
(board, mission/goal, constituency, leadership,
legal status)
• Management Practices
(organizational structure, information
management, administration procedures,
personnel, planning, program development,
program reporting)
• Human Resources
(human resources development, staff roles,
work organization, diversity issues, supervisory
practices, salary and benefits)
12
• Financial Resources
(accounting, budgeting, financial/inventory
Example 3. Excerpt From an Adaptation of the OCAT
USAID/Madagascar developed a capacity assessment tool based on the OCAT, but tailored it to
its own need to measure 21 partner institutions implementing reproductive health programs,
including the Ministry of Health. The mission tried to measure different types of organizations
and compare them by creating a standardized instrument to use with all the organizations.
Combining the OCAT results with additional information from facilitated discussions, the mission was able to summarize how different types of organizations perceived different aspects of
their capacity and recommend future strengthening programs.
Some of the difficulties that USAID/Madagascar encountered when using the tool included
having to translate questions from French to Malagasy, possibly losing some of their meaning;
finding that some respondents were unable to answer some questions because they had no
experience with the part of the organization to which the questions referred; discovering that
some respondents had difficulty separating the subject area of the questionnaire (family planning) from their work in other health areas; and having difficulty scheduling meetings because
of the organizations’ heavy workload. Moreover, the mission noted that the instrument is based
on perceptions and is self-scored, with the resulting potential for bias.a
Below is an excerpt from the “communications/extension to customers” component of the
OCAT used by USAID/Madagascar. The entire questionnaire is in annex 4.
Classification Scale
0 Nonexistent or out of order
1 Requires urgent attention and upgrading
2 Requires overall attention and upgrading
3 Requires upgrading in certain areas, but neither major nor urgent
4 Operating, but could benefit from certain improvements
5 Operating well in all regards
Communications/Extension to Customers
a. The institution has in each clinic a staff trained and competent in
counseling all customers.
1 2 3 4 5
b. The institution is able to identify and develop key messages for exten- 1 2 3 4 5
sion among potential customers, and it can produce or obtain materials
for communicating such messages.
c. A well-organized community extension is practiced by the clinic’s
staff or other workers affiliated with the institution, whether they are
salaried or volunteers. A system exists for supervising extension workers and monitoring their effectiveness.
13
1 2 3 4 5
controls, financial reporting)
• Service Delivery
(sectoral expertise, constituency, impact
assessment)
The IDF and the OCAT are similar in several
ways, but the processes differ. The OCAT uses
an assessment team that conducts research before completing the assessment sheet. For the
IDF, organization members meet and fill out the
sheet (determine their capacities) without the
intermediate data collection step (the OCAT,
by design, relies on evidence to supplement
perceptions when conducting an assessment,
and the IDF does not). The OCAT’s data-gathering step allows for systematic cross-checking
of perceived capacities with actual or observable “facts.” It is more inductive, building up to
the capacity description, while the IDF attempts
to characterize the organization along the development continuum from the beginning. The
OCAT categorizes an organization’s capacity
areas into one of four developmental stages.
Unlike the IDF, which uses the stages as the criteria by which members rate their organization,
the OCAT uses them as descriptors once the
rating has been done.
• External Relations
(constituency relations, inter-NGO collaboration, public relations, local resources, media)
• Sustainability
(program/benefit sustainability, organizational
sustainability, financial sustainability, resource
base sustainability)
After gathering data, the assessment team
meets to reach a consensus on the rating of
each element. With the help of an OCAT rating sheet, averages can be calculated for each
capacity area. These numeric scores indicate
the relative need for improvement in each area.
They also correspond to a more qualitative description of the organization’s developmental
stage. Each capacity area can be characterized
as nascent, emerging, expanding, or mature. DYNAMIC PARTICIPATORY
OCAT provides a table (similar to the IDF), INSTITUTIONAL DIAGNOSIS
“NGO Organizational Development—Stages
and Characteristics” that describes organiza- Background
tional capacities at each stage of development.
The dynamic participatory institutional diagnoProduct
sis (DPID) was developed by the Senegal PVO/
NGO support project in conjunction with the
The OCAT provides numeric ratings for each New TransCentury Foundation and Yirawah Incapacity area. In addition, it gives organizations ternational. It is a rapid and intensive facilitated
a description of their capacity areas in terms assessment of the overall strengths and weakof progressive stages of organizational develop- nesses of an organization. This methodology
ment. This information can be presented graph- explores member perceptions of an organizaically as well as in narrative form.
tion and the organization’s relationship with its
environment. DPID is highly participatory; an
Assessment
organization assesses itself in the absence of
external benchmarks or objectives to take full
The OCAT identifies areas of organization- advantage of its specific context, such as culture
al strength and weakness and tracks related and attitudes.
changes from one measurement period to the
next.
Process
14
Example 4. An Application of DPID
Since the DPID is such an individualized and flexible tool, every application will be different.
The DPID does not lend itself easily to an example as do the other tools in this Tips. Below
is an anecdote about one West African organization’s use of the DPID as reported by the
Senegal DPIPVO/NGO support project.
A Federation of Farmers’ Cooperatives with about 15,000 members in the Sahel was looking for a unique and efficient approach to redress some of the organization’s problems. The
federation suffered from internal strife and a tarnished reputation, impeding its ability to raise
funds. Through DPID, the federation conducted a critical in-depth analysis of its operational
and management systems, resulting in the adoption of “10 emergency measures” addressing
leadership weaknesses, management systems, and operational procedures. Subsequently, the
organization underwent internal restructuring, including an overhaul of financial and administrative systems. One specific result of the DPID analysis was that federation members gained
more influence over the operations of the federation.
An outside facilitator conducts the DPID over
5 to 10 days. It takes place during a series of
working sessions in which the facilitator leads
an organization’s members through several
stages: discussion of the services; operations
and results of the organization; exploration of
the issues affecting the organization; and summarization of the “state of the organization.”
During the discussions, members analyze the
following features of the organization:
They examine each element with reference to
institutional behavior, human behavior, management, administration, know-how, philosophy
and values, and sensitive points.
• Identity
Assessment
• Mission
Unlike the previously described tools, the DPID
does not use ranking, scoring, or questionnaires,
nor does it assess the organization along a continuum of developmental stages. Assessment
is based purely on group reflection. The DPID
requires a facilitator experienced in leading a
group through this type of analysis.
• Means and Resources
• Environment
• Management
• Internal Operations
• Service Provided and Results
Product
A written description of the state of the organization can result from the working sessions.The
analysis is qualitative without numeric scoring.
The DPID is open ended but somewhat systematic in covering a predefined set of organizational functions. Because of its flexibility, the
DPID is organization specific and should not
15
ORGANIZATIONAL CAPACITY
INDICATOR
Dynamic Participatory
Institutional Diagnosis
Background
From 1994 through 1997, the Christian Reformed World Relief Committee (CRWRC)
conducted research on organizational capacitybuilding with the Weatherhead School of Management at Case Western Reserve University
and more than 100 local NGOs around the
world. The results of this research led them to
replace their earlier system, the Skill Rating System, with an approach to capacity building and
assessment based on “appreciative inquiry.” Appreciative inquiry is a methodology that emphasizes an organization’s strengths and potential
more than its problems. It highlights those qualities that give life to an organization and sustain its ongoing capacity. Rather than providing
a standardized tool, the organizational capacity
indicator assumes that capacity monitoring is
unique to each organization and in the organization’s own self-interest. The organizational
capacity indicator (OCI) builds ownership because each organization creates its own capacity
assessment tool. Capacity areas are self-defined
and vary from organization to organization.
Features
• Difficult to compare across organizations
• Difficult to compare the same organization over time
• Capacity areas and criteria for measurement are loosely defined
• Produces qualitative description of an
organization’s capacity
• Assessment done with the help of an
outside facilitator
with the organization’s staff
Process
Although organizations create their own tool
under the OCI, they all follow a similar probe used to compare organiza tions. Nor is it a
cess in doing so. As they involve all partners
rigorous means of monitoring an organization’s
and stakeholders as much as possible, the parchange over time. Since the DPID does not use
ticipants “appreciate” the organization’s history
external standards to assess institutional caand culture.Together they explore peak experipacities, it should not be used to track accountences, best practices, and future hopes for the
ability. Collecting information from the DPID,
organization. Next, the participants identify the
as well as using it, should offer organizations a
forces and factors that have made the organizaprocess to assess their needs, improve commution’s positive experiences possible. These benications, and solve problems around a range of
come the capacity areas that the organization
organizational issues at a given moment.
tries to monitor and improve.
Next, the participants develop a list of “provoca16
ences related to each capacity component. The
organization should monitor itself by this process twice a year. The results of the assessment
should be used to encourage future development, plans, and aspirations.
tive propositions” for each capacity area. These
propositions, visions of what each capacity area
should ideally look like in the future, contribute
to the overall objective: that each organization
will be able to measure itself against its own
vision for the future, not some external standard. Each capacity area is defined by the most
ambitious vision of what the organization can
become in that area. Specific indicators or behaviors are then identified to show the capacity
area in practice. Next, the organization designs
a process for assessing itself and sharing experi-
Product
Each time a different organization uses the
methodology, a different product specific to
that organization is developed. Thus, each tool
will contain a unique set of capacity areas, an
Example 5. Excerpt From an OCI Tool
The following is an excerpt of one section from the capacity assessment tool developed by
CRWRC’s partners in Asia, using the OCI method. (The entire tool can be found in annex 5.)
It offers a menu of capacity areas and indicators from which an organization can choose and
then modify for its own use. It identifies nine capacity areas, and under each area is a “provocative proposition” or vision of where the organization wants to be in that area. It provides an
extensive list of indicators for each capacity area, and it describes the process for developing
and using the tool. Staff and partners meet regularly to determine their capacity on the chosen indicators. Capacity level can be indicated pictorially, for example by the stages of growth
of a tree or degrees of happy faces.
Capacity Area
A clear vision, mission, strategy, and set of shared values
Proposition
Our vision expresses our purpose for existing: our dreams, aspirations, and concerns for the
poor. Our mission expresses how we reach our vision. Our strategy expresses the approach
we use to accomplish our goals. The shared values that we hold create a common understanding and inspire us to work together to achieve our goal.
Selected Indicators
• Every person can state the mission and vision in his or her own words
• There is a yearly or a six-month plan, checked monthly
• Operations/activities are within the vision, mission, and goal of the organization
• Staff know why they do what they’re doing
• Every staff member has a clear workplan for meeting the strategy
• Regular meetings review and affirm the strategy
17
• Possible to measure well-defined capacity areas across well-defined criteria
organization capacity areas like the DPID does.
The OCI is the only tool presented in this paper in which the capacity areas are entirely self
defined. It is also unique in its emphasis on the
positive, rather than on problems. Further, the
OCI is more rigorous than the DPID, in that it
asks each organization to set goals and develop
indicators as part of the assessment process. It
also calls for a scoring system to be developed,
like the more formal tools (PROSE, IDF, OCAT).
Because indicators and targets are developed
for each capacity area, the tool allows for relatively consistent measurement over time. OCI
is not designed to compare organizations with
each other or to aggregate the capacity measures of a number of organizations; however, it
has proven useful in allowing organizations to
learn from each other and in helping outsiders
assess and understand partner organizations.
THE YES/NO CHECKLIST OR
“SCORECARD”
• Produces numeric or pictorial score on
capacity areas
Background
Organizational Capacity
Indicator
NGOs/PVOs; adaptable to other types of
organizations
Features
• Difficult to comparably measure across
organizations
A scorecard/checklist is a list of characteristics
or events against which a yes/no score is assigned. These individual scores are aggregated
and presented as an index. Checklists can effectively track processes, outputs, or more general
characteristics of an organization. In addition,
they may be used to measure processes or outputs of an organization correlated to specific
areas of capacity development.
• Assessment done internally
with organization’s staff
evaluation process, and scoring methods. In
general, the product comprises a written description of where the organization wants to be
in each capacity area, a list of indicators that can Scorecards/checklists can be used either to
be used to track progress toward the targeted measure a single capacity component of an
organization or several rolled together. Scorelevel in a capacity area, and a scoring system.
cards/checklists are designed to produce a
quantitative score that can be used by itself or
Assessment
as a target (though a scorecard/checklist withLike the DPID, the OCI is highly participatory out an aggregate score is also helpful).
and values internal standards and perceptions.
Both tools explicitly reject the use of external
standards. However, the OCI does not desi nate
18
Process
The Yes/No Checklist
“Scorecard”
To construct a scorecard, follow these general
steps: First, clarify what the overall phenomena
to be measured are and identify the components that, when combined, cover the phenomenon fairly well. Next, develop a set of characteristics or indicators that together capture the
relevant phenomena. If desired, and if evidence
and analysis show that certain characteristics
are truly more influential in achieving the overall result being addressed, define a weight to be
assigned to each characteristic/indicator. Then
rate the organization(s) on each characteristic
using a well defined data collection approach.
The approach could range from interviewing
organization members to reviewing organization documents, or it could consist of a combination of methods. Finally, if desired and appropriate, sum the score for the organization(s).
All types of organizations
Features
• Cross-organizational comparisons can
be made
• Measures well-defined capacity areas
against well-defined criteria
Product
• Possible to balance perceptions with
empirical observations
• Produces numeric score on capacity
areas
A scorecard/checklist results in a scored listing
of important characteristics of an organization
and can also be aggregated to get a summary
score.
• Assessment can be done by an external
evaluator or internally
• Data collected through interviews, observation, documents, involving a limited
number of staff
Assessment
A scorecard/checklist should be used when the
characteristics to be scored are unambiguous.
There is no room for “somewhat” or “yes, but . .
.” with the scorecard technique.The wording of
each characteristic should be clear and terms
should be well defined. Because scorecards/
checklists are usually based on observable facts,
processes, and documents, they are more objective than most of the tools outlined in this Tips.
This, in turn, makes them particularly useful for
cross-organizational comparisons, or tracking
organizations over time; that is, they achieve
better measurement consistency and comparability. Yet concentrating on observable facts can
be limiting, if such facts are not complemented
19
with descriptive and perceptionbased information. Though a person outside the organization
frequently completes the scorecard/checklist,
self-assessment is also possible. Unlike other
tools that require facilitators to conduct or
interpret them, individuals who are not highly
trained can also use scorecards. Further, since
scorecards are usually tightly defined and specific, they are often a cheaper measurement
tool.
Example 6. A Scorecard
USAID/Mozambique developed the following scorecard to measure various aspects of institutional capacity in partner civil society organizations. The following example measures democratic governance.
Increased Democratic Governance Within Civil Society Organizations
Characteristics
Score Multiplied Weight
By
1. Leaders (board member or equivalent) of the
X
3
CSO electedby secret ballot. No=0 pts.Yes=1 pt.
2. General assembly meetings are adequately
announced at least two weeks in advance to all
members (1 pt.) and held at least twice a year (1
pt.). Otherwise=0 pt.
3. Annual budget presented for member approval. No=0 pts.Yes=1 pt.
4. Elected leaders separate from paid employees.
No=0 pts.Yes=1 pt.
5. Board meetings open to ordinary members
(nonboard members). No=0 pts.Yes=1 pt.
X
2
X
2
X
2
X
1
Weighted
Score
Total
external relations and internal governance.
development professionals also
MEASURING INDIVIDUAL Organizational
use other tools to measure specific capacity
ORGANIZATIONAL
areas. Some drawbacks of these tools are that
they require specialized technical expertise and
COMPONENTS
they can be costly to use on a regular basis.
Other tools may require some initial training
In some cases, USAID is not trying to strength- but can be much more easily institutionalized.
en the whole organization, but rather specific Below we have identified some tools for meaparts of it that need special intervention. In many suring selected organizational components.
cases, the best way of measuring more specific (You will find complete reference information
organizational changes is to use portions of the for these tools in the resources section of this
instruments described. For instance, the IDF Tips.)
has a comparatively well-developed section
on management resources (leadership style, STRUCTURE AND CULTURE
participatory management, planning, monitoring and evaluation, and management systems). The Preferred Organizational Structure instruSimilarly, the OCAT has some good sections on ment is designed to assess many aspects of or20
ganizational structure, such as formality of rules,
communication lines, and decision-making. This
tool requires organizational development skills,
both to conduct the assessment and to interpret the results.
HUMAN RESOURCES AND THEIR
MANAGEMENT
First, the development of indicators should be
driven by the informational needs of managers,
from both USAID and the given relevant organizations-- to inform strategic and operational
decisions and to assist in reporting and communicating to partners and other stakeholders.
At times, there is a tendency to identify or design a data collection instrument without giving
too much thought to exactly what information
will be needed for management and reporting.
In these situations, indicators tend to be developed on the basis of the data that have been
collected, rather than on what managers need.
More to the point, the development of indicators should follow a thorough assessment of
informational needs and precede the identification of a data collection instrument. Managers should first determine their informational
needs; from these needs, they should articulate
and define indicators; and only then, with this
information in hand, they would identify or
develop an instrument to collect the required
data. This means that, in most cases, indicators
should not be derived, post facto, from a data
collection tool. Rather, the data collection tool
should be designed with the given indicators in
mind. Second, indicators should be developed
for management decisions at all levels (input indicators, output indicators, process indicators,
and outcome/impact indicators). With USAID’s
increased emphasis on results, managers sometimes may concentrate primarily on strategic
indicators (for development objectives and
intermediate results). While an emphasis on
results is appropriate, particularly for USAID
managers, tracking operational-level information for the organizations supported through
a given Agency program is critical if managers
are to understand if, to what degree, and how
the organizations are increasing their capacities. The instruments outlined in this paper can
provide data for indicators defined at various
management levels.
Many personnel assessments exist, including the
Job Description Index and the Job Diagnostic
Survey, both of which measure different aspects
of job satisfaction, skills, and task significance.
However, skilled human resource practitioners
must administer them. Other assessments, such
as the Alexander Team Effectiveness Critique,
have been used to examine the state and functioning of work teams and can easily be applied
in the field.
SERVICE DELIVERY
Often, a customer survey is one of the best
ways to measure the efficiency and effectiveness of a service delivery system. A specific
customer survey would need to be designed
relative to each situation. Example 7 shows a
sample customer service assessment.
DEVELOPING INDICATORS
Indicators permit managers to track and understand activity/program performance at both
the operational (inputs, outputs, processes)
and strategic (development objectives and intermediate results) levels. To managers familiar
with the development and use of indicators, it
may seem straightforward to derive indicators
from the instruments presented in the preceding pages. However, several critical points will
ensure that the indicators developed within the
context of these instruments are useful to managers.
Finally, indicators should meet the criteria out21
Example 7. A Customer Service Assessment
1. In the past 12 months, have you ever contacted a municipal office to complain about something such as poor
city services or a rude city official, or any other reason?
________No ________Yes
If YES:
1a. How many different problems or complaints did you contact the municipality about in the last 12 months?
________One ________Two ________Three to five ________More than five
1b. Please describe briefly the nature of the complaint starting with the one you feel was most important.
1._______________________________________________
2._______________________________________________
3._______________________________________________
2. Which department or officials did you contact initially regarding these complaints?
____Mayor’s office
____Council member
____Police
____Sanitation
____Public works
____Roads
____Housing
____Health
____Other________________________________________
2a. Were you generally satisfied with the city’s response? (IF DISSATISFIED, ASK: What were the major
reasons for your dissatisfaction?)
_____Response not yet completed
_____Satisfied
_____Dissatisfied, never responded or corrected condition
_____Dissatisfied, poor quality or incorrect response was provided
_____Dissatisfied, took too long to complete response, had to keep pressuring for results, red tape, etc.
_____Dissatisfied, personnel were discourteous, negative, etc.
_____Dissatisfied, other_____________________________
3. Overall, are you satisfied with the usefulness, courtesy and effectiveness of the municipal department or
official that you contacted?
_____Definitely yes
_____Generally yes
_____Generally no (explain)__________________________
_____Definitely no (explain)__________________________
Survey adapted from Hatry, Blair, and others, 1992.
22
lined in USAID’s Automated Directives System
and related pieces of Agency guidance such as
CDIE’s Performance Monitoring and Evaluation Tips #6, “Selecting Performance Indicators,” and Tips #12, “Guidelines for Indicator
and Data Quality.” That is, indicators should be
direct, objective, practical, and adequate. Once
an indicator has been decided upon, it is important to document the relevant technical details:
a precise definition of the indicator; a detailed
description of the data source; and a thorough
explanation of the data collection method. (Refer to Tips #7, “Preparing a Performance Monitoring Plan.”)
data). If a result refers to multiple organizations,
it might be useful to frame an indicator in terms
of the number or percent of the organizations
that meet or exceed a given threshold score or
development stage, on the basis of an aggregate
index or the score of a single element for each
organization. The key is to ensure that the indicator reflects the result and to then identify the
most appropriate and useful measurement instrument.
RESULTS-LEVEL INDICATORS
Example 8 includes real indicators used by USAID missions in 1998 to report on strategic objectives and intermediate results in institutional
capacity strengthening.
USAID managers spend substantial time and
energy developing indicators for development
objectives and intermediate results related to
institutional capacity. The range of the Agency’s
institutional strengthening programs is broad,
as is the range of the indicators that track the
programs’ results. Some results reflect multiple
organizations and others relate to a single organization. Additionally, of those results that relate to multiple organizations, some may refer
to organizations from only one sector while
others may capture organizations from a number of sectors. Results related to institutional
strengthening also vary relative to the level of
change they indicate-- such as an increase in institutional capacity versus the eventual impact
generated by such an i crease-- and with regard to whether they reflect strengthening of
the whole organization(s) or just one or several
elements. It is relatively easy to develop indicators for all types of results and to use the instruments outlined in this Tips to collect the necessary data. For example, when a result refers
to strengthening a single organization, across
all elements, an aggregate index or “score” of
institutional strength may be an appropriate indicator (an instrument based on the IDF or the
scorecard model might be used to collect such
PRACTICAL TIPS FOR A
BUSY USAID MANAGER
This TIPS introduces critical issues related to
measuring institutional capacity. It presents a
number of approaches that managers of development programs and activities currently use
in the field. In this section we summarize the
preceding discussion by offering several quick
tips that USAID managers should find useful as
they design, modify, and implement their own
approaches for measuring institutional capacity.
1. Carefully review the informational needs of
the relevant managers and the characteristics of the organization to be measured to
facilitate development of indicators. Identify
your information needs and develop indicators
before you choose an instrument.
23
2. To assist you in selecting an appropriate
measurement tool, ask yourself the following
questions as they pertain to your institutional
capacity measurement situation. Equipped
with the answers to these questions, you
Example 8.
Selected Institutional Capacity Indicators From USAID Missions
Indicator
• Number of institutions meeting at least
80% of their targeted improvements
To Measure
Institutions strengthened (entire organization)
• Amount of funds raised from non-USAID
Institutions more financially sustainable
sources
• Number of organizations where USAID
contribution is less than 25% of revenues
• Number of organizations where at least
five funding sources contribute at least 10%
each
• Percent of suspected polio cases investiga- Organization’s service delivery systems
tee within 48 hours
strengthened
• Number of governmental units displaying
Local government management capacities
improved practices, such as open and trans- improved
parent financial systems, set organizational
procedures, accountability, participatory
decision-making, by-laws and elections
can scan the “features list” that describes
every tool in this paper to identify which
measurement approaches to explore further.
• Is the objective to measure the entire organization? Or is it to measure
specific elements of the organization?
If the latter, what are the specific capacity areas of functions to be measured?
• How will the information be used?
To measure change in an organization over time? To compare organizations with each other?
To inform procurement decisions?
To hold an organization accountable
for achieving results or implementing
reforms?
• What type of organizations are you
measuring? Are there any particular
measurement issues pertaining to
this type of organization that must
be considered?
• How participatory do you want the
measurement process to be?
• What is the purpose of the intervention? To strengthen an organization?
24
• Will organization members themselves or outsiders conduct the
assessment?
• What product do you want the measurement tool to generate?
and involve some form of ordinal scaling/
scoring. When reviewing data, managers
should therefore zero in on the direction and
general degree of change. Do not be overly
concerned about small changes; avoid false
precision.
• Do you want the measurement process to be an institution-strengthening exercise in itself?i. Do you need
an instrument that measures one
organization? Several organizations
againstindividual criteria? Or several organizations against standard
criteria?
5. Cost matters-- and so does the frequency
and timing of data collection. Data need to
be available frequently enough, and at the
right point in the program cycle, to inform
operational and strategic management decisions. Additionally, the management benefits
of data should exceed the costs associated
with their collection.
3. If you are concerned about data reliability, apply measurement instruments consistently over
time and across organizations to ensure data
reliability. You can adapt and adjust tools as
needed, but once you develop the instru- 6. The process of measuring institutional capacity
can contribute substantially to increasing an orment, use it consistently.
ganization’s strength. A number of measurement approaches are explicitly designed as
4. When interpreting and drawing conclusions
learning opportunities for organizations;
from collected data, remember the limits of the
that is, to identify problems and suggest rerelevant measurement tool. Most methods for
lated solutions, to improve communication,
measuring institutional capacity are subjecor to facilitate a consensus around future
tive, as they are based on the perceptions
priorities
of those participating in the assessment,
This TIPS was prepared for CDIE by Alan Lessik and
Victoria Michener of Management Systems International.
Bibliography
RESOURCES
Booth, W.; and R. Morin. 1996. Assessing Organizational Capacity Through Participatory
Monitoring and Evaluation Handbook. Prepared for the Pact Ethiopian NGO Sector
Enhancement Initiative. Washington: USAID.
Center for Democracy and Governance. 1998. Handbook of Democracy and Governance
25
Program Indicators.Washington: U.S. Agency for International Development.
Christian Reformed World Relief Committee. 1997. Partnering to Build and Measure
Organizational Capacity. Grand Rapids, Mich.
Cooper, S.; and R. O’Connor. 1993. “Standards for Organizational Consultation: Assessment and Evaluation Instruments.” Journal of Counseling and Development 71: 651-9.
Counterpart International. N.d. “CAP Monitoring and Evaluation Questionnaire.”
—N.d. “Manual for the Workshop on Development of a Training and Technical Assistance Plan (TTAP).”
—N.d. “Institutional Assessment Indicators.”
Drucker, P.; and C. Roseum. 1993. How to Assess Your Nonprofit Organization with Peter Drucker’s Five
Important Questions: User Guide for Boards, Staff,Volunteers and Facilitators. Jossey--Bass .
Eade, D. 1997. Capacity-Building: An Approach to People-Centred Development. Oxford: Oxfam.
Fowler, A.; L. Goold; and R. James. 1995. Participatory Self Assessment of NGO Capacity. INTRAC
Occasional Papers Series No. 10. Oxford.
Hatry, H.; L. Blair; D. Fisk; J. Grenier; J. Hall; and P. Schaenman. 1992. How Effective Are Your Community
Services? Procedures for Measuring Their Quality. Washington: The Urban
Institute.
International Working Group on Capacity Building of Southern NGOs. 1998. “Southern NGO Capacity Building: Issues and Priorities.” New Delhi: Society for Participatory Research in Asia.
International Working Group on Capacity Building for NGOs. 1998. “Strengthening Southern NGOs: The Donor Perspective.” Washington: USAID and The World Bank.
Kelleher, D. and K. McLaren with R. Bisson. 1996. “Grabbing the Tiger by the Tail: NGOs
Learning forOrganizational Change.” Canadian Council for International Cooperation.
Lent, D. October 1996. “What is Institutional Capacity?” On Track: The Reengineering Digest. 2 (7): 3. Washington: U.S. Agency for International Development.
Levinger, B. and E. Bloom. 1997. Introduction to DOSA: An Outline Presentation.
http://www.edc.org/int/capdev/dosafile/dosintr.htm.
Lusthaus, C., G. Anderson, and E. Murphy. 1995. “Institutional Assessment: A Framework for
Strengthening Organizational Capacity for IDRC’s Research Partners.” IDRC.
26
Mentz, J.C.N. 1997. “Personal and Institutional Factors in Capacity Building and Institutional
Development.” European Centre for Development Policy Management Working Paper
No. 14.
Morgan, P.; and A. Qualman. 1996. “Institutional and Capacity Development, Results-Based
Management and Organisational Performance.” Canadian International Development
Agency.
New TransCentury Foundation. 1996. Practical Approaches to PVO/NGO Capacity Building:
Lessons from the Field (five monographs). Washington: U.S.Agency for International
Development.
Pact. N.d. “What is Prose?”
—1998. “Pact Organizational Capacity Assessment Training of Trainers.” 7-8 January.
Renzi, M. 1996. “An Integrated Tool Kit for Institutional Development.”Public Administration
and Development 16: 469-83.
—N.d. “The Institutional Framework: Frequently Asked Questions.” Unpublished paper.
Management Systems International.
Sahley, C. 1995. “Strengthening the Capacity of NGOs: Cases of Small Enterprise Development
Agencies in Africa.” INTRAC NGO Management and Policy Series. Oxford.
Save the Children. N.d. Institutional Strengthening Indicators: Self Assessment for NGOs. UNDP. 1997.
Capacity Assessment and Development. Technical Advisory Paper No. 3, Management
Development and Governance Division. New York.
Bureau for Policy and Program Coordination. 1995. USAID-U.S. PVO Partnership. Policy
Guidance. Washington: U.S. Agency for International Development.
Office of Private and Voluntary Cooperation. 1998. USAID Support for NGO Capacity-Building:
Approaches, Examples, Mechanisms. Washington: U.S. Agency for International
Development.
—1998. Results Review Fiscal Year 1997. Washington: U.S. Agency for International
Development.
NPI Learning Team. 1997. New Partnerships Initiative: A Strategic Approach to Development
Partnering. Washington: U.S. Agency for International Development. 23
USAID/Brazil. 1998. Fiscal Year 2000 Results Review and Resource Request.
USAID/Guatemala. 1998. Fiscal Year 2000 Results Review and Resource Request.
27
USAID/Indonesia. 1998. Fiscal Year 2000 Results Review and Resource Request.
USAID/Madagascar. 1998. Fiscal Year 2000 Results Review and Resource Request.
—1997. Institutional Capacity Needs Assessment.
USAID/Mexico. 1998. The FY 1999--FY 2003 Country Strategy for USAID in Mexico.
USAID/Mozambique. 1998. Fiscal Year 2000 Results Review and Resource Request.
USAID/West Bank--Gaza. 1998. Fiscal Year 2000 Results Review and Resource Request.
Whorton, J.; and D. Morgan. 1975. Measuring Community Performance: A Handbook of
Indicators, University of Oklahoma.
World Bank. 1996. Partnership for Capacity Building in Africa: Strategy and Program of Action.
Washington.
World Learning. 1998. Institutional Analysis Instrument: An NGO Development Tool.
Sources of Information on Institutional Capacity Measurement Tools
Discussion-Oriented Organizational Self-Assessment:
http://www.edc.org/int/capdev/dosafile/dosintr.htm.
Institutional Development Framework: Management Systems International. Washington.
Organizational Capacity Assessment Tool: http://www.pactworld.org/ocat.html Pact.Washington.
Dynamic Participatory Institutional Diagnostic: New TransCentury Foundation. Arlington,Va.
Organizational Capacity Indicator: Christian Reformed World Relief Committee. Grand
Rapids, Mich.
Smith, P.; L. Kendall; and C. Hulin. 1969. The Measurement of Satisfaction in Work and
Retirement. Rand McNally.
Hackman, J.R.; and G.R. Oldham. 1975. “Job Diagnostic Survey: Development of the Job
Diagnostic Survey”
Journal of Applied Psychology 60: 159-70.
Goodstein, L.D.; and J.W. Pfeiffer, eds. 1985. Alexander Team Effectiveness Critique:The 1995
Annual: Developing Human Resources. Pfeiffer & Co.
28
Bourgeois, L.J.; D.W. McAllister; and T.R. Mitchell. 1978. “Preferred Organizational Structure:
The Effects of Different Organizational Environments Upon Decisions About
Organizational Structure.” Academy of Management Journal 21: 508-14.
Kraut, A. 1996. Customer and Employee Surveys: Organizational Surveys:Tools for Assessment and
Change. Jossey-Bass Publishers. 24
29
NUMBER 16
1ST EDITION 2010
TIPS
CONDUCTING MIXED-METHOD EVALUATIONS
ABOUT TIPS
INTRODUCTION
This TIPS provides guidance on
using a mixed-methods approach
for
evaluation
research.
Frequently, evaluation statements
of work specify that a mix of
methods be used to answer
evaluation questions. This TIPS
includes the rationale for using a
mixed-method evaluation design,
guidance for selecting among
methods (with an example from
an evaluation of a training
program) and examples of
techniques for analyzing data
collected with several different
methods
(including
―parallel
analysis‖).
MIXED-METHOD
EVALUATIONS
DEFINED
A mixed-method evaluation is
one that uses two or more
techniques or methods to collect
the data needed to answer one or
more evaluation questions. Some
of the different data collection
methods that might be combined
in
an
evaluation
include
structured
observations,
key
informant interviews, pre- and
post-test surveys, and reviews of
government statistics. This could
involve the collection and use of
both quantitative and qualitative
data to analyze and identify
findings
and
to
develop
conclusions in response to the
evaluation questions.
1
RATIONALE FOR
USING A MIXEDMETHOD
EVALUATION DESIGN
There are several possible cases
in which it would be highly
beneficial to employ mixedmethods in an evaluation design:
 When a mix of different
methods is used to collect data
from different sources to
provide independent estimates
of key indicators—and those
estimates complement one
another—it
increases
the
validity of conclusions related
to an evaluation question. This
is referred to as triangulation.
(See TIPS 5: Rapid Appraisal,
and Bamberger, Rugh and
Key Steps in Developing a Mixed-Method Evaluation Design and Analysis
Strategy
1. In order to determine the methods that will be employed, carefully review the purpose of the evaluation and the
primary evaluation questions. Then select the methods that will be the most useful and cost-effective to answer
each question in the time period allotted for the evaluation. Sometimes it is apparent that there is one method
that can be used to answer most, but not all, aspects of the evaluation question.
2. Select complementary methods to cover different aspects of the evaluation question (for example, the how and
why issues) that the first method selected cannot alone answer, and/or to enrich and strengthen data analysis
and interpretation of findings.
3. In situations when the strength of findings and conclusions for a key question is absolutely essential, employ a
triangulation strategy. What additional data sources and methods can be used to obtain information to answer
the same question in order to increase the validity of findings from the first method selected?
4. Re-examine the purpose of the evaluation and the methods initially selected to ensure that all aspects of the
primary evaluation questions are covered thoroughly. This is the basis of the evaluation design. Develop data
collection instruments accordingly.
5. Design a data analysis strategy to analyze the data that will be generated from the selection of methods chosen
for the evaluation.
6. Ensure that the evaluation team composition includes members that are well-versed and experienced in applying
each type of data collection method and subsequent analysis.
7. Ensure that there is sufficient time in the evaluation statement of work for evaluators to fully analyze data
analysis
canof conducting
also  Mixed-method
evaluations
generated[2006]
from eachfor
method
employedricher
and to realize
the and
benefits
a mixed method evaluation.
Mabry
further
explanation and descriptions of
triangulation strategies used in
evaluations.)
provide a better understanding
of the context in which a
program operates.
 When reliance on one method
There are a number of additional
benefits derived from using a mix
of methods in any given
evaluation.
alone may not be sufficient to
answer all aspects of each
evaluation question.
 When the data collected from
one method can help interpret
findings from the analysis of
data collected from another
method.
For
example,
qualitative data from in-depth
interviews or focus groups can
help
interpret
statistical
patterns from quantitative data
collected through a randomsample survey. This yields a
 Using
mixed-methods can
more readily yield examples of
unanticipated
changes
or
responses.
 Mixed-method
evaluations
have the potential of surfacing
other key issues and providing
a deeper understanding of
program context that should
be considered when analyzing
data and developing findings
and conclusions.
2
often yield a wider range of
points of view that might
otherwise be missed.
DETERMINING
WHICH METHODS TO
USE
In a mixed-method evaluation,
the evaluator may use a
combination of methods, such as
a survey using comparison
groups in a quasi-experimental or
experimental design, a review of
key documents, a reanalysis of
government statistics, in-depth
interviews with key informants,
focus groups, and structured
observations. The selection of
methods, or mix, depends on the
nature of the evaluation purpose
and the key questions to be
addressed.
SELECTION OF DATA
COLLECTION
METHODS – AN
EXAMPLE
The selection of which methods
to use in an evaluation is
driven by the key evaluation
questions to be addressed.
Frequently,
one
primary
evaluation method is apparent.
For
example,
suppose
an
organization wants to know
about the effectiveness of a pilot
training program conducted for
100 individuals to set up their
own small businesses after the
completion of the training.
The evaluator should ask what
methods are most useful and
cost-effective to assess the
question of the effectiveness of
that training program within the
given time frame allotted for the
evaluation. The answer to this
question must be based on the
stated outcome expected from
the training program. In this
example, let us say that the
organization’s expectations were
that, within one year, 70 percent
of the 100 individuals that were
trained will have used their new
skills and knowledge to start a
small business.
What is the best method to
determine whether this outcome
has been achieved? The most
costeffective
means
of
answering this question is to
survey 100 percent of the
individuals who graduated from
the training program using a
close-ended questionnaire.
It
follows that a survey instrument
should be designed to determine
if these individuals have actually
succeeded in starting up a new
business.
While this sounds relatively
straightforward, organizations are
often interested in related issues.
If less than 70 percent of the
individuals started a new business
one year after completion of the
training,
the
organization
generally wants to know why
some
graduates
from
the
program were successful while
others were not. Did the training
these individuals received actually
help them start up a small
business? Were there topics that
should have been covered to
more thoroughly prepare them
for the realities of setting up a
business?
Were there other
topics that should have been
addressed?
In summary, this
organization wants to learn not
only whether at least 70 percent
of the individuals trained have
started up a business, but also
how effectively the training
equipped them to do so. It also
wants to know both the strengths
and the shortcomings of the
training so that it can improve
future training programs.
The organization may also want
to know if there were factors
outside the actual intervention
that had a bearing on the
training’s success or failure. For
example, did some individuals
find employment instead? Was
3
access to finance a problem? Did
they conduct an adequate market
analysis? Did some individuals
start with prior business skills?
Are there factors in the local
economy, such as local business
regulations, that either promote
or discourage small business
start-ups? There are numerous
factors
which
could
have
influenced this outcome.
The selection of additional
methods to be employed is,
again, based on the nature of
each aspect of the issue or set
of related questions that the
organization wants to probe.
To continue with this example,
the evaluator might expand the
number of survey questions to
address issues related to the
effectiveness of the training and
external factors such as access to
finance.
These
additional
questions can be designed to
yield additional quantitative data
and to probe for information
such as the level of satisfaction
with the training program, the
usefulness
of
the
training
program
in
establishing
a
business, whether the training
graduate
received
a
small
business start-up loan, if the size
of the loan the graduate received
was sufficient, and whether
graduates are still in the process
of starting up their businesses or
instead have found employment.
Intake data from the training
program on characteristics of
each trainee can also be
examined to see if there are any
particular characteristics, such as
sex or ethnic background, that
can be correlated with the survey
findings.
It is important to draw on
additional methods to help
explain the statistical findings
from the survey, probe the
strengths and shortcomings of
the training program, further
understand issues related to
access to finance, and identify
external factors affecting success
in starting a business.
In this
case, the evaluation design could
focus on a sub-set of the 100
individuals to obtain additional
qualitative information.
A
selected group of 25 people
could be asked to answer an
additional series of open-ended
questions during the same
interview session, expanding it
from 30 minutes to 60 minutes.
Whereas asking 100 people
open-ended questions would be
better than just 25 people, costs
prohibit interviewing the entire
group.
Using
the
same
example,
suppose the organization has
learned
through
informal
feedback that access to finance is
likely a key factor in determining
success in business start-up in
addition to the training program
itself.
Depending on the
evaluation
findings,
the
organization may want to design
a finance program that increases
access to loans for small business
start-ups. To determine the
validity of this assumption, the
evaluation design relies on a
triangulation approach to assess
whether and how access to
finance for business start-ups
provides further explanations
regarding success or failure
outcomes. The design includes a
plan to collect data from two
other sources using a separate
data collection method for each
source. The first data source
includes the quantitative data
from the survey of the 100
training
graduates.
The
evaluation designers determine
that the second data source will
be the managers of local banks
and credit unions that survey
respondents reported having
approached for start-up loans.
In-depth interviews will be
conducted
to
record
and
understand policies for lending to
entrepreneurs trying to establish
small businesses, the application
of those policies, and other
business practices with respect to
prospective clients.
The third
data source is comprised of bank
loan statistics for entrepreneurs
who have applied to start up
small businesses. Now there are
three independent data sources
using different data collection
methods to assess whether
access to finance is an additional
key factor in determining small
business start-up success.
In this example, the total mix of
methods the evaluator would use
includes the following: the survey
of all 100 training graduates, data
from open-ended questions from
a subset of graduates selected for
longer interviews, analysis of
training intake data on trainee
characteristics,
in-depth
interviews with managers of
lending institutions, and an
examination of loan data. The
use of mixed-methods was
necessary because the client
organization in this case not only
wanted to know how effective the
pilot training course was based
4
on its own measure of program
success, but also whether access
to finance contributed to either
success or failure in starting up a
new business. The analysis of the
data will be used to strengthen
the training design and content
employed in the pilot training
course, and as previously stated,
perhaps to design a microfinance
program.
The last step in the process of
designing
a
mixed-method
evaluation is to determine how
the data derived from using
mixed-methods will be analyzed
to produce findings and to
determine the key conclusions.
ANALYZING DATA
FROM A MIXEDMETHOD
EVALUATION –
DESIGNING A DATA
ANALYSIS STRATEGY
It is important to design the data
analysis strategy before the
actual data collection begins.
Having done so, the evaluator
can begin thinking about trends
in findings from different sets of
data to see if findings converge
or diverge.
Analyzing data
collected from a mixture of
methods is admittedly more
complicated than analyzing the
data derived from one method.
This entails a process in which
quantitative and qualitative data
analysis strategies are eventually
connected to determine and
understand key findings. Several
different techniques can be used
to analyze data from mixedmethods approaches, including
parallel
analysis,
conversion
analysis,
sequential
analysis,
multilevel analysis, and data
synthesis. The choice of analytical
techniques should be matched
with the purpose of the
evaluation using mixed-methods.
Table 1 briefly describes the
different analysis techniques and
the situations in which each
method is best applied.
In
complex
evaluations
with
multiple issues to address, skilled
evaluators may use more than
one of these techniques to
analyze the data.
EXAMPLE OF
APPLICATION
Here we present an example of
parallel
mixed-data
analysis,
because it is the most widely
used analytical technique in
mixed-method evaluations. This
is followed by examples of how
to resolve situations where
divergent findings arise from the
analysis of data collected through
a triangulation process.
PARALLEL
MIXED-DATA
ANALYSIS
Parallel mixed-data analysis is
comprised of two major steps:
Step 1: This involves two or
more analytical processes. The
data collected from each method
employed must be analyzed
separately.
For example, a
statistical analysis of quantitative
data derived from a survey, a set
of height/weight measures, or a
set of government statistics is
conducted. Then, a separate and
independent
analysis
is
conducted of qualitative data
derived from, for example, indepth interviews, case studies,
focus groups, or structured
observations
to
determine
emergent
themes,
broad
patterns, and contextual factors.
The main point is that the
analysis of data collected from
each
method
must
be
conducted independently.
Step 2: Once the analysis of the
data generated by each data
collection method is completed,
the evaluator focuses on how the
analysis and findings from each
data set can inform, explain,
and/or strengthen findings from
the other data set. There are two
possible
primary
analytical
methods for doing this – and
sometimes both methods are
used in the same evaluation.
Again, the method used depends
on the purpose of the evaluation.
 In cases where more than one
method is used specifically to
strengthen
and
validate
findings for the same question
through a triangulation design,
the evaluator compares the
findings from the independent
analysis on each data set to
determine if there is a
convergence of findings. This
method is used when it is
critical to produce defensible
conclusions that can be used to
inform
major
program
decisions (e.g., end or extend a
program).
 To interpret or explain findings
from
quantitative
5
analysis,
evaluators use findings from
the analysis of qualitative data.
This method can provide a
richer analysis and set of
explanations affecting program
outcomes that enhance the
utility of the evaluation for
program
managers.
Conversely,
patterns
and
associations arising from the
analysis of quantitative data
can inform additional patterns
to look for in analyzing
qualitative data. The analysis
of qualitative data can also
enhance the understanding of
important program context
data. This method is often used
when program managers want
to know not only whether or
not a program is achieving its
intended results, but also, why
or why not.
WHEN FINDINGS
CONVERGE
DO
NOT
In cases where mixed-method
evaluations employ triangulation,
it is not unusual that findings
from the separate analysis of
each
data
set
do
not
automatically converge. If this
occurs, the evaluator must try to
resolve the conflict among
divergent findings. This is not a
disaster. Often this kind of
situation
can
present
an
opportunity to generate more
nuanced
explanations
and
important additional findings that
are of great value.
One method evaluators use when
findings from different methods
diverge is to carefully re-examine
the raw qualitative data through
a second and more in-depth
content analysis. This is done to
determine if there were any
factors or issues that were missed
when these data were first being
organized for analysis.
The
results of this third layer of
analysis can produce a deeper
understanding of the data, and
can then be used to generate
new interpretations.
In some
cases, other factors external to
the program might be discovered
through contextual analysis of
economic, social or political
conditions or an analysis of
operations and interventions
across program sites.
Another approach is to reanalyze
all the disaggregated data in
each data set separately, by
characteristics of the respondents
as appropriate to the study, such
as age, gender, educational
background, economic strata,
etc., and/or by geography/locale
of respondents.
The results of this analysis may
yield other information that can
help to resolve the divergence of
findings. In this case, the
evaluator should attempt to rank
order these factors in terms of
frequency of occurrence. This
further analysis will provide
additional explanations for the
variances in findings. While most
professionals build this type of
disaggregation into the analysis
of the data during the design
phase of the evaluation, it is
worth reexamining patterns from
disaggregated data.
Evaluators should also check for
data quality issues, such as the
validity of secondary data sources
or possible errors in survey data
from incomplete recording or
incorrect coding of responses.
(See TIPS 12: Data Quality
Standards.) If the evaluators are
still at the program site, it is
possible to resolve data quality
issues with limited follow-up data
collection by, for example,
conducting in-depth interviews
with key informants (if time and
budget permit).
In cases where an overall
summative program conclusion is
required, another analytical tool
that is used to resolve divergent
findings is the data synthesis
method.
(See Table 2.) This
method rates the strength of
findings generated from the
analysis of each data set based
on the intensity of the impact
(e.g., on a scale from very high
positive to very high negative)
and the quality and validity of the
data. An overall rating is assigned
for each data set, but different
weights can then be assigned to
different data sets if the evaluator
knows that certain data sources
or methods for collecting data
are stronger than others.
Ultimately, an index is created
based on the average of those
ratings to synthesize an overall
program effect on the outcome.
See McConney, Rudd and Ayres
(2002) to learn more about this
method.
REPORTING ON
MIXED-METHOD
EVALUATIONS
Mixed-method
evaluations
generate a great deal of data,
6
and, to profit from the use of
those methods, evaluators must
use and analyze all of the data
sets. Through the use of mixedmethod evaluations, findings and
conclusions can be enriched and
strengthened. Yet there is a
tendency to underuse, or even
not to use, all the data collected
for the evaluation. Evaluators can
rely too heavily on one particular
data source if it generates easily
digestible and understandable
information for a program
manager. For example, in many
cases data generated from
qualitative
methods
are
insufficiently analyzed. In some
cases only findings from one
source are reported.
One
way
to
prevent
underutilization of findings is to
write a statement of work that
provides the evaluator sufficient
time to analyze the data sets
from each method employed,
and hence to develop valid
findings, explanations, and strong
conclusions that a program
manager
can
use
with
confidence.
Additionally,
statements of work for evaluation
should require evidence of, and
reporting on, the analysis of data
sets from each method that was
used to collect data, or
methodological justification for
having discarded any data sets.
REFERENCES
Bamberger, Michael, Jim Rugh and Linda Mabry. Real World Evaluation: Working Under Budget,
Time, Data and Political Constraints, Chapter 13, ―Mixed-Method Evaluation,‖ pp. 303-322, Sage
Publications Inc., Thousand Oaks, CA, 2006.
Greene, Jennifer C. and Valerie J. Caracelli. ―Defining and Describing the Paradigm Issue in Mixedmethods Evaluation,” in Advances in Mixed-Method Evaluation: The Challenges and Benefits of
Integrating Diverse Paradigms, Green and Caracelli eds. New Directions for Evaluation. Josey-Bass
Publishers, No. 74, Summer 1997, pp 5-17.
Mark, Melvin M., Irwin Feller and Scott B. Button. ―Integrating Qualitative Methods in a
Predominantly Quantitative Evaluation: A Case Study and Some Reflections,‖ in Advances in
Mixed-Method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms, Green
and Caracelli eds. New Directions for Evaluation. Josey-Bass Publishers, No. 74, Summer 1997, pp
47-59.
McConney, Andrew, Andy Rudd, and Robert Ayres. ―Getting to the Bottom Line: A Method for
Synthesizing Findings Within Mixed-method Program Evaluations,‖ in American Journal of
Evaluation, Vol. 3, No. 2, 2002, pp. 121-140.
Teddlie, Charles and Abbas Tashakkori, Foundations of Mixed-methods Research: Integrating
Quantitative and Qualitative Approaches in the Behavioral Science, Sage Publications, Inc., Los
Angeles, 2009.
7
TABLE 1 – METHODS FOR ANALYZING MIXED-METHODS DATA1
Analytical
Method
Brief Description
Best for…
Parallel
Two or more data sets collected using a mix of
Triangulation designs to look for
methods (quantitative and qualitative) are analyzed
convergence of findings when the strength
independently. The findings are then combined or
of the findings and conclusions is critical,
integrated.
or to use analysis of qualitative data to
yield deeper explanations of findings from
quantitative data analysis.
Conversion
Two types of data are generated from one data source
Extending the findings of one data set, say,
beginning with the form (quantitative or qualitative) of
quantitative, to generate additional
the original data source that was collected. Then the
findings and/or to compare and potentially
data are converted into either numerical or narrative
strengthen the findings generated from a
data. A common example is the transformation of
complimentary set of, say, qualitative data.
qualitative narrative data into numerical data for
statistical analysis (e.g., on the simplest level,
frequency counts of certain responses).
Sequential
A chronological analysis of two or more data sets
Testing hypotheses generated from the
(quantitative and qualitative) where the results of the
analysis of the first data set.
analysis from the first data set are used to inform the
analysis of the second data set. The type of analysis
conducted on the second data set is dependent on the
outcome of the first data set.
Multilevel
Qualitative and quantitative techniques are used at
Evaluations where organizational units for
different levels of aggregation within a study from at
study are nested (e.g., patient, nurse,
least two data sources to answer interrelated evaluation
doctor, hospital, hospital administrator in
questions. One type of analysis (qualitative) is used at
an evaluation to understand the quality of
one level (e.g., patient) and another type of analysis
patient treatment).
(quantitative) is used in at least one other level (e.g.,
nurse).
Data
Synthesis
A multi-step analytical process in which: 1) a rating of
Providing a bottom-line measure in cases
program effectiveness using the analysis of each data
where the evaluation purpose is to provide
set is conducted (e.g., large positive effect, small
a summative program-wise conclusion
positive effect, no discernable effect, small negative
when findings from mixed-method
effect, large negative effect; 2) quality of evidence
evaluations using a triangulation strategy
assessments are conducted for each data set using
do not converge and appear to be
“criteria of worth” to rate the quality and validity of each
irresolvable, yet a defensible conclusion is
data set gathered; 3) using the ratings collected under
needed to make a firm program decision.
the first two steps, develop an aggregated equation for
Note: there may still be some divergence in
each outcome under consideration to assess the overall
the evaluation findings from mixed data
strength and validity of each finding; and 4) average
sets that the evaluator can still attempt to
outcome-wise effectiveness estimates to produce one
resolve and/or explore to further enrich the
overall program-wise effectiveness index.
analysis and findings.
1
See Teddlie and Tashakkori (2009) and Mark, Feller and Button (1997) for examples and further explanations of parallel data analysis.
See Teddlie and Tashakkori (2009) on conversion, sequential, multilevel, and fully integrated mixed-methods data analysis; and
McConney, Rudd, and Ayers (2002), for a further explanation of data synthesis analysis.
8
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including USAID’s
Office of Management Policy, Budget and Performance (MPBP). This publication was written by Dr.
Patricia Vondal of Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
9
NUMBER 17
1ST EDITION, 2010
TIPS
CONSTRUCTING AN EVALUATION
REPORT
ABOUT TIPS
INTRODUCTION
This TIPS has three purposes. First,
it provides guidance for evaluators
on the structure, content, and style
of evaluation reports. Second, it
offers
USAID
officials,
who
commission evaluations, ideas on
how to define the main deliverable.
Third, it provides USAID officials
with guidance on reviewing and
approving evaluation reports.
The main theme is a simple one: how
to make an evaluation report useful to
its readers. Readers typically include
a
variety
of
development
stakeholders and professionals; yet,
the most important are the
policymakers and managers who
need credible information for
program or project decision-making.
Part of the primary purpose of an
evaluation usually entails informing
this audience.
To be useful, an evaluation report
should address the evaluation
questions and issues with accurate
and data-driven findings, justifiable
conclusions,
and
practical
recommendations. It should reflect
the use of sound evaluation
methodology and data collection,
and report the limitations of each.
Finally, an evaluation should be
written with a structure and style
that promote learning and action.
Five common problems emerge in
relation to evaluation reports.
These problems are as follows:
• An unclear description of the
program strategy and the specific
results it is designed to achieve.
• Inadequate description of the
evaluation’s purpose, intended
uses, and the specific evaluation
questions to be addressed.
• Imprecise analysis and reporting
of quantitative and qualitative data
collected during the evaluation.
1
• A lack of clear distinctions
between findings and conclusions.
• Conclusions
that
are
not
grounded in the facts and
recommendations that do not
flow logically from conclusions.
This guidance offers tips that apply
to an evaluation report for any type
of evaluation — be it formative,
summative (or impact), a rapid
appraisal evaluation, or one using
more rigorous methods.
Evaluation reports should be readily
understood and should identify key
points clearly, distinctly, and
succinctly. (ADS 203.3.6.6)
A PROPOSED
REPORT OUTLINE
Table 1 presents a suggested outline
and approximate page lengths for a
typical evaluation report.
The
evaluation team can, of course,
modify this outline as needed. As
indicated in the table, however,
some elements are essential parts of
any report.
This outline can also help USAID
managers define the key deliverable
in an Evaluation Statement of Work
(SOW) (see TIPS 3: Preparing an
Evaluation SOW).
We will focus particular attention
on the section of the report that
covers findings, conclusions, and
recommendations. This section
represents the core element of the
evaluation report.
BEFORE THE
WRITING BEGINS
Before the report writing begins, the
evaluation team must complete two
critical tasks: 1) establish clear and
defensible findings, conclusions, and
recommendations
that
clearly
address the evaluation questions;
and 2) decide how to organize the
report in a way that conveys these
elements most effectively.
FINDINGS,
CONCLUSIONS, AND
RECOMMENDATIONS
One of the most important tasks in
constructing an evaluation report is
to organize the report into three
main elements: findings, conclusions,
and recommendations (see Figure
1). This structure brings rigor to
the evaluation and ensures that each
element can ultimately be traced
back to the basic facts. It is this
structure that sets evaluation apart
from other types of analysis.
Once the research stage of an
evaluation is complete, the team has
typically collected a great deal of
data in order to answer the
evaluation questions. Depending on
the methods used, these data can
include observations, responses to
survey questions, opinions and facts
from key informants, secondary data
from a ministry, and so on. The
team’s first task is to turn these raw
data into findings.
Suppose, for example, that USAID
has charged an evaluation team with
answering the following evaluation
question (among others):
“How adequate are the prenatal
services provided by the Ministry of
Health’s
rural
clinics
in
Northeastern District?”
To answer this question, their
research in the district included site
visits to a random sample of rural
clinics,
discussions
with
knowledgeable health professionals,
and a survey of women who have
used clinic prenatal services during
the past year. The team analyzed
the raw, qualitative data and
identified the following findings:
• Of the 20 randomly-sampled rural
clinics visited, four clinics met all
six established standards of care,
FIGURE 1.
ORGANIZING KEY ELEMENTS
OF THE EVALUATION
REPORT
Recommendations
Proposed actions for management
Conclusions
Interpretations and judgments
based on the findings
Findings
Empirical facts collected during the
evaluation
while the other 16 (80 percent)
failed to meet at least two
standards. The most commonly
unmet standard (13 clinics) was
“maintenance of minimum staffpatient ratios.”
• In 14 of the 16 clinics failing to
meet two or more standards, not
one of the directors was able to
state the minimum staff-patient
ratios for nurse practitioners,
nurses, and prenatal educators.
TYPICAL PROBLEMS WITH FINDINGS
Findings that:
1.
Are not organized to address the evaluation questions — the reader must
figure out where they fit.
2.
Lack precision and/or context —the reader cannot interpret their relative
strength.
Incorrect: “Some respondents said ’x,’ a few said ’y,’ and others said ’z.’”
Correct: “Twelve of the 20 respondents (60 percent) said ’x,’ five (25
percent) said ’y,’ and three (15 percent) said ’z.’ ”
3.
Mix findings and conclusions.
Incorrect: “The fact that 82 percent of the target group was aware of the
media campaign indicates its effectiveness.”
Correct: Finding: “Eighty-two percent of the target group was aware of the
media campaign.” Conclusion: “The media campaign was effective.”
2
TYPICAL PROBLEMS WITH
CONCLUSIONS
Conclusions that:
1.
2.
3.
4.
Restate findings.
Incorrect: “The project met its
performance targets with respect
to outputs and results.”
Correct: “The project’s strategy
was successful.”
Are vaguely stated.
Incorrect: “The project could
have been more responsive to its
target group.”
Correct: “The project failed to
address the different needs of
targeted women and men.”
Are based on only one of several
findings and data sources.
Include respondents’ conclusions,
which are really findings.
Incorrect: “All four focus groups
of project beneficiaries judged the
project to be effective.”
Correct: “Based on our focus
group data and quantifiable data on
key results indicators, we conclude
that the project was effective.”
• Of 36 women who had used their
rural clinics’ prenatal services
during the past year, 27 (76
percent) stated that they were
“very
dissatisfied”
or
“dissatisfied,” on a scale of 1-5
from “very dissatisfied” to “very
satisfied.” The most frequently
cited reason for dissatisfaction
was “long waits for service” (cited
by 64 percent of the 27
dissatisfied women).
• Six of the seven key informants
who offered an opinion on the
adequacy of prenatal services for
the rural poor in the district
noted that an insufficient number
of prenatal care staff was a “major
problem” in rural clinics.
These findings are the empirical facts
collected by the evaluation team.
Evaluation findings are analogous to
the evidence presented in a court of
law or a patient’s symptoms
identified during a visit to the
doctor. Once the evaluation team
has correctly laid out all the findings
against each evaluation question,
only then should conclusions be
drawn for each question. This is
where many teams tend to confuse
findings and conclusions both in
their analysis and in the final report.
Conclusions represent the team’s
judgments based on the findings.
These are analogous to a court
jury’s decision to acquit or convict
based on the evidence presented or
a doctor’s diagnosis based on the
symptoms. The team must keep
findings and conclusions distinctly
separate
from
each
other.
However, there must also be a clear
and logical relationship between
findings and conclusions.
In our example of the prenatal
services evaluation, examples of
reasonable conclusions might be as
follows:
• In general, the levels of prenatal
care staff in Northeastern
District’s
rural
clinics
are
insufficient.
• The Ministry of Health’s periodic
informational bulletins to clinic
directors regarding the standards
of prenatal care are not sufficient
to ensure that standards are
understood and implemented.
However, sometimes the team’s
findings from different data sources
are not so clear-cut in one direction
as this one. In those cases, the team
must weigh the relative credibility of
the data sources and the quality of
the data, and make a judgment call.
The team might state that a
definitive conclusion cannot be
made, or it might draw a more
3
guarded conclusion such as the
following:
“The preponderance of the
evidence suggests that prenatal
care is weak.”
The team should never omit
contradictory findings from its
analysis and report in order to have
more
definitive
conclusions.
Remember,
conclusions
are
interpretations and judgments made
TYPICAL PROBLEMS WITH
RECOMMENDATIONS
Recommendations that:
1. Are unclear about the action to be
taken.
Incorrect: “Something needs to be
done to improve extension
services.”
Correct: “To improve extension
services, the Ministry of Agriculture
should implement a comprehensive
introductory training program for all
new extension workers and annual
refresher training programs for all
extension workers. “
2. Fail to specify who should take
action.
Incorrect: “Sidewalk ramps for the
disabled should be installed.”
Correct: “Through matching grant
funds from the Ministry of Social
Affairs, municipal governments
should install sidewalk ramps for the
disabled.”
3. Are not supported by any findings
and conclusions
4. Are not realistic with respect to
time and/or costs.
Incorrect: The Ministry of Social
Affairs should ensure that all
municipal sidewalks have ramps for
the disabled within two years.
Correct: The Ministry of Social
Affairs should implement a gradually
expanding program to ensure that all
municipal sidewalks have ramps for
the disabled within 15 years.
on the basis of the findings.
Sometimes we see reports that
include conclusions derived from
preconceived notions or opinions
developed
through
experience
gained outside the evaluation,
especially by members of the team
who have substantive expertise on a
particular topic. We do not
recommend this, because it can
distort the evaluation. That is, the
role of the evaluator is to present
the findings, conclusions, and
recommendations in a logical order.
Opinions outside this framework
are then, by definition, not
substantiated by the facts at hand. If
any of these opinions are directly
relevant to the evaluation questions
and come from conclusions drawn
from prior research or secondary
sources, then the data upon which
they are based should be presented
among the evaluation’s findings.
FIGURE 3
OPTIONS FOR REPORTING
FINDINGS, CONCLUSIONS,
AND RECOMMENDATIONS
OPTION 1
FINDINGS
Evaluation Question 1
CONCLUSIONS
OPTION 2
EVALUATION
QUESTION 1
Findings
Conclusions
Recommendations
EVALUATION
QUESTION 2
Findings
RECOMMENDATIONS Conclusions
Evaluation Question 2 Recommendations
OPTION 3
Mix the two approaches. Identify which
evaluation questions are distinct and which
are interrelated. For distinct questions, use
option 1 and for the latter, use option 2.
FIGURE 2
Tracking the linkages is one way to help ensure a credible report, with
information that will be useful.
Evaluation Question #1:
FINDINGS
CONCLUSIONS
RECOMMENDATIONS
XXXXXX
YYYYYY
ZZZZZZ
XXXXXX
ZZZZZZ
XXXXXX
YYYYYY
Once conclusions are complete, the
team is ready to make its
recommendations.
Too often
recommendations do not flow from
the team’s conclusions or, worse,
they are not related to the original
evaluation purpose and evaluation
questions. They may be good ideas,
but they do not belong in this
section of the report.
As an
alternative, they could be included in
an annex with a note that they are
derived
from
coincidental
observations made by the team or
from team members’ experiences
elsewhere.
Using our example related to rural
health clinics, a few possible
recommendations could emerge as
follows:
• The
Ministry
of
Health’s
Northeastern
District
office
should develop and implement an
annual prenatal standards-of-care
training program for all its rural
clinic directors. The program
would cover….
• The Northeaster District office
should
conduct
a
formal
assessment of prenatal care
staffing levels in all its rural clinics.
• Based on the assessment, the
4
ZZZZZZ
Northeastern
District
office
should establish and implement a
five-year plan for hiring and
placing needed prenatal care staff
in its rural clinics on a mostneedy-first basis.
Although
the
basic
recommendations should be derived
from conclusions and findings, this is
where the team can include ideas
and options for implementing
recommendations that may be based
on their substantive expertise and
best
practices
drawn
from
experience outside the evaluation
itself. Usefulness is paramount.
When developing recommendations,
consider practicality. Circumstances
or resources may limit the extent to
which a recommendation can be
implemented. If practicality is an
issue — as is often the case — the
evaluation team may need to ramp
down recommendations, present
them in terms of incremental steps,
or suggest other options. In order
to be useful, it is essential that
recommendations be actionable or,
in other words, feasible in light of
the human, technical, and financial
resources available.
Weak connections between findings,
conclusions, and recommendations
can undermine the user’s confidence
in evaluation results. As a result, we
encourage teams—or, better yet, a
colleague who has not been
involved—to review the logic before
beginning to write the report. For
each evaluation question, present all
the findings, conclusions, and
recommendations in a format similar
to the one outlined in Figure 2.
Starting with the conclusions in the
center, track each one back to the
findings that support it, and decide
whether the findings truly warrant
the conclusion being made. If not,
revise the conclusion as needed.
Then track each recommendation to
the conclusion(s) from which it
flows, and revise if necessary.
CHOOSE THE BEST
APPROACH FOR
STRUCTURING THE
REPORT
Depending on the nature of the
evaluation questions and the
findings,
conclusions,
and
recommendations, the team has a
few options for structuring this part
of the report (see Figure 3). The
objective is to present the report in
a way that makes it as easy as
possible for the reader to digest all
of the information. Options are
discussed below.
Option 1- Distinct Questions
If all the evaluation questions are
distinct from one another and the
relevant findings, conclusions, and
recommendations do not cut across
questions, then one option is to
organize the report around each
evaluation question. That is, each
question will include a section
including its relevant findings,
conclusions, and recommendations.
Option 2- Interrelated
Questions
If, however, the questions are
closely interrelated and there are
findings,
conclusions,
and/or
recommendations that apply to
more than one question, then it may
be preferable to put all the findings
for all the evaluation questions in
one section, all the conclusions in
another,
and
all
the
recommendations in a third.
Option 3- Mixed
If the situation is mixed—where a
few but not all the questions are
closely interrelated—then use a
mixed
approach.
Group
the
interrelated questions and their
findings,
conclusions,
and
recommendations into one subsection, and treat the stand-alone
questions and their respective
findings,
conclusions,
and
recommendations in separate subsections.
The important point is that the team
should be sure to keep findings,
conclusions, and recommendations
separate and distinctly labeled as such.
Finally, some evaluators think it
more useful to present the
conclusions first, and then follow
with the findings supporting them.
This helps the reader see the
“bottom line” first and then make a
judgment as to whether the
conclusions are warranted by the
findings.
OTHER KEY
SECTIONS OF THE
REPORT
THE EXECUTIVE
SUMMARY
The Executive Summary should
stand alone as an abbreviated
version of the entire report. Often
it is the only thing that busy
managers read. The Executive
Summary should be a “mirror
image” of the full report—it should
contain no new information that is
not in the main report.
This
principle also applies to making the
Executive Summary and the full
report equivalent with respect to
presenting positive and negative
evaluation results.
Although all sections of the full
report are summarized in the
Executive Summary, less emphasis is
given to an overview of the project
and the description of the evaluation
purpose and methodology than is
given to the findings, conclusions,
and recommendations. Decisionmakers
are
generally
more
interested in the latter.
The Executive Summary should be
written after the main report has
been drafted. Many people believe
that a good Executive Summary
should not exceed two pages, but
there is no formal rule in USAID on
this. Finally, an Executive Summary
should be written in a way that will
entice interested stakeholders to go
on to read the full report.
DESCRIPTION OF THE
PROJECT
Many evaluation reports give only
cursory
attention
to
the
development
problem
(or
opportunity) that motivated the
project in the first place, or to the
5
FIGURE 4. SUMMARY OF EVALUATION DESIGN AND METHODS (an illustration)
Evaluation
Question
Type of Analysis
Conducted
Data Sources and
Methods Used
Type and Size of
Sample
1. How
adequate are
the prenatal
services
provided by
the Ministry
of Health’s
(MOH) rural
clinics in
Northeastern
District?
Comparison of rural
clinics’ prenatal
service delivery to
national standards
MOH manual of rural
clinic standards of care
Structured observations
and staff interviews at
rural clinics
Twenty clinics,
randomly sampled
from 68 total in
Northeastern District
Three of the originally
sampled clinics were closed
when the team visited. To
replace each, the team
visited the closest open
clinic. As a result, the sample
was not totally random.
Description, based on
a content analysis of
expert opinions
Key informant interviews
with health care experts in
the district and the MOH
Ten experts identified
by project & MOH
staff
Only seven of the 10
experts had an opinion
about prenatal care in the
district.
Description and
comparison of ratings
among women in the
district and two other
similar rural districts
In-person survey of
recipients of prenatal
services at clinics in the
district and two other
districts
Random samples of 40
women listed in clinic
records as having
received prenatal
services during the
past year from each of
the three districts’
clinics
Of the total 120 women
sampled, the team was able
to conduct interviews with
only 36 in the district, and
24 and 28 in the other two
districts. The levels of
confidence for generalizing
to the populations of service
recipients were __, __, and
__, respectively.
“theory of change” that underpins
USAID’s intervention. The “theory
of change” includes what the project
intends to do and the results which
the activities are intended to
produce. TIPS 13: Building a Results
Framework is a particularly useful
reference and provides additional
detail on logic models.
If the team cannot find a description
of these hypotheses or any model of
the project’s cause-and-effect logic
such as a Results Framework or a
Logical Framework, this should be
noted. The evaluation team will
then have to summarize the project
strategy in terms of the “if-then”
propositions that show how the
project designers envisioned the
interventions as leading to desired
results.
In describing the project, the
evaluation team should be clear
about what USAID tried to improve,
eliminate, or otherwise change for
the better. What was the “gap”
between conditions at the start of
the project and the more desirable
conditions that USAID wanted to
establish with the project? The team
should indicate whether the project
design documents and/or the recall
of interviewed project designers
offered a clear picture
of the specific economic and social
factors that contributed to the
problem — with baseline data, if
available. Sometimes photographs
and
maps
of
before-project
conditions, such as the physical
characteristics and locations of rural
prenatal clinics in our example, can
be used to illustrate the main
problem(s).
It is equally important to include
basic information about when the
project was undertaken, its cost, its
intended beneficiaries, and where it
was implemented (e.g., country-wide
or only in specific districts). It can
be particularly useful to include a
6
Limitations
map that shows the project’s target
areas.
A good description also identifies
the organizations that implement the
project, the kind of mechanism used
(e.g., contract, grant, or cooperative
agreement), and whether and how
the project has been modified during
implementation.
Finally,
the
description
should
include
information about context, such as
conflict or drought, and other
government or donor activities
focused on achieving the same or
parallel results.
THE EVALUATION
PURPOSE AND
METHODOLOGY
The credibility of an evaluation
team’s findings, conclusions, and
recommendations rests heavily on
the quality of the research design, as
well as on data collection methods
and analysis used. The reader needs
to understand what the team did
and why in order to make informed
judgments
about
credibility.
Presentation of the evaluation design
and methods is often best done
through a short
summary in the text of the report
and a more detailed methods annex
that
includes
the
evaluation
instruments. Figure 4 provides a
sample summary of the design and
methodology that can be included in
the body of the evaluation report.
From a broad point of view, what
research design did the team use to
answer each evaluation question?
Did the team use description (e.g.,
to document what happened),
comparisons (e.g., of baseline data
or targets to actual data, of actual
practice to standards, among target
sub-populations or locations), or
cause-effect research (e.g., to
determine whether the project
made a difference)? To do causeeffect analysis, for example, did the
team use one or more quasiexperimental approaches, such as
time-series analysis or use of nonproject comparison groups (see
TIPS 11: The Role of Evaluation)?
More specifically, what data collection
methods did the team use to get the
evidence needed for each evaluation
question? Did the team use key
informant interviews, focus groups,
surveys,
on-site
observation
methods, analyses of secondary data,
and other methods? How many
people did they interview or survey,
how many sites did they visit, and
how did they select their samples?
and developing the findings and
conclusions that follow in the
report. The reader needs to know
these limitations in order to make
informed judgments about the
evaluation’s
credibility
and
usefulness.
Most evaluations suffer from one or
more constraints that affect the
comprehensiveness and validity of
findings and conclusions. These may
include overall limitations on time
and
resources,
unanticipated
problems in reaching all the key
informants and survey respondents,
unexpected problems with the
quality of secondary data from the
host-country government, and the
like. In the methodology section, the
team
should
address
these
limitations and their implications for
answering the evaluation questions
When writing its report, the
evaluation team must always
remember the composition of its
audience. The team is writing for
policymakers,
managers,
and
takeholders, not for fellow social
science
researchers
or
for
publication in a professional journal.
To that end, the style of writing
should make it as easy as possible
for the intended audience to
understand and digest what the
team is presenting. For further
suggestions on writing an evaluation
in reader-friendly style, see Table 2.
7
READER-FRIENDLY
STYLE
TABLE 1. SUGGESTED OUTLINE FOR AN EVALUATION REPORT1
Element
Approximate
Number of
Pages
Description and Tips for the Evaluation Team
Title Page
1 (but no page
number)
Essential. Should include the words “U.S. Agency for International
Development” with the acronym “USAID,” the USAID logo, and the
project/contract number under which the evaluation was conducted. See
USAID Branding and Marking Guidelines (http://www.usaid.gov/branding/)
for logo and other specifics. Give the title of the evaluation; the name of
the USAID office receiving the evaluation; the name(s), title(s), and
organizational affiliation(s) of the author(s); and the date of the report.
Contents
As needed, and
start with
Roman
numeral ii.
Essential. Should list all the sections that follow, including Annexes. For
multi-page chapters, include chapter headings and first- and second-level
headings. List (with page numbers) all figures, tables, boxes, and other
titled graphics.
Foreword
1
Optional. An introductory note written by someone other than the
author(s), if needed. For example, it might mention that this evaluation is
one in a series of evaluations or special studies being sponsored by USAID.
Acknowledgements
1
Optional. The authors thank the various people who provided support
during the evaluation.
Preface
1
Optional. Introductory or incidental notes by the authors, but not material
essential to understanding the text. Acknowledgements could be included
here if desired.
Executive Summary
2-3; 5 at most
Essential, unless the report is so brief that a summary is not needed. (See
discussion on p. 5)
Glossary
1
Optional. Is useful if the report uses technical or project-specific
terminology that would be unfamiliar to some readers.
Acronyms and
Abbreviations
1
Essential, if they are used in the report. Include only those acronyms that
are actually used. See Table 3 for more advice on using acronyms.
I. Introduction
5-10 pages,
starting with
Arabic numeral
1.
Optional. The two sections listed under Introduction here could be
separate, stand-alone chapters. If so, a separate Introduction may not be
needed.
Description of the
Project
The Evaluation Purpose
and Methodology
II. Findings, Conclusions,
and Recommendations
Essential. Describe the context in which the USAID project took place—
e.g., relevant history, demography, political situation, etc. Describe the
specific development problem that prompted USAID to implement the
project, the theory underlying the project, and details of project
implementation to date. (See more tips on p. 6.)
Essential. Describe who commissioned the evaluation, why they
commissioned it, what information they want, and how they intend to use
the information (and refer to the Annex that includes the Statement of
Work). Provide the specific evaluation questions, and briefly describe the
evaluation design and the analytical and data collection methods used to
answer them. Describe the evaluation team (i.e., names, qualifications, and
roles), what the team did (e.g., reviewed relevant documents, analyzed
secondary data, interviewed key informants, conducted a survey, conducted
site visits), and when and where they did it. Describe the major limitations
encountered in data collection and analysis that have implications for
reviewing the results of the evaluation. Finally, refer to the Annex that
provides a fuller description of all of the above, including a list of
documents/data sets reviewed, a list of individuals interviewed, copies of
the data collection instruments used, and descriptions of sampling
procedures (if any) and data analysis procedures. (See more tips on p. 6.)
20-30 pages
Essential. However, in some cases, the evaluation user does not want
recommendations, only findings and conclusions. This material may be
8
TABLE 1. SUGGESTED OUTLINE FOR AN EVALUATION REPORT1
Element
Approximate
Number of
Pages
Description and Tips for the Evaluation Team
organized in different ways and divided into several chapters. (A detailed
discussion of developing defensible findings, conclusions, and
recommendations and structural options for reporting them is on p 2 and
p. 5)
III. Summary of
Recommendations
1-2 pages
Essential or optional, depending on how findings, conclusions and
recommendations are presented in the section above. (See a discussion of
options on p. 4.) If all the recommendations related to all the evaluation
questions are grouped in one section of the report, this summary is not
needed. However, if findings, conclusions, and recommendations are
reported together in separate sections for each evaluation question, then a
summary of all recommendations, organized under each of the evaluation
questions, is essential.
IV. Lessons Learned
As needed
Required if the SOW calls for it; otherwise optional. Lessons learned
and/or best practices gleaned from the evaluation provide other users, both
within USAID and outside, with ideas for the design and implementation of
related or similar projects in the future.
Some are
essential and
some are
optional as
noted.
Essential. Lets the reader see exactly what USAID initially expected in the
evaluation.
Annexes
Statement of Work
Evaluation Design and
Methodology
Essential. Provides a more complete description of the evaluation
questions, design, and methods used. Also includes copies of data
collection instruments (e.g., interview guides, survey instruments, etc.) and
describes the sampling and analysis procedures that were used.
List of Persons
Interviewed
Essential. However, specific names of individuals might be withheld in order
to protect their safety.
List of Documents
Reviewed
Essential. Includes written and electronic documents reviewed, background
literature, secondary data sources, citations of websites consulted.
Dissenting Views
If needed. Include if a team member or a major stakeholder does not agree
with one or more findings, conclusions, or recommendations.
Recommendation
Action Checklist
Optional. As a service to the user organization, this chart can help with
follow-up to the evaluation. It includes a list of all recommendations
organized by evaluation question, a column for decisions to accept or reject
each recommendation, a column for the decision maker’s initials, a column
for the reason a recommendation is being rejected, and, for each accepted
recommendation, columns for the actions to be taken, by when, and by
whom.
1
The guidance and suggestions in this table were drawn from the writers’ experience and from the “CDIE Publications
Style Guide: Guidelines for Project Managers, Authors, & Editors,” compiled by Brian Furness and John Engels, December
2001. The guide, which includes many tips on writing style, editing, referencing citations, and using Word and Excel is
available online at http://kambing.ui.ac.id/bebas/v01/DEC-USAID/Other/publications-style-guide.pdf. Other useful
guidance: ADS 320 (http://www.usaid.gov/policy/ads/300/320.pdf ; http://www.usaid.gov/branding; and
http://www.usaid.gov/branding/Graphic Standards Manual.pdf.
9
TABLE 2. THE QUICK REFERENCE GUIDE FOR A READER-FRIENDLY TECHNICAL STYLE
Writing Style—
Keep It Simple
and Correct!
Avoid meaningless precision. Decide how much precision is really necessary. Instead of “62.45
percent,” might “62.5 percent” or “62 percent” be sufficient? The same goes for averages and other
calculations.
Use technical terms and jargon only when necessary. Make sure to define them for the unfamiliar
readers.
Don’t overuse footnotes. Use them only to provide additional information which, if included in the
text, would be distracting and cause a loss of the train of thought.
Use Tables,
Charts and Other
Graphics to
Enhance
Understanding
Avoid long, “data-dump”paragraphs filled with numbers and percentages. Use tables, line graphs, bar
charts, pie charts, and other visual displays of data, and summarize the main points in the text. In
addition to increasing understanding, these displays provide visual relief from long narrative tracts.
Be creative—but not too creative. Choose and design tables and charts carefully with the reader in
mind.
Make every visual display of data a self-contained item. It should have a meaningful title and headings
for every column; a graph should have labels on each axis; a pie or bar chart should have labels for
every element.
Choose shades and colors carefully. Expect that consumers will reproduce the report in black and
white and make copies of copies. Make sure that the reader can distinguish clearly among colors or
shades among multiple bars and pie-chart segments. Consider using textured fillings (such as hatch
marks or dots) rather than colors or shades.
Provide “n’s” in all displays which involve data drawn from samples or populations. For example,
the total number of cases or survey respondents should be under the title of a table (n = 100). If a
table column includes types of responses from some, but not all, survey respondents to a specific
question, say, 92 respondents, the column head should include the total number who responded to
the question (n = 92).
Refer to every visual display of data in the text. Present it after mentioning it in the text and as soon
after as practical, without interrupting paragraphs.
Number tables and figures separately, and number each consecutively in the body of the report.
Consult the CDIE style guide for more detailed recommendations on tables and graphics.
Punctuate the
Text with Other
Interesting
Features
Put representative quotations gleaned during data collection in text boxes. Maintain balance
between negative and positive comments to reflect the content of the report. Identify the sources
of all quotes. If confidentiality must be maintained, identify sources in general terms, such as “a clinic
care giver” or “a key informant.”
Provide little “stories” or cases that illustrate findings. For example, a brief anecdotal story in a text
box about how a woman used a clinic’s services to ensure a healthy pregnancy can enliven, and
humanize, the quantitative findings.
Use photos and maps where appropriate. For example, a map of a district with all the rural clinics
providing prenatal care and the concentrations of rural residents can effectively demonstrate
adequate or inadequate access to care.
Don’t overdo it. Strike a reader-friendly balance between the main content and illustrative material.
In using illustrative material, select content that supports main points, not distracts from them.
Finally…
Remember that the reader’s need to understand, not the writer’s need to impress, is paramount.
Be consistent with the chosen format and style throughout the report.
Sources: “CDIE Publications Style Guide: Guidelines for Project Managers, Authors, & Editors,” compiled by Brian
Furness and John Engels, December 2001 (http://kambing.ui.ac.id/bebas/v01/DEC-USAID/Other/publications-styleguide.pdf); USAID’s Graphics Standards Manual (http://www.usaid.gov/branding/USAID_Graphic_Standards_Manual.pdf);
and the authors extensive experience with good and difficult-to-read evaluation reports.
10
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including Gerry Britan and
written by Larry Beyna of Management Systems International (MSI).
Tel: (202) 712-1158
gbritan@usaid.gov
11
NUMBER 18
1ST EDITION, 2010
TIPS
CONDUCTING DATA QUALITY ASSESSMENTS
ABOUT TIPS
THE PURPOSE OF
THE DATA QUALITY
ASSESSMENT
Data quality assessments (DQAs)
help managers to understand how
confident they should be in the
data used to manage a program and
report on its success. USAID’s
ADS notes that the purpose of the
Data Quality Assessment is to:
“…ensure that the USAID
Mission/Office
and
Assistance Objective (AO)
Team are aware of the
strengths and weaknesses of
the data, as determined by
applying the five data quality
standards …and are aware
of the extent to which the
data integrity can be trusted
to influence management
decisions.” (ADS 203.3.5.2)
This purpose is important to keep
in mind when considering how to
do a data quality assessment. A
data quality assessment is of little
use unless front line managers
comprehend key data quality issues
and are able to improve the
THE DATA QUALITY
STANDARDS
Five key data quality standards are
used to assess quality. These are:
• Validity
• Reliability
• Precision
• Integrity
• Timeliness
A more detailed discussion of each
standard is included in TIPS 12:
Data Quality Standards.
WHAT IS REQUIRED?
USAID POLICY
While managers are required to
understand data quality on an
ongoing basis, a data quality
assessment must also be conducted
at least once every three years for
those
data
reported
to
Washington. As a matter of good
management, program managers
may decide to conduct DQAs
more frequently or for a broader
range of data where potential
issues emerge.
The ADS does not prescribe a
specific way to conduct a DQA. A
variety of approaches can be used.
Documentation may be as simple
1
as a memo to the files, or it could
take the form of a formal report.
The most appropriate approach
will
reflect
a
number
of
considerations,
such
as
management need, the type of data
collected, the data source, the
importance of the data, or
suspected data quality issues. The
key is to document the findings,
whether formal or informal.
A DQA focuses on applying the
data
quality
standards
and
examining
the
systems
and
approaches for collecting data to
determine whether they are likely
to produce high quality data over
time. In other words, if the data
quality standards are met and the
data collection methodology is well
designed, then it is likely that good
quality data will result.
This “systematic approach” is
valuable because it assesses a
broader set of issues that are likely
to ensure data quality over time (as
opposed to whether one specific
number is accurate or not). For
example, it is possible to report a
number correctly, but that number
may not be valid1 as the following
example demonstrates.
Example:
A program works
across a range of municipalities
(both urban and rural).
It is
reported that local governments
have increased revenues by 5%.
These data may be correct.
However, if only major urban areas
have been included, these data are
not valid.
That is, they do not
measure the intended result.
VERIFICATION OF DATA
Verification of data means that the
reviewer follows a specific datum
to its source, confirming that it has
supporting documentation and is
accurate—as is often done in
audits.
The DQA may not
necessarily verify that all individual
numbers reported are accurate.
The ADS notes that when assessing
data from partners, the DQA
should focus on “the apparent
accuracy and consistency of the
data.” As an example, Missions
often report data on the number of
individuals trained. Rather than
verifying each number reported,
the DQA might examine each
project’s system for collecting and
maintaining those data. If there is a
good system in place, we know
that it is highly likely that the data
produced will be of high quality.
“…data used for
management purposes
have different standards
than data used for
research.
Having said this, it is certainly
advisable to periodically verify
actual data as part of the larger
Project managers may:
Choose a few indicators to verify
periodically throughout the course
of the year.
Occasionally spot check data (for
example, when visiting the field).
HOW GOOD DO DATA
HAVE TO BE?
Refer to TIPS 12: Data Quality
Standards for a full discussion of all
the data quality standards.
1
In development, there are rarely
perfect data. Moreover, data used
for management purposes have
different standards than data used
2
for research. There is often a
direct trade-off between cost and
quality.
Each manager is
responsible for ensuring the highest
quality data possible given the
resources and the management
context. In some cases, simpler,
lower-cost approaches may be
most appropriate. In other cases,
where indicators measure progress
in major areas of investment,
higher data quality is expected.
OPTIONS AND
APPROACHES FOR
CONDUCTING DQAS
A data quality assessment is both a
process for reviewing data to
understand
strengths
and
weaknesses
as
well
as
documentation. A DQA can be
done in a variety of ways ranging
from the more informal to the
formal (see Figure 1).
In our
experience, a combination of
informal, on-going and systematic
assessments work best, in most
cases, to ensure good data quality.
INFORMAL OPTIONS
Informal approaches can be ongoing or driven by specific issues as
they emerge.
These approaches
depend more on the front line
manager’s in-depth knowledge of
the program.
Findings are
documented by the manager in
memos
or
notes
in
the
Performance Management Plan
(PMP).
Example:
An
implementer
reports
that
civil
society
organizations (CSOs) have initiated
50 advocacy campaigns.
This
number seems unusually high. The
project
manager
calls
the
Implementer to understand why
the number is so high in
FIGURE 1. OPTIONS FOR CONDUCTING DATA QUALITY ASSESSMENTS- THE CONTINUUM
Informal Options
• Conducted internally by the
AO team
• Ongoing (driven by
emerging and specific
issues)
• More dependent on the
AO team and individual
manager’s expertise &
knowledge of the program
• Conducted by the program
manager
• Product: Documented in
memos, notes in the PMP
comparison to previously reported
numbers and explores whether a
consistent
methodology
for
collecting the data has been used
(i.e., whether the standard of
reliability has been met).
The
project manager documents his or
her findings in a memo and
maintains that information in the
files.
Informal approaches should be
incorporated into Mission systems
as a normal part of performance
management. The advantages and
disadvantages of this approach are
as follows:
Advantages
• Managers
incorporate
data
quality as a part of on-going
work processes.
• Issues can be addressed and
corrected quickly.
• Managers establish a principle
that data quality is important.
Disadvantages
• It is not systematic and may not
be complete. That is, because
informal
assessments
are
normally driven by more
Semi-Formal Partnership
• Draws on both
management expertise and
M&E expertise
• Periodic & systematic
• Facilitated and coordinated
by the M&E expert, but AO
team members are active
participants
• Product: Data Quality
Assessment Report
immediate
management
concerns, the manager may miss
larger issues that are not readily
apparent (for example, whether
the data are attributable to
USAID programs).
• There is no comprehensive
document that addresses the
DQA requirement.
• Managers may not have enough
expertise to identify more
complicated data quality issues,
audit
vulnerabilities,
and
formulate solutions.
SEMI-FORMAL /
PARTNERSHIP OPTIONS
Semi-formal or partnership options
are characterized by a more
periodic and systematic review of
data quality. These DQAs should
ideally be led and conducted by
USAID staff. One approach is to
partner
a
monitoring
and
evaluation (M&E) expert with the
Mission’s AO team to conduct the
assessment jointly.
The M&E
expert can organize the process,
develop
standard
approaches,
facilitate
sessions,
assist
in
identifying potential data quality
issues and solutions, and may
3
Formal Options
• Driven by broader
programmatic needs, as
warranted
• More dependent on
external technical expertise
and/or specific types of data
expertise
• Product: Either a Data
Quality Assessment report
or addressed as a part of
another report
document the outcomes of the
assessment. This option draws on
the experience of AO team
members as well as the broader
knowledge and skills of the M&E
expert.
Engaging front line
mangers in the DQA process has
the additional advantage of making
them more aware of the strengths
and weaknesses of the data—one
of the stated purposes of the DQA.
The advantages and disadvantages
of this approach are summarized
below:
Advantages
• Produces a systematic and
comprehensive report with
specific recommendations for
improvement.
• Engages AO team members in
the data quality assessment.
• Draws on the complementary
skills of front line managers and
M&E experts.
• Assessing data quality is a matter
of understanding trade-offs and
context in terms of deciding
what data is “good enough” for a
program. An M&E expert can be
useful in guiding AO team
members through this process in
order to ensure that audit
vulnerabilities are adequately
addressed.
• Does not require
external team.
a
large
These types of data quality
assessments require a high degree
of rigor and specific, in-depth
technical expertise.
Advantages
and disadvantages are as follow:
Disadvantages
Advantages
• The Mission may use an internal
M&E expert or hire someone
from the outside.
However,
hiring an outside expert will
require additional resources, and
external contracting requires
some time.
• Produces a systematic and
comprehensive assessment, with
specific recommendations.
• Examines data quality issues with
rigor and based on specific, indepth technical expertise.
• Because of the additional time
and planning required, this
approach is less useful for
addressing immediate problems.
• Fulfills two important purposes,
in that it can be designed to
improve data collection systems
both within USAID and for the
beneficiary.
FORMAL OPTIONS
Disadvantages
At the other end of the continuum,
there may be a few select situations
where Missions need a more
rigorous and formal data quality
assessment.
• Often conducted by an external
team of experts, entailing more
time and cost than other
options.
Example:
A Mission invests
substantial funding into a highprofile program that is designed to
increase the efficiency of water use.
Critical performance data comes
from the Ministry of Water, and is
used both for performance
management and reporting to key
stakeholders,
including
the
Congress. The Mission is unsure as
to the quality of those data. Given
the high level interest and level of
resources invested in the program,
a data quality assessment is
conducted by a team including
technical experts to review data
and
identify
specific
recommendations
for
improvement. Recommendations
will be incorporated into the
technical assistance provided to the
Ministry to improve their own
capacity to track these data over
time.
• Generally
involvement
managers.
less
direct
by front line
• Often examines data through a
very technical lens.
It is
important to ensure that
broader management issues are
adequately addressed.
THE PROCESS
The Mission will also have to
determine
whether
outside
assistance is required.
Some
Missions have internal M&E staff
with the appropriate skills to
facilitate this process.
Other
Missions may wish to hire an
outside M&E expert(s) with
experience in conducting DQAs.
AO team members should also be
part of the team.
DATA SOURCES
Primary Data: Collected directly by
USAID.
Secondary Data: Collected from and
other sources, such as implementing
partners, host country governments,
other donors, etc.
STEP 2. DEVELOP AN
OVERALL APPROACH
AND SCHEDULE
The team leader must convey the
objectives, process, and schedule
for conducting the DQA to team
members. This option is premised
on the idea that the M&E expert(s)
work closely in partnership with
AO
team
members
and
implementing partners to jointly
assess data quality. This requires
active participation and encourages
managers to fully explore and
understand the strengths and
weaknesses of the data.
For purposes of this TIPS, we will
outline a set of illustrative steps for
the middle (or semi-formal/
partnership) option.
In reality,
these steps are often iterative.
INDICATORS TO BE
INCLUDED IN THE
REVIEW
DQA TEAM
It is helpful to compile a list of all
indicators that will be included in
the DQA. This normally includes:
Identify one person to lead the
DQA process for the Mission. This
person is often the Program
Officer or an M&E expert. The
leader is responsible for setting up
the
overall
process
and
coordinating with the AO teams.
4
• All indicators reported to
USAID/Washington (required).
• Any indicators with suspected
data quality issues.
• Indicators for program areas
that are of high importance.
This list can also function as a
central guide as to how each
indicator is assessed and to
summarize where follow-on action
is needed.
STEP 4. CATEGORIZE
INDICATORS
With the introduction of standard
indicators,
the
number
of
indicators that Missions report to
USAID/Washington has increased
substantially. This means that it is
important to develop practical and
streamlined
approaches
for
conducting DQAs. One way to do
this is to separate indicators into
two categories, as follows:
Outcome Level Indicators
Outcome level indicators measure
AOs or Intermediate Results (IRs).
Figure 2 provides examples of
indicators at each level.
The
standards for good data quality are
applied to results level data in
order to assess data quality.
The data quality assessment
worksheet (see Table 1) has been
developed as a tool to assess each
indicator against each of these
standards.
Output Indicators
Many of the data quality standards
are not applicable to output
indicators in the same way as
outcome level indicators.
For
example, the number of individuals
trained by a project is an output
indicator. Whether data are valid,
timely, or precise is almost never
an issue for this type of an
indicator. However, it is important
to ensure that there are good data
collection and data maintenance
systems in place. Hence, a simpler
and more streamlined approach
can be used to focus on the most
relevant issues. Table 2 outlines a
sample matrix for assessing output
indicators. This matrix:
• Identifies the indicator.
• Clearly outlines
collection method.
the
data
• Identifies key data quality issues.
• Notes whether further action is
necessary.
• Provides specific information on
who was consulted and when.
STEP 5. HOLD WORKING
SESSIONS TO REVIEW
INDICATORS
Hold working sessions with AO
team members.
Implementing
partners may be included at this
5
point as well. In order to use time
efficiently, the team may decide to
focus these sessions on resultslevel indicators. These working
sessions can be used to:
• Explain the purpose and process
for conducting the DQA.
• Review data quality standards for
each results-level indicator,
including the data collection
systems and processes.
• Identify issues or concerns that
require further review.
STEP 6. HOLD SESSIONS
WITH IMPLEMENTING
PARTNERS TO REVIEW
INDICATORS
If the implementing partner was
included in the previous working
session, results-level indicators will
already have been discussed. This
session may then focus on
reviewing the remaining outputlevel indicators with implementers
who often maintain the systems to
collect the data for these types of
indicators. Focus on reviewing the
systems and processes to collect
and maintain data. This session
provides a good opportunity to
identify solutions or recommenddations for improvement.
STEP 7. PREPARE THE
DQA DOCUMENT
As information is gathered, the
team should record findings on the
worksheets provided.
It is
particularly important to include
recommendations for action at the
conclusion of each worksheet.
Once this is completed, it is often
useful to include an introduction
to:
• Outline the overall approach and
methodology used.
• Highlight key data quality issues
that are important for senior
management.
• Summarize
recommendations
for
improving
performance
management systems.
AO
team
members
and
participating implementers should
have an opportunity to review the
first draft. Any comments or issues
can then be incorporated and the
DQA finalized.
STEP 8. FOLLOW UP ON
ACTIONS
Finally, it is important to ensure
that there is a process to follow-up
on recommendations.
Some
recommendations
may
be
addressed internally by the team
handling management needs or
audit vulnerabilities. For example,
the AO team may need to work
with a Ministry to ensure that data
can be disaggregated in a way that
correlates precisely to the target
group. Other issues may need to
be addressed during the Mission’s
portfolio reviews.
CONSIDER THE
SOURCE – PRIMARY
VS. SECONDARY
DATA
PRIMARY DATA
USAID is able to exercise a higher
degree of control over primary
data that it collects itself than over
secondary data collected by others.
As a result, specific standards
should be incorporated into the
data collection process. Primary
data collection requires that:
• Written procedures are in place
for data collection.
• Data are collected from year to
year using a consistent collection
process.
• Data
are
collected
using
methods
to
address
and
minimize sampling and nonsampling errors.
• Data are collected by qualified
personnel and these personnel
are properly supervised.
• Duplicate data are detected.
• Safeguards are in place to
prevent unauthorized changes to
the data.
• Source
documents
are
maintained and readily available.
• If the data collection process is
contracted
out,
these
requirements
should
be
incorporated directly into the
statement of work.
SECONDARY DATA
Secondary data are collected from
other sources, such as host
country
governments,
implementing partners, or from
other organizations. The range of
control that USAID has over
secondary data varies.
For
example, if USAID uses data from a
survey commissioned by another
donor, then there is little control
over
the
data
collection
methodology. On the other hand,
USAID does have more influence
over
data
derived
from
implementing partners. In some
cases,
specific
data
quality
requirements may be included in
the contract. In addition, project
performance management plans
6
(PMPs) are often reviewed or
approved by USAID. Some ways in
which to address data quality are
summarized below.
Data from Implementing
Partners
• Spot check data.
• Incorporate specific data quality
requirements as part of the
SOW, RFP, or RFA.
• Review data quality collection
and maintenance procedures.
Data from Other Secondary
Sources
Data from other secondary
sources includes data from host
countries, government, and other
donors.
• Understand the methodology.
Documentation often includes a
description of the methodology
used to collect data.
It is
important to understand this
section so that limitations (and
what the data can and cannot
say) are clearly understood by
decision makers.
• Request a briefing on the
methodology, including data
collection
and
analysis
procedures, potential limitations
of the data, and plans for
improvement (if possible).
• If data are derived from host
country organizations, then it
may be appropriate to discuss
how assistance can be provided
to strengthen the quality of the
data. For example, projects may
include technical assistance to
improve management and/or
M&E systems.
TABLE 1. THE DQA WORKSHEET FOR OUTCOME LEVEL INDICATORS
Directions: Use the following worksheet to complete an assessment of data for outcome level indicators against the
five data quality standards outlined in the ADS. A comprehensive discussion of each criterion is included in TIPS 12
Data Quality Standards.
Data Quality Assessment Worksheet
Assistance Objective (AO) or Intermediate Result (IR):
Indicator:
Reviewer(s):
Date Reviewed:
Data Source:
Is the Indicator Reported to USAID/W?
Criterion
Definition
1. Validity
Do the data clearly and adequately
represent the intended result? Some issues
to consider are:
Face Validity. Would an outsider or an
expert in the field agree that the indicator is
a valid and logical measure for the stated
result?
Attribution. Does the indicator measure
the contribution of the project?
Measurement Error. Are there any
measurement errors that could affect the
data? Both sampling and non-sampling error
should be reviewed.
2. Integrity
Do the data collected, analyzed and
reported have established mechanisms in
place to reduce manipulation or simple
errors in transcription?
3. Precision
Are data sufficiently precise to present a fair
picture of performance and enable
management decision-making at the
appropriate levels?
4. Reliability
Do data reflect stable and consistent data
collection processes and analysis methods
over time?
5. Timeliness
Are data timely enough to influence
management decision-making (i.e., in terms
of frequency and currency)?
Yes or No
Explanation
Note: This criterion requires the reviewer to
understand what mechanisms are in place to
reduce the possibility of manipulation or
transcription error.
Note: This criterion requires the reviewer to ensure
that the indicator definition is operationally precise
(i.e. it clearly defines the exact data to be collected)
and to verify that the data is, in fact, collected
according to that standard definition consistently
over time.
A Summary of Key Issues and Recommendations:
7
Table 2. SAMPLE DQA FOR OUTPUT INDICATORS: THE MATRIX APPROACH
Document
Source
Data
Source
Data Collection Method/
Key Data Quality Issue
Further Action
Additional Comments/
Notes
AO or IR
Indicators
1.
Number of investment
measures made consistent
with international investment
agreements as a result of
USG assistance
Quarterly
Report
Project
A
A consultant works directly with the
committee in charge of simplifying procedures
and updates the number of measures
regularly on the website
(www.mdspdres.com). The implementer has
stated that data submitted includes
projections for the upcoming fiscal year
rather than actual results.
Yes. Ensure that
only actual results
within specified
timeframes are
used for
reporting.
Meeting with COTR 6/20/10
and 7/6/10.
2.
Number of public and private
sector standards-setting
bodies that have adopted
internationally accepted
guidelines for standards
setting as a result of USG
assistance
Semi-Annual
Report
Project
A
No issues. Project works only with one body
(the Industrial Standards-Setting Service) and
maintains supporting documentation.
No.
Meeting with COTR and COP
on 6/20/10.
3.
Number of legal, regulatory,
or institutional actions taken
to improve implementation
or compliance with
international trade and
investment agreements due
to support from USGassisted organizations
Quarterly
Report
Project
A
Project has reported “number of Regional
Investment Centers”. This is not the same as
counting “actions”, so this must be corrected.
Yes. Ensure that
the correct
definition is
applied.
Meeting with COTR, COP,
and Finance Manager and M&E
specialist on 6/20/10. The
indicator was clarified and the
data collection process will be
adjusted accordingly.
4.
Number of Trade and
Investment Environment
diagnostics conducted
Quarterly
Report
Projects
A and B
No issues. A study on the investment
promotion policy was carried out by the
project. When the report is presented and
validated the project considers it
“conducted”.
No.
Meeting with CTO and COPs
on 6/25/10.
8
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including Gerry Britan and
written by Michelle Adams-Matson, of Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
9
NUMBER 19
1ST EDITION, 2010 DRAFT
TIPS
RIGOROUS IMPACT EVALUATION
ABOUT TIPS
WHAT IS RIGOROUS
IMPACT
EVALUATION?
Rigorous impact evaluations are
useful for determining the effects
of
USAID
programs
on
outcomes.
This
type
of
evaluation allows managers to
test development hypotheses by
comparing changes in one or
more specific outcomes to
changes that occur in the
absence
of
the
program.
Evaluators
term
this
the
counterfactual. Rigorous impact
evaluations
typically
use
comparison groups, composed of
individuals or communities that
do not participate in the
program. The comparison group
FIGURE 1. DEFINITIONS OF IMPACT EVALUATION
• An evaluation that looks at the impact of an intervention on final welfare
outcomes, rather than only at project outputs, or a process evaluation which
focuses on implementation.
• An evaluation carried out some time (five to ten years) after the
intervention has been completed, to allow time for impact to appear.
• An evaluation considering all interventions within a given sector or
geographical area.
• An evaluation concerned with establishing the counterfactual, i.e., the
difference the project made (how indicators behaved with the project
compared to how they would have been without it).
is examined in relation to the
treatment group to determine
the effects of the USAID program
or project.
Impact evaluations may be
defined in a number of ways (see
Figure 1). For purposes of this
TIPS, rigorous impact evaluation
1
is defined by the evaluation
design (quasi-experimental and
experimental) rather than the
topic being evaluated. These
methods can be used to attribute
change at any program or project
outcome level, including
Intermediate Results (IR), sub-IRs,
and Assistance Objectives (AO).
Decisions about whether a
rigorous impact evaluation would
be appropriate and what type of
rigorous impact evaluation to
conduct are best made during
the program or project design
phase, since many types of
rigorous impact
evaluation can only be utilized if
comparison groups are
established and baseline data is
collected before a program or
project intervention begins.
WHY ARE RIGOROUS
IMPACT
EVALUATIONS
IMPORTANT?
A rigorous impact evaluation
enables managers to determine
the extent to which a USAID
program or project actually
caused observed changes.
A Performance Management Plan
(PMP) should contain all of the
tools necessary to track key
objectives (see also TIPS 7
Preparing
a
Performance
Management Plan).
However,
comparing
data
from
performance indicators against
baseline values demonstrates
only
whether
change
has
occurred,
with
very
little
information about what actually
caused the observed change.
USAID program managers can
only say that the program is
correlated with changes in
outcome, but cannot confidently
attribute that change to the
program.
FIGURE 2. A WORD ABOUT WORDS
Many of the terms used in rigorous evaluations hint at the origin of these
methods: medical and laboratory experimental research. The activities of a
program or project are often called the intervention or the independent
variable, and the outcome variables of interest are known as dependent
variables. The target population is the group of all individuals (if the unit of
analysis or unit is the individual) who share certain characteristics sought by
the program, whether or not those individuals actually participate in the
program. Those from the target population who actually participate are
known as the treatment group, and the group used to measure what would
have happened to the treatment group had they not participated in the
program (the counterfactual) is known as a control group if they are selected
randomly, as in an experimental evaluation, or, more generally, as a
comparison group if they are selected by other means, as in a quasiexperimental evaluation.
There are normally a number of
factors, outside of the program,
that might influence an outcome.
These are called confounding
factors. Examples of confounding
factors include programs run by
other donors, natural events (e.g.,
rainfall, drought, earthquake,
etc.), government policy changes,
or even maturation (the natural
changes that happen in an
individual or community over
time). Because of the potential
contribution
of
these
confounding factors, the program
manager cannot claim with full
certainty that the program
caused the observed changes or
results.
In some cases, the intervention
causes all observed change. That
is, the group receiving USAID
assistance will have improved
significantly while a similar, nonparticipating group will have
stayed roughly the same.
In
other situations, the target group
may have already been improving
and the program helped to
accelerate that positive change.
Rigorous
evaluations
are
2
designed to identify the effects of
the program of interest even in
these cases, where both the
target
group
and
nonparticipating groups may have
both changed, only at different
rates. By identifying the effects
caused by a program, rigorous
evaluations
help
USAID,
implementing partners and key
stakeholders
learn
which
program or approaches are most
effective, which is critical for
effective
development
programming.
WHEN SHOULD
THESE METHODS BE
USED?
Rigorous impact evaluations can
yield very strong evidence of
program effects. Nevertheless,
this method is not appropriate
for all situations.
Rigorous
impact evaluations often involve
extra costs for data collection and
always require careful planning
during program implementation.
To determine whether a rigorous
impact evaluation is appropriate,
potential cost should be weighed
against the need for and
usefulness of the information.
Rigorous
impact
evaluations
answer
evaluation
questions
concerning the causal effects of a
program.
However, other
evaluation designs may be more
appropriate for answering other
types of evaluation questions.
For example, the analysis of ‘why’
and ‘how’ observed changes,
particularly unintended changes,
were produced may be more
effectively answered using other
evaluation methods, including
participatory evaluations or rapid
appraisals. Similarly, there are
situations
when
rigorous
evaluations, which often use
comparison groups, will not be
advisable, or even possible. For
example, assistance focusing on
political parties can be difficult to
evaluate using rigorous methods,
as this type of assistance is
typically offered to all parties,
making the identification of a
comparison group difficult or
impossible. Other methods may
be more appropriate and yield
conclusions
with
sufficient
credibility
for
programmatic
decision-making.
rigorous
impact
While
evaluations are sometimes used
to examine the effects of only
one
program
or
project
approach,
rigorous
impact
evaluations are also extremely
useful for answering questions
about the effectiveness of
alternative
approaches
for
achieving a given result, e.g.,
which of several approaches for
improving farm productivity, or
for delivering legal services, are
most effective.
Missions should consider using
rigorous evaluations strategically
to answer specific questions
about the effectiveness of key
approaches.
When multiple
rigorous evaluations are carried
out across Missions on a similar
topic or approach, the results can
be used to identify approaches
that can be generalized to other
settings, leading to significant
advances
in
programmatic
knowledge. Rigorous methods
are often useful when:
 Multiple
approaches
to
achieving desired results have
been suggested, and it is
unclear which approach is the
most effective or efficient;
 An approach is likely to be
replicated if successful, and
clear evidence of program
effects are desired before
scaling up;
 A program uses a large amount
of resources or affects a large
number of people; and
 In general, little is known about
the effects of an important
program or approach, as is
often the case with new or
innovative approaches.
PLANNING
Rigorous methods require strong
performance
management
systems to be built around a
clear, logical results framework
(see TIPS 13 Building a Results
Framework). The development
hypothesis should clearly define
the logic of the program, with
3
particular emphasis on the
intervention
(independent
variable) and the principal
anticipated results (dependent
variables), and provides the basis
for the questions that will be
addressed by the rigorous
evaluation.
Rigorous evaluation builds upon
the indicators defined for each
level of result, from inputs to
outcomes, and requires high data
quality.
Because
quasiexperimental and experimental
designs typically answer very
specific evaluation questions and
are generally analyzed using
quantitative methods, they can
be paired with other evaluation
tools and methods to provide
context, triangulate evaluation
conclusions, and examine how
and why effects were produced
(or not) by a program. This is
termed mixed method evaluation
(see TIPS 16, Mixed Method
Evaluations).
Unlike
most
evaluations
conducted by USAID, rigorous
impact evaluations are usually
only possible, and are always
most effective, when planned
before project implementation
begins.
Evaluators need time
prior to implementation to
identify appropriate indicators,
identify a comparison group, and
set baseline values. If rigorous
evaluations are not planned prior
to implementation, the number
of potential evaluation design
options is reduced, often leaving
alternatives that are either more
complicated or less rigorous. As
a result, Missions should consider
the feasibility of and need for a
Observed Change
Outcome of Interest
FIGURE 3. CONFOUNDING EFFECTS
Program
Effect
Confounding
Effect
Baseline
Follow-up
= Target Group
= Comparison Group
rigorous evaluation prior to and
during project design.
DESIGN
Although
there
are
many
variations, rigorous evaluations
are divided into two categories:
quasi-experimental
and
experimental. Both categories of
rigorous evaluations rely on the
same basic concept - using the
counterfactual to estimate the
changes caused by the program.
The counterfactual answers the
question, “What would have
happened to program participants
if they had not participated in the
program?” The comparison of
the
counterfactual
to
the
observed change in the group
receiving USAID assistance is the
true measurement of a program’s
effects.
While
before
and
after
measurements of a single group
using a baseline allow the
measurement of a single group
both with and without program
participation, this design does
not control for all the other
confounding factors that might
influence the participating group
during program implementation.
Well constructed, comparison
groups provide a clear picture of
the effects of program or project
interventions on the target group
by
differentiating
program/project effects from the
effects of multiple other factors in
the environment that affect both
the target and comparison
groups. This means that in
situations where economic or
other factors affecting
both
groups make everyone better
off, it will still be possible to see
the additional or incremental
improvement caused by the
program or project, as Figure 3
illustrates.
QUASI-EXPERIMENTAL
EVALUATIONS
To estimate program effects,
quasi-experimental designs rely
on measurements of a nonrandomly selected comparison
group. The most common means
for selecting a comparison group
is
matching,
wherein
the
4
evaluator ‘hand-picks’ a group of
similar units based on observable
characteristics that are thought to
influence the outcome.
For
example, the evaluation of
an
agriculture program aimed at
increasing crop yield might seek
to
compare
participating
communities
against
other
communities with similar weather
patterns,
soil
types,
and
traditional crops, as communities
sharing
these
critical
characteristics would be most
likely to behave similarly to the
treatment group in the absence
of the program.
However, program participants
are often selected based on
certain characteristics, whether it
is level of need, motivation,
location, social or political factors,
or some other factor.
While
evaluators can often identify and
match many of these variables, it
is impossible to match all factors
that might create differences
between the treatment and
comparison groups, particularly
characteristics that are more
difficult to measure or are
unobservable, such as motivation
or social cohesion. For example,
if a program is targeted at
WHAT IS EXPERIMENTAL AND
QUASI-EXPERIMENTAL
EVALUATION?
Experimental design is based on a
the selection of the comparison and
treatment group through random
sampling.
Quasi-experimental design is
based on a comparison group that
is chosen by the evaluator (that is,
not based on random sampling).
FIGURE 4.
QUASI-EXPERIMENTAL EVALUATION OF THE KENYA NATIONAL CIVIC EDUCATION PROGRAM
PHASE II (NCEP II)
NCEP II, funded by USAID in collaboration with other donors, reached an estimated 10 million individuals through
workshops, drama events, cultural gatherings and mass media campaigns aimed at changing individuals’ awareness,
competence and engagement in issues related to democracy, human rights, governance, constitutionalism, and
nation-building. To determine the program’s impacts on these outcomes of interest, NCEP as evaluated using a
quasi-experimental design with a matched comparison group.
Evaluators matched participants to a comparison group of non-participating individuals who shared geographic and
demographic characteristics (such as age, gender, education, and involvement with CSOs). This comparison group
was compared to the treatment group along the outcomes of interest to identify program effects. The evaluators
found that the program had significant long term effects, particularly on ‘civic competence and involvement’ and
‘identity and ethnic group relations, but had only negligible impact on ‘Democratic Values, Rights, and
Responsibilities’. The design also allowed the evaluators to assess the conditions under which the program was
most successful. They found confirmation of prior assertions of the critical role in creating lasting impact of multiple
exposures to civic education programs through multiple participatory methods.
- ‘The Impact of the Second National Kenya Civic Education Programme (NECP II-URAIA) on Democratic Attitudes,
Values, and Behavior’, Steven E. Finkel and Jeremy Horowitz, MSI
communities that are likely
succeed, then the target group
might be expected to improve
relative to a comparison group
that was not chosen based on the
same factors. Failing to account
for this in the selection of the
comparison group would lead to
a biased estimate of program
impact. Selection bias is the
difference
between
the
comparison group and the
treatment group caused by the
inability to completely match on
all characteristics, and the
uncertainty
or
error
this
generates in the measurement of
program effects.
Other
common
quasiexperimental designs, in addition
to matching, are described below.
Non-Equivalent Group Design.
This is the most common quasiexperimental design in which a
comparison group is hand-picked
to match the treatment group as
closely as possible. Since handpicking the comparison group
cannot completely match all
characteristics with the treatment
group, the groups are considered
to be ‘non-equivalent’.
significantly different except in
terms of eligibility for the
program. Because of this, the
group just above the cut-off
serves as a comparison group for
those just below (or vice versa) in
a regression discontinuity design.
Regression
Discontinuity.
Programs often have eligibility
criteria based on a cut-off score
or value of a targeting variable.
Examples
include
programs
accepting only households with
income
below
2,000
USD,
organizations registered for at
least two years, or applicants
scoring above a 65 on a pre-test.
In each of these cases, it is likely
that individuals or organizations
just above and just below the
cut-off value would demonstrate
only marginal or incremental
differences in the absence of
USAID assistance, as families
earning 2,001 USD compared to
1,999 USD are unlikely to be
Propensity Score Matching. This
method is based on the same
rationale as regular matching: a
comparison group is selected
based on shared observable
characteristics with the treatment
group.
However, rather than
‘hand-picking’ matches based on
a small number of variables,
propensity score matching uses a
statistical process to combine
information
from
all
data
collected
on
the
target
population to create the most
accurate matches possible based
on observable characteristics.
5
Interrupted Time Series.1 Some
programs will encounter
situations where a comparison
group is not possible, often
because the intervention affects
everyone at once, as is typically
the case with policy change. In
these cases, data on the outcome
of interest are recorded at
numerous intervals before and
after the program or activity take
places. The data form a timeseries or trend, which the
evaluator analyzes for significant
changes around the time of the
intervention. Large spikes or
drops immediately after the
intervention signal changes
caused by the program. This
method is slightly different from
the other rigorous methods as it
does not use a comparison group
to rule out potentially
confounding factors, leading to
increased uncertainty in
evaluation conclusions.
Interrupted time series are most
effective when data are collected
regularly both before and after
the intervention, leading to a
long time series, and alternative
causes are monitored.
EXPERIMENTAL EVALUATION
In an experimental evaluation, the
treatment
and
comparison
groups are selected from the
target population by a random
process. For example, from a
target
population
of
50
communities that meet the
1
Interrupted time series is normally
viewed as a type of impact evaluation.
It is typically considered quasiexperiemental although it does not use a
comparison group.
eligibility (or targeting) criteria of
a program, the evaluator uses a
coin flip, lottery, computer
program, or some other random
process to determine the 25
communities that will participate
in the program (treatment group)
and the 25 communities that will
not (control group, as the
comparison group is called when
it is selected randomly). Because
they use random selection
processes,
experimental
evaluations are often called
randomized
evaluations
or
randomized
controlled
trials
(RCTs).
Random selection from a target
population into treatment and
control groups is the most
effective tool for eliminating
selection bias because it removes
the possibility of any individual
characteristic
influencing
selection. Because units are not
assigned to treatment or control
groups
based
on
specific
characteristics, but rather are
divided
randomly,
all
characteristics that might lead to
selection bias, such as motivation,
poverty level, or proximity, will be
roughly equally divided between
the treatment and control
groups.
If an evaluator uses
random assignment to determine
treatment and control groups,
she might, by chance, get two or
three
very
motivated
communities in a row assigned to
the treatment group, but if the
program is working in more than
a handful of communities, the
number
of
motivated
communities will likely balance
6
out between treatment
control in the end.
and
Because
random
selection
completely eliminates selection
bias, experimental evaluations are
often easier to analyze and
provide more credible evidence
than quasi experimental designs.
Random assignment can be done
with any type of unit, whether the
unit is the individual, groups of
individuals (e.g., communities or
districts),
organizations,
or
facilities (e.g., health center or
school) and usually follows one of
the designs discussed below.
Simple Random Assignment.
When the number of program
participants has been decided
and additional eligible individuals
are identified, simple random
assignment through a coin flip or
lottery can be used to select the
treatment group and control
groups.
Programs
often
encounter
‘excess
demand’
naturally (for example in training
programs, participation in study
tours, or where resources limit
the
number
of
partner
organizations),
and
simple
random assignment can be an
easy and fair way to determine
participation while maximizing
the
potential
for
credible
evaluation conclusions.
Phased-In Selection. In some
programs, the delivery of the
intervention does not begin
everywhere at the same time. For
capacity or logistical reasons,
some units receive the program
intervention earlier than others.
This type of schedule creates a
natural opportunity for using an
FIGURE 5.
EXPERIMENTAL EVALUATION OF THE IMPACTS OF EXPANDING CREDIT ACCESS IN
SOUTH AFRICA
While commercial loans are a central component of most microfinance strategies, there is much less consensus on
whether consumer loans are also for economic development. Microfinance in the form loans for household
consumption or investment has been criticized as unproductive, usurious, and a contributor to debt cycles or traps.
In an evaluation partially funded by USAID, researchers used an experimental evaluation designed to test the impacts
of access to consumer loans on household consumption, investment, education, health, wealth, and well-being.
From a group of 787 applicants who were just below the credit score needed for loan acceptance, the researchers
randomly selected 325 (treatment group) that would be approved for a loan. The treatment group was surveyed,
along with the remaining 462 who were randomly denied (control group), eight months after their loan application to
estimate the effects of receiving access to consumer credit. The evaluators found that the treatment group was more
likely to retain wage employment, less likely to experience severe hunger in their households, and less likely to be
impoverished than the control group providing strong evidence of the benefits of expanding access to consumer
loans.
-‘Expanding Credit Access: Estimating the Impacts’, Dean Karlan and Jonathan Zinman,
http://www.povertyactionlab.org/projects/print.php?pid=62
experimental design. Consider a
project where the delivery of a
radio-based
civic
education
program was scheduled to
operate in 100 communities
during year one, another 100
during year two, and a final 100
during year three. The year of
participation can be randomly
assigned. Communities selected
to participate in year one would
be designated as the first
treatment group (T1). For that
year, all the other communities
that would participate in Years
Two and Three form the initial
control group. In the second
year, the next 100 communities
would become the second
treatment group (T2), while the
final 100 communities would
continue to serve as the control
group. Random assignment to
the year of participation ensures
that
all
communities
will
participate in the program but
also maximizes evaluation rigor
by reducing selection bias, which
could be significant if only the
most motivated communities
participate in Year One.
Blocked
(or
Stratified)
Assignment. When it is known in
advance that the units to which a
program intervention could be
delivered differ in one or more
ways that might influence the
program outcome, (e.g., age, size
of the community in which they
are located, ethnicity, etc.),
evaluators may wish to take extra
steps to ensure that such
conditions are evenly distributed
between
an
evaluation’s
treatment and control groups. In
a simple block (stratified) design,
an evaluation might separate
men and women, and then use
randomized assignment within
each block to construct the
evaluation’s
treatment
and
control groups, thus ensuring a
specified number or percentage
7
of men and women in each
group.
Multiple Treatments. It is
possible that multiple approaches
will be proposed or implemented
for the achievement of a given
result. If a program is interested
in
testing
the
relative
effectiveness of three different
strategies or approaches, eligible
units can be randomly divided
into three groups. Each group
participates in one approach, and
the results can be compared to
determine which approach is
most effective. Variations on this
design can include additional
groups to test combined or
holistic approaches and a control
group to test the overall
effectiveness of each approach.
COMMON
QUESTIONS AND
CHALLENGES
While
rigorous
evaluations
require significant attention to
detail in advance, they need not
be impossibly complex. Many of
the most common questions and
challenges can be anticipated and
minimized.
COST
Rigorous evaluations will almost
always cost more than standard
evaluations that do not require
comparison groups.
However,
the
additional
cost
can
sometimes
be
quite
low
depending on the type and
availability of data to be
collected.
Moreover, findings
from rigorous evaluations may
lead to future cost-savings,
through improved programming
and more efficient use of
resources over the longer term.
Nevertheless, program managers
must anticipate these additional
costs, including the additional
planning requirements, in terms
of staffing and budget needs.
ETHICS
The use of comparison groups is
sometimes criticized for denying
treatment
to
potential
beneficiaries.
However, every
program has finite resources and
must select a limited number of
program participants. Random
selection of program participants
is often viewed, even by those
beneficiaries
who
are
not
selected, as being the fairest and
most transparent method for
determining participation.
A second, more powerful, ethical
question emerges when a
program
seeks
to
target
participants that are thought to
be most in need of the program.
In
some
cases,
rigorous
evaluations require a relaxing of
targeting
requirements
(as
discussed in Figure 6) in order to
identify enough similar units to
constitute a comparison group,
meaning that perhaps some of
those identified as the ‘neediest’
might be assigned to the
comparison group. However, it is
often the case that the criteria
used to target groups do not
provide a degree of precision
required to confidently rankorder
potential
participants.
Moreover, rigorous evaluations
can help identify which groups
benefit most, thereby improving
targeting for future programs.
SPILLOVER
Programs are often designed to
incorporate ‘multiplier effects’
whereby program effects in one
community naturally spread to
others nearby.
While these
effects help to broaden the
impact of a program, they can
result in bias in conclusions when
the effects on the treatment
group spillover to the comparison
group. When comparison groups
also benefit from a program, then
they no longer measure only the
confounding effects, but also a
portion of the program effect.
This leads to underestimation of
program impact since they
8
FIGURE 6. TARGETING IN
RIGOROUS EVALUATIONS
Programs often have specific
eligibility requirements without
which a potential participant could
not feasibly participate.
Other
programs target certain groups
because of perceived need or
likelihood of success. Targeting is
still
possible
with
rigorous
evaluations, whether experimental
or quasi-experimental, but must be
approached in a slightly different
manner. If a program intends to
work in 25 communities, rather than
defining
one
group
of
25
communities that meet the criteria
and participate in the program, it
might be necessary to identify a
group of 50 communities that meet
the eligibility or targeting criteria
and will be split into the treatment
and comparison group.
This
reduces the potential for selection
bias while still permitting the
program to target certain groups.
In situations where no additional
communities meet the eligibility
criteria and the criteria cannot be
relaxed, phase-in or multiple
treatment approaches, as discussed
below, might be appropriate.
appear better off than they would
have been in the absence of the
program.
In some cases,
spillovers can be mapped and
measured but, most often, they
must be controlled in advance by
selecting treatment and control
groups or units that are unlikely
to significantly interact with one
another.
A special case of
spillover occurs in substitution
bias wherein governments or
other donors target only the
comparison group to fill in gaps
of service. This is best avoided by
ensuring coordination between
the
program
and
development actors.
other
SAMPLE SIZE
During the analysis phase,
rigorous evaluations typically use
statistical tests to determine
whether any observed differences
between
treatment
and
comparison groups represent
actual differences (that would
then, in a well designed
evaluation, be attributed to the
program)
or
whether
the
difference could have occurred
due to chance alone. The ability
to make this distinction depends
principally on the size of the
change and the total number of
units in the treatment and
comparison groups, or sample
size. The more units, or higher
the sample size, the easier it is to
attribute change to the program
rather than to random variations.
During
the
design
phase,
rigorous
impact
evaluations
typically calculate the number of
units (or sample size) required to
confidently identify changes of
the size anticipated by the
program. An adequate sample
size helps prevent declaring a
successful project ineffectual
(false negative) or declaring an
ineffectual project successful
(false positive). Although sample
9
size calculations should be done
before each program, as a rule of
thumb,
rigorous
impact
evaluations are rarely undertaken
with less than 50 units of analysis.
RESOURCES
This TIPS is intended to provide
an introduction to rigorous
impact evaluations.
Additional
resources are provided on the
next page for further reference.
Further Reference
Initiatives and Case Studies:
-
-
-
-
Office of Management and Budget (OMB):
o http://www.whitehouse.gov/OMB/part/2004_program_eval.pdf
o http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf
U.S. Government Accountability Office (GAO):
o http://www.gao.gov/new.items/d1030.pdf
USAID:
o Evaluating Democracy and Governance Effectiveness (EDGE):
http://www.usaid.gov/our_work/democracy_and_governance/technical_areas/dg_office/eval
uation.html
o Measure Evaluation:
http://www.cpc.unc.edu/measure/approaches/evaluation/evaluation.html
o The Private Sector Development (PSD) Impact Evaluation Initiative:
www.microlinks.org/psdimpact
Millennium Challenge Corporation (MCC) Impact Evaluations:
http://www.mcc.gov/mcc/panda/activities/impactevaluation/index.shtml
World Bank:
o The Spanish Trust Fund for Impact Evaluation:
http://web.worldbank.org/WBSITE/EXTERNAL/EXTABOUTUS/ORGANIZATION/EXTHDNETW
ORK/EXTHDOFFICE/0,,contentMDK:22383030~menuPK:6508083~pagePK:64168445~piPK:6
4168309~theSitePK:5485727,00.html
o The Network of Networks on Impact Evaluation: http://www.worldbank.org/ieg/nonie/
o The Development Impact Evaluation Initiative:
http://web.worldbank.org/WBSITE/EXTERNAL/EXTDEC/EXTDEVIMPEVAINI/0,,menuPK:39982
81~pagePK:64168427~piPK:64168435~theSitePK:3998212,00.html
Others:
o Center for Global Development’s ‘Evaluation Gap Working Group’:
http://www.cgdev.org/section/initiatives/_active/evalgap
o International Initiative for Impact Evaluation: http://www.3ieimpact.org/
Additional Information:
-
-
Sample Size and Power Calculations:
o http://www.statsoft.com/textbook/stpowan.html
o http://www.mdrc.org/publications/437/full.pdf
World Bank: ‘Evaluating the Impact of Development Projects on Poverty: A Handbook for
Practitioners’:
o http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,contentM
DK:20194198~pagePK:148956~piPK:216618~theSitePK:384329,00.html
Poverty Action Lab’s ‘Evaluating Social Programs’ Course: http://www.povertyactionlab.org/course/
10
TIPS publications are available online at [insert website]
Acknowledgements:
Our thanks to those whose experience and insights helped shape this publication including USAID’s
Office of Management Policy, Budget and Performance (MPBP). This publication was written by Michael
Duthie of Management Systems International.
Tel: (202) 712-1158
gbritan@usaid.gov
11