How to Build A Successful Mentoring Program Elements of Effective Practice Using the

How to Build
A Successful Mentoring Program
Using the
Elements of Effective Practice
TM
A S T E P - B Y- S T E P T O O L K I T F O R P R O G R A M M A N A G E R S
Don’t Forget the Free Tools!
Please note that this PDF does not include all of the tools available on
the CD-ROM. You can review and download the tools you want
directly from the web site:
Tools for Designing and Planning
http://www.mentoring.org/find_resources/tool_kit/design/
Tools to Manage a Program for Success
http://www.mentoring.org/find_resources/tool_kit/management/
Tools to Structure Effective Program Operations
http://www.mentoring.org/find_resources/tool_kit/operations/
Tools to Establish Evaluation Criteria and Methods
http://www.mentoring.org/find_resources/tool_kit/eval/
Section VII.
How to Establish Evaluation Criteria and Methods
The final mentoring program element listed in the second edition of the Elements of Effective Practice is program evaluation. The following are among the many
reasons1 programs should conduct evaluations:
• To increase understanding of effective practices in
youth mentoring relationships and programs;
• To make the programs accountable to the entities
that support them;
• To promote effective resource allocation (i.e. to
identify the most deserving recipients of scarce
funds);
• To avoid unintended harmful effects of
interventions;
• To increase the effectiveness of programs through a
feedback/continuous quality improvement process;
and
• To provide direct benefits for case managers,
mentors and youth when evaluation of individual
relationships is built into the evaluation plan.
To ensure the quality and effectiveness of your program, you’ll need to do the following:
• Develop a plan to measure program processes;
• Develop a plan to measure expected outcomes; and
• Create a process to reflect on and disseminate
evaluation findings.
The ultimate success of your program depends on how
well you are able to assess its effectiveness, address any
weaknesses and demonstrate that it is meeting established goals and objectives. With a comprehensive
evaluation process in place, you can do the following:2
• Provide objective feedback to program staff and
participants about whether they’re meeting their
goals;
• Identify achievements and milestones that warrant
praise and increase motivation;
• Pinpoint problems early enough to correct them;
• Assure funders and supporters of your program’s
accountability;
• Build credibility in the community that your
program is vital and deserves support; and
• Quantify experiences so that your program can help
others.
MEASURE PROGRAM PROCESS
Your plan for measuring program process should
include the following:
• Selecting indicators of program implementation
viability and volunteer fidelity, such as training
hours, meeting frequency and relationship
duration; and
• Developing a system for collecting and managing
specified data.
MEASURE EXPECTED OUTCOMES
Your plan for measuring expected outcomes should
include the following:
• Specifying expected outcomes;
• Selecting appropriate instruments to measure
outcomes, such as questionnaires, surveys and
interviews; and
• Selecting and implementing an evaluation design.
H O W T O E S TA B L I S H E VA L U AT I O N C R I T E R I A A N D M E T H O D S
163
CREATE A PROCESS TO REFLECT ON AND
DISSEMINATE FINDINGS
The final stage of program evaluation includes the following activities:
• Refining the program design and operations based
on the findings; and
• Developing and delivering reports to program
constituents, funders and the media (at a
minimum, yearly; optimally, each quarter).
The article “Gauging the Effectiveness of Youth
Mentoring,” written by Dr. Jean Rhodes for
MENTOR’s Research Corner, is reprinted below.
(The text of the article has been edited to meet the
needs of the tool kit.) It analyzes the components outlined above for conducting a thorough process and
outcome evaluation of a mentoring program. It will be
an invaluable reference for you as you determine how
best to develop an evaluation plan for your mentoring
program.
“GAUGING THE EFFECTIVENESS OF
YOUTH MENTORING”
BY DR. JEAN RHODES
The practice of evaluating one’s own efforts is as natural
as breathing. Cooks taste their own gravy and sauce, cabinetmakers run their hands over the wood to decide when
a piece is smooth enough, and basketball players watch to
see whether their shots go in. Indeed, it would be most
unwise after turning on the hot water to neglect to check
the water temperature before stepping into a shower stall.
— Posavac & Carey (1997)
Although program evaluation is not as natural or
spontaneous as this sort of self-evaluation, most programs engage in some form of monitoring. Sometimes
it’s as simple as asking mentees and mentors about
their experiences; in other cases, it involves large-scale,
rigorous experimental designs.
Of course, programs are more apt to launch the
former, less complicated types of evaluation. Such
164
evaluations do not require the same level of expertise,
are far less expensive, place minimal burden on participants and staff, and can yield useful findings. For
example, simple exit interviews can provide staff with
important and immediate feedback about programs.
So, you might ask, why not stop there? A primary reason is that funders need more convincing evidence
that programs are actually reaching their objectives.
Thus, accountability has increasingly involved moving
beyond simple descriptions to demonstrating that specific goals have been met. Knowing your options will
help you make informed decisions about the scope and
rigor of your design.
Determining the Impact of Your Program
We’ll cover several options, ranging from the simple to
the more complicated.3 We’ll begin with a strategy
that relies on comparing your program with others
(i.e., using benchmarks) to determine whether you are
having an effect. Some of the more intensive evaluation approaches (e.g., quasi-experimental designs), on
the other hand, might require the expertise of an outside evaluator, such as a graduate student or a professor from a local university. The cost of an outside evaluation tends to vary according to its intensity, but
programs should budget between $5,000 and $10,000
for the expertise.
Using Benchmarks
Without actually conducting an evaluation, programs
can sometimes draw on findings that have been linked
to outcomes in similar programs. In other words, findings from other studies can be used as benchmarks
against which to gauge a program’s relative effectiveness.4 This approach is feasible when your program
has these characteristics:
• It is targeting similar youth to the evaluated
program;
• It is reasonably similar in terms of relationship
structure and content to the evaluated program;
and
• It has met or exceeded the evaluated program’s
quality standards.
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
SECTION VII
What can we infer from other evaluations? DuBois and
his colleagues conducted a meta-analysis of 55 evaluations of one-to-one youth mentoring programs.5 The
analysis summarized the results of each study and calculated effect sizes (the magnitude of impact) across
the entire group of studies. Modest effects of mentoring programs were found across fairly diverse programs, but larger effect sizes emerged under the following conditions:
• Relationship quality. Although duration is
probably the single best benchmark, research found
that the quality of a mentoring relationship can
predict positive outcomes above and beyond how
long the relationship lasts.
MENTORING TOOL
See “Gauging
the Effectiveness of
• Youth were somewhat vulnerable but had not yet
succumbed to severe problems.
• Relationships were characterized by
1. More frequent contact;
2. Emotional closeness; and
3. A duration of six months or longer.
Youth Mentoring
Questionnaire” in
When responses to a questionnaire used in this
research indicated a positive, nonproblematic relationship, that relationship
tended to last longer and have
more positive effects.
Section VII on the CD.
• Programs were characterized by practices that
increased relationship quality and longevity, including these:
Although benchmarks can be
enormously useful, they may
not provide the level of detail or rigor that programs
or funders desire. Moreover, at this stage, benchmarks
can only be applied to one-to-one programs. Thus, it
is often necessary to conduct a structured evaluation.
The Nuts and Bolts of Evaluating
Mentoring Programs
1.
2.
3.
4.
Intensive training for mentors;
Structured activities for mentors and youth;
High expectations for frequency of contact;
Greater support and involvement from
parents; and
5. Monitoring of overall program
implementation.
MENTORING TOOL
See program
evaluation tools in
Section VII on the CD.
Since greater numbers of these practices predicted
more positive outcomes for youth in mentoring programs, one-to-one programs that have met these criteria can assume positive outcomes. Additionally,
research by Roffman, Reddy and Rhodes on one-toone programs has provided two relatively simple
benchmarks against which similar one-to-one mentoring programs can measure themselves to ensure that
relationships will have positive effects:6
• Duration. Because duration tends to imply strong
relationships and programs, it may be the single
best benchmark of program effectiveness. Across
several studies, longer durations have been
associated with stronger effects.
Types of Program
Evaluation
There are two major types of
program evaluation: process
evaluations and outcome evaluations.
• Process evaluations focus on whether a program is
being implemented as intended, how it is being
experienced, and whether changes are needed to
address any problems (e.g., difficulties in recruiting
and retaining mentors, high turnover of staff, high
cost of administering the program).
• Outcome evaluations focus on what, if any, effects
programs are having. Designs may compare youth
who were mentored to those who were not or may
examine the differences between mentoring
approaches. Information of this sort is essential for
self-monitoring and can address key questions
about programs and relationships.
H O W T O E S TA B L I S H E VA L U AT I O N C R I T E R I A A N D M E T H O D S
165
Process evaluations of mentoring programs usually
involve data from interviews, surveys and/or program
records that shed light on
MENTORING TOOL
the following areas:
•
•
•
•
See forms to track
Number of new matches;
program activity and
Types of activities;
participant feedback in
Length of matches;
Sections V and VI on
the CD.
Frequency and duration of
meetings; and
• Perceptions of the relationship.
Information of this sort is essential for self-monitoring
and can answer key questions about programs and
relationships.
Despite the importance of such information, outcome
evaluations have become increasingly important for
accountability; therefore, the rest of this section will
focus on the issues and decisions involved in conducting an outcome evaluation.
Outcome evaluations of mentoring programs usually
involve data from surveys, interviews, records and so
forth, including the following:
• Mentees’ reports of their grades, behavior and
psychological functioning;
• Teachers’ reports of mentees’ classroom behavior;
• Mentors’ reports of their well-being;
• Parent-child relationships; and
• High-school graduation rates.
Tips for and traps in conducting an
outcome evaluation
Measuring outcomes
• Select outcomes that are most:
a. Logically related to (and influenced by)
the program;
b. Meaningful to you; and
c. Persuasive to your funders.
166
• Be realistic. You are better off building a record of
modest successes, which keep staff and funders
motivated, than focusing on “big wins,” which may
be unrealistic and, when not achieved,
demoralizing.
• Collect outcome data after the youth and mentors
have been meeting for some time, long enough to
expect that some changes in the youth have
occurred.
Determining sources of data
• Obtain information from multiple sources,
including reports from mentees, mentors, parents,
caseworkers and so on.
• Select multiple criteria rather than just one
outcome (e.g., grades, drug use, attitudes).
• Use standardized questionnaires.
a. Questionnaires that have been scientifically
validated are more convincing to funders—and
provide a better basis for cross-program
comparisons—than surveys you might develop
on your own.
b. Such surveys are available for public use
through tool kits. The Search Institute has one
available (What’s Working: Tools for Evaluating
Your Mentoring Program) for purchase (see list
of Additional Resources below) and The
Mentor Center links to several free online
resources.
c. The Juvenile Justice Evaluation Center provides
links to questionnaires that are likely to be of
interest to mentoring programs, including
questionnaires about delinquency, drug and
alcohol use, ethnic identity, peer relations, and
psychological measures.
Selecting an outcome evaluation
Outcome evaluations generally are of two major types:
single-group and quasi-experimental designs.
• Single-group designs are the simplest and most
common types of evaluation. They are less
intrusive and costly and require far less effort to
complete than the more ambitious methods we will
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
SECTION VII
describe. An example of a single-group evaluation is
a questionnaire administered to participants at the
completion of the program (post-test only) or
before and after the program (pre-test/post-test).
• Quasi-experimental designs help evaluators identify
whether the program actually causes a change in
program participants, using controls to eliminate
possible biases. An example of a quasi-experimental
design is a pre-test administered at the beginning of a
program and a post-test at the completion of the
program to both the target mentoring group and a
matched comparison group that does not receive
mentoring.
Single-group designs
• Post-test only
a. Programs commonly use this design to
determine how mentees are doing at the end of
a mentoring program. Post-test evaluations can
help determine whether the mentees have
achieved certain goals (e.g., not dropping out of
school) that match the
program’s implicit or
MENTORING TOOL
explicit goals.
See pre- and post-tests
b. Such evaluations also
for mentors and
help discover whether
mentees in Section VII
mentors are satisfied
on the CD.
with the program.
c. Such an evaluation
cannot indicate whether the participant has
changed during the program, only how the
participant is functioning at the end of the
program.
• Pre-test/post-test designs
a. Programs use this design when they want to
determine whether mentees actually improved
while they were in the program. With this type
of evaluation, program staff survey how each
participant is doing at the time of enrollment in
the mentoring program and then after
completion of the program (e.g., 6 or 12
months after the pre-test). By comparing the
results of the pre-test and post-test, staff can see
whether the mentee improved.
b. This evaluation cannot indicate whether the
program caused the improvement. Many viable
alternative interpretations could explain the
change, including these:
• Maturation—natural change that occurred
simply as a result of the passage of time; and
• History—events that occurred between the
time the participants took the pre-test and
the post-test.
c. Other problems with interpreting findings from
this design include the following:
• Self-selection—The experimental group
might differ from the comparison group in
some systematic way. For example, quite
possibly only the mentees who benefited
most remained in the program long enough
to take the post-test.
• Regression to the mean—A mentee who is
functioning extremely poorly at the
program’s onset might improve naturally over
time. Mentees might enlist in programs
when they are most distressed and then
naturally return to a higher level of
functioning as time passes.
d. Even if one cannot identify the cause of a
mentee’s improvement, a pre-test design can be
useful in other ways:
• The evaluator can look at
differences within the
See MENTOR’s
group. For instance, do
Research Corner for
youth who receive more
the article on
frequent or enduring
“Mentoring and Race”
mentoring benefit most?
by Dr. Rhodes:
• The evaluator can
www.mentoring.org/
determine whether certain
research_corner
mentee characteristics are
related to achieving
program goals. For instance, do boys benefit more
than girls? Do minorities in same-race matches
benefit more than those in cross-race matches?
H O W T O E S TA B L I S H E VA L U AT I O N C R I T E R I A A N D M E T H O D S
167
Quasi-experimental designs
Despite their potential benefits, single-design evaluations seldom help evaluators identify whether the program is the cause of change in program participants. To
determine that, one needs to conduct evaluations of
slightly greater complexity. Such designs are called
quasi-experimental because, if carefully planned, they
can control for many of the biases described above. This
kind of evaluation comes in a variety of types, such as
time-series. We will focus on one common type of program evaluation: one that uses a comparison group.
Comparison group designs
• The most direct way to rule out alternative
explanations is to observe additional youth who
have not been part of the program but are similar
in other ways to the program youth. By including a
comparison group, evaluators can isolate the effects
of the program from the effects of other plausible
interpretations of change.
• A comparison group design also helps put in
perspective modest improvements or unexpected
declines. Take, for example, the landmark
evaluation of Big Brothers Big Sisters of America’s
mentoring program.7 Although youth in both the
mentored and control groups showed increases in
academic, social-emotional, behavioral and
relationship problems over the period of time being
studied, the problems of the mentored group
increased at a slower rate.
• One vexing problem with comparison group
studies is finding a comparison group that is
sufficiently similar to the mentored group. Parents
who seek out mentoring programs for their
children may devote more attention to their kids at
home than do parents of youth who are not
mentored. Similarly, young people who willingly
enlist in a mentoring program may differ (in terms
of motivation, compliance, etc.) from those who do
not enlist. The Big Brother Big Sisters study got
around this potential problem by selecting both
groups from the organization’s waiting list.
Unfortunately, many programs either do not
keep a waiting list or are not willing to deliberately
withhold their program from eligible and motivated
participants.
168
The Bottom Line
People in the mentoring field tend to believe implicitly
that mentoring benefits young people and that, therefore, expensive evaluations are an unnecessary drain on
precious resources. Given the choice of spending
money on evaluation or extending their services, many
mentoring programs will gladly choose the latter.
Although understandable, such choices may be shortsighted. We should not necessarily assume that all
mentoring programs are equally beneficial—and we
still have a lot to learn about the many newer types of
mentoring programs (e.g., site-based, group, peer, ementoring). Convincing evaluations are needed to
assess the effectiveness of both traditional one-to-one
mentoring programs and newer approaches. Such
work will play an important role in the expansion of
high-quality mentoring programs.
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
SECTION VII
Checklist of Program Progress:
PROGRAM EVALUATION
As you work to ensure program quality and effectiveness as outlined in the Elements of Effective Practice, use
the checklist below to gauge your progress. Checking
off the items on this list indicates that you are putting
the proper components in place to grow a quality, sustainable program.
If your program is already well established, you can use
the checklist to gauge the soundness of your current
policies, procedures and organizational structure.
Note: The design, focus and structure of your program
may mean that some of these components will not be
applicable or will need to be modified to match your specific program structure.
1. Develop a Plan to Measure Program Process
❑ We have selected appropriate instruments to
measure outcomes, such as questionnaires,
surveys and interviews.
❑ Our program has carefully considered whether
to use an outside evaluator or our staff.
❑ We have selected and implemented an
evaluation design.
❑ We have established a timeline for conducting
the evaluation.
❑ Our evaluation is being implemented and we are
collecting and analyzing evaluation data.
3. Create a Process to Reflect on and Disseminate
Evaluation Findings
Use of evaluation data for program enhancement
2. Develop a Plan to Measure Expected Outcomes
Design and implementation of program
evaluation
❑ Our program uses evaluation results to improve
our internal systems and procedures.
❑ Our program uses evaluation results to improve
and enhance the desired outcomes for youth.
❑ Our program understands the importance of
conducting a program evaluation.
❑ We have identified the processes and outcomes
that we would like to measure in our evaluation.
❑ We use evaluation results in marketing the
program to prospective volunteers and
community partners.
❑ We have developed a plan to measure program
process.
❑ We use evaluation results to increase the funding
and sustainability of the program.
❑ We have selected indicators of program
implementation viability and volunteer fidelity,
such as training hours, meeting frequency and
relationship duration.
❑ Our program interprets and uses our evaluation
results honestly.
❑ We have developed a system for collecting and
managing specific data.
❑ We developed and delivered reports to program
constituents, funders and the media at least
annually.
❑ We have specified expected outcomes.
❑ We have refined the program design and
operations based on the findings.
Adapted from Checklist of Program Progress, Oregon Mentors, Youth Mentoring: A Primer for Funders, The Connecticut
Mentoring Partnership and Elements of Effective Practice, Second Edition, MENTOR/National Mentoring Partnership.
H O W T O E S TA B L I S H E VA L U AT I O N C R I T E R I A A N D M E T H O D S
169
SECTION VII
Additional Resources
Program Outcomes Evaluation
• Analyzing Outcome Information, Harry Hatry, Jake Cowan and Michael Hendricks, 2003, The Urban Institute
www.urban.org
• An Evaluation Study of Mentoring Programs in Connecticut, The Connecticut Mentoring Partnership, 2003
Entire Report: www.preventionworksct.org/pdf/FINALREPORT.pdf
Executive Summary: www.preventionworksct.org/pdf/ExecSumm.pdf
• Connections Newsletter on Evaluation, Vol. 5, Issue 3, Summer 2003, Friends for Youth Mentoring Institute
www.homestead.com/prosites-ffy/resourcesinfo.html
• Evaluating Your Program: A Beginner’s Self Evaluation Workbook for Mentoring Programs, Information
Technology International and Pacific Institute for Research and Evaluation, 2000)
www.itiincorporated.com/sew_dl.htm
• Focus Groups: A Practical Guide for Applied Research, third edition, Richard A. Krueger and Mary Anne
Casey, 2000
www.sagepub.com/book.aspx?pid=3591
• Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation,
and Evaluation, Matthew Chinman, Pamela Imm and Abraham Wandersman, 2004
www.rand.org/publications/TR/TR101/
• Handbook of Youth Mentoring, the SAGE Program on Applied Developmental Science, edited by David L.
DuBois and Michael J. Karcher, 2005
i. Program Evaluation chapter, Jean Grossman
ii Assessment of Mentoring Relationships chapter, John Harris and Mike Nakkula
www.mentoring.org/youth_mentoring_handbook
• Juvenile Justice Evaluation Center Online, Office of Juvenile Justice and Delinquency Prevention and Justice
Research and Statistics Association
www.jrsa.org/jjec
• Key Steps in Outcome Management, Harry P. Hatry and Linda M. Lampkin, The Urban Institute, 2003
www.urban.org
• “Learning from Logic Models in Out-of-School Time,” Harvard Family Research Project, 2000
www.gse.harvard.edu/hfrp/projects/afterschool/resources/learning_logic_models.html
• Measuring Program Outcomes: A Practical Approach, United Way of America, 1996
http://national.unitedway.org/outcomes/resources/mpo
• Online Outcome Measurement Resource Network, United Way of America,
http://national.unitedway.org/outcomes/resources/What/OM_What.cfm
H O W T O E S TA B L I S H E VA L U AT I O N C R I T E R I A A N D M E T H O D S
171
• Outcome Measurement: What and Why—An Overview, United Way of America, 2002
http://national.unitedway.org/outcomes/files/TPsOMWhatandWhy.pdf
• Performance Measures in Out-of-School Time Evaluation, outlines the academic, youth development, and
prevention performance measures currently used by out of school time (OST) programs to assess their
progress and the corresponding data sources for these measures. Out-of-School Time Evaluation Snapshots
series of Harvard Family Research Project
www.gse.harvard.edu/hfrp/projects/afterschool/resources/snapshot3.html
• Research Corner from Dr. Jean Rhodes, Mentoring.org
www.mentoring.org/research_corner
• Surveying Clients About Outcomes, Martin D. Abravanel, The Urban Institute, 2003
www.urban.org
• What’s Working: Tools for Evaluating Your Mentoring Program, Search Institute, 2001
www.mentoring.org/whats_working
• W. K. Kellogg Foundation Evaluation Handbook, 1998
www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf
172
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
SECTION VII TOOLS ON CD
* Select tools denoted with an asterisk also appear in the print version of the tool kit.
Program Evaluation
• How to Select a Survey to Assess
Your Adult-Youth Mentoring
Program*........................................... 173
• Mentor Program Evaluations (Program
Coordinator, Mentor, Mentee)
• Teacher Report on the Match
• Pre-Post Teacher Survey
• Gauging the Effectiveness of
Youth Mentoring Questionnaire
.
• Youth Pre-Program Survey
• Youth Follow-Up Survey
• Mentor Evaluation Form
(Mentee Impact)
• Mentor Evaluation Form
(Mentor Impact)
• Mentor Program: Parent Survey
• Logic Model* .................................... 177
Note: The reliability/validity of some of the sample evaluation tools contained herein is unknown.
PROGRAM EVALUATION TOOLS
HOW TO SELECT A SURVEY TO ASSESS YOUR
ADULT–YOUTH MENTORING PROGRAM1
By John Harris, Applied Research Consulting and Michael Nakkula, Harvard Graduate School of Education
The assessment of mentoring relationship quality (MRQ) is fundamentally important to your mentoring program. In addition to helping you demonstrate the efficacy of your services, assessments of MRQ can help you
identify and maintain best practices for the youth you serve and the mentors you support. Timely and appropriate assessment can inform match supervision and ongoing mentor training, assist with the detection of problems
in a match or simply provide evidence of success to funders and mentors (who frequently fail to appreciate the
difference they make). Effective use of assessments may facilitate the development and maintenance of more
durable and high-quality matches.
Match advisors in many programs conduct regular check-ins with participants to informally assess MRQ, and
this personal supervision is critical to the maintenance of successful matches. However, a survey can be a useful
addition to such check-ins (e.g., to satisfy a formal evaluation requirement). It also may be integrated into programming processes in ways that augment match supervision. To be a useful addition, a survey must generate
(at a minimum) meaningful, accurate data that touches on important aspects of the match, such as closeness or
instrumentality (the degree to which a match fosters growth for the served youth). To yield more meaningful
insight, a survey should assess a broader array of perspectives on MRQ. If you want to integrate a survey more
fully into your program’s processes, you should choose a survey that conforms particularly closely to your program’s goals and assesses the broadest variety of perspectives on MRQ.
So, what should you look for in a survey that measures MRQ? First and foremost, it should be supported by
scientific proof of its usefulness or validity evidence—evidence that it really measures what it says it measures. The
best test of this criterion is whether an instrument has been incorporated into a study that was published in a
peer-reviewed journal. Only a handful of existing instruments meet this criterion, and we have provided brief
notes about them below. A survey can have strong validity evidence without being published, but if you consider
an unpublished instrument, you will need to contact the author to find out about its validity evidence. The fact
that a survey is used widely does not mean it was designed with sufficient scientific rigor.
If an instrument has sufficient validity evidence, you need to determine whether it assesses a useful range of
MRQ indicators and whether the ones it assesses are important to your program. Existing research and our own
experience have convinced us that to fully understand MRQ in a given relationship it is important to consider
three categories of indicators: those that pertain only to what goes on between a mentor and a child, including
relational/experiential indicators (e.g., compatibility, closeness); instrumental/goal-oriented indicators (e.g., degree
of focus on personal and academic growth, satisfaction with received support); and external, environmental indicators (e.g., programmatic influence, parental influence). Surveys can assess these indicators from a variety of perspectives: subjective indicators that reflect how participants feel about their match; objective indicators that reflect
actual match activities; positive reflections of MRQ (e.g., youth is satisfied with the match); or negative reflections
of MRQ (e.g., youth is dissatisfied).
Finally, the survey you choose should feel useful to you. It should ask questions that seem important to you and
match your program’s mentoring model (e.g., community-based, school-based), its goals (e.g., academically
1 Note: This is a synopsis (with some verbatim passages) of sections from Nakkula, M. J., & Harris, J. T. (in press).
Assessment of Mentoring Relationships. In DuBois, D. L., & Karcher, M. J. (Eds.), Handbook of Youth Mentoring
(pp. 100–117). Thousand Oaks, CA: Sage Publications Space limitations preclude a more in-depth consideration of some
points, but these are covered in detail within the chapter.
H O W T O S E L E C T A S U RV E Y T O A S S E S S Y O U R A D U LT – Y O U T H M E N T O R I N G P R O G R A M
173
focused, career focused or purely relationship focused) and its constituents (e.g., age, gender, and literacy level).
Other things to consider include the survey’s use of clear and age-appropriate language, the amount of time needed to administer it and the amount of insight it yields after it has been administered.
NOTES ON INSTRUMENTS WITH READILY AVAILABLE VALIDITY EVIDENCE
The following surveys are among those with the strongest available validity evidence. We provide only a few notes
about each to help you begin your consideration of which survey to use. If you would like more information
about any of them, you can read about them in the cited articles or contact the authors directly. Also, each is
reviewed in detail in the chapter of the Handbook of Youth Mentoring cited above.
Youth–Mentor Relationship Questionnaire (YMRQ; Roffman et al.) 2
• Designed for primary- and secondary school students (ages 9–16) (15 items in 4 subscales).
• Strengths: validity evidence published in peer-reviewed journal; correlates with length of match and academic
performance; derived from sample of items used in Public/Private Ventures’ landmark study of mentoring
(Grossman & Tierney, 1998).
• Limitations: negativity tendency among the survey’s items may limit its usefulness.
• Scope: assesses positive and negative subjective perspectives on relational–experiential and instrumental
indicators; does not measure objective or environmental dimensions.
The Youth Survey (Public/Private Ventures, 2002) 3
• Designed for primary and secondary school students (ages 9–16) (19 items in 3 subscales).
• Strengths: derived from the same sample of items as the YMRQ; comes closest to offering
standardized norms.
• Limitations: no published information about validation efforts or reliability of subscales.
• Scope: measures positive and negative subjective aspects of relational–experiential dimensions of the match;
does not assess objective, instrumental or environmental dimensions.
Match Characteristics Questionnaire v2.0 (Harris & Nakkula, 2003a) 4
• Designed for mentors of primary and secondary school students (62 items, 15 subscales).
• Strengths: validity evidence of earlier version (v1.1) published in a peer-reviewed journal; 5 is completed by
mentors; broad scope; has been successfully integrated into match supervision processes at the Yavapai
(Arizona) Big Brothers Big Sisters agency; correlates with academic outcomes.
• Limitations: validity evidence supporting version 2.0 not yet published.
• Scope: assesses positive, negative, subjective and objective perspectives on relational–experiential, instrumental
and environmental indicators.
Youth Mentoring Survey (Harris & Nakkula, 2003b) 6
• Designed for mentors of primary and secondary school students (45 items, 9 subscales).
• Strengths: broad scope; complements, correlates with Match Characteristics Questionnaire; has been
successfully integrated into match supervision processes at the Yavapai Big Brothers Big Sisters agency;
correlates with academic outcomes.
• Limitations: validity evidence not yet published.
• Scope: assesses positive and negative, subjective and objective, relational–experiential and instrumental
dimensions of MRQ; does not assess environmental indicators.
Relational Health Indices–Mentoring Scale (RHI-M) (Liang et al., 2002) 7
• Designed for female college students (11 items in 3 subscales).8
• Strengths: validity evidence published in peer-reviewed journal; unique theoretical perspective; provides an
assessment of natural mentoring relationships.
174
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
• Limitations: difficult to generalize findings from study involving female college students at liberal arts
women’s college.
• Scope: assesses subjective relational–experiential dimensions with some items related to instrumentality;
does not measure negative, objective or environmental dimensions.
Unnamed Mentoring Scale (Darling et al., 2002)9
• Designed for college students (4 items in 1 subscale).10
• Strengths: validity evidence published in peer-reviewed journal; demonstrated to be useful in two diverse
cultures (U.S./Japan); provides an assessment of natural mentoring relationships.
• Limitations: narrow scope; use of dichotomous (yes or no) ratings.
• Scope: assesses subjective ratings of instrumentality; does not measure negative, objective,
relational–experiential or environmental dimensions.
Instruments other than those reviewed above could be applied to MRQ assessment, but they lack sufficient
validity evidence to support their widespread use. For instance, Information Technology International (Mertinko
et al., 2000)11 and Big Brothers Big Sisters of America (Lyons & Curtis, 1998)12 have developed brief youth and
adult instruments that assess elements of relationship quality but are not supported by reliability and validity
evidence. A handful of researchers have developed qualitative designs to augment or complement their
quantitative work. DuBois et al. (2002)13 and Keller, Pryce and Neugebauer (2003)14 have made important
contributions that could inform your decisions about qualitative data collection.
SUMMARY
Given the free and easily accessible nature of the instruments described here, it may not be necessary to use all
of the subscales of specific instruments or even to use only one instrument. While longer instruments that assess
more constructs can generate more complete insight on relationship quality, this comprehensiveness may come
at a cost. Both youth and adults can become bored or frustrated by scales if they are too long, particularly if
they require multiple administrations or appear to contain undue overlap between items in the subscales.
Because the utility of MRQ assessments may be greatest when incorporated into regular programming
infrastructure, it is important to encourage participants’ buy-in. In such cases, participants should be made
aware at the outset that they will be asked to complete surveys regularly and should be helped to understand
why this process is important.
You will want to think carefully about when you administer the surveys. Although baseline data are prized in
program evaluation, it does not make sense to assess match quality before a relationship has had a chance to
develop. We believe it is most advantageous to administer MRQ assessments after the match has been meeting
regularly for about four months, to allow the match to progress beyond the initial awkwardness or honeymoon
stage. The interval between the initial and follow-up assessments should likewise allow sufficient time for the
relationship to evolve, likely about six months for the second administration and another six months for the
third. Thus, a typical administration schedule might be 4, 10 and 16 months after the match is made.
For matches that are still meeting after 18 months, a longer interval is likely to suffice.
Finally, survey instruments such as those described here may be easily administered but require the summation
and interpretation of scores, which will be enhanced by the involvement of trained researchers/evaluators.
Such external support for analysis ensures accuracy and lends credibility to interpretations of the data. While
professional evaluation support can be difficult for programs to afford, partnership with external evaluators is
vital to ensure that the interpretations upon which programming decisions and funding may be based have been
drawn accurately and responsibly from the data.
H O W T O S E L E C T A S U RV E Y T O A S S E S S Y O U R A D U LT – Y O U T H M E N T O R I N G P R O G R A M
175
2 Roffman, J., Ranjini, R., Rhodes, J., & Grossman, J. B. (in press). Promoting successful youth mentoring relationships:
A preliminary screening questionnaire. Journal of Primary Prevention.
3 Public/Private Ventures. (2002). Technical assistance packet #8: Measuring the quality of mentor-youth relationships:
A tool for mentoring programs. Portland, OR: Northwest Regional Educational Laboratory. Retrieved April 23, 2003,
from http://www.nwrel.org/mentoring/packets.html.
4 Harris, J. T., & Nakkula, M. J. (2003a). Match Characteristics Questionnaire v2.0. Unpublished measure, Harvard
Graduate School of Education, Cambridge, MA. Findings associated with the previous version (v1.1) may be found in
Karcher, M. J., Nakkula, M. J., & Harris, J. T. (in press). Developmental mentoring match characteristics: The effects
of mentors’ efficacy and mentees’ emotional support seeking on the perceived quality of mentoring relationships.
Journal of Primary Prevention.
5 Karcher, M. J., Nakkula, M. J., & Harris, J. T. (in press). Developmental mentoring match characteristics: The effects
of mentors’ efficacy and mentees’ emotional support seeking on the perceived quality of mentoring relationships.
Journal of Primary Prevention.
6 Harris, J. T., & Nakkula, M. J. (2003b). Youth Mentoring Survey. Unpublished measure, Harvard Graduate School of
Education, Cambridge, MA.
7 Liang, B., Tracy, A., Taylor, C. A., Williams, L. M., Jordan, J. V., & Miller, J. B. (2002). The relational health indices:
A study of women’s relationships. Psychology of Women Quarterly, 26, 25–35.
8 The RHI has been adapted for school-aged students, though no validity evidence for that instrument is yet available.
9 Darling, N., Hamilton, S., Toyokawa, T., & Matsuda, S. (2002). Naturally occurring mentoring in Japan and the
United States: Roles and correlates. American Journal of Community Psychology, 30(2), 245–270.
10 The RHI has been adapted for school-aged students, though no validity evidence for that instrument is yet available.
11 Mertinko, E., Novotney, L., Baker, T., & Lang, J. (2000). Evaluating your program: A beginner’s self-evaluation workbook
for mentoring programs. Washington, DC: Office of Juvenile Justice and Delinquency Prevention.
12 Lyons, M., & Curtis, T. (1998). Program Outcome-Based Evaluation (POE). Philadelphia, PA: BBBSA.
13 DuBois, D. L., Neville, H. A., Parra, G. R., & Pugh-Lilly, A. O. (2002). Testing a new model of mentoring.
New Directions for Youth Development, 93(Spring), 21–57.
14 Keller, T.E., Pryce, J.M., & Neugebauer, A. (2003) Observational methods for assessing the nature and course of
mentor-child interactions. Unpublished manual, University of Chicago.
176
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
LOGIC MODEL FOR GIRLPOWER!
177
FACILITIES
• Space for workshops and goalsetting and progress sessions
MATERIALS
• Program manual
• Supplies (participant handouts
and notebooks, disposable
cameras, picture puzzles,
refreshments, workshop
props)
* This program was developed through collaboration between Big
Brothers Big Sisters of Metropolitan Chicago (BBBS) and the
Girls Mentoring Project at University of Illinois at Chicago (UIC),
David DuBois, Ph.D., Director. FTE = full-time equivalent.
PREVENTION PROGRAMS
FOR GIRLS
• Lack of effectiveness
• Lack of gender-specific
strategies and content
EVALUATION
• Built-in program evaluation
activities
DIRECT SERVICES
• Bi-monthly workshop series
for mentors/youth: focused
on relationship and team
building, promotion
of healthy self-esteem,
prevention of risk behaviors/
promotion of healthy
behaviors (11 workshops total)
• Goal-setting and progress
sessions for individual
matches
• Between-session structured
activities for matches during
workshop series (Power
Builders)
• Continued one-on-one
interactions between mentors
and youth following
workshop series to 1-year
mark (includes Power
Builders)
• Group reunion session
DOSAGE
• Avg. # of workshop sessions
attended by mentors and
youth
• Parent attendance at
orientation & talent
show/graduation
• Avg. # of supervision
contacts/check-ins for
mentors/parents/youth
• Avg. # of goal-setting and
progress sessions completed
• Avg. hours of weekly one-oneone mentor/youth interactions
• Avg. # of Power Builders
completed
• % of evaluation materials
completed by staff/mentors/
youth/parents
• % of relationships sustained
one year
LONG TERM
• lrisky health behaviors:
substance use, violencerelated, unsafe sexual
behavior, self-harm, etc.
• jpositive health behaviors:
exercise, diet/nutrition, etc.
• lmental health problems:
internalizing (e.g., depression)
and externalizing (e.g.,
conduct disorder)
• jpositive mental health:
happiness, life satisfaction
• jsocial, educational,
occupational functioning at
later stages of development
INTERMEDIATE
• jself-esteem/self-efficacy
beliefs
• jsocial competence
• jskills for avoiding risky
behaviors/engaging in positive
health behaviors
• jquality of relationships with
parents, peers, and other
adults
INITIAL
• j social support from
non-parental adult (mentor):
emotional, companionship,
instrumental, informational
• jhealth-related
knowledge/attitudes
• jgender and racial identity
FIDELITY
• Implementation of training
sessions for staff and mentors
• Implementation of workshops
& reunion
• Quality of implementation of
training sessions, workshops,
supervision/check-ins, goalsetting, and progress sessions
• Mentor/staff satisfaction with
training
• Mentor/parent/youth
satisfaction with workshops,
supervision/ check-ins, goalsetting sessions, program
materials
• Youth/mentor/parent
satisfaction with mentoring
relationship
SUPPORTS
• Staff training and supervision
• Mentor training
• Bi-monthly supervision of
mentors and parent/youth
check-ins
FINANCIAL
• NIMH Grant Funding
• BBBS Subcontract
EARLY ADOLESCENT
URBAN, MINORITY GIRLS
• Low self-esteem
• Depression
• Victimization
• Health risk behaviors:
Diet/nutrition, exercise,
substance use, violence, risky
sexual behavior, self-harm
• Academic underachievement
PERSONNEL
• 2.20 FTE BBBS Staff
• Services of community
agencies (workshop
presentations)
• 10 BBBS female volunteer
mentors
• Consultation: UIC Research
Team
Program
Outcomes
Program
Outputs
Program
Activities
What are the benefits of
participating in the program?
Program
Inputs
What happens as a result of
the program?
Needs
What goes on in the program?
What goes into the program?
What needs does the
program address?
Logic Model for GirlPOWER!*
NOTES
179
NOTES
180
Section VIII.
Wrap-Up and Feedback
SUMMARY
Once again, congratulations on your commitment to quality youth mentoring.
Regardless of whether you reviewed the tool kit in its entirety or reviewed only those sections that were relevant
to your specific needs, we hope that you found it useful and that it included all the information and resources
you need to follow the guidelines in the Elements of Effective Practice. We encourage you to refer to the tool kit
often as you continue to build and strengthen your mentoring program.
WE NEED YOUR FEEDBACK
To ensure that the tool kit meets your needs, we are seeking your feedback on its content and suggestions for
improvement. Your feedback will enable us to enhance the online version of the tool kit (www.mentoring.
org/eeptoolkit) and to respond to emerging mentoring trends. Please complete the Tool Kit Evaluation Form
that follows, and mail or fax it to:
MENTOR/National Mentoring Partnership
Attention: National Mentoring Institute
1600 Duke Street, Suite 300
Alexandria, VA 22314
Fax: 703-226-2581
Thank you in advance for your input. We invite you to visit Mentoring.org often for the latest in mentoring
news, information and resources. Together, we can connect more of America’s young people with caring adult
mentors.
WRAP-UP AND FEEDBACK
181
SECTION VIII
How to Build a Successful Mentoring Program
Using the Elements of Effective Practice
EVALUATION FORM
What materials in the tool kit did you find most useful for your mentoring program?
_________________________________________________________________________________
_________________________________________________________________________________
What materials in the tool kit did you find least useful for your mentoring program?
_________________________________________________________________________________
_________________________________________________________________________________
Please rate the following sections:
Section III: Introduction to Mentoring and Program Building
Narrative
❑ Excellent
❑ Good
❑ Fair
Section IV: How to Design and Plan a Mentoring Program
Narrative
❑ Excellent
❑ Good
❑ Fair
List of Additional Resources
❑ Excellent
❑ Good
❑ Fair
Checklist of Program Progress
❑ Excellent
❑ Good
❑ Fair
Tools
❑ Excellent
❑ Good
❑ Fair
What tools in Program Design and Planning did you find most useful?
_________________________________________________________________________________
_________________________________________________________________________________
What tools in Program Design and Planning did you find least useful?
_________________________________________________________________________________
_________________________________________________________________________________
E VA L U AT I O N F O R M
183
Section V: How to Manage a Program for Success
Narrative
❑ Excellent
❑ Good
❑ Fair
List of Additional Resources
❑ Excellent
❑ Good
❑ Fair
Checklist of Program Progress
❑ Excellent
❑ Good
❑ Fair
Tools
❑ Excellent
❑ Good
❑ Fair
What tools in Program Management did you find most useful?
_________________________________________________________________________________
_________________________________________________________________________________
What tools in Program Management did you find least useful?
_________________________________________________________________________________
_________________________________________________________________________________
Section VI: How to Structure Effective Program Operations
Narrative
❑ Excellent
❑ Good
❑ Fair
List of Additional Resources
❑ Excellent
❑ Good
❑ Fair
Checklist of Program Progress
❑ Excellent
❑ Good
❑ Fair
Tools
❑ Excellent
❑ Good
❑ Fair
What tools in Program Operations did you find most useful?
_________________________________________________________________________________
_________________________________________________________________________________
What tools in Program Operations did you find least useful?
_________________________________________________________________________________
_________________________________________________________________________________
184
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE
SECTION VIII
Section VII: How to Establish Evaluation Criteria and Methods
Narrative
❑ Excellent
❑ Good
❑ Fair
List of Additional Resources
❑ Excellent
❑ Good
❑ Fair
Checklist of Program Progress
❑ Excellent
❑ Good
❑ Fair
Tools
❑ Excellent
❑ Good
❑ Fair
What tools in Evaluation Criteria and Methods did you find most useful?
_________________________________________________________________________________
_________________________________________________________________________________
What tools in Evaluation Criteria and Methods did you find least useful?
_________________________________________________________________________________
_________________________________________________________________________________
Feedback
Please provide your comments on the overall content of the tool kit and any suggestions for additional
information or improvement.
Please send the completed Tool Kit Evaluation Form by mail or fax to:
MENTOR/National Mentoring Partnership
Attention: National Mentoring Institute
1600 Duke Street, Suite 300
Alexandria, VA 22314
Fax: 703-226-2581
E VA L U AT I O N F O R M
185
ENDNOTES
Section II
1.
Elements of Effective Practice, second edition, MENTOR/National Mentoring Partnership, 2003.
Section IV
1.
Some information is adapted from Business Guide to Youth Mentoring, The Connecticut Mentoring
Partnership.
Section V
1.
Document Kit, Friends for Youth, Mentoring Institute, 2003.
2.
Dr. Susan G. Weinberger, president, Mentor Consulting Group, 2004.
3.
Partnerships for Success: A Mentoring Program Manual, The Enterprise Foundation and the United Way of
America, 1990.
4.
Partnerships for Success, 1990.
5.
Partnerships for Success, 1990.
6.
Adapted from “Getting your Message Heard: Understanding Nonprofit Marketing,” MENTOR/National
Mentoring Partnership Online Community Forum, November 2003, Mentoring.org/community.
7.
The Community Collaboration Manual, The National Assembly of National Voluntary Health and Social
Welfare Organizations, 1991.
Section VI
1.
“Mentoring: A Promising Strategy for Youth Development,” Susan M. Jekielek, M.A., Kristin A. Moore,
Ph.D., Elizabeth C. Hair, Ph.D., and Harriet J. Scarupa, M.S., Child Trends Research Brief, Washington, D.C.,
February 2002, p. 6.
2.
Allstate Insurance Co. Mentoring Program, Dr. Susan G. Weinberger, Norwalk, CT: Mentor Consulting Group,
2000.
3.
“Mentoring in America 2002,” MENTOR/AOL Time Warner Foundation, Mentoring.org/poll.
4.
Partnerships for Success: A Mentoring Program Manual, The Enterprise Foundation and the United Way of
America, 1990.
5.
“Mentoring in America 2002.”
6.
Mentoring: A Guide for Local Broadcasters, National Association of Broadcasters and Harvard Mentoring
Project, 2002.
ENDNOTES
187
7.
“Mentoring Matters: A National Survey of Adults Mentoring Young People,” The Commonwealth Fund,
K.T. McLearn, D. Colasanto and C. Schoen, in J.B. Grossman (ed.), Contemporary Issues in Mentoring,
Philadelphia: Public/Private Ventures, 1998, pp. 67–83.
8.
Connections Newsletter, Friends for Youth Mentoring Institute, Fall 2003. www.friendsforyouth.org
9.
Adapted from Vision to Reality: Mentoring Program Development Guide, The Maryland Mentoring Partnership,
2003.
10.
Mentor screening information compiled from The ABCs of Mentoring, Mentoring Partnership of Long Island,
2003 and adapted from Mentoring Manual: A Guide to Program Development and Implementation, The Abell
Foundation, Inc., Baltimore, MD.
11.
Partnerships for Success, 1990.
12.
Vision to Reality, 2003.
13.
Adapted in part from Mentoring in the Faith Community: An Operations Manual for Program Coordinators,
MENTOR/National Mentoring Partnership and The Mentoring Partnership of New York, 2002.
Section VII
188
1.
Dr. David DuBois, Institute for Health Research and Policy, University of Illinois at Chicago, 2004.
2.
Partnerships for Success, 1990.
3.
Substantial portions of these sections were adapted from Program Evaluation: Methods and Case Studies,
E.J. Posavac & Carey, Englewood Cliffs, NJ: Prentice-Hall; and “Assessing the Effectiveness of Mentoring
Programs,” J.B. Grossman and A. Johnson, in J.B. Grossman (ed.), Contemporary Issues in Mentoring.
Philadelphia: Public/Private Ventures, 1998.
4.
Grossman and Johnson, 1998.
5.
“Effectiveness of Mentoring Programs for Youth: A Meta-Analytic Review,” D.L. DuBois, B.E. Holloway,
J.C. Valentine, and H. Cooper, American Journal of Community Psychology, 30 (2002), pp. 157–197.
6.
“Toward Predicting Successful Youth Mentoring Relationships: A Preliminary Screening Questionnaire,”
J. Roffman, R. Reddy, and J. Rhodes, 2002. (Submitted for publication.)
7.
Making a Difference: An Impact Study of Big Brothers/Big Sisters (reissue of 1995 study), Joseph P. Tierney and
Jean Baldwin Grossman, with Nancy L. Resch, 2000. A publication of Public/Private Ventures.
HOW TO BUILD A SUCCESSFUL MENTORING PROGRAM USING THE ELEMENTS OF EFFECTIVE PRACTICE