Mission Skills Assessment - 2017 NAIS Annual Conference

Transcription

Mission Skills Assessment - 2017 NAIS Annual Conference
Mission Skills Assessment
Assessing Character Traits in Middle School Students
1 Assessment
INDEX Mission Skills
Independent Schools Data Exchange
www.indexgroups.org
617-413-8382
Mission Skills Assessment
Mission Skills Assessment: A Tool To Alter
The Way Schools Think About Education
MSA measures six character strengths—Teamwork, Creativity, Ethics,
Resilience, Curiosity, Time Management—that have proven essential
for success in school and in life. The assessment gives each school a more
scientific way to measure its curriculum’s success at meeting the goals
outlined in its mission.
INDEX Mission Skills Assessment
2
Who We Are
•  INDEX Elementary Schools: National benchmarking and
best practices group of 28 Independent PS – 8/9 coeducational day schools with 400+ students.
•  Center for Academic and Workforce Readiness and Success,
Educational Testing Services (ETS).
INDEX Mission Skills Assessment
3
MSA Schools
Duke School (NC)
Episcopal Day School (GA)
Far Hills Country Day School (NJ)
The Foote School (CT)
Grace-Saint Luke’s Episcopal School (TN)
Green Vale School (NY)
Greenwich Country Day School (CT)
The Lexington School (KY)
Marin Country Day School (CA)
New Canaan Country School (CT)
Old Trail School (OH)
The Pike School (MA)
Rippowam Cisqua School (NY)
Rodeph Sholom School (NY)
Rumson Country Day School (NJ)
Saint Martin’s Episcopal School (GA)
Shady Hill School (MA)
Shore Country Day School (MA)
St. Patrick’s Episcopal School (DC)
Trinity Episcopal School (NC)
INDEX Mission Skills Assessment
4
What is the Mission Skills Assessment
The MSA is a longitudinal assessment measuring and
benchmarking student achievement and improvement in core
mission skills.
•  Teamwork
•  Creativity
•  Ethics
•  Resilience
•  Curiosity
•  Time Management
INDEX Mission Skills Assessment
5
Roadmap
•  R&D
•  ETS
•  Constructs
•  Cognitive Labs
•  Pilot Test
•  Fall Test
•  Spring Test
•  Horizons
• 1st Report
•  Fall Test
•  2nd Report
INDEX Mission Skills Assessment
•  Fall Test
•  INDEX Schools
•  3rd Report
6
The Assessment
Web-based.
Student self-assessment
•  2 30-minute tests for approximately 60 minutes total.
•  Assess once per year.
•  SJT and other student-completed performance measures.
•  6th, 7th, and 8th graders.
•  The test is fun! There are no wrong answers.
•  Teacher-rater assessment
•  Teacher rates each student individually
•  Outcome data (e.g. test scores, grades, absences, etc.)
•  Institutional focus. No tracking of individual performance.
• 
• 
INDEX Mission Skills Assessment
7
Noncognitive Skills Research
INDEX Mission Skills Assessment
8
What are Noncognitive skills?
•  Not traditionally “tested” by standardized
assessments
•  Not new but newly important; labeled “21st century
skills”
•  Essential capacities necessary for success in school
and in life
•  Often taught implicitly, but can be taught explicitly
INDEX Mission Skills Assessment
9
Attitudes &
Beliefs
Social &
Emotional
Qualities
Habits &
Processes
Personality
Attitudes
(evaluations)
Emotion
Management
Time
Management
Openness to
Experience
Intelligence
Theories
Test Anxiety
Goal Setting
Conscientiousness
Self-Efficacy
Self-Regulation
Test-taking
Strategies
Extraversion
Self-Concept
Coping with
Stress
Study Skills
Agreeableness
Confidence
Teamwork
Self-Monitoring
Emotional
Stability
Lipnevich, MacCann, & Roberts (2011)
INDEX Mission Skills Assessment
10
Disposition Toward School
GPA
Class
Absences
Discipline
Infirmary
Visits
SSAT
.27
-.04
-.05
-.04
Conscientiousness
.21
-.31
-.23
-.22
Teacher ratings
(noncognitive)
.38
-.35
-.42
-.26
INDEX Mission Skills Assessment
11
Grades
rho
Conscientiousness
Openness
Agreeableness
Emotional Stability
Extraversion
Cognitive ability
Primary Secondary
.28
.21
.24
.12
.30
.05
.20
.01
.18
.58
-.01
.24
Tertiary
.23
.07
.06
-.01
-.03
.23
Poropat, A. (2009). Psychological Bulletin
INDEX Mission Skills Assessment
12
Academic Success
Noncognitive skills correlate positively
with achievement on NAEP and PISA.
Noncognitive factors predict college grades
as strongly as cognitive variables do.
ETS, (2008); Poropat, (2009)
INDEX Mission Skills Assessment
13
Life Trajectory
Early childhood interventions do not
raise IQ, but improve noncognitive
skills which affect education,
employment, earnings, and crime.
Heckman et al. (2010)
INDEX Mission Skills Assessment
14
Life Satisfaction
Noncognitive variables…have been
demonstrated to predict happiness,
health, marital satisfaction, and peer
relationships.
Diener & Lucas (1999); Bogg & Roberts (2004); Watson, Hubbard, & Wiese, (2000); JensenCampbell et al. (2002)
INDEX Mission Skills Assessment
15
Demographics
The benefits of higher noncognitive skills
are demonstrable across IQ, socioeconomic class, gender, and race.
Tough, Paul. How Children Succeed: Grit, Curiosity and the Hidden Power of Character.
(2012)
INDEX Mission Skills Assessment
16
Ethics
100%
Creativity
67%
Love of Learning
73%
Teamwork
40%
INDEX Mission Skills Assessment
Resilience
67%
Analysis of
mission
statements of 20
independent
middle schools
17
Analysis of university mission statements
(Oswald et. al., 2004)
Intellectual
Interpersonal
Intrapersonal
Knowledge and mastery
of general principles
Appreciation for
diversity
Social responsibility and
citizenship
Continuous learning,
intellectual interest and
curiosity
Leadership
Physical and
psychological health
Artistic and cultural
appreciation
Interpersonal skills
Career orientation
Adaptability and life
skills
Perseverance
Ethics and integrity
INDEX Mission Skills Assessment
18
“Very Important”
Percent Rating
*
*
*
*
*
*
The Conference Board, Corporate Voices for Working Families, Partnership for 21st Century
Working Skills, and the Society for Human Resource Management.
INDEX Mission Skills Assessment
19
The Character Imperative
1) Research shows that these skills lead to
positive educational and life outcomes;
2) Colleges, universities, and employers are
seeking these skills in students and
workers; and
3) We must demonstrate the value we add.
INDEX Mission Skills Assessment
20
Guiding Children to Achieve their
Full Potential
Students are provided with a distinct
advantage if their schools excel in training
noncognitive skills. Independent Schools
can lead the way in designing innovative
character assessments and intentional
classroom strategies.
INDEX Mission Skills Assessment
21
Constructs in MSA
INDEX Mission Skills Assessment
22
Teamwork
Models of teamwork include
•  Cooperation
•  Influence
•  Conflict resolution
•  Guiding others
100
2nd most valued work
attribute according to Are they
Really Ready to Work? report
60
90
80
70
50
Ma
th
IT
Wr
E
E
C
R
W
L
C
L
O
T
W
itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
al
i
n
i
w
g
nC
E
n
v
h
r
o
g
s
i
g
Th
hip m. ork thic
Le ty
om
ink Com
arn
.
ing
p
ing
In high school students, teacher reports of teamwork were correlated with grades
across several subjects (Wang, MacCann, Zhuang, Liu, & Roberts, 2009)
INDEX Mission Skills Assessment
23
Creativity
•  Originality, progressiveness, or imagination
•  Dictionary.com
•  Tasks measuring creativity, “require examinees fairly
quickly to think of…a series of responses fitting the
requirements of the task….”
•  Carroll, 1993; p. 428
INDEX Mission Skills Assessment
24
Creativity
100
5th
most valued work
attribute according to Are
they Really Ready to Work?
report
90
80
70
60
50
Ma
th
IT
Wr
E
E
C
R
Wr
L
C
L
O
T
Wo
itte thics nglis ritic ead
itin ifelo reat eade ral C eam
rk
a
i
n
i
w
g
nC
n
v
h
rsh
lT
om
gL
ity
ork Ethi
hin g Co
om
i
.
c
ea
p
kin
mp
rn i
.
g
ng
Creativity predicts graduate school outcomes beyond GRE verbal and mathematics
scores (Frederickson & Ward, 1975; Bennett & Rock, 1995; Bennett & Rock, 1998)
INDEX Mission Skills Assessment
25
Ethics
Defined in our study as
cheating and concern for
others
One of the most valued work
attribute according to Are they
Really Ready to Work? report
100
90
80
70
60
50
Ma
th
IT
Wr
E
E
C
R
W
L
C
L
O
T
W
itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
al
i
n
i
w
g
nC
n
v
h
r
o
g
sh
ork Ethi
g
Th
Le ity
om
ip m.
c
ink Com
a
rn i
.
ing
p
ng
INDEX Mission Skills Assessment
26
Resilience
•  A level of adaptability that allows individuals to survive
and thrive in adverse conditions
•  Dent and Cameron (2003)
•  One way we measure resilience in the MSA is to measure
core self-evaluation – or how capable, worthy, and effective
a person feels.
INDEX Mission Skills Assessment
27
2002 Income
1979 Core Self Evaluation
Judge and Hurst (2007)
INDEX Mission Skills Assessment
28
Intrinsic Motivation / Love of
Learning / Curiosity
•  What people will do without external inducement
(Malone & Lepper, 1987)
•  In a study of 3rd – 8th graders, intrinsic motivation was
positively correlated with:
•  Grades: r = .34
•  Standardized Test Scores = r = .27
•  However…
INDEX Mission Skills Assessment
29
(Lepper, Corpus, & Iyengar, 2005)
INDEX Mission Skills Assessment
30
Time Management
Determining one’s needs, setting goals to meet needs, and
prioritizing and planning to meet goals (Lakein, 1973).
Planning
Organization
Grades
.21**
.38**
Grades
(6 months later)
.25**
.42**
Liu, Rijmen, MacCann, & Roberts (2009)
INDEX Mission Skills Assessment
31
How are we doing it?
INDEX Mission Skills Assessment
32
Our GPS:
Multi-Trait Multi-Method (MTMM) Design
Self Ratings
Not directly
observable
True
True
Resiliency
Resilience
Alternative
Method
“Triangulation”
Measurement
Method
Teacher
Ratings
The underlying view of measurement in the MTMM analysis is that to measure a theoretical construct,
different measures, each with its own bias, are selected. Bias that is due to method effects is reduced through a
triangulation process.
Baron & Kashy, 1992, p. 170
INDEX Mission Skills Assessment
34
Item Types
INDEX Mission Skills Assessment
35
Self-Report
INDEX Mission Skills Assessment
36
Situational Judgment Test
INDEX Mission Skills Assessment
37
Biographical Data
INDEX Mission Skills Assessment
38
Creativity Performance Test
INDEX Mission Skills Assessment
39
Teacher Ratings
INDEX Mission Skills Assessment
40
Resilience:
Representative Items
Situational Judgment Test
You are feeling stressed about the amount of
homework that you have been given by your
teacher(s). Below are some ways that you
might think, feel, or act in this situation, right
at the time that you feel stressed-out. Rate how
often you do each activity when you feel
stressed. How do you think, feel, or act when
you are stressed from having too much
homework to do?
1.  I take control and say to myself: I can do
this!
INDEX Mission Skills Assessment
Self-Report
I overcome challenges and set
backs.
Resilience
Teacher Ratings
This student is easily
discouraged.
41
What did we find?
INDEX Mission Skills Assessment
42
Participation
• 
Fall 2011
•  18 INDEX schools
•  2,081 students
• 
Spring 2012 (optional assessment)
•  13 INDEX schools
•  1,677 students
• 
Fall 2012
•  20 INDEX schools
•  > 2,600 students
•  Data being analyzed
INDEX Mission Skills Assessment
43
Average'Construct'Scores'By'Gender
103
Average'Construct'Score
102
101
100
99
98
Male
97
Female
96
95
94
2011 2012
2011 2012
Creativity
Ethics
2011 2012
2011 2012
Intrinsic9Motivation
Resilience
Construct
INDEX Mission Skills Assessment
2011 2012
2011 2012
Teamwork
Time9Management
44
The MSA Demonstrates
Evidence of Reliability
INDEX Mission Skills Assessment
45
Reliability
Construct!
Time Management!
Resilience!
Intrinsic Motivation!
Creativity!
Teamwork!
Ethics!
2012
2011
Reliability! Reliability!
0.86!
0.86!
0.85!
0.84!
0.90!
0.89!
0.88!
0.88!
0.85!
0.77!
0.85!
0.90!
INDEX Mission Skills Assessment
SAT
Reliability
Math
.91
Critical
Reading
.91
46
The MSA Demonstrates
Evidence of Validity
INDEX Mission Skills Assessment
47
The MSA Predicts
•  Grades
•  Teacher Ratings of Student Quality
•  “How would you rate this student compared to other students in
your school?”
INDEX Mission Skills Assessment
48
r = .56
2012 Student Quality Rating
Creativity and Student Quality Rating
4
3.5
3
2.5
2
1.5
1
0.5
0
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Creativity
INDEX Mission Skills Assessment
49
How “Big” Are Our
Correlations?
Examples from other research (Meyer et al, 2001)
Variable 1
Variable 2
Correlation
Sugar Consumption
Child’s Behavior
.00
Aspirin
Heart Attack Death
.02
Parental Divorce
Child Well-Being
.09
Lead Exposure
Child IQ
.12
Ibuprofen
Pain Reduction
.14
Alcohol
Aggressive
Behavior
.23
Viagra
Sexual Functioning
.38
Gender
Height
.67
INDEX Mission Skills Assessment
50
6th r = .29
7th r = .59
8th r = .58
Spring 2012 GPA
Time Management and GPA by Grade
4
3.8
3.6
3.4
3.2
3
2.8
2.6
2.4
2.2
2
6
7
8
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Time Management
INDEX Mission Skills Assessment
51
The MSA Predicts
•  Student Well Being
• My#life#is#going#well#
• My#life#is#just#right#
• I#would#like#to#change#many#things#in#my#life#
• I#wish#I#had#a#different#kind#of#life#
• I#have#a#good#life#
• I#have#what#I#want#in#life#
• My#life#is#be;er#than#most#kids’#
INDEX Mission Skills Assessment
52
r = .33
Resilience!and!Well!Being!
5!
2012!Well!Being!
4.9!
4.8!
4.7!
4.6!
4.5!
4.4!
4.3!
4.2!
4.1!
A!-!Top!20%!
B!-!60%-79%!
C!-!40%-59%! D!-!20%-39%! E!-!Bo7om!20%!
2011!Resilience!
INDEX Mission Skills Assessment
53
r = .31
2012 Well Being
Ethics and Well Being
5
4.9
4.8
4.7
4.6
4.5
4.4
4.3
4.2
4.1
A - Top 20%
B - 60%-79%
C - 40%-59%
2011 Ethics
INDEX Mission Skills Assessment
D - 20%-39% E - Bottom 20%
54
There is initial evidence that the
MSA predicts just as well as
standardized tests
INDEX Mission Skills Assessment
55
Correlation with Student Quality
MSA and ERB Scores: Correlations with Student
Quality Ratings
0.6
0.5
0.4
0.50
0.46
0.45
0.44
0.38
0.31
0.3
0.16
0.2
0.15
0.1
0
INDEX Mission Skills Assessment
56
MSA and ERB Scores: Correlations with GPA
Correlation with GPA
0.7
0.6
0.5
0.59
0.56
0.52
0.49
0.41
0.4
0.28
0.3
0.2
0.27
0.21
0.1
0
INDEX Mission Skills Assessment
57
Correlation with Well Being
MSA and ERB Scores: Correlations with Well Being
0.3
0.25
0.2
0.28
0.23
0.22
0.22
0.19
0.14
0.15
0.1
0.04
0.05
0.01
0
INDEX Mission Skills Assessment
58
Current and Next Steps
•  Continual refinement of MSA assessment
(adding interests)
•  Assessing underserved populations with MSA
•  Gathering interest from additional schools
INDEX Mission Skills Assessment
59
Implications for Schools
INDEX Mission Skills Assessment
60
Implications for Schools
•  Wide-ranging and deep:
•  Why?
•  What?
•  How?
INDEX Mission Skills Assessment
61
Why? Mission and Vision
•  Reexamine school mission and how
measured
•  Strength of mind and strength of character
•  Develop a Vision for Character
INDEX Mission Skills Assessment
62
How?
Programsintentional and
well-designed
High-quality
training and
support
Evidencebased
Evaluation
INDEX Mission Skills Assessment
63
What? Programs
How can we teach these skills better and how will we know we
are succeeding?
•  Deliberate, explicit and implicit instruction
•  Proven interventions
•  Performance Character Audit
•  Articulated, spiraled, integrated curriculum
•  Grade-level goals
•  Updated and analyzed curriculum maps
INDEX Mission Skills Assessment
64
Programs
•  Social Emotional Learning, advisories, personal
awareness, and co-curricula
•  Shared lexicon, common language-Glossary of
Terms for Character
•  Best Practices-follow through, evaluation,
refinement
INDEX Mission Skills Assessment
65
What? Training and Support
•  Ongoing education–school leaders, faculty,
parents
•  Shared vision, vocabulary, and resources
•  Participation in regional and national educator
character summits
•  Ongoing partnerships and dialogue
INDEX Mission Skills Assessment
66
What? Evidence-based
Evaluation
•  Updated Progress Reports
•  Parent and student conferences
•  Faculty and parent education summits
•  Greater emphasis: Self-reflection and meta-cognition
•  Surveys
INDEX Mission Skills Assessment
67
Evidence-based Evaluation
•  Integrated assessments
•  Participation in new assessments (MSA)
•  Benchmarking
•  Data for national accreditation and new
criterion
INDEX Mission Skills Assessment
68
Measuring what we value:
Criterion 13:
The Standards require a school to provide evidence
of a thoughtful process, respectful of its mission,
for the collection and use in school decision
making of data (both external and internal) about
student learning.
INDEX Mission Skills Assessment
69
Improvement Phases
Envision
Implement
Evaluate and Refine
Commitment tied to
mission
Needs and resources
assessment
Professional
Development &
Partnerships
Create clear vision
Action Plans,
integration, and
enhancements
Communication and
engagement
Build engagement
Evidence-based
strategies and programs
•  What is working?
•  How do we know?
•  How can we improve
teaching and learning?
INDEX Mission Skills Assessment
70
Community Responses
•  Excitement!
•  Shared understanding
•  Partnership
•  Shared commitment to studying results and asking the next
questions…
INDEX Mission Skills Assessment
71
Your School, Your Mission
Look to your mission, look to your alumni, look
to your purpose and your school’s view of what
our society needs the most: creative and
innovative thinkers, people of fine judgment,
leaders, collaborators, and…then select or design
the instruments which will help you make these
predictions.
Martin, Jonathan. Lessons from Law School Admissions: More from USC Attributes that Matter, 21K12
Blog, January 23, 2013.
INDEX Mission Skills Assessment
72
Thank you!
Questions?
Comments?
FOR MORE INFORMATION, PLEASE CONTACT:
INDEX
INDEPENDENT SCHOOL DATA EXCHANGE
LISAPULLMAN@INDEXGROUPS.ORG
WWW.INDEXGROUPS.ORG
MSA Steering Committee:
Chuck Baldecchi
Head of School
The Lexington School (KY)
CBaldecchi@thelexingtonschool.org
Tim Bazemore
Head of School
New Canaan Country School (CT)
TBazemore@countryschool.net
Dave Michelman
Head of School
Duke School (NC)
Dave.Michelman@dukeschool.org
Ned Murray
Head of School
Episcopal Day School (GA)
nmurray@edsaugusta.com
Jeremy Burrus
Research Scientist
ETS
jburrus@ets.org
Jennifer Phillips
Director of Educational Advancement
Far Hills Country Day School (NJ)
jphillips@fhcds.org
INDEX Mission Skills Assessment
Lisa Pullman
Executive Director, INDEX
lisapullman@indexgroups.org
Rich Roberts
Managing Principal Research Scientist
ETS
rroberts@ets.org
73