Mission Skills Assessment - 2017 NAIS Annual Conference

Transcription

Mission Skills Assessment
Assessing Character Traits in Middle School Students
1 Assessment
INDEX Mission Skills
Independent Schools Data Exchange
www.indexgroups.org
617-413-8382
Mission Skills Assessment
Mission Skills Assessment: A Tool To Alter
The Way Schools Think About Education
MSA measures six character strengths—Teamwork, Creativity, Ethics,
Resilience, Curiosity, Time Management—that have proven essential
for success in school and in life. The assessment gives each school a more
scientific way to measure its curriculum’s success at meeting the goals
outlined in its mission.
INDEX Mission Skills Assessment
2
Who We Are
•  INDEX Elementary Schools: National benchmarking and
best practices group of 28 Independent PS – 8/9 coeducational day schools with 400+ students.
•  Center for Academic and Workforce Readiness and Success,
Educational Testing Services (ETS).
3
MSA Schools
Duke School (NC)
Episcopal Day School (GA)
Far Hills Country Day School (NJ)
The Foote School (CT)
Grace-Saint Luke’s Episcopal School (TN)
Green Vale School (NY)
Greenwich Country Day School (CT)
The Lexington School (KY)
Marin Country Day School (CA)
New Canaan Country School (CT)
Old Trail School (OH)
The Pike School (MA)
Rippowam Cisqua School (NY)
Rodeph Sholom School (NY)
Rumson Country Day School (NJ)
Saint Martin’s Episcopal School (GA)
Shady Hill School (MA)
Shore Country Day School (MA)
St. Patrick’s Episcopal School (DC)
Trinity Episcopal School (NC)
4
What is the Mission Skills Assessment
The MSA is a longitudinal assessment measuring and
benchmarking student achievement and improvement in core
mission skills.
•  Teamwork
•  Creativity
•  Ethics
•  Resilience
•  Curiosity
•  Time Management
5
Roadmap
•  R&D
•  ETS
•  Constructs
•  Cognitive Labs
•  Pilot Test
•  Fall Test
•  Spring Test
•  Horizons
• 1st Report
•  Fall Test
•  2nd Report
•  Fall Test
•  INDEX Schools
•  3rd Report
6
The Assessment
Web-based.
Student self-assessment
•  2 30-minute tests for approximately 60 minutes total.
•  Assess once per year.
•  SJT and other student-completed performance measures.
•  6th, 7th, and 8th graders.
•  The test is fun! There are no wrong answers.
•  Teacher-rater assessment
•  Teacher rates each student individually
•  Outcome data (e.g. test scores, grades, absences, etc.)
•  Institutional focus. No tracking of individual performance.
• 
• 
7
Noncognitive Skills Research
8
What are Noncognitive skills?
•  Not traditionally “tested” by standardized
assessments
•  Not new but newly important; labeled “21st century
skills”
•  Essential capacities necessary for success in school
and in life
•  Often taught implicitly, but can be taught explicitly
9
Attitudes &
Beliefs
Social &
Emotional
Qualities
Habits &
Processes
Personality
Attitudes
(evaluations)
Emotion
Management
Time
Management
Openness to
Experience
Intelligence
Theories
Test Anxiety
Goal Setting
Conscientiousness
Self-Efficacy
Self-Regulation
Test-taking
Strategies
Extraversion
Self-Concept
Coping with
Stress
Study Skills
Agreeableness
Confidence
Teamwork
Self-Monitoring
Emotional
Stability
Lipnevich, MacCann, & Roberts (2011)
10
Disposition Toward School
GPA
Class
Absences
Discipline
Infirmary
Visits
SSAT
.27
-.04
-.05
-.04
Conscientiousness
.21
-.31
-.23
-.22
Teacher ratings
(noncognitive)
.38
-.35
-.42
-.26
11
Grades
rho
Conscientiousness
Openness
Agreeableness
Emotional Stability
Extraversion
Cognitive ability
Primary Secondary
.28
.21
.24
.12
.30
.05
.20
.01
.18
.58
-.01
.24
Tertiary
.23
.07
.06
-.01
-.03
.23
Poropat, A. (2009). Psychological Bulletin
12
Academic Success
Noncognitive skills correlate positively
with achievement on NAEP and PISA.
Noncognitive factors predict college grades
as strongly as cognitive variables do.
ETS, (2008); Poropat, (2009)
13
Life Trajectory
Early childhood interventions do not
raise IQ, but improve noncognitive
skills which affect education,
employment, earnings, and crime.
Heckman et al. (2010)
14
Life Satisfaction
Noncognitive variables…have been
demonstrated to predict happiness,
health, marital satisfaction, and peer
relationships.
Diener & Lucas (1999); Bogg & Roberts (2004); Watson, Hubbard, & Wiese, (2000); JensenCampbell et al. (2002)
15
Demographics
The benefits of higher noncognitive skills
are demonstrable across IQ, socioeconomic class, gender, and race.
Tough, Paul. How Children Succeed: Grit, Curiosity and the Hidden Power of Character.
(2012)
16
Ethics
100%
Creativity
67%
Love of Learning
73%
Teamwork
40%
Resilience
67%
Analysis of
mission
statements of 20
independent
middle schools
17
Analysis of university mission statements
(Oswald et. al., 2004)
Intellectual
Interpersonal
Intrapersonal
Knowledge and mastery
of general principles
Appreciation for
diversity
Social responsibility and
citizenship
Continuous learning,
intellectual interest and
curiosity
Leadership
Physical and
psychological health
Artistic and cultural
appreciation
Interpersonal skills
Career orientation
Adaptability and life
skills
Perseverance
Ethics and integrity
18
“Very Important”
Percent Rating
*
*
*
*
*
*
The Conference Board, Corporate Voices for Working Families, Partnership for 21st Century
Working Skills, and the Society for Human Resource Management.
19
The Character Imperative
1) Research shows that these skills lead to
positive educational and life outcomes;
2) Colleges, universities, and employers are
seeking these skills in students and
workers; and
3) We must demonstrate the value we add.
20
Guiding Children to Achieve their
Full Potential
Students are provided with a distinct
advantage if their schools excel in training
noncognitive skills. Independent Schools
can lead the way in designing innovative
character assessments and intentional
classroom strategies.
21
Constructs in MSA
22
Teamwork
Models of teamwork include
•  Cooperation
•  Influence
•  Conflict resolution
•  Guiding others
100
2nd most valued work
attribute according to Are they
Really Ready to Work? report
60
90
80
70
50
Ma
th
IT
Wr
E
E
C
R
W
L
C
L
O
T
W
itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
al
i
n
i
w
g
nC
E
n
v
h
r
o
g
s
i
g
Th
hip m. ork thic
Le ty
om
ink Com
arn
.
ing
p
ing
In high school students, teacher reports of teamwork were correlated with grades
across several subjects (Wang, MacCann, Zhuang, Liu, & Roberts, 2009)
23
Creativity
•  Originality, progressiveness, or imagination
•  Dictionary.com
•  Tasks measuring creativity, “require examinees fairly
quickly to think of…a series of responses fitting the
requirements of the task….”
•  Carroll, 1993; p. 428
24
Creativity
100
5th
most valued work
attribute according to Are
they Really Ready to Work?
report
90
80
70
60
50
Ma
th
IT
Wr
E
E
C
R
Wr
L
C
L
O
T
Wo
itte thics nglis ritic ead
itin ifelo reat eade ral C eam
rk
a
i
n
i
w
g
nC
n
v
h
rsh
lT
om
gL
ity
ork Ethi
hin g Co
om
i
.
c
ea
p
kin
mp
rn i
.
g
ng
Creativity predicts graduate school outcomes beyond GRE verbal and mathematics
scores (Frederickson & Ward, 1975; Bennett & Rock, 1995; Bennett & Rock, 1998)
25
Ethics
Defined in our study as
cheating and concern for
others
One of the most valued work
attribute according to Are they
Really Ready to Work? report
100
90
80
70
60
50
Ma
th
IT
Wr
E
E
C
R
W
L
C
L
O
T
W
itte thics nglis ritic ead ritin ifelo reat eade ral C eam ork
al
i
n
i
w
g
nC
n
v
h
r
o
g
sh
ork Ethi
g
Th
Le ity
om
ip m.
c
ink Com
a
rn i
.
ing
p
ng
26
Resilience
•  A level of adaptability that allows individuals to survive
and thrive in adverse conditions
•  Dent and Cameron (2003)
•  One way we measure resilience in the MSA is to measure
core self-evaluation – or how capable, worthy, and effective
a person feels.
27
2002 Income
1979 Core Self Evaluation
Judge and Hurst (2007)
28
Intrinsic Motivation / Love of
Learning / Curiosity
•  What people will do without external inducement
(Malone & Lepper, 1987)
•  In a study of 3rd – 8th graders, intrinsic motivation was
positively correlated with:
•  Grades: r = .34
•  Standardized Test Scores = r = .27
•  However…
29
(Lepper, Corpus, & Iyengar, 2005)
30
Time Management
Determining one’s needs, setting goals to meet needs, and
prioritizing and planning to meet goals (Lakein, 1973).
Planning
Organization
Grades
.21**
.38**
Grades
(6 months later)
.25**
.42**
Liu, Rijmen, MacCann, & Roberts (2009)
31
How are we doing it?
32
Our GPS:
Multi-Trait Multi-Method (MTMM) Design
Self Ratings
Not directly
observable
True
True
Resiliency
Resilience
Alternative
Method
“Triangulation”
Measurement
Method
Teacher
Ratings
The underlying view of measurement in the MTMM analysis is that to measure a theoretical construct,
different measures, each with its own bias, are selected. Bias that is due to method effects is reduced through a
triangulation process.
Baron & Kashy, 1992, p. 170
34
Item Types
35
Self-Report
36
Situational Judgment Test
37
Biographical Data
38
Creativity Performance Test
39
Teacher Ratings
40
Resilience:
Representative Items
Situational Judgment Test
You are feeling stressed about the amount of
homework that you have been given by your
teacher(s). Below are some ways that you
might think, feel, or act in this situation, right
at the time that you feel stressed-out. Rate how
often you do each activity when you feel
stressed. How do you think, feel, or act when
you are stressed from having too much
homework to do?
1.  I take control and say to myself: I can do
this!
Self-Report
I overcome challenges and set
backs.
Resilience
Teacher Ratings
This student is easily
discouraged.
41
What did we find?
42
Participation
• 
Fall 2011
•  18 INDEX schools
•  2,081 students
• 
Spring 2012 (optional assessment)
•  1,677 students
• 
Fall 2012
•  > 2,600 students
•  Data being analyzed
43
Average'Construct'Scores'By'Gender
103
Average'Construct'Score
102
101
100
99
98
Male
97
Female
96
95
94
2011 2012
2011 2012
Creativity
Ethics
2011 2012
2011 2012
Intrinsic9Motivation
Resilience
Construct
2011 2012
2011 2012
Teamwork
Time9Management
44
The MSA Demonstrates
Evidence of Reliability
45
Reliability
Construct!
Time Management!
Resilience!
Intrinsic Motivation!
Creativity!
Teamwork!
Ethics!
2012
2011
Reliability! Reliability!
0.86!
0.86!
0.85!
0.84!
0.90!
0.89!
0.88!
0.88!
0.85!
0.77!
0.85!
0.90!
SAT
Reliability
Math
.91
Critical
Reading
.91
46
The MSA Demonstrates
Evidence of Validity
47
The MSA Predicts
•  Grades
•  Teacher Ratings of Student Quality
•  “How would you rate this student compared to other students in
your school?”
48
r = .56
2012 Student Quality Rating
Creativity and Student Quality Rating
4
3.5
3
2.5
2
1.5
1
0.5
0
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Creativity
49
How “Big” Are Our
Correlations?
Examples from other research (Meyer et al, 2001)
Variable 1
Variable 2
Correlation
Sugar Consumption
Child’s Behavior
.00
Aspirin
Heart Attack Death
.02
Parental Divorce
Child Well-Being
.09
Lead Exposure
Child IQ
.12
Ibuprofen
Pain Reduction
.14
Alcohol
Aggressive
Behavior
.23
Viagra
Sexual Functioning
.38
Gender
Height
.67
50
6th r = .29
7th r = .59
8th r = .58
Spring 2012 GPA
Time Management and GPA by Grade
4
3.8
3.6
3.4
3.2
3
2.8
2.6
2.4
2.2
2
6
7
8
E - Bottom D - 20%-39% C - 40%-59% B - 60%-79% A - Top 20%
20%
2011 Time Management
51
The MSA Predicts
•  Student Well Being
• My#life#is#going#well#
• My#life#is#just#right#
• I#would#like#to#change#many#things#in#my#life#
• I#wish#I#had#a#diﬀerent#kind#of#life#
• I#have#a#good#life#
• I#have#what#I#want#in#life#
• My#life#is#be;er#than#most#kids’#
52
r = .33
Resilience!and!Well!Being!
5!
2012!Well!Being!
4.9!
4.8!
4.7!
4.6!
4.5!
4.4!
4.3!
4.2!
4.1!
A!-!Top!20%!
B!-!60%-79%!
C!-!40%-59%! D!-!20%-39%! E!-!Bo7om!20%!
2011!Resilience!
53
r = .31
2012 Well Being
Ethics and Well Being
5
4.9
4.8
4.7
4.6
4.5
4.4
4.3
4.2
4.1
A - Top 20%
B - 60%-79%
C - 40%-59%
2011 Ethics
D - 20%-39% E - Bottom 20%
54
There is initial evidence that the
MSA predicts just as well as
standardized tests
55
Correlation with Student Quality
MSA and ERB Scores: Correlations with Student
Quality Ratings
0.6
0.5
0.4
0.50
0.46
0.45
0.44
0.38
0.31
0.3
0.16
0.2
0.15
0.1
0
56
MSA and ERB Scores: Correlations with GPA
Correlation with GPA
0.7
0.6
0.5
0.59
0.56
0.52
0.49
0.41
0.4
0.28
0.3
0.2
0.27
0.21
0.1
0
57
Correlation with Well Being
MSA and ERB Scores: Correlations with Well Being
0.3
0.25
0.2
0.28
0.23
0.22
0.22
0.19
0.14
0.15
0.1
0.04
0.05
0.01
0
58
Current and Next Steps
•  Continual refinement of MSA assessment
(adding interests)
•  Assessing underserved populations with MSA
•  Gathering interest from additional schools
59
Implications for Schools
60
Implications for Schools
•  Wide-ranging and deep:
•  Why?
•  What?
•  How?
61
Why? Mission and Vision
•  Reexamine school mission and how
measured
•  Strength of mind and strength of character
•  Develop a Vision for Character
62
How?
Programsintentional and
well-designed
High-quality
training and
support
Evidencebased
Evaluation
63
What? Programs
How can we teach these skills better and how will we know we
are succeeding?
•  Deliberate, explicit and implicit instruction
•  Proven interventions
•  Performance Character Audit
•  Articulated, spiraled, integrated curriculum
•  Grade-level goals
•  Updated and analyzed curriculum maps
64
Programs
•  Social Emotional Learning, advisories, personal
awareness, and co-curricula
•  Shared lexicon, common language-Glossary of
Terms for Character
•  Best Practices-follow through, evaluation,
refinement
65
What? Training and Support
•  Ongoing education–school leaders, faculty,
parents
•  Shared vision, vocabulary, and resources
•  Participation in regional and national educator
character summits
•  Ongoing partnerships and dialogue
66
What? Evidence-based
Evaluation
•  Updated Progress Reports
•  Parent and student conferences
•  Faculty and parent education summits
•  Greater emphasis: Self-reflection and meta-cognition
•  Surveys
67
Evidence-based Evaluation
•  Integrated assessments
•  Participation in new assessments (MSA)
•  Benchmarking
•  Data for national accreditation and new
criterion
68
Measuring what we value:
Criterion 13:
The Standards require a school to provide evidence
of a thoughtful process, respectful of its mission,
for the collection and use in school decision
making of data (both external and internal) about
student learning.
69
Improvement Phases
Envision
Implement
Evaluate and Refine
Commitment tied to
mission
Needs and resources
assessment
Professional
Development &
Partnerships
Create clear vision
Action Plans,
integration, and
enhancements
Communication and
engagement
Build engagement
Evidence-based
strategies and programs
•  What is working?
•  How do we know?
•  How can we improve
teaching and learning?
70
Community Responses
•  Excitement!
•  Shared understanding
•  Partnership
•  Shared commitment to studying results and asking the next
questions…
71
Your School, Your Mission
Look to your mission, look to your alumni, look
to your purpose and your school’s view of what
our society needs the most: creative and
innovative thinkers, people of fine judgment,
leaders, collaborators, and…then select or design
the instruments which will help you make these
predictions.
Martin, Jonathan. Lessons from Law School Admissions: More from USC Attributes that Matter, 21K12
Blog, January 23, 2013.
72
Thank you!
Questions?
Comments?
FOR MORE INFORMATION, PLEASE CONTACT:
INDEX
INDEPENDENT SCHOOL DATA EXCHANGE
LISAPULLMAN@INDEXGROUPS.ORG
WWW.INDEXGROUPS.ORG
MSA Steering Committee:
Chuck Baldecchi
Head of School
The Lexington School (KY)
CBaldecchi@thelexingtonschool.org
Tim Bazemore
Head of School
New Canaan Country School (CT)
TBazemore@countryschool.net
Dave Michelman
Head of School
Duke School (NC)
Dave.Michelman@dukeschool.org
Ned Murray
Head of School
Episcopal Day School (GA)
nmurray@edsaugusta.com
Jeremy Burrus
Research Scientist
ETS
jburrus@ets.org
Jennifer Phillips
Director of Educational Advancement
Far Hills Country Day School (NJ)
jphillips@fhcds.org
Lisa Pullman
Executive Director, INDEX
lisapullman@indexgroups.org
Rich Roberts
Managing Principal Research Scientist
ETS
rroberts@ets.org
73

Mission Skills Assessment - 2017 NAIS Annual Conference

Transcription

Similar documents

Maumelle Sports Association

THE SYMPATH PROJECT KEY FACTS

CYRM Flyer - The Resilience Research Centre

2014 Fire Service New Product Guide - Asset Library

FUEGO Fire-fighting Helmets

Just for Parents 2015 - University Counseling Center

SAS Creativity

qdoba mexican grill - LMS Commercial Real Estate

Martin Gibson

T Making Effective Master Service agreements Natural Gas Producers

SAMPLE of MSA program requirements with At Central Michigan University

SAMPLE of MSA program requirements with CGSC Transfer In-Residence Credits

July 2014

How To Establish And Operate A Successful

Why We Should Use Noncognitive Variables With Graduate and Professional Students