Lecture 8

4/16/2015
Language Testing and Assessment
Lecture 8
Brózik-Piniel, Katalin
Construct:
Interact successfully in a foreign language
Comprehension and production
 CEFR-
Overall Oral Production B2
Can give clear‚ detailed descriptions and
presentations on a wide range of subjects related to
his/her field of interest‚ expanding and supporting
ideas with subsidiary points and relevant examples.
1
4/16/2015
 What
affects candidates’ performances
which should be controlled for as much as
possible in high-stakes tests:
(construct irrelevant facets)
 Individual characteristics
 Task characteristics
 Other participants
In order to ensure the reliability and validity of speaking
examinations, as a means of standardizing the exam:
 design tasks carefully (select relevant+variety of activities)
 assessor and interlocutor (2 examiners)
 providing an Interlocutor Frame (all the instructions,
suggested question prompts needed ) and guidelines for
behavior
 train interlocutors and assessors
 employing analytic assessment scales that have been piloted
and revised;
 monitoring interlocutors and assessors.
 Benchmark performances
(Csépes & Együd, 2004, p. 25)
2
4/16/2015

Individual mode
 Questions of status
 Not life-like
 Interaction controlled by interviewer
 the examiner’s L1 background or acquaintanceship
with the candidate can influence candidate
performance
 the interlocutor’s acquaintanceship with candidates
may also influence the communication strategies
(help request) used by the participants
Paired mode
Testing candidates in pairs and small groups is
motivated by:
-dissatisfaction with the oral interview as
the only test format
-search for new tasks that can elicit
different patterns of interaction
-desire to generate positive washback on
teaching by encouraging more interaction
-to mirror good language teaching practice
-time saving and cost effective
(Csépes & Együd, 2004, p. 33)
3
4/16/2015






Characteristics of paired exams
- Mismatch between proficiency/personality
+ WTC
+ high level involvement
+ large language sample
+ control, ease
Ask ONLY from a set of pre-specified options
indicated on the interlocutor’s sheet.
The interlocutor is only allowed to make comments
that are in compliance with the guidelines for
conducting the exam.
The interlocutor has to select and ask follow-up
questions even if the candidate manages to talk about
the pictures at length.
It is not necessary for the interlocutor to use the
question prompts in the order listed.
Thank you and proceed to the next part of the
examination or Thank you. That will do.
(Csépes & Együd, 2004, p. 52)
4
4/16/2015
• interview
• picture-based individual long turn
• discussion
• role-play
 Personal information
 Opinion
Possible problems:
 Not life-like
 Variation in elicitation can lead to different
judgments of language ability (can be
counterbalanced by standardized interlocutor
behavior)
5
4/16/2015
SAMPLE 1
SAMPLE2
Where are you from? Tell me about
your town/village.
Where do you live?
How do you spend your free time?
Do you have friends?
What do you like most about
television?
Do you like watching TV?
Tell me about your family.
Do you have any brothers or sisters?
How do you think American
culture influences our life?
Why do you think many countries see
the
American influence as a threat to
their national identity?
How have the technological
inventions of the last century
changed our life?
Which scientific inventions of the
20th century do you think have
changed people’s lives the most?
p. 41
In interviews the interlocutor should follow specific guidelines:
• Use global questions for elicitation.
• Use wh-questions instead of yes/no questions whenever possible.
• Never ask more than one question at a time.
• Do not talk more than necessary: refrain from making unnecessary
comments.
• Do not interrupt or finish what the candidate wants to say.
• Do not ask questions that require background knowledge.
• Avoid ambiguous and embarrassing questions.
• Use genuine questions and avoid display questions.
• Maintain eye contact with the candidate when talking to him/her.
-Task in accordance with proficiency level
-Provide candidate with maximum scope of response
(Csépes & Együd, 2004, p. 40 and p. 42)
6
4/16/2015
Using pictures
+easy to find
+economic and effective
+freedom to show mastery
+deficiencies in reading comprehension cannot prevent
candidates from doing well
(Csépes & Együd, 2004, p. 51)
Inadequate use of pictures may stem from:
- not challenging enough, not enough stimuli
- is culture-dependent, needs special background knowledge
- topic distressing, offensive, violent or taboo
- surreal, abstract and symbolic pictures
- bizarre, unrealistic situations
- too many pictures to compare and contrast
- only one picture, too simple, leads to description
- pictures too similar for comparison and contrast
- static pictures
- over-crowded pictures (Csépes & Együd, 2004, p. 52)
7
4/16/2015
Paired and individual mode
Two-way task (interaction)
Negotiation of own opinion – come to an agreement
(planning, listing, ranking, solving a problem etc.) –
specific focus is needed
 Avoid eliciting two monologues



GUIDELINES FOR WRITING PAIRED TASKS
Item writers should bear in mind that
• the reading input should be minimal
• the task instructions should be read out by the Interlocutor
• instructions should be clear
• contributions to the interaction should be balanced
• comparable tasks
• the interaction should be task-based
• the tasks have to be guided but not fully controlled
• controversial topics seem to be more likely to generate a
discussion than neutral topics;
• distressing, offensive, violent or taboo topics should be avoided;
• the topic should be something on which it is reasonable to expect
candidates to have an opinion
• define the context of the conversation very carefully
• Candidates should be given a chance to voice their own opinion
rather than argue for something that they cannot identify with.
(Csépes & Együd, 2004, p. 90)
8
4/16/2015














Involve simulating roles we take on in our everyday lives
Give candidates familiar roles. Give them a standardized, clear and
concise role description.
Instruct them about their expected contributions to the role-play
situation.
Assessment is based on the process and not the result.
Select prompts/input text and design the task so that it generates the
appropriate amount of varied language at the required level.
Use prompts that are clear, and appropriate for the target age.
The language level of verbal prompts should be below the tested
level.
Do not use distressing, offensive, violent or taboo topics. Treat
sensitive topics with care.
Do not use surreal, abstract, puzzling or symbolic pictures at lower
levels.
Do not require students to use their imagination or creativity as
these are not to be tested. (Csépes & Együd, 2004, p.102-131)
Narrative tasks
Instruction tasks
Description tasks
Prepared monologue
9
4/16/2015
Aim for consistency (intra-rater reliability: self-check, go
back to the first performance)

Rating checklists can be adapted and used in learningrelated settings (Luoma, 2004)

 Rating checklists with useful and comprehensive descriptors should
provide useful informaation for the learner
 Peer assessment: learners become more aware of the learning goals,
use criteria that learners comprehend(should be task-related, good idea
to develop the criteria together with the learners, listing features of task
achievement)

Feedback: concrete, descriptive, and relates the
performance to the goals (using feedback reports and
clearly formulated speaking goals) (Luoma, 2004)

Share the learning objective(s) for the talk activity clearly. Encourage pupils to
reflect on how they are learning through using talk.
Discuss and agree criteria for success in advance of an oral activity or task. If
possible, listen to/watch examples first, or model a successful outcome yourself.
Different tasks will require different types of speaking and listening, and hence
specific success criteria.
Set up peer response partners to provide feedback and to help each other improve.
Encourage pupils to use a talk log or journal and to note down successful
contributions they have made, or to assess the contributions of others, and to reflect
on what they have learned. Provide them with vocabulary/terminology to aid
reflection and analysis.
Use pupil observers to focus on specific aspects of talk and provide feedback to
other groups. Use video on occasions to enable pupils to observe and comment on
their own performance.
Plan time at the end of a specific oral activity to discuss and debrief with pupils. Use
quick pair discussion first or “thinking” time before inviting comment. Focus on what
went well before looking at areas for improvement.





10
4/16/2015
Key skills for assessment










Evidence for assessment
Target for improvement
(examples are given for guidance)
Communicating clearly and
imaginatively
Structuring and sustaining talk
Adapting talk to different situations
and for different audiences
Adopting different roles and using a
range of techniques
Present ideas clearly and confidently
Listening carefully to a presentation,
reading, programme, discussion etc.
Understanding key points, ideas and
views
Listen carefully to identify key points and
understand ideas
Participating in discussions
Listening carefully to others’
contributions
Making helpful and sustained
contributions to discussions
Contribute effectively to group discussions
Using standard English appropriately
Use standard English in formal situations
Speaking and Listening
targets
Examples of follow-up actions

Listen carefully to identify
key points and understand
ideas


Contribute effectively to
group discussions





Present ideas clearly and
confidently




Vary expression and
vocabulary to suit the
audience and purpose
Vary expression and vocabulary to suit the
audience and purpose




Prepare helpful frameworks for the task or topic before you listen and use them to record key points as
you listen
Listen for and list key “naming” words for the topic and their synonyms, or topic sentences that mark
key stages in the argument. Practise reconstructing the text using these lists.
Provide a timed summary for someone else of the topic you’ve just listened to.
Revise “golden rules” for effective group work, or devise your own list.
Listen to another group’s discussion and make notes on what is helpful and not so helpful in getting the
task done or reaching agreement. Decide what you can change about your own contributions in the
light of this list. Ask a friend to monitor your progress.
Give yourself a set number of counters to “spend” when you make a helpful contribution to discussion.
You must use them all but cannot contribute once they are all used.
Practise using helpful prompt phrases for taking turns in discussion e.g.
I agree with …because…; I can see what you’re saying but….; a new point I want to make is…; to sum
up, we seem to be saying that….
Practise making notes on a topic or task in different formats, and using them to structure your talk.
Choose a subject or process that you know a lot about and write very brief notes about it, e.g. 6 key
words in sequence. “Teach” someone else your topic, using only these notes. Ask them for feedback
about clarity and confidence.
Practise giving 15 second/30 second/60 second talks on a topic, building up the detail each time.
Prepare notes on a topic with a friend, then get them to ask you questions about it. Use the notes to
prompt your answers.
Highlight key words in the task that give information about who the text is for, and what its purpose is.
Plan key points under these headings and list examples of suitable words and phrases to use.
Ask someone to take the role of the intended audience, listen to your talk and give you suggestions
about what needs changing.
Record yourself giving your talk and review its effectiveness.
11
4/16/2015
Construct: receptive skill, it varies from test to test
(what do we wan to measure)- ways and reasons
 It involves following arguments, recognizing attitudes,
obtaining the gist (Hughes, 2003)


variety of purposes

a range of texts

different kinds of information









No replay
Chunks
Redundancy
Reduced forms
Performance variables
Colloquial expressions
Accents
Speed
Intonation, rhythm, stress
12
4/16/2015

selective attention (level of attention)

listening for understanding / filtering / hearing


skills / subskills
ability to understand gist, main points, details, infer
speakers’ attitudes, opinions or feelings

Understanding:
local meaning
full linguistic meaning
inferred meaning





Bottom-up, top-down
Specific information vs. Gist (global)
Intensive listening vs. Extensive listening
Stress-intentions of the speaker
Sound discrimination
13
4/16/2015
Type: monologues, dialogues , group conversations
 Form: description, narration, etc.
 Length
 Speed
 Dialect
 Accent
 Authentic (slightly edited) texts intended for listening

Domain: personal, social, academic, vocational
 Genre: group discussion, debate, lecture,
presentation, talks, report


Topic: abstract/concrete, complex/simple
(propositionally/linguistically), familiar/unfamiliar,
technical/general

Medium: live, broadcast, recorded
14
4/16/2015








Short answer questions
Multiple-choice items
Note taking
Matching and sequencing
Information transfer (e.g. charts)
Gap filling
Partial dictation
Transcription (numbers, words spelled out)






Based on texts heard (not written) !!!
Authentic texts
Can be written in L1
Trialing
Items set sufficiently far apart
Key words-warning signal
15
4/16/2015
 All texts must be authentic.
 Texts must not be taken from
printed material in other
examinations or English language teaching materials.
 The topic of the texts must be accessible to the
students’ age group.
 Texts must not be offensive, distressing or violent.
 Texts must be of a suitable length.
 Texts must be of a suitable level of difficulty.
 A text must have a title unless it is used as part of the
item.
(Fehérváryné & Pizorn, 2003)







Have a clear idea what skill and sub-skill(s) you intend to test
and select the task and texts/prompts accordingly.
The task type must be one which is familiar to the students who
are tested.
Tasks must be accessible to the students’ age group.
Tasks must not be offensive, distressing or violent.
The time allocated for each task must be sufficient for
somebody who has the ability being tested to complete the task
comfortably within the time limit.
Students must be able to see easily how the task relates to the
text.
No more than one testing technique should be used in each
task. Do not switch from multiple-choice to single-word
answers in a single task, for example.
(Fehérváryné & Pizorn, 2003)
16
4/16/2015

There should be a minimum of five items in a task.
Items must follow the text sequence.
Items must be spread evenly through the text.
It must not be possible to answer any item without reference to the text.
This must be checked carefully.
In non-sequencing tasks, items must not be interdependent – students
should not need one answer in order to find another.
Items must not overlap.
Two items must not have similar answers.
Items must have a complete answer key on a separate sheet. All possible
good answers and predictable wrong answers should be provided.
The items in each task must be numbered sequentially.
One item in each task must have an answer provided as an example. The
example item, written in italics, should be marked 0 (zero) and should
precede the other items.
The example answer must be provided in the form in which candidates are
expected to write their answers and, where appropriate, it must be handwritten(italics).
“Find the wrong answer” type items are not acceptable in multiple choice
tasks.
Each item must score one point.
Clear and sufficient rubrics (Fehérváryné & Pizorn, 2003)



Good quality
If live speech – training is needed to be reliable
Familiarization with the items













17
4/16/2015
Spelling and grammar mistakes are not taken into
account

Listening Log
Task: Listening to Recorded Passages
Name:
__________________________
Here is your task:
Keep this listening log in your binder/folder. Fill it out each time that you finish
listening to a text.
Date
Type of Text
Topic
Main Idea
Comment
The passage that I found the easiest to understand was ___________ because
_________.
The hardest passage to understand was _________ because ________.
Source: www.caslt.org
18