September 2012 Project title: What is Speaking Proficiency? (WiSP)

September 2012
Project title: What is Speaking Proficiency? (WiSP)
Project funded by the Netherlands Organisation for Scientific Research by a grant
awarded to Hulstijn and Schoonen (NWO grant 254-70-030). Official project name:
Unraveling second language proficiency.
Funding period: 2004-2010.
Research team:
Nivja de Jong
Margarita Steinel,
Arjen Florijn,
Rob Schoonen
Jan Hulstijn
Project leaders: Jan Hulstijn and Rob Schoonen
Institution: Amsterdam Center for Language and Communication (ACLC), Faculty of
Humanities, University of Amsterdam
Contact:
Jan H. Hulstijn, Professor of second language acquisition, University of Amsterdam
Email: j.h.hulstijn@uva.nl
Summary
The goal of this project (2004-2010) is to investigate the componential structure of
second-language speaking proficiency. In three studies, we gauge the relative weights
of different components that determine the speaking proficiency of adult speakers of
Dutch as a second language at intermediate and advanced proficiency levels.
In study 1, 200 learners of Dutch as a second language and a control group of 50
native speakers performed eight speaking tasks, differing in content complexity
(simple vs complex), discourse type (descriptive vs argumentative) and formality
(informal vs formal). With structural equation modeling (SEM), we modeled
communicative success in these speaking tasks on the basis of the following
components expected to underlie speaking proficiency: vocabulary and grammar
knowledge, pronunciation quality, speed of lexical retrieval, articulation speed,
pronunciation speed, sentence building speed. Personality (extraversion), measured by
means of a questionnaire, served as a potential mediating variable.
Studies 2 and 3 examine the role of L1 transfer (Turkish or English) in Dutch L2 with
respect to grammar knowledge and knowledge of verb subcategorization frames in
relation to level of L2 speaking proficiency (study 2) and with respect to fluency in
terms of pausing behavior and rate of speech (study 3). Approximately 25 Turkish and
25 English learners of Dutch participated in these studies. Data collection has been
completed (study 2) or almost completed (study 3). Additionally, we have developed
a tool to automatically measure aspects of fluency, allowing us to investigate how
performance in terms of fluency is related to linguistic knowledge and skills using the
data from study 1.
1
The three studies, while unraveling the construct of L2 speaking proficiency, aim to
contribute to theories of speaking and to theories of second language acquisition, in
combination with language assessment.
PROJECT PUBLICATIONS
1.
De Jong N.H. , Steinel, M.P, Florijn, A.F., Schoonen, R. & Hulstijn J. H. (2012).
Facets of speaking proficiency. Studies in Second Language Acquisition, 34, 5-34.
Abstract
This study examined the componential structure of second language (L2) speaking
proficiency. Participants (181 nonnative and 54 native speakers) performed eight
speaking tasks and six tasks tapping nine linguistic skills. Performance in the speaking
tasks was rated by a panel of judges on functional adequacy, forming the dependent
variable in subsequent analyses (structural equation modeling). Separately, the
following independent variables were assessed: linguistic knowledge in two tests
(vocabulary and grammar), linguistic processing skills (four reaction time measures
obtained in three tasks: picture naming, delayed picture naming and sentence building)
and pronunciation skills (speech sounds, word stress, and intonation). All linguistic
skills, with the exception of two articulation measures in the delayed picture naming
task, were significantly and substantially related to functional adequacy of speaking,
explaining 76% of the variance. This provides substantial evidence for a componential
view of L2 speaking proficiency, consisting of language-knowledge components and
language-processing components. The componential structure of speaking proficiency
was almost identical for the 40% subjects at the lower and the 40% subjects at the
higher end of the functional adequacy distribution (n = 73 each), showing no support
for Higgs and Clifford’s (1982) relative contribution model, predicting that, while L2
learners become more proficient over time, the relative weight of component skills
may change.
2.
De Jong, N.H., Steinel, M.P., Florijn, A., Schoonen, R., & Hulstijn, J.H. (2012).
Linguistic skills and speaking fluency in a second language. Applied Psycholinguistics.
FirstView Article, August 2012, pp 1 24. DOI: 10.1017/S0142716412000069,
Published online: 14 March 2012.
Abstract
This study investigated how individual differences in linguistic knowledge
andprocessing skills relate to individual differences in speaking fluency. Speakers of
Dutch as a second language (N = 179) performed eight speaking tasks, from which
several measures of fluency were derived such as measures for pausing, repairing, and
speed (mean syllable duration). Additionally, participants performed separate tasks,
designed to gauge individuals' second language linguistic knowledge and linguistic
processing speed. The results showed that the linguistic skills were most strongly
related to average syllable duration, of which 50% of individual variance was
explained; average pausing duration, on the other hand, was only weakly related to
linguistic knowledge and processing skills.
2
3.
Hulstijn, J.H., Schoonen, R., de Jong, N.H., Steinel, M.P., & Florijn, A.F. (2012).
Linguistic competences of learners of Dutch as a second language at the B1 and
B2 levels of speaking proficiency of the Common European Framework of
Reference for Languages (CEFR). Language Testing, 29, 202-220.
Abstract
This study examines the associations between the speaking proficiency of 181 adult
learners of Dutch as a second language and their linguistic competences. Performance
in eight speaking tasks was rated on a scale of communicative adequacy. After
extrapolation of these ratings to the Overall Oral Production scale of the Common
European Framework of Reference for Languages (CEFR) (Council of Europe, 2001),
80 and 30 participants (on average per speaking task) were found to be, respectively,
at the B1 and B2 levels of this scale. The following linguistic competences were
tapped with non-communicative tasks: productive vocabulary knowledge, productive
knowledge of grammar, speed of lexical retrieval, speed of articulation, speed of
sentence building, and pronunciation skills. Discriminant analyses showed that all
linguistic competences, except speed of articulation, discriminated participants at the
two levels of oral production. Subsequent comparisons showed that the distance
between B1ers and B2ers was smaller in knowledge of high-frequency words than in
knowledge of medium- and low-frequency words. Extrapolation from scores on the
vocabulary test yielded estimations of productive vocabularies of, on average, 4,000
and 7,000 words for B1ers and B2ers, respectively. The grammar test assessed
grammatical knowledge in ten domains. B2ers were found to outperform B1ers on all
parts of the test. Thus, the differences in lexical and grammatical knowledge of B1ers
and B2ers appear to be a matter of degree, rather than a matter of category or domain.
The paper ends with a research agenda for a linguistic underpinning of the CEFR.
4.
De Jong, N.H., Steinel, M.P., Florijn, A., Schoonen, R., & Hulstijn, J.H. (2012). The
effect of task complexity on native and nonnative speakers’ functional adequacy,
aspects of fluency, and lexical diversity. In A. Housen, F. Kuiken, & I. Vedder (Eds.),
Dimensions of L2 performance and proficiency. Investigating complexity, accuracy
and fluency in SLA (pp. 121-142). Amsterdam, Netherlands: Benjamins.
Abstract
This study investigated how task complexity affected native and
nonnative speakers’ speaking performance in terms of a measure of
communicative success (functional adequacy), three types of fluency
(breakdown fluency, speed fluency, and repair fluency), and lexical
diversity. Participants (208 nonnative and 59 native speakers of
Dutch) carried out four simple and four complex speaking tasks.
Task complexity was found to affect the three types of fluency in
different ways, and differently for native and nonnative speakers.
With respect to lexical complexity, both native and nonnative
speakers produced a wider range of words in complex tasks
compared to simple tasks. Results for functional adequacy revealed
3
that nonnative speakers scored higher on simple tasks, whereas
native speakers scored higher on complex tasks. We recommend that,
in future research examining effects of task types on task
performance, the notion of functional adequacy be included.
5.
Hulstijn, J.H., de Jong, N.H., Steinel, M.P., Florijn, A.F., & Schoonen, R. (2012). Hoe
groot is het verschil tussen B1 en B2? Verschillen in kennis van woordenschat en
grammatica tussen NT2-leerders op B1- B2-niveau van spreekvaardigheid.
International Neerlandistiek, 50, 201-217.
Abstract
This study examined lexical and grammatical knowledge of adult learners of Dutch as
a second language (L2) at the B1 and B2 speaking-proficiency levels of the Common
European Framework of Reference for Languages. In a sample of 208 Dutch L2
learners, 80 and 30 participants were found to be proficient in speaking at the B1 and
B2 levels, respectively, as assessed in eight computer-administered speaking tasks.
Participants also performed paper-and-pencil tests of knowledge of vocabulary and
grammar. Average vocabulary sizes were obtained of 4000 and 7000 words (with
standard deviations of 1623 and 1456) in the B1 and B2 groups, respectively. Overall
performance on the grammar test, which assessed knowledge in ten grammatical
domains, was significantly higher in the B2 group than in the B1 group, with average
correct scores of 86% and 71%, respectively. For each of the ten domains of grammar,
examples are given of features that B1ers and B2ers did, or did not yet, control.
6.
De Jong, N.H., & Wempe, T. (2009). Praat script to detect syllable nuclei
and measure speech rate automatically. Behavior Research Methods, 41, 385-390.
Abstract
In this article, we describe a method for automatically detecting syllable nuclei in
order to measure speech rate without the need for a transcription. A script written in
the software program Praat (Boersma & Weenink, 2007) detects syllables in running
speech. Peaks in intensity (dB) that are preceded and followed by dips in intensity are
considered to be potential syllable nuclei. The script subsequently discards peaks that
are not voiced. Testing the resulting syllable counts of this script on two corpora of
spoken Dutch, we obtained high correlations between speech rate calculated from
human syllable counts and speech rate calculated from automatically determined
syllable counts. We conclude that a syllable count measured in this automatic fashion
suffices to reliably assess and compare speech rates between participants and tasks.
7.
De Jong, N.H. , Steinel, M.P, Florijn, A., Schoonen, R. & Hulstijn J. H. (2007). The
effect of task complexity on fluency and functional adequacy of speaking
performance. In Van Daele, S., Housen, A., Pierrard M., Kuiken F. & Vedder, I.
(Eds.), Complexity, Accuracy and Fluency in second language Use, Learning and
4
Teaching (pp. 53-63). Brussels: Koninklijke Vlaamse Academie van België voor
Wetenschappen en Kunsten.
Abstract
In a large-scale study, we investigate how task complexity affects native and nonnative speakers’ speaking performance in terms of fluency and communicative
adequacy. Participants carried out four simple and four complex speaking tasks.
Fluency was measured automatically using scripts written in PRAAT (Boersma &
Weenink, 2007). For non-native speakers, we found that complex tasks lead to less
fluent speech, in line with Robinson’s Cognition Hypothesis (Robinson, 2001).
However, native speakers’ fluency increased when tasks were complex. Results for
communicative adequacy were similar to the results for fluency, in that non-native
speakers scored higher on simple tasks, whereas native speakers scored higher on
complex tasks. We discuss the results in the light of Robinson’s Cognition Hypothesis
and Skehan and Foster's Limited Attentional Capacity Model (Skehan & Foster, 2001).
8.
De Jong, N.H., Groenhout, R., Schoonen, R. & Hulstijn J.H.. L2 fluency: speaking
style or proficiency? Correcting measures of L2 fluency for L1 behavior. Revision
under review. Applied Psycholinguistics. The work in which this paper reports was
supported not only by NWO (grant 254-70-030) but also by a grant from Pearson
Language Testing awarded to N.H. de Jong.
Abstract
In second language (L2) research and testing, measures of oral fluency are used as
diagnostics for proficiency. However, fluency is also determined by personality or
speaking style, raising the question to what extent L2 fluency measures are valid
indicators of L2 proficiency. In this study, we obtained a measure of L2 (Dutch)
proficiency (vocabulary knowledge), L2 fluency measures, and fluency measures that
were corrected for L1 behavior from the same group of Turkish and English native
speakers (N = 51). For most measures of fluency, except for silent pause duration,
both the corrected and the uncorrected measures significantly predicted L2
proficiency. For syllable duration, the corrected measure was a stronger predictor of
L2 proficiency than the uncorrected measure. We conclude that for L2 research
purposes, as well as for some types of L2 testing, it is useful to obtain corrected
measures of syllable duration to measure L2-specific fluency.
ADDITIONAL DOCUMENTS
Wijers, M. (2010). Marking or masking disfluency? An investigation of the role of
lexical fillers in distinguishing native speakers from second language learners of
Dutch. University of Amsterdam. Student term paper supervised by Jan Hulstijn and
Nivja de Jong.
Abstract
This paper explores the role of lexical fillers in the perception and measurement of
fluency in the speech of native speakers (NSs) and non-native speakers (NNSs) of Dutch.
Although lexical fillers are used to express hesitation gaps and the need for processing
5
time, their role has often been neglected in previous research on fluency, which has
mainly focused on non-linguistic fluency aspects (pauses, speech rate etcetera.). Three
speaker groups were investigated for the purpose of this study: 21 native speakers of
Dutch, 19 highly proficient L2 speakers and 21 intermediate L2 speakers of Dutch. All
learners had a Germanic language as their mother tongue. The data were collected in four
separate speaking tasks within the research project Unravelling Second Language
Acquisition. Participants in this project were rated by naive raters. It showed that there
were great individual differences but that the amount of lexical fillers, unlike the amount
of pseudo-lexical fillers (uh, uhm) and non-lexical fillers, did not influence the rated
fluency. On the basis of this result it was concluded that the amount of lexical fillers
cannot be used as a fluency measure. However, it also indicates that lexical fillers are a
very effective way to compensate for non-lexical and pseudo-lexical fillers, which do
influence perception of fluency. In the current paper, it is argued that lexical fillers are
often unjustly neglected in second language learning because the use of lexical fillers can
be a useful fluency strategy in second language learning.
2.
Dieteren, C. (2011). The influence of a first language on the acquisition of Dutch as a
second language. University of Amsterdam. Student term paper supervised by Jan
Hulstijn and Margarita Steinel.
6