Supervisors’ oral feedback on HDR student writing A criterion-based approach

Supervisors’ oral feedback on HDR
student writing
A criterion-based approach
Dr Bronwen Dyson
bronwen.dyson@sydney.edu.au
Supervisors’ oral feedback
›  For Higher Degree Research students, feedback constitutes:
“… “a major, if not the major, form of instruction” (Bitchener, Basturkmen &
East (2010, p.82).
o Written feedback provided on student drafts.
o Oral feedback provided in meetings, either before or after drafting.
›  Limited research on supervisory feedback, particularly oral feedback.
Largest study (Paré, 2010, p. 107) concludes that advisors’ spoken
comments are “often ambiguous, enigmatic and coded – that is saturated
with meaning, but difficult to understand”.
›  So, is supervisors’ oral feedback ambiguous & difficult to understand?
›  Argument: Writing criteria can reveal whether or not there is a lack of
clarity - or any other problem - in supervisors’ oral feedback and so should
be more widely used.
2
Studies of supervisor oral feedback
›  Paré (2010, 2011): questionnaire & recorded meetings of large number of
Canadian Humanities and Sciences supervisors.
›  Argues that advisors “struggle to articulate implicit knowledge”, e.g.
›  I think the information is there but I have two main points about it. One is
that I think it should be a bit more focused. ... The other comment is to, I
don’t know, firm it up, I suppose. ... I’d like more numbers, I suppose. ... So
that my general feeling is that the chapter itself ... should be put within a
slightly bigger box for the committee (2011, 61-62).
›  However:
›  Analysis is broad-brush.
›  Learning is not measured, despite the conclusion that the feedback is
“difficult to understand”.
3
Studies of supervisor oral feedback (2)
›  Bitchener, Basturkmen, East & Meyer (2011): questionnaire & written
feedback of 35 NZ Humanities, Commerce & Sciences/Mathematics
supervisors.
›  Shows supervisors perceive a range of feedback types e.g. oral + written.
›  Meetings have various aims & characteristics e.g. talk/listen vs address
omissions.
›  Problems e.g. some do not hold meetings (2/35).
›  Suggest classification of written feedback using speech acts e.g. “Perhaps
do X” = ‘suggesting’ (cf Kumar & Stracke, 2007).
›  However:
›  No analysis of actual meeting practices.
›  No mention of writing criteria, which could provide a more direct link with
the writing than speech acts.
4
Research questions
›  So, little research on supervisors’ oral feedback on thesis writing, none on
learning but the suggestion of some problems & the potential of writing
criteria not yet explored.
›  Research questions:
(1)  How successful is supervisors’ oral feedback, as measured by a set of
writing criteria?
(2)  In what ways could writing criteria guide this feedback?
5
The study
Case study 1
Doctoral advisor - male, US-trained.
Student - female, Cantonese-speaking, IELTS: 6.5.
Case study 2
Doctoral advisor - female, US-trained.
Student - female, Mandarin-speaking, IELTS: 8.
Data
Fortnightly meetings audio-recorded & transcribed.
8 one-hour sessions (4 per case study).
6
Data analysis: the criteria
MASUS writing criteria to assess strengths & weaknesses(Bonanno & Jones, 2007)
5 criteria
A
Use of source material
B
Structure and development of
the argument
C
Academic writing style
D
Grammatical correctness
E
Qualities of presentation
o 25 sub-criteria e.g.
•  B
o Genre is appropriate to the task
o Clear focused thesis statement
o Choice of Theme & New reflects
structure …
•  E
•  Paragraphing reflects essay
structure …
7
Data analysis (2)
Student learning
•  Measured by ‘uptake’, an optional move in which the learner responds to
feedback on a gap in his/her knowledge (Ellis, Basturkmen & Loewen,
2001, p. 286).
•  Uptake an early sign of learning.
Moves & episodes
•  Feedback & learning analyzed as:
•  Moves: Utterances within a turn or over adjoining turns relating to a single
sub-criterion.
•  Episodes: One or more moves relating to a single Area e.g. A or more than
one Area e.g. A & B.
8
Findings: Feedback moves
Majority of feedback in Areas A & B (95.78%)
A (Use of source material)
B Structure and development of the
argument)
42.46%
(N = 262)
53.32%
(N = 329)
Most relevant data is employed
e.g. I think (author) and (author)
obviously are very good for this, I have
to see them in here (DA 2, 1:222)
Genre is appropriate to the task
e.g. So I think this might become a short
separate chapter on its own after this
long chapter (DA 1, 1:336)
9
Findings: Feedback moves (2)
Minority of feedback in Areas C, D & E
C: Academic writing
style
D: Grammatical
correctness
E: Qualities of
presentation
0.16%
(N = 1)
0.16%
(N = 1)
3.89%
(N = 24)
Appropriate lexis
e.g. Like this uh . her‘herausforderung’ right
which I still don't
understand right (DA 1, 4:
160)
Clause structure
e.g. Yeah this is a run-on
sentence (DA 2, 1: 119)
Other qualities of
presentation
e.g. It's a bit long I think
(DA 1, 4: 48)
10
Findings: Feedback episodes
Majority of feedback episodes combine Areas A & B
A
B
C
D
E
A+B
Other
combinat
ions
13.35%
(N = 75)
16.90%
(N = 95)
0
0
0
61.03%
(N = 343)
49%
(N = 8.72)
DA 2: [B] And then you say, look how a whole style ... has merged in China and
outside China, following the success of (title). Right. ... [A] And then I think it would
be great to just go through some newspapers and do you know what I mean?
Student 2: Yeah. (DA 2, 4: 182-185)
11
Findings: + or - feedback
More feedback on inappropriate than appropriate writing
Strengths
Weaknesses
8.75%
(N = 54)
91.24%
(N = 563)
Critical evaluation of evidence
e.g. That was excellent, … I like how you
say there's a gap obviously because
there's all this stuff on Chinese American
literature and nothing on very recent …
migrants. (DA 2, 1: 129-131)
Focused thesis statement
e.g. But I think that what's missing there
is why are you looking at the specific
points that you talk about? (DA 2, 1: 35)
12
Findings: Effect on learning
Majority of uptake in Areas of main feedback. Less uptake than feedback.
A+B feedback episodes produce half the uptake.
%
A
B
C
D
E
A+B
Total N
Feedback 43
53
0.2
0.2
3.9
61
617
Uptake
60
0
0
3.2
30
154
37
Feedback & uptake on A:
DA 2: So my big question was ... why are you using (author) in your thesis? …
Student 2: I think that the context that (author) uses this concept is different from
my context. (2, 1: 9-10)
13
Discussion: RQ 1
How successful is supervisors’ oral feedback, as measured by a set of writing criteria?
Successful and not-so-successful:
ü Fine-tuned awareness of criteria A & B & their sub-criteria
Important counter-evidence of Paré’s (2010, 2011) conclusion of ambiguity.
Humanities’ language-savvy supervisors.
o  Little attention to C, D & E
Problematic given difficulties at micro-level (e.g. Cooley & Lewkowitz, 1997).
Although, written feedback on micro-level (Bitchener, Basturkmen & East, 2010).
o  Episodes zigzagged across Areas A & B
Problematic given difficulties with Area B (Bitchener & Basturkmen, 2006)
Bitchener et al. (2011): written feedback ‘piles up’ functions.
o  More feedback on weaknesses than strengths
Hyland & Hyland (2001): written feedback ‘sugars the pill’.
o  More feedback than uptake
Not just due to ‘ambiguity’ as in Paré (2010, 2011), but range of factors.
14
Discussion: RQ 2
In what ways could writing criteria guide this feedback?
MASUS could guide supervisors’ feedback by:
›  Developing skills in providing clear feedback on criteria A & B & their subcriteria.
›  Encouraging attention to C, D & E since often errors occur & are repeated
in these Areas.
›  Fostering a focus on specific Areas, especially ones which pose particular
difficulties e.g. B.
›  Creating greater balance between feedback on strengths & weaknesses.
›  Thereby, fostering greater balance between feedback & uptake.
15
Conclusion: a criterion-based approach
›  Study contributes to the limited research on supervisors’ oral feedback.
›  Suggests that this feedback is not problematic in ways previously indicated
but in some new ways.
›  Highlights the value of writing criteria, specifically MASUS, as a framework
for analyzing & strengthening supervisors’ oral feedback.
›  Provides a direct link between the oral feedback & the writing.
›  Limitations: A small sample from Humanities which does not test potential
of criterion-based approach.
›  Future research: Widen the disciplinary lens, trial a criterion-based
approach & consider effect on student writing.
16
›  Thank you!
›  Bronwen.dyson@sydney.edu.au
›  Postgraduate Writing
›  Faculty of Arts & Social Sciences
›  University of Sydney
17
References
›  Bitchener, J., & Basturkmen, H. (2006). Perceptions of the difficulties of postgraduate L2 thesis students writing the discussion section.
Journal of English for Academic Purposes, 5, 4-18.
›  Bitchener, J., Basturkmen, H., & East, M. (2010). The focus of supervisor written feedback to thesis/dissertation students. International
Journal of English Studies, 10(2), 79-97.
›  Bitchener, J., Basturkmen, H., East, M., & Meyer, H. (2011). Best practice in supervisor feedback to thesis students. Wellington, New
Zealand: The National Centre for Tertiary Teaching Excellence.
›  Bonanno, H., & Jones, J. (2007). Measuring the Academic Skills of University Students: The MASUS procedure, a diagnostic
assessment. Sydney: University of Sydney, Learning Centre.
›  Casanave, C. P., & Hubbard, P. (1992). The writing assignments and writing problems of doctoral students: faculty perceptions,
pedagogical issues, and needed research. English for Specific Purposes, 11(1), 33-49.
›  Cooley, L., & Lewkowitz, J. (1997). Developing awareness of the rhetorical and linguistic conventions of writing a thesis in English:
addressing the needs of EFL/ESL postgraduate students. In A. Duszak (Ed.), Culture and styles of academic discourse (pp. 113-129).
Berlin, Germany: Mouton de Gruyter.
›  Ellis, R., Basturkmen, H. & Loewen,S. (2001). Learner uptake in communicative ESL lessons. Language Learning 51(2), 281-318.
›  Hyland, F. & Hyland, K. (2001), Sugaring the pill: Praise and criticism in written feedback. Journal of Second Language Writing 10(3),
185-212.
›  Kumar, V., & Stracke, E. (2007). An analysis of written feedback on a PhD thesis. Teaching in Higher Education, 12(4), 461-470.
›  Paré, A. (2010). Making sense of supervision: Deciphering feedback. In P. Thomson & M. Walker (Eds.), The Routledge doctoral
student's companion: Getting to grips with research in education and the social sciences (pp. 107-115). New York, NY: Routledge.
›  Paré, A. (2011). Speaking and writing: Supervisory feedback and the dissertation. In L. McAlpine & C. Amundsen (Eds.), Doctoral
education: Research-based strategies