o - DORAS

A n Investigation into the Impact o f Controlled English Rules on the
Comprehensibility, Usefulness, and Acceptability of MachineTranslated Technical Documentation for French and German Users
A dissertation submitted to Dublin City University in fulfilment
o f the requirements for the degree o f Doctor o f Philosophy.
Johann Roturier M.A.
School o f Applied Languages and Intercultural Studies
N ovem ber 2006
2 V olum es (Volum e 1)
Supervisors: Prof. Jenny W illiams, D r M inako O'Hagan, & Dr
Sharon O'Brien
ii
Declaration
‘1 hereby certify that this material, which I now submit for assessment on the
programme o f study leading to the award o f Doctor o f Philosophy is entirely my own
work and has not been taken from the work o f others save and to the extent that such
work has been cited and acknowledged within the text o f my work.*
Student Number:
(Candklate)
Date:__________________0 f -
Q1
521
2. I
Acknowledgements
There are many people to whom I am indebted due to their contribution to this
dissertation over the last three years. First of all, I would like to thank my academic
supervisors, Prof. Jenny Williams, Dr. Minako O’Hagan, and Dr. Sharon O'Brien,
not only for their invaluable advice and recommendations, but also for their
encouragement and eternal patience. I am extremely grateful to them for pushing
me to explore new horizons and tackle topics from a different angle. Thanks to
them, the experience has been all the more diverse and enjoyable.
I also wish to thank my industrial supervisor, Dr. Fred Hollowood, for his
availability, advice, and sense of direction. On numerous occasions, he has helped
me address what seemed like unsolvable 'level 9' problems. I am extremely grateful
to him for providing me with resources when I needed them. Without him, his team
in Dublin, and Symantec in general, this research could not have been carried out.
I am also grateful for the support I have received from all Symantec staff who have
helped during these three years. Without Symantec's financial support, without their
authorisation to allow me to access some of their resources and users, this
dissertation would never have come to life. To all Symantec employees who have
involved in one way or another in this research, I say a huge "Thank you!"
I would like to acknowledge the advice received from Dr. Sabine Lehmann from
Acrolinx at the early stages of this research. Thank you very much for stimulating
discussions on test suites and everything else!
I would also like to thank my family and friends for always supporting me despite
the distance, and putting up with my prolonged silence when I was writing up this
dissertation.
Last but not least, to Dympna, without whom I simply would not have completed
this dissertation. Regardless of the time when things were good or bad, you have
always been there to help me and support me, and this will never be forgotten.
V
C h a p te r 6 : A n a ly s in g th e im p a c t o f CL ru le s
o n t h e r e c e p tio n o f M T d o c u m e n ts ................................................................. 1 7 7
6 .1 . O bjectives of the present c h a p te r...........................................................177
6 .2 . Prelim inary evaluation of MT d o c u m e n ts .............................................178
6 .3 . Analysis of th e distribution of resp o n ses...................
184
6 .4 . Analysis of th e responses obtained for the first d o c u m e n t........... 188
6 .5 . Analysis of th e responses obtained for the second d o c u m e n t... 193
6 .6 . Im p a c t of CL rules on the acceptability of MT d o cu m en ts............ 198
6 .7 . D ata analysis conclusions.......................................................................... 201
C h a p te r 7: C o n c lu s io n ............................................................................................ 2 0 4
7 .1 . The o b jectives.................................................................................................204
7 .2 . F in d in g s............................................................................................................ 205
7 .3 . Review of th e m ethodology used............................................................ 207
7 .4 . Future challenges and research o p p o rtu n ities...................................211
7 .5 . Final w o rd s ...................................................................................................... 212
R e f e r e n c e s ....................................................................................................................2 1 4
APPENDIX A. List of CL rules selected for th e evaluation ..................
APPENDIX B. A vailable rules in custom version of acrocheck™ ....
APPENDIX C. Penn T re e b a n k Tag S et .......................................................
APPENDIX D. Exam ples of m odifications of raw segm ents ...............
APPENDIX E. T ran scrip t of th e evaluation guidelines
given to French and G erm an evaluators (first study) .......................
APPENDIX F. Evaluation results for French .............................................
APPENDIX G. E valuation results for G erm an ..........................................
APPENDIX H. List of violated CL rules
in th e selected docum ents ...........................................................................
APPENDIX I. Source docum ents used to g en erate
French and G erm an MT docum ents .........................................................
APPENDIX J. French MT docum ents (n o t fo rm a tte d ) ..........................
APPENDIX K. G erm an MT docum ents (n o t fo rm a tte d ) ......................
APPENDIX L. S u rv ey results ..........................................................................
APPENDIX M. Evaluators results (second study) ................................
APPENDIX N .T ran scrip t of th e instructions
given to th e French and G erm an evaluators (second s tu d y )
A -l
B -l
C -l
D -l
E -l
F -l
G -l
H -l
1-1
J -l
K -l
L -l
M -l
N -l
Abstract
Johann Roturier: An investigation into the impact of Controlled English rules
on the comprehensibility, usefulness, and acceptability of machine-translated
technical documentation for French and German users.
Previous studies suggest that the application of Controlled Language (CL) rules can
significantly improve the readability, consistency, and machine-translatability of
source text. One of the justifications for the application of CL rules is that they can
have a similar impact on several target languages by reducing the post-editing effort
required to bring Machine Translation (M l’) output to acceptable quality. In certain
situations, however, post-editing services may not always be a viable solution.
Web-based information is often expected to be made available in real-time to ensure
that its access is not restricted to certain users based on their locale. Uncertainties
remain with regard to the actual usefulness of MT output for such users, as no
empirical study has examined the impact of CL rules on the usefulness,
comprehensibility, and acceptability of MT technical documents from a Web user's
perspective. In this study, a two-phase approach is used to determine whether
Controlled English rules can have a significant impact on these three variables. First,
individual CL rules are evaluated within an experimental environment, which is
loosely based on a test suite. Two documents are then published and subject to a
randomised evaluation within the framework of an online experiment using a
customer satisfaction questionnaire. The findings indicate that a limited number of
CL rules have a similar impact on the comprehensibility of French and German
output at the segment level. The results of the online experiment show that the
application of certain CL rules has the potential to significantly improve the
comprehensibility of German MT technical documentation. Our findings also show
that the introduction o f CL rules did not lead to any significant improvement of the
comprehensibility,
usefulness,
and
acceptability
documentation.
viii
of French
MT
technical
List of Figures
Figure 1 -1 : Technical support docum ent view ed in a W eb b ro w ser........................27
Figure 1 -2 : Edited s c re e n s h o t................................................................................................ 29
Figure 1 -3 : Non-localised screenshot in a localised d o c u m e n t................................ 30
Figure 1 -4 : Localised screenshot in a technical support d o c u m e n t........................31
Figure 1 -5 : Missing a lte rn a tiv e te x t in HTML c o d e ......................................................... 31
Figure 1 -6 : In fo rm a tio n w orkflow in a W eb-based a rc h ite c tu re ..,,........................ 36
Figure 1 -7 : Q uery results displayed in W ordS m ith Tools 4 .0 's C oncord...............43
Figure 3 -1 : Linguistic an n o tation of to k e n s ...................................................................... 67
Figure 3 -2 : Rule creation w o r k flo w ...................................................................................... 69
Figure 3 -3 : Selection of S ystran's Translation O p tio n s................................................ 76
Figure 4 -1 : G eneric score p er CL ru le
..........................
99
Figure 5 -1 : French disclaim er used w ithin a custom te m p la te ............................... 153
Figure 5 -2 : C u sto m er satisfaction form for technical support d o c u m e n ts
154
Figure 5 -3 : French subm ission form for online s u r v e y ............................................ 158
Figure 5 -4 : G erm an subm ission form displayed in a pop-up w in d o w ............... 160
Figure 5 -5 : In crease d visibility of th e first d o c u m e n t.................................................166
Figure 5 -6 : T ran slatio n w orkflow using M T ................................................................... 168
Figure 5 -7 : Exam ple o f reference user d iction ary e n tr ie s ...............
170
Figure 6 -1 : Evaluators' first answ ers for do cum ent 'N A V 0 5 _ 1 3 1 '...................... 179
Figure 6 -2 : Evaluators' second answ ers for d o cu m en t 'N A V 0 5 _ 1 3 1 '................ 180
Figure 6 -3 : Evaluators' first answ ers for do cum ent N A V 0 5 _ 1 2 1 ......................... 181
Figure 6 -4 : Evaluators' second answ ers for d o cu m en t N A V 05_12 1 .................. 182
Figure 6 -5 : Responses subm ission m e th o d .......................
186
Figure 6 -6 : D istribution of su b m itted re s p o n s e s ....................................................... 187
Figure 6 -7 : Users' answ ers to th e third question for docum ent 'N A V 0 5 _ 1 3 1 ' 188
Figure 6 -8 : Users' answ ers to th e first question for docum ent 'N A V 0 5 _ 1 3 1 '. 192
Figure 6 -9 : Users' answ ers to the third question for docum ent 'N A V 0 5 _ 1 2 1 ' 194
Figure 6 -1 0 : Users' answ ers to th e first question for docum ent 'N A V 0 5 _ 1 2 1 ' 197
Figure 6 -1 1 : Users' answ ers to the fourth question (both d o c u m e n ts )
198
Figure 6 -1 2 : Users' answ ers to th e fifth question (both d o c u m e n ts ).................. 200
List of Tables
Table 1 -1 : Exam ples of technical tokens presen t in th e c o rp u s ....................
41
T ab le 3 -1 : Scores used by th e e v a lu a to rs ..........................................
86
T ab le 4 -1 : In te r-e v a lu a to r a g r e e m e n t.................................................................................92
Table 4 -2 : Evaluators' consistency on identical s e g m e n ts ......................................... 94
T ab le 4 -3 : Rules w ith
m ajo r positive im pact in both lan g u ag es .............. 101
T ab le 4 -4 : List of rules with lim ited positive im pact in both lan g u ag es
Ill
Table 4 -5 : Rules w ith no im pact in both la n g u a g e s ....................................................118
T ab le 4 -6 : Rules th a t show signs of n eg ative im pact in both la n g u a g e s
122
T ab le 4 -7 : Rules w ith
m ore positive im pact in French than in G e rm a n
126
Table 4 -8 : Rules w ith
m ore positive im pact in G erm an than in F ren ch
134
T ab le 4 -9 : Final selection of CL rules based on th e ir e ffe c tiv e n e s s ................... 143
T ab le 5 -1 : N u m b er of answ ers received (M ay 3 1 s t - June 13th 2 0 0 6 ) ............ 165
T ab le 5 -2 : N u m b er o f rule violations and words per d o c u m e n t.............................172
Table 5 -3 : Types of CL rules violated in th e selected d o cu m en ts..........................172
Table 5 -4 : Violations of rule 51 in th e selected d o cu m en ts................................... 174
Table 6 -1 : Response rates (June 6th - July 9 th 2 0 0 6 ) ............................................. 184
x
Abbreviations
AECMA
AI
CASL
CL
CLAW
CMS
CSS
CTE
GIFAS
GMS
GUI
HTML
HTTP
IDE
LGP
LISA
LSP
MT
NLP
NP
PDF
PE
POS
PP
RBMT
RSS
SE
TM
UI
URL
VP
W3C
XML
XSL-FO
XSL-T
Association Européenne des Constructeurs de Matériel Aérospatial
Artificial Intelligence
Controlled Automotive Service Language
Controlled Language
Controlled Language Application Workshop
Content Management System
Cascading Style Sheet
Caterpillar Technical English
Groupement des Industries Françaises Aéronautiques et Spatiales
Globalisation Management System
Graphical User Interface
HyperText Markup Language
HyperText Transfer Protocol
Integrated Development Environment
Language for General Purpose
Localisation Industry Standards Association
Language for Specific Purpose
Machine Translation
Natural Language Processing
Noun Phrase
Portable Document Format
Post-Editing
Part Of Speech
Prepositional Phrase
Rule-Based Machine Translation
Rich Site Summary
Simplified English
Translation Memory
User Interface
Uniform Resource Locator
Verb Phrase
World Wide Web Consortium
extensible Markup Language
extensible Style Language Formatting Objects
extensible Style Language Transform
Introdu ction
Introduction
In 2003 this researcher was involved in a Machine Translation (MT) evaluation
project from which the idea for this dissertation originated. This project was
conducted within the localisation department of a software publisher, Symantec,
which specialises in security and availability solutions. During this project, special
attention was to be paid to the way source content was authored so as to reduce the
Post-Editing (PE) of MT output to a minimum (Roturier, 2004). Such an objective
was based on a long series of initiatives that have been undertaken in the field of
technical communication, whereby publishers have attempted to increase the
comprehensibility of their source content for both humans and machines by
implementing a Controlled Language (CL). Huijsen (1998: 2) gives the following
definition of a CL:
A CL is an exp licitly defined restriction o f a natural language that
sp ecifies constraints on lexicon , grammar, and style.
This definition can be supplemented by the fact that it is almost always used to
'write clear technical documentation in a particular domain'. (Power et al., 2003:
115). Industry-based projects (such as Bemth: 1998, Kamprath et al.: 1998,
Godden: 1998, or Rychtyckyj: 2002) have shown that the quality of MT output can
be significantly improved by removing complexity and ambiguity from source
technical documentation. By placing strict linguistic and pragmatic restrictions on
the source text, the quality of MT output can be improved to such an extent that it
seems possible to significantly reduce translation turnaround and costs. Using this
approach, the traditional translation process may be effectively replaced by a PE
step to bring the quality of MT output to an acceptable standard. This has been
empirically confirmed by O'Brien (2006: 177), who discovered that 'controlling the
input to MT leads to faster post-editing'.
Until now, however, no research has been conducted to determine whether the
application of CL rules can have a positive impact on the reception of machinetranslated technical documents that are not post-edited prior to their publishing and
dissemination to users. The aim of the present dissertation is to address this question
and make a useful contribution to the field by focusing on the reception of machinetranslated technical documents from a Web user's perspective.
2
Objectives o f the study
The present study has two main objectives. First, it will attempt to determine
whether some individual CL rules are more effective than others in improving the
comprehensibility of MT output in two target languages when no post-editing is
performed. If this were the case, the most effective rules could be used as authoring
standards when developing specific Web-based technical documentation. In this
study, the 'comprehensibility' of a translation is based on the definition provided by
T.C. Halliday, who states that comprehensibility refers to 'the ease with which a
translation can be understood, its clarity to the reader.’ (quoted in Van Slype, 1979:
62). In order to achieve this objective, the following questions will have to be
answered: are specific rules always effective regardless of the target language? Are
there any cases where the rule may have a negative impact on the comprehensibility
of the target text? In these cases, should the scope of the rules be reduced to allow
for exceptions?
The second objective of this study is to investigate whether the application of
certain CL rules can influence the reception of machine-translated documents by a
specific category of information consumers: users of software products who require
technical information in their own language. The decision to solicit feedback from
real users stems from the general lack of involvement of users in the evaluation
process of translations as end products (Lauscher, 2001: 166). It appears important
to fill this gap since 'user feedback can make a valuable contribution to an
assessment programme; the aim of which is to evaluate the quality of
documentation after it is distributed' (Marlow, 2005: 36). The ease in accessing
users of Symantec products is therefore a unique opportunity to address this
question by involving users in the evaluation process of machine-translated
documents within the framework of a case study.
In order to achieve this second objective, the following question will have to be
answered. From a Web user's perspective, can the application of CL rules
significantly improve the comprehensibility, usefulness, and acceptability of online
machine-translated documents? In this study the term 'usefulness' is based on the
definition provided by Richardson (2004: 247), whereby 'customers feel that an
3
article helped them solve their problem'. Several studies conducted at Microsoft and
Cisco (Richardson, 2004, Jaeger 2004) have indeed suggested that refined MT
output could be useful to users. The term 'refined MT output1is used to reflect any
customisation an MT system may have received to be used in a specific domain
without post-editing. Such customisation may include the use of specific user
dictionaries or the design of extra rules. As mentioned by Allen (2003: 298),
however, it is also 'important to determine to what extent MT output texts are
acceptable' for their users. In this study, the term 'acceptability' is based on the
fourth standard of textuality defined by De Beaugrande and Dressier (1981: 7). This
standard concerns 'the text receiver's attitude that the set of occurrences should
constitute a cohesive and coherent text having some use or relevance for the
receiver'. Since this definition of acceptability contains the term 'use', the concepts
of acceptability and usefulness must be further clarified. Acceptability does not only
refer to the relevance a text has for its receiver, but also to the manner in which its
textual characteristics are going to be accepted, tolerated, or rejected by its receivers.
Lassen (2003: 76) infers that if a document does not meet specific expectations in
terms of style, layout, and content, readers who belong to the discourse community
'would consider it unacceptable as a specimen of the genre'. In the context of
software technical support documentation, the discourse community encompasses
users of software products who need to find solutions to their problems. The
machine-translated solutions some users are being provided with may be useful in
certain cases, but are these documents acceptable as specimens of the genre? One
possible way to answer this question is to determine whether users are willing to
consult such documents again in the future. Based on De Beaugrande and Dressler's
terminology (1981: 8), this will allow us to examine whether the potential
'disturbances' contained in machine-translated documents are being 'tolerated' by
users of technical support documentation.
In this study the choice of the target languages was based on the ability of this
researcher to analyse MT output, thus French and German were selected. It was
important to select two languages that were not too similar, such as two Romance
languages. It was also decided to focus on the output of a single MT system rather
than to compare the impact of the rules on the outputs produced by several MT
systems. This choice was prompted for several reasons. It was felt that the effort
4
involved in adding corpus-specific terminology to several systems’ user dictionaries
would be too great. Since commercial MT systems rely on proprietary terminology
formats, the interchange of user dictionary resources from one MT system to the
next is not automatically possible. Besides, evaluation costs could spiral if more
than one MT system were used, since the time required for the evaluation process
would increase significantly. Instead of having to reduce the amount of evaluation
material, it was preferable to use only one MT system. Systran Web Server 5.0 was
selected for a number of reasons. First, this system is used to power most Webbased translation portals today. Second, it is a broad coverage MT system so all
sentences will be parsed in one way or another. Finally, this product was available
in Symantec's localisation department. Symantec's user dictionaries of this MT
system contain domain-specific terminology, thus reducing the terminological
customisation required prior to the evaluation process.
Context and m otivations
The benefits of CLs abound in the marketing documentation of certain CL checker
providers (Smart, 2006: 8). A CL checker may be defined as an application
designed to flag linguistic structures that do not comply with a predefined list of
formalised CL rules. However, one should not forget that CLs may also have sideeffects, since the acceptability of the source text could be affected by the
introduction of rules. This danger is expressed by Huijsen (1998: 12), who states
that 'some writing rules may even do more harm than good'. This problem was
foreseen during the implementation of Caterpillar Technical English (CTE) at
Caterpillar, but internal study findings revealed that CTE versions of source
documents were well received by English readers. Not only were Caterpillar's
translation costs reduced with the introduction of CTE, but the acceptability of
source texts was also enhanced (Hayes et al., 1996: 87). Focus group testing
revealed that the CTE versions of sentences were easier to understand, while the
CTE versions of documents were easier to skim thanks to the low incidence of
dependent sentences. This example confirms that a careful selection of specific CL
rules is essential in achieving well-defined objectives.
5
However, deploying a large set of CL rules is sometimes not feasible due to time
constraints. Having to check that a text conforms to a large set of CL rules will take
time to complete, even with the use of a CL checker. In the worst case scenario,
such a pre-editing process might take longer that the actual post-editing of the MT
output. For instance, Govyaerts (1996: 139) reports that the controlled authoring
process at Alcatel Bell could take up to 20% longer than the traditional process,
thus affecting the writers and the whole document production workflow.Identifying
such
pitfalls was one of the objectives mentioned by Schachtl (1996: 143)during
the design of a German CL for Siemens documentation. Siemens saw the benefits of
using CL to improve MT, but they did not want to 'simplify the source text too
much, obstruct the process of authoring and cause the texts to get too long' (ibid).
Based on these recommendations, the impact of individual rules should therefore be
evaluated empirically to determine whether the impact of a given rule justifies its
implementation in a systematic way. However, Nyberg et al. (2003: 276) remark
that ‘the motivation for individual writing rules is generally based on intuition
rather than on empirical evidence'. This may be due to the fact that determining the
role that a specific individual CL rule plays in a given rule set is a challenge, as
stated by Douglas and Hurst (1996: 94):
W hile it m ight be p o ssib le to evaluate the quality o f a docum ent
conform ing to a sp ecific CL, this does not a llo w us to say
anything about the effects o f particular elem ents in the definition
o f the CL.
Trying to address this challenge was therefore a key motivation for the undertaking
of this study due to the paucity of empirical research conducted on the impact of
individual rules.
The undertaking of this study was also spurred by increasing demands for Webbased translated documents. Following a Global Strategies Summit organised by the
Localisation Industry Standards Association (LISA), LISA's Chief Analyst, Bill
Pullman (2006), admitted that nobody knows the actual size of the opportunities
that the localisation industry will be able to generate in years to come. What is
certain, however, is that the World Wide Web has dramatically changed the way in
which content must be localised to be made available to users. With the emergence
of a wide range of new communication channels, such as Rich Site Summary (RSS)
6
news feeds or alert text messages that are generated from databases, the life-cycle of
content has been greatly reduced. This means that the time available to localise
content is constantly being reduced. Being able to localise all content is further
complicated by the size and growth rates of databases in which content is stored. As
far as multilingual communication is concerned, one of the challenges for
publishers lies in making their content accessible to most of their global customers,
and preferably within a very short timeframe. For many publishers the use of
Machine Translation (MT) therefore presents itself as a promising cost-effective
solution to address these challenges and produce translated content in timely
manner. However, Web authoring guidelines for technical documentation are often
not sufficiently precise to ensure the machine-translatability of source content. If
stricter rules were used as standards, the performance of MT systems may be
improved. The present study will focus on these challenges by examining the
impact o f specific CL rules on the comprehensibility, usefulness, and acceptability
of machine-translated technical documents.
Scope o f the study
This study will focus on the reception of machine-translated documents from a Web
user's perspective, but will not evaluate the usability of the documents. Usability
evaluations focus on the users’ actions rather than just on their opinions (Dray,
2004: 31), by studying, for instance, the interaction of users with a given Web
interface. However, Web sites now tend to use or reuse chunks of information that
originate from databases. This information is then published in multiple formats,
using tools that ensure that form is always separated from content. The form of a
message is also sometimes referred to as the ‘package’ (O’Hagan & Ashworth,
2002: 67), dealing with the layout or the selection of fonts and graphics. The final
look and feel o f a Web document therefore often stems from the template that is
used to display the content of databases. Since the purpose of this study is to focus
on the content rather than on the form or ‘package’, a traditional usability study
does not seem to be appropriate.
7
In this study, it was decided to focus on a coipus originating from the IT domain.
The corpus contains documents whose main purpose is to instruct, using procedural
text. Selecting a corpus from the IT domain is motivated by several reasons. First of
all, the CL and MT paradigm may not be new, but it has so far almost exclusively
been the preserved territory of the automotive, aeronautic, and telecommunication
industries. Apart from Rank Xerox’s use of controlled English input to facilitate
MT (Adams et al., 1999: 250), a CL and MT project in the IT industry has yet to
publish successful results such as those achieved in the automotive sector by
companies such as Caterpillar (Kamprath et al., 1998) or Ford (Rychtyckyj, 2006).
Times seem to be changing, as SAP was the first company in the software sector to
introduce a CL application (Lieske et al., 2002: 1) to ensure the consistency, quality,
and machine-translatability of its English and German source documentation (ibid,
Schäfer, 2003: 140). Sun Microsystems have also introduced CL in their authoring
workflow in 1999, and in 2002 were a year away from introducing MT in their
production cycle (Akis & Sisson, 2002: 4). However, no empirical study has yet
focused on evaluating the impact of CL rules on the reception of Web-based
technical support documentation, despite claims that source quality is a critical
factor for the success of such as process (Warburton, 2003, Jaeger, 2004).
This empirical study will be based on a two-phase approach and will fill a gap in the
field of CL evaluation, by focusing on the impact of individual rules at the segment
and document levels. There is currently no standard methodology available to
evaluate the impact of individual CL rules. This study intends to set up an
environment that could be reused in future projects to assess the performance of
other MT systems with other language pairs, provided that the system uses English
as the source language. This study also intends to deliver a clear and unique
evaluation of the impact of CL rules on MT output from a Web user’s perspective.
This will help us determine whether the reception of machine-translated documents
can be influenced by the introduction of certain rules.
8
The structure o f the dissertation
The aim of Chapter 1 is twofold. First, it will further explore the context of the
present study to discuss the main characteristics of Web-based multilingual
technical documentation. This review is performed to identify a text type that is
suitable for the present study. Second, it will present the challenges involved in the
design, compilation, and description of a corpus corresponding to the selected text
type.
In Chapter 2, the concept of CL will be discussed in the light of its use with MT, by
reviewing existing CL rules and machine translatability guidelines. The main
objective of this review is to select a set of frequent individual MT-oriented CL
rules so that the impact that they have on the comprehensibility of MT segments can
be evaluated.
Chapter 3 will present the methodology used to extract segments from the corpus
and set up an evaluation environment in order to evaluate two sets of MT segments.
These MT segments will be obtained by machine-translating two sets of source
segments. The first set will contain violations of the rules harvested in Chapter 2,
while the second set will contain segments rewritten according to specific CL rules.
Chapter 4 will analyse the results of the first evaluation round, by comparing the
scores attributed to the two sets of MT segments created during the setup phase of
the evaluation environment described in Chapter 3. The discussion will be based on
a comparison of the results obtained for the two target languages. At the end of this
chapter, a final list of rules will be drawn and used in the second part of this study.
Chapter 5 will describe the methodology used for the second experimental part of
the study, whereby two sets of online documents will be randomly presented to a
sample of users of Symantec products. The design of this online experiment and the
customer satisfaction questionnaire used will be discussed.
9
C h ap ter 6 w ill present the findings o f the second part o f the study. T he discussion
will be based on a com parison o f the im pact o f the rules at the do cu m en t level, by
con trastin g results obtained from French and G erm an users.
In C h ap ter 7, the tw o objectives o utlined ea rlie r w ill be revisited to determ ine
w h ether they have been m et. T he findings o f b o th parts o f the study w ill be
sum m arised and the m ethodology used will be critiqued. N ew research questions
arising from th e fin d in g s o f this study w ill be p resen ted to possibly p av e the w ay for
fu tu re research.
10
Chaptcr 1
11
Chapter 1: Designing a corpus of Webbased technical documentation
1 . 1 . Objectives o f the present chapter
This chapter is divided into two sections. Section 1.2 will further explore the
context of the present study to discuss the main characteristics of Web-based
multilingual technical documentation. The interaction between Web design
guidelines and localisation requirements is reviewed in the context of an MT
process. This review is performed to identify a text type that is suitable for the
present study. Section 1.3 will then present the challenges involved in the design,
compilation, and description of a corpus corresponding to the selected text type.
1.2. C haracteristics o f m ultilingual W eb-based
technical docum entation
1. 2. 1.
W e b lo c a lis a tio n b o ttle n e c k
The area of Web localisation is sometimes perceived as the ‘fastest-growing area in
the translation sector today’ (O’Hagan & Ashworth 2002: ix). This is no surprise
considering the ever increasing amount of content to be translated in a veiy limited
period of time. Even though localisation does not only involve translation,
publishers are often striving for a simultaneous publication of their information in
multiple languages. As far as multilingual Web sites are concerned, Esselink (2001:
17) also warns that ‘the frequency of updates has raised the challenge of keeping all
language versions in sync (...), requiring an extremely quick turnaround time for
translations.’ However, providing information before it becomes obsolete is
sometimes not possible for publishers, and some content is published exclusively in
the language in which it has been authored. Yunker (2003: 75) remarks that ‘unless
the target audience consists of only bilinguals, this approach is bound to leave
people feeling left out.’ This is confirmed by O’Hagan & Ashworth (2002: 52) who
point out that ‘readers require a given Web page to be in their language to allow for
real-time browsing and information gathering’. In 2000, the International Data
Corporation carried out a survey within the framework of the Atlas II project. Based
on the results obtained from 29,000 Web users, they estimated that by 2003, 50 per
cent of Web users in Europe would be likely to favour sites in their native language
12
(Myerson 2001:14). This trend is noteworthy, as the number of Internet users that
are non-native English speakers grows by over 140 million per year (Levin, 2005:
45). The lack of global distribution and accessibility has been highlighted by Pym
(2004: 9) and is reflected by three types of locales:
■
The ‘participative’ locale consists of users who are able to access
information in a language they can understand. These users are then
able to act upon the information they have accessed.
■
The ‘observational’ locale consists of users for whom it is too late to
do anything with the information they access. They are able to
access it in their own language, but by the time this information is
translated, it is obsolete.
■
The ‘excluded’ locale consists of users who are never given the
chance to gain access to information in a language they can
understand.
For certain users, this accessibility problem may be alleviated by the use of online
MT services provided by portals such as Google or AltaVista Babel Fish. If
excluded users cannot get access to information in their own language, they may
resort to online MT services to attempt to understand foreign language information.
1. 2 . 2 .
F r e e M T s e r v ic e s o r r e f i n e d M T s e r v ic e s ?
The use of MT for the translation of Web content has been described by Hutchins
(2004: 16):
C lo sely related to the use o f M T for translating texts for
assim ilation purposes is their use for aiding bilingual (or cross­
language) com m unication and for searching, accessin g and
understanding foreign language inform ation from databases and
w ebp ages.
With this new approach, users of free online MT services may be able to get the
'gist' of a Web page thanks to raw MT output. 'Raw MT output' is used to refer to
the translation produced by a general purpose MT system, which was not
customised in any way for its end-users. Informal reports suggest that these users
show a pragmatic attitude towards the capabilities of these systems (Somers, 2003 :
523), which may help them overcome the barriers of a communication deadlock.
However, such a practice ‘goes against the general recommendations for the use of
13
MT that have been made over the last decade or so’ (ibid). As indicated by Yang &
Lange (2003: 200), it is difficult to evaluate the usefulness of MT output for these
users. This seems especially true when the type of content published is domainspecific and requires a certain level of quality. For instance, in the case of online
technical documentation, users must be able to act on the information they are given
to read. A general purpose MT system whose dictionaries do not contain specific
terminology will probably prove inadequate in these situations. In order to try and
address this issue in a cost-effective manner, certain software publishers, such as
IBM, Cisco, and Microsoft have therefore started providing online refined MT
content to the users of their online knowledge bases (Warburton, 2003, Jaeger, 2004,
Richardson, 2004).
Uncertainties, however, remain as to whether refined MT output can be accepted
and used effectively as a translation product by Web users, especially when source
content is not controlled. Controlling source content in a Web localisation context
may be regarded as a Web globalisation task, since its objective is to ensure that as
many locales as possible can gain access to a certain type of information. Section
1.2.3 will review some of the guidelines used for the development of Web content
to determine the extent to which they contribute to making content more accessible
and translatable in a Web globalisation context.
1. 2 . 3 .
W e b g lo b a lis a tio n
According to Pym's terminology (2004: 9), the localisation process attempts to
transform ‘excluded’ locales into ‘participative’ locales rather than ‘observational’
locales. Since this process might involve the publication of information in multiple
languages, publishers must plan ahead to ensure they can quickly cater for all their
multilingual customers. This challenge is commonly regarded as a Web
globalisation issue, whereby Web content should be designed and maintained with
localisation in mind (Esselink, 2003: 68). From a translation perspective, efforts are
being made by working groups within the World Wide Web Consortium (W3C) to
standardise the handling of specific content within XML documents, so as to
simplify the localisation process of Web content. The W3C is the organisation that
works
on
creating
and
maintaining
Web
standards.
For
instance,
the
Internationalisation Tag Set (ITS) provides translatability rules so that translators
14
know whether an element must be translated or n o t'1. This approach seems
particularly adapted to technical information, which often contains technical words
or chunks of text that should not be parsed by a general MT system. However, nontranslatable content may not be systematically marked-up. This issue is raised by
Schachtl (1996: 145), who states that 'a common fault in software documentation is
the insufficient marking of chunks of object code in running text.' The metadatabased approach of the ITS also proposes the possible tagging of ambiguous terms to
ease the translation process. From an MT perspective, the challenge lies in ensuring
that MT systems can understand these rules and act on them. Whereas the ITS
focuses on the translatability and localisability of Web content, specific Web design
guidelines deal with the actual accessibility of the content. These guidelines are
reviewed in the next two sections.
1 . 2. 3 . 1.
W e b a c c e s s ib ility g u id e lin e s
In his foreword to Maximum Accessibility: Making your Web Site More Usable for
Everyone (Slatin and Rush: 2003), Nielsen states that Web usability and Web
accessibility are ‘two tightly intertwined concepts’, because content that is made
accessible to a certain group of users ‘tends to be usable for all users’. Existing
linguistic Web accessibility guidelines (W3C, 1999) focus on simplicity and clarity,
so it may be hypothesised that they share certain features with CL rules. In this light,
violations of CL rules may impact Web accessibility. This use of the term
‘accessible’ does not correspond to the strict definition of ‘Web accessibility’ used
by Slatin & Rush (2003: 3). For them, Web sites are deemed ‘accessible when
individuals with disabilities can access and use them as effectively as people who
don’t have disabilities’. This strict use of the term ‘accessible’ refers to section 508
of the US Rehabilitation Act of 1973, which was amended by Congress in 1998.
However, the semantic scope of this term has since widened to encompass groups
of users that contain more than just users with physical disabilities. For instance,
Nielsen
(2000:
309)
mentions
that
cognitive
disabilities
should not be
underestimated.
1 For more information, see http://www.w3.Org/TR/its/#translate [Last accessed: September, 18th
2006]
15
It may even be argued that people who are not competent in a certain language can
be affected in the same way as, say, visually-impaired people. The W3C has
identified these new challenges, and use the term ‘Web accessibility’ to refer to a
wide range of situations. In their guidelines, they suggest that ‘many users may be
operating in contexts very different’ from the situations that Web developers are
familiar with. Certain users may not be able to perceive or interact with Web
content, which prevents them from having access to the information they need. The
W3C (1999) draws the following list of users who may find certain Web content
non-accessible:
1.
‘They may have a text-only screen, a small screen, or a slow Internet
connection.
2.
They may have an early version of a browser, a different browser
entirely, a voice browser, or a different operating system.
3.
They may not have or be able to use a keyboard or mouse.
4.
They may be in a situation where their eyes, ears, or hands are busy
or interfered with.
5.
They may have difficulty reading or comprehending text.
6.
They may not speak or understand fluently the language in which the
document is written’.
This list shows that challenges to Web accessibility can originate from a wide range
o f issues. They can be of a technical nature, as in situations 1 and 2, of a physical
nature as in situations 3, 4, and 5, and of a cognitive nature as in situations 5 and 6.
Technical and physical issues, which mainly pertain to the form of Web content, are
abundantly addressed in the W3C guidelines and specialised literature. However,
cognitive issues, which are often linked with linguistic accessibility, seem to be
neglected. If we were to use Klare’s definition of accessibility (1963), we could say
that the ‘ease of understanding or comprehension due to the style of writing’ is
absent in non-accessible Web sites. This is valid for Web content which is not
localised, and therefore non-accessible for the group of users who are not competent
in the language in which the source content has been published. But to some extent,
this also applies to Web content which is too difficult to comprehend, even for
certain native speakers, due to their lack of technical expertise. Some users may
16
then be able to physically and technically access specific Web content without being
able to get the answer they may be looking for. On the other hand, people with the
proper technical background may not be able to physically or technically gain
access to Web content they would able to understand. Specific writing guidelines
used to make Web content more accessible and comprehensible are discussed in the
next section.
1. 2 . 3 . 2 .
W r it in g g u id e lin e s f o r W e b c o n te n t
Linguistic Web accessibility guidelines are not always the main concern of Web
accessibility experts. In their preface to Maximum Accessibility: Making your Web
Site More Usable fo r Everyone, Slatin and Rush (2003: xxiii) inform readers that
they will tell them ‘how to make the World Wide Web more accessible and more
usable for everyone around the world’. In this same preface, they mention the
W3C’s 14th guideline concerning the clarity and the simplicity of the documents.
Yet, they do not refer to this guideline anywhere else in their book.
More detailed writing guidelines are presented by Spyridakis (2000: 376) to
produce comprehensible Web content. She mentions that certain text features will
impact the reading process:
I f readers can d evote less attention to low er level tasks such as
decod ing letters, w ords, and syntactic structures, they w ill have
m ore attention available for higher level tasks, such as com bining
text-based inform ation with other text-based inform ation and also
w ith inform ation stored in long term m em ory.
In order to avoid such processing issues, Spyridakis provides a list of guidelines that
are destined to improve the comprehensibility and translatability of Web content .
However, most of these guidelines are not precise enough to be implemented on a
systematic basis, an example being the use of 'simple sentence structures and
internationalized words and phrases'. This lack of precision in the definition of
specific guidelines was also found in Designing Web Usability: the Practice o f
Simplicity (Nielsen, 2000: 101). Nielsen provides general advice without getting
into great detail. He mentions that the three main guidelines for writing for the Web
2 T hese guidelines are available at:
hKp://w w w .tiw tc.w ashiniiton.eclu/research/m ibs/ispvridaki.s/O uicklist C om p reh en sib le W eb Pages
[Last a ccessed on A u gu st, 2 5 th 2 0 0 6 ]
17
con cern text c o n c ise n e ss, con tent scan n ab ility, and the p resen ce o f hyperlinks. T he
first tw o g u id elin es share o n e co m m o n point: th ey are d ifficu lt to apply on a
sy stem atic basis due to their general nature. T h ey do n ot m ake ex p licit the m eans
w h ich can be u sed to render con ten t clearer and sim pler. T h ese g u id elin es’ lack o f
p rec isio n d oes n ot b en efit users o f tech n ical docu m en tation .
T he
first
g u id elin e,
c o n cisen ess,
ec h o e s
a
general
principle
of
tech nical
co m m u n ication , w h ereb y w riters are often asked to a v o id verb osity (D 'A gen ais and
Carruthers, 1985: 100; G erson and G erson , 2000: 31; R am an and Shaim a, 2004:
187). H o w ev er, e x c e s s iv e c o n c ise n e ss m ay so m etim es h ave a n egative im pact on
the clarity o f m e s sa g e s , due to am b igu ities introduced b y th e com p ression o f
inform ation (B yrn e, 2 0 0 4 : 2 4 ). B y re m o v in g w ords su ch as articles or prep ositions
from the sou rce tex t, the clarity o f a m e ssa g e m ay b e affected . T his issu e is lik e ly to
occur i f g u id elin es are too stringent. For instance, G erson and G erson (2000: 2 4 3 )
su g g est that W eb con ten t sen ten ces sh ou ld contain b etw een 10 and 12 w ords. I f this
g u id elin e
w ere
to
be
en forced
in
a sy stem a tic
m anner,
essen tial
syntactic
co m p o n en ts w o u ld be rem o v ed from certain sen ten ces.
N e x t, th e scan n ab ility o f W eb con tent seem s m otivated by tw o factors. First, it is
b e lie v e d that u sers take 20 -3 0 % lon ger to read from a screen than from a p age, and
seco n d , th ey rarely read a sou rce tex t in its entirety (N ie lse n , 2000: 104). T hey tend
to fo c u s on the part o f th e tex t that m o st interests them . P ym (2004: 187) ev e n g o es
as far as sa y in g that users are ‘no lon ger readers’ due to the lo ss o f d iscu rsive
linearity o f texts.
T his re v ie w o f W eb con ten t and W eb a cc essib ility g u id elin es su g g ests that p recise
ru les are required i f M T sy stem s are to be used e ffe c tiv e ly for the p roduction o f
u sefu l W eb -b a sed translated inform ation. A s b riefly d iscu ssed in the introduction,
certain C L rules m a y h a v e the p oten tial to co m p lem en t th ese gu id elin es in order to
im p rove the m ach in e-tran slatab ility o f W eb -b ased tech n ical docum entation. B efore
su ch rules can be p ro p o sed as standards, their e ffe c tiv e n e ss m u st be evalu ated at the
seg m en t le v e l and d o cu m en t le v e l. In order to a ch iev e th ese ob jectives, test content
is required. S e c tio n 1.3 d isc u sse s th e w a y in w h ich test con tent w ill be assem b led ,
by
ex a m in in g
th e
c h a llen g es
in v o lv e d
descrip tion .
18
in
corpus
d esign ,
com p ilation ,
and
1.3» Corpus design, com pilation, and description
1 .3 .1 .
I m p a c t o f t h e s t u d y ’s o b j e c t i v e s o n c o r p u s d e s i g n
r e q u ir e m e n ts
K en n ed y (1998: 7 0 ) rem arks that the ‘op tim al d e sig n o f a c o ip u s is h igh ly
d ependent on the purpose for w h ich it is intended to b e u se d ’. T he ob jectives o f this
study m ust therefore be a sse sse d to determ ine their im p act on corpus d esign. In this
study, the corpus w ill be u sed to a ch iev e the fo llo w in g o b jectiv es in a tw o-p h ase
approach:
■
to p rovid e m u ltip le v io la tio n s o f CL ru les to evalu ate th e im pact o f
th ese rules on the com p reh en sib ility o f refin ed M T segm en ts
■
to p rovid e gen u in e tech n ical d ocu m en ts that w ill b e m achin e
translated
and
p u b lish ed
w ith in
the
fram ew ork
o f an
on lin e
exp erim en t in v o lv in g users w h o require lo c a lise d inform ation
A
sp ecific
corpus m u st b e
id en tified
in the
field
o f W eb -b ased tech nical
d ocu m en tation so that th ese tw o o b jectiv es can b e a ch ieved .
T he tw o o b jectiv es h a v e im p lica tio n s w ith regard to th e au dien ce o f th e tex t type
u sed in this study. T ech n ica l d ocu m en tation m ay b e prod uced for a w id e range o f
u sers, so the le v e l o f tech n ical inform ation w ill vary d ep en d in g on th e audience
targeted. In the IT d om ain, tw o m ain groups o f users can be id en tified b ased on the
d istin ction that is o ften m ade b etw een h om e and en teip rise softw are products:
■
H o m e users, w h o m ay or m ay n ot h a v e any tech n ica l k n o w led g e
■
E nterprise u sers, su ch as system or netw ork adm inistrators, w h o
sh ou ld h ave an ex c e lle n t tech n ical b ackground
E v en th ou gh th e d ifferen ce is n ot alw ays ea sy to
ascertain, m o st tech nical
d ocu m en tation targets at lea st on e o f th o se ca teg o ries o f users. T his d ifferen ce,
h o w ev er,
has
an in flu e n c e
on
the co m m u n ica tiv e
fu n ction
and
localisation
requirem ents o f a g iv en te x t type.
T h is separation b etw e en general and p rofession al u sers im p acts on the features o f
th e sp eech act that w ill be u sed in the com m u n ication p rocess. Sager (19 9 4 : 52)
u se s the term speech act ‘to d esign ate fu n ction ally coh eren t interactions perform ed
b y m eans o f lin g u istic s ig n a ls’. In short, th e lin g u istic features o f any p ub lish ed
19
tech n ical m e ssa g e w ill b e b ased on the typ e o f interlocutors in volved in the sp eech
act. T ech n ical con tent d evelop ers are sp ecia lists, w h ereas readers or users w ill
either b elo n g to a lay p erson category or a sp ecia list category. T he d ich otom y
in v o lv e d w ith the participants o f the sp eech act w ill therefore im pact on the
lin g u istic m aterialisation o f the tech n ica l m e ssa g e . A cco rd in g to Sager (1994: 45),
the lan gu age u sed b etw een sp ecia lists w ill h over b etw een ‘artificial la n g u a g e’ and
‘sp ecial subject la n g u a g e’ u sed in an overall ‘sp ecia lised d isco u rse’. O n the other
hand, the lan gu age u sed b etw e en sp ecia lists and lay-p erson s w ill only u se ‘artificial
lan gu age ele m e n ts’ u sed in an overall ‘general d iscou rse and popularised sp ecialist
d isc o u r se’. S in ce th e te x t typ e ch o sen for this study m u st not be too sp ecific, it
sh ould be targeted at h om e users, w h o are m ore lik e ly to require translated
d ocu m en tation than their enterprise counterparts.
T he seco n d ob jectiv e a lso h as tech n ica l im p lication s. A s M T content w ill have to be
p u b lish ed in a rea l-life situ ation, the repository o f the ch o sen tex t type should a llo w
for the storage o f lo c a lis e d content. K en n ed y (ib id) also in sists on three central
issu e s in corpus d esign : w h eth er th e corpus sh ou ld b e static or d ynam ic, the
represen tativeness o f th e corpus, and its size. T h ese issu e s w ill be rev iew ed in the
lig h t o f our o b jectiv es so as to ob tain a fin al list o f corpus d esign requirem ents.
1 .3 .l . i .
S ta tic o r d y n a m ic c o rp u s ?
T he m ain o b jectiv e o f th is study is n ot to m on itor the evolu tion o f a certain
tech n ica l authoring sty le o v er tim e. Rather, raw textu al data is required for the first
part o f th is study, so that the im p act o f CL rules on th e com p reh en sib ility o f refined
M T output can b e evalu ated at the seg m en t lev el. O lohan (2 0 0 4 : 4 5 ) m aintains that
a static corpus p ro v id es a ‘sn apsh ot o f aspects o f th e lan gu age at a particular point
in tim e .’ T his d o es n o t m ean, h ow ever, that the fo c u s sh ould on ly be p laced on
con ten t that has b een w ritten in a lim ited period o f tim e.
I f v io la tio n s o f ex istin g C L rules are to be fou n d in th e corpus, the content should
id e a lly h ave b een authored o v er a num ber o f years and still b e a ccessed b y users.
S o far th e term s ‘c o n te n t’ and ‘d o cu m en ts’ h ave b een u sed w ith ou t p rovid in g any
strict d efin ition s. T rad ition ally, tech n ical d ocu m en ts (or articles) are co m p o se d o f
20
con tent w h ich can be updated on an irregular b asis (o n ly w h en n e w inform ation is
a v a ilab le), or on a regular b asis (to m atch the quarterly or yearly release o f a n ew
product version ). So a tech nical d ocu m en t m ay w e ll h ave b een ph ysically created a
fe w years ago, but som e o f its con ten t m a y h ave sligh tly or rad ically altered over
tim e. A cco rd in g to H am m erich and H arrison (2 0 0 2 : 2), the term 'content' refers to
the 'written m aterial on a W eb site', w h ereas the 'visuals refer to all form s o f d esign
and graphics'. T he selected corpus sh ou ld therefore integrate any con tent that cou ld
b e m achine-translated in the secon d part o f the study, ev en i f it w a s written a fe w
years ago. S o m e o f th e lin g u istic constructs u sed in the con tent d ev elo p ed a fe w
years ago m a y no lon ger appear in n e w d ocu m en ts. For instance, certain content
d ev elo p ers m a y h ave stopped u sin g certain syn tactic patterns or sp ecific phrases.
H o w ev e r, th ese patterns m ay still be p resent in older tech n ical d ocu m en ts, so they
sh ould n ot b e om itted from the evalu ation in the first part o f the study. T his leg a cy
co n ten t m ay n ot be as valuab le as n e w content, but it is w orth in vestigatin g becau se
it m a y still b e relevant for certain users.
T ech n ical con tent is subject to frequent updates. T hus, it w a s anticipated in the
d om ain o f th is study that there w o u ld b e a d iscrep an cy b etw een the textual content
o f the d ocu m en ts gathered for the d esig n o f the corpus, and the textu al content to be
m achin e-tran slated in the secon d part o f the study. B y the start o f the seco n d part o f
th is study, so m e o f the d ocu m en ts m ay h ave b een updated. S in ce the overall
o b jec tiv e o f th is study w as to u se real con tent for the evalu ation o f CL rules, the
updated v er sio n s o f th e d ocu m en ts w ere used.
1 . 3 . 1 .2 . R e p r e s e n t a t i v e n e s s o f t h e c o r p u s
O ne o f the o b jec tiv es o f this study is to fo c u s on the effe c tiv e n e ss o f ind ividu al CL
ru les. In order to be able to evalu ate their im p act e ffe c tiv e ly , v io la tio n s o f th ese
rules m u st be fou n d in the corpus.
I f the selected tex t ty p e is to o sp e c ific , certain CL ru les m a y not be violated .
T he te x t type ch o sen for the d esig n o f the corpus sh ould therefore not be too
s p e c ific so that gen u in e rule v io la tio n s can be found. A s m en tion ed b y O lohan
(2 0 0 4 : 4 5 ), the represen tativen ess o f the corpus affects the ‘d egree o f co n v ic tio n ’
w h en it co m es to m ak in g gen eralisation s based on the data an alysis. It is therefore
esse n tia l to ensure that:
21
“
T he ch osen tex t typ e is not sp e c ific to S ym an tec or the Internet
security industry. T he text type sh ould be co m m o n in the softw are
industry
*
T he syn tactic and textual features o f the con tent should not be
con sid ered as su b lan gu age characteristics. T h ey sh ould ideally be
ech oed in other ty p es o f d ocu m en tation so as to obtain a balanced
corpus
■
T he content h as b een authored b y a range o f d evelop ers, w h o m ay
or m ay n ot be n ative speakers o f E n glish . T his w ill h elp s us obtain a
varied list o f CL rule v io la tio n s
■
T he con ten t to b e in clu d ed in th e corpus sh ou ld contain tech nical
inform ation d estin ed for h om e users w h o n eed lo ca lised material
■
T he con tent sh ou ld b e p u b lish ed v ia th e W eb
■
T he u nd erlyin g con ten t repository sh ould already contain lo ca lised
m aterials to en su re the sea m le ss storage o f future m achine-translated
con tent
T h ese con sid eration s are esse n tia l to take into accou n t to ensure that the present
study acts as a gen u in e ca se study. I f certain C L rules p rove e ffec tiv e, th ey m ay be
reused in the future for sim ilar te x t typ es in the industry. L alaude et al. (1996: 112)
state that certain ru les are so m e tim es d o cu m en t-sp ecific, w h ile others do tend to be
gen eric and m ay b e ap p lied to d ifferen t typ es o f d ocu m en ts.
I . 3 . I . 3 . S iz e o f t h e c o r p u s
T he s iz e o f the corpus sh ould reflect the lin g u istic d iversity requirem ent, so that a
w id e range o f v io la tio n s o f CL ru les can be id en tified . H ow ever, the corpus m ust
n ot b e extrem ely large for th e fo llo w in g reasons. Firstly, it can be tim e-co n su m in g ,
ev e n for a sp ecia lised p ie c e o f softw are, to p rocess a large corpus.
T he exp erim en tal approach u sed to extract ex a m p les from the corpus w ill be
d escrib ed in greater d etail in Chapter 3. B ut it is essen tial to take into accoun t
hardware resou rces w h en d esig n in g the corpus. T he larger the corpus, the lon ger it
tak es to perform an a n a ly s is-o f so m e sort, e sp e c ia lly w h en th e o b jectiv e is to refine
that a n alysis o v er tim e. For that reason, a large corpus in the region o f a m illio n
w o rd s had to b e d iscou n ted .
22
M oreover, th e seco n d o b jectiv e o f this study is to evalu ate the im p act o f certain CL
rules w ith in an o n lin e exp erim en t u sin g m achine-translated m aterial. In order to
p u b lish this m aterial, source content can b e m an u ally ed ited so as to con form to the
CL rules id en tified in the first part o f th is study. D u e to tim e constraints, o n ly a
lim ited num ber o f d ocu m en ts w ill h ave to b e edited prior to their m achin e
translation. It is therefore n ecessary to h ave en o u g h d ocu m en ts to ch o o se from , but
n ot essen tial to h ave an exh au stive set o f d ocu m en ts.
F in ally, adding m ore d ocu m en ts or parts o f d ocu m en ts to a co ip u s w h en th ose
d ocu m en ts b e lo n g to the sam e te x t type and dom ain can lead to a ‘con ten t b ottle­
n e c k ’. E v en th ou gh th e w ord count and the s iz e o f the corpus increase, there is little
n e w data. B o w k er & P earson (2002: 4 8 ) therefore w rite that ‘it is generally
accep ted that corpora intended for L angu age for S p e cific Purpose stu d ies can be
sm aller than th o se u sed for L angu age for G eneral Purpose stu d ies (..). In our
ex p erien ce, w e ll-d e sig n e d corpora that are an yw h ere from about ten thousand to
several hundreds o f th ou sand s o f w ords in siz e h ave p roved to be ex c ep tio n a lly
u sefu l in L SP stu d ie s’. B a sed on th ese g u id elin es, it w a s d ecid ed to lo o k for a text
ty p e that w o u ld be ab le to prod uce a corpus s iz e o f ap proxim ately
2 0 0 ,0 0 0
1 .3 .2 .
S e le c tin g t h e id e a l t e x t ty p e f o r a c a s e s tu d y
At
start o f th e
the
study,
in
2004,
S y m a n tec’s tech n ical
con tent
w ords.
focu sed
p red om in antly o n secu rity ap plication s. A s so m e d ocu m en ts m ay be sem an tically
related, but h a v e d ifferen t com m u n ica tiv e p urp oses, and in turn different syntactic
d istributions, it w a s essen tia l to re v ie w several te x t ty p es to se e w hether th ey m et
th e corpus requirem ents. T h is q uick r e v ie w w a s essen tia l to selec t a tex t typ e that
w a s n ot s p e c ific to S ym an tec or to th e Internet security industry, but rather,
co m m o n in the softw are industry. I f th e te x t typ e w e re sp ecific to S ym an tec, it
w o u ld b e d ifficu lt to con d u ct a relevant ca se study. For instance, tech n ical articles
fo c u sin g o n the d escrip tion o f security threats are sp ecific to the Internet security
industry. B e sid e s, th e rep etitiven ess o f their con tent w o u ld m ak e it d ifficu lt to find
g en u in e ex a m p les o f v io la tio n s o f CL rules. T h is tex t type u ses a lan gu age that can
be d escrib ed as a su b lan gu age due to the c lo se d nature o f the syn tactic structures
b ein g u sed and th e large am ount o f non-translatable elem en ts. S in ce the ob jectiv e o f
23
this study is to evalu ate the e ffe c tiv e n e s s o f as m an y CL ru les as p o ssib le, a m ore
gen eric tex t typ e w a s required. S p ecia l attention w a s paid to tech n ical support
docu m en tation sin ce th is typ e o f te x t is very co m m o n in the softw are industry.
1 . 3 . 2 .1 . T e c h n i c a l s u p p o r t d o c u m e n t a t i o n
W eb -b ased tech n ical support d ocu m en tation is o ften u sed b y softw are com p an ies as
an after-sales serv ice p rovided to users ex p erien cin g tech n ical p rob lem s w ith a
particular softw are ap plication . M a r lo w (1 9 9 5 : 4 4 ) states that support d ocum ents
'provide additional task-related in form ation n ot oth erw ise covered in the user
m anual to h elp users so lv e particular problem s'. T he purpose o f th ese d ocu m en ts is
to reduce cu stom er con tacts b y o fferin g on lin e tech n ical con ten t so that global
cu stom ers can find an sw ers to their q u estion s or re so lv e their p rob lem s by
th em se lv es. A c co rd in g to Freem an (2 0 0 6 : 4 ), th is type o f con ten t is 'cost-reducing
co n ten t 1 b eca u se it h elp s redu ce cu stom er serv ice co sts. From a fin ancial p ersp ective,
users m ay also b en efit from th is typ e o f con tent, e sp e c ia lly i f th ey h ave to pay for
support serv ices. M arlow (20 0 5 : 5) b e lie v e s that 'users w ou ld prefer n ot to contact
te ch n ica l support tech n ician s, and th is is e sp e c ia lly true w h en support system s are
chargeable'. H o w ev er, sp ecific inform ation p ertaining to W eb -b ased tech nical
support d ocu m en tation is scarce in th e literature on tech n ical d ocu m en tation , w h ich
o ften fo c u se s o n user's gu id es and o n lin e h elp sy stem s (Price and Korman: 1993;
M arlow : 1995; G erson and G erson: 2 0 0 0 ; R am an and Sharma: 2 0 0 4 ). T his type o f
inform ation is eq u a lly scarce in th e literature related to o n lin e con tent d evelop m en t
(Spyridakis: 2 0 0 0 ; H am m erich and H arrison: 2 0 0 2 ).
In a traditional m od el, th ese tech n ica l support d ocu m en ts are stored in k n o w led g e
b a ses, w h ich are sp ecia l databases for k n o w le d g e m an agem en t u sin g a w ell-d efin ed
cla ssifica tio n structure. T he d ocu m en ts are th en p u b lish ed v ia the W eb so that they
can b e e a sily a cc essed b y users. U sers h ave then tw o w a y s to a cc ess th ese
d ocu m en ts. T h ey m a y fo llo w a link present in som e o f th e softw are application's
d ia lo g b o x es so as to be redirected to an o n lin e docu m en t. T h ey can also go to the
co m p a n y ’s W eb site and search the k n o w le d g e b ase on lin e. T he m eth od u sed w ill
dep en d on the ty p e o f p rob lem en cou n tered b y users. L et u s take the exam p le o f a
user ex p erien cin g an error m e ssa g e w h en trying to in stall a product. Such an error
m e ssa g e m ay b e due to a gen u in e bug in the ap plication , or to a m istak e m ade by
24
the user during the in stallation p rocess. In the secon d scenario, a d ocum ent
ex p la in in g h o w to avoid this issu e m ay h ave b een p ub lished on lin e i f the issu e w as
exp erien ced b y other cu stom ers. T he error m essage's d ia lo g b o x seen by the user
m a y th en contain a h yperlink referring to this on lin e tech nical docum ent.
Certain d ocu m en ts are also authored to sh o w users h o w to perform com m on tasks
w ith
their
ap plication s,
rather
than
to
fix
a
tech nical
problem .
From
a
co m m u n ica tio n p ersp ective, the authors o f enterprise tech n ical support d ocum ents
address sp ecia lists, w h ereas con su m er tech n ical support d ocu m en ts p rovide an
en viron m en t w h ere sp ecia lists ad vise lay persons. E ven th ou gh certain h om e users
m a y b e com puter literate, it is m o st lik e ly that p eo p le w h o n eed to read th ose
d o cu m en ts are p eo p le w h o cannot address the initial problem . T he im portance o f
lo c a lise d con su m er m aterial is therefore crucial to ensure the a ccessib ility o f the
con ten t to all o f the users. It is h y p o th esised that m o st h om e users w ith a b asic
k n o w le d g e o f E n g lish w ou ld n ot be able to s o lv e a p rob lem by reading instructions
in a lan gu age th ey barely understand. A s m o st o f tech n ical support con tent is
p rescriptive, m issin g or m isinterp reting a step can h ave n eg a tiv e con seq u en ces. T his
fact a lo n e see m s to ju stify th e translation o f con su m er tech n ical support content into
v ariou s target lan gu ages. R ega rd less o f the reasons for n ot fu lly lo ca lisin g W eb
co n ten t (tim e, cost, lack o f resou rces), the con seq u en ces o f h avin g W eb con tent that
is o n ly partly lo c a lise d sh ou ld not b e underestim ated. T he ab sen ce o f W eb
lo ca lisa tio n can b e com pared to the ab sen ce o f lo ca lised tech n ical docum entation.
H oft (1 9 9 5 : 2 0 5 ) b e lie v e s that ‘n on -n ative readers o f the source langu age can
b eco m e d issa tisfied w ith a tech n ical m anual in th e sou rce lan gu age and project this
d issa tisfa ctio n on to th e co m p a n y .’
T h is ch a llen g e has a lso b een p oin ted ou t b y V an der M eer (20 0 3 : 181):
It is no longer su fficient to translate the product docum entation
and lo ca lise the softw are, all enterprise information must be
adapted to the global m arketplace in w hich the com pany wants to
be seen as market leader. ( . . . ) Current users w ill buy online, get
support and get training online.
A s a forem en tion ed , translating every p u b lish ed tech n ical support d ocu m en t is a
c h a llen g e for large softw are co m p a n ies, in clu d in g S ym an tec, due to the size o f their
25
k n o w le d g e b ases. T he am ount o f d iv erse con tent th ey con tain appears a g ood
starting p lace to d esig n a corpus.
A s far as th is study is con cern ed , it is im portant that th e lin gu istic co v era g e o f this
typ e o f con tent corresp on ds to the represen tativen ess criterion w h ich w as d efin ed
earlier. It m a y b e argued that S ym an tec con su m er tech n ical support con tent o n ly
ap p lies to S ym an tec con su m er products and users. A s S ym an tec con su m er products
are m o stly W in d o w s-b a sed , th e d om ain is also reduced. B u t this study fo c u se s on
th e typ es o f lin g u istic structures u sed in a particular te x t typ e rather than on its
lex ica l com p osition . T he fin d in gs, w h ich w ill be m ad e based on a particular sam ple
o f S y m a n tec -sp e cific con su m er tech n ical d ocu m en tation , m ay also apply to other
tech n ical
support
d o cu m en ts
w ith in
the
softw are
industry.
E very
softw are
p u b lish in g com p an y p ro v id es tech n ical support to their cu stom ers, regard less o f the
typ e o f product th ey m anufacture. It is u n lik ely that a slig h tly different sem antic
field w o u ld u nderm ine th e fin d in gs o f this study b ased o n a S y m a n tec-sp ecific
corpus. O f cou rse, th ese cla im s m u st b e ju stified in greater detail by exam in in g
S ym an tec con su m er tech n ica l support d ocu m en tation . T his w ill h elp u s prod uce a
m a cro -lev el
d escrip tion
of
this
tex t
typ e
and
determ ine
w h eth er
certain
characteristics o f tech n ica l support d ocu m en tation can b e fou n d in other te x t types.
T he n ex t sec tio n fo c u se s on the textu al characteristics o f th is tex t type from a user's
p o in t o f v ie w .
1 .3 .2 .2 . C h a r a c t e r is tic s o f a t e c h n ic a l s u p p o r t d o c u m e n t: t h e
u s e r ’s p e r s p e c t i v e
B a sed on th e c la ssific a tio n o f m acro te x t ty p es in sc ien ce and tech n o lo g y g iv en by
S ager (1 9 9 4 : 8 5 ), tech n ica l support d ocu m en ts see m to fit in the ‘m e m o ’ category,
b eca u se th ey are ‘con cern ed w ith th e im m ed iate interaction o f participants and
to p ic s ’. U sers n eed to co n su lt tech n ical support d ocu m en tation w h en th ey h ave
tech n ica l p rob lem s or q u estio n s that rem ain u nan sw ered . A s m en tion ed p reviou sly,
th ey m ay g et direct a c c e s s to a particular article b y click in g on a h yperlink from
w ith in their ap plication . T h ey m ay also get a cc ess to a solu tion by g oin g o n lin e and
search in g th e tech n ica l su pp ort’s k n o w le d g e base. In the seco n d scenario, th ey can
v ie w the m o st freq u en tly a c c e sse d d ocu m en ts after selec tin g search criteria su ch as
product n am e and v e r sio n n am e. E nd-users m ay a lso enter a k eyw ord corresponding
26
to their query to retrieve a list o f search hits, from w h ich th ey can selec t the
d o cu m en t that b est corresp on ds to their n eed s. A ty p ica l tech nical support d ocum ent
as v ie w e d in a W eb b row ser is sh o w n in Figure 1-1 b elo w .
^
s u p p o rt
Symantec*
:;me & hunts oHica&maH business
un ite d states
rate this document
global sites
p rodocta and
ices
Docum ent 10:2 0 0 4 0 9 2 8 1 4 5 1 4 4 0 6
Last M o d ifie d :09/05¿2005
!s document
purchase
support.
security response
dow nloads
about symarttec
After clicking Install Norton Antivirus, a cursor appears briefly, and Norton
Antivirus 2005 stops responding
S itu a tio n :
After inserting your Norton A n tiviru s 2005 CD, you see the first installation screen W hen you click Insla
Norton A n tiviru s, the installation does not run An hourglass cursor appears briefly, but no other window
appears
search
feedback
S o lu t io n :
<© 1 9 9 5 -2 0 0 5 S y m a n te c
C o rp o ra tio n ,
A ll rig hts reserved.
L e g a l N otices
P riv a c y P o lic y
B e fo re y o u b e g in : The information in th is document applies to a problem that happens whan installing
Norton A n tiviru s from the installation CD If you downloaded Norton A n tiviru s and are hatvrng a sim ilar
problem installing the program, read Norton A n tiv iru s 2005 installation stoos rescondmn aftei eatraclinn
downloaded files
Because th is situation has a variety of causes, no one solution will work in every case Begin with
"Solution 1 : Make sure that your com puter is virus free " You will be directed as to what to do next.
Solution 1 : Make sure that your computer is virus free
Figure 1-1: Technical support document viewed in a Web browser
A
typ ical
W eb -b ased
S ym an tec
con su m er tech n ica l
support d ocu m en t is
a
d ocu m en t that can b e d isp layed in a traditional W eb brow ser. I f a tex t-o n ly W eb
b row ser is u sed , the d ocu m en t m a y n ot b e co m p lete b ecau se o f the p o ssib le
p resen ce o f n o n -tex t elem en ts su ch as graphics.
1.3.2.2.1. Linguistic elem ents o f graphics
S o m e o f the grap h ics p resent in tech n ical support d ocu m en ts are 'screenshots',
sh o w in g sp e c ific parts o f an en viron m en t in w h ich som eth in g happens or n eed s to
b e done. T he term ‘en v iro n m en t’ refers here to th e G raphical U ser Interface (G U I)
o f a program or a set o f program s. T he u se o f screen sh ots estab lish es a strong
co n n ectio n b e tw e e n tech n ical support d ocu m en ts and softw are user gu id es, the
latter con tain in g sec tio n s that sh o w users h o w to u tilise a particular p ie c e o f
softw are. B yrne (2 0 0 4 :3 ) d efin es user gu id es as 'the interface b etw een com puter
sy stem s and hum an users'. In user gu id es, sec tio n s are often illustrated w ith
27
screen sh ots w h o se purpose is to gu id e users in step -b y-step situ ations, su ch as
a ctivatin g
ap plication .
a particular fu n ction , m o d ify in g
certain settin gs,
S in ce screen sh ots so m e tim es perform
or rem ovin g
an
the sam e fu n ction as text
instru ction s, on e m ay w on d er w h y o n e is used instead o f the other, or w h y both are
so m e tim es u sed together. E lem en ts o f answ ers to th is q u estion m ay b e found in a
study (Fukuoka et al., 1 9 9 9 ) w h ich fou n d that A m erican and Japanese users b e lie v e
that m ore graphics, rather than few er, m ak e instructions easier to fo llo w . T his study
a lso revealed that users prefer a co m b in a tio n o f te x t and graphics, w h ich th ey
b e lie v e w o u ld b e m ore e ffe c tiv e than te x t-o n ly instructions.
F rom a sem io tic p ersp ectiv e, screen sh ots p lay an ic o n ic role (D irv en & V erspoor,
1998: 4 ), b ecau se th ey p rovid e users w ith a rep lication o f th e environm ent w ith
w h ic h th ey are interacting. S creen sh ots m ay a lso p rovid e an illustration o f so m e o f
the steps users should fo llo w to fix a problem . T ech n ical support screen sh ots can
so m e tim es be ed ited b y con ten t d ev elo p ers to p rovid e extra inform ation to users.
Inform ation can b e added u sin g te x t or graphical d raw ings, su ch as arrows or circles,
to draw th e attention o f th e user to a certain part o f th e replicated G U I. T his is
e x e m p lifie d by F igu re 1-2:
28
W indow s 95/98/M e/NT/2000
1
2.
3.
Cliquez sur D é m a rier > R e c h e ic lie r ou Trouver > Fichiers ou dossiers
Assurez-vous que (C:J soit bien sélectionné dans la boîte de dialogue Rechercher dans et que
l'option Inclure les sous-dossiers ait également été choisie.
Dans le champ “ Nommé" ou "R echercher...", sa isisse z-o u copiez/collez- les noms de
fichiers suivants :
dow nloads
Cliquez sur Trouver m ain ten ant ou R echercher m aintenant. Les résultats sont affichés
dans le volet de droite ou dans la partie inférieure de la fenêtre. Il peut y en avoir plus d'un. Le
dossier que vous recherchez est celui dont le chemin d'accès finit par \LiveUpdate.
Name
L_l Downloads
f*~l
D o iA in lo a tk < r ~
6 f downloads.gif
Downloads
5
6,
7.
In Folder
r-'iQ fu-unip r'^
CADocumenls and SettingsVAII Users^Applicatiori Data\Symantec\LiveU pdale.
C:\PragMm l-n e s w o lllT llliy \B tjC
C:\Documents and Settings\Adrninistrator\Recent
Cliquez deux fois sur ce dossier.
Supprimez tous les fichiers ou dossiers contenus dans le dossier \Live U p d at e\D owri Io a d s .
Dans la plupart des cas, il s'agira de Autoupdt.trg et Livetri zip. Un ou plusieurs dossiers
peuvent également s'y trouver. Supprimez-les tous
Exécutez LiveUpdate de nouveau. Si le problème persiste, passez à la section intitulée "Un
fichier Livetri zip corrompu".
Figure 1-2: Edited screenshot
T h ese elem en ts are ex a m p les o f an in d ex in g p rin cip le (D irven & V erspoor, 1998: 5)
b eca u se th ey draw the attention o f th e user to a particular action that should be
perform ed, or to the result o f an action. T his a llo w s users to iso la te the com pon en t
o f the G U I w h ic h requires action . A s a result o f th is q uick link b etw e en form and
m ean in g, screen sh ots m ay rep lace procedural sen ten ces con tain ing instructions to
fin d the lo ca tio n o f a graphical item , b e it a button, a tab, a pane, a w in d o w , a m enu
bar, a m en u item , or a radio button. T h ese elem en ts m ay also h ave an icon ic
fu n ctio n b y rep la cin g th e action that th e u ser should perform on on e o f th ese item s:
to click , to ch eck , to u nch eck , or to enter a w ord.
From a m u ltilin g u a l co m m u n ica tiv e p ersp ectiv e, th o se screen sh ots sh ou ld o f course
be in the lan gu age o f the users so that their prim ary ico n ic fu n ction can be fu lly
perform ed. H o w ev e r, th is is n ot a lw a y s p o ssib le , b ecau se third-party E nglish
ap p lication s are n ot alw a y s lo ca lised . A d ocu m en t that is w ritten in French but
w h ic h con tain s n o n -lo c a lise d screen sh ots, w h ere the langu age u sed in the U I is
E n g lish , is th erefore boun d to b a ffle French u sers, as sh ow n b y F igure 1-3:
29
Poui configurer NetZip 7.5.1 afin que LiveUp<l*ite puisse reconnaître les fichiers .zip comm e
étant des fichiers et non des tlossieis
1.
2
Lancez NetZip.
Cliquez sur Outils puis sur Options
3.
Cliquez sur I'onglet Dossiers NetZip
Figure 1-3: Non-localised screenshot in a localised document
T he hand ling o f screen sh ots is therefore a co m p lex lo ca lisa tio n p rocess, w h ich is
b ey o n d the cap ab ilities o f an M T system . A s sh o w n b y the p reviou s exam p le, it is
so m etim es d ifficu lt or im p o ssib le for a hum an translator to find the corresp on din g
screen sh ot in h is or her o w n lan gu age. T he tim e required to perform such a search
during the translation p ro cess sh ould therefore n ot b e underestim ated. T his is not
th e o n ly draw back o f screen sh ots w h en th ey are in clu d ed in tech n ical support
d ocu m en ts. S creen sh ots can also create a cc essib ility issu e s for users w ith ey e sig h trelated d isab ilities. I f screen sh ots are n ot accom p an ied b y alternative text, th ey m ay
b e ignored b y a c c e ssib ility to o ls such as screen narrators. F igu res 1-4 and 1-5 sh o w
an excerp t o f a tech n ica l support d ocu m en t w ith a screen sh ot, and the und erlyin g
H T M L co d e from w h ich the alternative tex t string is m issin g:
30
4
5.
Klicken Sie im Registrierungseditor auf den Schlüssel HKEY_LOCAL_MACHINE.
Klicken Sie im Menü Sicherheit auf Beiechtigunge.
Berechtigungen für IIKEY_l.OtAL_M ACHINE
J jx j
Sicherheitseinstellungen
Hinzufügen...
Name
Administratoren (ADMIN-HATSM7FEVAA...
ß ü EINGESCHRÄNKTER ZUGRIFF
_ Entfernen^
f S Jeder
SYSTEM
Berechtigungen:
Zulassen
Verweigern
□
Lesen
Vollzugriff
n .
Erweitert...
p
Vererbbare übergeordnete Berechtigungen übernehmen
OK
Abbrechen
I
Übernehmen
J
Figure 1-4: Localised screenshot in a technical support document
<li>Klicken Sie im Regi stri erungsedi tor auf den Schlüssel HKEY_LOCAI MACHINE.
<li>Klicken Sie im Menü <b>Si cherhei t</b> auf <b>Berechtigunge</b>. <br>
<br>
ci mg
src="/SUPPORT/lNTER/navintl.nsf/bflfb726f6e7892985256ef50048caf2/0d312f63245654a5B0256d6400396«
d2/Soluti on/0. lECS?OpenElement&amp; Fi eldElemFormat=gi f" wi dth="370" hei ght="439" al t='‘"><br>
<br>
Figure 1-5: Missing alternative text in HTML code
B e sid e s, a screen sh ot m ay im p act o n th e reliab ility o f a d ocu m en t over tim e, or at
lea st b a ffle users ru m iing old er v ersion s o f th e product for w h ich the d ocu m en t w as
o rig in ally intended. T h e ic o n ic fu n ction o f screen sh ots w a s m en tion ed b eca u se o f
th e link it created w ith a g iv e n G U I at a certain p oin t in tim e. H o w ev er, that link
m a y w e ll b e altered or ev en broken i f the G U I ch a n g ed over tim e. For instance, a
d o cu m en t ap p lies to several v ersio n s o f an operating sy stem w h en the te x t u sed in
th e d ocu m en t d o es n ot fo cu s o n any particular v ersion . I f a screen sh ot is introduced,
th e d ocu m en t m a y b e p erc eiv ed as v e r sio n -sp e c ific b y certain users. I f the
31
screen sh ot d oes n ot ex a ctly m atch their en viron m en t, certain users m ay be tem pted
to think that the d ocu m en t d o es not apply to them .
To so m e exten t, th is also ap plies to v id e o clip s that are so m etim es linked to
tech n ical support d ocu m en ts. T h ese v id e o c lip s contain step -b y-step tutorials
d esig n ed to h elp u sers fin d an an sw er to their question. From a localisation
p ersp ective, th is typ e o f elem en t is ev e n m ore co m p lex than static screenshots,
w h ile h avin g the sam e pragm atic fu n ction as p lain text. Plain text is, o f course, the
m o st com m on elem en t am on g the textu al elem en ts that are contained in tech nical
support docum ents.
1.3.2.2.2. Textual content
T he screen sh ot in F igu re 1-1 ou tlin ed the typ ical structure o f S y m a n tec’s tech n ical
support d ocu m en ts, w h ich are b ased o n a tem p late. Certain section s are com pu lsory,
but the tex t th ey con tain is n ot subject to any con trol3. A ll d ocu m en ts co n sist o f a
title, w h ich su m m arises the tech n ical issu e or q u estion encountered b y the user, and
o f a b o d y o f te x t con tain in g several section s. T itles are d irectly fo llo w e d b y a
‘S itu ation ’ sec tio n
w h ic h
exp an ds
on
the
inform ation
present
in
the title.
T he ‘S itu ation ’ se c tio n estab lish es an im m ed iate o n e-to -o n e relationsh ip b etw een
the w riter and th e user, as sh o w n b y th e frequent u se o f the seco n d p erson personal
pronoun ‘Y o u ’. B y reading th is section , users sh ou ld then b e able to determ ine
w h eth er th ey are reading the right d ocu m en t. T he situation that is described in th ese
first introductory lin es sh ould be exactly th e sam e as th e on e th ey w ere exp erien cin g
b efore a c c e ssin g the d ocu m en t. The 'Situation' sec tio n is p ivotal to the resolu tion o f
u sers’ p rob lem s, b eca u se the su bsequ en t sec tio n s o f the d ocu m en t are based on the
a ssu m p tion that users are exp erien cin g a sp e c ific prob lem in a sp ecific con text w ith
a sp e c ific feature o f a sp ecific application.
3 In 2005, Symantec decided to deploy an application with CL checking capabilities (acrocheck ™)
to improve the consistency, readability, and machine-translatability o f its new technical support
documents. The selection o f the corpus was earned out prior to this deployment.
32
T he ‘S o lu tio n ’ sec tio n is also alw a y s p resent in S y m a n tec’s tech n ical support
d ocu m en ts. It con tain s all the inform ation n ecessa ry to an sw er a q uestion, or so lv e a
problem . A s sh o w n in F igure 1-2, it m ay con tain a ‘B efo r e y o u b e g in ’ seg m en t to
rem ind users that the solu tion s described in th e sectio n relate to a sp ecific problem .
T he rest o f the ‘S o lu tio n ’ sectio n con tain s a list o f steps that users m ust fo llo w to
r eso lv e their p rob lem s. T he nature o f th is sec tio n is procedural sin ce the steps users
h ave to fo llo w are often presented u sin g num bered lists. Price and K on n an (1993:
2 2 7 ) state that the aim o f su ch a procedure is 'to c a n y out a sm a ll-sca le task ’. T his is
a ch ieved b y en su ring that 'each step sp e c ifie s an action readers take on th e w ay
tow ard a cc o m p lish in g their goal' (ibid: 2 3 3 ). M o st o f th e list item s therefore start
w ith im p erative verbs. In certain ca se s, series o f P reposition al Phrases (PP) are u sed
b efore th e im p erative verb, as in: 'In the m ain w in d o w , on the left side, click N orton
A ntiSpam '.
S y m a n tec’s te ch n ica l support d ocu m en ts m a y also contain tw o other typ es o f
sections: a ‘T ech n ica l In form ation ’ sectio n and a ‘R e fere n c e’ section . T hese
d escrip tive sec tio n s are not alw ays essen tia l for so lv in g the issu e d iscu ssed in the
docum ent. Instead, th ey p rovid e a sta llin g p o in t for users w h o w ant to learn h o w to
avoid future issu e s. For instance, the ‘re fer en ce’ sectio n m ay p oin t to internal
tech n ical d ocu m en ts w ith in th e sam e k n o w le d g e b ase, but it m ay also refer to thirdparty m aterial. T h e se link s m ay b e seen as in stan ces o f intertextuality b ecau se
con tent d ev elo p ers assu m e that a certain tech n ica l issu e w ill b eco m e clearer i f users
b eco m e fam ilial' w ith the subject field . T he n ex t sec tio n o f this chapter fo c u se s on
the textu al elem en ts that facilitate intertextuality: hyperlinks.
1.3.2.2.3. Intertextuality: the role o f hyperlinks
A typ ical textu al feature o f tech n ical support d ocu m en ts lie s in their use o f
h yperlink s to refer u sers to background articles or to related articles. T h o se related
articles m ay p ro v e in valu ab le in the resolu tion o f a prob lem , e sp ec ia lly i f users
realise that th ey h a v e b een lo o k in g at the w ro n g docum ent. H yperlinks deconstruct
the traditional m o n o lith ic structure o f a d ocu m en t, by offerin g users th e p o ssib ility
to get a cc ess to in form ation from several d ocu m en ts. A s lon g as users can find all
the inform ation th ey require, it d o es not m atter w h eth er th ey have read so m e or all
o f on e or m ore d ocu m en ts. W hat is, h o w ev er, crucial in this approach, is to have
33
hyperlink s referring to target con tent w h ich is in the sam e language as the original
content. L egitim ate frustration can so m etim es em anate w h en the lo ca lisa tio n chain
is broken in th e m id d le o f a quest for inform ation. T his asp ect o f technical
d ocu m en ts w ill be borne in m ind in the seco n d part o f our study, to ensure that
p ub lish ed d ocu m en ts can eith er stand on their o w n or refer to lo ca lised docum ents.
N o w that the m ain textu al characteristics o f tech n ica l support d ocu m en ts have been
h ig h ligh ted from th e u ser’s p oin t o f v ie w , it is p o ssib le to proceed to a d etailed
re v ie w o f th e con tent from th e d ev elo p er’s p ersp ective.
1 .3 .2 .3 . C h a r a c t e r i s t i c s o f t e c h n i c a l s u p p o r t c o n t e n t : t h e
d e v e l o p e r ’s p e r s p e c t i v e
1 .3 .2 .3 .1 . Technical support knowledge base
T hus far, the fo c u s h as b een p laced on tech n ica l support d ocum ents from a user's
p ersp ective, w ith little fo c u s on the situations that arise w h en users n eed to con su lt a
tech n ical support d ocu m en t. E nd u sers’ typ ical n eed s can be an alysed b y exam in in g
th e cla ssifica tio n u sed to store tech n ical support articles in S y m a n tec’s k n o w led g e
b ase. Sager (1994:
101) states that the ex ch a n g e o f k n o w led g e is ‘th e b asic
co n d ition for su c c e ss in n on -p h atic co m m u n ica tio n ’. W h en d evelop ers add n e w
con ten t to the k n o w le d g e b ase, th ey m u st h ave g o o d reason to do so, w h ich is often
to bridge the k n o w le d g e gap b etw e en th em se lv es and the users. T he categories used
b y d evelop ers to la b el d ocu m en ts are therefore a g o o d indicator o f w h at is u sually
requ ested b y the u sers.
1 .3 .2 .3 .2 . Classification o f knowledge
W h en a d ocu m en t p ro v id es a so lu tio n for users exp erien cin g a general tech nical
p rob lem , the article is c la ssifie d into su b -categories b ased on the issu e encountered:
w h eth er it is a co m p a tib ility issu e, a third-party issu e, or a fu n ction ality issu e. The
particular feature o f the ap p lication in w h ich the p rob lem occurs can a lso b e u sed as
a filter. A d ocu m en t can th u s b e c la ssified in several categories, so as to m a x im ise
ch a n ces o f retrieval w h en th e inform ation is required. S uch d ocu m en ts are w ritten
to p rovid e an an sw er to an op en q u estion introduced b y an interrogative pronoun
su ch as ‘w h y ’, ‘h o w ’, ‘w h a t’, ‘w h ic h ’ or ‘w h e r e ’. A ccord in g to Price and K orm an
(1 9 9 3 : 3 2 0 ), su ch d o cu m en ts sh ould be lim ited to on e 'unit o f inform ation w h ich is
th e topic'. For in stan ce, a u ser m ay n ot be ab le to u se a sp ecific feature o f the
34
program and w o u ld lik e to k n ow the reasons w h y. T his type o f d ocu m en t p rovides
an an sw er a p osteriori to h elp users r e so lv e a p rob lem that has already occurred.
S p e c ific tech n ical p rob lem s are o ften cla ssifie d in an ‘Error M e s s a g e ’ category,
b eca u se the error m e ssa g e num ber a llo w s for q uick and ea sy id en tification o f the
problem .
In other ca se s, users m ay be lo o k in g for m ore inform ation prior to exp erien cin g a
p roblem . T h ey m ay h ave searched to no avail in th e o n lin e h elp or in the user
m anual. In th is scenario, the typ e o f q u estion asked is a clo sed q u estion , starting
w ith a m od al verb or an auxiliary. For ex a m p le, users m ay w on d er w hether it is
p o ssib le to install tw o versio n s o f a product on th e sam e m achin e, or w h eth er a n ew
product is capable o f p erform in g a sp e c ific task. T he d ocu m en ts that contain
an sw ers to su ch q u estion s are, unsu rp risingly, c la ssified as Frequently A sk ed
Q u estion s (F A Q ) d ocu m en ts.
T ech n ica l support d ocu m en ts so m e tim es require both ty p es o f con ten t - tech nical
instructions and F A Q s - esp e c ia lly w h en th e initial q u estion has m ore than one
so lu tion . In th o se ca se s, d ocu m en ts tend to p oin t to other d ocu m en ts rather than
in clu d in g all solu tion s w ith in th e sam e docum ent. T h is separation o f content into
reusab le inform ation chunks is a clear ex a m p le o f the reuse p h ilo so p h y that certain
co m p a n ies h a v e adopted. A t C aterpillar, for instance, th ese chunks w ere called
Inform ation E lem en ts (H ayes et al., 1 9 9 6 ). C ontent d evelop ers are increasin gly
en cou raged to u se an ‘author on ce, p u b lish m an y tim e s ’ approach. T his approach is
particularly v isib le for the d ev elo p m en t o f user m an u als and o n lin e h elp , w h ereb y
the em p h asis is p la ced on sec tio n s o f texts, also k n ow n as to p ics, rather than on
actual d ocu m en ts. D e v elo p e rs are en cou raged to w rite se lf-su ffic ie n t chunks o f text
that can b e reused w h e n n eed s b e, regard less o f the output form at. T he te ch n o lo g y
fa cilitatin g th is approach often relies on a C ontent M an agem en t S y stem (C M S )
b ack -en d , w h ich stores th ese units o f inform ation.
W h en a k n o w le d g e b ase is lin k ed to a C M S , the con tent th en b eco m es availab le for
p u b lica tio n w ith in oth er typ es o f docu m en tation . T his m eans that con tent w ritten for
a tech n ica l support d ocu m en t m ay b e reused in a future m anual or on lin e help.
T his leverage opportunity o ffered b y a C M S is op tional, but sh o w s that technical
35
support content m a y find its w a y into other ty p es o f d ocum entation, w h ich fu lfils
th e g en eralisation requirem ent ou tlined earlier. From a translation p ersp ective, the
translation requirem ent rests on the con tent rather than on the final tech n ical support
docum ent. B efo r e d ealin g w ith the con tent, it is n ecessary to understand the
so m etim es c o m p lex relationsh ips b etw e en tech n ical support content, docum ent,
k n o w le d g e b ase, and C M S. F igure 1-6, o u tlin es th ese relationships:
Inform ation Tier
Middle Tier
Client Tier
Content
authored by
developers
Dynamic HTML
document
_5L
w
Knowledge base
Web server
A
A
—
Content
Management
System
■4------------
Web browser
1
1
1
M --------- — .* »
.
».Í
Figure 1-6: Information workflow in a Web-based architecture
T h is diagram in d icates h o w tech nical support con ten t m a y b e d irectly authored in a
k n o w le d g e b ase, and then p ub lished d y n a m ica lly in H T M L b y u sin g C ascad in g
S ty le S h eets (C S S ). W hen users request a particular tech n ical d ocu m en t on lin e, the
W eb server outputs the corresp on din g W eb p a g e based on the latest v er sio n o f the
content. W h en u p d ates are required, d ev elo p ers th en m o d ify th e database content,
w ith ou t a ffectin g th e H T M L display. T his ex p la in s w h y the sam e d ocu m en t m ay
lo o k slig h tly d ifferen t from on e v isit to th e n ext. Ideally, ch an ges m ad e to ex istin g
con tent are tracked w ith in th e database or b y an underlying C M S, b eca u se source
con ten t u pdates m u st b e translated in all target lan gu ages. In sec tio n 1.3.3, the focu s
is p la ced o n th e extraction o f con tent from variou s k n o w le d g e b ases in order to
c o m p ile th e corpus.
36
1.3 .3 * C o m p i l i n g t h e c o r p u s
l . 3 .3 .1 . S e le c tin g p o p u la r c o n t e n t
A s d isc u sse d in sec tio n
d ocu m en ts
d estin ed
1.3.1, it is n ecessary to fo cu s on tech n ical support
for h om e u sers,
due to
the assu m p tion that translated
d ocu m en tation is m ore relevant for h o m e users than for sy stem adm inistrators.
H o w ev er, th e range o f S ym an tec products d estin ed for su ch users is quite large,
w ith an estim ated cu m u lative cu stom er base o f 4 m illio n French and G erm an users4.
A t th e start o f this study, ap proxim ately 3 0 different con su m er products w ere
a v a ilab le from the se le c tio n list on the tech n ical support W eb site. In order to m eet
the d esig n requirem ents outlined earlier, the m o st popular products w ere selected to
con centrate e x c lu siv e ly on their resp ectiv e tech n ical support k n o w le d g e bases. In
term s o f corpus size, the o b jectiv e o f
several
products.
T his
d e cisio n
2 0 0 ,0 0 0
presented
w ords m ade it n ecessary to target
a
fe w
ad vantages
from
a
rep resen tativeness p oin t o f v iew . It m ean t that th e content extracted w as authored
b y several con tent d evelop ers. A fter ch ec k in g the n am es o f the authors o f various
a rticles in th e k n o w le d g e b a ses th e m se lv e s, it turned out that the articles related to
N o rto n A n tiV iru s™ , N orton Internet S ecu rity™ and N orton S ystem W ork s™ had
been authored b y at lea st 15 d ifferen t con ten t d evelop ers. T he risk o f u sin g data
resultin g from authors’ id io sy n cra sies w a s partly avoided. T hou gh there m ay seem
to be a large num ber o f authors, th e tech n ical support con tent d ev elo p m en t took
p la ce over 7 years. T h is 7-year tim efram e is also an advantage w ith regard to the
rep resen tativen ess o f th e corpus.
T here w ere other reason s for selec tin g th ese three S ym an tec products, on e reason
b ein g the p opularity o f S y m a n tec’s N o rto n A n tiV iru s™ product. A study con d ucted
b y th e 1DC, a global provider o f m arket in tellig e n c e for the inform ation te ch n o lo g y
and teleco m m u n ica tio n s industries, sh o w ed that S ym an tec h eld a 40.4% share o f
th e w o rld w id e antivirus softw are m arket (B urke, 2004: 5). In its sum m ary report,
B urke p rovid es the fo llo w in g d efin itio n for antivirus softw are:
4 Information received from Hesham Abuelata, Senior Director, Global Programs (Symantec
Consumer Marketing group) on August 8th, 2006.
37
A ntivirus softw are scans hard drives, em ail attachments, floppy
disks, W eb pages, and other types o f electronic traffic (e.g.,
instant m essaging [IM ] and short m essage service [SM S]) for any
know n or potential viruses, m alicious code, Trojans, or spyware.
N orton A n tiV iru s™ is key to S ym an tec reven u es. T his is illustrated by the high
num ber o f tech n ica l support d ocu m en ts that h a v e b een translated into French and
German: 40% and 50% resp ectiv ely . T h ese figu res sh o w that m ore than h a lf the
docu m en ts rem ain in E n g lish b eca u se o f a lack o f resources to translate all n ew
source content. H o w ev e r, d ocu m en ts are o n ly translated d ep en din g on the critical
nature o f the p h en o m en o n th ey describ e.
T w o other p roducts, N orton Internet S ecu rity™ and N orton S ystem W ork s™ , also
b elo n g to the N orton lin e o f products as bundled su ites o f ap plication s. T h ese suites
con tain ap p lication s su ch as N orton A n tiV iru s, N orton P ersonal F irew all, N orton
G host, or N o rto n A n tiS p am . M arket figu res are n ot availab le for su ites o f
ap plication s. H o w ev er, B urke reveals that N orton products are p re-installed on 50
m illio n n e w PC s w o rld w id e ea ch year (2 0 0 4 : 2 2 ) due to S y m a n tec’s relationship
w ith PC m anufacturers. D e sp ite th is sig n ifica n t figure, the translation ratio for
tech n ical support d ocu m en ts related to th o se tw o products is m u ch lo w er than for
N o rton A n tiV iru s™ .
25%
o f N orton
Internet Security d ocu m en ts are b eing
translated into French com pared to 27% for G erm an, w h ereas 15 % o f N orton
S y stem W ork s d ocu m en ts are translated into French com pared to 22% for Germ an.
For th e three products, the average translation ratios are 30% for French and 35%
for G erm an, clea rly in d icatin g an opportunity to find an alternative w a y to translate
the rem ainder o f th e d ocu m en ts.
1 .3 .3 .2 . C h a lle n g e s o f c o r p u s c o m p ila t io n
T he p ro cess o f te x t extraction to build a corpus presents a num ber o f ch allen ges,
w h ich are d escrib ed b y O lohan (20 0 4 : 4 5 ). S h e n o tes that p erm ission m u st be
granted from the cop yrigh t h old er ‘to m ak e and h old a co p y o f a text electronically'
(2 0 0 4 : 50). T his p erm issio n w a s granted by S ym an tec sin ce the researcher w as
in v o lv e d in a p ilo t project. T he n ex t issu e sh e raises con cern s the form at in w h ich
the corpus should b e stored so that it can later b e p rocessed b y a particular p iece o f
softw are.
38
1 .3
*3-2.1. Storage form at o f the corpus
T he inner w ork in gs o f the ap plication ch o sen to extract segm en ts from the corpus
w ill b e d escrib ed in Chapter 3. T his particular application required th e corpus to be
stored into text form at. T w o so lu tio n s w ere availab le to con vert the H T M L
tech n ical support d ocu m en ts in tex t form at. T he first w a s to a cc ess the various
H T M L p a g es v ia the S ym an tec tech n ica l support W eb site, and save th em as text
file s w ith th e h elp o f a W eb brow ser. T he H T M L cou ld also h ave b een accessed and
d irectly co p ied and pasted and saved as text. In th ese tw o scen arios, a ted iou s p o st­
p ro cessin g stage w o u ld h ave b een required to rem ove unw anted tags or superfluous
lin e fe ed characters that m ay h ave b een inserted during the con version process.
B e sid e s, all H T M L -gen erated d ocu m en ts related to th e products id en tified w o u ld
h a v e b een d ifficu lt to locate. D u e to the nature o f S y m a n tec’s tech n ical support
W eb site, it w as n ot p o ssib le to get a v ie w o f all o f th e ex istin g d ocu m en ts for a
sp e c ific product. I f su ch a list w e re availab le, it w ou ld have b een p o ssib le to create
a program
to
craw l
th o se p a g es
and
sa v e the resultin g
con verted
content
a u tom atically. S in ce an alternative m eth od w a s required, sp ecia l a cc ess w a s
requested to a cc ess th e k n o w le d g e b ase itself, sim ilar to th e a cc ess that content
d ev elop ers and translators h ave. A p riv ileg ed v ie w o f all sou rce tech n ical content
per product w a s obtained. For each article from the list, relevant sectio n s w ere
co p ied and p asted into a te x t file. A m acro w a s then w ritten to extract the
adm inistrative inform ation that sh ould n ot be includ ed in the te x t file itself, but in a
referen ce file. T his referen ce file w a s u sed to track the fo llo w in g inform ation:
■
T he internal referen ce num ber o f the d ocu m en t
■
T he versio n n am es o f th e product
■
T he v ersion n am es o f th e platform
■
T he first p u b lication date o f the d ocum ent
■
T he last update o f the d ocu m en t
■
T he lan gu ages in w h ic h the d ocu m en t is availab le
■
T he num ber o f screen sh ots present in each d ocu m en t
A ll o f th is inform ation w a s retained so as to b e u sed as p o ssib le filters in the
rem ainder o f the study. It w as also d ecid ed to add the date w h en the d ocu m en t w a s
extracted and to m ark the category in w h ich this d ocu m en t w a s cla ssified . W hen
39
u sin g the p riv ileg ed v ie w per product, all d ocu m en ts w ere cla ssified according to
several categories, in clu d in g in stallation issu e s, F A Q s, com p atib ility issu es, general
inform ation, fu n ction ality issu e s, error m e ssa g e s, or p rod u ct-sp ecific issu es. A s one
d o cu m en t cou ld b e referen ced in m ore than o n e category, it w a s n ecessary to ensure
that the con tent w a s extracted o n ly o n c e from the k n o w le d g e base. O ther ch allen ges
encou n tered w ith this approach are d escrib ed b elo w .
1.3.3.2.2. Content extraction post-processing
T he requirem ent to save th e con ten t as tex t-o n ly m eant that all the database's
form atting inform ation w a s lost, a ffectin g titles, b ulleted lists, cell row s, and
h yperlink s. From a corpus an alysis p o in t o f v ie w , this content had to b e correctly
id en tified and parsed during the a n a ly sis p rocess. T w o n e w lin e characters w ere
required b etw een tw o ind ep en d en t seg m en ts to ensure a correct id en tification o f
seg m en t boundaries. For instance, a title w ith ou t a final p unctuation mark, such as a
fu ll stop, w a s separated b y tw o n e w lin e characters to ensure that it w ou ld be
a n alysed sep arately from the first sen ten ce o f the fo llo w in g paragraph. N e w line
characters w ere au tom atically inserted at sp e c ific location s to separate non-related
seg m en ts. T h ese rep lacem en ts w ere p erform ed on all o f th e tex ts that w ere
extracted from th e k n o w le d g e b ase. A m o n g the 3 8 9 d ocu m en ts co llec ted , 140
co n cern ed N orton Internet Security. 177 w ere related to N orton A n tiV iru s™ , and
th e rem ainin g 72 cam e from the N o r to n S ystem W ork s k n o w led g e base. S p ecific
characteristics o f the corpus w ere id en tified w h en calcu latin g w ord cou n ts and
sen ten ce len gth cou n ts.
1 .3 .4 .
S p e c ific c o r p u s c h a r a c te r is tic s
T he siz e o f the corpus w a s first calcu lated w ith a corpus lin g u istic to o l, W ordSm ith
T o o ls 4 .0 (S cott, 2 0 0 4 ), w h ich sh o w ed that it con tain ed 2 2 4 ,6 2 4 tok en s for a total o f
3 ,7 1 9 d istin ct ty p es. H o w ev er, the w ord count produced by this to o l o n ly provided a
raw tok en isation o f the texts. T he w ord cou n t w as then re-run w ith an application
w h o s e tok en iser had b een fin e-tu n ed to deal w ith sp ecific tok en s such as virus
n a m es, U R L s, or file ex ten sio n s. T h is ap plication , acroch eck ™ (A c ro lin x , 2 0 0 5 ),
p rod uced a total w ord cou n t o f 2 2 0 , 5 6 1 . T able 1-1 sh o w s ex a m p les o f su ch tokens.
40
T o k e n c la s s
E x a m p le
URL
w w w .sym an tec .co m
D irectory Path
C :\te m p
File extension
.g if
Virus nam e
W 3 2 .S o b e r@ m m
Table 1-1: Examples of technical tokens present in the corpus
T h e d ifferen ce b etw een the w ord cou n ts ind icates that a fin e-tun ed ap plication w as
required to h andle the tech n ical tok en c la sses that are inherent to tech n ical support
d o cu m en ts. T his also su g g ests that particular attention w o u ld h ave to be p laced on
th ese elem en ts w h en u sin g a gen eral p u rpose M T system .
T he len g th o f sen ten ces w a s a lso calcu lated , and 1167 sen ten ces w ere found to have
m ore than 25 w ords in the corpus. S u ch a h igh num ber m ay b e exp lain ed b y the
frequent u se o f lo n g error m e ssa g e s and d ocu m en t titles em bedd ed in the m id d le o f
sen ten ces. For instance, a d ocu m en t title u sed in a p o st-m o d ify in g p o sitio n is a
tex tu al characteristic o f the corpus. A s d iscu ssed earlier, h yperlink s estab lish in g a
referen ce to other d ocu m en ts are co m m o n p la ce in tech n ical support docum ents.
O ne o f the characteristics o f tech n ica l support con tent is that th ese hyperlinks can
s o m e tim es b e u sed in an a p p o sitiv e p o sitio n , as sh ow n in the fo llo w in g exam ple:
A fter you dow nload the program, fo llo w the instructions in the
docum ent Error: "Unable to initialise virus scanning engine. .
after running W in dow s System Restore or installing. N oiton
A n tiV irus 2 0 0 3 /2 0 0 4 .
In the a b ove ex a m p le, there is n o en d -o f-sen ten ce character b etw een the word
‘d o cu m en t’ and the n am e o f th e d ocu m en t, w h ich is a hyperlink. Such a sen ten ce is
b ou n d to b e p rob lem atic for an M T sy stem b ecau se its len gth introduces co m p lex ity
and am b iguity. T he ab sen ce o f a punctuation mark such as a co lo n or a com m a
m ea n s that a d isam b igu ation issu e m u st be resolved. E ven though the w ord ‘Error’
is cap italised , it m ay b e p o ssib le to m is-p arse the first part o f th e sen ten ce by
a n a ly sin g it as: ‘fo llo w the instru ction s in the Error o f the d ocu m en t:... ’.
41
T he u se o f ap p ositive elem en ts is n ot restricted to error m e ssa g e s. In technical
support docu m en tation , and in softw are d ocu m en tation in general, U ser Interface
(U I) op tion s are often used as p re-m od ifiers, as sh o w n b y the fo llo w in g exam p le
taken from the corpus:
At the Extract File dialog box, in the Restore from box, type the
follow ing where <CD-ROM drive> is the drive letter o f your CDROM drive:
In the ab ove exam p le, tw o U I op tion s ('Extract File', and 'R estore from ') are u sed as
p re-m od ifiers. W h en su ch U I op tion s are not form atted or m arked in the source text,
a m b iguity is introd uced and com p reh en sib ility issu e s m ay arise. T his is another
characteristic o f softw are-related tech n ica l d ocu m en tation that w ill n eed to be
ex a m in ed in detail prior to th e M T p rocess in the seco n d part o f this study.
Apart from the gen eration o f raw num bers o f to k en s and ty p es present in the co ip u s,
no
further le x ic a l
statistics w ere
generated from
the corpus.
From
an M T
p ersp ective, le x ic a l acq u isitio n is crucial to the cu stom isation o f user dictionaries,
b eca u se
any term
that is
not p resent in the
sy ste m ’s dictionaries
w ill be
m istranslated or n ot translated at all. In order to avoid d up licatin g M T dictionary
w ork, o n e o f the first steps in d esig n in g a CL is o ften to standardise w ords and
term s b ased on clea r ly -d efin ed m ean in gs. T his approach is su m m ed up by N yb erg
and M itam ura (1 9 9 6 : 77):
A key elem ent in controlling a source language is to restrict the
authoring o f texts such that only a pre-defined vocabulary is
utilized.
H o w ev er, the aim o f th is study is n ot to d efin e a strict con trolled le x ic o n for a
particular text typ e, ev e n th ou gh it is boun d to im prove th e quality o f the M T output.
In an id eal scenario, H a y es et al. (1 9 9 6 : 9 1 ) state that the con trolled 'vocabulary
database w ou ld p rovid e th e E n g lish side o f a lo ca l dictionary database.' There
w o u ld b e m erits in stu d yin g the distribution o f sy n o n y m s and variants present in the
c o ip u s, and th e su b seq u en t effort in v o lv e d in a system atic standardisation o f such
term in ology. B u t this effort fa lls ou tsid e the rem it o f th is study, sin ce such a
con trolled le x ic o n w o u ld o n ly pertain to S ym an tec d ocu m en tation . In this study, it
is therefore assu m ed that th e u se o f unusual sy n o n y m s and term variants w ou ld
degrade M T output. T h is assu m p tion is based on the com p reh en sib ility issu e s that
42
m a y occu r w h en w riters u se n e w term s to refer to alread y-d efin ed con cep ts
(W arburton, 2 001: 6 7 8 ). It m a y b e true that n e w term s h ave to be coin ed to describe
n e w ideas, esp e c ia lly in h ig h ly te ch n o lo g ica l spheres. B u t w h at is problem atic from
a co m p reh en sib ility p ersp ectiv e is th e u se o f sy n o n y m s, or w ords w ith unusual
sp ellin g or parts o f sp eech . A s K irkm an (19 9 2 : 97) p oin ts out, th ey tend ‘to surprise
the readers, and distract their atten tion ’, on e ex a m p le b ein g 'install' u sed as a noun.
Instead, the fo c u s w ill be p la ced on th e extraction o f segm en ts v io la tin g sp ecific CL
ru les, sin ce th ese ru les cou ld p o ssib ly be u sed in other environm ents. E xistin g
corpus lin g u istic to o ls, such as W ord S m ith T o o ls 4 .0 m ay be u sed to id en tify basic
lin g u istic patterns ad dressed by sp e c ific CL rules. O ne exam p le is B em th and
G d a n iec’s 2 1 st rule, w h ich states that the pattern '(s)' should not be used to indicate
p lu r a l’ (2001: 194). F igure 1-7 sh o w s the results o f a con cordan ce query for the
sim p le pattern *(s) u sin g W ord S m ith T o o ls 4 .0 ’s C oncord m odule:
l£ Concoid
File
Edit
~
View
Compute
Settings
—
'
L J SH
Help
ISetfTaq
N C oncordance
J
1 (scanned duftnq a manual scan Product(s) Norton AntiVirus 2005, NortonH
2005 3 U ser License O perating S ystern(s): W in do w s 2000, W in do w s XP Date
W o r d # 1 * )|a s l| .
os l| .
527
0 527
0 527
0 538
538
0 538
3
type. C lick O K Em ail C lient E>:clusion(s) to enter B ecky! Mail This w ill be the
223
0 223
0 223
4
th e se problem s. Read only the se ction (s) that apply to you. To proceed if you
816
0 816
0 816
5
alerts that you see, scan yo ur hard drive(s) for Internet-enabled a pplications For
477
2
6
7
th is m essage: "The follow ing product(s) m ust be uninstalled before all
Control Panel to remove the product(s). If you do not w ish to remove the
0 477
0 477
24
0
24
0
24
52
0
52
0
52
os|
Figure 1-7: Query results displayed in WordSmith Tools 4.0’s Concord
T h is approach w ork s w e ll for b asic patterns, but is n ot suited to iden tify m ore
c o m p le x lin g u istic structures on a corpus that h as n ot b een tagged. T he approach
u sed to extract v io la tio n s o f rules w ill be d escrib ed in Chapter 3, on ce a set o f CL
ru les has b een se lec ted for th e evalu ation . T he rule selec tio n p rocess is d iscu ssed in
se c tio n 2 .4 o f C hapter 2.
43
1.4. Sum m ary
In the first part o f th is chapter, so m e o f the localisation requirem ents for W eb-based
tech n ical d ocu m en tation h ave been review ed . S ectio n
1.2 sh ow ed that M T is
in crea sin gly b ein g used to address a W eb localisation bottlen eck d esp ite a lack o f
clear and p recise W eb content d ev elo p m en t ru les. In order to address th is issue,
p recise CL rules m ust be rev iew ed and evaluated em p irically so that the m ost
e ffe c tiv e rules can be isolated . T h is em pirical evalu ation w ill be carried out by
u sin g gen u in e content d estin ed for gen u in e users. In sec tio n 1.3 o f this chapter, a
corpus o f tech nical support d ocu m en tation has been a ssem b led to p rovid e the test
content that w ill be u sed in the rem ainder o f th is study. O nce a list o f clearlyd efin ed CL rules is co m p iled in Chapter 2, v io la tio n s o f th ese rules w ill be extracted
from the corpus and used for the first phase o f the evalu ation p rocess.
44
Chapter 2
45
Chapter 2: Identifying MT-oriented CL
rules
2.1. Objectives o f the p resen t chapter
T he con cep t o f C on trolled L anguage (C L ) w a s b riefly presented in th e introduction.
In this chapter, it is further exp lored to a ch iev e tw o ob jectives. The first ob jective is
to ex a m in e various C L evalu ation fram ew orks in order to o p tim ise the evalu ation
m e th o d o lo g y u sed in this study. T he seco n d o b jectiv e is to harvest a list o f clearlyd efin ed C L rules w h o se im pact w ill be evalu ated in Chapter 4 , u sin g segm en ts
extracted from th e corpus. In order to a ch iev e th is seco n d ob jectiv e, several related
con cep ts are d isc u sse d in sectio n 2 .2 so that th e h arvesting o f e x istin g CL rules
fo c u se s on the rules that are su p p osed to im p rove the m achine-translatability o f
E n g lish sou rce content.
2.2. Controlled Language, sublanguage, and
translatability
T he so lu tio n to the cross-cultural com m u n ication d ead lock presented in section
1.2.1
o f Chapter 1 so m etim es lie s in a co m p lete autom ation o f the translation
p ro cess b y u sin g M ach in e T ranslation (M T ). O ne o f th e b est-k n ow n ex a m p les o f an
M T im p lem en tation is probably that o f the d o m a in -sp e cific M T sy stem derived
from the T A U M -M E T E O project and w h ich has b een u sed sin c e 1976 in order to
sim u lta n eo u sly p rod u ce w eather forecast b u lletin s in E n glish and French (Isab elle
1987: 2 7 4 ). H o w ev e r, the su cce ss o f th is sy stem m ain ly relies on th e v ery restricted
nature o f the lan gu age u sed in the su bject fie ld o f w eather forecast bulletin s. Such a
lan gu age is d efin ed as a su blan guage, b eca u se it u se s a restricted lex ic o n and a
lim ited num ber o f syn tactic structures.
A su blan guage is d efin ed as a clo sed subset o f sen ten ces and w ords that is shared
b y a 'com m un ity o f speakers' (K ittredge, 1982: 111). In cid en tally, th ese patterns can
so m etim es h over o n the v erg e o f gram m aticality, w h en ‘th ey are n ot u sual in the
standard la n g u a g e ’ (K ittredge, 2003: 4 3 7 ). I f th is su blan guage is to be p ro cessed by
an M T sy stem , th e M T sy stem m u st b e d esign ed to h andle su ch unusual sen tences.
T he T A U M M E T E O project sh o w ed that a very sp ecia lised subject field cou ld yield
46
M T output o f p u b lish ab le q uality w h en u sin g a d o m a in -sp ecific M T system
(Lehrberger & B ourbeau, 1988: 51). B ut N irenburg (19 8 7 : 12), am ong others,
w arns that it m ay be d ifficu lt to find a ‘s e lf-su ffic ie n t and u sefu l su b lan gu age’ that
ju s tifie s the d esig n o f su ch an M l' system . T he m ain d ifferen ce b etw een a
su blan guage and a con trolled lan gu age is that a su blan guage is n ot artificially
con trolled, it ju st h appens to h ave a lim ited num ber o f lin g u istic features. The
clo se d nature o f a su b lan gu age is due to the restricted dom ain it describes. On the
other hand, a C L is a su bset o f a natural lan gu age that has b een sp ecifica lly
restricted w ith regard to th e le x ic o n , syn tax, and sty le it u ses. T he lex ica l and
gram m atical restriction s that d efin e a C L are th erefore the results o f w ell-th ou gh tout ch o ices. T h ese features are further d escrib ed in the n ext section .
2 .2 .1 . T h e e m e r g e n c e o f C o n t r o lle d L a n g u a g e
T he origin s o f C L can be traced back as early as the 1 9 3 0 s w ith C harles K. O gd en ’s
Basic English (O gd en , 1 9 3 0 ). O gden d esig n ed a restricted lex ic o n that w as
co m p rised o f 850 w ord s. T h is restricted lan gu age w a s ‘inten ded to be used both as
an international lan gu age and as a fou n d ation for learning standard E n g lish ’
(H u ijsen , 1998: 5). It is w orth n otin g that O g d e n ’s B a sic E n g lish w a s never
d esig n e d w ith translation in m ind, but rather to so lv e am b igu ity p rob lem s such as
sy n o n y m y or p o ly se m y for readers o f E n g lish texts. In cid en tally, th ese readers w ere
inten ded to b e b oth n a tiv e speakers o f E n g lish and n on -n ative speakers o f E nglish .
O g d e n ’s ideas w ere th en em u lated a fe w d ecad es later in the au tom otive industry
w ith the introduction o f Caterpillar Fundam ental E n g lish (C F E ). T his CL w as
‘inten ded for u se b y n o n -E n g lish speakers, w h o w o u ld be able to read service
m an u als w ritten in C FE after so m e b asic train in g’ (N y b erg et al., 2 003: 2 6 1 ). A
su rvey o f C L s (A d riaen s & Schreurs, 1992: 5 9 5 ) in d icates that this CL w a s quick ly
fo llo w e d by Sm art’s P lain E n g lish Program (P E P ) and W h ite’s International
L a n gu age for S erv in g and M ain ten an ce (IL S A M ). T he latter gave birth to the
S im p lified E n g lish (S E ) rule set d ev elo p ed b y the A sso c ia tio n E uropéenne des
C onstructeurs de M atériel A érospatial (A E C M A ). T he S E rule set w en t on to
47
b eco m e a standard in the authoring o f aircraft m ain ten an ce docum entation^. A ll o f
th ese projects had on e characteristic in com m on : th ey used C L s that w ere d esign ed
to im p rove the co n sisten cy , readability and com p reh en sib ility o f their source text
for hum an readers. T h ese CLs are therefore o ften regarded as H um an-O riented C Ls,
and accord in g to L u x and D auphin (1996: 194) are not adequate for Natural
L angu age P ro c essin g (N L P ) due to their lack o f form alisation and ex p licitn ess. T his
can be seen b y so m e o f the va g u en ess asso cia ted w ith A E C M A SE's rules for
d escrip tive w ritin g, su ch as rule 6.2, 'Try to vary sentence lengths and constructions
to keep the text interesting'.
2 .2 .2 .
C o n tr o lle d la n g u a g e a n d t r a n s la t a b ilit y
A n oth er project w orth m en tion in g is th e collab oration b etw een the C arnegie
G ro u p /L ogica and D ie b o ld (H ayes, 1996: 89, M oore, 2 0 0 0 ), w h o se o b jec tiv es w ere
slig h tly d ifferen t than the aforem en tion ed p rojects. Instead o f u sin g CL to im p rove
e x c lu siv e ly
th e
co n sisten cy ,
com p reh en sib ility,
and
readability
of
source
d ocu m en tation , D ie b o ld Inc w as interested in introd ucing a CL to also op tim ise its
e x istin g
translation
w o rk flo w ,
w h ich
w as
u sin g
translators
and
translation
m em ories. T he op tim isa tio n o f th is w o r k flo w w a s to b e d one b y 'reducing
w ordcount,
in creasin g
leverageab le
sen ten ces,
and
redu cin g
the
am ount
of
e x p en siv e term in ology' (M oore, 2 000: 51). D e sp ite reporting savin gs o f 25% in
translation co sts w ith the introduction o f C L , M oore also m en tion s that other
b en efits w ere harder to quantify, su ch as cu stom er satisfaction or fe w er support
calls.
5 AECMA SE was renamed ASD STE in 2004, after AECMA joined other European organizations
to form the AeroSpace Defence Industries o f Europe (ASD). SE became Simplified Technical
English (STE), and the SE guide became the ASD STE-100 Specification. Since AECMA SE is the
most commonly used name, it will be used in the remainder o f this study.
2 . 2 .3 * M a c h in e - o r ie n t e d C o n t r o lle d la n g u a g e
In contrast to hum an-oriented C L s or C L s for translation, M achin e-O riented CLs
intend to im p rove the com p reh en sib ility o f source te x ts by N L P applications.
(H u ijsen , 1998: 2). T h ese ap p lication s are n ot n ece ssa rily M T sy stem s, as d iscu ssed
in sec tio n 2 .2 .3 .1 .
2 .2 .3 .1 . P u r e M a c h in e - O r ie n t e d C L
Certain C L s have b een d esig n ed to im p rove the com p reh en sib ility o f text by
program s u sin g lo g ic program m ing or A rtificial In tellig en ce (A I) com pon en ts.
A ttem p to C ontrolled E n glish (A C E ) fa lls into this category, b ein g 'a com puter
p ro cessab le subset o f E n g lish for w ritin g requirem ent sp ecifica tio n s for softw are 1
(F u ch s and S chw itter, 1996: 125). B y rem o v in g am b iguity from the source text,
autom atic
valid ation
o f th ese
sp ecifica tio n s
is
p o ssib le.
T his
strict
CL
is
characterised b y th e v ery lo w num ber o f structures perm itted. For instance, o n ly
d eclarative sen ten ces u sin g th ird -p erson singular sim p le p resent verbs are allow ed .
A n oth er exam p le is the CL that w a s d esig n ed at Ford (R ychtyckyj: 2 0 0 6 ), so that
'engineers can w rite clear and c o n c is e a ssem b ly instructions that are u nam bigu ous
and m achine-readable'. T his CL, n am ed Standard L angu age, restricts the num ber o f
verbs (1 6 9 ) and the typ e o f sen ten ces (im p erative) that can b e u sed b y certain Ford
w riters. A s am b igu ity is rem o v ed from th ese task instructions, the A rtificial
In tellig en ce com p on en t o f Ford's G lob al Study P rocess A llo ca tio n S ystem can
d eterm ine the 'length o f tim e that this task w ill require' (ib id). T he su cce ss o f this
C L is w orth reporting b ecau se Ford is n o w u sin g a general purpose M T system ,
Systran,
to
R ychtyckyj
translate
reports
instru ction s
(ib id)
w ritten
that certain
in
Standard
u n con ven tion al
L anguage.
structures
H o w ev er,
allow ed
by
Standard L angu age, su ch as 'R obot S p o t-w eld the Object' cau sed prob lem s for
Systran. T his ex a m p le con firm s that n ot all am b igu ity issu e s w ill b e h andled in the
sam e w a y b y N L P ap p lication s, so th e e ffe c tiv e n e ss o f certain CL rules m ay not be
reproduced from o n e M T sy stem to th e n ext. T his v ie w has b een exp ressed by
H u tch in s and S om ers (1 9 9 2 : 9 4 ), w h o state that ‘it d o es not really m atter w h at kind
o f am b igu ity the sy stem is up against; w hat m atters is w hether the sy stem has the
relevan t data for d isa m b ig u a tio n ’. W h en a C L is u sed in con ju n ction w ith an M T
49
sy stem , th e rem oval o f le x ica l, structural, and referential am b iguity issu es m ay be
regarded as a p re-p rocessin g step in an M T w o rk flo w . C lem en cin (1996: 32) states
that a m ach in e-orien ted CL attem pts to 'sim p lify and n orm alise the lin gu istic
con ten t o f d ocu m en ts in order to m atch the cap acities o f autom atic translation tools'.
T he n ex t sec tio n fo c u se s o n ex a m p les o f such CLs.
2 .2 .3 .2 . C o n tr o lle d L a n g u a g e f o r M a c h in e T r a n s la tio n
T he first com p an ies to really m ak e u se o f a CL to reduce their translation costs w ere
R ank X e ro x u sin g a S Y S T R A N M T system (A d am s et al., 1999: 2 5 0 ) and Perkins
E n g in es (P ym , 1 990) in the 1 9 8 0 s. V ariou s com p an ies so o n im itated them , and one
o f the m o st su cce ssfu l projects to co m b in e CL w ith M T w a s the collab oration
b etw e en Caterpillar and C arnegie M e llo n U n iv ersity throughout the 1990s. W hereas
Perkins A p p roved C lear E n g lish (P ym , 1 990) u sed a sm all num ber o f rules (1 0 ) and
a sm a ll le x ic o n , the C L d ev elo p ed at Caterpillar w a s characterised by its strictness.
As
a revam p ed v er sio n
o f C FE , Caterpillar T ech n ical E n g lish
(C T E ), w as
s p e c ific a lly d esig n ed to im p rove the clarity o f the sou rce tex t so as to rem ove
a m b igu ities during th e au tom atic translation p rocess (K am prath et al., 1998).
D e sp ite a sign ifican t p rod u ctivity h it in sou rce authoring (H ayes et al., 1996:
8 6
),
w h ich m ay be ex p la in ed b y interactive d isam b igu ation s that authors had to perform ,
th e introd uction o f CTE's 140 C L ru les and con trolled term in ology enabled the
h ea v y
m ach in ery
p u b lish in g
m anufacturer
m achin e-tran slated
to
sig n ifica n tly
d ocu m en tation
redu ce
in
translation
m u ltip le
costs
languages.
by
T he
particularly h ig h and cu m b ersom e num ber o f rules can be exp lain ed b y the fact that
th e M T sy stem u sed , w h ich w a s b ased on the K A N T M T system (M itam ura et al.,
1 9 9 1 ), in v o lv e d an interlin gua p rocess. T he abstract representation o f the source
tex t, ob tained after the parsing o f the E n g lish sen ten ces, had to be universal, so as to
g en erate sen ten ces in m u ltip le target langu ages. In order to m ake this p rocess as
e ffic ie n t as p o ssib le, N y b er g and M itam ura (19 9 6 : 7 7 ) con clu d e that any am biguity
had to b e reso lv ed
in th e sou rce te x t to
ensure translation accuracy. T his
im p lem en tation o f C L for M T sh o w ed that the accuracy o f the M T output depended
h e a v ily on the le v e l o f control present in th e source. T he fact that such large
co m p a n ies
com m itted to
su ch
a paradigm m a y b e
exp lain ed b y
im proved
co m m u n ica tio n b e tw e e n d ev elo p m en t groups and lo ca lisa tio n groups. A m an t
(2 0 0 3 : 5 6 ) ex p la in s that for a lo n g tim e, 'm em bers o f both field s (translation and
50
tech n ical w ritin g) p erceived their p ro fessio n a l a ctiv ities as separate from one
another'.
A round th e sam e tim e, sim ilar CL and M T projects at G eneral M otors (G odden,
1 9 9 8 ) and IB M (B em th , 1998) sh o w ed that CL rules co u ld sig n ifica n tly im prove
the quality o f M T output in variou s lan gu age pairs. Other b en efits w ere, h ow ever,
m ore d ifficu lt to quantify. T his w a s m en tio n ed b y G od d en and M ean s (19 9 6 : 109),
w h o reported that b en efits su ch as higher cu stom er satisfaction cou ld not be
m easured but argued strongly for the im p lem en tation o f the C ontrolled A u tom otive
S erv ice L an gu age (C A S L ) rule set. In the present study, su ch a variable w ill n ot be
studied directly, but fin d in g s related to the accep tab ility and u sefu ln ess o f M T
d ocu m en ts m ay h elp d eterm ine h o w sa tisfied users are w ith M T docum ents.
2 .2 .4 .
D e v e lo p m e n t o f C L r u le s
D eterm in in g w h ich C L rules are the m o st su itable for a sp e c ific environm ent is, o f
cou rse, co m p lica ted b y the p roliferation o f C L rules. M an y C L s and CL ru les have
em erged
in
th e
last
tw o
d eca d es,
w h eth er
as
p rototyp es
or
as
real-life
im p lem en tation s. In a paper p resented at th e fifth C on trolled L angu age A p p lication
W orkshop (C L A W ), P o o l (2 0 0 6 ) re v ie w ed CL projects carried out in the last 25
years (w h ether the C L w a s aim ed at th e W eb , m ach in e reason in g, m achin e
translation, or hum an in tellig ib ility ), and fou n d '41 projects that attem pted to d efin e
w ritten
con trolled
v arieties
o f E n glish ,
E speranto,
French,
G erm an,
G reek,
Japanese, M andarin, S pan ish, or S w ed ish'. A ll o f th ese C L s did not n ecessarily
attem pt to control E n g lish as sou rce langu age. E xam p les o f C L s in lan gu ages other
than
E n g lish
in clu d e
GIF A S
R ation alized
French
(B arthe
et
al.,
1 999),
S ca n ia S w ed ish for S w ed ish (A lm q v ist and S agvall H ein , 1996: 159), M odern G reek
(V a ssilio u et al., 2 0 0 3 : 185), or S iem en s-D ok u m en tation d eu tsch (S ch ach tl 1996) for
G erm an. V e ry often , h o w ev er, the o b jectiv e for the introduction o f source control is
sim ilar across C L projects or prototypes: im p rove the com p reh en sib ility o f source
d o cu m en ts, be it for hum ans or m ach in es, b y redu cin g or rem o v in g am biguity and
c o m p le x ity from sou rce input. O n that p oint, on e m ay w on d er w h eth er a m achin eo rien ted CL is a su b set o f a h um an-oriented CL, or w h eth er th e reverse statem ent is
true. For in stan ce, H u ijsen (1 9 9 8 : 2 ) states that certain rules, such as th e one
restricting the u se o f pronouns or e llip se s, w ill be m ore h elp fu l for m ach in es than
51
for h um ans, and that certain rules w ill b e m ore h elp fu l for hum ans than for
m ach in es, such as the p o sitio n o f d ep en dent cla u ses w ith regard to the m ain clause.
O n the other hand, R euther (2003: 131) found that 'readability rules w ere a subset o f
translatability rules', w h ich a llo w ed her to co n clu d e that 'translatability ensures
readability, w h ereas the reverse statem ent w a s o n ly true to so m e extent'.
T his p roliferation o f projects su g g ests that CL ru les vary accord in g to th e language
d irections, but a lso from on e M T sy stem to the n ext, or from on e te x t typ e to the
n ext. T h is has b een con firm ed b y a study (O ’B rien, 2 003: 111), w h ich fou n d that
eig h t E n g lish C L s shared o n ly on e co m m o n rule, ‘th e rule that p rom otes short
sen te n c es’. T he fa ct that CL rule sets are so ind ividu al su g g ests that the introduction
o f e x istin g C L s sh ou ld not b e p erform ed w ith ou t a n y evaluation. H o w ev er, H artley
and Paris (2 0 0 1 : 3 2 2 ) in sist that 'the corresp on d in g m id d le territory b etw een
d ocu m en t typ e d escrip tion s and sen ten ce rules - the con trolled lan gu age gap - is
underexplored'. In order to b e able to port C L ru les from o n e typ e o f d ocu m en t to
the n ext, or from o n e en viron m en t to th e n ex t, the e ffe c tiv e n e ss o f a rule m ust first
b e evalu ated . S p e c ific evalu ation m eth od s are re v ie w ed in the n ext section .
2.3. Evaluation o f CL rules
2 .3 .1 .
F a c to r s h a m p e r in g th e e v a lu a tio n o f C L r u le s
C L rule sets su ch as A E C M A SE fo c u s o n p rin cip les su ch as clarity and sim p licity
to rem o v e am b ig u ity and co m p lex ity from sou rce content. Y et, N y b erg et al. (2003:
2 7 6 ) state that ‘p resen t-d ay h um an-orien ted C L s are often n ot sp ecified very
p rec ise ly
and
co n sisten tly .
T his
ca u ses
co n fu sio n
in
their ap plication
and
c o m p lica tes e v a lu a tio n .’ W h en certain CL rules are described in gen eric term s,
w riters or con ten t d ev elo p ers m ay apply m ore ch an ges than they are required to. If
d ev elop ers perform
u n d esirab le and u n ex p ected
ch an ges, it is therefore not
surprising that so m e w ritin g ru les ‘m ay even do m ore harm than g o o d ’ (ib id, 2003:
105). N y b e r g et a l.’s statem en t is con firm ed by so m e o f the A E C M A S E ’s rules
(E uropean A s so c ia tio n o f A ero sp a ce Industries, 2 0 0 1 ). For instance, rale 4.1 states
that w riters sh ou ld ‘k eep to on e to p ic per sen te n c e’. S in ce the con cep t o f ‘to p ic ’ is
n o t d efin ed w ith o b jec tiv e criteria, the adheren ce to this rule m ay vary from one
writer to the n ext. T he lack o f granularity in the d efin itio n o f CL rules m ay also be
ex p la in ed
by
the
in su ffic ien t
lin g u istic
52
background
o f w riters
and
content
d ev elop ers in general. I f the lin g u istic p h en om en on addressed b y a CL rule is
d escrib ed in m in ute detail u sin g lin g u istic term in o lo g y , content d evelop ers m ay not
be able to im p lem en t the required change.
T heir potential lack o f lin g u istic k n o w le d g e m ay prevent th em from understanding
w h ich w ord or phrase sh ould be altered, rep laced or rem oved . T his p rob lem has
b een id en tified b y van der Eijk et al. (1 9 9 6 : 6 4 ) w h o state the fo llo w in g :
Grammar restrictions often can only be expressed in a linguistic
jargon that is not alw ays easy to explain to authors, w ho norm ally
are dom ain experts w ith no or lim ited linguistic background.
T his issu e is esp e c ia lly relevant w h en the CL rule is o n ly proscrip tive, becau se
w riters are n ot told w h at th ey are a llo w ed to w rite. T his issu e has been raised by CL
ch eck er d ev elo p ers w h o state that so m e o f the A E C M A SE's ex a m p les 'do not
a lw a y s represent the b est advice' (W o jcik and H olm b ack , 1996: 2 6 ). U ncertainties
surrounding reform ulation s can th en also arise i f a CL ch eck in g ap p lication d oes
n ot p rovid e any alternative rew riting o f p rob lem atic sen ten ces. T his w a s the case
w ith tw o o f the ch eck ers that w ere d ev elo p ed to im p lem en t so m e o f th e A E C M A
SE's rules: the B o e in g SE ch ecker (W o jcik et al. 1990) and th e E U R O C A S T L E
ch eck er (C lem en cin , 1996: 4 0 ). W o jcik and H olm b ack (ibid: 2 3 ) m en tion that the
B o e in g S E C h ecker did not 'attempt to p rop ose rev isio n s o f sentences', w h ile
C lem en cin (1 9 9 6 : 4 0 ) found that the reform ulation o f sen ten ces cou ld not b e fu lly
com pu ted based o n th e inform ation generated by the lin g u istic an alysis o f the
E U R O C A S T L E ch ecker.
2 .3 .2 .
E v a lu a t io n o f C L r u l e s e ts
E m pirical stu d ies h a v e often fo c u se d on the e ffe c ts o f sp ecific CL rule sets, such as
the S E rule set d e v e lo p e d by A E C M A . For in stan ce, Shubert et al. (1 9 9 5 a ) focu sed
on th e effec t o f S E o n the com p reh en sib ility o f A irp lane Procedure d ocum ents
u sin g a co m p reh en sion test, w h ile Shubert et al. (1 9 9 5 b ) and Spyridakis et al.
(1 9 9 7 ) fo c u se d on the effe c t o f SE o n the translatability o f procedural docum ents.
W h ile the p reviou s stu d ies fo c u se d on h um an-oriented CL rules, B em th (1 9 9 9 )
fo c u se d on th e im p act o f a set o f CL rules o n M T output. T he u sefu ln ess o f a
m achin e-tran slated
tech n ica l
d ocu m en t
that
had
b een
p re-ed ited
u sin g
the
E a sy E n g lish A n a ly ze r’s recom m en dation s (B em th , 1998) w as evalu ated by n ative
53
speakers in the target lan gu age. She reports that the num ber o f ‘u s e fu l’ translations
ju m p ed from
6 8
% to 93% w h en the recom m en d ation s w ere taken into accoun t prior
to the M T p ro cess u sin g the L M T M T system . A sim ilar study w a s perform ed by
B ernth & G dan iec (20 0 1 : 2 0 8 ), w h ereb y an E n glish sam p le tex t o f 69 Q & A
sen ten ces in the d om ain o f plant care in stru ction s w as rew ritten accord in g to 13
rules. T he u sefu ln e ss o f th ese p rin cip les w a s then evaluated at the text and sen tence
le v e ls b y com parin g in tellig ib ility sco res g iv e n by a group o f three n ative speakers
in each target lan gu age (French, G erm an, and Spanish). The M T output w as
p roduced b y d ifferen t M T sy stem s for each lan gu age pair. Q uality im p rovem en ts
ranging from 4% to 15% w ere reported at the d ocu m en t le v e l, and from 25% to
36% at the sen ten ce le v e l (ibid).
A n oth er study (R am irez P o lo & H aller, 2 0 0 5 ) fo c u se d on th e e ffe c tiv e n e ss o f the
G erm an CL rule set u sed in th e M U L T IL IN T ch ecker (S ch m id t-W igger, 1998) on
the com p reh en sib ility and p o st-ed ita b ility o f M T outputs p roduced b y several M T
sy stem s. D e Preux's study (2 0 0 5 ) fo c u se d on the e ffe c tiv e n e ss o f the CTE and SE
rule sets on th e com p reh en sib ility and overall quality o f M T output b y cou n ting
lin g u istic errors. T he results ob tain ed in all o f th ese stu d ies w ere lim ited to sp ecific
rule sets, and did n ot m ak e any cla im s w ith regard to the e ffe c tiv e n e ss o f individual
rules.
T h is o v e r v ie w o f the evalu ation o f CL rule sets w a s lim ited to stu d ies w h o se
fin d in gs h ave b een p ub lished. A s m en tio n ed by B ernth & G d an iec (2001: 2 0 7 ), the
ex a ct results o f certain evalu ation stu d ies w ere n ever p u b lish ed for con fid en tiality
reasons. T his is th e ca se w ith the stu d y con d u cted by G odd en at G eneral M otors,
w h ere the effe c t o f th e C A S L rules (G od d en , 1 9 9 8 ) w a s evaluated on French M T
output. T he ev a lu a tio n p rocess w a s con d ucted b y a p rofession al translator and a
b ilin g u a l
au to m o tiv e
tech n ician .
T his
b alan ced
evalu ation
approach
seem s
ad van tageou s as it in v o lv e d a real user o f tech n ical docu m en tation . In G odden's
study, th e im p act o f th e CL ru les on M T output w a s also a ssessed in term s o f p o st­
ed itin g co sts, and su bsequ en t translation co sts savin gs. T his factor is o ften u sed in
ev a lu atin g the e ffe c tiv e n e s s o f C L rule sets, but it fails to p rovid e a clear indication
o f the e ffe c tiv e n e ss o f each C L rule. T his gap has b een filled b y an em pirical study
54
fo c u sin g o n the im p act o f n eg a tiv e translatability indicators on p o st-ed itin g effort
(O 'B rien, 2 0 0 6 ).
2 .3 .3 .
E v a lu a tio n o f in d iv id u a l C L r u le s
D u e to the lack o f em pirical stu d ies in the field o f CL, fo cu s is so m etim es p laced on
th eoretical lin g u istic issu es. A sh ortcom in g in th is approach lies in th e relevan ce o f
th e ex a m p les ch o sen to dem onstrate the e ffe c tiv e n e ss o f som e rules. It m ay be true
that certain 'garden path' sen ten ces are p rob lem atic for M T system s, but on e m ay
w o n d er h o w frequent and system atic th o se sen ten ces are in a rea l-life environm ent.
For in stan ce, let us con sid er th e ex a m p le p rovided by B em th and G dan iec (2001:
185) to ju s tify th e ap plication o f the fo llo w in g rule: 'Do not omit relative pronouns;
write that (which, who, etc.) explicitly'. T he exam p le th ey p rovid e w ithout
m en tio n in g its origin is: 'The cotton shirts are m ad e from co m es from A rizona'
(ib id). T h is ex a m p le is particularly co n fu sin g , but is it representative o f all the
v io la tio n s that th is rule en co m p a sses? T h is issu e has been p oin ted out b y K in g and
F alkedal (1 9 9 0 : 2 1 4 ), w h o state that ‘no attem pt sh ould be m ade to think up or
im port from th e literature tricky e x a m p le s’.
W h en CL rules are b ein g evalu ated as a w h o le , the n egative im p act o f som e m ay be
m ask ed b y the p o sitiv e effects o f others. T his v ie w has b een ech oed b y N y b erg et al.
(2 0 0 3 : 2 5 7 ), w h o b e lie v e that ‘it is unclear w h at th e contribution o f each ind ividu al
w ritin g rule is to th e overall e ffe c t o f th e C L ’. S everal studies h ave exam in ed the
e ffe c tiv e n e ss o f in d ivid u al CL rules. For instance, M oller (2 0 0 3 ) evalu ated the
e ffe c tiv e n e ss o f 5 SE ru les on the q uality o f M T output, but her study w a s not
backed up em p irica lly and lack ed p recise evalu ation m etrics. R ochford (2 0 0 5 ) and
M cC arthy (2 0 0 5 ) also geared th e evalu ation p ro cess tow ards the im pact o f sp ecific
ru les on the q uality o f M T output. In her study, R ochford (2 0 0 5 ) fo c u se d on eight
ru les, u sin g b etw e en o n e and four ex a m p les to test the im pact o f each rule on the
M T output o f three M T system s. M cC arthy (2 0 0 5 ) evaluated m ore rules (2 2 ), but
o n ly u sed tw o ex a m p les to evalu ate their im p act on the M T quality o f tw o target
la n g u ages. T h ese tw o approaches vary slig h tly and su g g est that there is a trad e-off
b etw een th e num ber o f rules that m ay b e evalu ated and the num ber o f segm en ts that
m a y be u sed for the evalu ation . W h en rules are evalu ated in d ivid u ally, a test suite is
often u sed. S u ch an evalu ation en viron m en t is d iscu ssed in sectio n 2 .3 .3 .1 .
55
2 . 3 -3 *1* T r a d i t i o n a l t e s t s u i t e s
B esid e s test corpora, another m eth od is availab le to evaluate the effe c tiv e n e ss o f CL
rules on the perform an ce or output o f an N L P ap p lication su ch as an M T system :
test su ites. For in stan ce, B aker et al. (1 9 9 4 ) u sed a test suite o f m ore than 800
sen ten ces to a sse ss the effe c tiv e n e ss o f d isam b igu ation strategies on the analyser o f
the K A N T M T system . T heir b est resu lts w ere obtained 'when the system w a s run
w ith constrained le x ic o n and gram m ar, no n ou n -n ou n com p ou n d in g and sem antic
restriction w ith a d om ain m odel' (ib id ), lead in g to an average num ber o f 1.5 parses
per sen ten ce, in stead o f 2 7 p arses w ith ou t th ese restrictions. N yb erg et al.
(2 0 0 3 :2 7 3 ) also add that 'about 95% o f the sen ten ces w ere a ssign ed a sin gle
interlingua representation.
B alkan (1 9 9 4 :2 ) d efin es a test su ite as ‘a co lle c tio n o f (u su ally) artificially
constructed inp u ts, w h ere each input is d esig n ed to probe a sy ste m ’s treatm ent o f a
sp ecific p h en o m en o n or set o f p h e n o m en a ’. L ehm ann et al. (1 9 9 6 : 7 1 1 ) refine this
d efin itio n b y stating that test su ites sh ould be system atic and exh au stive. S uch an
eva lu ation strategy appears to be particularly su ited to assess the effe c tiv e n e ss o f
in d ivid u al CL rules. B alkan (ib id) argues that in a test suite, ‘n eg a tiv e data can be
d erived sy stem a tica lly from p o sitiv e data by vio la tin g gram m atical constraints
a sso ciated w ith th e p o sitiv e data ite m ’. T his approach cou ld b e reversed to derive
p o sitiv e data from n eg a tiv e data.
2 .3 .3 .2 . R e f in in g t h e te s t s u ite m o d e l
D a ta v io la tin g a sp e c ific CL rule co u ld ea sily be turned into con trolled data and
b oth M T outputs cou ld be com pared to a ssess the e ffec ts o f introducing the CL rule.
In th is case, q u estio n s m ay arise as to w h eth er a sin g le test sen ten ce is en ou gh to
a ssess th e e ffe c t o f a sp ecific rule that govern s the u se o f a p recise lin gu istic
p h en om en on . For in stan ce, K in g and Falkedal (1 9 9 0 : 2 1 3 ) b e lie v e that the test suite
sh ould com p rise ‘at least tw o test inputs for each structure’. I f an im p rovem en t is
n oted w ith the first pair o f sen ten ces, another pair o f sen ten ces u sin g the sam e
lin g u istic p h en o m en o n sh ould be a ssessed to con firm w h eth er th e im p rovem en t can
be reproduced. O n ce a satisfactory num ber o f test sen ten ces has been u sed and
ev alu ated for a g iv e n lin g u istic feature, it m ay be p o ssib le to gen eralise and
co n clu d e that th e C L rule im p roves sp e c ific characteristics o f the M T output. The
56
ch o ic e o f sp e c ific evalu ation m etrics d eterm in es the m anner in w h ich M T output
has b een im proved.
L arge test su ites fo c u sin g on a large num ber o f rules and ex am p les are probably
m ore reliab le, but th ey can ea sily b eco m e unm anageable w h en a hum an evaluation
m o d el is used. For instance, on e o f the C A S L rules, 'Do not use more than 25 w>ords
per sentence', cou ld b e evalu ated u sin g an extrem ely large num ber o f com bin ations
o f syn tactic structures. The num ber o f p o ssib le reform ulations w o u ld be also very
large sin c e no standard reform ulation e x ists for such a rule. U sin g a fu lly system atic
and ex h a u stiv e approach is therefore n ot p o ssib le. A balance m ust, h ow ever, be
a ch ieved . T his is n ece ssa ry to ensure that no h asty co n clu sio n s are m ade w ith
regard to the im p act o f a rule that has o n ly b een evaluated w ith a fe w sen tences
co n ta in in g sp e c ific patterns. It is p o ssib le to m ake the im p act o f a CL rule m ore
sig n ifica n t than it really is w h en sp e c ific am b igu ou s ex am p les are selected . For
in stan ce, te stin g the im p act o f a rule su ch as 'Do not om it relative pronouns and a
form o f the verb "be " in a relative clause', w ith ex a m p les that o n ly contain partitive
structures, su ch as 'a list o f ad dresses used', w o u ld underm ine the external validity
o f the evalu ation . T he se le c tio n o f CL rules u sed and evaluated in this study is
d isc u sse d in sectio n 2 .4 .
2.4. Selection o f CL rules for the evaluation
2 .4 .1 . Id e n t if ic a t io n o f M T - o r ie n t e d C L r u le s o r
m a c h in e t r a n s la t a b ilit y g u id e lin e s
T he m ain ch a llen g e w ith m ach in e-orien ted C L ru les is that th ey are often
proprietary and th u s often u npublished. T his lack o f v isib ility rein forces the
co n fu sio n that is o ften a ssociated w ith C L rules. For instance, the CL rules that
w ere
d efin ed
at
X e ro x ,
A lca tel
B e ll,
G eneral M otors,
Caterpillar,
or
Sun
M icr o sy stem s h ave n ever b een m ad e co m p letely p ub lic. Their resp ective authors or
u sers (A d a m s et al.: 1999; A d riaen s & Schreurs: 1992; G odd en & M eans: 1996;
K am prath et al.: 1998; A k in s & S isson : 2 0 0 2 ) h ave tou ch ed o n general gu id elin es
that w e re u sed for th e d esig n o f so m e o f the rules, but w ith ou t revealin g the
s p e c ific s o f the ru les. T his p rotection ist attitude contrasts w ith the open approach
u sed for the p u b lica tio n o f general m anuals o f style b ased on in -h ou se gu id elin es
(M ic ro so ft
Corporation:
1998;
Sun M icrosystem s:
57
2 0 0 3 ).
Certain com pan ies
(or departm ents w ith in a g iv en com p an y) see m to be con cern ed w ith th e pub lication
o f assets th ey con sid er as valu ab le, regard less o f the standardisation b en efits they
m ay ob tain from an o p en p o lic y . T h is trend, h ow ever, m ay be ch an gin g as sh ow n
by the creation o f a U ser G roup
co n sistin g o f n in e glob al com p an ies and
corporations around th e con cep t o f C on trolled L angu age u sin g a q uality assurance
ap plication w ith CL ch eck in g cap ab ilities, acroch eck ™
. 6
T ech n ical com m u n ication w o u ld b en efit from a clearly d efin ed set o f CL rules, i f
that set o f rules added v a lu e to con ten t w ith in a m u ltilin gu al en vironm ent. In order
to estab lish a list o f C L rules to evalu ate em p irically, the fo llo w in g sou rces o f CL
rules w ere identified:
P erk in ’s P A C E (P ym , 1 990)
■
A lc a te l’s C O G R A M (A d riaens and Schreurs, 1 995)
■
C A S L rules in B em th and G daniec (2 0 0 1 : 199)
■
E a sy E n g lish (B em th , 1 998)
K A N T CE (M itam ura, 1 9 9 9 )
T he ty p e o f CL rules u sed in the p rojects ab ove vary greatly. T he first on e b elongs
to th e category o f lo o se ly d efin ed C L s (H u ijsen , 1 998) b eca u se its sp ecifica tio n is
n o t very p recise. O n the other hand, K A N T CE's is d efin ed as a strict C L b ecau se o f
its large num ber o f w e ll-d e fin e d rules. A s m en tion ed p reviou sly, this set o f rules is
not fu lly availab le, but so m e ex a m p les o f its p rin cip les are found in the literature
(M itam ura, 1999). T his statem ent a lso ap p lies to the other rule sets presented in this
list.
B e sid e s M T -oriented CL rules, general g u id elin es on m ach in e translatability are
also p resent in th e literature. Past projects (G d aniec, 1994, B ernth & M cC ord , 2 0 0 0 ,
U n d erw o o d & Jongejan, 2 0 0 1 ) h ave sh o w n that there are w a y s to m easure the
m ach in e translatability o f a sou rce text. O ne w a y to d escrib e translatability concerns
the gen eration o f 'g r o s s m easures o f sen ten ce com p lexity' (H a y es et al., 1996: 90).
6 For more information, see: http://www.acrolinx.com/news_release_2006_06_07_en.php
58
In a p rim itive form , this p rocess in v o lv e s cou n tin g the fo llo w in g p h en om en a and
attributing so m e penalties:
Sentence length, numbers o f com m as, prepositions, and
conjunctions, supplem ented by restrictions on som e locally
checkab le grammatical phenom ena, such as passive and - in g
verbs (ibid)
P en alties are ap plied w h en the sou rce te x t v io la tes certain lin g u istic rules. The
fo llo w in g p u b lication s in the field o f m ach in e translatability w ere then considered
as a starting p oin t tow ards the id en tification o f sp e c ific rules:
■
L o g o s T ranslatability in d ex (G d aniec, 1994)
■
B em th and G d a n iec’s m ach in e translatability g u id elin es (Bernth &
G dan iec, 2 0 0 1 )
■
IB M 's W riting G u id elin es for the W eb S ph ere T ranslation Server 7
■
Systran's gu id elin es: Preparing E n g lish T ext for M T (Systran, 2 0 0 5 )
It sh ou ld be n oted that so m e o f the rules overlap w ith on e another. For instance, rule
22 in B ernth and G d a n iec’s m ach in e translatability gu id elin es (20 0 1 : 193) states
that the sla sh character 7 ’ sh ould n ot b e u sed to separate alternative w ord s, as in
‘u se r /sy ste m ’. T he sc o p e o f th is rule is sim ilar to the orthographical g u id elin e
p rovid ed b y M itam ura (1999: 4 7 ), stating that ‘the u se o f the slash character should
be co n sisten tly sp e c ifie d ’. A lso , the rule con cern in g the u se o f ex p licit relative
p ron oun s can b e fou n d in K A N T C L (M itam ura 1999: 4 7 ), P A C E (P ym , 1990: 85),
and B e m th and G d a n iec’s m ach in e translatability gu id elin es (20 0 1 : 184).
M an y sou rces, in clu d in g M itam ura (19 9 9 : 4 7 ), P ym (1990:
8 6
), and B em th and
G d a n iec (2 0 0 1 : 189) a lso recom m en d that w riters 'avoid ellipses, except in clearly
defined cases'. H allid ay and H asan (19 7 6 :
143) d efin e an ellip tical item as
so m eth in g w h ic h ‘le a v e s sp ecific structural slots to be filled e lse w h e r e .’ A ccord in g
to the p reviou s so u rces, relyin g o n an M T sy stem to fill th ese gaps is bound to result
in incorrect p arses o f sou rce inputs, w h ich w ill lead to incorrect output generation.
7 These guidelines were mentioned in Torrejon and Rico (2002) and available at:
http://publib.boulder.ibm.com/voice/ndfs/white napeis/MT Guidelines.pdf [Last accessed: April,
25th, 2004]. This link, however, no longer points to the document.
59
S in ce th e sc o p e o f this rule is large, it w a s d ecid ed to split it into tw o rules: a rule
g o v ern in g the e llip sis o f verbs, and a rule g o vern in g the e llip sis o f subjects. It w as
also d ecid ed to om it lex ica l ellip se s from th e evalu ation . T his d ecisio n w a s m ade on
the assu m p tion that reduced form s o f term s degrade M T output. For instance, i f the
term 'drop dow n' is u sed in sou rce te x t w h ereas an M T dictionary con tain s the term
'drop d o w n m enu', the q uality o f M T output is bound to be affected.
T he m o tiv a tio n s that led to the d esig n o f the rules con tain ed in th e 9 sources
presented earlier are n ot d etailed , sin c e w h at is particularly d ifficu lt for an M T
sy stem is w e ll d ocu m en ted in th e literature (N irenburg, 1989, H utch ins and Som ers,
1 992, A rn old , 2 0 0 3 ). F or instance, H utch ins (20 0 3 : 5 0 5 ) id en tifies sp e c ific issu es,
su ch as ‘the reso lu tio n o f le x ic a l and structural am b igu ities b oth w ith in langu ages
and b etw e en la n g u a g es’. Other ty p es o f am b igu ity in clu d e referential am b iguities
w h ich m ay span several sen ten ces. K aplan (2003: 7 2 ) p rovid es p recise exam p les o f
syn tactic am b iguity, su ch as d ep en d en cy issu e s w h ich m ay lead to parse failures.
K ittredge (2 0 0 3 : 4 3 7 ) states that parse failu res at th e lex ica l le v e l can be triggered
by 'lexical item s that h ave different p a rt-o f-sp eech or freq u en cy o f occurren ce from
th e norm o f the w h o le language'. A fter a fu ll re v ie w and reorgan isation o f th ese
rules, 54 unique rules w ere shortlisted. T h ese ru les are p resented in A p p en d ix A .
T he r e v ie w and reorganisation o f th e CL rules obtained from th e p u b lic dom ain w as
perform ed u sin g a sp e c ific strategy, w h o se m ain purpose w a s to iso la te rules
d ealin g w ith sp e c ific lin g u istic p henom en a. N o n e o f the rules availab le from the
p ub lic d om ain cam e w ith a form alised rule d escription, so ex a m p les in E nglish ,
w h en availab le, had to b e com pared w ith on e another to determ ine w hether tw o
rules co v ered th e sam e lin g u istic p h en om en on . In so m e ca ses, h o w ev er, so m e o f the
rules co lle c te d w ere p rovid ed w ith ou t ex a m p les and exp lan ation s, so certain rules
had to b e com pared and c la ssifie d based on their titles, regard less o f the p recision o f
th ese titles. For th is reason, th e sco p e o f co llec ted rules or 'negative sen tence
properties' (G d an iec, 1994: 100) p resented in A p p en d ix A is so m etim es m u ch w ider
than th e d escrip tion o f the fin al rule ch o sen for the p resent study's evaluation. For
instance, th e first rule evalu ated in th is study, 'Avoid ambiguous coordinations by
repeating the head noun, or by changing the w ord order', is based o n Bernth's rule
(1 9 9 8 : 3 3 ). T he am b igu ity issu e addressed b y this rule is includ ed in the generic
60
p ro b lem o f 'coordinations' that G d an iec regards as a n eg a tiv e sen ten ce property
(ibid). F o cu sin g on a sp e c ific p h en om en on , h o w ev er, w a s th e preferred solu tion to
try and ob tain a system atic reform u lation for m o st exam p les.
2 .4 .2 .
C la s s ific a tio n o f C L r u le s
T he list in A p p en d ix A con tain s ru les that are d esig n ed to rem o v e am biguity and
c o m p le x ity from sou rce content. T h is p rin cip le is quite gen eric so a m ore detailed
cla ssifica tio n o f th ese rules into variou s lin gu istic categories m ay b e required. On
in itial an alysis, it w o u ld appear that th e typ e o f lin gu istic p h en om en on addressed
w ith a CL rule w ill im p act its form al d esig n and su bsequ en t com putational
im p lem en tation . T h is is con firm ed b y O ’B rien (20 0 3 : 106), w h o found that the
c la ssifica tio n o f CL ru les varies accord in g to the lin g u istic fram ew ork that is u sed
to d escrib e a lan gu age. T he c la ssifica tio n sh e u sed in her paper d iv id es CL rules
b ased on lex ica l, syn tactic, and textu al phenom en a. T his w a s also th e taxon om y
u sed in the C O G R A M project at A lca tel B e ll (A driaens & Schreurs, 1992). A s
o u tlin ed in the K A N T CL p roject (M itam ura, 1999: 4 7 ), it is also p o ssib le to
separate C L rules d ep en d in g on th e lin g u istic u nit that is b ein g addressed b y th e CL
rule, w h eth er it is a lex ica l unit, a phrasal unit, a sentential unit, or a textual unit.
B o th approaches im p ly that the c la ssifica tio n o f a CL rule d ep en ds on the le v e l o f
a n a ly sis that is supported b y the C L ch eck in g ap plication u sed to im p lem en t CL
rules. T raditionally, m o st ch eck ers h ave operated at a sen ten ce le v e l. For instance,
C lem en cin (19 9 6 : 3 4 ) states that 'the E U R O C A S T L E ch eck er w orks at the sentence
le v e l and h as v ery little k n o w le d g e o f th e context.' T h is is con firm ed b y B em th
(2 0 0 6 :
1
) w h o m en tio n s that 'little attention has been p aid to d ocu m en t or discourse
le v e l ch ecking'. In a paper p resen ted at the fifth C L A W w ork sh op (ib id), she
rev ea led that IB M 's E a sy E n g lish A n a ly zer w a s b ein g updated to address problem s
occurring at the d ocu m en t le v e l.
A s far as th is stu d y is con cern ed , a CL ch ecker w ill be u sed to extract test sen ten ces
from th e corpus. T h is CL ch eck er, w h ich w ill be p resented in sectio n 3.2.2.1 o f
C hapter 3, d o es n ot perform any gram m atical parse o f th e sou rce text. Y et, phrase
and sen ten ce le v e l p h en o m en a can still be id en tified u sin g a pattern m atching
approach. T hanks to th e fle x ib ility o f th is approach, it d o es n ot seem n ecessary to
attem pt to fit CL rules into s p e c ific categories initially.
61
2.5* Sum m ary
In this chapter, the con cep t o f CL w a s d iscu ssed from a historical p ersp ective in
order to rev iew the various m e th o d o lo g ies that h ave been used to date for the
evalu ation o f CL rules. Several se ts o f C L rules and m achine-translatability
g u id elin es have also been review ed in order to isolate the rules that have been m ost
often im p lem en ted. T his re v ie w h elp ed us draw a list o f ex istin g rules w h o se im pact
w ill be an alysed in Chapter 4. B efore p erform in g su ch an a n a ly sis, v io la tio n s o f
th ese C L rules m ust be identified in the corpus presented in Chapter 1. The
m eth o d o lo g y used for the id en tification o f rule v io la tio n s is d iscu ssed in section 3.2
o f Chapter 3.
62
C
i *
*
P
t
63
e
r
3
Chapter 3: Setting up an environment for
the evaluation of MT segments
3.1. Objectives o f the p resen t chapter
A list o f M T -orien ted CL rules w a s id en tified in th e p reviou s chapter. T h e ob jective
o f this chapter is to set up an evalu ation en viron m en t so that tw o sets o f M T
seg m en ts can b e evaluated. T he first set w ill contain v io la tio n s o f the rules
h arvested in Chapter 2 , w h ile the seco n d set w ill contain segm en ts rewritten
accord in g to sp e c ific C L rules. S ectio n 3 .2 d escrib es the w ay in w h ich a CL checker
is u sed to extract seg m en ts con tain in g v io la tio n s o f th ese CL rules from the corpus.
T he tw o sets o f seg m en ts are then u sed as test data in a test suite environm ent.
S ectio n 3.3 d isc u sse s th e param eters that w ere u sed w h en m achin e-tran slatin g the
tw o sets o f segm en ts. F in ally, the last sec tio n o f th is chapter d escrib es the
ev a lu ation m etrics that are em p lo y ed to an alyse th e M T segm en ts.
3.2. Identifying violations o f CL rules
3 .2 .1 .
D a t a s e le c tio n r e q u ir e m e n t s f o r t h e te s t s u ite
A s d iscu ssed in the p reviou s chapter, d esig n in g an exh au stive test suite is not
a ch iev a b le w ith in the sco p e o f a study fo c u sin g on th e evalu ation o f a large set o f
rules (5 4 ). B e sid e s, in order to b e ex h a u stiv e, on e m ay be tem p ted to u se artificial
structures that n ev er occu r in rea l-life d ocu m en ts. For th ose reasons, it is preferable
to fo llo w K in g and F alkedal's recom m en d ation (19 9 0 : 2 1 4 ), w h ereb y ‘the ch o ice o f
w h at test inputs to in clu d e w ill b e inform ed by the study o f the corpus o f actual
te x ts ’.
T he first o b jec tiv e o f this study is to evalu ate the im pact that clearly-d efin ed CL
rules h ave on th e com p reh en sib ility o f M T segm en ts. T he test su ite u sed to ach ieve
th is ob jectiv e th erefore n eed s to be popu lated w ith sen ten ces from the corpus, as
lo n g as th ese sen te n c es v io la te clea rly -d efin ed CL rules. T he resultin g evalu ation
en viron m en t sh ou ld b e see n as a hybrid m o d el, b orrow in g from both test suites and
test corpora. A s d isc u sse d in sec tio n 2 .3 .3 .2 o f Chapter 2, the num ber o f sen ten ces
extracted per CL rule w ill depend on the lin g u istic scop e o f each CL rule. Certain
C L rules m a y require m ore test inputs than others to ensure that no hasty
64
co n clu sio n s are b ein g drawn. T he extraction p rocess w ill also b e restricted by the
co v erage o f the corpus. Certain v io la tio n s o f CL rules m ay be absent from the
corpus, but no attem pt w ill b e m ade to create artificial exam p les. T he ch o sen hybrid
eva lu ation en viron m en t is m ore fle x ib le than a test suite as it is not based on
sy stem a ticity , w h ile th e co v erage o f its con tent is broader than that o f sm all sam ple
texts. T he approach u sed to extract data is described in the n ext section.
3 .2 .2 .
D a t a e x t r a c t io n u s in g a C L c h e c k e r
In order to id en tify rule v io la tio n s to p op u late the test su ite, it w a s d ecid ed to u se a
C L ch ecker. F ou vry and B alk an (1 9 9 6 : 179) state that CL ch eck ers are 'com p lex
program s, (u su ally) con tain in g parsers and (gram m ar) ch eck ers, and bring in their
o w n extra lim itation s and characteristics, w h ic h are a su bset o f th e natural language'.
For in stan ce, a C L ch eck er m a y fla g a sen ten ce that con tain s a relative clau se that is
n o t introduced by a relative pronoun. In order to fin d su ch rule v io la tio n s, the
ch eck er m u st b e pre-program m ed to id en tify sp e c ific structures. T o extract data
corresp on d in g to v io la tio n s o f th e 5 4 rules selected in the p reviou s chapter, the CL
ch eck er m u st b e fle x ib le en o u g h so that sim p le rules can be added i f they are not
p resent b y default. T here is a lim ited num ber o f com m ercial CL checkers available
on the m arket, n a m ely th e B o e in g C h ecker (W o jcik et al., 1 990), the M a x it C hecker
(Sm art, 2 0 0 5 ), the C on trolled L an gu age A uthoring T e ch n o lo g y to o l p u b lish ed by
the
Institute
of
A p p lied
Inform ation
S c ien ce
(IA I,
2 0 0 2 ),
and
A c ro lin x ’s
a cro ch eck ™ (A c ro lin x , 2 0 0 5 ). I f a C L ch eck er w a s to b e u sed to extract sen ten ces
from tech n ica l d ocu m en tation in the IT d om ain, it w a s essen tial that it cou ld handle
unusual tok en c la sse s such as th o se id en tified in T able 1-1. T herefore, it w as
d ecid ed to u se S ym an tec's v ersio n o f acroch eck ™ 2 .6 , w h ich had b een cu stom ised
to h andle such tok en s. It also con tain ed gen eric and S y m a n tec -sp e cific rules, so
v a lu a b le cu sto m isa tio n tim e w a s saved . T h ese rules are referen ced in A p p en d ix B .
a cro ch eck ™ is presented in further d etail to illustrate h o w it w a s used to extract test
data for th e test suite.
65
3 . 2 . 2 . 1 . a c r o c h e c k ’s t e c h n o l o g y
acro ch eck ™
from A c ro lin x
is b ased on the F L A G te ch n o lo g y described in
B redenkam p et al. (2 0 0 0 ). Its architecture is b ased on a server/clien t m od el,
w h ereb y d ocu m en ts can be ch eck ed w ith in a clien t ap plication , such as M S W ord or
A dobe
Fram em aker v ia
a p lu g-in .
acroch eck ™
also
co m e s w ith
a cu stom
ap plication , acroch eck ™ B atch C lien t, that a llo w s for several d ocu m en ts to be
ch eck ed sim u ltan eou sly, a cro ch eck ’s server contains tw o m ain parts: a lin gu istic
softw are com p on en t (lin gw are) and a report-generating m od u le.
T he report-
generating m od u le is u sed to store inform ation related to the vio la tio n o f rules.
O n ce a d ocu m en t is ch eck ed , th is inform ation is m ade availab le to the clien t
application. E nd-users are th en n o tified o f the m o d ifica tio n s required for the
d ocu m en t to be com p lian t w ith the p red efin ed set o f rules. T he lingw are com pon en t
is a c o lle c tio n o f data file s u sed to fin d rule v io la tio n s. Its o p en architecture creates
an opportunity to u se a cro ch eck ™ to extract v io la tio n s o f rules. S p ecific lin gu istic
patterns o f interest can b e iso la ted b y cu sto m isin g rules and u sin g acroch eck ’s
reporting m od u le. In order to fu lly understand h o w cu sto m ised rules are created,
a cr o ch eck ™ ’s lin gw are (A cro lin x lin g u istic en g in e™ ) m u st b e described in greater
detail.
3.2.2.1.1. Acrolinx Linguistic Engine ™
A cro lin x
lin gu istic
e n g in e ™ ’s te ch n o lo g y
is
b ased
on
the id en tification o f
p red efin ed patterns, w h ic h occu r in a lin g u istica lly annotated string corresponding
to th e d ocu m en t that is b ein g ch eck ed . S en ten ce boundaries are first identified ,
b efo re sen ten ces are to k en ised to determ ine w ord boundaries. E ach tok en present in
a sen ten ce then r e c e iv e s a tok en cla ss d ep en din g on its lin gu istic attributes. For
instance, a file path id en tified by a capital letter, fo llo w e d by a c o lo n and a back
slash , and a cap italised w ord (su ch as C :\W IN D O W S ) receiv es a tok en class
‘file p a th ’. T ok en s w ith u n am b igu ou s tok en cla sses are th en au tom atically attributed
on e o f th e 35 Part O f S p eech (P O S ) tags from th e Pen n T reebank tag set, w h ich is
d escrib ed in M itch ell et al. (1 9 9 3 ) and referen ced in A p p en d ix C. In the p reviou s
e x a m p le, the file path tok en au tom atically re ceiv es a proper noun (N N P ) PO S,
b eca u se it can n ot b e an a ly sed in any other w ay.
66
O nce tok en cla sse s h ave b een attributed to all tok en s, a tagger is used to mark each
to k en w ith a P O S. T he tagger u sed is b ased on the statistical TnT tagger (Brants,
2 0 0 0 ). W ords m issin g from the le x ic o n can be handled thanks to a statistical
approach, and am b igu ou s tok en s from the le x ic o n can be d isam b igu ated by
ex a m in in g con textu al to k en s w ith in a th ree-tok en w in d o w . A n ex a m p le is sh ow n in
F igure 3-1 b elow :
Error:
(¿^Sentence Feature Struct.., S3
"Symantec Log Viewer has encountered an i n t e r n ^
Outline =
Situation:
When ¡starting|the Norton AntiVirus Activity log, you
AppMartie: cclgview.exe AppVer: 103.0.3. S ModName: msvcj
.When]
P 0 5 = WRB
TOK = "When"
TOKEN = J o p
: TAGGER = Jo p
: MORPH
TOKCLA5S = FirstCapltalWord
NOTERM = true
Mo Civet-: 7.0.9466.0 Offset:
000013(36
Solution:
This error message can be caused b y an infection.
. starting
POS » VBG
TOK = "starting*
TOKEN = Jo p
TAGGER = J o p
RELIABLE = true
Do
Hake sure that the computer is virus-free
You m ay see this error message if your computer has a
CONFIDENCE = 57
[ TAGS - [VEG. JJ] |
To quickly scan for common infections
MORPH
Connect to the I n ternet.
Right-click the following link:
TO KCLASS = LowerWord
NOTERM = true
l+j 2
the
g
¿5 3 Norton
POS = NNP
TOK**
v
"Norton"_________
■SplitView Problems ¿if ¡Rule Error Overview SS
Figure 3-1: Linguistic annotation of tokens
In the a b ove figu re, th e tok en 'starting' is an am b igu ou s w ord sin ce it cou ld b e an
ad jective (JJ) or a present participle (V B G ). T he tagger therefore had to se le c t one
o f th ese tw o P O S . T he upper b ox in d icates that V B G w a s selected , probably due to
the fo llo w in g w ord, 'the', w h ich is a determ iner (D T ). A structure JJ + D T is indeed
v ery u n lik ely in E n g lish , esp e c ia lly w h en th e p reced in g w ord is a con ju n ction
('W hen'). O nce ea c h tok en has b een attributed a PO S by the tagger, a m orp h ological
a n alysis o f each tok en is perform ed to p rovid e additional inform ation, such as the
b ase form o f a w ord. In short, each tok en o f the string re ceiv es th e fo llo w in g
lin g u istic inform ation:
■
A to k en form (such as 'starting')
■
A to k en cla ss (su ch as 'L ow er ca se w ord')
■
M o r p h o lo g ica l inform ation su ch as th e
to k en (su ch as 'start' for 'starting')
67
lem m atised form o f the
■
PO S w ith a co n fid en ce indicator (su ch as 'VBG ')
■
T erm inform ation i f th e term is present in a term base
O bjects o f varyin g c o m p le x ity can then b e created u sin g som e o f th ese lin gu istic
features to form patterns. T he p resen ce o f illeg a l patterns is then ch eck ed and
reported on ce ccrtain co n d itio n s h ave b een m et. F or instance, a com m a, fo llo w e d by
a coordin ation con ju n ction (C C ), fo llo w e d b y a form o f 'be' cou ld rou gh ly ind icate
an instance o f an e llip sis o f subject. Patterns, or ob jects, are therefore d efin ed first,
and rules are th en created by fo c u sin g on the interaction o f th ese ob jects. The
fo rm alism u sed by acroch eck ™ is su m m ed up by L iesk e et al. (2002:
6
):
The formalism can be described as a pattern matching language
(including regular expressions, negation, etc.) over linguistic
feature structures. A rule interpreter applies the rules to each
sentence o f the input text and if a pattern matches, then the
corresponding rule with its assigned action is triggered.
S im p le ob ject and rule d efin itio n can b e ea sily p erform ed u sin g acroch eck ’s
Integrated D e v e lo p m e n t E n vironm ent (ID E ), w h ic h em p lo y s a p lu g-in to interface
w ith th e E clip se P latform ap plication (E clip se F oundation, 2 0 0 5 ), as sh ow n in
F igure 3-1. T his en viron m en t w orks as a traditional acroch eck ™ clien t application.
T he o n ly d ifferen ce lie s in the fact that o n ce file s are ch eck ed , their fu ll lin gu istic
annotation data are m ad e availab le so that rules can b e ed ited and refined w h en
'false alarm s or m isse d critiques' (F ou vry and B alkan , 1996: 185) are d iscovered.
T h is
en viron m en t p ro v id es a uniq ue
opportunity to create
sim p le
rules
to
co m p lem en t th e ru les already p resent in Sym an tec's version o f acroch eck ™
(A p p en d ix B ). T he rule creation p rocess is d escrib ed in sec tio n 3 .2 .2 .2 .
3 .2 .2 .2 . C r e a t io n o f s im p le r u le s
S p e c ific rules from th e fin al list o f 5 4 rules w ere identical in sco p e to so m e o f the
rules p resent in S ym an tec's v er sio n o f acroch eck ™ , on e exam p le b ein g rule 4 4 ,
'Avoid splitting infinitives unless the emphasis is on the adverb'. Other rules also
shared so m e co v era g e. For instance, ru les 3 and 3 6 from th e final list w ere
en co m p assed b y th e sam e rule in S ym antec's v er sio n o f acroch eck ™ , 'Do not use
subjectless non-finite clauses a t the start o f sentences'. D iv id in g this rule into tw o
separate rules w a s n o t p o ssib le due to the sem antic im p lication s o f such rules. In
order to extract v io la tio n s o f rule 3 separately from v io la tio n s o f rule 36, sem antic
inform ation is required to determ ine w h eth er th e subject o f a m ain clau se is the
68
sa m e as that o f a n o n -fin ite clau se. T h is lim itation m ean t that v io la tio n s o f th ese
tw o ru les w o u ld b e m an u ally separated after th e extraction p rocess. F in ally, certain
rules from th e fin al list had to b e created from scratch, in clu d in g the three rules
g o v ern in g the u se o f p ronouns (rules 17, 18, and 19). A gain , b e in g able to separate
v io la tio n s o f th ese three rules accurately w o u ld b e v e r y d ifficu lt w ith ou t sem antic
inform ation. B a sic patterns had th erefore to b e created so as to extract as m an y
p oten tial rule v io la tio n s as p o ssib le. A m an u al re v ie w p rocess w o u ld then determ ine
w h ere th ey sh ould b e includ ed in the test suite. F igure 3 -2 su m m arises th e strategy
that w a s adopted to extract v io la tio n s o f su ch rules:
Second Step
Iterative
First Step
CL rule in plain
English
List of violations
in the corpus
f
_______
_L
acrocheckID E
acrocheck
Batch Client
A
Coded CL rule
Corpus
Figure 3-2: Rule creation workflow
In th e a b ove diagram , the term 'plain E nglish' is u sed to refer to th e d escrip tion s o f
C L ru les from the fin al list. For instance, rule 15 in p lain E n glish or laym an's term s
is 'Do not use an ambiguous form o f "have”'. W h en this 'rule' is cod ed , it w ill lo o k
for th e au xiliary 'have', fo llo w e d b y an ob ject o f varyin g nature and len gth , fo llo w ed
b y a p ast-p articip le, as in ex a m p le 82: 'Sym antec d oes n ot recom m en d h avin g
m u ltip le firew alls in stalled on th e sam e com puter.' L iesk e et al. (20 0 2 :
6
) also found
that th e u s e o f the p a ssiv e v o ic e can b e ea sily fla g g ed w ith su ch a form alism w h en
an y in flecte d form o f lie ' occurred togeth er w ith a past p articip le in o n e sentence.
T h e iterative first step for th e creation o f su ch rules con cern s the im p lem en tation o f
b a sic restriction s to ensure that gen u in e patterns are returned. T h e se rules w ere
69
tested separately by ch eck in g several texts from the corpus u sin g acroch eck ™ IDE.
O nce the p rec isio n o f the rules w a s im p roved to avoid too m any fa lse p o sitiv e s,
th ey w ere u sed as u niq ue rules on th e w h o le c o ip u s u sin g acroch eck ™ B atch Client.
Issu es con cern in g p recisio n and recall are d isc u sse d in the n ext section .
3.2.2.2.1. Precision and recall
D u e to the nature o f the task, it w as n ot a lw a y s p o ssib le to create rules w ith high
p recision , w h ereb y o n ly gen u in e v io la tio n s o f th e rules w ou ld b e flagged . The
accuracy o f C L ch eck ers is often m easured w ith p recision and recall scores.
P recisio n is m easu red by com paring the num ber o f correctly fla g g ed errors against
the total num ber o f errors flagged . O n the other hand, recall is m easured by
com parin g th e num ber o f correctly fla g g ed errors w ith the total num ber o f errors
actually occurring (A d riaens & M ack en , 1995: 126). T rying to im p rove on e score
has c o n seq u en ces o n the other. T he m ain o b jec tiv e o f th is task w a s to id en tify
v io la tio n s
o f CL ru les that con tain ed
d ifferen t lin gu istic elem en ts to
avoid
p opu latin g the test su ite w ith sim ilar patterns. L o w p recision w as therefore not an
issu e sin c e a m anual re v ie w p ro cess w o u ld be perform ed to rem ove fa lse p o sitiv es.
It is w orth in sistin g that su ch 'rules' cou ld n ev er b e u sed in a real authoring
en viron m en t
b efore
b ein g
fu lly
tested
and
refined.
The
con seq u en ces
of
im p lem en tin g C L ru les that h ave not b een fu lly tested w ith data on w h ich they are
u sed m u st n ot be underestim ated. W o jcik and H olm b ack (19 9 6 : 3 0 ) found that
tech n ical w riters u sin g the B o e in g S E ch eck er w ere m ore frustrated by errors in
p recisio n than errors in recall. T his w a s con firm ed by Barthe (1996: 50) w h o insists
o n the im portan ce o f updating rules to a v o id 'unjustified errors', be it at the lex ica l
or syn tactic le v e ls.
3.2.2.2.2. Challenges in rule creation
T he sec o n d ch a llen g e to be addressed w ith regard to rule d esign con cern ed certain
d efau lt settin g s o f acrocheck"s lin g u istic resou rces. For instance, rule 4 8 , ‘Question
marks should only be used in direct qu estion s’, p roved d ifficu lt to im p lem en t
w ith o u t ch an gin g a cr o ch eck ™ ’s sen ten ce sp litting rules. I f a q u estion mark is used
in the m id d le o f a sen ten ce, an M T system m ay segm en t the sen ten ce incorrectly.
T his w a s n oted w h en translating the fo llo w in g sen ten ce w ith Systran 5.0:
70
[El In the drop -d ow n m enu, under the "What do you w ant to do?" pane, ch o o se
M an u ally, and click OK.
A
D an s le m enu déroulant, sou s le « Q ue so u h a itez-v o u s faire ? » le vo let,
ch o isisse n t m an u ellem en t, et cliq u en t sur OK.
In th e ab ove exam p le, the q u estion mark is u sed in a direct question, but it d oes not
stand for an end o f sen ten ce character, b ecau se th e w h o le question q u alifies the
w ord ‘p a n e ’. In order to iden tify sim ilar in stan ces o f this p h en om en on in the corpus,
a rule w a s created u sin g acroch eck ™ ID E . H o w ev e r, a lin g u istic p h en om en on that
is p rob lem atic for on e N L P ap p lication is bound to create p rob lem s for other N L P
ap plication s. T his issu e w as h ig h lig h ted w h en attem pting to co d e such a rule by
ex a m in in g patterns w h ich co n sisted o f noun ob jects fo llo w in g a q u estion mark
character and op tional q uotation m arks. W h en ch eck in g the segm en tation o f the
a b o v e ex a m p le, it appeared that the q uestion mark character had b een interpreted as
an end o f sen ten ce marker. B y d efault, sen ten ce boundary m arkers w ere inserted
a u tom atically after q u estion m ark characters. T his m eant that acrocheck's sen tence
sp litting rules had to b e edited so that the CL rule cou ld be im p lem en ted . O nce this
issu e w a s resolved , it w a s p o ssib le to u se the rule on the entire corpus to extract
v io la tio n s o f th is rule. T he r e v ie w p rocess o f th ese v io la tio n s is described in section
3 .2 .2 .3 .
3 .2 .2 .3 . P o p u la tin g t h e te s t s u ite w it h r a w s e g m e n ts
acro ch eck ™ w a s u sed to extract test sen ten ces (or groups o f sen ten ces) from the
corpus. T h ese extracted sen ten ces are referred to as raw segm en ts, w h ich w ere
m an u ally re v ie w ed to obtain a final list o f segm en ts to in clu d e in the test suite.
F o llo w in g K in g and F a lk ed a l’s recom m en d ation (19 9 0 : 2 1 3 ), at least tw o segm en ts
ft
•
had to b e selected for each o f the 54 rules . For certain ru les, it w a s n ecessary to
in clu d e m ore than tw o raw segm en ts b ecau se o f th e p rod u ctive nature o f th e rule.
For in stan ce, the sc o p e o f rule 12, 'Use the active voice when you know who or what
did the action ’, is w e ll-d e fin e d and its reform ulation s are o b v io u s. A s d iscu ssed in
sec tio n 3 .2 2 .2 , a form o f 'be' fo llo w e d by a p ast-participle and 'by' w ill return m ost
v io la tio n s o f th is rule. R eform u latin g such a sen ten ce can b e d one co n sisten tly in
8 All o f the examples used in the test suite are provided in Appendices F and G.
71
m o st ca ses, sin c e the agen t m ust be turned into a subject. H o w ev er, the len gth o f the
agent can vary greatly, b ein g so m e tim e s co m p o sed o f a sin g le w ord (as in exam ple
64, 'W indow s') or o f a lon g n ou n phrase (as in exam p le 67, 'a sp e c ific registry entry
that w a s n ot d eleted during the u nin stallation o f a p reviou s N orton program'). In
order to evalu ate the im p act o f th is C L rule e ffe c tiv e ly u sin g different structures, it
w a s d ecid ed to extract as m an y ty p es o f agen ts as p o ssib le w h ile k eep in g in m ind
that the test suite's con tent w o u ld h a v e to be m an u ally evalu ated and analysed.
T w e lv e raw seg m en ts out o f th e 2 4 4 v io la tio n s o f this rule found in the w h o le
corpus w ere therefore includ ed in th e test suite. It m u st b e stressed that th ese raw
seg m en ts w ere n ot ran d om ly in clu d ed in the test suite for tw o reasons. First, som e
o f th e 2 4 4 v io la tio n s found in the corpus cou ld h ave b een fa lse p o sitiv es. Including
random seg m en ts in th e test suite w o u ld h ave affected the va lid ity o f th e experim ent.
S eco n d , the issu e o f structure d iv ersity m en tion ed in this sectio n w o u ld have been
co m p rom ised . Certain ru les th erefore contain three or four tim e s the num ber o f
ex a m p les o f others in the fin al test su ite. In order to ensure that all CL rules receive
the sam e treatm ent during the evalu ation , an average score w ill be calcu lated based
on th e num ber o f seg m en ts th ey w ere a ssessed w ith. T his w ill be further detailed in
sec tio n 4 .2 .2 .1 o f Chapter 4.
A n oth er criterion u sed for the se le c tio n o f raw seg m en ts w a s that th ey sh ould be as
co n text-in d ep en d en t as p o ssib le in order to facilitate th e m achin e translation process
and th e su bsequ en t evalu ation b y hum an evaluators. S egm en ts are often parsed at a
sen tential le v e l b y M T sy stem s, but prelim inary tests revealed that M T output could
d iffer slig h tly d ep en d in g on th e con text. For this reason, con textu al sen ten ces w ere
a lso in clu d ed in raw seg m en ts in w h ich anaphora resolu tion issu e s w ere apparent.
For instance, i f the first su bject or object o f a sen ten ce w as exp ressed b y an
in d efin ite th ird -p erson pron oun ( ‘it’ in ex a m p le 9 8 ), the p reviou s sen ten ce includ ing
the referent w a s added to th e raw seg m en t to p rovide the M T system w ith an
opportunity to resort to a d isam b igu ation strategy.
F in a lly, on e
last step had to
be perform ed b efore raw segm en ts cou ld be
reform ulated into p ost-C L seg m en ts. K in g and Falkedal (1990: 2 1 3 ) state that ‘test
inputs d esig n ed prim arily to illu m in ate the sy ste m ’s treatm ent o f sp ecific source
la n g u age p h en o m en a sh ou ld a v o id the introduction o f “n o is e ” triggered by an
72
in ju d iciou s ch o ice o f lex ica l m aterial’. T h is recom m en d ation w a s also applied to
su perflu ous syn tactic structures that h ave no direct im p act on the problem atic
lin g u istic feature addressed b y the CL rule. In the p resent study, the e ffe c tiv e n e ss o f
th e C L rules w ill b e evalu ated by hum an evaluators. T est inputs, and corresponding
M T outputs, sh ould therefore be rid o f m o st elem en ts that cou ld m ake the
ev a lu ation p rocess m ore cu m b ersom e as lon g as th ey h ave no direct in flu en ce on
th e lin g u istic p h en o m en o n under study. T he fo llo w in g exam p le sh o w s h o w this
strategy w as used:
■
R u le
8
: 'Do not om it the relative pronoun and a form o f the verb 'be'
in a relative clause when the post-m odifier is an -ing word'.
■
R aw segm ent: I f y o u still have the prob lem , then it can be cau sed by
an op en driver or b y an application n ot
responding to the shutdow n
request.
■
E lem en ts w ith no direct in flu en ce on the lin gu istic p h en om en on
addressed b y the rule: con d ition al cla u se, pronoun
■
Pre-C L segm ent: T he problem can b e cau sed b y an op en driver or
by an ap plication not
■
responding to the sh utdow n request.
P ost-C L segm ent: T h e problem can be cau sed b y an op en driver or
b y an ap p lication th at is
not responding to th e shutdow n request.
O ther ex a m p les o f m o d ifica tio n s perform ed on the raw segm en ts to obtain pre-C L
seg m en ts are p resen ted in A p p en d ix D . T he sem antic sco p e o f certain segm en ts w as
so m e tim es reduced or exp an ded d ep en d in g on the typ e o f m od ification perform ed.
A s far as the o b jec tiv e o f this study w a s con cern ed , a tr a d e-o ff w a s required
b etw e en h avin g gen u in e test sen ten ces and sim p ler sou rce te x t sen ten ces so that the
ev a lu ation p rocess w o u ld n ot be ham pered b y u n n ecessarily co m p lex sen ten ces. It
sh ould b e n oted that th is step w a s lim ited to w e ll-d e fm e d rep lacem en ts or rem ovals.
A s each CL rule h ad to b e evaluated separately, m od ification s based on other
clea rly -d efin ed C L rules w ere n ot introduced. For instance, on e cou ld h ave d ecid ed
to ensure that all o f the p a ssiv e form s present in raw segm en ts w ere turned into
a ctiv e form s in pre-C L segm en ts. A v o id in g p a ssiv e form s is ind eed a CL rule that is
o ften fou n d in the literature (B em th & G daniec, 2 0 0 1 : 190, Farrington, 1996: 18).
B u t w h at i f th is C L rule so m e tim es d oes m ore harm than good ? In su ch a case,
73
so m e o f the test su ite’s seg m en ts cou ld be contam inated and the internal valid ity o f
the study w o u ld be underm ined. F o cu sin g on on e ch an ge at a tim e is sim ilar to the
approach taken by M cC arthy (2 0 0 5 : 12), w h o evalu ated the e ffec tiv en ess o f eigh t
CL rules by u sin g sen ten ces ‘con tain in g o n ly on e p rob lem w h ich the CL rule w ou ld
aim to address'.
O nce the raw seg m en ts w ere rid o f 'noise', the resultin g pre-C L segm en ts w ere
m an u ally turned into p o st-C L segm en ts b y fo llo w in g the rew riting instructions o f
the CL ru les w h en th ey w e re availab le in th e fin al list o f rules. M ost o f the
uncertainty surrounding C L ru les stem s from the v a g u en ess associated w ith the
reform ulation s that w riters or con tent d evelop ers are ex p ected to m ake. A k ey
con cern in the p resent stu d y w a s therefore to ensure that rew ritings w ere as
co n sisten t as p o ssib le so that the im p act o f the rule did n ot vary too m u ch from on e
reform ulation to th e n ext. For this reason, pre-C L seg m en ts w ere all turned into
p ost-C L seg m en ts by th e researcher. It w ou ld , o f cou rse, h ave b een interesting to
study w h eth er d ifferen t p e o p le w o u ld h ave u sed th e sam e reform ulation for a g iv en
lin g u istic feature. H o w ev e r, it w a s fe lt that su ch a stu d y w a s b eyon d th e scop e o f
the present research. B e s id e s , u sin g con sisten t reform ulation s w as som etim es
im p o ssib le due the sc o p e o f th e rules. For instance, rule 2 , 'Do not use ambiguous
attachments
o f non fin ite
clauses
(present-participles)' en co m p a sses
several
a m b igu ou s u ses o f - i n g w ord s introducing n o n -fin ite clau ses. R eform ulations
therefore differed in ex a m p les
1 0
and
1 2
selec ted to test the e ffec tiv en ess o f this
rule:
M F ix es com pu ter p rob lem s u sin g N orton A n tiV iru s.
HI Y o u n eed m o re a ssista n ce rem o v in g a threat.
In th ese tw o ex a m p les, the nature o f the clau se introduced b y the - i n g w ord is
d ifferent. In ex a m p le 10, th e - i n g w ord introduces a cla u se o f m anner that requires
th e p rep osition 'by' to re m o v e the am biguity. O n the other hand, exam p le 12
co n tain s a n o n -fin ite p u rp o siv e cla u se that requires an in fin itiv e verb introduced by
'to' to rem ove the am b igu ity. O n ce th ese m o d ifica tio n s w ere perform ed, the sets o f
3 0 4 pre-C L and p o st-C L seg m en ts w ere ready to b e m ach in e translated. S ection 3.3
o f th is chapter d escrib es the param eters that w ere u sed for th is p rocess.
74
3 »3 - U sing an MT system to translate Pre-CL and
post-CL segm ents
A s ou tlin ed in the introduction, the m ain ob jectiv e o f this study w as to fo c u s on the
F rench and G erm an output prod uced b y a sin g le M T system , Systran W eb Server
5 .0 , rather than to com pare the e ffe c tiv e n e ss o f CL rules on the output o f several
M T sy stem s. T he w a y in w h ich this sy stem w a s u sed is described in sectio n 3 .3 .1 .
3 .3 .1 .
D e te r m in in g M T p a r a m e te r s
A cco rd in g to A rnold et al. (19 9 4 : 163), an im portant factor in the evalu ation o f the
lin g u istic q uality o f M T output is th e co n tex t in w h ich the system is evaluated,
either in a b lack -b ox co n tex t or a g la ss-b o x content. T his param eter is d iscu ssed in
the n ex t section .
3 . 3 . 1 .1 . U s i n g a n M T s y s t e m i n a B l a c k b o x c o n t e x t
W h ite (20 0 3 : 2 2 5 ) states that a ‘g la ss-b o x v ie w lo o k s in sid e the translation en gin e
to see i f its co m p o n en ts each did w h at w a s ex p ected o f th em in the cou rse o f the
translation p r o c e ss.’ O n th e
other hand,
a b la ck -b o x
con text ap plies w h en
a ssessm e n t can o n ly b e perform ed b y w orkin g w ith inputs and outputs (A rnold et al.,
ibid). A n ex a m p le o f g la ss-b o x evalu ation is fou n d in B aker et al. (1 9 9 4 : 90), w h o
‘exp erim en ted w ith th e K A N T an alyzer in order to determ ine the effec ts o f different
d isam b igu ation str a teg ies’. In su ch a setting, th e M T sy stem generates lo g s o f all
operations perform ed, in clu d in g th e d isam b igu ation strategies used during the
a n alysis o f sou rce input. In term s o f CL rule e ffe c tiv e n e ss, a g la ss-b o x approach is
ad vantageous b e c a u se it m ay a llo w researchers to ch eck w h eth er a sp e c ific CL rule
has rem oved a m b ig u ity from sou rce input, and a void ed co m p lex d isam bigu ation
strategies. W ith in a b la c k -b o x con text, exam in in g M T output cannot p rovide this
le v e l o f inform ation. It m ay b e p o ssib le to d eterm ine w h eth er the sou rce input w a s
correctly an alysed , but it is im p o ssib le to k n o w th e co n fid en ce w ith w h ich the M T
output w a s generated. T he system u sed in th is study, Systran W eb Server 5.0, w as
u sed in su ch a co n tex t. T his M T system , h o w ev er, is sh ip p ed w ith a w id e variety o f
clien t ap plication s that m ay b e u sed to request translation jo b s from the M T en gin e.
S in ce the test su ite a ssem b led for th e first part o f this study w a s stored in an M S
E x c e l spreadsheet, it w a s d ecid ed to translate its con ten ts w ith the Systran p lu g-in
for M S E x cel. T h is m eant that a sin g le en viron m en t w a s u sed for the w h o le
75
ev a lu ation p rocess. O n ce the segm en ts w ere m achin e-tran slated , an extra colu m n
w a s added to the spread sh eet so that scores cou ld be attributed to every seg m en t by
the evaluators. T he translation p rocess w a s perform ed b y m ak in g sure that S ystran’s
translation op tion s w ere set to reflect th e sty le o f th e con tent to be m achin etranslated. For both lan gu age pairs, the p o lite form w a s u sed w ith the seco n d person
pronoun set to ‘plural’, as sh ow n in F igure 3-3.
i ranslation Options
^
V
V
Formatting
Support Document Sub-Languages
No
Preserve Textual Formatting
Ssgmenting spaees
Text paragraph definition
Auto
DN7 font list
Symbol.Win gd in g s, Webd in gs
• ■*
Segmentation character list
X
detection
Linguistic options
Do Not Translate Capitalized W ords
No
Source spellcheck
Yes
Imperative choice
Imperativs
if
Personal pronoun definition
1st Person Singular Gender
Masculine
i 2nd Person
Gender
Number
Pa liteiln formal
1st Person Plural Gender
Masculine
Plural
Polits
Masculine
Figure 3-3: Selection of Systran’s Translation Options
F or both lan gu age pairs, it w a s also d ecid ed to u se th e im p erative m od e b y default,
sin c e tech n ica l support con ten t is often characterised b y procedural steps that users
are in vited to fo llo w . H o w ev e r, on e op tion w a s sp e c ific to the E n g lish to G erm an
la n g u age pair: the activation o f the n e w G erm an sp ellin g .
L astly, sp ecia lised d iction aries w ere se lec ted to ensure that M T output w o u ld n ot be
m ea n in g less. T o a ch iev e that ob jectiv e, S ystran ’s S c ie n c e s dictionary w a s activated
u sin g the C om p u ters/D ata p ro ce ssin g d om ain. S y m a n tec’s o w n user d iction aries
w e re also u sed o n ce th ey w ere cu sto m ised to h and le any te rm in o lo g y sp ecific to the
te st sen ten ces ch o se n for the evalu ation . T he ad vantages and draw backs o f this
approach, as w e ll as the issu e s en cou n tered during th e cu stom isation p rocess, are
d isc u sse d in sectio n 3 .3 .1 .2 .
76
3 *3 -i» 2 . U s i n g r e f i n e d u s e r d i c t i o n a r i e s
B efo r e th e con tents o f the test su ite cou ld b e translated and evalu ated , dom ainsp e c ific
te rm in o lo g y
had to
be
id en tified
and
added to
the
system 's
U ser
D iction aries. T his sectio n d escrib es th e approach taken to refine Sym antec's U ser
D ictio n a ries b efore the final translation p rocess w as conducted for the tw o sets o f
seg m en ts con tain ed in the test suite. In the rem ainder o f this dissertation, the set o f
translated p re-C L seg m en ts w ill b e referred to as M T output A and the set o f postCL seg m en ts as M T output B. B oth sets o f seg m en ts w ere m ach in e translated a first
tim e to id en tify m istranslation s o f tech n ica l term s, or ab sen ce o f translations for
u n k n o w n term s in b oth outputs. C ertain ind ividu al term s and com p ou n d s w ere
in correctly generated by the M T sy stem for both the set o f pre-C L segm en ts, and
th e set o f p ost-C L segm en ts. It w a s therefore n ecessary to add th em to the user
d iction aries before p erform in g the fin al evalu ation . T o perform this task, N yb erg
and M itam ura's recom m en d ation s (1 9 9 6 :7 9 ) w ere follow ed :
W henever p ossib le, a system should parse longer strings o f
technical w ords as single, atom ic units o f m eaning rather than
com p osition ally-derived structures. ( . . . ) Phrasal verb-particle
constructions such as 'abide by' are also easier to analyze if taken
as a unit.
T h is last p oin t is ex e m p lifie d b y o n e o f C O G R A M 's pair o f approved lex ica l term s,
'set up', and d isap p roved term , 'establish' (A d riaens, 1996: 2 2 6 ). I f the phrasal verb
is to b e preferred to the sin g le form o f a verb, th en it m u st b e present in the M T
sy stem 's d ictionary. T o ensure the internal valid ity o f the exp erim en t, it w as
esse n tia l that both outputs re ceiv ed the sam e treatm ent, and that u nk n ow n term s
occurring in both outputs w o u ld be added to the d ictionaries. B rym an and Cramer
(2 0 0 1 : 9) p oin t ou t that ‘an intern ally va lid study is on e w h ich p rovid es firm
e v id e n c e o f ca u se and e ffe c t’. I f a causal relationship w ere to b e established
b etw e en the im p rovem en t o f the M T output and the ap plication o f a CL rule, both
M T outputs sh ou ld b e handled in th e sam e w ay. A list o f u nk n ow n or m istranslated
term s w a s co m p iled and translated into both French and G erm an. T his list o f term s
w a s th en im ported into S y m a n tec’s u ser d ictionaries and cod ed u sin g Systran’s
In tu itive C od in g te c h n o lo g y d escrib ed in Senellart et al. (2001: 5). T he m ain
ad vantage o f th is approach w a s the sp eed at w h ich term s w ere im ported and coded,
w h ere other M T sy stem s m ay h a v e required additional lin gu istic inform ation.
C ertain entries generated p rob lem s during the co d in g p rocess, but th ey w ere ea sily
77
identified by S ym an tec's Systran u ser d iction ary experts w h o added sp ecia l intuitive
or co d in g clu es (Systran, 2 005: 146) to ensure that the system w o u ld u se the entry
properly. T he fo llo w in g exam p le dem on strates the u se o f co d in g clu es for an
E n g lish to G erm an entry:
®
p oin t to > z e ig e n a u f
E3 to p oin t to > ze ig e n a u f (g o v e m s_ a c c u sa tiv e )
In the ex a m p le m arked w ith a tick, tw o c lu e s h a v e b een u sed. First o f all, th e source
tok en ‘p o in t’ is recogn ised as a verb during th e co d in g p rocess thanks to the w ord
‘t o ’ w h ich p reced es it. B esid e s, the G erm an p rep osition ‘a u f o f th is particular verb
n eed s to b e fo llo w e d by an accu sa tiv e form , h en ce the p resen ce o f the secon d
co d in g clu e w ith in brackets (g o v e m s_ a c c u sa tiv e ). T he seco n d clu e is an exam p le o f
an advanced la n g u a g e-sp e cific co d in g c lu e 9.
T his approach sh o w e d certain lim itation s, h o w ev er, as so m e entries w ere still used
incorrectly in th e target text for no apparent reason. It w o u ld h a v e b een p o ssib le to
refm e th e u ser dictionary entries over and over again, to try and a ch iev e 100%
accuracy, but th is p ro cess w ou ld h ave b een to o tim e-co n su m in g . B e sid e s, in certain
ca ses, it is not p o ssib le to obtain the d esired result, e sp ec ia lly w h en the form o f the
target w ord d o es not m atch that o f th e sou rce w ord. T h is w a s the ca se w ith the verb
pair 'to click ' and 'klicken a u f, w h ich w a s n ot u sed as intended in the Germ an
output. B y d efault, th e G erm an p rep osition ‘a u f w a s n ot generated b y the M T
sy stem
w h en
the E n g lish
verb
'click' w a s
u sed
w ith ou t the
corresponding
p rep osition ‘o n ’ 10. It w a s therefore d ecid ed to hardcode th e p rep osition 'auf in the
user d ictio n a ry ’s entry. T his p rep osition , h ow ever, sh ould be located in various
p la c es in th e target tex t d ep en din g on th e typ e o f sen ten ce, b e it im perative or
p urposive. A s it w a s h ardcoded in th e user dictionary entry, it a lw a y s fo llo w e d the
verb, w h ich im p acted on G erm an w ord order in m any ca ses. A fter sp en din g som e
9 The Coding Reference Table for German is available at:
http://www.svstransoft.com/support/Dict$/Tables/latesl/de,html [Last accessed: 25/06/2006]. When
last accessed, this table did not include the following clue: (govems_accusative).
10 This issue was reported to Systran and was fixed in a subsequent version o f Systran Web Server
delivered to Symantec, with the implementation o f a specific coding clue.
78
tim e trying to find a so lu tio n to tw eak the entry, it w a s d ecid ed that it w o u ld be
easier to perform an autom atic rep lacem en t b efore subm itting the output to the
evaluators. T he u n d erlyin g con cep t beh in d th is replacem en t is described b elo w .
3 .3 .1 .2 .1 . A u t o m a t i c m o d if ic a t io n s
A s m en tion ed p rev io u sly , recurring p rob lem s can im pact on the M T output in an
u n ex p ected m anner; an exam p le b ein g th e verb pair ‘to c lic k ’, ‘k lick en a u f . A s
m en tion ed by E lm in g (2006: 2 1 9 ), ‘o n e o f the great advantages o f ru le-b ased M T as
o p p o sed to statistical M T is its transparency, ( . . . ) w h ich m ak es th e errors very
c o n siste n t’. T h ese errors m ay be reported to the M T system d evelop er, but it m ay
take so m e tim e for a solu tion to b e fou n d. In the m ean tim e, tw o so lu tio n s are
a v a ilab le to the M T end-user: accep t the error as a PE task and rectify it during that
p ro cess, or fin d a w a y to au tom atically correct th e problem . T he secon d approach
ranges from pattern m atch in g rep lacem en ts (S e n e z , 1998: 2 9 4 ) to transform ationb ased corrections (E lm in g, ibid ). A sim p le solu tion w a s adopted to ensure that a
co m m o n verb, su ch as ‘to c lic k ’, w o u ld n ot a ffect the overall results o f the
evalu ation . T he test su ite con tain ed thirty occu rren ces o f th e w ord ‘c lic k ’, so thirty
ex a m p les w ere p o ten tia lly affected b y th is issu e. E valu ation scores m ay h ave been
affected b eca u se o f th is gram m atical inaccuracy. B efo re the M T output w as
subm itted to evalu ators, a glob al rep lacem en t w a s created and p erform ed on the
G erm an output, b y restoring th e pattern (\b ([D d ]o p p el-)? [K k ]lick en \b ) (\bauf\b)
(\b S ie \b ) as $1 $3 $ 2 . O nce th is rep lacem en t w a s perform ed u sin g a m acro, th e M T
output w a s ready to b e evaluated.
3.4. Evaluation o f the MT segm ents
A lm o s t forty years after th e sem inal report o f the A u tom atic L angu age P rocessin g
A d v iso ry C o m m ittee (A L P A C : 1 966) w a s p ub lished, W h ite (2003: 2 1 2 ) p oin ts out
that ‘reliab le, e ffic ie n t and reusab le m e th o d s’ to evalu ate M T output quality are still
d ifficu lt to find. T in s is due to the fact that ‘a ssessin g translation quality is n ot ju st a
p rob lem for M T : it is a practical p rob lem that hum an translators face ( . . . ) due to the
num ber o f m an y p o s sib le tran slation s’ (A rnold et al., 1994: 161). In order to b ypass
the issu e s that are inherent to hum an evalu ation s, several autom atic evalu ation
m eth od s h ave b een d ev elo p ed in the last num ber o f years. M o st o f th ese autom atic
eva lu ation m eth od s fo c u s o n th e sim ilarity or d iv erg en ce ex istin g b etw een an M T
79
output and on e or several referen ce translations. In order to select the b est-su ited
approach for th e present study, th e ad vantages and draw backs o f both autom atic and
hum an evalu ation strategies are d iscu ssed in the n ext section .
3 .4 .1 .
R e v ie w o f e x is tin g e v a lu a t io n m e t r ic s
W hether an evalu ation o f M T output u ses hum an or autom atic m etrics, ch o ices
m u st be m ad e w ith regal'd to the se le c tio n o f th ese m etrics as no ‘evalu ation
requirem ents or standards for M T output e x is t’ (G ou gh, 2 003: 113). T he selectio n
o f sp ecific m etrics d ep en ds on various factors, ranging from tim e and budget
constraints to the nature o f th e stu d y’s o b jectiv es. P rosp ective m etrics for an
autom atic evalu ation and a hum an or m anual evalu ation are rev iew ed in sectio n s
3 .4 .1 .1 and 3 .4 .1 .2 resp ectiv ely .
3 . 4 . 1 .1 . R e v i e w o f a u t o m a t i c e v a l u a t i o n m e t r i c s
E xam p les o f au tom atic m etrics in clu d e the W ord Error Rate (W E R ) m etric
described in Jufrasky & M artin (2002: 2 7 1 ) and D abbadie et al. (20 0 2 : 10); B L E U
(P apineni et al., 2 0 0 2 ); N IS T (D od d in gton , 2 0 0 2 ); and P recision and R eca ll (Turian
et al., 2 0 0 3 ). U sin g autom atic evalu ation m etrics that em p lo y referen ce translations
w o u ld require at least o n e set o f referen ce translations p roduced from both pre-C L
and p ost-C L seg m en ts, and com parin g the results obtained for both sets. T he first
stu m b lin g b lock o f su ch an approach w o u ld con cern referen ce translations. Should
p o st-ed ited v ersio n s o f refined M T output be used as referen ce translations? Or
should hum an translations be p roduced from scratch? S tu dies u sin g autom atic
ev a lu ation m etrics tend to fo c u s on referen ce translations p roduced b y hum an
translators. For in stan ce, C ou gh lin (2003: 6 5 ) u sed M icro so ft T ranslation M em ories
(T M ), and G ou gh (2 0 0 3 : 114) u sed T M s p rovid ed by Sun M icrosystem s. A sligh tly
different approach is described in H ájic et al. (20 0 3 : 163), w h ereb y reference
translations are p rod u ced by p ost-ed itors u sin g refined M T output con tain ed in a
TM .
80
R eg a rd less o f the approach taken, au tom atic evalu ation m etrics are often said to be
an in e x p e n siv e alternative to hum an evalu ation (P apineni et al., 2002: 318).
P rev iou sly, w e sa w that n e w sets o f data require referen ce translations, w h ich m ight
be m ore e x p e n siv e to produce than p erform in g a m anual evalu ation o f th e M T
output. P o p e sc u -B e lis et al. (2 0 0 2 : 17) m en tion that this budgetary issu e can
b eco m e m ore p rob lem atic i f several referen ce translations are u sed. A u tom atic
m etrics su ch as B L E U
and P recisio n and R ecall support m u ltip le referen ce
translations
string,
per
sou rce
so
several
translations
cou ld th eoretically
be
em p lo y ed to evalu ate the quality o f th e M T output. Such an approach, h ow ever,
w o u ld m ak e the ev a lu a tio n co sts spiral w ith ou t guarantee o f any b en efit. G ough
(2 0 0 3 : 113) states that ‘the num ber o f referen ce translations can a ffect th e scores
obtained in an au tom atic ev a lu a tio n ’, w h ich sh o w s that autom atic m etrics are not
en tirely reliable. T h is v ie w has b een ech o ed in a stu d y perform ed b y Turian et al.
(2 0 0 3 :
8
), w h o se ‘m o s t im portant fin d in g is that, ev e n though hum an evalu ation o f
M T is its e lf in c o n siste n t and n ot very reliab le, autom atic M T evalu ation m easures
are ev e n le s s reliab le and are still very far from b ein g able to replace hum an
ju d g m en t.’ T h is statem en t contradicts the c o n clu sio n s exp ressed in C ou gh lin (2003:
6 9 ) and P ap ineni et al. (2 0 0 2 : 3 1 8 ). P ap ineni et al. state that ‘B L E U ’s strength is
that it correlated h ig h ly w ith hum an ju d g m e n ts’, and C ou gh lin d escrib es it as a
‘h ig h ly reliab le alternative to hum an ev a lu a tio n ’. T h ese statem ents can n ot be
con firm ed in all evalu ation situ ations, as it transpires that B L E U 's scores are m ore
u sefu l for large corpora rather than sin g le sen ten ces (P apineni et al., 2 002: 318).
The purpose o f th is study is to fo c u s o n a spectrum o f im p rovem en ts p rovided by
C L ru les, and hum an ju d g em en ts appear to b e a m ore reliab le solution . T he n ext
sec tio n re v ie w s the m etrics traditionally u sed w ith in hum an evalu ation fram ew orks,
b ased on the o b jec tiv e o f th e present study.
3 .4 .1 .2 . R e v ie w o f h u m a n e v a lu a tio n m e tr ic s
T he hum an ev a lu a tio n o f M T output q uality is o ften regarded as a p erilou s exercise
due to the su b jectiv ity and bias that is inherent to each evaluator. W hite et al. (1994:
1 9 3 ) stress that ‘ju d g m en ts as to the correctn ess o f a translation are h igh ly
su b jective, e v e n a m o n g exp ert hum an translations and translators’. T his p rob lem is
aggravated b y the fa c t that certain evaluators m ay b e b iased w h en evalu atin g M T
output. For in stan ce, i f th ese evaluators are p ro fessio n a l translators, th ey m ay regard
81
M T as a threat. R eg a rd less o f the evalu ation m etrics used, this factor w ill
c o n sc io u sly or u n c o n sc io u sly in flu en ce the w a y th ey score M T output. It is
therefore n ecessary to u se a group o f evaluators to m a x im ise the reliability o f the
results, and d ecid e on w e ll-d e fin e d evalu ation task s that th ese evaluators sh ould
perform .
V an S lyp e (1 9 7 9 :1 2 ) states that ‘translation quality is not an absolute con cep t, and
that it has to be a sse sse d rela tiv ely , ap p lyin g several d istin ct criteria illu m in ating
each sp ecial asp ect o f th e q uality o f the translation ’. A w id e range o f criteria have
b een used to date, and fin d in g th o se that correspond b est to a g iv en study largely
dep en ds on the o b jec tiv es o f this study. V an S ly p e (ib id ) m en tion s that th ese criteria
can operate at several lev els:
■
at th e co g n itiv e le v e l, w h en the p urpose o f th e study is to evalu ate
the in tellig ib ility , the accuracy, or th e u sa b ility o f the M T output
■
at th e e c o n o m ic le v e l, w h en the o b jec tiv e is to m easure the effort, or
th e tim e that w o u ld b e required to bring th e M T output to a certain
quality le v e l.
In the p resent study, th e o b jec tiv e is to isolate CL ru les that w ill p rove e fficien t on
both counts. Id eally, th ey w ill a llo w for the output to be su fficien tly accurate and
com p reh en sib le for u sers, so that the p ost-ed itin g p ro cess can b e b ypassed. T his
dual ob jective m u st th erefore b e reflected in the criteria that w ill b e u sed by the
group o f evaluators sco rin g th e M T output.
3 .4 .1 .2 .1 . C o u n t in g lin g u is t ic e r r o r s
O ne p o ssib le evalu ation m eth od is to count and com pare th e num ber o f translation
errors in both M T outputs. C ou n tin g errors in M T output is an approach su g g ested
by W eissen b orn in V an S ly p e (1 9 7 9 :1 0 2 ), A rnold et al. (19 9 4 : 164), and F lanagan
(1 9 9 4 ). It is also favou red by m e th o d o lo g ies su ch as the B lack jack m eth od, (IT R
Ltd, 2 0 0 2 ) or the S o c ie ty o f A u to m o tiv e E n gin eerin g (S A E )’s J2450 m etrics (2 0 0 1 ).
T his approach h as b e e n u sed in studies fo c u sin g on th e evalu ation o f the
effe c tiv e n e ss o f C L ru les, b y D e Preux (2 0 0 5 :3 ) and M cC arthy (20 0 5 : 26).
H o w ever, this approach p resents several d isad van tages. Firstly, cou n tin g the
num ber o f errors is n o t totally ob jective sin ce it introduces ju d gm en t calls. For
82
instance, the S A E J 2 4 5 0 m etric c la ssifie s errors as either ‘ser io u s’ or ‘m in or’, but
a ck n o w led g es that it is ‘im p o ssib le to d efin e th e n otion o f a serious or m inor error’
(S A E , 2 001: 2 ). T his rem ark g o e s against their stated ob jective: 'to establish a
co n sisten t standard against w h ic h the quality o f translation o f au tom otive service
inform ation can b e o b je c tiv e ly m easured' (S A E , 2 0 0 1 : 1). S econ d ly, it m ay be
p o ssib le to ch eck w h eth er the num ber o f errors is low er in M T output B , and
co n clu d e that the ap p lication o f the C L rule w as effec tiv e. H o w ev er, it w ou ld not
tell us h o w e ffe c tiv e it w a s w ith regard to the com p reh en sib ility o f th e output.
F in ally, on e sh ould n ot u nderestim ate the am ount o f tim e required for evaluators to
b eco m e fam iliar w ith th ese error categories (S A E J2450 o n ly u se 7 categories, but
Flanagan and B lack jack u se m ore than 2 0 ). C o n sisten cy in cou n ting and c la ssify in g
errors m ay be d ifficu lt to a ch iev e ev e n w ith clear instructions. T his ch allen ge w a s
reported by M cC arthy (2 0 0 5 : 3 0 ), w h o se evalu ation study w a s ham pered by the
d ifficu lties en cou n tered by evaluators in categorisin g errors due to the 'subjective
nature o f the task as w e ll as th e "knock-on" e ffe c t o f the different errors'. A m ore
practical and e ffic ie n t approach m u st b e found to avoid the steep learning curve
asso ciated w ith su ch an evalu ation strategy.
3 .4 .I.2 .2 . In te llig ib ility a n d f id e lity
T w o o f the m o st freq u en tly u sed hum an evalu ation m etrics o f M T output quality are
in tellig ib ility and fid elity . For T .C . H allid ay, com p reh en sib ility and in tellig ib ility
are sy n o n y m o u s term s and refer to the ‘ea se w ith w h ich a translation can be
u nderstood, its clarity to th e reader.’ (quoted in V an S lyp e, 1979: 62). D istin ction s
are so m e tim es m ad e b etw e en the tw o term s, d ep en d in g on the type o f m aterial that
is b ein g evaluated (iso la ted seg m en ts or fu ll translation o f a docum ent). A rnold et al.
(1 9 9 4 : 161) h old that the 'in telligib ility o f a translated sen ten ce is affected by
gram m atical
errors,
m istranslation s
and untranslated
w o r d s.’ T his
has
been
con firm ed b y exp erim en ts con d u cted b y R eed er (20 0 4 : 2 3 1 ). She d iscovered that
so m e error typ es are im m ed ia tely apparent and predicators o f in telligib ility. T hese
in clu d e ‘incorrect pron oun translation, in con sisten t p rep osition translation and
incorrect p u n ctu a tio n .’ T he introd uction o f CL rules in the sou rce tex t should
redu ce th ese co m p reh en sib ility issu e s in the target text. B e sid e s, a m achin etranslated sen ten ce can so m e tim es b e very d ifficu lt to understand w h en it contains
am b iguity. For th is reason, d isam bigu ation
83
is often regarded as a com m on
translation task to b e perform ed on the target tex t (L offler-L aurian, 1996:
8 6
). A s
certain CL rules rem o v e am b iguity from the sou rce text, am biguity sh ould also
disappear from the target text.
O n the other hand, fid elity is con cern ed w ith th e accuracy o f the inform ation
co n v ey e d (W h ite, 2 003: 2 1 6 ). W h ile b ein g gram m atical and com p reh en sib le, a
sen ten ce m ay be inaccurate i f am b igu ou s w ords h ave b een m istranslated. T h ese tw o
m etrics fall into the category o f intu itive ju d g m en ts m ention ed by W hite et al.
(1 9 9 4:
193): ‘E valu ation m u st ex p lo it in tu itive ju d gm en ts w h ile constraining
su b jectivity
in w a y s that m in im ise
id iosyn cratic sou rces o f variance in the
m ea su rem en t.’ H o w ev e r, in tellig ib ility and fid elity are often m easured w ith graded
sca les, w h ich m ay im p act the o b jectiv ity o f th e results. T he other drawback o f u sin g
th ese tw o m etrics sep arately is that it exten d s th e tim e required for th e evalu ation
p ro cess, w h ich in turn, in creases the co st o f the operation. W hen u sin g tw o separate
m etrics, C ou gh lin (2 0 0 3 : 64) m en tion s that o n e has ‘to determ ine the im portance o f
on e characteristic o v er another w h en d ecid in g w h at accep tab le quality i s ’. In her
study, w h ich fo c u se d o n the correlation b etw een autom ated and hum an a ssessm en t
o f M achin e T ranslation quality, sh e asked evalu ators to u se a uniq ue scale o f 4
v a lu e s to m easure th e accep tab ility o f the output. T h is sim p le approach integrated
criteria con cern in g b o th in tellig ib ility and accuracy characteristics, but w a s easier to
u se and p rocess than i f th e tw o criteria had b een evalu ated separately.
F in d in g appropriate m etrics for M T evalu ation b ased o n sp ecific requirem ents is a
p ro cess on w h ich th e F ram ew ork for M ach in e T ranslation E valuation (FE M T I) has
b een w ork in g sin c e 2 0 0 2 (H o v y et al., 2 0 0 2 ). FE M T I is an in itiative o f the
International Standards in L angu age E n gin eerin g (IS L E ), w h o se aim is to cla ssify
variou s m e th o d o lo g ies u sed for th e evalu ation o f a w id e range o f M T sy ste m s’
characteristics, in clu d in g the q uality o f M T output. P o p escu -B elis et al. (20 0 5 :
6
)
state that th e ‘m o st origin al a sp ect o f FE M T I is a m ap p in g from the co n tex t o f u se
o f th e M T sy stem to the quality ch aracteristics’ that m u st b e m easured w ith w e lld efin ed m etrics. S in ce M T output is u sed for d issem in a tio n p urposes in th is study,
FE M T I 2 0 0 5 su g g ests that the m ost im portant evalu ation m etrics are accuracy and
w ell-fo rm ed n ess. I f the M T output is ungram m atical or contains p unctuation errors,
it w ill be m ore d ifficu lt to understand. L ik e w ise , i f th e translated inform ation is not
accurate, users m ay do m ore harm than good to their com p u tin g en viron m en t w h en
84
tryin g to fix a p rob lem u sin g the m achin e-tran slated d ocu m en t as referen ce m aterial.
E valuatin g M T output u sin g m etrics incorporating referen ces to accuracy and w e llfo rm ed n ess th erefore seem ed essen tial. E v en th ou gh FE M T I also m en tion s that
sty le is m ore im portant than sp eed w h en M T is u sed for d issem in ation p urposes, the
strict translation turnaround requirem ents ou tlin ed in sectio n 1.2.1 o f Chapter 1
p revented style from b ein g taken into con sideration . It w a s d ecid ed to u se a sim ilar
approach in the p resent study based o n the ty p e o f con tent con tain ed in the corpus,
and the co n tex t o f the u se o f M T . S in ce th is tech n ical content is p erishable and
lik e ly to be updated o n a regular b asis, translations m u st be obtained q u ick ly so as
to b e d issem in ated to a large p o o l o f users. T he approach com b in in g accuracy,
com p reh en sib ility, and w e ll-fo rm e d n ess m etrics, and th e scalc u sed are d escrib ed in
further detail in sec tio n 3 .4 .2 .
3 .4 .2 .
S e le c t io n a n d d e s ig n o f m e t r ic s
T he m ain draw back o f C o u g h lin ’s approach is that her m etrics attem pted to
m easure the accep tab ility o f a set o f translations prod uced by various M T sy stem s,
w ith ou t tak ing u sers into account. V a n S lyp e (1 9 7 9 :1 3 ) stresses that ‘accep tab ility
can be e ffe c tiv e ly m easu red o n ly b y a su rvey o f fin a l u sers.’ It is interesting to note
that a sen ten ce can b e ‘com p reh en sib le, g iv en en ou gh con text or tim e to w ork it
o u t,’ (C ou gh lin , 2 0 0 3 : 6 4 ) b y an evaluator w h o h as had a cc ess to the source text,
but w h at about en d -u sers w h o rely e x c lu siv e ly on th e translated m aterial? The
m etrics ch o se n in th e first part o f the p resent study in clu d ed this requirem ent, based
on the d e c isio n to in v o lv e gen u in e users in the sec o n d part o f the study. T he scale
co n ta in in g four separate v a lu e s is presented in T able 3-1:
Criteria
Score
Excellent MT output (E)
Read the MT output first. Then read the Source Text (ST).
Your understanding of the MT output is not improved by the
reading of the ST because the MT output is satisfactory and
would not need to be modified: it is syntactically correct and it
uses
proper
terminology,
stylistically perfect.
even
However, the
though
it
may
not
be
MT output performs its
primary function, which is to convey information accurately.
An end-user who does not have access to the ST would be able
to understand the MT output.
..
85
1
‘i
11
Criteria
Score
Read the MT output first. Then read the source text.
Good MT output (G)
Your understanding of the MT output is not improved by the
reading of the ST even though the MT output contains minor
grammatical
mistakes
(word
order/punctuation
errors/word
formation/morphology). You would not need to refer to the ST
to correct these mistakes.
An end-user who does not have access to the source text could
possibly understand the MT output.
Read the MT output first. Then read the source text.
Medium MT output (M)
Your understanding of the MT output Is improved by the
reading of the ST, due to significant errors in the MT output
(textual
coherence/
textual
pragmatics/
word
formation/
morphology). You would have to re-read the ST a few times to
correct these errors in the MT output.
An end-user who does not have access to the source text could
only get the gist of the MT output.
Read the MT output first. Then read the source text.
Poor MT output (P)
Your understanding only derives from the reading of the ST, as
you could not understand the MT output. It contained serious
errors in any of the categories listed above, including wrong
POS. You could only produce a translation by dismissing most
of the MT output and/or re-translating from scratch.
An end-user who does not have access to the source text
would not be able to understand the MT output at all.
_
—
—
_J
Table 3-1: Scores used by the evaluators
A s sh ow n in T able 3 -1 , the sco p e o f th ese m etrics is n ot restricted to a sp ecific
q uality criterion. T h is w ill a llo w for an assessm en t o f the M T output on several
le v e ls, co m b in in g criteria that others h ave d ecid ed to separate. For instance,
R am irez P olo and H aller (20 0 5 : 7) d ecid ed to separate com p reh en sib ility and p osted itab ility w ith in their com p arison evalu ation o f three M T sy stem s. T he scores
ch o sen
in
th e
p resent
study,
h o w ev er,
w ill
p rovid e
ind ication s
on
the
co m p reh en sib ility o f M T output and the p o ssib le efforts that w o u ld be required to
bring the seg m en ts to a p o st-ed ited version . B y attributing an 'Excellent' score to an
M T output, evalu ators ju d ge that the seg m en t d o es not n eed to be m od ified . A sk in g
evaluators to take p o st-ed itin g m o d ifica tio n s into accoun t during the evalu ation w a s
d eem ed n ecessa ry to respon d to th e q u estion raised by O 'Brien (2006: 177):
86
I f an e v a lu a to r ra te s a se n te n c e th a t h as b een m a c h in e tran slated
as "ex cellen t", d o es th is m e a n th a t the se n ten c e w ill n o t require
p o st-e d itin g ?
In this study, the attribution o f 'E xcellent' scores m a y be com pared w ith the
v a lid ation o f 100% T M m atch es. T he su g g ested seg m en t is read, com pared w ith the
origin al and validated. O n the other hand, 'M edium ' and 'Poor' scores w ill sh o w that
the M T output m ay be cu m b ersom e for p ost-ed itors to correct. B ased on K rings'
fin d in g s (2 0 0 1 : 5 4 1 ), h o w ev er, it w ill n ot be p o ssib le to assu m e that a 'Poor'
seg m en t w o u ld take lo n g er to p o st-ed it than a 'M edium ' segm en t. In h is study,
K rings (ib id ) fou n d out that ‘m ed iu m q uality ca u ses m ore d ifficu lt PE o v er a ll’.
A t another le v e l, th ese sco res w ill also sh o w h o w com p reh en sib le the M T outputs
w o u ld be for en d -u sers. T h is is facilitated b y the em p h asis that w as p laced on
accuracy, com p reh en sib ility , and w e ll-fo rm e d n ess o f th e output w ith the higher
score ('E xcellen t'). In th eory, 'Good' and 'E xcellent' sco res w o u ld be u n d erstood by
en d -u sers that h ave n o a c c e ss to th e sou rce text. In order to a ch iev e th is ob jective,
evaluators w ere asked to read target and sou rce seg m en ts in turn to try and v ie w
output from an en d -u sers’ p ersp ectiv e. T h ese tw o situ ations are sim ilar up to a
certain point. E valuators h a v e a c c e ss to the sou rce tex t to ch eck that th ey
u nd erstood correctly all that th ey had read. B e sid e s, the lim itation s o f this artificial
com p arison are o b v io u s, sin c e evaluators are lik e ly to find the M T output m ore
co m p reh en sib le due th eir m ore in-depth k n o w le d g e about the top ic than end-users.
T h is is n ot surprising, as D an k s and G riffin (1 9 9 7 : 172) n ote that ‘i f readers have
k n o w le d g e that is sp e c ific to th e text, th en p ro cessin g is fa cilita ted .’ D e sp ite this
lim itation , it w a s essen tia l to in clu d e this requirem ent w ith regard to end -u sers, as
lo n g as the evalu ation w a s p erform ed b y evalu ators w h o w ere fam iliar w ith the
a u d ien ce targeted b y th is ty p e o f tech n ical content. O ther evalu ation requirem ents
also had to be co n sid ered w h en ch o o sin g the evaluators. T h ese requirem ents are
p resen ted in sec tio n 3 .4 .3 .
3 .4 .3 .
E v a lu a tio n r e q u ir e m e n ts
W h ite (2003: 2 1 9 ) rem arks that ‘very ordinary th in g s can affect so m e o n e ’s ability
to b e co n sisten t in th eir ju d gm en ts. S p e cifica lly , th ey w ill g et tired, bored, hungry,
or fed up w ith the p ro ce ss o f ev a lu a tin g .’ T his statem en t is particularly relevant, i f
th e evalu ation p ro ce ss is to b e p erform ed b y p eo p le w h o are n ot fo cu sed on th e task.
87
A ch o ic e had to be m ade w ith regard to th e selec tio n o f evaluators. Should
S ym an tec in -h ou se translators and review ers be used for the evalu ation p rocess, or
should an external a g en cy be hired? A fter con sid erin g the pros and con s o f each
alternative, it w a s d ecid ed to c h o o se th e form er op tion. T he m ain reason for
se le c tin g in -h ou se translators and review ers w as their fam iliarity w ith the top ic and
S y m a n tec -sp e cific term in o lo g y . B e sid e s, it w a s essen tial that both sets o f M T
outputs w ere scored by the sam e p erson w ith in a d efin ed tim efram e. T hese
param eters w o u ld be d ifficu lt to control i f the evalu ation task w a s outsourced. It
w a s d ecid ed to u se four evaluators for each langu age pair. T his d ecisio n w as in line
w ith D y so n and H annah’s recom m en d ation (19 8 7 : 166) to u se no less than three
evaluators, and A rnold et al.'s co n v ictio n that ‘four scorers is the m inim um ; a b igger
group w o u ld m ak e th e results m ore re lia b le’ (1994: 162). For both G erm an and
F rench, tw o o f th e evalu ators w ere translators w ith p o st-ed itin g exp erien ce, w h ile
th e other tw o w ere lin g u istic review ers or experts w ith ou t p ost-ed itin g exp erience.
U s in g evaluators that w ere fam iliar w ith a sp e c ific typ e o f con tent also m eant that
th ey m ay b e m ore stringent w h en ju d g in g the quality o f the output.
A s an addition to th e m etrics presented earlier, the evaluators w ere g iv en the
fo llo w in g extra g u id elin es. T h ey w ere told to co m p lete th e evalu ation o f M T output
A (3 0 4 ex a m p les for a total o f 4 4 5 6 source w ords) b efore starting the evalu ation o f
M T output B (3 0 4 e x a m p les for a total o f 4 6 4 5 sou rce w ord s). This gu id elin e w as
introduced to m in im ise the e ffe c t o f reading an output for a secon d tim e, ev e n i f
slig h tly different from th e first initial one. W h ite (2 0 0 3 : 2 1 8 ) w arns that ‘the second
tim e th ey see th e translation, th ey h ave an inform ed idea o f w h at the ex p ressio n is
su p p osed to say, and th is affects their ju d gm en t o f w h eth er it actually says it or n o t.’
S ectio n 4 .2 o f Chapter 4, w h ich fo c u se s on the c o n sisten cy and the reliab ility o f the
resu lts, w ill sh o w w h eth er th is ob jective w a s ach ieved .
E valuators w ere not told w h ich inputs had b een treated b y CL rules. T h ey a lso did
n ot k n o w w hether th e outputs had b een m ixed . It w a s a co n scio u s d ecisio n n ot to
m ix th e outputs at random sin ce it w a s assu m ed that the tim e allocated to perform
th e evalu ation (tw o w e e k s) and the size o f the test suite w o u ld prevent evaluators
from b ein g in flu en ced b y their p reviou s scores. The evalu ation p rocess can be
regarded as a ‘sin g le -b lin d ex p erim en t’ (L ev in e & Stephan, 2 005:
88
6
) b eca u se o n ly
the researcher k n e w w h ic h segm en ts w ere con trolled . T he purpose o f th is approach
w as to ensure that evalu ators w o u ld n ot be in flu en ced in an y w ay, thus m a x im isin g
the v a lid ity o f the results p resented and d isc u sse d in Chapter 4.
3.5. Sum m ary
T his chapter p resented the en viron m en t that w a s u sed to carry out an evalu ation o f
M T segm en ts. S ectio n 3 .2 o f th is chapter fo c u se d o n the selec tio n o f data u sed in
th e test su ite en viron m en t that w a s sp e c ific a lly d esig n ed for th is study. T his
en viron m en t con tain ed tw o sets o f seg m en ts that w ere extracted from the corpus
based on their v io la tio n s o f sp ecific CL rules. S ectio n 3.3 o f th is chapter d etailed
th e
param eters
u sed
to
m achin e-tran slate
th ese
segm en ts.
F in ally,
p oten tial
ev a lu ation m etrics w ere rev iew ed in the lig h t o f th is stu d y’s o b jectiv es. S p e cific
m etrics w ere d esig n e d b ased on tw o w id e ly -u se d M T evalu ation criteria. Chapter 4
w ill present th e a n a ly sis o f th e evalu ation results.
89
Chapter 4
90
Chapter 4: Analysing the impact of
individual CL rules at the segment level
4.1. Objectives o f the p resen t chapter
In th is chapter, the results o f the evalu ation p ro ce ss described in Chapter 3 are
analysed. To m eet the first o b jectiv e o f this study, th e an alysis o f th ese results w ill
be u sed to d eterm in e w h eth er certain C L ru les are m ore e ffe c tiv e than others in
im p rovin g th e co m p reh en sib ility o f M T output at th e segm en t lev el. S ectio n 4 .2 o f
this chapter p resents the m eth od u sed to an a ly se th e results. In S ectio n 4 .3 , each
rule is re v ie w ed b ased on its le v e l o f im p act o f th e com p reh en sib ility o f M T
segm en ts. F in ally, th e m ain fin d in g s o f th is an a ly sis are sum m arised in sec tio n 4.4.
4.2. Preparing for the analysis o f the results
In order to an alyse q uan titatively th e data c o lle c te d from the evalu ation p rocess, it
w a s d ecid ed to rep lace the scores g iv en b y th e evaluators b y n um eric values.
'E xcellent' w a s rep laced b y 4 , 'Good' b y 3, 'M edium ' b y 2, and 'Poor' b y 1. T hese
replacem en ts w e re essen tia l to b e able to m easu re con tin u ou s variables, su ch as the
m edian score o f a g iv e n ex a m p le, or th e im p rovem en t o f its q uality thanks to the
rew riting. For in stan ce, i f a G erm an M T ex a m p le A re ceiv es 3 M scores and on e G
score, its m ed ian sco re is 2. I f th e corresp on din g M T output B r e c e iv e s 2 'Good'
scores and 2 'E xcellent' scores, its m ed ian score is 3 .5 . T he d ifferen ce b etw een the
tw o m edian sco res g iv e n to ea ch exam p le can th en b e calculated. In th e exam p le
u sed earlier, a raw score o f 1.5 is obtained w h en the m ed ian scores are calculated
and subtracted from o n e another (3 .5 - 2 ). T h e fo llo w in g form ula sh o w s h o w
exam ples' raw sco r es are calcu lated for each target language:
E x a m p le 's ra w sco re = E x a m p le B 's M ed ia n M e d ia n
E x am p le A 's
T he d ecisio n to u se m ed ian scores instead o f average scores is m ade to ensure the
c o n sisten cy and reliab ility o f th e results. S ectio n 4.2.1 d isc u sse s th ese issu e s by
p ro v id in g an o v e r v ie w o f th e resu lts o f the evalu ation p rocess.
91
4 .2 .1 .
C o n s is te n c y a n d r e l ia b il it y o f t h e r e s u lts
A s m en tion ed in H o v y et al. (20 0 2 :
6
), ‘an im portant m atter is inter-evaluator
agreem ent, reported on by m o st careful ev a lu a tio n s.’ T he n ext section fo c u se s on
the reliab ility o f th e evalu ation results, based on the agreem ent am on g evaluators.
4 . 2 . 1 .1 . R e l i a b i l i t y o f t h e r e s u l t s
B efo re ex a m in in g in detail the scores that h ave b een g iv en to M T outputs A and B ,
the le v e l o f general agreem en t am on g the evaluators o f the sam e lan gu age pair is
a ssessed . Four evalu ators w ere used, so relative agreem ent is ach ieved w h en a
m ajority o f three evalu ators g iv e s sim ilar, better, or w orse scores to both outputs,
regardless o f th e ty p e and le v e l o f ch ange. For instance, on e German evaluator m ay
d ecid e to g iv e 'M edium ' scores to both M T output A and M T output B o f a
particular segm en t. In th is ca se, no im p rovem en t is n oted b etw een the tw o segm en ts.
L ik e w ise , a seco n d evaluator m ay d ecid e to g iv e 'Good' scores to both outputs,
co n firm in g the lack o f im p rovem en t. O n th e other hand, i f the last tw o evaluators
d ecid e to g iv e M T output A 'M edium' scores, and M T output B 'Good' scores, an
im p rovem en t is n oted . In this situation, h o w ev er, a co n sen su s is not reached sin ce a
m ajority o f three evalu ators is not obtained. T his situation is referred to in this
sectio n as a d isagreem en t. T able 4-1 sh o w s the le v e l o f con sen su s am on g the
evaluators b ased o n the total num ber o f seg m en ts (304):
Target
Language
Agreement
(Worse)
Disagreement
German
33% (9 9 /3 0 4 )
French
34% (1 0 3 /3 0 4 )
4% (1 1 /3 0 4 )
Agreement
(Same)
41% (126/304 )
,
Agreement
(Better)
22% (68/304)
.
2% (6 /3 0 4 )
31% (93/304)
32% (102/304 )
Table 4-1: lnter-evaluator agreement
T h ese results sh o w that at least three G erm an evaluators agreed on 205 segm en ts
(67% ) w ith regard to th e general trend o f ch an ge from M T output A to M T output B
(sam e, better, or w o r se ), w h ile at least three French evaluators agreed on 201
seg m en ts
(67% ).
D e sp ite
h a vin g
the
ex a ct
sam e
instructions, th ese
results
dem onstrate that evalu ators did n ot interpret th em in the sam e w ay, or d ecid ed to
rate output d ifferen tly.
92
T his lack o f sy stem a tic co n sen su s is con firm ed b y the d ifferen ces ex istin g at the
seg m en t lev el b etw een the scores p ro v id ed by evaluators. T h ese d ifferen ces m ay
n ot be the m ain fo c u s o f this study, but it is w orth estab lish in g h o w d ivid ed
evaluators w ere w h en attributing sco res to sp e c ific segm en ts. Our data sh o w that
scores o f {4 ,4 ,1 ,1 } for a sp ecific seg m en t n ever occur for both sets o f French and
G erm an M T outputs, but that scores o f {1 ,2 ,3 ,4 } appear tw ic e (in the French M T
output A data set).
E valuators h ave h esitated b etw een 'Good' and 'M edium ' scores for a sm all num ber
o f segm en ts. S co res o f { 3 ,3 ,2 ,2 } appear in the data set, representing
8
% o f the
G erm an results (for both M T output A and M T output B ), 7% o f the French results
for M T output A and 3% o f the French results for M T output B . For th ese segm en ts,
a fifth evaluator m a y h ave h elp ed esta b lish a pattern.
O verall, th ese num bers con firm that th e results o f th e evalu ation are o n ly congruent
for about tw o-th irds o f the seg m en ts in each lan gu age. W h en segm en ts are analysed
in d etail in the n ex t part o f this chapter, the fo c u s w ill be p laced on the segm en ts for
w h ich a co n sen su s has b een reached. A fter exam in in g th e general trend o f
a greem ent am on g evalu ators, the c o n siste n c y o f the scores p rovided by the
evaluators is scrutinised.
4 .2 .1 .2 . C o n s is te n c y o f t h e r e s u lts
In so m e ca ses, certain M T output B ex a m p les w ere identical to M T output A
ex a m p les, b eca u se b oth inputs w ere sem a n tica lly identical and the CL rule had no
im p act o n the com p reh en sib ility o f th e M T output. It is therefore p o ssib le to ch eck
w h eth er evaluators a ssig n ed co n sisten tly th e sam e score to both M T outputs. For
instance, B ernth (1 9 9 8 : 3 5 ) recom m en d s that relative pronouns sh ould not be
om itted in front o f p ast-p articip les. T h is rule (rule 9 in A p p en d ix A ) w a s evaluated
w ith eigh t seg m en ts, by introducing a relative p ronoun and a form o f 'be' in M T
output B . T he rep lacem en t is sh o w n as fo llo w s:
0
R epeat th is step for each registry k e y listed in th e error.
A
W ied erh olen S ie d iesen Jobstepp fur je d e n R egistrieru n gssch lu ssel, der im
F ehler au fgefuh rt wird.
0
R epeat th is step for each registry k e y that is listed in the error.
93
B
W ied erh olen S ie d ie sen Jobstepp fiir jed en R eg istrieru n gssch lu ssel, der im
Fehler aufgefiihrt wird.
In the ab ove ex a m p le (e x a m p le 41 in ap p en d ices F and G ), G erm an M T output A
and M T output B are id en tical, but the scores g iv e n by G erm an evaluators w ere not
co n sisten t across both v er sio n s ( { 2 ,3 ,3 ,3 } for M T output A and { 3 ,4 ,4 ,2 } for M T
output B ). S uch a situ ation w a s reproduced w ith 41 other seg m en ts in G erm an, but
o n ly 2 9 seg m en ts in French, sh o w in g that rew ritings had le ss im pact on Germ an
output B than French output B .
T ab le 4 -2 sh o w s the num ber o f id en tical segm en ts in b oth lan gu age pairs and the
co n siste n c y o f each evaluator:
Evaluator l ' s
consistency
Target
L anguage
G erm an
, French
Evaluator 2's
consistency
Evaluator 4 's
consistency
Evaluator 3 's
consistency
8 8 % ( 3 7 /4 2 )
6 4 % ( 2 7 /4 2 )
8 1 % ( 3 4 /4 2 )
6 4 % ( 2 7 /4 2 )
4 0 % (12 /3 0 )
6 0% (1 8 /3 0 )
73% (22 /3 0 )
7 3 % (2 2 /3 0 )
Table 4-2: Evaluators' consistency on identical segments
T h e se p ercen tages s h o w that the G erm an evaluators w ere m u ch m ore con sisten t
w ith their o w n scorin g, averagin g 74.25% co n sisten cy on identical segm en ts, as
o p p o se d to 6 1 .5 % for th e F rench evaluators, strongly affected by the 40%
co n siste n c y o f evalu ator 1. T h is lack o f co n sisten cy m ay b e exp lain ed by the
d ifficu lty for hum an evalu ators to m aintain th e sam e le v e l o f con centration w h en
th e evalu ation task is rep etitive, and requests a fo c u se d attention to detail. It also
corroborates K rin g s’ fin d in g s (2 0 0 1 : 11) that evaluators have a ten d en cy to change
internal criteria o v er tim e. It co u ld be argued that ask in g them to score both
seg m en ts in su c c e ssio n w o u ld h a v e lim ited this c o n sisten cy p roblem , sin ce they
co u ld h ave com pared both seg m en ts and adjusted their scores. H ow ever, the
o b jectiv e o f this study w a s n o t to perform a com parison o f both segm en ts during the
ev a lu ation p rocess itse lf, but rather during th e an alysis o f the data co llec ted during
th is evalu ation .
94
K rings (ib id) also exp lain s that in so m e ca se s evaluators and ‘p ost-ed itors b eco m e
so accu stom ed to the phrasing p roduced by the M T output that th ey w ill no longer
n o tice w h en som eth in g is w ron g w ith it'. In the p resent evalu ation , the scorin g
variation w a s often perform ed in favou r o f M T output B. In the ab ove exam p le,
three o f the G erm an evaluators th ou ght that the a b ove exam p le had been im proved
b y the rew riting, w h ereas on e o f th em thought that it had b eco m e w orse. H ow ever,
th e ten d en cy to favou r M T output B as op p osed to M T output A w a s n ot as clearcut in both langu ages. 4 6 ca se s o f in con sisten t scorin g w ere n oted across the four
F rench evaluators, and in 80% o f the ca ses, the variation w as translated into an
im p rovem en t. H o w ev er, o n ly 58% o f the 43 ca ses o f in c o n siste n c y in G erm an w ere
p o sitiv e im p rovem en ts. It sh ou ld be m en tion ed that a lm ost all o f th ese n egative or
p o sitiv e variations returned a d ifferen ce o f
1
, w h ereb y, for instance, an evaluator
attributed a 'M edium ' score to a segm en t A and a 'Good' score for an identical
seg m en t B . O n ly six ca se s o f in co n sisten cy sh o w ed a variation o f 2 am on g the
F rench evaluators, and tw o ca ses am on g the G erm an evaluators. S in ce French
evaluators p roved less co n sisten t and m ore in clin ed to favour M T output B than
their G erm an counterparts, a refined scorin g m ech a n ism m ust be u sed to a sse ss the
e ffe c tiv e n e ss o f each rule. T he se le c tio n o f su ch a scorin g m ech anism is d iscu ssed
in the n ex t section , w h ere the results o f the evalu ation are presented for ea ch rule.
4 .2 .2 .
S e le c t io n o f s c o r in g m e c h a n is m s f o r t h e a n a ly s is
o f th e r e s u lts p e r C L r u le
A co m b in ed approach is u sed to an alyse th e resu lts o f this evalu ation so that the
im p act o f CL ru les o n the com p reh en sib ility o f M T output can be a ssessed in a
reliab le m anner. B a sed on th e issu e s o f co n sisten cy and reliab ility d iscu ssed in the
p rev iou s sectio n s, u sin g th e evaluators' average sco res obtained for both v er sio n s o f
a seg m en t is n ot an appropriate scorin g m ech an ism . Such average scores w o u ld be
im p acted b y unusual sco res p rovided b y ind ividu al evaluators for sp ecific segm en ts,
be it ou t o f tired n ess, frustration, or keyboard errors. M ed ian scores are therefore
u sed for each segm en t.
95
4 .2 .2 .1 . Id e n t if y in g a g e n e r ic s c o re f o r e a c h C L r u le
T o ju stify th is c h o ice, the sets o f scores obtained for exam p le 41 in Germ an are
presented again: { 2 ,3 ,3 ,3 } for M T output A and { 3 ,4 ,4 ,2 } for M T output B . I f
average sco res w ere calcu lated for each o f th ese tw o sets o f scores, 2 .7 5 and 3.25
w o u ld b e ob tained for M T output A and M T output B resp ectively. T he score 2.75
is m islea d in g b eca u se it d oes n ot reflect the fact that m ost evaluators thought this
seg m en t w a s understandable d esp ite th e m in or gram m atical errors it contains. On
the other hand, a m ed ian score o f 3 p rovid es an accurate picture o f the con sen su s
obtained w ith three o f the evaluators. S u ch d ifferen ces are essen tial to take into
accoun t w h en ex a m in in g the am ount o f im p rovem en t p rovided b y a CL rule's
rew riting recom m en d ation . T he raw score o f th is exam p le is th en calcu lated by
subtracting th e m ed ian score o f M l ’ output A from that o f M T output B , w h ich
y ie ld s 0.5 (3.5 - 3). S in ce th e 54 C L ru les h ave been evalu ated w ith a different
num ber o f ex a m p les, their gen eric scores m ust b e calcu lated b y averagin g all o f
their exam p les' raw sco res, as sh ow n b y th e fo llo w in g form ula:
G eneric score = ^ (E xam ple B's M edian - Exam ple A's M edian-)
N um ber o f E xam ples
Separate g en eric sco res are calcu lated per target lan gu age to com pare and contrast
the results obtained for each o f th e rules in French and G erm an. T h ese scores,
h o w ev er, are u n lik ely to be su fficien t to draw final co n clu sio n s on th e overall
e ffe c tiv e n e ss o f all ru les. Our understanding o f certain rules m ay h ave to be refined
b y lo o k in g at s p e c ific ex a m p les, b efore stating that the rule is a lw a y s effe c tiv e or
in e ffe ctiv e. B y u sin g th is approach alon e, it w o u ld n ot b e p o ssib le to find out
w h eth er certain ex a m p les w ith in each rule w ou ld have p rovided better results, had
th ey n ot b een p o llu ted by other v io la tio n s. A m icro-evalu ation o f rules providing
m ix ed results see m s n ecessa ry to avoid th is issu e. V an S ly p e (19 7 9 : 14) m ention s
that on e o f th e m eth o d s u sed during a m icro-evalu ation o f translation quality
im p lie s a d ia g n o stic le v e l, w h ereb y an ‘an a ly sis o f the cau ses o f errors input, and an
a n a lysis o f th e sou rce la n g u a g e’ is perform ed. B y perform ing an evalu ation o f the
results at th e ex a m p le le v e l, it should b e p o ssib le to determ ine w h y a CL rule had a
sp ecific typ e o f im p act, b e it p o sitiv e , n eg a tiv e, or zero. It sh ould th en be p o ssib le to
96
determ ine w hether the scop e o f a rule cou ld b e restricted by elim in atin g patterns
that did not im p rove th e M T output, or in certain ca ses m ade it w orse.
4 . 2 . 2 . 2 . U s i n g ’e x t r a ’ s c o r e s t o r e f i n e t h e a n a ly s is
S in ce on e o f our o b jec tiv es is also to iso la te the rules that p rovid e th e m ost
im portant im p rovem en ts, a secon d set o f scores is required for each rule. K n ow in g
that a rule im p roves M T output is on e thing, but k n ow in g by h o w m u ch it im p roves
M T output is another. In order to p rovid e answ ers to th is question, ex a m p les that
h ave b een greatly im p roved b y the C L rules' rew riting w ill be counted. W hen u sing
th is seco n d approach, the scorin g agreem en t d iscrep an cies noted earlier m u st be
taken into accou n t to ensure that the rules' scores do not originate from in con sisten t
scoring.
T he fo llo w in g restriction s w ill therefore apply for the generation o f su ch 'extra'
scores:
1
■
E xam p les w ith id en tical M T output A and M T output B are
ex clu d ed
■
E x a m p les for w h ich a co n sen su s w as n ot reached b y at least three
evaluators are exclu d ed . A n im p rovem en t m u st b e noted in the raw
scores o f three evaluators.
T h ese
'extra'
scores
are further d ivid ed
to
take into
accoun t the
le v e l
of
im p rovem en t brought b y the ap p lication o f a g iv en CL rule. T w o le v e ls o f p o sitiv e
'extra' scores are d efin ed as fo llo w s:
■
For p o sitiv e 'extra' sco res o f le v e l 1, the m ed ian score o f an
exam p le's output A m u st b e inferior to 3, and th e m ed ian score o f
the corresp on din g output B m u st b e superior or eq u al to 3. T he CL
rule's
rew riting
ensured
a
m ajor
im p rovem en t
in
the
com p reh en sib ility o f th e M T output, sin ce a m ajority o f evaluators
rated th e exam ple's output A as 'Poor' or 'M edium' and the exam ple's
output B as 'Good' or 'E xcellent'.
■
For p o sitiv e 'extra' scores o f le v e l 2, the m edian score o f an
exam p le's output A m u st b e equal to 3, and the m edian score o f the
corresp on din g output B m u st b e equal to 4. T he C L rule's rew riting
97
ensured a m inor im p rovem en t in the com p reh en sib ility o f the M T
output, since a m ajority o f evaluators rated the exam p le's output A
as 'Good' and th e exam p le's output B as 'Excellent'.
T w o le v e ls o f n eg a tiv e 'extra' scores are d efin ed as fo llo w s:
■
For n eg a tiv e 'extra' sco res o f le v e l 1, the m ed ian score o f an
exam p le's output A m u st b e equal or superior to 3, and the m edian
score o f the corresp on din g output B m ust be inferior to 3. T he CL
rule's
rew riting
had
a
m ajor
n egative
im pact
on
the
com p reh en sib ility o f the M T output, sin ce a m ajority o f evaluators
rated the exam ple's output A as 'Good' or 'E xcellent' and the
exam p le's output B as 'M edium ' or 'Poor'.
■
For n eg a tiv e 'extra' sco res o f le v e l 2, the m ed ian score o f an
exam p le's output A m u st be equal to 4, and the m edian score o f the
corresp on din g output B m u st be equal to 3. T he CL rule's rew riting
had a m inor n eg a tiv e im p act on the com p reh en sib ility o f the M T
output, sin ce a m ajority o f evaluators rated the exam ple's output A
as 'E xcellent' and the exam p le's output B as 'Good'.
T he rem ainder o f th is chapter con centrates on the distribution o f CL rules based on
the im p act su g g ested b y their scores, in clu d in g their 'extra' scores.
4 .2 .2 .3 . D is t r ib u t io n o f C L r u le s b a s e d o n t h e i r s c o re s
A s ex p la in ed in the p reviou s section , gen eric scores for each CL rule are calculated
b y d iv id in g the sum o f each e x a m p le ’s raw score b y th e num ber o f exam p les
selec ted for ea ch rule. For instance, rule 4 8 , 'Use only ",because", never "since" in a
subclause o f reason', m en tion ed in A d riaen s and Schreurs (1992: 5 9 8 ), w as
evalu ated u sin g four ex a m p les. A total o f eigh t M T outputs w a s scored b y four
evaluators, so the resu lts for this ru le’s first ex a m p le lo o k as fo llo w s in German:
{ 2 ,2 ,2 ,1 } for M T output A (w ith a m ed ian o f 2 ), and { 2 ,2 ,2 ,2 } for M T output B
(w ith a m ed ian o f 2 ). T h is e x a m p le ’s raw score w a s therefore zero. T his zero figure
w a s added to th e ru le’s other three e x a m p le s’ raw scores (0, 0, and 0 .5 ) to produce a
g en eric score o f 0 .1 2 5 for th e C L rule in G erm an. On the other hand, the rule
p rovid ed a gen eric sco re o f 0 .6 2 5 in French. T he sam e approach w as u sed for all
ru les, and a sn ap sh ot o f th ese results is p rovided in F igure 4 -1 . T he scatter plot
98
diagram is not used to sh o w the evalu ation o f a variable over tim e, but rather to
com pare the distribution o f the scores per C L rule in each target language.
Generic score per CL rule
2
4
♦
1.5
♦
♦ ♦
♦
•
♦
^
*
1
*
0.5
0
•
"♦
«
*
■
,
t
- i
%
■
’
*
«
♦
♦“
♦
*♦
•
■
-05
*
.
.....
* * IV I* f i t 13' H «*■ H If T* t t I * i t IS 21 U 19 S t ( I i t i f J * ) |
* **
♦♦
♦
♦
♦ ♦
♦
■*
♦ French
■ G erm an
•
»
■
»
H 1» « 3»*» 414* *» * f O
HIIf i t <•> S t-11 « «
M
R a te n u m b e r
_____
Figure 4-1: Generic score per CL rule
A t first glan ce, C L ru les appear to b e m ore e ffe c tiv e for French than for German.
W ith th ese scores alon e, it is, h ow ever, im p o ssib le to advance w ith certainty any
exp lan ation s for this trend. T h e rules cou ld b e m ore effe c tiv e in French due to the
relative p oor q uality o f French M T output A com pared to that o f G erm an M T
output A . T h e y cou ld also b e m ore e ffe c tiv e b eca u se o f the inferior quality o f
G erm an output B com pared to that o f French output B . A com bin ation o f both
exp lan ation s is also p o ssib le. Furtherm ore, a rule m ay h a v e a rela tiv ely h ig h generic
score, but fail to return any p o sitiv e ’extra’ scores. T his w as the ca se w ith rule 13,
'Do not use the impersonal pa ssive voice'. T h is rule obtained a gen eric score o f 1 in
F rench, but n on e o f the three ex a m p les returned ’extra’ scores that sh ow a
substantial im p rovem en t in the com p reh en sib ility o f the output. T his d iscrepancy
b etw een gen eric and ’extra’ scores su g g ests that the latter scores should b e u sed as
th e m ain criterion to d ecid e on the e ffe c tiv e n e ss o f a particular rule. In order to
c la s s ify ru les, the fo llo w in g categories are g o in g to b e used:
■
R u les that h a v e a m ajor p o sitiv e im pact on the com p reh en sib ility o f
M T output in b oth lan gu ages
■
R u les that h a v e a lim ited p o s itiv e im p act on the com preh en sib ility
o f M T output in b oth lan gu ages
■
R u les that h ave no im p act in b oth langu ages
99
■
R u les that h ave a n eg a tiv e im pact in both lan gu ages
■
R u les that m o stly im p rove th e com p reh en sib ility o f M T output in
on e lan gu age pair
A fter calcu latin g 'extra' scores for ea c h rule, rules w ere p laced in on e o f th ese five
categ ories based on 'frequency' scores that w ere calcu lated out o f their French and
G erm an 'extra' scores. T he fo llo w in g form u la w a s u sed to calcu late the 'frequency 1
score o f each rule:
(((N u m b er o f p o sitiv e 'extra' scores o f category l)* 1 0 0 )/N u m b e r o f E xam p les) +
(((N u m b er o f p o sitiv e 'extra' scores o f category 2 )* 1 0 0 )/N u m b er o f E x a m p les)/2 (((N u m b er o f n eg a tiv e 'extra' scores o f category l)* 1 0 0 )/N u m b e r o f E xam p les) (((N u m b er o f n eg a tiv e 'extra 1 scores o f category 2 )* 1 0 0 )/N u m b er o f E x a m p les)/2
T h is form u la c o nfirm s that 'extra' scores o f category 2 w ere n ot rew arded in the
sam e m anner as 'extra' scores o f category
1
, so as to reflect the le v e l o f
co m p reh en sib ility im p rovem en t brought b y th e CL rule. S in ce th e present study w as
the first to u se th is typ e o f scorin g m ech a n ism , no p re-d efm ed category w as
a v a ilab le to d eterm ine th reshold s for rules w ith m ajor p o sitiv e im p act or rules w ith
lim ited im pact. It w a s therefore d ecid ed to populate th ese fiv e categories usin g
arbitrary criteria, w h ic h are d etailed in sec tio n s 4.3.1 to 4 .3 .6 .
4.3. A nalysis o f the results per CL rule
4 .3 .1 .
C L r u le s t h a t h a v e a m a jo r p o s itiv e im p a c t in b o th
la n g u a g e s
For a rule to be in itia lly con sid ered as h avin g substantial im pact on both French and
G erm an M T output, th e fo llo w in g requirem ent had to be met: the freq uency score
o f the rule m ust b e greater than 33 for both French and G erm an. A s m en tion ed in
the p reviou s sectio n , th is th reshold w a s ch o sen arbitrarily. T his threshold, h ow ever,
su g g ests that th e rule is very e ffe c tiv e at lea st every three segm en ts. T he 11 rules
that m et this requirem ent are d isc u sse d in turn to d eterm ine w hether th ese rules
sh ould rem ain in th is category. T h ese 11 rules are listed in T able 4-3 in no
particular order (the freq u en cy sco res h ave b een rounded up):
100
German
'extra'
score 1
German
’extra'
score 2
French
'extra'
score 1
(+ )
(+ )
(+ )
4
0
4
French
'extra'
score 2
(+)
0
54
6
2
6
1
33
33
1
0
1
0
4
50
62
0
2
1
Rule 3 5 : Avoid
unusual
punctuation.
2
50
SO
1
0
1
0
Rule 5: Do not
use m ore than
I 2 5 words p er
sentence.
3
33
100
1
0
3
0
Rule 1: Avoid
ambiguous
coordinations
b y repeating
the head noun,
o r by changing
the w ord
order.
8
37.5
37.5
3
Rule 22: Do
n o t coordinate
verbs o r verbal
phrases when
the verbs do
n o t have the
same
' transitivity.
9
56
56
4
Rule 4 1 : Avoid
em bedded
p arenthetical
expressions
introduced by
commas or
1dashes.
5
50
70
2
Rule 25: Two
p arts o f a
conjoined
sentence
should be o f
the sam e type.
3
67
Rule 4 8 : Use a
question m ark
only a t the end
o f a direct
question.
3
67
Rule number
and name
Number
of
examples
German
freq.
score
French
freq.
score
Rule 5 4 : Check
the spelling.
6
67
67
Rule 52: Use
single-w ord
verbs.
11
64
Rule 4: Do not
om it hyphens
in Noun +
A djective or
Noun + Past
Participle
structures.
3
Rule 19: Do
n o t use
pronouns th a t
have no
specific
referent.
*
2
0
'
33
3
4
1
3
1
1
0
2
0
1
0
2
0
2
Table 4-3: Rules with major positive impact in both languages
101
R u le 5 4 , 'Check the spellin g’, sh o w ed sign ifican t im p rovem en ts in both languages,
sin ce m issp elt w ord s such as 'Runing' or 'retailing' rem ained untranslated in the
target outputs. Systran 5.0's a u to -sp ellin g correction fu n ction , h ow ever, correctly
id en tified th e sp ellin g error in the fo llo w in g segm ent: 'This sou ld be "Permit AH'".
S uch results con firm that m issp elt w ord s affect the com p reh en sib ility o f the
corresp on din g M T output. E lim in atin g such errors in the source con tent is therefore
a priority, w h ic h w ill also im p rove th e cred ib ility o f sou rce d ocu m en ts. H am m erich
and H arrison (20 0 2 : 115) m en tion that 'typos and other errors su g g est that the
con tent m ay also be erroneous'.
T he results obtained for rule 5 2 , ’Use single-w ord verbs' con firm ed that w h en a verb
is u sed in con ju n ction w ith a particle, and this p article is an alysed as a preposition,
the sou rce input m ay b e parsed in correctly. Incorrect parses w ill then greatly affect
the com p reh en sib ility o f M T
output, esp ec ia lly w h en the particle d oes not
im m ed ia tely fo llo w th e verb, as sh o w n b y the fo llo w in g exam ple:
HI T his d ocu m en t d escrib es h o w to turn N orton A n tiV iru s 2 0 0 5 Internet W orm
P rotection on and off.
T h ese results con firm the recom m en d ation s o f Fuchs and S ch w itter (19 9 6 : 127),
M itam ura (1 9 9 9 : 4 7 ), and B em th and G dan iec (20 0 1 : 187). T he sco p e o f th is rule is
n ot lim ited to th e u se o f verbs w ith p articles. R ew ritin gs ad dressing co m p lex verbs,
su ch as 'get to work' returned 'extra' scores o f le v e l 1 in b oth French and Germ an
output w h en th ese co m p le x verbs w ere replaced w ith a sim p le verb su ch as 'use'.
O ne o f the rules operating at the le x ic a l le v e l is rule 4 , 'Do not omit hyphens in
Noun + Adjective or Noun + Past Participle structures'. T he im p lem en tation o f
su ch a rule w a s ju stified B em th and G dan iec (2001: 193), w h o realised that it w as
‘quite co m m o n (at least in U .S . E n g lish ) to om it the h yp h en b etw e en a noun and a
p o st-m o d ify in g past p a rticip le.’ M itam ura (1999: 4 7 ) also n oted that ‘hyphenation
should be co n sisten tly sp e c ifie d ’, w h ich appeared to be a m ore appropriate an gle to
stu d y the e ffe c tiv e n e ss
o f this rule,
sin ce p ost-m od ifiers can so m etim es be
a d jectives instead o f past p articip les, as in ‘v iru s-free’, or ‘industry-standard’. T his
rule w a s evalu ated w ith three ex a m p les and p rovid ed alm ost identical scores for
F rench and G erm an. It sh ou ld be p oin ted out that the h yphenated w ords had been
102
co d ed as dictionary entries to study the im p act o f the sligh t typ ograp h ical change
introduced b y the hyphen. T he rule o n ly p roved e ffe c tiv e for exam p le 18, w h ich
con tain ed an ad jective as p o st-m o d ifier, 'virus free'. T he other tw o exam p les used
(e x a m p le s 19 and 2 0 ), w ere properly h andled b y the M T system , so the rew ritings
p ro v ed o tio se. B a sed on th ese m ix e d results, it is d ifficu lt to gen eralise but the
sig n ifica n t im pact o f the rew riting for ex a m p le 18 (w ith tw o 'extra' scores o f lev el
1) su g g ests that th is rule is eq u ally im portant for both French and Germ an.
T he n ex t rule to p rovid e in d ication s o f sig n ifica n t im p rovem en ts in both French and
G erm an con cern s a sp ecific part o f the rule p rohibiting the u se o f pronouns. R u le 19,
'Do not use pronouns that have no specific referent', ec h o e s on e o f the C O G R A M
ru les ("It" m u st refer to a p reced in g noun. A d riaen s & Schreurs, 1995: 130). T his
rule w a s evalu ated u sin g four ex a m p les and the rew rites clearly im p roved the
co m p reh en sib ility o f b oth M T outputs. P ronouns w ith no sp e c ific referent, such as
'it' in 'm akes it ea sy t o ...', w ere n ot h andled correctly b y the M T system in both
la n g u ages. R eferential am b igu ity w a s also introduced in ex am p les 114 and 117, in
w h ich p ron oun s w ere p laced after a noun th ey did not refer to. From a CL
im p lem en ta tio n p ersp ective, h o w ev er, id en tify in g th ese sp ecific ca ses o f referential
a m b igu ity is boun d to be a d ifficu lt task for a CL ch ecker w ith ou t any sem antic
a n alysis.
T he d escrip tion o f rule 35 m ay sou nd v ery gen eric, e sp ec ia lly as it w as on ly
evalu ated
w ith
tw o
exam p les:
'Avoid unusual punctuation'. T w o
exam p les
co n ta in in g unu su al p un ctuation u sage w ere fou n d in the corpus w h en extracting
sen ten ces for other rules. It w a s d ecid ed to test this rule as a separate rule to
ev a lu ate the im p act o f su ch v io la tio n s. E ach segm en t returned an 'extra' score o f
le v e l 1 either for French or G erm an. For in stan ce, the original segm en t o f exam p le
2 0 3 con tain s d ash es that are u sed to insert explanatory com m en ts in the m id dle o f a
sen tence: ‘T y p e —or co p y and p aste—th e fo llo w in g file n a m e s’. S in ce the dashes are
n o t separated b y sp a ces, an an alysis p rob lem occurred during the translation process,
and th e first w ord o f the sen ten ce w a s to k en ised a lo n g sid e th e tw o fo llo w in g dashes.
T he e ffe c tiv e n e s s o f th e corresp on d in g rew rite w as, h o w ev er, m ore sign ifican t in
G erm an and F rench due to u n exp ected n o ise created b y the m istranslation o f the
h om ograp h 'copy' in French. In the other exam p le, w h ere a com m a had been
103
incorrectly p la ced before a noun, the ex a ct sam e prob lem happened in G erm an. The
effe c tiv e n e ss o f the rew rite w a s blurred by the m istranslation o f the hom ograph
'copy' as a verb instead o f a noun. T h is d etailed an alysis w a s h elp fu l in id en tifyin g
tw o clear ca ses o f punctuation u sage that sh ou ld be avoid ed b efore an M T process.
R ule 5, w h ich lim its the num ber o f w ord s to 2 5 , w as evaluated w ith three exam p les
on ly. It m u st be said that n o standard rew rites ex ist for this rule, so the evaluation
results depend h ea v ily on the ex a m p les that are ch osen to test this rule. O ne could
ev en argue that su ch a rule is to o v a g u e to generate con sisten t im p rovem en ts, but i f
no restriction is p laced on sen ten ce len gth , co m p lex sen ten ces su ch as the fo llo w in g
m ay b e found:
I f y o u h a v e W in d o w s M e and y o u h ave run W in d o w s S y stem R estore and
y o u see the error m e ssa g e "U nable to in itialize the virus scan n in g engine"
b efore y o u see error (3 0 1 9 ,6 ), th en fo llo w the instructions in the d ocu m en t
Error:
"U nable to in itia lize virus scanning en gin e.
. ." after running
W in d o w s S ystem R estore or in stallin g N orton A n tiV iru s 2 0 0 3 /2 0 0 4 .
T his rule therefore sh o w ed clear sig n s o f im p rovem en ts in b oth French and Germ an,
w ith a co m m o n 'extra' score o f le v e l 1 obtained by exam p le 2 1 . It sh ould b e said,
h o w ev er, that n ot all sen ten ces that are com p lian t w ith this rule w ill produce
com p reh en sib le output, e sp e c ia lly i f essen tial w ord s are om itted due to co n cisen ess.
T h is issu e w ill b e further d isc u sse d w h en the rules govern in g the u se o f e llip se s are
d iscu ssed . D e sp ite th ese lim itation s, redu cin g sen ten ce len gth is crucial. T his w as
con firm ed by the fact that th is rule w a s the o n ly on e in the set o f 54 to return
p o sitiv e 'extra' scores for all three ex a m p les in French.
S ig n s o f clear im p rovem en ts w ere also obtained w ith rule 1, 'Avoid ambiguous
coordinations by repeating the head noun, or by changing the w ord order'. T his
rule, w h ich b u ild s on Systran's recom m en d ation to w rite co m p lete com p ou n d s w hen
p o ssib le (Systran, 2005: 176), w a s evalu ated w ith eigh t exam p les. B em th (1 9 9 8 :3 3 )
a lso states that ‘a real am b iguity occurs w h en a con join ed noun phrase p rem od ifies
a noun. In th is ca se th e sco p e o f the p rem od ification can be h igh ly a m b ig u o u s’. This
see m s to be esp e c ia lly true w h en the coordin ation is punctuated w ith com m as. Out
o f the eig h t ex a m p les u sed to test this rule, tw o returned 'extra' scores in both
104
F rench and G erm an. E xam p le 2 p rovid ed the b est result for both lan gu ages, w h ich
returned an 'extra' score o f le v e l
1
, sin c e sev e n evaluators thought that the fo llo w in g
ex a m p le w as im p roved by the rewrite:
El On the S ch ed u le tab, fill in the S ch ed u le Task, Start tim e, and S ch ed ule
T ask D a ily field s.
A
Sur l'on glet P lan ification , c o m p lé te z le tem p s de tâche, de Dém arrer de
program m e, et les cham ps T âch e q u otid ien n e p lan ifiée.
E l On the S ch ed u le tab, fill in th e S ch ed u le T ask field , the Start tim e field , and
the S ch ed u le T ask D a ily field .
B
Sur l'on glet P lan ification , co m p lé te z le cham p T âche p lan ifiée, le cham p
D ébu t, et le cham p T âch e q u otid ien n e p lan ifiée.
In the origin al ex a m p le, 'fields' is a h ead n ou n u sed at the end o f the sen ten ce to
en co m p ass th e three op tion fie ld s that m u st b e com p leted b y users. T his type o f
structure m ay n ot b e to o am b igu ou s for a hum an reader, ev en th ou gh the end o f the
sen ten ce has to b e reach ed b efore it can be fu lly understood. H o w ev er, the French
output A sh o w s that the sou rce structure created an alysis p rob lem s for the M T
sy stem . T he translation 'champs' in M T output A w as not reco g n ised as b ein g the
head o f several com p ou n d s, w h ic h w a s om itted tw ic e to avoid repetitions. This
d ecisio n w a s tak en to m ake the ST as c o n c ise as p o ssib le. C o n cisen ess, h ow ever
o ften lead s to am b igu ity. I f w ord s are m issin g in th e ST, M T sy stem s m ust gu ess
w h eth er certain w ord s ou ght to b e restored in the TT. T his issu e is h igh ligh ted b y
B ernth and G d an iec (2 0 0 1 : 2 0 7 ) w h o stress that the ‘o m issio n o f inform ation, be it
e llip sis or m issin g sen ten ce d elim iters, ca u ses the M T sy stem to go into w h at (they)
m igh t call “ g u e ssin g m o d e ’” . A s sh o w n w ith the p reviou s exam p le, this strategy
p ro v es c o stly in sp e c ific cases. N o t o n ly is the syn tax o f the TL affected , but the
F rench M T output A sh o w s that certain u ser dictionary entries w ere n ot used during
the translation p rocess.
U sin g
ellip tical structures m ay h ave the unfortunate
co n seq u en ce o f n egatin g dictionary w ork p erform ed prior to the M T process.
It is a co m m o n dictionary p ractice to separate head n oun s from m od ifiers. Petrits
(2 0 0 1 : 9) m en tio n s that the E uropean C om m ission 's d iction aries are d ivid ed into
‘d iction aries for in d ivid u al w ord s and d iction aries for e x p re ssio n s’. T his strategy,
w h ich a v o id s th e d u p lication o f d iction ary w ork, can be im p lem en ted in Systran 5.0
105
w ith the u se o f looku p fin d operators (Systran, 2 005: 136). For instance, a m ain user
dictionary m ay contain an entry for th e w ord 'field' p reced ed by a looku p operator.
T his look u p operator w ill p oin t to any ex p ressio n con tain ed in th e referen ce
d ictionary. 'Schedu le Task', 'Start tim e', and 'Schedu le T ask D aily' w o u ld all be
includ ed in the referen ce dictionary. T h is tech n iq u e has tw o advantages: Firstly,
d up licate w ork is avoid ed , b ecau se there is no need to h ave both 'Schedu le Task
field' and 'Schedu le Task' as separate entries. S eco n d ly , the translation o f the
syn tactic relationsh ip b etw een the head noun 'field' and its p rem odifiers can
overw rite the M T s y s te m ’s default com p ou n d in g rule w h ich m ay n ot be appropriate
in certain ca ses. For instance, the p rep o sitio n 'de' w o u ld be added b y d efau lt to link
the head 'champ' w ith th e p ostm od ifier ex p ressio n corresp on din g to 'Schedu le Task'.
In this particular exam p le, this p rep osition is superfluous and w o u ld h ave to be
rem o ved during the p o st-ed itin g p rocess. C on sid erin g th e freq uency o f th ese
com p ou n d in g
structures in tech n ical
con ten t in w h ich
softw are
op tion s
are
d escrib ed , it see m s c o st-e ffe c tiv e to avail o f this referen cin g dictionary strategy.
T yp ical n om in al groups includ e w ords su ch as 'tab', 'menu', 'list', 'folder', or 'box',
b ein g p reced ed b y u ser interface op tion s su ch as 'Schedule Task', 'V iew ', 'Save as',
or 'Scan now '. T h ese op tion s are u sed as ep ith ets to m o d ify the head n ou n d esp ite
the fact that th ey con tain verbs. B lo o r & B lo o r (2004: 139) m en tion that 'the
fu n ction o f M od ifier can be realised b y variou s w ord cla sses, m o st frequently by
determ iners, num erals and ad jectives as Prem odifiers'. W h en it co m es to tech nical
support d ocu m en tation , and m ore gen erally softw are docu m en tation , verbs m u st be
added to this list, as sh o w n in the ab ove ex a m p le w ith ‘S ch ed u le Task'.
R u le 2 2 deals w ith th e u se o f coordinated verbs that do not share the sam e valen cy.
O ne o f th em m ay be u sed in a direct transitive w a y , w h en the other is used
ind irectly w ith a p rep osition . T his rule sh ould therefore be regarded as a sp ecific
part o f the rule su g g estin g that coordinated verbs should b e avoided (M itam ura,
1999: 4 7 ). C oordin ations are in d eed n eg a tiv e translatability indicators u sed in the
L o g o s translatability in d ex (G d aniec, 1 994). R u le 22 returned a very h igh num ber
o f 'extra' scores o f le v e l 1 (three for French and four for G erm an) and le v e l 2 (four
for F rench and for G erm an), sh o w in g that the com p reh en sib ility o f M T output can
be sig n ific a n tly im p roved w h en the ob ject is inserted after the first verb, and a
106
pronoun is u sed after the secon d . T he fo llo w in g exam p le (exam p le 133) returned
'extra' scores o f le v e l 2 in both French and G erm an:
0
S croll to and u n ch eck the fo llo w in g entry.
A
A lle z à et d é selec tio n n ez l'entrée suivante.
A
Blättern S ie z u und d eak tivieren S ie d en fo lg e n d e n Eintrag.
E l S croll to the fo llo w in g entry and u n ch eck it.
B
A lle z à l'entrée su ivan te et d éselectio n n ez-la .
B
B lättern S ie zu m fo lg en d en E intrag und d eak tivieren S ie ihn.
T he results ob tained for rule 2 2 m u st b e contrasted w ith th o se obtained for rule 23.
R u le 23 w a s evalu ated w ith 3 ex a m p les, w h ich did n ot return any 'extra' scores. T he
rew ritings w ere in e ffe c tiv e b eca u se the origin al sen ten ces p roduced p erfectly
co m p reh en sib le translations. B a sed on th ese results, it is p o ssib le to con clu d e that
the sco p e o f th e v er y restrictive origin al rule con cern in g the u se o f coordinated
verbs sh ould b e red u ced to that o f rule 2 2 . T his recom m en d ation sh ould prove
effe c tiv e as lo n g as target lan gu ages are able to translate sou rce transitive verbs in a
sim p le transitive w a y , w ith ou t resorting to phrasal verbs. A n exam p le o f a potential
p rob lem is fou n d in ex a m p le 72 in French w ith th e translation o f the verb
'bookmark'.
R u le 4 1 , 'Avoid em bedded parenthetical expressions introduced by commas or
dashes', b u ild s o n Systran's recom m en d ation to 'avoid m u ltip le stacking' (Systran,
2005:
174). D ep en d en t cla u ses are also u sed b y G d an iec (1 9 9 4 ) as n egative
translatability ind icators. W h ile th is rule m ay b e regarded as an M T sy stem -sp ecific
rule, its rew ritin gs attem pt to separate lo n g sen ten ces into shorter on es. T h is typ e o f
approach is b ased o n th e reu se p h ilo so p h y d isc u sse d in sectio n 1.4.2.2 o f Chapter 1,
and e x e m p lifie d b y ex a m p le 227:
IE] I f y o u can n ot ch a n g e th e drive letter, as a w orkaround to the prob lem , you
can u se th e W in d o w s S u b st.ex e u tility to tem porarily d esign ate a fold er as
th e c: drive.
E l Y o u can n ot ch an ge th e drive letter. A s a w orkaround to this p roblem , y ou
can u se th e W in d o w s S u b st.ex e u tility to tem porarily d esign ate a fold er as
th e c: drive.
107
T he u se o f tw o sim p le seg m en ts in th e con trolled exam p le ensured that 'extra' scores
o f le v e l 1 w ere ob tain ed in both French and G erm an. B reaking sen ten ces into tw o
seg m en ts w a s n ot a lw a y s p o ssib le, but a reordering o f the original sentences'
elem en ts im p roved th e com p reh en sib ility o f the corresponding M T outputs. T his
w a s the ca se w ith ex a m p le 230:
[HI A clean b o o t is sim ilar to, but m ore th orou gh than, clo sin g all applications.
0
W h ile
A clean b o o t is sim ilar to c lo sin g all ap p lication s but it is m ore thorough.
the
translations
o f the
con trolled
ex a m p le
w ere
n ot
p erfect,
their
co m p reh en sib ility w a s im p roved w ith the rem oval o f the interpolated clau se and the
introduction o f a pronoun. T his ex a m p le su g g ests o n ce m ore that th e u se o f
p ronouns d o es n ot n ece ssa rily h ave a n eg a tiv e im p act on the quality o f the M T
outputs as lo n g as th ey h a v e clear referents.
T he e ffe c tiv e n e ss o f rule 2 5 , T w o p a rts o f a conjoined sentence should be o f the
same type', w a s evalu ated w ith three ex a m p les. T w o p o sitiv e 'extra' scores o f le v e l 1
w ere ob tained for F rench, but o n ly on e w a s obtained for Germ an. T he results
ob tained for th is rule su g g ests that im p rovem en ts can h ave p o sitiv e im p act in both
langu ages.
R u le 4 8 , 'Use a question m ark only at the end o f a direct question' is ex p licitly
m en tio n ed b y A d riaen s and Schreurs (19 9 2 : 5 9 5 ). Such a rule is ju stified by the fact
that the overall a n a ly sis o f sou rce input can b e a ffected i f the M T sy stem attributes
en d -o f-sen ten ce properties to q u estion m arks appearing in the m id d le o f segm en ts.
O ne can ea sily understand w h y tech n ical w riters m ay d ecid e to inclu d e punctuation
m arks in the m id d le o f sen tences; th ey w an t their d escrip tion to m atch ex a ctly the
con tent o f the U ser Interface (U I), w h o se m en u s m ay contain direct q u estion s, as
sh o w n w ith th e fo llo w in g exam ple:
0
In the "W hat do y o u w a n t to call th is rule?" field , type M essen g er S ervice
and c lic k N e x t.
A
In d em „ W ie m o ch ten S ie d iese R e g e l n en n en ??“ Feld, T yp M essen ger
S erv ice u nd k lic k e n S ie a u f W eiter.
108
E3 In the "What do y o u w ant to call th is rule" field , type M essen g er S ervice,
and click N e x t.
B
Im „W ie m o ch ten S ie d iese R e g el n en n en ?“ F eld g eb en S ie M essen ger
S ervice ein und k lick en S ie a u f W eiter.
In this exam p le (ex a m p le 2 6 5 ), three out o f four G erm an evaluators th ou ght that the
com p reh en sib ility o f the M T output w a s im p roved to su ch an exten t that it w ou ld
b eco m e understandable for users. T his w a s p rob ably b eca u se the sen ten ce w as
an alysed as on e seg m en t instead o f tw o w h en th e q u estion mark w a s rem oved . It is
interesting to n o tice that q u estion m arks w ere au tom atically inserted in both
G erm an outputs. T h is, h o w ev er, w a s n ot th e ca se in th e French outputs, w h ich
su g g ests that d ifferen t pre- or p o st-p ro ce ssin g ru les m ay b e u sed b y th e M T system
d ep en d in g on the target langu age. T his assu m p tion can n ot be verified w ith ou t any
a cc ess to the system 's internal rules. In French, the effe c tiv e n e ss o f the rule w a s not
v isib le , sin ce M T output B w a s a ffected b y th e m istranslation o f the verb 'type',
w h ich w a s an alysed and translated as a noun. R e so lv in g am b iguity issu e s created by
h om ograp h s w ill h ave to be taken into accou n t in th e n ex t part o f the study.
T h is rule w a s th e la st rule c la ssifie d as h a v in g a m ajor p o sitiv e im pact in both
la n gu ages. S ectio n 4 .3 .2 d isc u sse s th e rules that h a v e a lim ited im pact on French
and G erm an outputs.
4 .3 .2 .
C L r u le s t h a t h a v e a lim i t e d p o s itiv e im p a c t in
b o t h la n g u a g e s
In order to q u alify as a CL rule w ith a lim ited p o s itiv e im p act on b oth French and
G erm an outputs, th e fo llo w in g requirem ent had to b e m et: the freq uency score o f
th e rule is greater than 0 in b oth lan gu ages but le s s than 33. T he threshold for this
freq u en cy score w a s se le c te d to reflect the fact that the rule returns 'extra' scores
infrequently. T able 4 -4 sh o w s the 11 rules that fe ll into th is category (in no
particular order).
109
German
'extra'
score 2
(+)
French
'extra'
score 1
(+)
0
French
'extra'
score 2
(+ )
2
1
0
1
0
14
2
0
1
0
17
22
1
1
1
2
7
21
14
1
1
0
2
Rule 5 1 : Avoid
-ing words.
8
25
25
2
0
1
2
Rule 2 0 : Do
n o t use an
ambiguous
form o f 'could'.
5
10
20
0
1
1
0
7
14
14
1
0
0
2
Rule 9: Do not
o m it relative
pronouns and
a form o f the
verb 'be' in a
re lativ e clause.
The postm od ifier is a
past-participle.
8
12.5
12.5
1
0
1
0
Rule 2: Do n ot
use ambiguous
attachm ents o f
n o n-finite
clauses
(p res en tparticiples).
4
25
25
1
0
1
0
Rule number
and name
Number
of
examples
German
freq.
score
French
freq.
score
Rule 53: Avoid
ungram m atical
constructions.
4
12.5
25
Rule 3 6 : The
subject o f a
non-finite
clause m ust be
the sam e as
■th a t o f the
m ain clause.
4
25
25
Rule 14: Do
n o t m ake noun
clusters o f
m ore than
three nouns.
7
29
Rule 17: Do
n o t use a
pronoun when
the pronoun
does n o t refer
to the noun it
im m ediately
follows.
9
Rule 18: Do
use pronouns,
even when the
pronoun
im m ediately
follows the
.noun it refers
to.
,Rule 2 8 : Omit
German
; 'extra'
score 1
(+)
0
unnecessary
' words.
110
Rule number
and name
Rule 29: Place
a purpose
clause before
the m ain
clause.
Number
of
examples
German
freq.
score
French
freq.
score
4
12.5
25
German
extra'
score 1
(+>
0
German
'extra'
score 2
(+)
1
French
'extra'
score 2
(+)
French
extra'
score 1
(+)
1
1
°
Table 4-4: List of rules with limited positive impact in both languages
R u le 5 3 , w h ich proscrib es th e u se o f ungram m atical con stru ction s, returned overall
results that w ere sim ilar in b oth lan gu ages. T his rule w a s evalu ated in the fo llo w in g
scenarios:
■
the m isu se o f th e p o s se s siv e ‘it s ’ or g en itiv e m arker ‘s ’. T w o
ex a m p les w ere fou n d in the corpus: ‘C annot locate ccS etM g r.ex e or
on e o f it's c o m p o n en ts’ and ‘the program s tech n ica l support’
■
the la ck o f su bject/verb
agreem ent,
an exam p le b ein g
‘T h ese
c o n flicts appears to o n ly happen w h en th e com puter is starting.’
T h e results su g g est that th e introduction o f th is rule had m ore im p act on the w e llform ed n ess o f the output than o n its com p reh en sib ility. O ne o f the ex am p les u sed to
test the e ffec tiv en ess o f proper agreem ent b etw e en su bject and verb ev e n did not
h ave any effe c t o n G erm an output, w h ereas it did for F rench (exam p le 296):
HI W in D octor and O ne B utton C heckup fo llo w strict g u id elin es for w hat they
con sid ers v a lid or invalid .
Ä
W in D o cto r u nd O ne B u tton C heckup fo lg e n strengen K orrekturlinien für,
w a s sie gü ltig oder u n g ü ltig betrachten.
A
W in D octor et O ne B utton C heckup su iven t le s d irectives strictes pour ce
qu'ils con sid ère v a lid e o u n on -valid e.
0
W in D octor and O ne B utton C heckup fo llo w strict gu id elin es for w h at th ey
con sid er va lid or invalid .
B
W in D o cto r und O ne B u tton C heckup fo lg e n stren gen K orrekturlinien für,
w as sie g ü ltig oder u n gü ltig betrachten.
B
W in D octor et O ne B utton C heckup su iven t le s d irectives strictes pour ce
qu'ils con sid èren t v a lid e ou n on -valid e.
111
A s d isc u sse d in sectio n 3 .3 .1 .1 , it is im p o ssib le to exp lain w h y a com m on source
a n a ly sis prob lem prod uced d ifferen t outputs w ith ou t h avin g a cc ess to the rules o f
the M T system .
R u le 3 6 , The subject o f a non-finite clause must be the same as that o f the main
clause', w a s evalu ated w ith four ex a m p les. T his rule is b ased on the gram m aticality
issu e raised b y B em th and G d an iec (2001: 178). O n ly on e ex a m p le returned an
'extra' score o f le v e l 1 in each target langu age. For French, it w a s exam ple 2 0 4
w h o se n on -fin ite cla u se w a s introduced by a past p articiple. For G erm an, it w as
ex a m p le 2 0 6 , w h o se n o n -fin ite cla u se w a s introduced b y an - i n g w ord. A careful
a n a ly sis
o f th ese
four
ex a m p les
revealed
that lin g u istic
n o ise
m asked
the
e ffe c tiv e n e ss o f the rule in G erm an (for exam p le, 'the d ia lo g b o x graphic m oves'
w a s m istranslated in ex a m p le 2 0 5 ). In three o f the G erm an outputs A (exam p les 204,
2 0 5 , and 2 0 6 ), a p erson al pron oun w as inserted in the cla u se p reced in g the m ain
cla u se b ased on the su bject o f th e m ain clau se. T his in sertion did not appear in
F rench outputs A sin c e an eq u ivalen t n on -fm ite cla u se can be also be used
a m b ig u o u sly in French. T he ch a llen g e in preserving am b igu ity in target languages
has b een d escrib ed b y A rn old (2 0 0 3 : 125), w h o b e lie v e s that source am biguity
cannot b e ignored in the sam e w a y in all langu ages. B ased on th is m icro-an alysis o f
th ese fou r seg m en ts, it can be co n clu d ed that th is rule is m ore effe c tiv e for Germ an
than for French w h en the subject o f a n on -fin ite cla u se introduced b y an - in g w ord
is n o t th e sam e as th e su bject o f th e fo llo w in g m ain clau se. W h en the n on-fm ite
cla u se is introduced b y a p ast p articip le, the rule is a lso e ffe c tiv e for the E nglish F rench lan gu age pair.
R u le 14, 'Do not make noun clusters o f more than three nouns' w a s evaluated w ith
sev e n ex a m p les and returned fairly identical gen eric scores for French and Germ an
(0 .6 ). T h is rule is o ften u sed in C L rule sets b ecau se o f the poten tial d ifficu lty for an
M T sy stem (and hum an readers) to determ ine the gram m atical relationsh ips existin g
b etw e en the elem en ts o f a lo n g string o f nouns. W hen co n n ectin g w ords are m issin g
b etw e en noun s, am b igu ity m a y b e introduced b y h om ographs. T his is sh ow n by the
pre-C L segm en t o f ex a m p le 79:
M N orton Internet W orm P rotection features
112
In th is pre-C L segm en t, 'features' cou ld b e an alysed as a n ou n or as a verb by an
M T system . In this particular case, the n ou n form w a s selected b y th e M T system so
the introduction o f the rule did n ot h ave an y effect:
A
F on ction n alités de N orton Internet W orm P rotection
EZI Features o f N orton Internet W orm P rotection
B
F on ction n alités de N orton Internet W orm P rotection
Several ty p es o f rew ritings e x ist for this particular rule, d ep en d in g on the
gram m atical relationsh ips o f the elem en ts o f the cluster. For instance, tw o typ es o f
g en itiv e structures can be used, either the S axon g en itiv e (ex a m p le 81) or the
N orm an g en itiv e (ex a m p le 79). O ther p rep osition s are so m etim es required, as
sh o w n w ith the introduction o f'fo r ' in ex a m p le 76. T he results obtained for th is rule
are therefore
lim ited
to
the
ex a m p les
selec ted
in
this
test
suite.
M aking
g en eralisation s for th is rule is d ifficu lt b eca u se am b iguou s p rep osition s, su ch as
‘a g a in st’ or ‘t o ’, m ay so m etim es b e u sed to split clusters. T h ese p rep osition s m ay
be translated d ifferen tly d ep en d in g on the co n tex t in the target langu age, so their
introd uction m ay produce an M T output that is n ot n ecessarily better than the
d efault output p roduced b y a g iv e n M T system . E xam p le 78 su g g ests that the
system 's default com p ou n d in g m ech a n ism w a s m ore e ffe c tiv e in French than in
G erm an.
T he
rule
introduced
in
th is
exam p le
had
le ss
im p act
on
the
co m p reh en sib ility o f th e French output than the G erm an output. T his exam p le
returned an 'extra' score o f le v e l 1 in G erm an, w hereas little im p rovem en t w as
v isib le in French.
R u le 17, 'Do not use a pronoun when the pronoun does not refer to the noun it
immediately fo llo w s' w a s evalu ated w ith 9 ex a m p les, and returned sim ilar gen eric
scores in F rench and G erm an. T he num ber o f 'extra' scores w a s also fairly
co n sisten t across the tw o lan gu ages. E xam p le 99 is w orth d iscu ssin g b eca u se it
con tain s a ca se o f referential am b iguity that w as w ell-h a n d led by the M T system .
0
I f an in fectio n is found, y o u w ill be sent to a to o l that rem o v es it.
T he pron oun 'it' con tain ed in the pre-C L seg m en t d oes not refer to the p reced in g
noun, but the correct translation o f the p ronoun w a s p roduced by the M T sy stem in
both French and G erm an. H o w ev er, both groups o f evaluators th ou ght that th e M T
113
output w a s m ore com p reh en sib le w h en a fu ll form o f the noun w a s repeated in the
translation. T his result su g g ests that th e w e ll-fo rm e d n ess o f a sen ten ce d oes not
ensure its com p reh en sib ility. O verall, o n e cannot con clu d e that the e ffec tiv en ess o f
this rule is v isib le on a con sisten t b a sis for the tw o langu ages evaluated. In certain
ca ses, th e M T sy stem perform ed the right d isam bigu ation , and in others, the
am b igu ity w a s accep tab le in th e target lan gu age. Such results w ere also noted w ith
rule 18, 'Do not use pronouns, even when the pronoun immediately fo llo w s the noun
it refers to'. S in ce the sty le o f the sou rce text m ay b eco m e very repetitive and
v erb o se w ith the re-introd uction o f n oun s, the ap plication o f su ch rules w ill depend
on the en viron m en t in w h ich the C L is d ep loyed .
A sim ilar co n clu sio n w a s also ob tained w h en exam in in g the results obtained for
rule 5 1 , 'Avoid -ing words'. D u e to the extrem e am b iguity associated w ith - in g
w ords, it w a s d ecid ed to evalu ate the e ffe c tiv e n e ss o f this rule u sin g
8
exam p les.
T he sco p e o f th is rule had to d iffer from ru les 2, 3, 3 8 , and 4 6 . T he French generic
score w a s h igh er than th e G erm an gen eric score, but 'extra' scores w ere m ore
co n sisten t across both lan gu ages, su g g estin g that the effe c tiv e n e ss o f this rule did
n ot favou r a particular target lan gu age. A s sh ow n by ex a m p le 2 7 7 in French, usin g
tw o - i n g w ord s in su c c e ssio n can a ffect the com p reh en sib ility o f th e M T output.
T his ex a m p le w a s b ased on a title found in the corpus, 'Preparing for installin g
N o rto n S ystem W ork s 2005'. 'Ing' w ord s are, h ow ever, very often used in h eadings,
so im p lem en tin g this rule in a system atic w a y w o u ld im pact the accep tab ility o f the
sou rce text. O ne p o ssib le so lu tio n to this prob lem m ay be to lim it the num ber o f
'ing' w ord s to o n e per sen ten ce, or to ensure that th ey do n ot occu r after certain
p rep o sition s (as sh o w n b y the e ffe c tiv e rew ritin gs for ex a m p les 281 and 2 8 2 for
French). In su ch ca ses, h o w ev er, th e rew rite m u st be handled correctly b y the M T
sy stem , w h ich w a s n ot th e ca se w ith ex a m p le 2 8 3 , w h o se p ost-C L segm en t w as
m ore co m p le x than th e pre-C L segm en t. S p e cific 'ing' w ords m ay also be targeted,
su ch as 'follow in g' w h en it is syn o n y m o u s w ith 'after' (as sh ow n by exam p le 2 7 9
w h ich returned an 'extra' score o f le v e l 1 in G erm an). U sin g such an approach w ill
be co n sid ered in the n ex t part o f this study.
R ule 2 0 , 'Do not use an ambiguous form o f "could", w a s evaluated w ith five
ex a m p les, w h o s e pre-C L seg m en ts con tain ed a form o f 'could' that w a s u sed w ith a
114
h y p oth etical m eanin g. O nly tw o o f th ese ex a m p les returned p o sitiv e 'extra' scores in
either French or G erm an (exam p les 121 and 119 resp ectiv ely ). A m icro an alysis o f
the seg m en ts sh o w ed that past form s o f 'can' w ere generated m ore often in G erm an
M T output A ('konnte' or 'konnten' instead o f the con d itional form , 'konnte' or
'lconnten') than in French. It is, h o w ev er, im p o ssib le to determ ine w h y th ese
d ifferen ces occurred (in ex a m p le
1 2 0
, for instance).
R u le 2 8 , 'Omit unnecessary words' m u st n ot b e con fu sed w ith rules 2 6 or 27. W h ile
the rules restricting e llip se s attem pt to p reserve k ey w ords, rule 2 8 attem pts to rid
sou rce tex t o f redundant or ta u to lo g ica l w ords. T his rule w a s evaluated w ith sev en
ex a m p les and returned con sisten t gen eric scores in French and G erm an. W ith the
rem oval o f sin g le w ord s such as ‘ju s t’ or ‘b a s ic ’, or lon ger exp ressio n s su ch as ‘and
so o n ’ w h ich are v o id o f any real m ean in g, the com p reh en sib ility o f M T output w as
im p roved . It is crucial to fin d th e right b alan ce b etw een k eep in g w ords that w ill
h elp the M T sy stem d isam bigu ate sou rce input, and rem ovin g th o se that w ill im pact
n e g a tiv e ly on the M T output.
R u le 9, 'Do not omit relative pronouns and a form o f the verb "be" in a relative
cla u se w h en th e p o st-m o d ifier is a past-participle.' T his rule w a s evaluated w ith
8
ex a m p les but o n ly o n e ex a m p le per lan gu age sh ow ed sig n s o f e ffec tiv en ess,
ex a m p le 4 0 for G erm an, and ex a m p le 4 2 for French. M ak ing gen eralisation s for this
rule is d ifficu lt but it is p o ssib le to state that th e p o ly se m o u s nature o f the verbs
u sed in th ese sen ten ces, 'caused' and 'made', m ay h ave p rovok ed an alysis problem s
for the M T sy stem w ith the pre-C L segm en ts.
D u rin g th e evalu ation o f rule 2 , 'Do not use ambiguous attachments o f non-fmite
clauses (present-participles)', o n ly on e 'extra' score o f le v e l
1
w a s returned in both
lan g u ages w ith ex a m p le 12. A fter perform ing a m icro-an alysis o f the four exam p les
w ith w h ich the rule w a s evalu ated , it appeared that the im p rovem en t introduced by
the rule in th is ex a m p le cou ld h ave b een in flu en ced b y tw o factors. It cou ld have
b een in flu en ced b y the verb that w a s present in the n o n -fin ite clau se. B ut it also
co u ld h ave b een in flu en ced b y th e typ e o f clau se that the am b iguou s - i n g w ord w a s
introducing. T hree ex a m p les con tain ed a clau se o f m am ier introduced b y the
p o ssib ly am b igu ou s verb 'using'. T his verb w as parsed and translated properly in all
115
three cases. O n the other hand, all evaluators agreed that the rew riting introduced in
ex a m p le 12 im p roved the co m p reh en sib ility o f the M T output. T he pre-C L segm en t
o f th is ex a m p le w as different from th e other three, sin c e it contained an - i n g w ord
introducing a n o n -fm ite p u ip o se clau se. T his result con firm s that the ch o ice o f
ex a m p les u sed to evalu ate the e ffe c tiv e n e ss o f CL rules can h ave som e im pact on
the final form alisation o f a rule. I f ex c e p tio n s are not added to a rule w h en such a
rule is form alised , the rule m ay b e u sed m ore often than it should be.
T he last rule to sh o w lim ited im p act in both lan gu ages is rule 2 9 , ’P lace a purpose
clause before the main clause'. T h is rule w a s evalu ated w ith four ex am p les that
con tain ed purpose cla u ses introduced b y in fin itiv e verbs. T h ese cla u ses w ere m oved
into first p o sitio n in th e p ost-C L seg m en ts. O n ly on e ex a m p le sh o w ed clear signs o f
im p rovem en t in both lan gu ages w h en th e rule w a s applied, ex a m p le 179:
HI F o llo w the instructions in th is sectio n to repair th e Intrusion D e tectio n files.
A
S u iv e z les instructions dans cette sec tio n de réparer les fich iers de d étection
d'intrusion.
A
B e fo lg e n S ie d ie A n w eisu n g e n in d ie sem A b sch n itt, die Intrusion D etection D a te ien zu reparieren.
0
T o repair the Intrusion D e te c tio n file s, fo llo w the instructions in this section.
B
Pour réparer le s fich iers de d étectio n d'intrusion, su iv e z les instructions dans
cette section .
B
U m d ie Intrusion D e te c tio n -D a te ie n
A n w eisu n g e n in d iesem A b sch nitt.
In th is
ex a m p le,
the
p rep osition
zu
reparieren,
‘to ’ introducing
the
b e fo lg e n
in fin itiv e
S ie
die
clau se w as
m istranslated in the French M T output A and left untranslated in the G erm an output
A . T h is w as n ot the ca se w ith th e three other exam p les. T hese results sh o w that the
M T output w a s n ot co n sisten tly im p roved b y the rule, but it is d ifficu lt to exp lain
w h y results are n ot h o m o g en eo u s. In future stu d ies, it cou ld b e interesting to
evalu ate the im p act o f th is rule on a lan gu age such as C h in ese, w h o se clau se order
is m u ch stricter than that o f F rench and G erm an.
116
4 -3 *3 * C L r u le s t h a t h a v e n o i m p a c t i n b o t h la n g u a g e s
C ertain C L rules w ere c la ssifie d as h a v in g no im pact in both lan gu ages w h en their
freq u en cy score w a s equal to 0. In so m e ca ses, certain rules returned a p o sitiv e
'extra' score w h ich w a s ca n celled out b y a n eg a tiv e 'extra' score. T h ese 10 rules are
p resen ted in T able 4-5:
0
French
'extra'
score 1
(+)
0
French
'extra'
score 2
(+)
0
Rule 7: Do n ot
o m it relative
pronouns and
a form o f the
verb 'be' in a
re lativ e clause.
The post­
m o d ifier is an
adjective.
3
0
0
German
'extra'
score 1
(+)
0
Rule 2 3 : Do
n o t coordinate
- verbs o r verbal
phrases th a t
share the
sam e object
3
0
0
0
0
0
0
Rule 3 2 : S ta rt
n e w sentences
, instead o f
using sem i­
colons.
3
0
0
0
0
0
0
3
0
0
0
0
0
0
Rule 3 9 :
A lw ays w rite
"in order to "
before an
in fin itive in a
purpose clause
■instead o f ju s t
1"to".
6
0
0
0
0
0
0
Rule 4 3 : Avoid
using " (s )" to
indicate plural.
4
0
0
0
0
0
0
Rule 4 6 : Do
n o t use a
particip le to
introduce an
adverbial
clause.
4
0
0
0
0
0
0
Rule 3 7 :
Ensure th a t
re lativ e
] pronouns o r
com plex
re lativ e
pronouns
im m ed iately
follo w th e ir
antecedent.
3
0
0
0
0
0
0
Rule number
and name
Rule 3 4 : Use
the s erial
com ma
Number
of
examples
German
freq.
score
French
freq.
score
117
German
'extra'
score 2
(+ )
0
German
'extra'
score 1
(+ )
1
German
'extra'
score 2
(+)
0
French
'extra'
score 1
(+)
0
French
'extra'
score 2
(+)
1
0
0
0
1
0
Number
of
examples
German
freq.
score
French
freq.
score
Rule 10: Do
not enclose
relative
clauses,
parenthetical
rem arks, or
extraneous
comments in
parentheses.
6
0
Rule 30:
Words such as
"all", "one", or
"none", m ay
n ot appear
alone.
5
0
Rule number
and name
Table 4-5: Rules with no impact in both languages
R ule 7, 'Do not om it relative pronouns and a form o f the verb "be" in a relative
clause, when the post-m odifier is an adjective', w a s evalu ated w ith three ex a m p les
that did n ot sh o w any sig n s o f im p rovem en t. T he la ck o f im p act in French can be
ex p lain ed b y the fa ct that relative p ronouns can also be optional in th is langu age.
T his w as con firm ed by the con sen su s ob tained from French evalu ators w h o
attributing 'E xcellent' scores to the three M T outputs A . T he lack o f im p act in
G erm an can b e exp lain ed b y the fact that relative pronouns w ere au tom atically
inserted in tw o o f the three M T outputs A . T h ese fin d in gs su g g est that this rule is
not e ffe c tiv e w h en th e p o st-m o d ifier im m ed ia tely fo llo w s the n ou n it q u alifies.
R esu lts m ay be d ifferen t i f am b igu ou s p artitive structures w ere u sed in front o f
p ost-n om in al m od ifiers.
T he im pact o f rule 23 w a s covered in sectio n 4 .3 .1 . T he n ex t tw o ru les w ith no
apparent im p act in b oth lan gu a ges con cern tw o p u n ctuation rules, rule 32 and rule
34. A m icro -a n a ly sis o f th ese ex a m p les con firm ed th e ab sen ce o f p o sitiv e im p act in
both languages.
R u le 39, 'Always w rite "in order to" before an infinitive in a purpose clause instead
o f ju s t "to". T his rule, m en tion ed b y B em th and G dan iec (2001: 187) and Systran
(2 0 0 4 : 172) w a s evalu ated w ith six exam p les. T he French gen eric score w a s
superior to the G erm an gen eric score, due to th e introduction o f th e m ore idiom atic
translation ‘afin d e ’ instead o f ‘p our’ in M T output B . T his im p rovem en t, h ow ever,
118
d o es not appear valu ab le en ou gh to ju s tify the introduction o f tw o extra w ords in
the sou rce text on a sy stem a tic b asis. T his fin d in g e c h o e s that o f O 'Brien (2006:
1 7 4 ), w h o fou n d that the p resen ce o f such a n egative translatability indicator ('to'
instead o f 'in order to') did n ot increase the p o st-ed itin g effort. T his should be good
n ew s to O B roin (2 0 0 5 : 3 7 ), w h o states that the elim in ation o f th ese tw o extra
w ord s can save 40% o f the translation co st for sen ten ces con tain ing fiv e words.
W h en re v ie w in g other ex a m p les in th e test suite, h o w ev er, tw o m istranslations o f
the p rep osition 'to' introducing purpose cla u ses w ere fou n d in the French output
(e x a m p le s 103 and 179). T he ty p ica l in fin itiv e structure w a s also m issin g from the
G erm an segm en ts for th ese exam p les.
T he introduction o f rule 4 3 , 'Avoid using "(s)" to indicate plural', did not im prove
th e com p reh en sib ility o f M T output b ased on the results obtained in both languages.
T his can be ex p la in ed by the fact that the M T sy stem reco g n ised this marker and
a u tom atically replaced it w ith a plural form in pre-C L segm en ts 2 3 7 and 239.
H o w ev e r, a singular form w a s generated in the translations o f pre-C L segm en t 2 3 8 ,
thus a ffectin g th e accuracy o f th e translation.
O verall, th ese results contrast w ith O'Brien's fin d in gs (2006: 177), w h ich sh ow ed
that th is n eg a tiv e translatability indicator led to h ig h p ost-ed itin g effort. T hese
d ifferen t fin d in gs m a y b e ex p la in ed b y several reasons. First o f all, this lin gu istic
p h en o m en o n m ay h ave b een h an d led in a d ifferen t m anner by th e M T system used
b y O 'Brien (IB M W eb S p h ere). S econ d , the m eth o d o lo g y sh e used to evaluate the
im p act o f this n eg a tiv e translatability indicator on p o st-ed itin g effort fundam entally
d iffers from the approach taken in the p resent study. H er subjects had to post-edit
M T output, w h ereas th e evaluators u sed in the present study had to rate M T output
b y talcing into accou n t h yp oth etical p ost-ed itin g requirem ents. T hese tw o different
ap proaches appear to h ave prod uced different results.
R u le 3 7 , 'Ensure that relative pronouns or complex relative pronouns immediately
fo llo w their antecedent' w a s ev alu ated w ith three ex a m p les, but n one returned any
extra p o sitiv e score. T he gen eric scores w ere also lo w in both langu ages. A m icro­
a n a ly sis o f the seg m en ts w a s p erform ed to determ ine w h eth er the e ffec tiv en ess o f
th is rule m ay h ave b een blurred b y other lin g u istic p hen om en a. T his w as the case in
ex a m p le 2 0 9 , w h ere th e replacem en t introduced in th e p ost-C L segm en t resolved
119
the initial issu e, but introduced referential am biguity. In the tw o other exam p les, the
w ord 'that' did n ot introduce a gen u in e relative clau se, but con tain ed a nom inal
projection. T h is typ e o f structure is related to so m e o f the structures covered b y rule
6
. In the pre-C L segm en ts o f ex a m p les 2 0 8 and 2 1 0 , h o w ev er, the w ord 'that' w as
a m b igu ou s b eca u se it w a s re co g n ise d as a relative p ronoun b y the M T system .
D e sp ite lead in g to ungram m atical M T output A , the evaluators' scores do not
su g g est any im p rovem en t w ith the rew riting. T his m ay b e partly exp lained by
in co n sisten t scoring. For in stan ce, th e French evaluators d ivid ed their four scores
into 'M edium ' and 'Good' scores for M T output A o f ex a m p le 2 1 0 , w h ich did not
lead to any im p rovem en t u sin g the scorin g m ech a n ism d escrib ed in sectio n 4 .2 .2 .2 .
R u le 10, 'Do not enclose relative clauses, parenthetical remarks, or extraneous
comments in parentheses', d iffers from rule 4 2 , sin ce it d oes not a llo w for any
m aterial w ith in parentheses. R u le 4 2 o n ly p roscribes parenthetical exp ression s that
break the syn tactic flo w o f a sen ten ce. T he e ffe c tiv e n e ss o f rule 10 w a s evaluated
u sin g six exam p les, w h ich prod uced differen t results d ep en d in g on the target
langu age. B efore ex a m in in g ex a m p les that returned n eg a tiv e scores, the exam p le
that returned p o sitiv e 'extra' scores is d iscu ssed sin ce it reveals source an alysis
d ifferen ces. T aking th is factor into accou n t is im portant b eca u se it has an im p act on
the gen eralisation s that can b e m ad e w ith regard to the e ffe c tiv e n e ss o f certain CL
rules. T he parenthetical ex p re ssio n in exam p le 4 8 see m s to h ave b een analysed
d ifferen tly b y the M T sy stem , sin c e it produced different outputs:
HI P redefined sy stem file s (m ust be p laced w ith in th e first 2 gigab ytes o f the
drive)
A
F ich iers sy stèm e p réd éfin is (d oit être p lacé dans le s 2 prem iers gigaoctets du
disq u e)
A
V orb estim m te S ystem d ateien (m ü ssen innerhalb der ersten 2 G igab ytes des
L aufw erks platziert w erd en )
T he French output sh o w s that there is no agreem ent b etw een the initial verb o f the
parenthetical ex p ressio n ('doit') and the subject o f the p reced in g segm en t, 'fichiers'.
T h is su g g ests that the parenthetical ex p ressio n has b een parsed w ith ou t taking into
accoun t th e first part o f the segm en t. T his is n ot th e case in G erm an output A , sin ce
the verb 'm üssen' agrees w ith 'System dateien'. N o n e th e le ss, a p o sitiv e 'extra' score
120
o f le v e l 1 w as ob tained for this seg m en t in G erm an, w h ile a p o sitiv e 'extra' score o f
le v e l 2 w as obtained in French.
O n th e other hand, n eg a tiv e 'extra' scores w ere returned for tw o exam ples: 43 in
French and 4 4 in G erm an. In th ese ex a m p les, the pre-C L segm en ts con tain ed a
parenthetical ex p re ssio n located at the end o f the segm en t. T he parenthetical
ex p ressio n s w ere therefore w e ll separated from the rest o f the segm en ts. In exam ple
4 3 , am b igu ity w a s u n e x p e cte d ly introduced in th e p ost-C L segm en t, as sh ow n by
the French ex a m p le, w h ic h re ceiv ed a n eg a tiv e 'extra' score o f lev el 2:
M D eterm in e
th e
address
of
the
listserv
(u su ally
sim ilar
to
listserv @ sy m a n te c.c o m , or te ch su p p @ sv m a n tec.co m ) .
A
D éterm in ez
l'adresse
du
listserv
(h ab itu ellem en t
sem b lab le
à
listserv @ sy m a n te c.c o m , ou à tech su p p @ sy m a n tec.co m ).
0
D eterm in e
th e
address
o f the
listserv,
w h ich
is
u su ally
sim ilar
to
listserv @ sy m a n te c.c o m , or te ch su p p @ sv m a n te c.co m .
B
D éterm in ez
l'adresse
du
listserv,
qui
est h abitu ellem en t
sem blab le
à
listserv @ sy m a n te c.c o m , ou de te ch su p p @ sy m a n te c.co m .
T he incorrect u se o f th e p rep o sitio n 'de' in front o f 'tech su p p@ sym an tec.com ' in M T
output B sh o w s that th e co m m a p reced in g 'or' sh ou ld h ave been rem oved from the
p o st-C L segm en t. T h is w o u ld h ave prevented its incorrect an alysis b y th e M T
system .
A m ore sign ifican t p rob lem w a s noted w ith ex a m p le 44. T he integration o f the
parenthetical co m m en t into the m ain seg m en t introduced co m p lex ity in the p ost-C L
segm en t, w h ich a ffected th e com p reh en sib ility o f the G erm an M T output. T his
fin d in g su g g ests that th e e ffe c tiv e n e ss o f this rule is v ery lim ited . T his w as
con firm ed b y ex a m in in g ex a m p les con tain ing parenthetical ex p ressio n s that w ere
em b ed d ed in the m id d le o f the segm en ts (in ex a m p les 45 and 4 6 ). R em o v in g the
parentheses in the p ost-C L seg m en ts did n ot im p rove th e com p reh en sib ility o f the
M T output in b oth French and G erm an.
121
T he e ffe c tiv e n e ss o f rule 3 0 , 'Words such as "all", "one", or "none", may not
appear alone' w a s evaluated u sin g fiv e exam p les. O n ly on e o f th em returned a
p o sitiv e 'extra' score o f le v e l 1 in French (ex a m p le 181), w h en "one" fo llo w e d a
p rep osition ("with"). O ne n eg a tiv e 'extra' score o f le v e l 1 w as obtained w ith
ex a m p le 185, d esp ite "none" b ein g u sed on its o w n in the pre-C L segm en t. T hese
results su g g est that the e ffe c tiv e n e ss o f th is rule is n ot con sisten t, and that its im pact
m in im al. In section 4 .3 .4 , the CL rules that sh o w sig n s o f n eg a tiv e im pact in both
lan g u ages are d iscu ssed .
4 .3 .4 .
C L r u l e s t h a t s h o w s ig n s o f n e g a t iv e i m p a c t i n
b o t h la n g u a g e s
O n ly tw o rules fall into this category, ev e n th ou gh so m e o f the ex am p les u sed for
their evalu ation returned p o sitiv e 'extra' scores. T he n eg a tiv e scores are presented in
T able 4 -6 w ith a "(-)" m arker in the first row .
French
’extra'
score 1
2
German
'extra'
score 2
(-)
0
1
French
'extra'
score 2
(-)
0
3
1
0
2
Rule number
and name
Number
of
examples
German
freq.
score
French
freq.
score
German
'extra'
score 1
C-)
Rule 4 2 : Do
n o t include
parenthesized
expressions in
a segm ent
unless the
segm ent is still
valid
syntactically
when you
rem ove the
parentheses
while leaving
the
parenthesized
expressions.
6
-33
0
Rule 12: Use
the active
voice when
you know who
o r w h at did
the action.
12
-30
4
Table 4-6: Rules that show signs of negative impact in both languages
T he e ffe c tiv e n e ss o f rule 4 2 , 'Do not include parenthesized expressions in a
segment unless the segm ent is still valid syntactically when you remove the
parentheses while leaving the parenthesized expressions', w a s evaluated w ith six
ex a m p les. T he sco p e o f this rule is different from that o f rule 10. W hereas rule 10
122
p roscrib es any parenthetical ex p re ssio n s occurring in th e m id d le o f sen ten ces, this
rule fo c u se s on parenthetical ex p re ssio n s that break the syn tactic flo w o f the
sen ten ce. T h is d ifferen ce is e x e m p lifie d w ith tw o exam ples:
El L o g on as th e user w h o sa w the error (and w h o n o w has adm inistrator
a cc ess), and con tin u e to th e n ex t section , (ex a m p le 45 u sed for rule 10)
El T he (3 0 2 1 ,2 ) error m e s sa g e p revents y o u from see in g the N orton A n tiv ir u s
interface, (ex a m p le 2 3 6 u sed for rule 4 2 ).
R u le 4 2 returned on e p o sitiv e 'extra' score in French for ex a m p le 4 5 , but none for
G erm an. T his m ay be ex p la in ed by a d ifferen t h and ling o f p unctuation marks in the
tw o lan gu age pairs, as sh o w n b y both M T outputs A:
h
D ie (3 0 2 1 .2 ) F eh lerm eld u n g hindert S ie am S eh en der N orton A n tiv ir u s
Sch nittstelle.
A
(L es 3 0 2 1 .2 ) m e ssa g e s d'erreur v o u s em p êch en t de v oir l'interface de N orton
A n tiv ir u s.
T he
parenthetical
ex p re ssio n
w as
correctly
an alysed
in th e E nglish-G erm an
la n g u age pair, sin c e th e translation fo llo w s th e w ord order o f the pre-C L segm ent.
T h is w a s n ot the ca se w ith th e E n glish -F ren ch lan gu age pair.
T he seco n d rule to return n eg a tiv e 'extra' scores in both lan gu ages in a con sisten t
m anner is rule 12, 'Use the active voice when you know who or what d id the action'.
T his rule w a s evalu ated u sin g a large num ber o f ex a m p les to d iversify the typ es o f
a g en tiv e structures present in pre-C L segm en ts. T he ex a m p les returning n egative
'extra' scores su g g est that so m e o f th e rew ritings p roved m ore am b iguou s than the
original sen ten ces. T h is is sh ow n b y exam p le 70,
El
Y o u w an t to k n o w th e m o st co m m o n load p oin ts that are u sed by viruses.
A
S ie m ôch ten d ie m eisten g em ein sa m en L adepunkte k ennen, die durch V iren
benutzt w erden.
0
Y o u w ant to k n o w th e m o st co m m o n load p oin ts that viru ses use.
B
S ie
m ôch ten
d ie
m e isten
g em ein sam en
V irusgebrauch.
123
L adepunkte
kennen
dass
T his ex a m p le su g g ests that the u se o f the active v o ic e in short relative clau ses can
introduce am b iguity. T his fin d in g w a s o n ly partly con firm ed w ith exam p le
6 8
, sin ce
it returned a n egative 'extra' score o f le v e l 1 in G erm an and a p o sitiv e 'extra' score
o f le v e l 1 in French.
Apart from ex a m p les 187 and 2 3 6 , th is is the o n ly exam p le in the test suite that
prod uced op p osite results in both lan gu ages. E xam p le
6 8
is sh ow n below :
1H1 S elect other p h y sica l d isk d rives that you w ant to be protected b y N orton
G oB ack .
Ä
C h o isisse z d'autres d isq u es p h y siq u es que v o u s v o u le z être protégé par
N orton G oB ack .
A
W äh len S ie andere p h y sik a lisc h e n Festplatten aus, die S ie v o n N orton
G oB ack g esch ü tzt w erd en m öchten .
0
S elect other p h y sica l d isk d rives that y o u w ant N orton G oB ack to protect.
B
C h o isisse z d'autres d isq u es p h ysiq u es que v o u s v o u le z que N orton G oB ack
protège.
B
W äh len S ie andere p h y sik a lisc h e n F estplatten aus, dass S ie N orton G oB ack
sch ü tzen w ü n sch en .
G erm an output B sh o w s that th e relative pronoun in the p ost-C L segm en t w as
a n a ly sed as a con ju n ction w ith th e introduction o f the active v o ic e in the relative
clau se. T his d o es n ot see m to h ave b een th e case in French, ev e n th ou gh 'que' is also
am b igu ou s in that lan gu age. T he com p reh en sib ility o f the French output w as
therefore im p roved b y th e introduction o f the CL. Apart from exam p le 64, this
ex a m p le is th e o n ly o n e that returned p o sitiv e scores across both langu ages, so the
e ffe c tiv e n e ss o f th e rule appears very lim ited overall. Such a fin din g g o es against
the recom m en d ation s p resent in the literature (B em th & G daniec, 2 001: 190,
W o jcik , 1998: 118, A d riaen s & M acken, 1995: 130). T his also su g g ests that further
eva lu ation is required to determ ine w h eth er the sco p e o f this rule should be reduced.
It w o u ld also b e v ery interestin g to study the e ffe c tiv e n e ss o f th is rule in other
la n g u ages, such as C h in ese, due to the le ss com m on and m ore restricted u se o f the
p a ssiv e in this lan gu age (R o ss, 2004: 2 4 1 ). For instance, C ardey et al. (2004: 4 0 )
had d ecid ed to ex c lu d e this structure from the C on trolled E n glish rule set th ey
d esig n ed for translating m ed ical p rotocols into C hinese.
124
4 . 3 -5 - C L r u l e s t h a t a r e m o r e e f f e c t i v e f o r F r e n c h t h a n
fo r G e rm a n
B a sed o n the gen eration o f 'extra' sco res and their associated freq uency score,
certain rules appeared to b e m ore e ffe c tiv e in French than in G erm an. R u les w ere
in itia lly p la ced in th is ca teg o ry w h en o n e o f tw o criteria w as met:
■
T h e rule's freq u en cy score for French w a s greater than
or equal to
33 and the rule's freq u en cy score for G erm an w a s less than 33.
■
T he rule's freq u en cy score for French w a s greater than 0 and the
rule's freq u en cy score for G erm an w a s le ss than or equal to 0.
O ne ex c e p tio n to th ese tw o criteria con cern s rule 13, w h o se French freq uency score
w a s 0, but w h o se gen eric score w a s 1 (com pared to a gen eric score o f 0 in Germ an).
T h is su g g ested that the rule w a s m ore e ffe c tiv e for French than for G erm an. T he 12
ru les corresp on din g to the a b o v e criteria are listed in T able 4-7:
German
'extra'
score 2
(-)
0
French
'extra'
score 1
(+)
1
French
'extra'
score 2
17
German
'extra'
score 1
(-)
0
0
25
0
0
0
2
6
0
17
0
0
1
0
Rule 2 1 : Do
n o t use the
slash to list
lexical item s.
7
14
64
0
0
4
1
Rule 11: When
appropriate,
use an article
(a , an, the)
before a noun.
12
0
21
0
0
2
1
Rule 13: Do
n o t use the
im personal
passive voice.
3
0
0
0
0
0
0
Rule number
and name
Number
of
examples
German
freq.
score
French
freq.
score
Rule 4 9 : Do
n o t use "this",
"that",
"these", o r
"those" on
th e ir own
6
0
Rule 45: Use
only
"because",
n ev er "since"
in a subclause
o f reason
4
Rule 3 3 : Use a
com m a before
a coordinated
clause.
125
0
Number
of
examples
German
freq.
score
French
freq.
score
German
'extra'
score 2
(-)
6
German
'extra'
score 1
{-)
1
Rule 16: Avoid
the use o f
I progressive
verb
9
-8
Rule 50: Keep
both parts o f a
verb together
4
0
50
Rule 31:
W henever
possible, the
use o f "w h-"
questions
should be
avoided
4
-12.5
Rule 26: Avoid
the ellipsis o f
verb.
6
Rule 27: Avoid
the ellipsis o f
subject.
Rule 47: Move
docum ent
nam es, erro r
messages, o r
section titles
to independent
segments.
Rule number
and name
o
French
'extra'
score 1
(+ )
0
French
'extra'
score 2
(+)...
1
1
0
2
0
75
O
1
3
0
16
50
1
0
2
2
14
7
43
1
0
6
0
6
17
67
1
0
4
0
„
.
Table 4-7: Rules with more positive impact in French than in German
T h e e ffe c tiv e n e ss o f rule 4 9 , 'Do not use "this", "that", "these", or "those" on their
own' w a s evalu ated u sin g s ix exam p les. O nly on e ex a m p le returned a p o sitiv e 'extra'
score o f le v e l 1 in French (ex a m p le 2 6 6 ). T he pre-C L seg m en t o f this exam p le
con tain ed a d eictic p ronoun introducing referential am b iguity. In G erm an, the
im p rovem en t brought b y th e ap p lication o f the rule w a s m ask ed b y th e co m p lex ity
o f the exam p le. B a sed on th ese results, th e e ffec tiv en ess o f th is rule appears very
lim ited in b oth lan gu ages.
R u le 4 5 , 'Use only "because", never "since" in a subclause o f reason' is present in
th e C O G R A M rule set (A driaens and M acken, 1995: 130). T his rule w a s evaluated
w ith four ex a m p les, and returned a French gen eric score that w as superior to the
G erm an gen eric sco re (0 .6 2 5 com pared to 0 .1 2 5 ). D iffere n c es b etw een the tw o
la n g u ages w ere also con firm ed b y tw o 'extra' scores o f le v e l 2 obtained in French,
su g g estin g that o n ly m arginal im p rovem en ts w ere introduced b y th e rule. T his m ay
b e ex p la in ed b y th e fact that in th e four ex a m p les th e M T system an alysed ‘s in c e ’
126
1
as a con ju n ction introducing a su b cla u se o f reason in both lan gu ages, and not as a
tem poral su b clau se, w h ich w o u ld h ave distorted the original m ean in g o f the source
input. T he superior gen eric score for French m ay b e exp lain ed by the m ore
id io m atic translation o f ‘b e c a u se ’ b y ‘parce q u e ’ instead o f ‘p u isq u e ’ for ‘sin c e ’.
T he effe c tiv e n e ss o f rule 33, 'Use a comma before a coordinated clause', w as
evaluated u sin g six exam p les. O n ly on e o f th em returned a p o sitiv e 'extra' score
(ex a m p le 196 for the E n glish -F ren ch lan gu age pair):
El Enter addresses on e at a tim e and separate th em b y a space.
A
S a isisse z le s adresses u n e par u n e et sép arées e lle s par un esp ace.
0
Enter ad dresses on e at a tim e, and separate them by a space.
B
S a isisse z le s ad resses u n e par une, et sép arez-les par un esp ace.
In th e ab ove exam p le, the M T sy ste m in correctly parsed the pre-C L segm en t, sin ce
‘sep arate’ w as translated as an ad jective. T he introduction o f the co m m a in the p ostCL segm en t rem oved the am b igu ity. In this particular ex a m p le, the seco n d verb o f
th e pre-C L
segm en t w as
a lso m istranslated in G erm an M T
output A .
The
e ffe c tiv e n e ss o f th is rule m ay n o t b e frequent in b oth lan gu age pairs, but can greatly
im p rove the com p reh en sib ility o f M T output in certain cases.
T he e ffe c tiv e n e ss o f rule 2 1 , 'Do not use the slash to list lexical items', w as
ev alu ated w ith se v e n ex a m p les. F iv e o f th em returned p o sitiv e 'extra' scores in
F rench, w h ereas o n ly on e o f th em returned a p o sitiv e 'extra' score in Germ an. T he
score d ifferen ces m a y b e ex p la in ed b y differen t an alysis and transfer rules in the
tw o lan gu age pairs. For in stan ce, M T outputs A d iffer greatly for exam p le 124:
El E n h an cem en ts/P rob lem s fix ed
A
L es p erfectio n n em en ts/p ro b lèm es les ont résolu
À
V erb esseru n gen /P rob lem e b eh ob en
W hereas the G erm an w ord order fo llo w s the E n g lish w ord order, extra w ords have
b een introduced in the F rench output, w h ich is not com p reh en sib le. In this
particular ex a m p le, the rule is m ore e ffe c tiv e for French than G erm an becau se
G erm an M T output A w a s m u ch m ore com p reh en sib le than French M T output A.
O ne exam p le, h o w ev er, returned p o sitiv e 'extra' scores in both lan gu ages (exam p le
127
127). A s a co n clu sio n , the p o sitiv e im p act o f this rule can be equally im portant in
both lan gu ages, e sp ec ia lly w h en th e slash is u sed to separate tw o n oun s or an
a d jective and a noun. T his is co n sisten t w ith O'Brien's co n clu sio n s (2006: 170),
w h o also fou n d that im p rovem en ts are lim ited w h en the slash separates other lex ica l
item s, su ch as tw o con ju n ction s (and/or).
R u le 11, 'When appropriate, use an article (a, an, the) before a noun’, returned 3
p o sitiv e 'extra' scores in French w h ereas it did not return any in Germ an. A s stated
b y H eald and Zajac (1 9 9 6 : 2 0 7 ), 'the e x p lic it u se o f the article is m otivated b y the
assu m p tion that it m ak es u nderstanding easier and h elp s to clear p o ssib le con fu sion
b etw e en n ou n and verbs.' T h is assu m p tion w as con firm ed b y the p o sitiv e 'extra'
sco res o f le v e l 1 obtained for ex a m p les 4 9 and 56 in French. In th ese tw o exam p les,
n oun s requiring an article fo llo w e d am b igu ou s w ords: an - i n g w ord in exam p le 49
and a su b jectless third p erson verb in the p resent ten se in ex a m p le 56:
El T he prob lem m igh t b e a result o f upgrading you r operating system w ithout
first u n in stallin g program s that are n ot com p atib le w ith W in d ow s X P.
0
B lo ck s repeated Internet attacks.
For ex a m p le 5 6 , the rule w a s m ore e ffe c tiv e in French than G erm an b ecau se the
a m b igu ou s verb w a s m istranslated in French M T output A , w h ereas it w as properly
translated in G erm an M T output A . In exam p le 4 9 , th e am b iguou s - in g w ord w as
m istranslated in b oth lan gu ages, but the co m p lex ity o f the sen ten ce m ask ed the
im p rovem en t in G erm an. In short, results sh ow ed that th e effec tiv en ess o f this rule
is n o t con sisten t, d esp ite its p oten tial p o sitiv e im pact o n th e com p reh en sib ility o f
M T output. It sh ould b e n oted , h o w ev er, that the description o f th is rule appears too
restrictive. In certain ca se s, su ch as exam p le 56, u sin g th e article 'the' in the post-C L
seg m en t is incorrect sin ce th e noun phrase contained in the segm en t has not b een
e x p lic itly d efin ed . A determ iner su ch as 'any' w o u ld b e m ore natural in the source
te x t but m a y cau se transfer p rob lem s due its various translations into both French
and G erm an. T his ex a m p le su g g ests that articles such as 'the' h ave the potential to
rem o v e am b iguity, but so m e tim es at a cost. S in ce the accep tab ility o f the source
tex t see m s to b e im p acted in certain cases, con siderin g an autom atic insertion o f
articles as part o f a p re-p ro cessin g p rocess m ay be en v isa g ed . S uch a p rocess, w h ich
h as b een su ggested b y N y b er g et al. (20 0 3 : 2 7 6 ), requires further studies.
128
T he effe c tiv e n e ss o f rule 13, 'Do not use the impersonal passive voice' w as
evalu ated w ith 3 ex a m p les w h ich all con tain ed the sam e structure, sin ce it w as the
o n ly structure o f this typ e found in th e corpus ('it is recom m ended'). T he rule w as
n ot effe c tiv e in G erm an sin c e the structure con tain ed in the pre-C L segm en t w as
correctly handled and translated in G erm an M T outputs A . T his w a s n ot th e ca se in
French M T outputs A , but no p o sitiv e 'extra' sco res w ere returned d esp ite a high
gen eric score. T his m ay exp lain ed by the fact that th e alternative structure u sed in
p ost-C L segm en t w a s translated fo llo w in g the w ord order o f the sou rce text, instead
o f u sin g a m ore id iom atic form u lation ('nous recom m en d on s que' instead o f 'nous
v o u s recom m en d on s de'). B a sed on th e ex a m p les u sed in this part study, th e im pact
o f this
rule
is
lim ited
to
m in or
im p rovem en ts
that do
not
guarantee the
co m p reh en sib ility o f th e French M T output.
The effe c tiv e n e ss o f rule 16, 'Avoid the use o f progressive verb participles' w as
evaluated u sin g n in e ex a m p les. E ight o f th em con tain ed a present p ro g ressiv e verb,
and on e o f th em con tain ed a past p ro g ressiv e verb. T he on ly exam p le to return a
p o sitiv e 'extra' sco re in French w a s th e ex a m p le con tain in g a past p ro g ressiv e verb
(w h ich is unusual for th is typ e o f verb in E n g lish ). T he translation o f the pre-CL
seg m en t resulted in a French im p erfect ten se, w h ic h w a s not as accurate as the
p a ssé co m p o sé p roduced in M T output B . T h is m in or im p rovem en t m u st be
contrasted w ith the deterioration noted w ith o n e o f the rew ritings in Germ an
(ex a m p le 92). T he redu ction o f the u n am b igu ou s p rogressive form from th e pre-C L
seg m en t into a sim p le form introduced am b igu ity in the p ost-C L segm en t. The
e ffec tiv en ess o f th is rule is therefore ex trem ely lim ited and cou ld h ave n egative
im p act in certain cases.
D iffere n c es in th e hand ling o f sp e c ific structures across lan gu age pairs w a s noted
w ith results ob tained for rule 31, 'Whenever possible, the use o f "wh-" questions
should be avoided'. S uch a rule d oes not see m to address any am b iguity, but rather
to address p oten tial d e fic ie n c ie s o f M T system s. T his rule p roved m u ch m ore
e ffe c tiv e for F rench than for G erm an, b eca u se G erm an M T outputs A w ere m uch
m ore com p reh en sib le than French M T outputs A . M edian scores for G erm an M T
outputs A w ere { 3 .5 , 4, 4 , 3 } com pared to {2 .5 , 2 .5 , 2 .5 , 1 .5 }. T h ese results sh ow
that sim p le and u n am b igu ou s interrogative structures w ere not handled properly in
129
I
the E nglish -F ren ch lan gu age pair. T h e rew ritin gs introduced in p ost-C L segm en ts
therefore im p roved the com p reh en sib ility o f French M T output.
T he e ffe c tiv e n e ss o f rule 5 0 , 'Keep both p a rts o f a verb together' w a s evaluated
w ith four exam p les. It returned tw o 'extra' scores in French but o n ly o n e in German.
A n eg a tiv e 'extra' score w as also returned in G erm an for exam p le 2 7 2 in w h ich tw o
con join ed particles w ere used in the pre-C L segm en t. The results obtained for this
last ex a m p le su g g est that the u se o f con join ed particles is handled m ore ea sily in
G erm an than in French, due to the fact that particles can also be u sed in this
lan gu age. O verall, th is rule m ay b e e ffe c tiv e in both langu ages, but its application
m a y co n flict w ith the ap plication o f rule 5 2 , 'Use single-w ord verbs'.
R u le 2 6 , 'Avoid the ellipsis o f verb', w a s evalu ated w ith six ex a m p les, due to the
variou s co n tex ts in w h ich verbal e llip se s can occur, w hether in a sin g le sen ten ce (as
in ex a m p le 155), or across tw o sen ten ces (as in exam p le 154). V erbal ellip se s can
b e found in patterns su ch as ‘i f it d o es n o t,’ or ‘but th e file s are n o t,’ in w h ich the
lex ica l verb is om itted . M ore 'extra' sco res w ere obtained in French. A ll six
ex a m p les w ere p o o rly h andled in M T output A in French and G erm an, ex cep t for ‘i f
n o t’, w h ic h returned an id iom atic translation in French, ‘sin o n ’ in exam p le 151.
T his ex a m p le w a s a sp ecial ca se, b eca u se b oth the verb and the subject w ere
om itted. E llip se s o f subject are d iscu ssed n ext.
R u le 2 7 , 'Avoid the ellipsis o f subject', co v ered tw o d istin ct structures: subject
e llip sis and a com b in ation o f su bject and 'verbal operator ellip sis'. T he latter
structure is cla ssifie d b y H allid ay and H asan (1976: 174) as a typ e o f verbal ellip sis.
In this structure, h o w ev er, the su bject is also om itted from the clau se, so it w as
d ecid ed to in clu d e it in rule 2 7 instead o f rule 26. T his secon d structure differs from
the first on e b eca u se both th e su bject and the operator h ave to be re-inserted in the
reform ulation, w h ich then con tain s a fin ite d ep en dent clau se. S egm en ts 156, 159,
165, and 166 w e re se le c te d to co v er the first typ e o f structure, w h ile the other 10
seg m en ts
w ere
ch o sen
to
evalu ate
the
im p act
o f operator
e llip sis
on
the
co m p reh en sib ility o f M T output. T h is rule returned m ore 'extra' scores for French
than for G erm an
( 6
com pared to 1). T he gen eric scores w ere also in favour o f
French (0 .8 for F rench and 0.1 for G erm an). T he results o f this rule, h ow ever,
130
should be an alysed in the light o f the results obtained for rule 36. W h en testin g rule
36, it w a s n oted that p ronouns w ere au tom atically inserted in G erm an output. The
sam e m ech an ism w a s again u sed by the M T sy stem to translate som e o f the pre-C L
seg m en ts in G erm an. T he m ech an ism w ork ed w e ll w h en the subject w a s the sam e
in b oth clau ses. For instance, ex a m p les 163 and 164 returned identical G erm an M T
outputs regardless o f the typ e o f sou rce seg m en t (pre-C L or p ost-C L ). T his is
ex p la in ed by the correct insertion o f subjects in the G erm an d ependent clau ses. The
sam e insertion is n ot perform ed in French, so the com p reh en sib ility o f M T outputs
A w a s greatly im p acted, as sh ow n w ith ex a m p le 164:
0
C lick O K w h en asked to con firm the d eletion .
A
C liq u ez sur O K une fo is dem an d é à con firm er la suppression.
0
C lick O K w h en you are asked to con firm the d eletion .
B
C liq u ez sur O K quand vo u s êtes in v ité à con firm er la suppression.
T he a b ove ex a m p le con firm s that the com p reh en sib ility o f French M T output can
be greatly im p roved b y the introduction o f rule 2 7 . T his exam p le also h igh ligh ts the
d ifferen ces that e x is t b etw een the M T sy s te m ’s transfer rules. H o w ev e r, the
autom atic insertion o f p ronouns in G erm an output d oes not w ork w h en the source
input is am b igu ou s, as sh o w n in exam p le 168:
0
V e rify that th e item is ch ecked. I f n ot ch eck ed , th en ch eck it.
A
Ü berprüfen S ie, dass das E lem en t überprüft wird. W en n S ie nicht überprüft
w erden, überprüfen S ie es dann.
0
V e rify that the item is ch eck ed . I f th e item is not ch eck ed , then ch eck it.
B
Ü berprüfen S ie, dass das E lem en t überprüft wird. W en n das E lem en t nicht
überprüft w ird, dann überprüfen S ie es.
In the a b ove ex a m p le, the p ronoun ‘S ie ’ w a s au tom atically inserted in M T output A ,
probably due to the p resen ce o f the im p erative verb ‘c h e c k ’ in the m ain clau se. In
th is exam p le, h o w ev e r, the e llip sis con cern ed a n ou n from the p reviou s sentence:
'item 1. T his su g g ests that e llip se s sh ould n ot span over several sen ten ces, d esp ite the
fact that the e ffe c tiv e n e ss o f th e rule w a s co m p letely m asked b y the m istranslation
o f the verb 'check'.
131
R u le 4 7 , 'Move document names, error messages, or section titles to independent
segments' is based on the textual feature o f tech n ical support docum entation
id en tified in sectio n 1.3.4 o f Chapter 1. T his rule w a s evalu ated w ith six exam p les,
and returned a h igh num ber o f p o sitiv e 'extra' scores for French (4). R esu lts w ere
n ot as clear-cut for G erm an due to the inherent c o m p le x ity o f certain exam p les
(su ch as 2 5 7 or 2 6 0 ). B esid e s, ex a m p le 2 5 9 revealed that short em bed d ed titles
m arked w ith distin ct punctuation m arks (su ch as pair o f d ouble q uotes) w ere not as
p rob lem atic for G erm an as th ey w ere for French. T he com p reh en sib ility o f the M T
output in this particular ex a m p le w as im p roved w ith the introducion o f the rule in
French, but n ot in G erm an.
A fter p erform in g a m icro -a n a ly sis o f all th e rules that appeared to be m ore effec tiv e
for French than G erm an, it transpires that certain rules can be c la ssifie d as
la n gu age-d ep en d en t rules. B a sed on the results obtained w ith the ex am p les u sed in
th e test suite, three rules sp ecifica lly fall into this category: rule 31, rule 13, and a
s p ecific part o f rule 16. T he rules present in the last category o f rules m u st undergo
th e sam e r e v ie w p ro ce ss to determ ine w h eth er certain rules im p rove sp e c ific a lly the
E nglish -G erm an lan gu age pair.
4 .3 .6 .
C L r u le s t h a t a r e m o r e e ffe c tiv e f o r G e r m a n t h a n
fo r F re n c h
B a sed on the gen eration o f gen eric and 'extra' scores, certain rules appeared to be
m ore e ffe c tiv e in G erm an than in French. R u les w ere in itially p la c ed in this
ca teg ory w h en on e o f tw o criteria w a s met:
■
T he rule's freq u en cy score for G erm an w a s greater than or equal to
33 and the rule's freq u en cy score for French w as less than 33.
■
T he rule's freq u en cy score for G erm an w a s greater than 0 and the
rule's freq u en cy score for French w a s le s s than or equal to 0.
132
T h ese rules are listed in T able 4 -8 b elow :
Rule number
and name
Rule 3 : Do n ot
s ta rt a
sentence w ith
an im plicit
subject in a
subjectless,
no n-finite
clause
prem odifying a
finite clause,
even when the
subject is the
sam e in both
clauses.
Number
of
examples
3
0
German
'extra'
score 1
(+)
1
German
'extra'
score 2
(+)
0
French
'extra'
score 1
(-)
0
German
freq.
score
French
freq.
score
20
French
’extra’
score 2
(-)
.
o
Rule 6: Do n ot
om it the
subordinating
word 'th at'
a fte r verbs or
verbal nouns
3
33
0
1
0
0
0
Rule 8: Do not
o m it relative
pronouns and
a form o f the
verb 'be' in a
relative clause.
The post­
m odifier is an ing word.
5
20
0
1
0
0
0
Rule IS : Do
not use an
ambiguous
form o f 'have'.
7
43
0
3
0
0
0
Rule 24:
'Repeat the
preposition,
conjunction, o r
infinitive
m a rke r in
coordinated
prepositional
phrases,
subordinate
clauses, or
infinitive
com plements'.
5
10
0
0
0
0
Rule 38:
R ew rite ingwords th a t are
com plements
o f o th e r verbs.
4
50
0
2
0
0
Rule 40: Avoid
very short
sentences
(less than 4
words).
5
20
0
1
0
0
'
133
[
0
0
j
Rule number
and name
Number
of
examples
German
freq.
score
French
freq.
score
German
'extra'
score 1
Rule 4 4 : Avoid
splitting
infinitives,
unless the
emphasis is on
the adverb.
5
20
0
1
German
'extra'
score 2
(+ )
0
French
'extra'
score 1
0
French
'extra'
score 2
(-)
0
Table 4-8: Rules with more positive impact in German than in French
R u le 3, 'Do not start a sentence with an im plicit subject in a subjectless, non-finite
clause prem odifying a fin ite clause, even when the subject is the same in both
clauses', w a s evalu ated w ith fiv e exam p les. O nly on e o f th em returned a p o sitiv e
'extra' score in G erm an (ex a m p le 15). T his exam p le w a s different from the other
four, sin ce the su b jectless n o n -fin ite cla u se con tain ed tw o - in g w ords. W h ile the
pre-C L segm en t w a s correctly parsed for th e E nglish -F rench lan gu age pair, it
prod uced output that receiv ed three m ed iu m scores in G erm an. In m o st ca ses, this
rule d o es n ot h ave any im p act in b oth lan gu ages. W h en a su b jectless n on -fin ite
cla u se is lim ited to o n e - i n g w ord , th e com p reh en sib ility o f G erm an output seem s
to be im p roved .
T he e ffe c tiv e n e ss o f rule
6
, 'Do not omit the subordinating w ord "that" after verbs
or verbal nouns', w a s evalu ated u sin g three exam p les. T he rule w a s not as effec tiv e
in French as it w a s in G erm an due to syn tactic d ifferen ces resultin g from the
ab sen ce o f th e con ju n ction in th e sou rce text. D esp ite the ab sen ce o f the conjunction
'que' in th e M T output A o f ex a m p le 2 4 , three French evaluators th ou ght the
sen ten ce cou ld b e understood b y en d -u sers. In the other tw o exam p les, exam p les 25
and 2 6 , th e subordinating con ju n ction 'que' w a s au tom atically inserted in French
output
d esp ite
b ein g
absent
from
the
pre-C L
segm ent.
In
G erm an,
the
corresp on din g con ju n ction 'dass' w a s o n ly inserted in exam p le 2 5 . E xam p le 26
therefore returned a p o sitiv e score o f le v e l 1 in G erm an due to sign ifican t syntactic
d ifferen ces b etw e en M T output A and M T output B . B ased on th ese three exam p les,
the rule appears to b e slig h tly m ore e ffe c tiv e in G erm an than in French.
134
T he e ffe c tiv e n e s s o f rule
8
, 'Do not om it relative pronouns and a fo rm o f the verb
'be' in a relative clause, when the post-m odifier is an -ing word', w a s evaluated w ith
fiv e ex a m p les. T his rule w a s n ot e ffe c tiv e for the E nglish -F rench lan gu age pair, and
o n ly e ffe c tiv e in o n e ex a m p le for the E n glish -G erm an lan gu age pair. For the
E n g lish -F ren ch lan gu age pair, all o f the present participles p resent in pre-CL
seg m en ts w ere parsed correctly. For the E nglish -G erm an lan gu age pair, fu ll relative
cla u ses w ere au tom atically inserted in four o f the M T outputs A , d esp ite the
ab sen ce o f relative pronouns in pre-C L segm en ts.
T he e ffe c tiv e n e ss o f rule 15, 'Do not use an ambiguous form o f 'have', w as
evalu ated w ith se v e n ex a m p les, and returned p o sitiv e 'extra' scores in G erm an only.
T he n eg a tiv e im p act origin atin g from the am b igu ities that the rule tries to address
w a s m ore v is ib le in G erm an M T ou tp uts A than in French M T outputs A . In the
pre-C L seg m en ts o f ex a m p les 82, 83, 84 and 85, the structure 'have' + past
participle is am b igu ou s, b ecau se it co u ld b e an alysed as a cau sative structure in
w h ic h the p ast-p articip le is u sed in a p a ssiv e w ay. T h is is sh ow n w ith the pre-CL
seg m en t o f ex a m p le 82:
HI S ym an tec d o es not recom m en d h av in g m u ltip le firew alls in stalled o n the
sam e com puter.
T his ex a m p le returned a p o sitiv e 'extra' score o f lev el 1 in G erm an, su g g estin g that
a rew ritin g u sin g th e a ctive v o ic e co u ld im p rove the com p reh en sib ility o f the M T
output. A sim ilar p o sitiv e 'extra' score w a s obtained w ith ex a m p le
8 6
, w h ich did
con tain a cau sative structure. T h is structure failed to be transferred accurately in
G erm an, w h ic h im p acted on th e com p reh en sib ility o f M T output A . T he results
obtained w ith th ese sev en ex a m p les su g g est that the rule w a s m ore e ffe c tiv e for
G erm an than French. W hen ex a m in in g other segm en ts p resent in the test suite,
h o w ev er, another am b igu ou s form o f 'have' w a s fou n d in on e o f th e p ost-C L
seg m en ts (in ex a m p le
153). T his seg m en t con tain ed a n egative form o f this
a m b igu ou s structure, w h ich w a s m istranslated in French output:
135
S
I f y ou do not h ave N orton S ystem W ork s installed, then contact your
com puter m anufacturer.
A
S i v o u s ne faites p as installer N orton S ystem W ork s, con tactez alors votre
fabriquant inform atique.
T his ex a m p le su g g ests that the rule co u ld also b e effe c tiv e for the E nglish -F rench
lan gu age pair w h en n eg a tiv e structures are used.
T he effe c tiv e n e ss o f rule 2 4 , 'Repeat the preposition, conjunction, or infinitive
marker in coordinated prepositional phrases, subordinate clauses, or infinitive
complements', w a s evalu ated u sin g fiv e exam p les. O nly on e o f th em returned a
p o sitiv e 'extra' score ( o f le v e l 2) in G erm an. T he pre-CL seg m en t o f exam p le 144,
w h ich
con tain ed
co n jo in ed p rep osition al phrases
separated b y a com m a,
is
p resen ted b e lo w w ith the tw o M T outputs A:
0
T he fo llo w in g so lu tio n s h a v e w ork ed for som e o f our cu stom ers, but n ot all
o f them .
A
D ie fo lg en d en L ôsu n gen haben fur ein ig e unserer K unden, aber n icht alle
funktioniert.
A
L es so lu tio n s su ivan tes ont fo n ctio n n é pour certains de n os clien ts, m ais
pour pas tous.
W h ile three G erm an evalu ators attributed 'Good' scores to M T output A , three
F rench evaluators g a v e 'E xcellent' scores d esp ite a w ord order problem ('pour pas')
introduced b y the ab sen ce o f the p rep osition in the pre-C L segm en t. B ased on this
a n a ly sis, it can n ot be stated that the rule is m ore e fficien t for G erm an than for
F rench. T he e ffe c tiv e n e ss o f th is rule, h o w ev er, w a s n ot con sisten t across th e three
ex a m p les con tain ing con join ed phrases that w ere not separated w ith com m as. In
o n e o f th e ex a m p les (e x a m p le 143), th e p rep osition w as introduced autom atically in
M T output A d esp ite b ein g ab sen t from th e pre-CL segm en t. T o m a x im ise the
e ffe c tiv e n e ss o f th is rule, its sco p e cou ld be reduced to co n jo in ed phrases separated
w ith com m as.
136
T he effe c tiv e n e ss o f rule 4 0 , 'Avoid very short sentences (less than 4 w ords)', w as
evaluated u sin g fiv e exam p les. O nly o n e exam p le returned a p o sitiv e 'extra' score o f
le v e l 1 (ex a m p le 2 2 1 ) in G erm an. In th is exam p le, the com p reh en sib ility o f G erm an
M T output A w a s a ffected b y the incorrect gen eration o f an in fin itiv e structure
(w ith the repetition o f 'zu'), rather than by th e length o f the pre-C L segm en t. The
scores obtained for th e other four ex a m p les con firm ed the lack o f e ffe c tiv e n e ss o f
this rule. T h ese four ex am p les con tain ed the w ord 'click', w h ich had b een cod ed
w ith a h igh priority in the u ser dictionary so as to be alw ays recogn ised as a verb.
W ith out this cu stom isation , results m a y h ave b een different in short sen ten ces, sin ce
'click' m ay h ave b een an alysed as a noun.
R ule 3 8 , 'Rewrite -ing words that are complements o f other verbs', is recom m en ded
b y B ernth and G dan iec (2 0 0 1 : 184). D ep en d in g on the v a len cy o f the verb th ey
fo llo w , such -in g w ords can either introduce a su b jectless clau se as a direct object,
or act as an adjunct in the m ain clau se. T his rule w a s evalu ated w ith 4 exam p les,
and returned tw o p o sitiv e 'extra' scores o f le v e l 1 in G erm an. O ne o f th ese exam p les
(ex a m p le 2 1 2 ) con tain ed a verb w h o se u su al com p lem en tation is a su b jectless -in g
cla u se, ‘co n tin u e’ (Q uirk et al., 1985: 118). T his structure, h ow ever, w a s not parsed
accurately b y the M T sy stem in the pre-C L segm en t. T he com p reh en sib ility o f the
G erm an translation o f the p ost-C L seg m en t w a s therefore im p roved thanks to the
rew riting. T his im p rovem en t w a s not v isib le in French, due a le x ic a l am biguity
introduced by th e h om ograp h 'test', w h ic h w a s an alysed as a noun. T his m icro­
a n a lysis su g g ests that th e rule w o u ld h ave b een e ffe c tiv e in French w ith a less
a m b igu ou s verb. T he h and ling o f the verb 'continue' in exam p le 2 1 2 contrasts w ith
th e output p roduced for ex a m p le 2 1 3 , w h ich con tain ed a sim ilar verb ( ‘try ’). In this
ex a m p le, th e structure w a s properly an alysed in both langu ages. In o n e o f the other
ex a m p les (ex a m p le 2 1 4 ), rew riting the -in g w ord as an in fin itiv e w a s o n ly p o ssib le
b y ch an gin g the verb it p reced ed ('require' to 'need'). B a sed on th ese results, the
e ffe c tiv e n e ss o f th is rule see m s to be lim ited to sp e c ific cases. B e sid e s, on e should
n ot underestim ate the p o ssib le
am b iguity that rew ritings m ay introduce. A n
in fin itiv e clau se introduced by 'to' cou ld be an alysed as a p urp osive clau se. In short,
th is rule has lim ited p o sitiv e im p act in both lan gu ages and can n ot alw ays be
im p lem en ted b y o n ly rew riting th e - i n g word.
137
The e ffe c tiv e n e ss o f rule 4 4 , 'Avoid splitting infinitives, unless the emphasis is on
the adverb', w a s evalu ated u sin g fiv e ex a m p les. O nly on e o f th em returned a
p o sitiv e 'extra' score o f le v e l 1 in G erm an (ex a m p le 2 4 5 ). The in fin itiv e that w as
split in th e p re-C L segm en t o f this ex a m p le w a s incorrectly translated as a noun in
G erm an M T output A . T his w a s not th e ca se in French, so th e e ffe c tiv e n e ss o f this
rule w as n ot v is ib le for the E n glish -F ren ch lan gu age pair in this particular exam ple.
In the four other ex a m p les, the rew riting did not im p rove the com p reh en sib ility o f
the G erm an output b eca u se G erm an M T outputs A and M T outputs B w ere identical
regard less o f th e p o sitio n o f the adverb in the sou rce segm en t. B ased on th ese
results alon e, it is n ot p o ssib le to co n clu d e that this rule w o u ld a lw a y s b e m ore
e ffic ie n t for G erm an than for French.
4.4. Findings
T he m ain c o n c lu sio n that can be d erived from the first part o f th is study is that the
e ffe c tiv e n e ss o f in d ivid u al C L rules is n ot h o m o g en eo u s. N o t on e sin g le rule w as
a lw a ys co n sisten tly e ffe c tiv e in both lan gu age pairs. T his fin din g m ay be exp lained
b y several reason s that w ill b e d iscu ssed in sec tio n s 4.4.1 to 4.4.8:
■
T he
im p rovem en t
translation
introduced
p rob lem s
triggered
by
the
by
CL
other
rule
is
lin g u istic
m ask ed
by
p henom en a
p resen t in the sou rce seg m en t (co m p lex ity , am b iguity, h om ography).
■
T he im p rovem en t introduced b y the CL rule is m ask ed or blurred by
translation
p rob lem s
(m o rp h o lo g ica l
triggered
p rob lem s,
by
d uplicated
u ncontrollable
w ords,
phenom ena
w ron g
personal
p ron oun s)
■
T he im p rovem en t introduced b y the CL rule is m ask ed b y scorin g
in c o n siste n c ies
■
T he rule is n ot e ffe c tiv e w ith sp ecific langu age pairs b ecau se the
M T sy stem properly h and led the lin g u istic p h en om en on that the rule
tried to address
■
T he rule is o n ly e ffe c tiv e w ith certain ex am p les b ecau se the scop e
o f th e rule is too large for the lan gu age pairs and M T system
selec ted in th is study
138
■
T he e ffe c tiv e n e ss o f a rule cannot a lw a y s be reproduced b ecau se a
standard reform ulation is so m e tim es n ot available
■
T he im pact o f a CL rule m ay be n eg a tiv e w ith certain ex a m p les i f
am b igu ity is introduced in th e reform ulation
■
T he rule is o n ly e ffe c tiv e in o n e lan gu age pair b ecau se an alysis and
transfer rules d iffer from on e lan gu age pair to the n ext
4 .4 .1 .
L in g u is t ic p h e n o m e n a m a s k in g t h e e ffe c tiv e n e s s
o f a n in d iv id u a l r u le
D e sp ite attem pting to rem o v e certain co m p lex and am b iguou s structures from preCL and p ost-C L seg m en ts, certain lin g u istic p h en om en a m ask ed the e ffec tiv en ess
o f certain rules. L ea v in g certain p rob lem atic lin g u istic p h en om en a in the sou rce text
w a s, h o w ev er, so m e tim es n ecessary to a v o id u nderm ining the external v a lid ity o f
the study. I f o n ly sim p le sen ten ces had b een u sed in the test suite, it w o u ld be
d ifficu lt to gen eralise and state that the rule is e ffe c tiv e for a particular text type.
For instance, a p rob lem atic instance o f p o ly se m y w a s n oted w ith ex a m p le 168,
w h ich con tain ed tw o verbs, 'check' and 'verify', that resulted in the sam e translation
in G erm an, 'überprüfen'. H o w ev er, o n e o f th e sou rce verbs, 'check', w a s u sed in the
sen se o f 'activate'. T he e ffe c tiv e n e ss o f rule 26 m ay h ave b een m ore v isib le w ith
th is ex a m p le i f a different translation had b een entered in the G erm an user
dictionary.
A p rob lem atic in stan ce o f h om ograp h y w a s also fou n d w ith th e w ord 'copy', w h ich
w a s so m etim es an a ly sed as n ou n instead o f a verb (in exam p le 203 for French) or as
verb instead o f a n ou n (in ex a m p le 2 0 2 for G erm an). T h ese lex ica l prob lem s, w h ich
can be exacerb ated b y a lack o f term in o lo g y standardisation, w ere n ot addressed
w ith in the sco p e o f th is study. T h ey con firm , h ow ever, the assum ption that w as
m ad e b efore em barking on the d esign o f the test suite: the u se o f standardised
sou rce and target term s w o u ld im p rove the com p reh en sib ility o f M T output. T his
recom m en d ation w ill be retained for th e n ex t part o f the study.
139
4 . 4 *2 ’ E x t r a p h e n o m e n a a f f e c t i n g t h e e f f e c t i v e n e s s o f
a n in d iv id u a l r u le
B e sid e s
lin g u istic
'noise' introduced b y p h en o m en a su ch as co m p lex ity and
a m b igu ity (b e it le x ic a l, syn tactic, or referen tial), certain translation problem s
originated u n e x p e cte d ly in the M T output. For instance, in exam p le 2 1 5 , the
p erson al p ronoun 'you' w a s translated in the singular form in French ('te') despite
th e researcher h av in g selec ted sp ecific translation op tions prior to the translation
p ro cess. In other p la c e s, certain w ord s w ere so m etim es duplicated for no apparent
reason, such as 'zu' in ex a m p le 2 2 1 . T he d u p lication o f such a w ord in G erm an M T
output A im p acted the com p reh en sib ility o f M T output A , and m ay h ave su ggested
erron eou sly that th e rule w a s e ffe c tiv e in this particular exam p le. T h ese problem s
w ere d ifferen t from th e w ord order p rob lem m en tio n ed in sectio n 3 .3 .1 .2 .1 , so they
w ere n ot fix e d prior to th e evalu ation p rocess.
4 .4 .3 . A m b ig u it y c o r r e c tly h a n d le d b y th e M T s y s te m
A s m en tion ed during the r e v ie w o f the evalu ation results, certain rules w ere
so m etim es in e ffe c tiv e b eca u se the M T sy stem correctly d isam bigu ated th e pre-CL
seg m en t and p rod uced com p reh en sib le translations. For instance, th is w a s th e case
w ith ex a m p le 56 in G erm an, w h ere th e h om ograp h 'blocks' w a s correctly an alysed
as a third p erson verb instead o f a plural noun. T h is w a s also the ca se w ith m o st o f
the ex a m p les u sed to evalu ate the e ffe c tiv e n e ss o f rule 3 9 , 'Always write "in order
to ” before an infinitive in a purpose clause instead o f ju s t "to'". Other exam p les
fou n d in th e test su ite, h ow ever, sh o w ed that th e d isam bigu ation w a s n ot alw ays
perform ed su c c e ssfu lly (ex a m p les 103 and 179). B ased on this d isco v ery , it seem s
that th e rule w o u ld
so m e tim es
be e ffe c tiv e .
D e cid in g
w h eth er it is
worth
im p lem en tin g su ch a rule d esp ite its infrequent e ffec tiv en ess w ill th en vary from
on e en viron m en t to th e n ext.
4 .4 .4 .
S c o r in g in c o n s is te n c ie s a m o n g e v a lu a to r s
D e sp ite all o f the efforts that w ere m ad e to ensure co n sisten t and reliab le evalu ation
results, certain sco res p rovided b y th e evaluators so m etim es did n ot correspond to
th e ev a lu a tio n criteria that had b een d efin ed . Individual scorin g in co n sisten cies
w ere a v o id ed in th e an a ly sis o f th e results b y fo c u sin g sp ecifica lly on the exam p les
for w h ic h a m ajority o f evalu ators agreed. T his approach, h ow ever, w a s som etim es
14 0
n ot su fficien t to elim in ate scorin g in c o n siste n c ies. For instance, this w a s the case
w ith the French M T output A o f ex a m p le 144, w h ich receiv ed three 'E xcellent'
scores from French evaluators d esp ite b ein g syn tactically incorrect. T his w a s also
the case w ith the F rench M T output A o f ex a m p le 1, w h ich received tw o 'E xcellent'
scores d esp ite fa ilin g to c o n v e y inform ation accurately. T he im p rovem en ts brought
about b y the introduction o f the rule w ere therefore not v isib le w h en calcu latin g
p o sitiv e 'extra' sco res for th ese particular ex a m p les. It should be said, h o w ev er, that
such p rob lem atic scorin g in c o n siste n c ies in v o lv in g a m ajority o f evaluators w ere
rare.
4 .4 .5 .
S c o p e o f th e in it ia l r u le
Certain rules w ere o n ly effe c tiv e w ith sp e c ific ex a m p les b ecau se the sco p e o f the
initial rule w a s too w id e to be e ffe c tiv e co n sisten tly w ith the M T sy stem se lec ted in
this study. T his w a s the ca se w ith rule 16, w h ich o n ly p roved effe c tiv e for the
E nglish -F ren ch la n g u a g e pair w h en the ex a m p le con tain ed a verb in a past
p ro g ressiv e form . T h is w a s also the case w ith rule 3, 'Do not start a sentence M’ith
an im plicit subject in a subjectless, non-finite clause prem odijying a fin ite clause,
even when the subject is the same in both clauses'. T h is rule w a s o n ly e ffe c tiv e for
the E n glish -G erm an lan gu age pair w h en the ex a m p le contained tw o su b jectless
n o n -fin ite clau ses at the start o f a sen tence. T his issu e w as also noted w h en
evalu atin g the e ffe c tiv e n e s s o f rule 3 6 , 'The subject o f a non-finite clause must be
the same as that o f the main clause'. Final co n clu sio n s for this particular rule varied
based o n the typ e o f n o n -fin ite clau se present in the exam p les. M achin e-orien ted
rules m u st therefore be d efin ed m ore strictly i f their e ffe c tiv e n e ss is to be evaluated
m ore con sisten tly.
4 .4 .6 .
R a n g e o f r e fo r m u la tio n s
C lo se ly related to th e issu e s d iscu ssed in sectio n 4.4.1 and 4 .4 .3 is the issu e
a sso ciated w ith m u ltip le rew riting alternatives. T h e range o f p o ssib le reform ulations
asso ciated w ith certain rules so m etim es p revented the e ffec tiv en ess o f a rule from
b ein g v isib le w ith all o f the ex a m p les selected . For instance, this w as the ca se w ith
rule 5, 'Do not use more than 25 words p e r sentence'. D eterm inin g p recisely
w hether shorter sen te n c es w ill alw ays im p rove the com p reh en sib ility o f M T output
seem s d ifficu lt to a c h ie v e due to the various w a y s in w h ich lo n g sen ten ces can be
141
reform ulated. F in d in g a correlation b etw e en sen ten ce len gth and M T output quality
for ind ividu al sen ten ces is a ch a llen g e that had b een id en tified b y G dan iec (1994:
100). Stricter rew rite recom m en d ation s are therefore n ecessary to im p rove the
co n sisten cy o f reform ulation s.
4 .4 .7 . T h e im p a c t o f a r u le m a y b e n e g a tiv e i f a m b ig u ity
is i n t r o d u c e d i n t h e r e f o r m u l a t i o n
S o m e o f the results ob tained in th is study also sh o w ed that certain reform ulations
c o u ld be m ore am b igu ou s than the origin al segm en ts. T he com p reh en sib ility o f M T
output B w a s so m e tim es n eg a tiv ely a ffected b y th e introduction o f a rule. T his w as
v is ib le w ith so m e o f the ex a m p les u sed to evalu ate the e ffec tiv en ess o f rule 12, 'Use
the active voice when you know who or w hat d id the action'. S in ce the effec tiv en ess
o f th is rule had to b e evalu ated in d ivid u ally, no extra w ords (su ch as articles) w ere
introduced in the p ost-C L segm en t. W hereas n ou n s and verbs w ere correctly
d isam bigu ated in th e p a ss iv e structure con tain ed in certain pre-C L segm en ts,
a m b igu ity w a s introd uced in certain p ost-C L seg m en ts w ith the u se o f the active
v o ic e . In order to a v o id su ch sid e -effec ts, the sco p e o f the rule cou ld b e reduced or
be u sed in con ju n ction w ith rule 11.
4 .4 .8 .
T h e e f f e c t iv e n e s s o f t h e r u l e is v is ib le i n o n e
la n g u a g e p a i r o n ly
B a sed on the resu lts obtained in th is study, certain ru les clearly im p roved the
com p reh en sib ility o f th e output corresp on din g to a sp e c ific langu age pair. T his w as
the ca se w ith rule 3 1 , 'Whenever possible, the use o f "w h q u e s t i o n s should be
avoided' w h en its e ffe c tiv e n e s s w a s evaluated w ith th e E nglish -F ren ch langu age
pair. H o w ev er, su ch a rule w a s not e ffe c tiv e w ith th e E n glish -G erm an lan gu age pair
b eca u se the p re-C L seg m en ts w ere correctly h an d led b y the M T system . A sim ilar
p h en o m en o n w a s a lso n oted w ith so m e o f the ex a m p les u sed to evalu ate rule 6, 'Do
not omit the subordinating w o rd 'that' after verbs or verbal nouns'. T he translation
o f the m issin g su bordin ating w ord w a s n ot au tom atically introduced as freq uently in
G en n an M T output A as in French M T output A . T he rule w as therefore m ore
e ffe c tiv e for the E n glish -G erm an lan gu age pair. T h e m aturity o f a lan gu age pair
sh ould therefore n ot b e underestim ated in the evalu ation o f the effe c tiv e n e ss o f CL
142
rules. E valu ation s w ith m ore lan gu age pairs are required to determ ine w hether the
results obtained in th is study w o u ld apply to d ifferen t lan gu age pairs.
4 .4 .9 .
S e le c tin g a f i n a l lis t o f r u le s
D e sp ite the ch a llen g es en countered during the evalu ation o f ind ividu al ru les and
sum m arised in the p reviou s sectio n s (4.4.1 to 4 .4 .8 ), a fin al distribution o f rules can
be drawn b ased on their im p act on the com p reh en sib ility o f French and G erm an M T
output. T he 28 m o st e ffe c tiv e rules are p resen ted in T able 4 -9 (in n o particular
order).
CL rule name
=1
Rule 1: Avoid ambiguous coordinations by repeating the head noun, or by changing the word
order._________________________________________________________________
Rule 2: Do not use ambiguous attachments of non-finite clauses (present-participles).
Rule 4 : Do not omit hyphens in Noun + Adjective or Noun + Past Participle structures
Rule 5: Do not use more than 25 words per sentence as long as the short sentence is not
ambiguous.
______ _______ ________ _____
Rule 6: Do not omit the subordinating word 'that' after verbs or verbal nouns.
Rule 9: Do not omit relative pronouns and a form of the verb 'be' in a relative clause. The postmodifier is a past-partidple.
Rule 14: Do not make noun clusters of more than three nouns.
Rule 15: Do not use an ambiguous form of 'have'.
Rule 17: Do not use a pronoun when the pronoun does not refer to the noun it Immediately
follows.
Rule 19: Do not use pronouns that have no specific referent.
Rule 20: Do not use an ambiguous form of 'could'.
Rule 21: Do not use the slash to list lexical items.
Rule 22: Do not coordinate verbs or verbal phrases when the verbs do not have the same valency.
Rule 25: Two parts of a conjoined sentence should be of the same type.
Rule 26: Avoid the ellipsis of verb.
Rule 27: Avoid the ellipsis of subject.
Rule 28: Omit unnecessary words as long as ambiguity is not introduced.
Rule 29: Place a purpose clause before the main clause.
Rule 35: Avoid unusual punctuation (dashes with no surroundlngiiSgaçesJ.
Rule 36: The subject of a non-finite clause must be the same as that of the main clause.
Rule 41: Avoid embedded parenthetical expressions introduced by commas or dashes.
Rule 47: Move document names, error m essages, or section titles to independent segments.
Rule 48: Use a question mark only at the end of a direct question
Rule 50: If a single-word verb cannot be used, keep both parts of a verb together.
Rule 51: Avoid - ing words in specific situations (gerunds as subjects, coordinated -In g words).
Rule 52: Use single-word verbs.
Rule 53: Avoid ungrammatical constructions (agreement and possessive).
Rule 54: Check the spelling.
Table 4-9: Final selection of CL rules based on their effectiveness
1 43
T he final selec tio n o f rules presented in T able 4 -9 is b ased on an evalu ation that
w a s perform ed u sin g a sp e c ific M T sy ste m and sp e c ific lan gu age pairs. T h ese rules
w ere selec ted b ased on th e fo llo w in g criteria:
■
th e rule has to return at least on e 'extra' score in b oth lan gu age pairs
(an 'extra' score o f category 1 or 2). O ne ex c ep tio n to this criterion
con cern s rule 6 'Do not omit the subordinating w ord "that" after
verbs or verbal nouns', sin ce th e m icro-evalu ation o f exam p le 24
rev ea led that th e w e ll-fo rm e d n ess o f th e French M T output cou ld be
im p roved w ith th e ap p lication o f th is rule.
■
the rule d o es n ot h ave any n eg a tiv e im p act in either target langu age.
O ne ex c e p tio n to th is criterion con cern rale 15, 'Do not use an
ambiguous form o f "have'", w h ich returned a h igh num ber o f 'extra'
scores in G erm an, and sh ow ed so m e poten tial im p rovem en t in
French during the m icro-evalu ation o f ex a m p le 153.
■
the rule d o es n ot c o n flict w ith a rule that has a
m ajor im p act for both
lan gu ages (for th is reason, rule 18, 'Do not use pronouns', w as
discard ed sin c e it co n flicts w ith reform ulation s triggered b y rule 22,
' Do not coordinate verbs or verbal ph rases when the verbs do not
have the same valency'.
B efo r e further cla im s can b e m ad e o n the e ffe c tiv e n e s s o f th ese rules, their
e ffe c tiv e n e ss m u st b e evalu ated at the d ocu m en t le v e l. S o m e o f th ese rules w ill be
u sed in th e seco n d part o f th is study i f th ese ru les are v io la ted in the d ocu m en ts
u sed for the o n lin e exp erim en t. T his strategy w ill h elp s us determ ine w h eth er their
ap p lication can im p rove the u sefu ln e ss, com p reh en sib ility, and accep tab ility o f M T
d ocu m en ts.
144
4 .5 - S u m m a r y
In th is chapter, an evalu ation o f th e effe c tiv e n e ss o f CL ru les on M T output w as
p erform ed, lead in g to four m ain fin d in gs. Firstly, a lim ited num ber o f C L rules
p roved freq uently e ffe c tiv e for the tw o lan gu age pairs u sed in this study. S econ d ly,
so m e o f the rules n eed to a cco m m o d a te ex cep tio n s. For instance, the L SP u sed in
th e corpus cou ld n o t operate w ith ou t k ey verbs such as 'back u p ’ or T og o n ’,
d esp ite c o n flictin g w ith o n e o f the m o st effe c tiv e rules (rule 52). Thirdly, certain
CL ru les, su ch as rules 7, 8, and 9 h ave traditionally b een con sid ered as guarantees
to im p rove m ach in e translatability. T he fin d in gs, h o w ev er, sh o w ed that the sco p e o f
su ch rules cou ld b e restricted to am b igu ou s ca ses o n ly , to a v o id a situation w here
the sty le o f th e sou rce te x t is n e g a tiv e ly affected w ith ou t n ecessa rily im p rovin g the
M T output. F in ally, o n ly a fe w ru les im p roved e x c lu siv e ly M T output in o n e target
lan gu age. T h ese rules can still b e ap p lied to the sou rce tex t as lon g as th ey h ave no
n eg a tiv e im p act on th e other lan gu age pair.
1 45
C hapter 5
14 6
Chapter 5: Setting up an online experiment
to evaluate the impact of CL rules at the
document level
5.1. Objectives o f the p resen t chapter
A t th e end o f Chapter 4 , e ffe c tiv e C L ru les w e re cla ssified based on th e typ e o f
im p act th ey had on the com p reh en sib ility o f M L output at the segm en t lev el. In the
present chapter, a num ber o f tech n ica l support d ocu m en ts are extracted from the
corpus, so as to evalu ate the im pact o f so m e o f th ese rules at the d ocu m en t level.
T he m e th o d o lo g y u sed to a ch iev e th is o b jectiv e is d iscu ssed in sectio n 5.2. A n
o n lin e exp erim en t u sin g a cu stom er satisfaction questionnaire is con d u cted to
c o lle c t users' reaction s to d ocu m en ts con tain in g m achine-translated content. The
first part o f th is chapter w ill fo c u s on the m eth o d o lo g ica l d ecisio n s m ad e during the
d esig n o f the on lin e exp erim en t. A fter re v ie w in g related stu d ies to op tim ise the
m e th o d o lo g y u sed in th is study, th e fin al exp erim en t setup is described in greater
d etail, a lon g w ith the strategy e m p lo y ed to m achine-translate the d ocu m en ts that
w ill b e p resen ted to gen u in e users. T h e sec o n d part o f th is chapter is con cern ed w ith
an ex a m in a tio n o f the d ocu m en ts se lec ted for the experim ent. T his exam in ation is
perform ed to d eterm in e the term in o lo g ica l entries that should b e en cod ed in the M T
system 's user d iction aries, and id en tify rule v io la tio n s present in th e original
d ocu m en ts. R eform u lation s w ill th en b e applied to original d ocu m en ts so as to
p roduce a set o f con trolled d ocu m en ts.
5.2. M ethodology selection
5 .2 .1 . R e v ie w o f d a t a c o lle c t io n te c h n iq u e s
A s n oted in the introduction, no em pirical study has ever b een sp ecifica lly
co n d u cted
to
evalu ate
th e
e ffe c tiv e n e ss
of
CL
rules
on
the
u sefu ln ess,
com p reh en sib ility, and accep tab ility o f m achine-translated d ocu m en ts from a W eb
user's p ersp ective. T here is therefore no standard m e th o d o lo g y to rely o n for data
c o llec tio n . In order to evalu ate the im p act o f the ind ep en d en t variable (the p resence
or a b sen ce o f C L rule v io la tio n s) o n the three dep en dent variab les exam in ed ,
several related stu d ies are re v ie w ed to d eterm ine the m o st suited m eth od ological
approach. A standard m e th o d o lo g y m ay not b e available, but certain com p on en ts
147
m ay b e reused from p reviou s related stu d ies. T w o related studies, w h ich fo c u s on
the u se o f SE in tech n ical d ocu m en tation (Shubert et al.: 1995; Spyridakis et al.:
1 9 9 7 ), are first rev iew ed . A nother related study is then review ed in the field o f
a cc essib ility and accep tab ility in tech n ical d ocu m en tation (L assen , 2 0 0 3 ). F inally,
tw o related stu d ies are rev iew ed in the fie ld o f o n lin e m achine-translated d ocu m en t
p u b lication and cu stom er satisfaction (R ichardson , 2 0 0 4 ; Jaeger, 2 0 0 4 ).
5 .2 .1 .1 . R e v i e w o f s t u d i e s f o c u s i n g o n s o u r c e d o c u m e n t s
A s d iscu ssed in se c tio n 2 .3 .1 .1 during a r e v ie w o f C L evalu ation m eth od s, the
e ffe c tiv e n e ss o f a set o f CL rules on the com p reh en sib ility o f a d ocu m en t has
so m etim es b een evalu ated u sin g com p reh en sion tests (Shubert et al., 1995). The
m ain strength o f this exp erim en t w a s its u se o f g en u in e SE and n on -S E d ocu m en ts tw o d ocu m en ts w ere u sed , each on e con tain in g tw o v ersion s o f the sam e procedure.
T he im p act o f the SE rule set w as th en evalu ated by com paring com p reh en sion
results obtained from variou s groups o f su bjects. In this study, the fo c u s w a s p laced
on th e control o f th e subjects sin ce 121 undergraduate students w ere u sed instead o f
gen u in e users. T h is approach did n ot fo llo w the recom m en d ation m ad e b y K in g
(1 9 9 6 : 198), w h o rem arks that an evalu ation study b ased on a com p reh en sion test
sh ould in v o lv e 'con sum ers as lo n g as th ey can b e identified'. O verall, the typ e o f
exp erim en t u sed b y Shubert et al. (ib id) see m s to b e a g o o d starting p o in t for the
present study as lo n g as subjects are gen u in e users o f tech n ical d ocum entation.
U sin g tw o d ifferen t v er sio n s o f the sam e d ocu m en t and com parin g the reactions
obtained from tw o groups o f subjects p rovid es a sou nd b a sis to determ ine i f the tw o
groups re ceiv e M T d ocu m en ts d ifferen tly. U sin g tw o d ocu m en ts a lso seem s
ad vantageou s to en su re that results can b e reproduced from on e d ocu m en t to the
next. A co m p reh en sio n test, h o w ev er, d o es n ot appear appropriate to obtain
reaction s
from
u sers,
as H olm b ack
et al.
(19 9 6 :
176)
state that 'taking a
com p reh en sion test is probably n ot the b est w a y to m easure com p reh en sion o f a
procedure ( . . . ) sin c e procedures are w ritten to b e perform ed, n ot quizzed'. T his
rem ark su g g ests that a u sab ility study cou ld be perform ed to determ ine w h eth er the
ap plication o f C L rules in procedural text can h ave an im pact on the ex e cu tio n o f
th is p rocedure (su ch as th e sp eed w ith w h ich it is execu ted ). H ow ever, a laboratorybased u sab ility stu d y p resents d isad van tages, su ch as the unfam iliar settin g in w h ich
users w ill be m on itored and the sm all sam p les o f subjects u sed (S p yridak is et al.,
14 8
2 0 0 5 : 2 4 6 ). A s m en tion ed in the introduction, a u sab ility study is not appropriate
w ith in the sco p e o f th is study. It is, h o w ev er, essen tia l that the evalu ation o f
d ocu m en ts is p erform ed by g en u in e users o f su ch d ocu m en ts to m a x im ise the
e c o lo g ic a l v a lid ity o f the study. G en uin e users co n su lt on lin e tech n ical support
d ocu m en ts b eca u se th ey h a v e en cou n tered a p rob lem or b ecau se th ey h ave a
q u estion that rem ains u nansw ered. G en uin e users are therefore different from
ex istin g cu stom ers w h o cou ld be con tacted and ask ed to evaluate a d ocu m en t that
d o es n ot con cern them .
For instance, th is w a s the approach that w a s taken b y L assen (2003: 7 9 ) w h o used
an o fflin e su rvey to in v e stig a te u ser attitudes to the n otion s o f a cc essib ility and
a ccep tab ility in te ch n ica l d ocu m en tation . In order to m in im ise n on -resp on se rates,
sh e contacted p ro sp ectiv e respon d en ts before m a ilin g th em a questionnaire. D esp ite
u sin g th is approach, resp on se rates w ere around 14% (ibid: 80). U sin g an o fflin e
approach p resents certain advantages, e sp e c ia lly w ith regard to tim e constraints.
G illh am (20 0 0 : 7 ) m en tio n s that o fflin e su rveys generate “le ss pressure for an
im m ed iate resp o n se sin c e respon d en ts can an sw er in their o w n tim e” . T o som e
extent, su ch a setup cou ld be reproduced in an on lin e en vironm ent, w h ereb y
exp erim en tal m achin e-tran slated d ocu m en ts co u ld be p osted on lin e. Fake users
cou ld th en b e con tacted and asked to g iv e their o p in io n s on certain asp ects o f the
translated content. W ith su ch an approach, h o w ev er, the real life scenario d iscu ssed
earlier w o u ld not b e in p lace. I f gen u in e users are to be u sed w ith in th e con text o f
an exp erim en tal d esig n , the data co lle c tio n instrum ent m u st be part o f the m aterial
that is b ein g evalu ated . In an on lin e con text, th is tech n iq u e is often im p lem en ted
w ith cu stom er sa tisfa ctio n feed b ack m ech an ism s. T w o ex am p les o f studies that
m ad e u se o f cu sto m er satisfaction rates to evalu ate certain characteristics o f
m achin e-tran slated con ten t are re v ie w ed in th e n ex t section .
149
5 .2.1.2. R e v i e w o f s t u d i e s f o c u s i n g o n t r a n s l a t e d d o c u m e n t s
R ich ard son (2 0 0 4 ) reports on a project con d u cted at M icrosoft in collab oration w ith
th e group resp o n sib le for the translation o f K n o w le d g e B ase product support
articles. In this p roject, m ore than a hundred th ou sand product support articles w ere
translated into S pan ish and Japanese u sin g M icrosoft's M T system , M S R -M T
(R ichardson et al., 2 0 0 1 ). D uring a four-m onth p ilo t study, R ob in son (2004: 2 4 7 )
m en tion s that feed b a ck w as obtained from S p an ish users b y 'surveying a sm all
sam p le o f the ap p roxim ately 6 0 ,0 0 0 v isits to th e w eb site'. D uring th is p eriod , 380
su rveys w ere ob tained, but it is not clear w h eth er all visitors w ere g iv en the chance
to fill in th e su rvey. I f th ey w ere, the resp on se rate is 0.65% for Spanish and 1.5%
for Japanese, su g g estin g that users' feed b ack is d ifficu lt to obtain. T h ese M icrosoft
users w ere asked to evalu ate certain characteristics o f the m achine-translated
K n o w le d g e B ase content. R ichardson d oes n ot p rovide the exact q u estion s that
users w ere ask ed to
answ er, but p rovid es rates in the fo llo w in g categories
(R ichardson, 2 004: 2 48):
■
% o f cu stom ers w h o are satisfied w ith K B
■
% o f cu stom ers w h o w ere h elp ed to so lv e their issu e s u sin g K B
(u sefu ln ess rate)
■
% o f cu stom ers w h o th ou ght inform ation is ea sy to understand
T h ese ca teg o ries are interesting b eca u se th ey share so m e sim ilarities w ith th e three
variab les
that
are
o f interest
in
th is
study
(accep tab ility,
u sefu ln ess,
and
co m p reh en sib ility). O ne o f th ese variab les, u sefu ln ess, is also found in a study that
w a s con d u cted at C isc o (Jaeger, 2 0 0 4 ). In O ctober 2 0 0 3 , C isco used a cu stom ised
Systran M T sy stem to translate m ore than 4 ,5 0 0 tech n ical support d ocu m en ts into
Japanese. O nce th e o n lin e translations w ere p u b lish ed , a W eb survey w a s u sed to
determ ine the d egree o f u sefu ln ess o f th ese M T d ocu m en ts for Japanese custom ers.
In January 2 0 0 4 , Jaeger reports that 461 resp o n ses w e re obtained. A gain , it is not
clear w h eth er all o f the ap proxim ately 1 0 0 ,0 0 0 visitors w ere asked to fill in a
cu stom er satisfaction su rvey. I f th ey w ere, the resp on se rate for th is particular study
is slig h tly less than 0.5% .
150
In both th e M icro so ft study and th e C isco study, the q uestion s used in custom er
sa tisfaction q u estionn aires see m to h ave fo c u se d on the characteristics o f the w h o le
k n o w le d g e base's con ten t rather than on sp e c ific d ocu m en ts. In the p resent study,
q u estion s
m u st relate
to
sp ecific
d ocu m en ts.
T he
experim ent's
d esig n
and
instrum ent are d isc u sse d in turn in th e n ex t tw o section s.
5 .2 .2 .
D e s ig n o f a n o n lin e e x p e r im e n t
B ased on the r e v ie w o f related stu d ies, a com b in ation o f several approaches has
b een id en tified to a c h ie v e th e seco n d o b jectiv e o f the study. A n o n lin e experim ent
m u st be set up, w h ereb y users w ill p rovid e their reaction s to m achine-translated
d ocu m en ts b y fillin g in a questionnaire. T he m ain advantages o f con d ucting an
o n lin e exp erim en t (or W eb exp erim en t) h ave b een h igh ligh ted b y R eip s (2002:
2 45):
■
ea se o f a cc ess to a large num ber o f d iverse participants
■
ea se o f a c c e ss for participants w ith voluntary participation
■
h igh external valid ity
■
a v o id a n ce o f tim e constraints
W hereas a laboratory-based exp erim en t w o u ld b e lim ited to a sm all num ber o f
participants e v o lv in g in an artificial en viron m en t, an o n lin e exp erim en t can reach
m an y participants w ith d iverse backgrounds and tech n ical settings. T he u se o f this
approach guarantees the ec o lo g ic a l va lid ity o f th e exp erim en t and en su res that the
fin d in gs can b e g en era lised to the popu lation .
H ew so n et al. (2 0 0 3 : 4 8 ) state that 'in an exp erim en tal d esig n the researcher
m an ipu lates the in d ep en d en t variab le(s) in order to m easure the e ffe c t on the
d ependent variables'. In the present study, the feed b ack p rovided by users w ill be
a n alysed to d eterm in e w h eth er the introduction o f C L rules (the independent
variable) has an y e ffe c t o n the fo llo w in g three d ependent variables: u sefu ln ess,
com p reh en sib ility, and accep tab ility o f the m achine-translated d ocu m en ts. The
ind ep en d en t variab le w ill b e m anipulated b y u sin g certain d ocu m en ts that contain
C L rule v io la tio n s and others that do not.
151
O ne p o ssib le w a y to set up the exp erim en t w o u ld be to select a sp ecific num ber o f
d ocu m en ts from the corpus, and apply certain CL rules to h a lf o f th ese docum ents.
T he d ocu m en ts w o u ld th en b e p ub lished, and users asked to g iv e their reaction s to
th ese
d ocu m en ts.
D iffere n c es
b etw e en
the
reaction s
obtained
for controlled
d ocu m en ts and original d ocu m en ts w o u ld th en b e m easured. T his approach w ou ld ,
h o w ev er, underm ine the internal v a lid ity o f the study, b ecau se o f con fou n d in g
variables. It w o u ld b e extrem ely d ifficu lt to draw co n clu sio n s w ith regard to the
e ffe c tiv e n e ss o f the ru les i f b oth sets o f d ocu m en ts w ere n ot equal b efore the
ap plication o f the rules.
T he approach u sed in th is study is to p u b lish th e sam e num ber o f original and
con trolled v ersio n s o f a set o f d ocu m en ts. W h en users a cc ess th ese d ocu m en ts, th ey
w ill then be ran d om ly p rovid ed w ith on e o f th e tw o version s and ask ed to fill in a
questionnaire b ased on their exp erien ce. S in ce this evalu ation exercise should be as
realistic as p o ssib le, th ey w ill n ot be asked to com pare b oth v ersion s. Their
ex p erien ce w ill be as c lo se as p o ssib le to w h at it w o u ld have b een w ith hum an
translation. T he o n ly d ifferen ce w ill co n sist in th e p resen ce o f a d isclaim er at the
start o f the d ocu m en t to warn them that th ey are about to read a m achine-translated
d ocu m en t. T his d iscla im er w ill also in sist on the sig n ific a n c e o f their feed b ack , and
on the fact that the m achin e-tran slated v er sio n o f the d ocu m en t is p rovided w ith in
th e fram ew ork o f a p ilo t project. A ccord in g to R eid (2000:
160), th ese tw o
requirem ents are am o n g the ‘b asic rules in persuading cu stom ers to p rovide
fe ed b a c k ’. C u stom ers m u st b e told that their o p in io n is valu ed , and m u st k n o w w h y
th e inform ation is required (ibid). A d h erin g to th ese p rinciples is essen tial to ensure
that both parties fu lly understand the im p lica tio n s o f su ch an evaluation.
F igu re 5-1 p resents th e d isclaim er that is u sed in the French docum ents:
152
p ro d u its e t se rv ic e s
Id e n tific atio n <Jocument:2CiO£D£3110421 (3 2 3
De rn ¡ è re tè v i sion:C g /3 1,;2ü3?-
im prim er ce document
a chats
su p p o rt
partenaires/revendeurs
security response
téléchargements
à propos de Symantec
recherche
Déni de responsabilité :
Nous tenons à vous informer que le présent-document a été traduit par l'intermédiaire d'un outil
automatique dans le cadre d'un projet pilote Nous vous serions donc infinim ent reconnaissants
de bien vouloir nous donner votre avis sur F utilité de cette traduction Auriez-vous une petite
m inute pour répondre à n otre q u e s tio n n a ire ?
Désactiver l'analyse du courrier électronique dans Norton Internet Security
v o ir e a v is
Situation;
# 1SSE -Z O T C S y m a n t e c
C c r g æ ts ti s n
AU r ig h ts i e s e r * e a
L = oal N o n ces
Vous ne voulez pas que Norton Internet Security analyse votre courrier électronique Vous pouvez
désactiver l'analyse du courrier électronique de Norton A n tiv iru s mais Norton Internet Security
analyse toujours votre courrier électronique Vous voulez savoir empêcher f^orton Internet Security
d'analyser votre courrier électronique
IraiBiBli* m S srftM " H n * n i
Solution?
Norton Internet Security peut analyser votre courrier électronique dans diverses façons. Pour
em pêcher Norton Internet S ecurity d analyser votre courrier électronique entièrement, com plétez des
étapes dans la section qui s'applique à votre version de Norton Internet Security. Cliquez sur l’icône
précédant une section afin de développer ( S J ou de réduire ( □ } cette section Consultez le
document intitulé Impossible de développer les sections dans un document de la base de doonees
_________ ______ ___
^ y i i r i r p u . ' ç.-fivoir com m ent résoudre ce problème
Figure 5-1: French disclaimer used within a custom template
F igu re 5-1 sh o w s that users w ill k n o w prior to their con su ltin g o f the d ocu m en t that
their feed b ack is required. T h is is n ece ssa ry to m a x im ise the num ber o f resp on ses
subm itted. O btaining sig n ifica n t resp on se rates is o n e o f the m ain ob jectiv es o f
su rv ey-b ased research. O ne o f the issu e s w ith q uestionn aires m en tion ed by G illh am
(2 0 0 0 :
10), h o w ev er, is the p oten tial lack o f m otiv a tio n from respondents.
A cco rd in g to h im , “fe w p eo p le are stron gly m otivated b y q uestionnaires u n less they
can se e it as h a v in g p erson al re lev a n ce” . In order to en cou rage users to p rovide
feed b ack , it w a s en v isa g ed to u se an in c en tiv e b y g iv in g aw ay a prize. A study
(G öritz, 2 0 0 6 ) fou n d that m aterial in c en tiv es increase resp on se and d ecrease
dropout in W eb su rveys. A draw w o u ld then h ave b een b ased on the co lle c tio n o f
all resp on d en ts’ an sw ers. T h is m eant that cu sto m ers’ id en tities had to b e co llected ,
w h ic h m ay h a v e d iscou raged so m e cu stom ers. I f o n ly em ail addresses had b een
co llec ted , certain u sers cou ld have b een tem p ted to subm it several form s u sin g
different em ail ad dresses to m a x im ise their ch an ces o f su ccess. In th e end, it w as
d ecid ed n ot to u se an in cen tiv e, but to em p h a sise the altruistic appeal. S ectio n 5.2.3
co v ers in detail the instru m en t that w ill b e u sed to c o lle c t feed b ack from users.
153
5.2.3. Instrumentation
5 .2 .3 .1 . P r o s a n d c o n s o f c u s to m e r s a tis fa c tio n f e e d b a c k
m e c h a n is m s
S y m a n tec’s tech n ica l support group u se s a W eb -b ased feed back m ech a n ism that
a llo w s th em to m on itor their custom ers' satisfaction le v e ls, as sh ow n by F igure 5-2.
Klicken Sie hier. um die enqlische Version dieses Dokuments anzuzeiqan
H in w e is : Bitte beachten Sie. dass aufgrund des Zeitbedarfs für die Übersetzung ins Deutsche das
englische Originaldokum ent in der Zwischenzeit m ö g lich e w e ise aktualisiert wurde, wodurch die
deutsche Version inhaltlich abweichen kann.
^ 4
D' eses Dokum ent drucken
B ew ertung des D o k u m e n ts |____________________________________________________________
W u r d e Ih ie F j a g e d u r d i d i e s e s
w e r d e n k ö n n te .
Ja
W
S i e « ¿ h a lte n h ie r b e i Ä e in e A n tw o rt, B itte s e -n d e n S i e le d ig lic h V a r s d l l e g e e i n ; d i e
d e ? V e r b e s s e r u n g a i e s s s D o k u m e n ts d ie n e r s .
N em
' \ J V ie ife ic b fc . k f o m i . s s
Ist d i e s e s D o k u m e n t g u t g e s c h f i s b s n u n d le ic h t n u v e r s t e h e n d
S e ü d e n S i e u n s V o f s c n l s g e d a z u , v«-ie ö i s G u a l i t a l d i e s e s D o k u m e n t s v e i b e s s s r l
C c fc u ro e n t D s a n h v c J te f7
L ö su r-j
!3©5k 5iS=-piOi>i£T&T5
K y K sä!*£ - s e r A n t w o r t e n t r i f f t za-
, S enden ]
F r o d u f t h e z e i c f t m / n g : f t c n & n A n tiV f c r u s 2 0 0 6
B e & ie fe s s y s t e r n : W in d iy .v s S -S iJ C . W i n d o w s X F
E r s t e l I u n g s d a t u 0 1: 1-0 / 0 5 / 2 0 0 5
Figure 5-2: Customer satisfaction form for technical support documents
T h is feed b a ck m e ch a n ism u ses a com b in ation o f clo sed and op en q u estion s, w h ich
m ak e it d ifficu lt to obtain sp ecific an sw ers. T he q u estion ‘W as you r q u estion
a n sw ered b y th is d o cu m en t? ’ seem s to o p recise b eca u se it a llo w s resp on d en ts to
su bm it an sw ers w ith o u t ind icatin g w h eth er th e d ocu m en t w a s u sefu l. A d ocu m en t
m ay n ot an sw er a sp e c ific question, but it m ay be u sefu l b y p oin tin g to other
referen ce m aterials. O n the other hand, the free tex t satisfaction b o x d oes not
en cou rage p recise an sw ers. For desperate or frustrated custom ers, this m ay e v e n be
seen as the o n ly w a y to request h elp from the organisation. T his w a s con firm ed by
e x a m in in g so m e o f th e feed back receiv ed in p reviou s m onths. V alid remarks w ere
m ix e d w ith a large num ber o f tech n ical support requests, su ch as ‘p le a se call m e at
th is num ber to te ll m e h o w t o . ..
154
S uch a feed b ack m ech a n ism is in p la ce b eca u se ‘it is gen erally agreed that the
organ ization m u st listen to cu stom ers to im p rove th e quality o f g o o d s and se r v ic e s.’
(A rm istead & Clark, 1992: 1). O f cou rse, the feed b a ck is p rovided b y cu stom ers on
a volun tary basis, and real con stru ctive feed b ack m ay b e scarce. S in ce there is no
sp ecia l in cen tiv e for users to p rovide feed b a ck - apart from a p rom ise o f im proved
serv ices - sa tisfied cu stom ers m ay n ot fe e l the urge to state that th e d ocu m en t they
co n su lted an sw ered their q u estion s. In m o st ca ses, their o n lin e exp osu re to the
tech nical support d ocu m en t w a s triggered b y a p rob lem they did n ot ex p ect in the
first p lace, so w h y sh ould th ey feel grateful to th e com p an y from w h ich th ey bought
th e product? O n th e other hand, d issa tisfie d cu stom ers can lea v e the W eb page
w ith o u t m ak ing a com m en t. T his m a k es it d ifficu lt to understand h o w th e service
co u ld be im p roved . In this study, sile n c e from b oth satisfied and d issa tisfied users
co u ld a ffect the accuracy and reliab ility o f th e results. T h is issu e is d iscu ssed in the
n ex t section .
5 .2 .3 .2 . E n s u r in g t h e a c c u r a c y a n d r e l ia b il it y o f t h e r e s u lts
In th e present study, the em p h asis m u st b e p la ced o n sp ecific characteristics o f
d ocu m en ts b y so lic itin g feed b ack from users w h o w ill b e ex p o sed to such m achin etranslated d ocu m en ts. T his approach p resents so m e issu e s w ith regard to the
reliab ility o f th e resu lts sin c e the q uality and v eracity o f an onym ous an sw ers cannot
be con trolled. H o w ev e r, H e w so n et al. (2 0 0 2 : 4 3 ) state that ‘in general w e m ay
a ssu m e that our participants are g iv in g gen u in e and accurate answers' (20 0 3 : 44).
B e sid e s, u sin g a n on ym ou s an sw ers sh ould reduce the p rob lem o f so cia l desirability
m en tio n ed b y D e V a u s (2 0 0 2 : 107). R esp on d en ts w ill n ot fe el under any pressure to
p ro v id e ‘g o o d ’ an sw ers w h en p rovid in g feed back.
O btaining as m an y an sw ers as p o ssib le is essen tial to ensure that th e statistical
p o w er o f th e exp erim en t is su fficien t. E van s et al. (2 0 0 4 :1 4 ) d efin e statistical p ow er
as 'the ab ility to d etect a d ifferen ce b etw e en treatm ent groups in an exp erim en t, i f a
true d ifferen ce exists'. In order to a v o id a ccep tin g fa lse h y p o th eses or rejecting true
h y p o th eses, the statistical p ow er o f th e g iv en exp erim en t m u st b e su fficien t. T his
can b e a ch iev ed b y in creasin g the num ber o f respon d en ts. E vans et al. (ib id) state
that w h en the 'pow er is n ot h ig h en ou gh , a study m a y n ot p roduce a statistically
sig n ifica n t result e v e n w h en there is a true difference'. Increased accuracy w ill
1 55
therefore be obtained b y redu cin g the sam p lin g error to a m inim um . D e V au s (2002:
81) m en tio n s that th e sam p lin g error o f a sam p le siz e o f 100 is 10%. T he longer the
exp erim en t stays o n lin e, the m ore accurate the results, i f the num ber o f respondents
in creases o v er tim e. T he com p on en ts o f the instrum ent used in the exp erim en t are
d escribed in the n e x t section .
5 . 2 . 3 . 3 . I n s t r u m e n t a t i o n d e s ig n
U sin g o p en q u estion s in a W eb su rvey p resen ts disad van tages w h en there is no
control over the se le c tio n o f respondents. In her study, G alesic (2006: 3 2 5 ) found
that 'open q u estion s see m to n eg a tiv ely a ffect interest in the questions'. In order to
m a x im ise resp on se rates, sh e m en tion s that resp on d en ts m ust be kept interested. A s
op en q u estion s sh o u ld be avoid ed , u sin g a short list o f sim p le clo sed q u estion s and
answ ers
appears
to
be
the
m o st
su itable
approach.
T his
also
fo llo w s
the
recom m en dation m ad e by Jackob and Z erback (20 0 6 : 20). It w as d ecid ed to u se
d ich otom ou s resp on ses, w h ereb y respon d en ts w o u ld b e asked to selec t on e o f tw o
alternatives ( ‘Y e s ’ and ‘N o ’). D e V au s (2 0 0 2 : 106) w arns that “the danger w ith
u sin g ‘d o n ’t k n o w ’ and ‘n o o p in io n ’ alternatives is that so m e respondents select
th em out o f laziness ” . In order to avoid th is prob lem , o n ly tw o radio b o x e s w ou ld
be d isp layed alon g ea ch question. S y m a n tec’s ex istin g q u estion ('D id this d ocu m en t
answ er your q uestion?') w a s retained and su p p lem en ted w ith four others fo llo w in g a
lo g ica l flo w . T h ese four extra q u estion s were:
■
W as this d ocu m en t u sefu l?
■
D id the q uality o f the translation ham per your com preh en sion ?
■
W o u ld y o u h ave preferred to read the E n g lish v ersion o f this
d ocu m en t?
■
I f y o u en countered another p rob lem in the future, w o u ld you con su lt
again
a
m achine-translated
d ocu m en t
from
the
Sym an tec
K n o w le d g e B ase?
T he first q u estion is ask ed to determ ine w h eth er a m achine-translated tech n ical
support d ocu m en t ca n h elp users so lv e their p rob lem s. P o sitiv e answ ers to this
q u estion sh ould sh o w that the u se o f refined M T output, w hether th e source
d ocu m en t is in its original form or con trolled form , g o e s b eyon d th e m ere
understanding o f th e overall th em e o f th e d ocu m en t. It m ay not be fu lly autom atic
156
h igh quality translation, but as V an der M eer (2 0 0 6 ) puts it, c lo se to ‘fu lly
autom atic u sefu l translation ’. T he d ifferen ce in term s o f 'yes' and 'no' answ ers
sh ou ld indicate w h eth er the u se o f C L rules can p la y a part in the overall co g n itiv e
p ro cess w ith w h ich th e users h a v e b e e n confronted. S in ce M T output is n ot as fluent
as hum an translation, and in so m e ca ses, n ot as accurate, users m ay find the
d ocu m en t u sefu l but at a cost. D u e to unusual syn tax or w ord order, it m ay take
th em lon ger to understand the inform ation con tain ed in the docum ent, but still find
it u sefu l. T he an sw ers to th e q u estion 'Did the q uality o f the translation ham per your
com prehension?' sh ou ld therefore h elp u s to fin d out w h eth er CL ru les can h elp M T
output b eco m e m ore com p reh en sib le, and in turn m ore acceptable.
A s stated in the introduction, th e con cep t o f a ccep tab ility is tigh tly intertw ined w ith
the con cep t o f u sefu ln e ss, sin c e the accep tab ility and u se o f a product or serv ice are
triggered b y an ‘im p lic it or e x p lic it co st-b en efit analysis' (S h ack el, 1990: 3 2 ). From
this statem ent, it is inferred that users w ill fin d m achine-translated d ocu m en tation
accep tab le w h en th ey tolerate so m e o f the textu al disturbances caused by an M T
p ro cess. In this case, the num ber o f gram m atical or sty listic disturbances d oes not
o u tw eig h the num ber o f textu al characteristics that u sers ex p ect to fin d in tech n ical
support d ocu m en tation . L assen (2003: 81) states that 'acceptability is an am b igu ou s
n o tio n that m ay im p ly gram m aticality to so m e respon d en ts, w h ile it m ay im p ly
sty listic accep tab ility to others'. W h en an alysin g th e results o f her survey on the
accep tab ility o f tech n ica l m anuals, L assen (ib id ) fou n d that the term 'acceptable'
had n ot b een u n d erstood in the sam e w a y b y all respondents. In order to avoid such
a situation in the p resen t study, it w a s d ecid ed to u se tw o separate q u estion s to
evalu ate th e accep tab ility o f m achin e-tran slated d ocu m en tation , w ith ou t u sin g the
term 'acceptable'.
T he first o f th ese tw o q u estion s asks users to state w hether th ey w o u ld have
preferred to read th e E n g lish v er sio n o f the d ocu m en t. T h is q uestion is b ein g asked
b eca u se the m achin e-tran slated d ocu m en t m a y lack so m e o f the ex p ected features
that m ak e the d o cu m en t a 'sp ecim en o f the genre' (L assen , 2003: 7 8 ). R egard less o f
their le v e l o f p ro ficie n c y in E n g lish and d esp ite the overall u sefu ln ess o f the
d ocu m en t, certain u sers m ay an sw er that th ey prefer reading an E n glish v er sio n o f
this d ocu m en t rather than a m achin e-tran slated v er sio n in their o w n language. This
157
w o u ld su g g est that th e intrinsic quality o f a d ocu m en t g o es b eyon d its u sefu ln ess.
N o t o n ly sh ould a d ocu m en t be u sefu l, but it sh ou ld also includ e indicators o f
cred ib ility and sty listic accuracy that m e et th e exp ectation s o f its receivers. T his
in v estig a tio n w ill b e rein forced b y the fin al q u estion , w h ic h w ill ask users to state
w h eth er th ey w o u ld co n su lt again a m achin e-tran slated d ocu m en t in th e future. T his
q u estion is ask ed to d eterm in e h o w accep tab le an M T d ocu m en t is to its receivers
sin ce the an sw ers to th is q u estion w ill depend on the results o f the users' costb en efit a n a ly sis o f the serv ice p rovided. I f M T d ocu m en ts are m issin g too m any k ey
sty listic and gram m atical features, it is inferred that th ey w ill b e rejected b y users
and n ot con su lted again in th e future. T he q u estion s u sed in the French su rvey are
sh o w n in F igure 5-3:
Aidez-nous à mieux connaître vos besoins !
Vous pouvez nous aider à améliorer la qualité de nos prestations en répondant aux questions
suivantes
yj Oui
Ce document vous a-t-il servi ?
Ce document a-t-il répondu à votre question ?
La qualité de cette traduction vous-a-t-elle empêché de comprendre ce
document ?
Auriez-vous préféré consulter la version anglaise de ce document ?
Si vous rencontrez un autre problème à l’avenir, consulteriez-vous de
nouveau un document de la base de données de Symantec traduit par un
outil automatique ?
\ V Non
o Oui o Non
0 Oui
Non
o Oui O Non
o
Oui
O Non
Figure 5-3: French submission form for online survey
F igu re 5-3 sh o w s that th e d esign o f the questionnaire w a s kept as sim p le as p o ssib le.
T h is fo llo w s the recom m en d ation m ade b y D illm a n et al. (1998: 4 ), w h o fou n d in a
study that resp o n se rates cou ld b e n eg a tiv ely a ffected b y 'fancy design'. F igure 5-3
a lso sh o w s that cu stom ers are en cou raged to p rovid e feed b ack so that the overall
serv ice can b e im p roved in the future. B y ask in g th em to rate th e u sefu ln ess lev el o f
M T output, users are im p lic itly told that the translation coverage o f tech n ical
support d ocu m en ts m ig h t b e exten ded in th e future. A ll d ocu m en ts w o u ld be
av a ilab le in E n g lish as w e ll as in a range o f target lan gu ages thanks to M T. U p on
su b m issio n o f th is form , u sers w ill b e o f cou rse thanked for the tim e th ey h ave
d ev o ted to fillin g ou t th e q uestionnaire. M ak ing sure that users k n o w w hat is g o in g
158
to b e d one w ith the inform ation, and thanking them , are tw o b asic rules o f custom er
feed b ack generation (R eid , 2 0 0 0 : 160).
O nce the overall d esig n o f th e instrum ent w a s co m p leted , it had to be im p lem en ted
so as to replace the e x istin g feed b ack form in certain d ocu m en ts taken from the
corpus. T he n ex t sectio n d escrib es the steps that w ere taken to put a prototype
questionnaire in p lace, b efore p u b lish in g it and m ak in g it availab le to users.
5 .2 .4 .
T e c h n ic a l im p le m e n ta tio n o f th e e x p e r im e n t a n d
in s tr u m e n t
A s d escrib ed in Chapter 1, S y m a n tec’s tech n ica l support d ocu m en ts are generated
from
variou s
k n o w le d g e
b ases.
D o cu m en ts
are
stored
in
la n g u a g e-sp ecific
k n o w le d g e b ases u sin g u n iq u e id en tification num bers and p re-defm ed tem plates.
T h ese d ocu m en ts m a y also b e referen ced from gen eric tech n ical support W eb p ages
resid in g on a separate W eb server. In order to im p lem en t the exp erim en t’s
sp ecifica tio n s ou tlin ed in th e p reviou s section , cu sto m isin g ex istin g W eb p ages w as
n ot the o n ly task required. O ne o f the ob jectives w a s to ensure that users w o u ld be
p rovided w ith either th e con trolled version or th e origin al v ersion o f a m achin etranslated docum ent. T h is m ean t that a ran d om isation m ech a n ism had to be put in
p la ce so that on e o f the d ocu m en ts w o u ld b e transparently generated to respond to a
user request. T o im p lem en t th is requirem ent, it w a s d ecid ed that both v ersion s o f
the d ocu m en t sh ould resid e in the sam e k n o w le d g e b ase location. A p ie c e o f
JavaScript co d e w o u ld th en h id e or d isp lay on e o f the tw o versio n s d ep en d in g o n a
random num ber gen erated at request tim e. S in ce b oth d ocu m en ts had to resid e in
the sam e location , a cu sto m tem p late had to be created. A prototype con tain in g a
con trolled sectio n and an original section w a s crafted and m ad e available for testin g
o n a stagin g server. T he m ain purpose o f this prototype w as to ensure that the
questionnaire d isp la y ed properly in W eb b row sers, and that answ ers w e re sent to
the right server.
U sin g this prototype a lso p roved extrem ely u sefu l in ad dressing a num ber o f issu es
that had not been en v isa g ed during the d esig n p h ases. D u e to the nature o f the
k n o w le d g e base's tem p late, h avin g a con trolled sec tio n and an original sec tio n w as
n o t su fficien t sin c e b oth v er sio n s o f th e d ocu m en t happened to be sharing som e
c o m m o n com p on en ts su ch as th e title. T his issu e w a s fix e d b y m ak ing sure that
159
ev ery field o f the tem p late w o u ld con tain tw o entries for each o f the fo llo w in g
d o cu m en t’s sections:
■
T itle
■
Situation
■
S o lu tio n
■
T ech n ical inform ation
■
S urvey
T w o other issu es w e re id en tified during th e testin g p h ase o f the questionnaire
prototype. First, it w a s realised that it w o u ld be ea sy for u sers to read th e d ocu m en t
and le a v e the p age w ith o u t co m p letin g the su rvey. It w a s therefore d ecid ed to u se a
p op -u p m ech a n ism that w o u ld trigger i f users le ft th e p age w ith ou t com p letin g the
survey. T he G erm an p op -u p is sh o w n in F igure 5.4:
Q d
Ö Kundenumfrage - Microsoft Internet Explorer
^
Symantec.
Kundenumfrage
Sie können uns dabei helfen, die Qualität dieser Website zu verbessern,
indem Sie die folgenden Fragen beantworten:
Fanden Sie dieses Dokum ent hilfreich?
Wurde Ihre Frage durch dieses Dokument beantwortet?
Hatten Sie aufgrund der Übersetzungsqualitat
Schwierigkeiten,, das Dokument zu verstehen?
Hätten Sie lieber die englische Version dieses
Dokuments gelesen?
Würden Sie im Falle eines weiteren Problems nochmals
ein Dokument aus Sym antecs Unterstützungsdatenbank
zu Rate ziehen, das maschinell übersetzt wurde?
o Ja
0 jJaa
o
o Ja
G ja
o Nein
o Nein
o Nein
o Nein
Ja
o Nein
| Senden
2.^'i r; : ' .T*£nt<
Figure 5-4: German submission form displayed in a pop-up window
F igure 5-4 sh o w s that radio b o x es w ere align ed on the right hand -sid e o f the p age to
a v o id back and forth e y e m o v em en ts. T h is d ecisio n w a s m ad e based on the study
co n d u cted b y B o w k er and D illm a n (2 0 0 0 ), w h o found that left-a lig n ed answ ers
co u ld n eg a tiv ely a ffect resp on se rates. O f cou rse, d isp layin g this p op-up w in d o w
d o es n ot guarantee that the su rvey w ill b e com p leted b y all users. Certain W eb
b row sers au tom atically b lock su ch w in d o w s, and users h ave to d ecid e w hether they
w an t to h ave them d isplayed. B e sid e s, c lo sin g su ch a w in d o w is quicker than
c lic k in g fiv e tim es in th e radio b o x e s and on ce on the 'Submit' button. Y et, users are
g iv e n the p o ssib ility to see that the su rvey w o u ld not take m ore than thirty secon d s
o f their tim e to be com pleted .
T he seco n d issu e that w a s n oted w h en the prototype w a s u sed is that certain users
w o u ld o n ly b e presented w ith the original v ersio n o f th e m achine-translated
d ocu m en t. S h ou ld th ey take the tim e to com p lete the survey, th ey sh ould be
rew arded w ith the p o ssib ility to co n su lt the con trolled versio n o f the docum ent.
It w a s th erefore d ecid ed to cu sto m ise the ‘Thank y o u ’ m e ssa g e by adding a link to
th e con trolled d ocu m en t. A s aforem en tion ed , it w a s im portant that u sers w o u ld on ly
p ro v id e feed b ack o n ce, so it w a s d ecid ed to u se tem porary co o k ies to prevent them
for su bm itting the form m ore than on ce. T h e p o ssib ility o f h a v in g m u ltiple
su b m issio n s is so m etim es regarded as a d isad van tage in a W eb experim ent (R eips,
2 0 0 2 : 2 4 5 ). U sin g tem porary c o o k ie s w a s selec ted as a m easure to address this
p roblem . O nce the q uestionn aire prototype w a s fm e-tu n ed , it w a s em bedded w ith in
d o cu m en ts se lec ted from the corpus. T h ese d ocu m en ts are p rovided in A p p en d ix I.
T he n ex t sec tio n d escrib es th is se le c tio n p rocess in detail.
161
5 *3 » Selecting docum ents for the experim ental
variations
5 .3 .1 .
C r it e r ia f o r t h e s e le c tio n o f d o c u m e n ts
In th e seco n d part o f this study, an o n lin e exp erim en t m u st be con d u cted u sin g at
lea st tw o d ocu m en ts. T hree initial criteria w ere id en tified to select th ese tw o
d ocu m en ts from the corpus for the exp erim en t. First, th ey m ust b e a ccessed b y a
su fficien t num ber o f u sers so that feed b a ck can be obtained. S econ d , th ey should
id e a lly n ot already h a v e b een translated into French and G erm an, so that ex istin g
hum an translations do n ot h a v e to b e rem oved . T his solu tion , h ow ever, cou ld be
e n v isa g ed i f the num ber o f an sw ers receiv ed during a p ilo t study su g g ests that
statistical tests m ay b e jeop ard ised b y sm all cell siz e s. F in ally, th ese source
d o cu m en ts m u st v io la te a su fficien t num ber o f CL rules. I f o n ly o n e or tw o rules are
v io la ted , the 'power' o f the exp erim en t m a y n ot b e su fficien t en ou gh (E vans et al.,
2 0 0 4 : 14).
5 .3 .1 .1 . R e le v a n c e o f t h e d o c u m e n ts f o r F r e n c h a n d G e r m a n
u s e rs
In order to d eterm ine w h eth er so m e sou rce d ocu m en ts are lik e ly to be con su lted by
a large group o f users o n ce m achin e-tran slated , o n e can lo o k at th e num ber o f W eb
hits that certain sou rce d ocu m en ts h ave receiv ed from French and G erm an users.
T w o h y p o th eses can b e draw n from th ese statistics. First, the p eo p le w h o have
a cc essed a sou rce d ocu m en t in E n g lish after reading a d ocu m en t in their ow n
la n g u age m ig h t h a v e preferred to co n su lt a translated versio n o f this docum ent.
T h ey m igh t b en efit from a m achin e-tran slated d ocu m en t i f th ey fa iled to fu lly
com preh en d th e E n g lish docu m en t. S econ d , i f certain F rench and G erm an users are
lo o k in g at sp ecific E n g lish d ocu m en ts, o n e m ay assu m e that th ese d ocu m en ts are
n ot U S -ce n tric, and th erefore relevant for the French and G erm an lo ca lised version s
o f th e product. P rovid in g a translation for th ese d ocu m en ts m igh t generate som e
interest from French and G erm an users.
W eb h its statistics w e re obtained for each E n g lish sou rce d ocu m en t present in the
corpus. T h ese statistics w ere gathered b y querying a tracking database, u sin g the
id en tification num ber o f each corresp on din g d ocu m en t. A s th e corpus had b een
162
stored in a static form at, updates to origin al d ocu m en ts had n ot b een m onitored. Yet,
certain d ocu m en ts had b een updated sin ce the start o f this study. B efore any
d ocu m en t co u ld be m achine-translated, updates had to b e recorded to ensure that all
m achin e-tran slated con tent w o u ld still be relevant. Other d ocu m en ts had b ecom e
o b so le te or in a ctiv e b y the tim e the statistics w ere harvested, w h ich facilitated the
se le c tio n p rocess. T he d o cu m en ts’ id en tifica tio n num bers, h o w ev er, never changed
and p rovided a reliab le m eth od to fin d out m ore about the w a y cu stom ers or users
a cc essed a particular docum ent. W h en a cc essin g a particular E n g lish technical
d ocu m en t, French or G erm an users co u ld h ave b een in on e o f the fo llo w in g seven
scenarios:
■
T h ey w ere redirected to the d ocum ent after en cou n terin g a problem
w ith their lo c a lise d program (or agent). A n error m e ssa g e asked
th em to trou b lesh oot th e issu e b y con su ltin g a particular tech nical
support docu m en t. T h is is the scenario that p rovid es the m ajority o f
W eb hits.
■
T h ey n avigated to th e d ocu m en t b eca u se it w a s present in a list o f
‘hot to p ic s ’ on the French or G erm an support W eb site.
■
A fter reading a tech n ica l support d ocu m en t in their o w n language,
they d ecid ed to read th e E n glish versio n (m ayb e b ecau se the E n glish
v er sio n w as m ore up-to-d ate).
■
T h ey fou n d a link p oin tin g to the d ocu m en t after querying a
k n o w le d g e b ase u sin g th e tech n ical support W eb site's search en gine
in their o w n langu age.
■
T h ey w ere p oin ted to th e d ocu m en t after reading a d ocu m en t on the
S ym an tec S ecu rity R e sp o n se W eb site, w h ich p rovid es inform ation
about viru ses and vu ln erabilities.
■
T h ey w ere redirected to the d ocu m en t after u sin g an A utom ated
Support
A ssista n t,
w h ic h
is
a
to o l
u sed
for
autom atic
troub lesh ooting.
A ll o f the W eb h its statistics obtained for th ese particular sou rces w ere added to
ev alu ate th e p opu larity o f th e co rp u s’ con tent w ith French and G erm an custom ers.
T his strategy is often u sed at S ym an tec to determ ine w h eth er it is w orth translating
1 63
a d ocu m en t into a certain langu age. A n an alysis o f th ese statistics revealed that few
E n g lish d ocu m en ts triggered any interest from French and G erm an users. T his m ay
be exp lain ed b y the lim ited E n g lish sk ills o f som e o f th ese users, w h ich cou ld
p revent th em from attem pting to co n su lt a d ocu m en t in E n glish . T h ese statistics
w ere con sid ered as an in d icative referen ce on ly, the real factor b ein g the num ber o f
W eb hits th ese d ocu m en ts w ill re ceiv e o n ce th ey are m achin e-tran slated and
p u b lish ed in French and German.
S o m e o f the d ocu m en ts receiv in g a h ig h num ber o f hits, h o w ev er, w ere to o short
and o n ly con tain ed link s to other d ocu m en ts. T h ey w ere n ot con sidered for
selec tio n b eca u se th ey did not p resent any com p reh en sib ility ch a llen g e. Other
d ocu m en ts w ere already translated into French and G erm an, so th ey w ere not
in itia lly con sidered . S o m e d ocu m en ts w ere sim p ly too im portant to b e u sed in such
an exp erim en t, su ch as the d ocu m en t 'Configuring Norton AntiVirus to provide
maximum virus protection', w h ic h d eals w ith a sen sitiv e top ic. O ne should not
forget that m o st users are S ym an tec cu stom ers en titled to tech n ical support. The
d istin ction b etw een users and cu stom ers is m ade b y R ic e (19 9 7 : 6) w h o w rites that
'custom ers are p e o p le w h o u se ser v ic es and p ay for th em w h ile users are often
p eo p le w h o u se the product but do n ot p ay for it'. S in ce S y m a n tec’s tech nical
support W eb sites are n ot restricted to cu stom ers through a lo g in a cc ess, both
cu stom ers and users have a ch an ce to get W eb -b ased tech n ica l support. B y
increasin g the exp osu re o f con tent that m ay n ot be com p reh en sib le, on e fa c es the
risk o f infuriating or e v e n lo sin g cu stom ers. A bad ex p erien ce related to an
u n so lv ed p rob lem can th en b e associated w ith th e brand. S in ce the 'brand im age is
w hat d istin g u ish es a product from its com petitors' (R ice: 129), risks m u st be
m in im ised .
T he fo c u s w a s th en p la ced o n d ocu m en ts o f average size, so that the am ount o f user
dictionary preparation prior to the M T p rocess w o u ld b e lim ited to a m inim um .
B o th procedural d ocu m en ts
and
d escrip tive
d ocu m en ts
w ere
con sid ered
for
selectio n . H o w ev er, o n ly procedural d ocu m en ts sh ow ed any sig n s o f interest from
French and G erm an users. For instance, the d escrip tive source d ocu m en t, 'Features
included in Norton SystemWorks Prem ier Edition', did not return any W eb hits from
F rench or G erm an users.
164
In th e end, three d ocu m en ts w ere ca refu lly re v ie w ed based on their num ber o f
receiv ed W eb hits:
■
'Turning on or turning o ff em ail scanning in Norton AntiVirus'
receiv ed 81 W eb hits from French users and 94 from G erm an users
over a three-m onth p erio d 11
■
'Error:
"Norton AntiVirus 2005 has encountered an internal
program erro r." (3009,1003)' receiv ed o n ly 2 W eb hits from French
users but 107 from G erm an users.
■
'Disabling em ail scanning in Norton Internet Security' receiv ed 106
W eb h its from F rench users and 107 from G erm an users over a
three-m on th period
T he v io la tio n s o f C L rules th ese d ocu m en ts con tain ed w ere then exam in ed . A fter
this rev iew , th e third d ocu m en t w a s discarded b eca u se it o n ly con tain ed tw o rule
v io la tio n s. T he first d ocu m en t con tain ed 14 rule v io la tio n s, and the seco n d 9.
5 .3 .1.2 . C o n d u c tin g a p ilo t s tu d y
C on cern s surrounded the se le c tio n o f the seco n d d ocu m en t based on the lo w
num ber o f re ceiv ed W eb h its from French users. It w a s therefore d ecid ed to conduct
a p ilo t study for tw o w e e k s to d eterm in e w hether a su fficien t num ber o f respon ses
w o u ld b e returned to con d u ct statistical tests. I f the num ber o f answ ers re ceiv ed w as
bound to jeo p a rd ise a statistical an alysis, a popular d ocu m en t w o u ld b e u sed instead.
W h en an a ly sin g th e resp on se rates after a tw o -w e e k period, the lack o f feed back
w a s o b v io u s, regard less o f the d ocu m en t version presented to users. T he second
d o cu m en t receiv ed 85 W eb hits, but o n ly fiv e questionnaires w ere subm itted, as
sh o w n in T able 5-1:
D o cu m e n t
N um ber of
h its for the
G erm an
do cum en t
N um ber of
G erm an
re sp o n d e n ts
N um ber o f
French
re sp o n d e n ts
N um ber of
h its fo r the
French
docum ent
NAV05_131
337
12
316
8
Document 2
41
2
44
3
Table 5-1: Number of answers received (May 31st - June 13th 2006)
11 This docum ent is referred to as ’N AV05_131' in the remainder o f this dissertation.
16 5
T able 5-1 sh o w s lo w resp on se rates that co u ld jeo p a rd ise the statistical an alysis that
had b een p lanned. In order to circu m ven t th is feed b ack drought, tw o d ecisio n s w ere
taken. First, it w a s d ecid ed to m ake the first d ocu m en t m ore v isib le on the French
and G erm an tech n ica l support h om e p a g es, as sh ow n b y F igure 5-5:
31 L es s g e n t * d u s u p p o r t e n lig n e n e p e u v e n t
ré p o n d re à v o s q u e s tio n s c o n c e rn a n t la s u p p r e s s io n
d e v ir u s . Si v o u s d e v e z c o n ta c te r u n te c h n ic ie n o c u r la
s u p p r e s s io n d 'u n v ir u s , v e u ille z c o n ta c te r n o tre
support a n tiviru s et logiciels espions
A id e à la s u p p re s s io n ou à la d é te c tio n d s v iru s et
de lo g ic ie ls esp io n s e t a u tre s q u e s tio n s liees a u x
v iru s e t a u x lo g ic ie l; e sp io n s,
contacter ie support
antivirus et logiciels espions
Aidez-nous à mieux connaître vos
___________ besoins !
Noue aimerions vous donner la
possibilité de consulter la plupart
de nos documents en français.
C'est donc pour cela que nous
menons actuellement une étude
cherchant à évaluer les avantages offerts par un
système de traduction automatique, Amiez-vous
quelques minutes pour lire l'un des documents
suivants et nous donner votre a v « en répondant a
cinq petites questions " Merci d’avance '
contacter le service clientèle
A ide s u r les a b o n n e m e n ts , l'e n re g is tre m e n t, les
re to u rs Ht p r o d u is , e t a u tre s q u e s tio n s
n o n -te c h n iq u e s .
contacter le service clientèle
• Activer/desscliver l'snalyss du «w ffier
étect'or.iaue dans H-.V
♦ DésBciivsv l'a a â ïy se du io u rrie r éJectrsm aue
Figure 5-5: Increased visibility of the first document
T he sec o n d d e c isio n w as to replace th e seco n d d ocu m en t w ith a m ore popular
docum ent. B a sed on the lo w num ber o f re sp o n ses receiv ed for the seco n d d ocum ent
(fiv e in total), it w a s d ecid ed to rem o v e an already-translated popular docum ent,
and rep lace it w ith m achine-translated content.
5 .3 .1 .3 . S e le c tin g a p o p u la r d o c u m e n t
T o d ecid e on a p opular d ocu m en t for both French and G erm an users, W eb hits w ere
u sed again. B ased o n the W eb traffic m on itored b etw e en the 6 th and the 13 th o f June,
the fo llo w in g d ocu m en t w as chosen: “Error: "N orton A n tiV iru s w a s unable to scan
your Instant M essen ger..." (3 0 2 1 ,4 ) after in stallin g N orton A n tiV iru s”
12
*
. This
d ocu m en t receiv ed 1112 v isits from French users and 1131 from G erm an users
during this tim efram e. T his h igh num ber o f v isits m ay b e exp lain ed by the ea se o f
a cc ess to th is d ocu m en t and the to p ic it covers. It con tain s instructions to h elp users
12 This document is referred as NAV05 121 in the rem ainder o f this dissertation.
166
ensure that the inform ation they ex c h a n g e u sin g a popular instant m essa g in g
a p plication (M S N M essen g er) is con sid ered safe by the S ym an tec product. D u e to
increasin g security paranoia, on e can ea sily understand w h y users w o u ld lik e to
s o lv e this p rob lem i f th ey see the error m e ssa g e con tain ed in the title o f this
d ocum ent. P rovid ed that th ey are on lin e - w h ich should be the ca se sin ce th ey are
u sin g instant m e ssa g in g te c h n o lo g y - c lic k in g a link em bedd ed in the error m essa g e
w ill redirect th em au tom atically to th e corresp on din g tech n ical support docum ent.
S uch h igh W eb h its statistics m ak e it th e third m o st popular d ocu m en t for the
F rench and G erm an N orton A n tiV iru s k n o w le d g e b ases, w h ich con firm s the risks
and the v isib ility a sso cia ted w ith th is study.
T his sectio n p rovid ed so m e inform ation w ith regard to the selec tio n and the
p u b lish in g o f d ocu m en ts. T he n ex t sec tio n w ill d escrib e the steps that w ere u sed to
prepare the d o cu m en ts prior to the m achin e-tran slation p rocess.
5 .3 .2 .
P r e p a r in g s o u rc e te x ts f o r M T
H o w original and con trolled d ocu m en t v ersio n s sh ould b e m achine-translated
dep en ds on o n e o f th e o b jec tiv es o f the p resent study. A s m u ch as p o ssib le, this
ca se study is b ased on a rea l-life scenario, w h ereb y users are asked to rate a
d ocu m en t that has b een alm ost entirely translated by an M T sy stem u sin g a
prod uction w o r k flo w . S ym an tec has b een u sin g M T for so m e tim e, so its current
M T p rocess is b riefly d isc u sse d to d eterm ine w h eth er som e o f its com p on en ts can
be u sefu l for th e p resen t study.
5 . 3 . 2 . 1 . L e v e r a g i n g S y m a n t e c ’s M T r e s o u r c e s
S y m a n tec’s M T p ro ce ss u ses a com b in ation o f T M and M T te c h n o lo g ie s w h ereb y a
k n o w le d g e b ase d o cu m en t is extracted from its k n o w le d g e base in an X M L format,
and an alysed again st a T rados T M (R oturier et al., 2 0 0 5 ). A ll source segm en ts that
do not return h igh fu z z y m atch es a b o v e a p red efin ed threshold valu e are then
au tom atically sent for translation to a Systran 5 .0 M T en gine. O nce th e segm en ts
are m achin e-tran slated , th ey are au tom atically im ported back into th e T M so that
translators can translate the n e w d ocu m en t u sin g a T M that contains a com bin ation
o f p reviou s m atch es and m achine-translated segm en ts. T his p rocess is sh ow n in
F igure 5-6:
167
Figure 5-6: Translation workflow using MT
T his p rocess co u ld b e adapted to set th e T M leverage threshold valu e to 99% so that
all sou rce seg m en ts that do n ot h ave an exact 100% m atch are sent to the M T
en gin e. T he sou rce d ocu m en t w o u ld then be pre-translated w ith the T M , so as to be
au tom atically im ported into th e k n o w le d g e b ase and p ub lished. S uch an approach
w o u ld p resent the advantage o f b ein g c lo s e to a rea l-life 2 4 /7 autom atic translation
service. Y et, a fe w issu e s arise. F irstly, th e d o cu m en ts’ original content w o u ld have
to b e ed ited in the k n o w le d g e b ase and au tom atically exp orted in an X M L format.
A s d isc u sse d earlier, a cu stom k n o w le d g e b ase tem p late w a s sp e c ific a lly created for
this study. T h is m ean s that the X M L exp ort fu n ction w o u ld h ave to be m o d ified to
h andle th is n e w tem p late, w h ich con tain s both the controlled and original version s
in th e sam e d ocu m en t. S uch d ev elo p m en t w ork seem ed b eyon d the sco p e o f this
study.
B e sid e s, u sin g a large T M w o u ld affect th e internal valid ity o f the results, sin ce an
u n k n ow n part o f the d o cu m en ts’ translation w o u ld originate from le g a c y hum an
translations. It w o u ld therefore b e im p o ssib le to con clu d e w ith co n fid en ce that the
application o f C L ru les im p roved the u sefu ln ess o f M T output sin ce the u se o f a T M
w o u ld invalid ate the results. O ne ex c ep tio n to th is d ecisio n to u se M T on its ow n
con cern s a C L rule that d eals sp e c ific a lly w ith segm en tation issu es and reuse. A s
d iscu ssed in th e p reviou s chapter, rule 4 7 states that elem en ts su ch as titles and
error m e ssa g e s sh ou ld be separated from running tex t and b e u sed as independent
16 8
segm en ts. T his typ e o f authoring d o es n ot o n ly p o se a p rob lem for M T , but it also
a ffects T M reu se, as sh o w n w ith the fo llo w in g ex a m p le w h ere the title is in italics:
El F o llo w the step s in Removing your Norton program using SymNRT to
rem o v e the program
A rticle titles a lw a y s appear on their o w n at the start o f a docum ent, so w h en
d ocu m en ts are translated or p ost-ed ited , th ey are stored as ind ep en d en t TM
seg m en ts. 100% m atch es can th en b e retrieved b y in d ep en d en t article titles, but not
by em b ed d ed o n es. In th is study, the T M com p on en t o f Systran 5 .0 w a s used
e x c lu siv e ly to store th e translation o f an article n am e. Apart from th is excep tion ,
d ecid in g to rely p u rely on refin ed M T output m eant that S y m a n tec’s M T user
d iction aries had to b e su pp lem ented w ith any n e w term occurring in the source
d ocu m en ts. T he fo llo w in g sectio n d eals sp e c ific a lly w ith the term in o lo g ica l w ork
perform ed prior to th e fin al M T p rocess.
5 .3 .2 .2 .
T e r m in o lo g ic a l w o r k p e r f o r m e d o n t h e s e le c te d
d o c u m e n ts
T w o factors gu id ed th e approach taken w ith th e term in o lo g y w ork p erform ed on the
E n g lish docu m en ts: th e accuracy requirem ents ou tlin ed in the p reviou s section , and
the s iz e o f the d ocu m en ts. S in ce it w a s d ecid ed that all im portant term s sh ould be
includ ed in th e te rm in o lo g y extraction p ro cess, a m anual approach w a s taken. The
approach u se d w a s based on th e m e th o d o lo g y described by A lle n (2 0 0 1 : 27),
w h ereb y sou rce tex ts are m achin e-tran slated in teractively to id en tify u n k n ow n and
m istranslated term s.
5.3.2.2.1. Interactive coding o f new and unknown terms
T he con ten t o f ea c h sou rce d ocu m en t w a s co p ied and pasted into Systran’s
interactive translation en viron m en t, Systran T ranslation P roject M anager. A project
w a s created for each sou rce d ocu m en t and in teractively m achine-translated u sin g
S y m a n tec’s ex istin g user dictionary a lo n g sid e Systran's C om puter/D ata p rocessin g
tech n ica l diction ary. A fter translating b oth sou rce d ocu m en ts (8 9 9 w ord s) a first
tim e, th e num ber o f term s origin ating from S y m a n tec’s ex istin g user dictionary w as
54 for French and 7 2 for G erm an. T his m ean t that m ore n ew term s had to be
en co d ed
for the E n glish -F ren ch lan gu age pair than for the E nglish -G erm an
169
langu age pair. A n y term that returned an incorrect translation due to a different
co n text or a m issin g translation w a s then m an u ally sent to a trilingual projectsp ecific u ser diction ary, translated, and en cod ed , b y u sin g Systran cod in g clu es in
certain situations. 31 n e w term s w ere en cod ed for French and 19 for G erm an. 10
sp ecific term s w ere also sent to a referen ce dictionary to handle U ser Interface (U I)
o p tion s by u sin g the looku p operators d escrib ed in sectio n 4.3.1 o f Chapter 4.
F igure 5 -7
sh o w s so m e entries con tain ing U I op tion s in the referen ce user
dictionary:
__
'’Y'OptionenY’" ipn)
“Options'* £>n)
’VOptionsY’“ ipn)
"Scan incoming Email'5(pn)
"V’Analyser les emails entrants''-/"' (pn)
:,Y'Bngehende E-Mails prufenY"1Ipn)
’’Scan outgoing Email” (pn)
''V!Analyser les emails soitanLsY'’' •Ipn}
"Y'.-Ausgehende E-Mails prüîenY"’ ipn)
"Permanently" {pn)
"V'En PermanenceY1" 5pn)
‘VDauerhaftY,,: {pn}
Figure 5-7: Example of reference user dictionary entries
A s d isc u sse d in sec tio n 1.3.4 o f Chapter 1, F igu re 5 -7 sh o w s that com m on w ords
such as 'Perm anently' or 'Options' had to b e carefu lly handled and en cod ed w h en
th ey occurred as U I o p tio n s in the source d ocu m en ts. T h ese entries w ere m arked
w ith d oub le q uotation m arks for ea se o f id en tifica tio n in the M T output. O verall,
the dictionary co d in g p ro cess to o k an hour for both lan gu age pairs, the m ost tim eco n su m in g task b ein g th e lo o k in g up o f ex istin g translations in referen ce TM s.
B ased on p reviou s stu d ies (su ch as B ab ych et al., 2 0 0 4 ), the im pact o f this
dictionary update w ork is ex p ected to im p rove co n sisten tly the p erform ance o f the
M T system . D eterm in in g the actual im p act o f th is dictionary update on the
recep tion o f M T d o cu m en ts fa lls ou tsid e the scop e o f th is study, but it is essen tial to
h igh ligh t that b oth v er sio n s o f th e d ocu m en ts had their term in ology en cod ed in the
user dictionary. C ertain ex c ep tio n s to this p rocess are d iscu ssed in sectio n 5 .3 .2 .2 .2 .
5.3.2.2.2. Exceptions to the coding process
S in ce so m e o f the C L ru les selected for this part o f the study operate at the le x ic a l
le v e l, not all n e w w ord s and term variants w ere co d ed in Systran’s user dictionary.
T his d ecisio n w a s m ad e to ensure that the effe c t o f th ese CL rules w o u ld be v isib le
in the con trolled M T output. The tw o CL rules con cern ed w ere 'Check the spelling'
and 'Omit unnecessary words'. I f sp ellin g m istak es w ere found in th e original
d ocu m en ts, th ey w o u ld o n ly be rectified in the con trolled docum ents.
170
The su b jectivity a sso cia ted w ith the rule 'Omit unnecessary words' should n ot be
underestim ated, b eca u se w h at see m s redundant or u n n ecessary to on e p erson m ay
sound sty listica lly leg itim a te to another. For instance, th e phrase 'if for so m e reason'
w as u sed in on e o f the sou rce d ocu m en ts. A fter exam in in g the M T output, it
transpired that this typ e o f phrase did n ot return an id io m a tic translation, so it could
have b een p o ssib le to add it to the user dictionary as a protected w ord seq u en ce.
S in ce the m ean in g o f this phrase is already en cap su lated in the com m on con ju n ction
'if, it w a s d ecid ed n ot to p ollu te the user dictionary u n n ecessarily. T he phrase 'for
so m e reason' w a s then rem oved from th e con trolled sou rce docum ent, but left in the
original version .
F in ally, the gu id elin e con cern in g the u se o f co n sisten t vocab u lary and term in ology
w as fo llo w e d in this study. A s m en tion ed in the p reviou s chapter, a p roject-sp ecific
dictionary o f gen eric w ord s w a s n ot co m p iled for th is study, but any gen eric lex ica l
in co n sisten cy in the sou rce d ocu m en ts w a s addressed in the controlled v ersio n s o f
the d ocu m en ts. For in stan ce, b oth sy n o n y m o u s phrases "keep so m eb od y from d oin g
som ething"
and
"prevent so m e b o d y from d o in g
som ething"
appeared in the
d ocu m en ts. A fter translating the sou rce d ocu m en ts in teractively, it em erged that the
seco n d phrase w a s h and led better by the M T sy stem than the first on e, so it w as
retained for the rew ritin g o f th e con trolled d ocu m en ts.
T his sec tio n has p resen ted the m e th o d o lo g y that w a s u sed to en cod e n e w and
m istranslated te rm in o lo g y into a sp e c ific M T u ser diction ary, and d iscu ssed so m e o f
the c h o ic e s m ad e during th e rew riting p rocess. T he n ex t section w ill expand on this
rew riting p rocess, by d etailin g the strategy u sed to turn original source d ocu m en ts
into con trolled d ocu m en ts.
5 .3 .3 .
E x p e r im e n ta l v a r ia tio n s
In order to id en tify rule v io la tio n s in the tw o d ocu m en ts, tw o approaches w ere
con sidered . First, th e su b set o f C L rules id en tified in the p reviou s chapter cou ld be
fo rm alised
and
integrated
in
acroch eck ™ .
T he rela tiv ely
sm all
size
o f the
d ocu m en ts, h o w ev er, did n ot ju stify su ch d ev elo p m en t. B e sid e s, so m e o f the rules
co u ld n ot be form alised , so a m anual ch eck in g and ap p lication p rocess w a s required.
C h eck in g
rule v io la tio n s
and
im p lem en tin g rew ritin gs w a s p erform ed w h en
171
ex a m in in g the sam p le corpus during the interactive M T p rocess u sin g Systran
T ranslation P roject M anager. E ach instance o f rule v io la tio n w as counted and added
to get a total num ber (# ) o f v io la tio n s per d ocu m en t, as sh o w n in T able 5-2.
Document
Number of CL rule
violations
Number of words
(original version)
Number of words
(controlled version)
NAV05 131
325
14
312
NAV05 121
574
18
579
Table 5-2: Number of rule violations and words per document
T able 5 -2 sh o w s little increase in w ord num bers w h en original d ocu m en ts are
turned into con trolled d ocu m en ts. T his table also h igh ligh ts the fact that o n ly a
lim ited num ber o f rule v io la tio n s occurred in the selec ted d ocu m en ts (the fu ll list o f
rule v io la tio n s p resent in th e origin al versio n s o f th ese tw o d ocu m en ts is provided
in A p p en d ix H ). T he C L ru les v io la ted in th ese tw o d ocu m en ts are presented in
T able 5-3:
Violations
NAV05 131
Violations
NAV05 121
Rule 5: Do not use more than 25 words per sentence.
2
4
Rule 17: Do not use a pronoun when the pronoun does not
refer to the noun it immediately follows.
-
1
Rule 24: Repeat the preposition in conjoined prepositional
phrases.
-
2
Rule 26: Avoid the ellipsis of verb.
-
1
Rule 28: Omit unnecessary words.
1
3
Rule 36: The subject of a non-flnlte clause must be the
same as that of the main clause.
-
1
Rule 41: Avoid embedded parenthetical expressions
introduced bv commas or dashes.
1
-
CL rule name
Rule 42: Do not include parenthesized expressions in a
segment unless the segment is still valid syntactically when
you remove the parentheses while leaving the parenthesized
expressions.
1
Rule 47: Move document names, error m essages, or section
titles to independent segments.
3
Rule 49: Do not use "this", "that", "these", or "those" on their own.
-
1
Rule 51: Avoid -inq words.
2
-
Rule 52: Use single-word verbs.
8
1
Table 5-3: Types of CL rules violated in the selected documents
T he num bers p resen ted in T able 5-3 su g g est that the original d ocum ents w ere not
co m p letely u n con trolled , sin c e v io la tio n s o f v ery e ffe c tiv e rules such as rules 19, 22,
3 5 , or 4 4 w ere u nfortu n ately n ot found in the origin al d ocu m en ts. O nly 9 rules from
the list o f 28 rules p resented in T able 4 -9 at th e end o f Chapter 4 w ere v io la ted in
172
the tw o d ocu m en ts selected for the o n lin e experim ent. T able 5-3 ind icates, h ow ever,
that so m e o f the rules v io la ted in the tw o d ocu m en ts do not originate from the final
list o f rules (n am ely ru les 2 4 , 4 2 , and 4 9 ). D u e to th e relative sm all num ber o f rules
v io la ted in the original d ocu m en ts, it w a s d ecid ed to im p lem en t three rules w ith
lim ited im pact in order to co m p en sate for the ab sen ce o f v io la tio n s o f m ore
e ffe c tiv e rules. A nother strategy m ay h ave co n sisted o f th e intentional introduction
o f rule v io la tio n s in the original v ersio n s. B ut it w o u ld h ave been d ifficu lt, i f not
im p o ssib le, to ensure that all CL ru les w ere broken the sam e num ber o f tim es.
B e sid e s, such an approach w o u ld h ave created an artificial situation that w ou ld
n ev er b e found in a rea l-life scenario. S o m e rules fa iled to b e v io la ted b ecau se their
rew ritin g overlap p ed w ith th e reform ulation o f sim ilar rules. For instance, rule 50,
’Keep both p a rts o f a verb together' w a s v io la ted on ce in th e selected docum ents.
T h e sen ten ce in w h ich th e v io la tio n occurred was:
Hi T his d ocu m en t g iv e s d irection s to turn em ail scan n in g on or off.
B a sed on this rule, the rew ritin g sh ould h ave been:
0
T his d ocu m en t g iv e s d irection s to turn on or turn o f f em ail scanning.
H o w ev e r, the original sen ten ce also v io la ted rule 5 2 , ' Use single-w ord verbs'. In this
particular exam p le, it se e m e d preferable to rew rite th e sen ten ce b ased on the secon d
rule, sin ce it w a s m ore e ffe c tiv e in the first part o f th e study:
0
T his d ocu m en t g iv e s d irection s to enab le or d isab le em ail scanning
O ther rules cou ld n ot b e ap p lied system atically. For instance, the ab ove exam p le
con tain s an - i n g w ord that is part o f a tech n ica l term 'em ail scanning' so it cannot be
reform ulated. A s m en tion ed in the p reviou s chapter, it w a s d ecid ed to reduce the
s c o p e o f rule 5 1 , 'Avoid -ing words'. In th is exp erim en t, th e application o f this rule
w as lim ited to tw o sp ecific cases: th e u se o f co n jo in ed - i n g w ord s and the u se o f a
gerund as subject, as sh o w n in T able 5-4.
173
R u le violated
Rule 51: Avoid -in g words
O rig in a l N A V 0 5 _ 1 3 1
Controlled NAV05 131
Turning on or turning off email
How to enable or disable
scanning in Norton Antivirus
email scanning in Norton
Antivirus
Rule 51: Avoid -in g words
Checking email for problems is
Norton Antivirus can scan
one Norton Antivirus task.
your email to check for
problems.
Table 5-4: Violations of rule 51 in the selected documents
S o m e o f the rules w ere n ot v io la ted in the selected d ocu m en ts b ecau se they are too
sp ecific. For instance, th e rule stating that A question mark should only be used at
the end o f a direct question' p roved very e ffe c tiv e in th e first part o f the study due
to the segm en tation issu e it caused. H o w ev er, the selected d ocu m en ts did not
con tain a v io la tio n o f this rule, so it w ill n ot be p o ssib le to take this rule into
accou n t w h en c o n clu sio n s are m ade o n the im p act o f C L rules at the d ocu m en t lev el.
O n ce tw o v ersio n s w ere availab le for each d ocu m en t, th ey w ere m an u ally uploaded
and form atted in the k n o w le d g e b ase b efore b ein g p ub lished. W hether th e sm all
num ber o f m o d ifica tio n s introduced in the con trollin g o f original d ocu m en ts is
su fficien t to sh o w sig n ifica n t d ifferen ces in users' reaction s m u st b e taken into
a cco u n t during the form u lation and testin g o f h y p o th eses. T his w ill b e d iscu ssed in
th e n ex t chapter.
5.4. Sum m ary
T h is chapter co v ered the tw o m ain steps u sed to set up an on lin e environm ent for
th e seco n d part o f the study: th e d esig n o f an on lin e questionnaire that is em bedded
w ith in exp erim en tal d ocu m en ts, and th e preparation o f th ese d ocu m en ts for an M T
p ro cess. T he d esig n o f th e q uestionn aire u sed in th is o n lin e environm ent w a s driven
by the requirem ent to obtain as m an y resp on ses as p o ssib le in a short tim efram e.
T h ese re sp o n ses w ere au tom atically c o llec ted and stored in a database, so as to be
exp orted in a C S V form at for an alysis. A fu ll an alysis and d iscu ssio n o f the results
w ill be p resented in the n ex t chapter. The o n lin e exp erim en t u sed tw o d ocum ents
co n ta in in g a con trolled v er sio n and an original version o f the sam e content, but
u sers w e re ran d om ly p resen ted w ith o n ly on e o f the tw o version s. T he French and
G erm an v ersio n s o f th ese d ocu m en ts are in clu d ed in A p p en d ices J and K
174
resp ectiv ely . O nce an alysed , the u se r s’ answ ers w ill h elp us determ ine w hether the
ap plication o f CL rules has any im p act on the u sefu ln e ss, com p reh en sib ility, and
a ccep tab ility o f M T d ocu m en ts ev e n w h en th ey are n ot p ost-ed ited .
175
C hapter 6
176
Chapter 6: Analysing the impact of CL
rules on the reception of MT documents
6.1. Objectives o f the p resen t chapter
In the p reviou s chapter, the approach u sed to im p lem en t an o n lin e exp erim en t w as
d iscu ssed . T he m ain con cern o f th is chapter is the an alysis o f th e resu lts o f the
o n lin e exp erim en t, b y exam in in g the resp o n ses subm itted b y users for tw o distinct
d ocu m en ts. In order to form ulate h y p o th e ses about the o u tcom e o f th ese results, the
d ocu m en ts u sed in th e o n lin e study h ave b een rated b y a group o f evaluators to
obtain a 'quality' benchm ark. T his group o f evaluators is d escrib ed in detail in the
sectio n 6 .2 .1 . T he an alysis o f th e data w ill b e p erform ed u sin g b oth d escrip tive and
inferential statistics. A n alpha le v e l o f 0 .0 5 or le s s is adopted for all statistical tests.
T h ese statistical results w ill b e d isc u sse d to answ er the seco n d research question:
from a W eb user's p ersp ective, d o es th e ap plication o f CL rules h ave any significant
im p act o n the u sefu ln e ss, the com p reh en sib ility, and the accep tab ility o f d ocu m en ts
co n ta in in g refined M T output? W h en an sw erin g th is q uestion, results w ill be
com p ared b etw e en the tw o lo ca les. T he distribution o f resp on se rates ob tained in
th is exp erim en t w ill first b e presented . T h e rem ainder o f th e chapter w ill d iscu ss
and com pare the an sw ers obtained from users for the tw o docum ents.
177
6.2. Prelim inary evaluation o f MT docum ents
6 .2 .1 . E v a lu a tio n s e tu p
In order to form ulate h y p o th eses about the effe c t o f C L rules on the characteristics
o f M T d ocu m en ts, it w a s d ecid ed to u se an expert group o f in d ep en d en t evaluators.
T he d ocu m en ts w ere a sse sse d by a sk in g q u estion s sim ilar to th ose users w ere asked
to answ er. T h is group o f evaluators w a s co m p o sed o f four external p rofession al
translators in each lan gu age. T h ese translators w ere either freelan ce translators or
e m p lo y ee s o f translation a g en cies that S ym an tec's localisation departm ent u ses on a
regular b asis. T h ese evalu ators w ere ask ed to an sw er three q u estion s for each o f the
d ocu m en t v ersio n s th ey w ere g iv e n to read. In order to avoid any bias w ith regard to
a certain v er sio n o f the d ocu m en ts, th ey w ere not told that the purpose o f the
ex p erim en t w a s to a ssess the e ffe c tiv e n e ss o f CL rules on M T output. A lso , they
w ere n ot g iv e n th e sou rce d ocu m en ts, sin c e the ob jectiv e o f th is parallel study w as
to reproduce the co n d itio n s in w h ich S ym an tec users w o u ld operate. T w o batches o f
d ocu m en ts w ere th en ran d om ly created. E ach batch com prised on e controlled M T
d o cu m en t v er sio n and on e original M T d ocu m en t version so that both version s o f a
g iv en d ocu m en t w e re n ot present in th e sam e batch. T he first batch o f docum ents
w a s d isp atch ed w ith sp ecific instru ction s, p rovid ed in A p p en d ix N . T h e evaluators
w ere first ask ed to report on the tim e it to o k th em to read the d ocu m en t, and then to
an sw er the tw o fo llo w in g questions:
■
D id th e quality o f the translation ham per your com preh en sion ?
■
W o u ld this d ocu m en t b e u sefu l for a S ym an tec user?
O nce the first round o f evalu ation w a s co m p leted and their an sw ers subm itted, they
w ere sent th e sec o n d batch o f d ocu m en ts w ith th e sam e instructions tw o days later.
T h ey m ay h ave n o ticed that som e o f th e d ocu m en ts w ere sim ilar to th o se th ey had
read tw o d ays earlier, but th ey w ere n ot asked to com pare both v ersion s. E ach set o f
*
*13
answ ers w a s p rovid ed in d ivid u ally for each d ocu m en t version .
T h ese an sw ers are exam in ed in d etail so that h y p o th eses can b e form ulated and
tested b a sed on the p ub lic's resp on ses.
13 T he evaluators' answ ers are included in A ppendix M.
178
6 .2 .2 .
F o r m u la t in g h y p o th e s e s
T he tw o d ocu m en ts used in this exp erim en t vary in len gth and co m p lex ity . The
original v er sio n o f N A V 0 5 _ 1 3 1 con tain s 3 2 5 w ord s and 2 sim p le procedural tasks.
T he d ocu m en t N A V 0 5
121 con tain s 5 7 4 w ord s, describ in g 2 d istin ct solu tion s
co m p risin g 3 and 2 tasks resp ectively. B ec a u se o f th ese textual d ifferen ces, the
evaluators' an sw ers w ere exam in ed sep arately so as to form ulate h y p o th eses that are
sp ecific to a particular docum ent. T h is approach is u sed to avoid form ulating and
testin g h y p o th eses assu m in g that the exp erim en tal m aterials and con d ition s are
sim ilar. T his assu m p tion had b een m ad e b y Shubert et al. (1 9 9 5 ) w h en testin g the
e ffec t o f SE o n the com p reh en sib ility o f tw o d ocu m en ts. A fter fin d in g d ifferen ces
in their resu lts, th ey had to re-exam in e the co m p lex ity o f the sou rce m aterials,
b efore co n clu d in g that the e ffec ts w ere lim ited to on e procedure (ibid: 173).
6 .2 .2 .l.
E v a l u a t i o n o f D o c u m e n t ’N A V o 5 _ i 3 i ’
In the rem ainder o f th is chapter, the term s 'Original' and 'Controlled' refer to the tw o
v ersion s o f the m achine-translated d ocu m en ts u sed in th e experim ent. T h ese term s
are u sed to q u a lify the translated d ocu m en ts, e v e n though CL rules w ere ap plied on
the sou rce d ocu m en ts. T his d istin ction is sim ilar to the d istin ction b etw een M T
output A and M T output B u sed in th e first part o f the study. B a sed on this
d istin ction , th e fo llo w in g answ ers w ere ob tained from evaluators w h en th ey w ere
asked the first q u estion after reading d ocu m en t 'N A V 05_131':
Did the q u a lity of th e tra n sla tio n h a m p e r y o u r c o m p re h e n sio n ?
1/Î
2
CD
I
3 ,
^
4
>
&
o
100%
i*—
S No
75%
.
_G
E
1.
I
25%1
controlled
■ Yes
controlled
original
original
German
French
Language and docum ent version
Figure 6-1: Evaluators' first answers for document 'NAV05_131'
1 79
T he
results
in
F igu re
6-1
su g g est
that
the
attitude
o f evaluators
on
the
co m p reh en sib ility o f th e M T output con tain ed in th is d ocu m en t is c lo s e ly related to
the u se o f C L rules for th e con trolled v er sio n o f the docum ent. T he introduction o f
CL rules see m s to h a v e p layed its part in rem o v in g am b igu ity and c o m p lex ity from
sou rce d ocu m en ts, w h ich in turn prod uced m ore com p reh en sib le M T output. The
four French evalu ators th ou ght that the con trolled v er sio n w a s ea sy to understand
d esp ite b ein g m achine-translated. H o w ev e r, o n ly on e o f th em th ou ght that th is w as
also the case w ith the original version . T he G erm an evaluators w ere unan im ou s in
fin d in g the origin al versio n d ifficu lt to understand. W hen confronted w ith the
con trolled v er sio n o f the d ocu m en t, three out o f four G erm an evaluators did not
h a v e any m ajor com p reh en sion prob lem . B a sed o n th ese results, the fo llo w in g
h y p o th esis can be form ulated:
H I: The French and German 'Controlled' versions o f document
N A V 05_131 are more comprehensible than the 'Original' versions.
French and German users presented with the 'Original' version o f
the document are more likely to experience comprehension
problems than users presented with the 'Controlled' version.
T his h y p o th esis w ill b e tested during the an a ly sis o f the data obtained from users.
T he fo llo w in g
an sw ers w ere ob tained for the seco n d q uestion, 'W ould this
d ocu m en t b e u sefu l for a S ym an tec user?':
Would th is docum ent be useful for a Sy m an te c user?
I No
o
7 5 %
_Q
E
1-
50%
50%
original
controlled
controlled
original
German
nch
Language and document version
Figure 6-2: Evaluators' second answers for document 'NAV05 131'
18 0
W hereas resu lts w ere rela tiv ely co n sisten t b etw een French and G erm an evaluators
for the first q u estion , their answ ers d iffer for th e secon d question. French evaluators
are u nan im ou s in th ink in g that the 'Controlled' versio n w o u ld b e u sefu l for users,
w h ereas G erm an evaluators are d ivid ed . T hree G erm an evaluators did not report
any com p reh en sion p rob lem s w h en a n sw erin g q u estion 1, but o n ly tw o o f th em
think that th e d ocu m en t w o u ld be u sefu l for users. H ow ever, the trends rem ain
sim ilar, sin ce a m ajority o f evaluators b e lie v e that 'Original' version w o u ld not be
usefu l for users.
H2: French and German users presented with the 'Original'
version o f document N A V 05_131 are less likely to find it useful
than users presented with the 'Controlled' version.
6 . 2 . 2 . 2 . E v a l u a t i o n o f D o c u m e n t ’N A V o 5 _ i 2 i ’
A sim ilar approach w a s u sed to form u late h yp oth eses w ith regard to certain
characteristics o f d ocu m en t N A V 0 5 _ 1 2 1 . T he answ ers to the first q u estion are
p resented in F igure 6-3:
Did the quality of the translation ham per your com prehension?
a
I
!
.
3
75%
-Q
E
,
50%
1
1
■
■
1
25%
controlled
originai
controlled
French
original
German
Language and document version
Figure 6-3: Evaluators' first answers for document NAV05_121
18 1
E3 No
■ Yes
The results ob tained for the seco n d d ocu m en t, N A V 0 5 _ 1 3 1 , d iffer from th ose
obtained for th e first docu m en t. R esu lts d iffer b etw e en the tw o lo ca les, sin ce m ost
G erm an
evaluators
en countered
com p reh en sion
p rob lem s
regardless
of
the
d ocu m en t v ersion . In th is situation, the e ffe c t o f CL rules m ay h ave b een m ask ed by
other lin g u istic p henom en a. On the other hand, French evaluators seem to think that
the 'Controlled' versio n is m ore co m p reh en sib le than the 'Original' one. T w o
different h y p o th eses therefore h ave to b e form ulated:
H3A: The French 'Controlled' version o f document N AV 05_121
seems to be more comprehensible than the 'Original' version.
French users presented with the 'Original' version o f the
document are more likely to experience comprehension problems
than users presented with the 'Controlled' version.
H 3B: The German 'Controlled' version o f document NAV05_121
does not seem to be more comprehensible than the 'Original'
version. German users presented with the 'Original' version o f the
document are as likely to experience comprehension problems as
users presented with the 'Controlled' version.
T he an sw ers ob tained for th e secon d q u estion are sh o w n in F igure 6-4:
Would this docum ent be useful for a Sy m an te c user?
E3 No
-Q
mm
E
5 0 %
25%
2 5 %
original
controlled
controlled
French
original
German
Language and document version
Figure 6-4: Evaluators' second answers for document NAV05_121
182
■ Yes
F igure
6-3
sh o w ed
that
tw o
French
evaluators
en countered
com preh en sion
p rob lem s w ith the 'Original' d ocu m en t, but the results p resented in Figure 6-4
ind icate that three o f th em b e lie v e the d ocu m en t w o u ld still be u sefu l for users. T his
attitude differs from the o n e ob served w ith G erm an evaluators for the p reviou s
d ocu m en t in F igu res 6-1 and 6 -2 . B a sed on th ese results, the fo llo w in g h yp oth eses
can be form ulated:
H4A:
F re n c h u se rs p re se n ted w ith th e 'O rig in al' v ersio n o f
d o c u m e n t N A V 0 5 _ 1 2 1 are as lik ely to fin d it usefu l as users
p re se n te d w ith th e 'C o n tro lle d ' versio n .
H4B:
G e rm a n u se rs p re se n te d w ith th e 'O rig in a l' v ersio n o f
d o c u m e n t N A V 0 5 _ 1 2 1 are less likely to fin d it useful th a n users
p re se n te d w ith th e 'C o n tro lle d ' versio n .
In order to test th ese h y p o th eses, the proportions o f p o sitiv e and n egative answ ers
obtained from groups o f u sers w ill be com pared b ased on the d ocu m en t version
th ey con su lted . T he se le c tio n o f a statistical test is described in the n ex t section .
6 .2 .3 .
S e le c tin g s t a t is t ic a l te s ts to te s t t h e h y p o th e s e s
S o m e o f th e h y p o th e ses (H 3 and H 4 ) are form ulated d ifferen tly d ep en d in g on the
lo ca le. T h e d istribu tion o f user resp on ses w ill therefore be p resented per target
la n gu age (or lo c a le ) and d ocu m en t versio n u sin g four independent groups o f
respondents. In order to test the form ulated h y p o th eses, resp on se d ifferen ces
b etw e en groups w ill b e ob served and com pared. A s m en tion ed b y L ev in e and
Stephan (20 0 5 : 137), 'the h y p o th esis test ex a m in es the d ifferen ce b etw een the
proportions o f tw o groups'. Z -tests w ill therefore be con d ucted to determ ine
w hether th e p roportion o f p o sitiv e or n eg a tiv e an sw ers obtained for a sp ecific
q u estion from a particular group (su ch as T ren ch O riginal') is sign ifican tly different
from the proportion o f p o s itiv e or n eg a tiv e answ ers obtained from another group
(su ch as 'French C ontrolled').
183
In order to test the h y p o th eses form ulated in the p reviou s section , a sig n ifica n ce
le v e l m ust b e ch osen . D e V au s (2 0 0 2 : 2 3 0 ) states that ‘the low er the sig n ifica n ce
le v e l, the m ore co n fid en t w e are that our ob served percen tage d ifferen ces reflect
real d ifferen ces in th e p op u lation ’. For d ich o to m o u s variables, h e su g g ests u sin g
tests u sin g ‘0.05 for sm all sam p les and 0.01 or lo w er for larger sa m p les’ (ibid).
B a sed on th e siz e o f th e p resent sam p les, a lev el o f sig n ific a n c e o f 0.05 w a s ch o sen
for the Z -tests.
S ectio n s 6 -3 , 6 -4 , 6 -5 , and 6-6 fo c u s o n the an alysis o f the resp on ses receiv ed from
users, starting w ith th e p u b lic’s resp on se rates.
6.3. A nalysis o f the distribution o f responses
6 .3 .1 . R e s p o n s e r a te s
In the p reviou s chapter, it w a s m en tion ed that resp on se rates obtained during the
p ilo t study w ere re la tiv ely lo w , w ith 4% o f resp on ses from G erm an users and 3%
from French users for d ocu m en t N A V 0 5 _ 1 3 1 . D uring the m on th fo llo w in g the p ilot
study, resp on se rates in creased slig h tly for this particular docum ent, as sh o w n in
T able 6 -1 , in w h ich resp o n se rates for d ocu m en t N A V 0 5 _ 1 2 1 are also presented:
D ocum ent: N A V 0 5 _ 1 3 1
Number of Web hits
Number of responses
% o f re sp o n d e n ts
D ocum ent: N A V05 121
Number of Web hits
Number of responses
% o f re sp o n d e n ts
Germ an
French
1870
1488
85
73
4 .7 0
4 .9 1
Germ an
French
1551
3180
39
54
2.51
1.70
Table 6-1: Response rates (June 6th - July 9th 2006)
T able 6-1 fo c u se s o n the resp o n se rates ob tained after the p ilo t study. T h ese rates
and num bers do n ot in clu d e the 12 G erm an an sw ers and the 8 French answ ers
re ceiv ed for N A V 0 5 _ 1 3 1 during the p ilo t study. T h ese 2 0 answ ers, h ow ever, w ill
b e u sed for the data an alysis.
184
T ab le 6-1 su g g ests that a large num ber o f S ym an tec users accessed the tw o
d o cu m en ts u sed in th e exp erim en t w ith ou t an sw erin g the questionnaire. B ased on
the ty p o lo g y o f n on -resp on d en ts estab lish ed b y B osn jak et al. (2001: 12), th ese
in d iv id u als m ay h a v e b een 'lurkers' w h o read the q u estion s but did n ot com p lete the
q uestionnaire. T h ey m a y a lso h ave b een in d ivid u als w h o started answ ering the
q uestionn aire but did n ot subm it it. D eterm in in g the proportion o f each category o f
n on-resp on d en ts is im p o ssib le. B e sid e s, B osn jak et al. (2001: 13) rem ind us that
W eb hits m ay n ot n ece ssa rily be triggered b y h um ans, but p o ssib ly by 'robots,
w o rm s, or w anderers'. I f this w ere th e case, the actual resp on se rates w o u ld be
high er than the o n es p resen ted in T able 6-1.
In se c tio n 5 .2 .4 o f C hapter 5, the tech n ical im p lem en tation o f the o n lin e experim ent
w a s d escrib ed by ex p la in in g that the ran d om isation p rocess w o u ld rely on storing
b oth v ersio n s o f the sam e d ocu m en t in the sam e k n o w le d g e base's location . D u e to
th is im p lem en tation , it is n ot p o ssib le to k n o w h o w m any hits w ere sp ecifica lly
re ceiv ed for the con trolled v ersio n s or the original v ersio n s o f a g iv en docum ent.
S in ce both d ocu m en t v er sio n s w ere p u b lish ed w ith in the sam e k n o w led g e base's
lo ca tio n , th ey did n o t r e ceiv e a unique d ocu m en t identifier a llo w in g for the tracking
o f sp ecific W eb hits.
T he tw o d ocu m en ts p u b lish ed for the duration o f th is study did not h ave th e sam e
ex p osu re. B a sed o n the num ber o f W eb hits, d ocu m en t N A V 0 5 1 2 1 p roved m ore
p opular w ith French users than w ith G erm an users. T his m igh t be due to the
tech n ical issu e d escrib ed in the docum ent, w h ich d eals sp ecifica lly w ith a thirdparty product (M S N M e ss e n g e r ).14
14 The users' results are included in Appendix L.
185
R esp o n se rates also vary greatly b etw een the tw o d ocu m en ts, but it is im p o ssib le to
d eterm ine w ith certainty the ca u ses o f th ese d ifferen ces. P o ssib le exp lanations m ay
in clu d e th e len gth , co m p lex ity and th e quality o f the docum ent. B esid e s, i f users did
n ot co n su lt d ocu m en ts in their entirety or stopped after the d escrip tion o f the first
so lu tion in d ocum ent N A V 0 5 _ 1 2 1 , th ey m ay n ot h ave reached the questionnaire
located at the b ottom o f th ese d ocu m en ts. T his assu m p tion seem s to be supported
by the u se o f the p op -u p m ech an ism , as sh ow n in F igure 6-5:
Questionnaire subm ission method
120 '
»
10D­
£
IO ­
"Cl
ID
S
E
S I} -
a static
40-
□ pop-up
r
20
-t:.rr
39%
44%
28%
French
German
French
19%
j
German
NAVQ5 131
NAVQE 121
Locale and docum ent name
Figure 6-5: Responses submission method
T he results sh o w that th e p op -u p m ech an ism p roved v ery u sefu l to en cou rage users
to su bm it feed b ack , e sp e c ia lly w ith d ocu m en t N A V 0 5 1 2 1 . T o som e exten t, this
u sa g e co n flicts w ith th e v ie w exp ressed by N ie lse n and Loranger (2006: 74), w h o
state that ‘em p irically, w e see m an y users c lo se p op -u p s as fast as p o ssib le — often
ev e n b efore the con ten t has b een rendered. ( . . . ) Sad to say, ev e n g ood p op -u p s are
rarely appropriate th ese days, h o w ev er, b ecau se ev il p op -u p s h ave tarnished their
repu tation .’ T he num ber o f re sp o n ses subm itted v ia the p op-up in this study d oes
n o t con firm this statem ent, sin c e a large proportion o f users u sed this m ech an ism to
su b m it their feed b ack for the lon ger d ocu m en t o f th e tw o , N A V 0 5 _ 1 2 1 . H o w ev er, it
sh ould b e m en tion ed that the p op -u p probably did not reach all users sin ce certain
W eb brow sers can b e co n fig u red to au tom atically b lo c k them .
186
6 .3 .2 .
D is tr ib u tio n o f re s p o n s e s b a s e d o n d o c u m e n t
v e r s io n
S in ce the o b jectiv e o f this study is to com pare users' resp on ses based on the
d o cu m en t versio n th ey w ere g iv en to con su lt and their lo ca le, four groups o f
respon d en ts are u sed for the an a ly sis o f each docum ent. T h ese four groups o f
respon d en ts are: 'French C ontrolled', 'French O riginal', 'G erm an C ontrolled', and
'Germ an O riginal'. T he distribution o f subm itted resp on ses b ased on lo ca le and
d ocu m en t v er sio n is sh o w n in F igu re 6.6:
Distribution
of
submitted surveys per locale and document version
120
(U
>
£
1 GO
m
BO*
^
60-■
■?
u
40'
■o
I
-Q
¿L
-
-
18
French
German
-
a original
□ controlled
German
French
NAVQ5 121
5
S3
40
2 7
-
MAVOS
Locale and document name
Figure 6-6: Distribution of submitted responses
T he a b ove figu re sh o w s that th e num ber o f resp on ses ob tained w a s alm ost the sam e
for each typ e o f d ocu m en t versio n . A total o f 133 resp on ses w a s subm itted w h en
th e M T d ocu m en t p resen ted to users corresp on ded to an origin al source docum ent,
w h ic h is slig h tly le s s than th e total o f 138 resp on ses receiv ed w h en th e M T
d o cu m en t origin ated from the con trolled versio n o f a d ocu m en t. T he distribution o f
resp o n ses per lo c a le is also h o m o g e n e o u s sin ce 136 resp on ses w ere obtained from
G erm an users, and 135 resp o n ses from French users. T h is distribution o f resp on ses
is ideal for statistical tests sin c e there is no sm all ce ll across the four groups o f
resp on d en ts for ea ch d ocu m en t.
187
6 .4 * A nalysis o f the resp on ses obtained for the
first docum ent
T he n ex t sec tio n fo c u se s on the an alysis o f the im pact o f sp ecific CL rules o n the
co m p reh en sib ility and u sefu ln ess o f th e first docum ent, N A V 0 5 _ 1 3 1 , w h ich
con tain s 14 rule v io la tio n s in its origin al version . T h ese rule v io la tio n s are d etailed
in A p p en d ix H.
6 .4 .1 .
E v a lu a tin g th e im p a c t o f C L r u le s o n th e
c o m p r e h e n s ib ility o f th e f ir s t d o c u m e n t
In th e exp erim en t, users w ere ask ed to p rovid e answ ers after con su ltin g a docum ent.
T his d o es not n ecessa rily m ean that th ey read the entire d ocu m en t b efore g iv in g
their an sw ers. For instance, certain users m a y h ave d ecid ed to fo cu s on a sp ecific
part o f the docum ent. S in ce d ocu m en t N A V 0 5 _ 1 3 1 con tain s tw o procedural tasks,
certain users m ay h ave read o n ly th e first section , the on e that d escrib es the task
required to en ab le em ail scan n in g. T akin g this factor into account, th e fo llo w in g
resp o n ses w ere ob tained from users:
Did the quality of the translation hamper your comprehension?
6001—
5 50ID
S:
if> 40E
Clj
—
O 30as
-Q
E
I yes
2010-
35
66%
m
I no
0controlled
controlled
original
original
German
French
Language and document version
Figure 6-7: Users' answers to the third question for document 'NAV05_131'
188
H y p o th esis H 1 is tested by com p arin g the proportions o f n egative answ ers obtained
from users to this particular q u estion b ased on the d ocu m en t version th ey consulted:
H I : T h e F ren ch an d G erm an 'C o n tro lle d ' v ersio n s o f d o cu m e n t
N A V 0 5 _ 1 3 1 are m o re c o m p re h e n sib le th an th e 'O rig in al' v ersio n s.
F re n c h a n d G erm an u se rs p re se n te d w ith the 'O rig in al' v ersio n o f
th e d o c u m e n t are m o re likely to e x p e rie n c e c o m p re h en sio n
p ro b le m s th a n u se rs p re se n te d w ith th e 'C o n tro lled ' v ersio n .
R esu lts ind icate that there is a sig n ific a n t d ifferen ce in the proportions o f n egative
an sw ers obtained from G erm an u sers w h o con su lted the O riginal v ersion o f the
docu m en t and th o se w h o con su lted th e C on trolled version . On the other hand, there
is no sig n ifica n t d ifferen ce in the proportions o f n eg a tiv e an sw ers obtained from
French u sers w h o con su lted the O riginal v er sio n o f th e d ocu m en t and th ose w h o
co n su lted th e C on trolled version . H y p o th e sis H I is therefore con firm ed by the
G erm an resu lts, but contradicted b y th e French results.
In order to p rovid e p oten tial exp lan ation s for th ese results, the M T output contained
in th e F rench 'Original' d ocu m en t m u st b e exam in ed to d eterm ine w hether so m e o f
the C L rules w ere m ore e ffe c tiv e in G erm an than in French. For instance, let us
con sid er th e third sen ten ce o f the 'Situation' section . In the 'Original' v ersion o f the
d ocu m en t, th is sen ten ce con tain ed a v io la tio n o f rule 52, stating that verbs w ith
particles sh ould be replaced w ith sin g le-w o rd verbs w h ere p o ssib le.
ŒD T his d ocu m en t g iv e s d irection s to turn em ail scan n in g o n or off.
A
C e d ocu m en t d onne d es d irection s à l'analyse du courrier électron iq ue de
tour en fo n ctio n ou hors fon ction .
0
T h is d ocu m en t g iv e s instru ction s to enab le or d isab le em ail scanning.
B
C e d ocu m en t d onn e d es instructions pour activer ou d ésactiver l'analyse du
courrier électron iq ue.
T he a b ove ex a m p les sh o w that in the original sen ten ce, the verb ‘turn’ and the
p articles 'on' and ' o f f w ere n ot an alysed properly b y th e M T en gin e, w h ich
p rod uced F rench M T output that w o u ld traditionally be regarded as incorrect and
u n in tellig ib le. In M T output A , ‘tu rn ’ is translated as a noun, and th e particles ‘o n ’
and ‘o f f are translated as phrases. In the 'Controlled' version , the am b iguity w as
rem o ved b y th e ap p lication o f th e rule, so correct an alysis and generation phases
w ere perform ed by th e M T sy stem , resultin g in a gram m atically correct M T output.
189
T he sam e prob lem occurred in the G erm an output:
[HI T h is d ocu m en t g iv es d irection s to tu m em ail scan n in g on or off.
A
D ie se s D ok u m en t gibt R ich tu n gen zu m U m drehung E -M ail-P rüfun g an oder
w eg.
E3 T his d ocu m en t g iv e s instructions to enab le or d isab le em ail scanning.
B
D ie s e s D ok u m en t erteilt A n w eisu n g e n , E -M a il-Prüfung zu aktivieren oder
z u deaktivieren.
D e sp ite fin d in g sim ilar ex a m p les o f an e ffe c tiv e rule application in both French and
G erm an output, results differ b etw e en th e tw o lo ca les w ith regard to the overall
im p act o f th e rules at th e d ocu m en t lev el.
A p o ten tial exp lan ation for d ifferen ces in the overall im pact o f CL rules on the
co m p reh en sib ility o f th is particular d ocu m en t lies in the textu al location o f
u n in tellig ib le sen ten ces. T he exam p le m en tion ed earlier is d irectly located after the
title o f the d ocu m en t, in a sectio n w h o se purpose is to p rovid e background
inform ation or an o v e r v ie w o f th e in stru ction s con tain ed in th e ‘ Solution' section.
T he ‘S itu a tio n ’ sectio n u su ally con tain s a fe w gen eric sen ten ces, su ch as:
C h e c k in g e m a il fo r p ro b le m s is o n e N o rto n A n tiV iru s task. E m ail
s c a n n in g c h e c k s each em ail. T h is d o cu m e n t gives d ire ctio n s to
tu rn e m a il scan n in g on o r off. T h e d ire ctio n s are fo r N o rto n
A n tiV iru s 2003 th ro u g h 2006.
O ne m a y argue that su ch a sec tio n is redundant sin ce m ost o f the inform ation is
already con tain ed in th e title o f a d ocu m en t such as ‘Turning on or turning o f f em ail
scan n in g in N o rto n A n tiV iru s’. For certain users, reading and acting on the
instru ction s con tain ed in the ‘S o lu tio n ’ sec tio n m ay be th e m ost critical part o f their
o n lin e tech n ical exp erien ce, so th ey m ay sk im over the ‘S itu ation ’ section . W hether
the sen te n c es o f th is sec tio n are totally in tellig ib le m ay n ot b e so relevant. O f course,
th is is o n ly a h y p o th e sis that w o u ld n eed to be con firm ed b y a m ore thorough
u sa b ility study.
19 0
This h yp oth esis, h ow ever, is supported b y the fact that fe w rules w ere broken in the
'Solution' section . B e sid e s, so m e o f the ru les v io la ted in this sectio n d id n ot seem to
h ave any im p act on the w e ll-fo rm e d n ess o f the output. A n ex a m p le o f rule 52,
w h ich w a s broken tw ic e in the 'Solution' sectio n , is p rovided b e lo w w ith the
corresp on din g translated sen ten ces sh o w in g no sig n o f im provem ent:
ISI T o turn on em ail scan n in g
A
Pour activer l'analyse du courrier électron iq u e
A
E -M ail-P rüfun g aktivieren
H
T o en ab le em a il scan n in g
B
Pour activer l'analyse du courrier électron iq u e
B
E -M ail-P rüfun g aktivieren
T he a b ove ex a m p le is different from th e on e p rovid ed earlier, b eca u se the particle
'on' w a s correctly an alysed b y th e M T system , probably due to th e fact that it
im m ed iately fo llo w e d the verb.
O verall, th ese results su g g est that F rench u sers did n ot regard the d ocu m en t as
h o listic a lly as their G erm an counterparts, probably fo c u sin g on the 'Solution'
section .
T h ese
results
m ay
a lso
su g g e st
that
French
users'
tolerance
for
u n in tellig ib le output m ay b e h igh er as lo n g as their overall on lin e exp erien ce results
in so lv in g their tech n ical prob lem . T h is typ e o f attitude already p rovid es som e
elem en ts o f in form ation w ith regard to h o w u sefu l this M T d ocu m en t w a s to users.
T h is a sp ect is d isc u sse d in sectio n 6 .4 .2 .
6 .4 .2 .
E v a lu a tin g t h e im p a c t o f C L r u le s o n th e
u s e fu ln e s s o f t h e f i r s t d o c u m e n t
T he first tw o q u estio n s that users w ere ask ed to an sw er con cern the u sefu ln ess o f
th e d ocu m en t th ey w e re con fronted w ith . S in ce the an sw ers to the seco n d q uestion
co u ld h ave b een in flu en ced b y a w id e range o f variab les, the an alysis o f the results
w ill fo c u s o n ly on th e first question. T he second question, 'Did the d ocum ent
answ er your question?,' w a s part o f S ym an tec's original cu stom er satisfaction
feed back m ech a n ism so it w a s retained in the questionnaire u sed in th is study. A s
d iscu ssed in sectio n 5 .2 .3 .1 , h o w ev er, th is q uestion w a s n ot p recise en ou gh to
d eterm ine the u sefu ln ess rate o f a d ocu m en t. It w as ex p ected that m ore clear-cut
1 91
results w o u ld be obtained w ith the q u estion 'W as th is d ocu m en t useful?', w h o se
distribution o f an sw ers is presented in figu re 6-8:
W as this d o c u m e n t useful?
■ yes
El no
Language and document version
Figure 6-8: Users’ answers to the first question for document 'NAV05_131'
T he results for q u estion 1 sh o w that in th e four groups o f respondents, at least 50%
o f respondents fou n d the d ocu m en t u sefu l. R egard less o f the v ersion users w ere
g iv en to con su lt, th ese initial results con firm that the refined M T output contained
in th is d ocu m en t w a s u sefu l for a m ajority o f users. H y p o th esis 2 w a s then
sta tistically tested:
H2: French and German users presented with the 'Original'
version o f document NAV05_131 are less likely to find it useful
than users presented with the 'Controlled' version.
R esu lts ind icated that there w as no sig n ifica n t d ifferen ce w h en com paring the
proportions o f p o sitiv e answ ers obtained from th e groups 'French Controlled' and
'French O riginal', and 'German C ontrolled' and 'German O riginal’. T h ese results,
w h ic h contradict the seco n d h yp oth esis, m a y b e exp lained b y the reason that has
b een ad van ced earlier. The m o st im portant sectio n o f the d ocu m en t is the 'Solution'
sectio n , sin ce it con tain s th e steps that m u st be perform ed by users to a cco m p lish a
task. V ery fe w CL rules w ere broken in th is section , so th e m achine-translated
con tent p resent in th is sectio n is alm ost id en tical in both version s o f the docum ent.
T he u sefu ln e ss o f th is section seem s to b e in flu en ced b y the intrinsic sim p licity o f
th e sen ten ces u sed in this section .
192
T he u sefu ln ess o f th is section , and p o ssib ly o f th e w h o le docu m en t, in its 'Original'
v ersion m ay also be in flu en ced b y another param eter. T his param eter is the correct
translation o f tech n ica l term s in b oth v er sio n s o f the docum ents. A s described in
Chapter 5, all tech n ica l term s, in clu d in g U ser Interface (U I) op tion s, w ere en cod ed
in the M T system 's u ser d iction aries. T h is p re-p rocessin g step w a s n ecessary to
ensure that th ese U I op tion s w o u ld perform their ic o n ic role, as d iscu ssed in section
1 .3.2.2.1 o f th e first chapter. T h e resu lts ob tained for th is particular d ocu m en t
su g g est that th is term in ological w ork w a s su fficien t for d ocu m en t N A V 0 5 _ 1 3 1 to
p rove u sefu l for m o st users, regard less o f the ap plication o f certain CL rules. T his
fin d in g m u st b e ex a m in ed in the lig h t o f th e results obtained for th e secon d
d ocu m en t, N A V 0 5 _ 1 2 1 .
6.5. A nalysis o f the resp on ses obtained for the
second docum ent
A s n oted in sec tio n 6 .2 .2 , th e seco n d d ocu m en t selec ted for th is exp erim en t is m ore
co m p lex than the first on e, sin ce it con tain s tw o so lu tio n s co m p o sed o f several tasks.
T h e ty p es and n um ber o f rules v io la ted in th e d ocu m en t N A V 0 5 _ 1 2 1 are also
different. T h ese rule v io la tio n s are in clu d ed in A p p en d ix H . Separate h y p o th eses
w ere therefore form u lated for th is d ocu m en t. T he im p act o f the CL rules on the
com p reh en sib ility o f the m achine-translated con ten t is first d iscu ssed .
19 3
6.5.1. Evaluating the impact o f CL rules on the
comprehensibility o f the second document
A s n oted in T able 6 -1 , fe w er resp o n ses w ere obtained for this particular d ocum ent
than for d ocu m en t N A V 0 5 _ 131, so c e lls are sm aller. T he resp on ses obtained for the
third q u estion are p resen ted in F igure 6-9:
Did the quality of the translation hamper your comprehension?
3025-
Qi
S
t/J 20C
<3
150
E3 ye s
-O
E
10-
19
70%
17
«3%
controlled
□ nc
controlled
original
French
original
German
Language and document version
Figure 6-9: Users’ answers to the third question for document 'NAV05_121'
T he results p resen ted in F igu re 6 -9 are u sed to test tw o separate h y p o th eses, the first
on e con cern in g resp o n ses ob tained from French users.
H3A:
T h e F re n c h 'C o n tro lle d ' v e rsio n o f d o cu m e n t N A V 05_121
seem s to be m o re c o m p re h e n sib le th a n th e 'O rig in al' v ersion.
F re n c h u se rs p re se n te d w ith th e 'O rig in a l' v e rsio n o f th e
d o c u m e n t are m o re likely to e x p erien c e co m p re h en sio n pro b lem s
th a n u se rs p re se n te d w ith the 'C o n tro lled ' v ersio n .
R esu lts ind icate that there is no sign ifican t d ifferen ce b etw een the proportions o f
n eg a tiv e an sw ers ob tain ed from French users w h o w ere presented w ith an 'Original'
v ersio n o f the d ocu m en t from th o se w h o con su lted a 'Controlled' v er sio n o f the
docu m en t. T he h ig h prop ortion o f p o sitiv e an sw ers obtained from French users w h o
co n su lted the 'Original' versio n o f the d ocu m en t N A V 0 5 1 2 1 m ay b e exp lain ed by
on e o f the a forem en tion ed argum ents. C ertain users m ay n ot h ave read the
docu m en t in its en tirety, and therefore n ot en cou n tered com p reh en sion problem s.
For instance, let us ex a m in e on e o f the sen ten ces that violated fiv e C L rules and its
translation in the 'O riginal' version:
194
El I f for som e reason it is not, con tin ue on to th e n ext sectio n for h o w to
m an u ally con figu re the latest version o f M S N M essen g er to w ork w ith
N orton A n tiv ir u s Instant M essen g er scanning.
A
Si pour q u elq u e raison elle n'est pas, co n tin u ez en fon ction à la section
suivante pour que la façon con figu re m an u ellem en t la dernière v er sio n de
M S N M e ssen g er pour fon ction n er a v ec l'analyse de m essagerie instantanée
de N orton A n tiv ir u s.
In the ab ove ex a m p le, the translated sen ten ce sh ould h ave created com preh en sion
p rob lem s for users (w ith the m istranslation o f the 'how to' clau se), but w ith ou t
d etailed feed b ack , it is d ifficu lt to ex p la in w h y th is did n ot occur. B e sid e s, the
q u estion users w ere ask ed to answ er cou ld o n ly be an sw ered b y 'yes' or 'no', so it
m ay h ave b een d ifficu lt for th em to report o n their com p reh en sion d ifficu lties. A
sca le o f v a lu es m ay h a v e p rovided m ore d etailed results.
T o so m e extent, th e sam e exp lan ation s m ust be ad vanced to exp lain the lack o f
im p act o f CL rules on the com p reh en sib ility o f the G erm an output. T his lack o f
sig n ifican t d ifferen ce w as n oted after testin g the seco n d h y p o th esis for this
particular docum ent:
H3B:
The
G erm an
'C o n tro lled '
v e rsio n o f d o c u m en t
'N A V 0 5 _ 1 2 1 ' d o es n o t seem to b e m ore c o m p reh en sib le than th e
'O rig in a l' v e rsio n . G erm an u sers p resen te d w ith th e 'O rig in al'
v e rsio n o f th e d o cu m e n t are as lik ely to experien ce
c o m p re h e n sio n p ro b le m s as u se rs p re sen ted w ith th e 'C o n tro lled '
v ersio n .
N o sign ifican t d iffer en ce w a s found b etw e en th e proportions o f n eg a tiv e answ ers
obtained from G erm an users w h o w ere p resented w ith an 'Original' v er sio n o f the
d ocu m en t and th o se w h o con su lted a 'Controlled' versio n o f the docum ent. T h ese
results sh o w , h o w ev e r, that it w a s essen tial to evalu ate separately th e im pact o f
certain CL rules on th e com p reh en sib ility o f M T d ocu m en ts. I f the results obtained
from users had b een
an alysed regardless o f the d ocu m en t ( N A V 0 5 1 2 1
or
N A V 0 5 _ 1 3 1 ), errors o f T ype 1 or 2 cou ld h ave b een m ad e during the an alysis. A s
stated b y K rey sz ig (1 9 9 9 : 1 1 1 8 ), a T yp e 1 error is to reject a true h y p o th esis, and a
T yp e 2 error is to a ccep t a fa lse h yp oth esis. T h ese results also con firm that on ce
certain CL ru les are ap plied togeth er, it is v ery d ifficu lt to determ ine their in d ivid u al
195
im pact. O n ce again, a m ore d etailed q ualitative stu d y m ay p rovide m ore in-depth
ex p lan ation s.
6 .5 .2 .
E v a lu a tin g th e im p a c t o f C L r u le s o n th e
u s e fu ln e s s o f t h e s e c o n d d o c u m e n t
T h e fin al tw o h y p o th eses con cern th e im p act o f th e C L rules on the u sefu ln ess o f
the d ocu m en t lN A V 0 5 _ 1 2 1 '. T w o separate h y p o th e ses w ere tested b ased on the
lo c a le in w h ich th e d ocu m en t w a s con su lted . T he first h yp oth esis con cern s the
im p act o f the CL ru les on the u sefu ln e ss o f th e F rench M T d ocu m en ts. T he secon d
h y p o th esis d eals w ith the im p act o f th e CL rules on th e u sefu ln e ss o f the G erm an
M T docum ents:
H4A: French users presented with the 'Original' version o f
document N A V 05_121 are as likely to find it useful as users
presented with the 'Controlled' version.
H4B: German users presented with the 'Original' version o f
document N A V 05_121 are less likely to find it useful than users
presented with the 'Controlled' version.
T h e se h y p o th e ses w e re tested by com parin g the proportions o f p o sitiv e answ ers
ob tain ed from th e grou p s o f users 'French C on trolled 1 and 'French O riginal1, and
'Germ an C ontrolled' and 'G erm an O riginal' re sp e ctiv ely . T h e results obtained from
th ese groups o f u sers are sh o w n in F igure 6 -1 0 . In both tests, no sign ifican t
d ifferen ce w a s fou n d b etw e en th e proportions o f p o sitiv e answ ers obtained from the
'Original' groups and th e 'Controlled' groups', so h y p o th e sis H 4 A m ust b e accepted
and H 4 B rejected.
196
W as this document useful?
original
controlled
original
controlled
German
French
Language and document version
Figure 6-10: Users’ answers to the first question for document 'NAV05_121'
T h ese results con firm w h at w a s h y p o th esised after the an alysis o f the results for
d o cu m en t N A V 0 5 1 2 1 . T he introduction o f extra C L rules in d ocu m en ts that do not
o rig in ally con tain m an y C L rule v io la tio n s is n ot the m o st im portant factor to take
into accou n t w h en a n a ly sin g the u sefu ln ess o f su ch tech n ical support docum ents.
T his statem ent lack s p recisio n , but it seem s very d ifficu lt to determ ine accurately
w h en a d ocu m en t b e c o m e s u ncontrolled , and w h en it w o u ld b en efit from the
ap plication o f extra C L rules so that th e u sefu ln ess o f corresp on din g M T d ocum ents
can b e sig n ifica n tly im p roved . T ranslated d ocu m en ts con tain in g refined M T output
appear to be u sefu l for m o st users, p rovided that th ey contain accurate translations
o f m o st tech n ical term s, tok en s, and U I options.
B a sed on th ese results, a fin al h yp oth esis m u st b e form ulated and tested, so that the
final q u estion o f th is study m ay be answ ered: d o es th e introduction o f extra CL
rules h ave any im p act on th e accep tab ility o f M T d ocu m en ts? T his h y p o th esis is
tested and d isc u sse d in th e n ex t section .
197
6.6. Im pact o f CL rules on the acceptability o f
MT docum ents
A s d iscu ssed in sectio n 5 .2 .3 .3 o f Chapter 5, th e con cep t o f accep tab ility is n ot easy
to grasp so it w a s d ecid ed n ot to ask th is q u estion d irectly to users. The
accep tab ility o f M T d o cu m en ts w a s evalu ated w ith the fo llo w in g tw o q uestions:
■
W ou ld y o u h ave preferred to read the E n g lish version o f this
d ocu m en t?
■
I f y o u w ere to en cou n ter another tech n ical problem in the future,
w o u ld y o u co n su lt again a m achin e-tran slated d ocum ent?
T he first o f th ese tw o q u estion s w a s asked to ch eck th e resp on d en ts’ n eed for
lo c a lise d tech n ical inform ation . A fter all, the original d ocu m en t is availab le in
E n g lish , so certain u sers m a y prefer to read the E n g lish version . H o w ev er, it has
b een assu m ed in th is stu d y that u sers o f con su m er softw are products m ay h ave a
lim ited k n o w le d g e o f E n g lish . H a v in g a cc ess to m achine-translated inform ation in
their o w n langu age m ig h t b e better than a m ere p is aller in certain situations. Figure
6-11 sh o w s the results ob tain ed for th e fourth q u estion for b oth docum ents:
Would you have preferred to read the English version of this document?
80-
<5
Û
3
S(>'
ro
c
tu
40-
<a
-Q
E
=
I yss
6 2
60
5 3
93%
58%
82%
4 7
20
7 2 %
originai
cnmtrollsd
controlled
French
original
German
Language and document version
Figure 6-11: Users' answers to the fourth question (both documents)
1 98
I no
It w a s d ecid ed to com b in e b oth d ocu m en ts for the an alysis o f th ese results after
ch eck in g that there w a s no sig n ifica n t d ifferen ce at the d ocu m en t le v e l w hen
com parin g the proportions o f p o sitiv e an sw ers p rovided by tw o distin ct groups. The
resu lts presented in Figure 6-11 clearly sh o w that regardless o f the docum ent
versio n , an o v erw h elm in g m ajority o f users w o u ld not h ave preferred to read the
E n g lish versio n o f th e docum ent. T h ese results su g g est that the E n glish sk ills o f
m o st users o f S ym an tec con su m er products are in su fficien t for them to find a
so lu tio n in original d ocu m en ts. M T d ocu m en ts therefore h ave the p oten tial to offer
a c c e ss to inform ation th ey w o u ld b e u nable to get. A s noted in the p reviou s section,
th e final q u estion in this seco n d study m u st be answ ered: d o es the introduction o f
extra CL rules h ave any im p act on the accep tab ility o f M T docu m en ts?
In order to an sw er th is q u estion it w a s d ecid ed to test tw o h yp oth eses. T he first one
fo llo w s the m od el that has b een u sed throughout the an alysis o f the results:
H5: Users presented with the 'Original' version o f a document are
less likely to consult again such a document than users presented
with a 'Controlled' version
T he seco n d h y p o th esis is p rom pted by so m e o f th e results obtained earlier. I f certain
u sers encou n ter com p reh en sion p rob lem s w h en con su ltin g a m achine-translated
d o cu m en t, th ey m ay be u n w illin g to co n su lt su ch d ocu m en ts again in the future.
B a se d on th e results obtained during th e testin g o f H y p o th esis H I , G erm an users
se e m to be le ss tolerant than French users w ith regard to the quality o f M T
d o cu m en ts. T he fo llo w in g h y p o th e sis is therefore form ulated:
H6: German users presented with an 'Original' version o f a
document are less likely to find this type o f document acceptable
than French users presented with an 'Original' version
199
The an sw ers obtained from both d ocu m en ts for the fifth q u estion are presented in
F igure 6-12:
Would you consult again a machine-translated document?
BO ■
60
-50
■ yes
13 ne
I
20-
29
45%
31%
ccntrotl&d
controlled
original
French
original
Gsrman
Language and document version
Figure 6-12: Users’ answers to the fifth question (both documents)
It w a s d ecid ed to test h y p o th eses H 5 and H 6 by co m b in in g the answ ers obtained for
both d ocu m en ts after ch eck in g that n o
sign ifican t d ifferen ce ex isted
at the
d o cu m en t le v e l. R esu lts, w h ich are p resented in F igure 6 -1 2 , did n ot reveal any
sig n ifica n t d ifferen ce w h en com parin g the proportions o f p o sitiv e answ ers obtained
from users presented w ith an 'Original' v ersio n o f a d ocu m en t w ith th ose obtained
from users p resented w ith a 'C ontrolled1 d ocu m en t. H y p o th esis H 5 therefore cannot
be co n firm ed b y th ese results.
O n the other hand, h y p o th e sis H 6 w a s con firm ed b y the results obtained from
F rench and G erm an users. T here is a sign ifican t d ifferen ce b etw e en the proportions
o f p o sitiv e an sw ers obtained from G erm an u sers p resented w ith an 'Original'
d o cu m en t and th o se ob tained from F rench users w ith sim ilar d ocu m en ts. From a
cu stom er serv ice p ersp ectiv e, h a v in g d ifferen ces in the w a y users respond to a type
o f d ocu m en t is prob lem atic. T h is p rob lem is am p lified by the fact that certain users
lack E n g lish sk ills w h ile n ot fin d in g M T output accep tab le en ou gh to con sider
co n su ltin g another M T d ocu m en t in the future. I f an M T service w as the o n ly
serv ice p rovided, th ese u sers or cu stom ers m igh t start lo o k in g elsew h ere for a
sim ilar product and service.
200
Possible explanations for these different attitudes towards MT output have already
been provided, one of them being the relative inferiority of the MT system’s
English-German pair compared to the English-French pair. The difference in word
order between the three languages is indeed more likely to penalise German than
French. Besides, more tolerance from the disciples of L’Académie Française with
regard to an MT version of their own language, even if imperfect, may account for a
different distribution of answers across the two locales.
6.7. Data analysis conclusions
One of the main findings of the second part of this study is that some of the
hypotheses formulated based on the expert opinions of translators had to be rejected.
This was the case for hypotheses HI for French, H2 for French and German, H3A,
and H4B. The lack of agreement between translators and users may be explained by
a variety of reasons. First of all, users and translators do not have the same
linguistic standards and expectations, so translators may be more likely to be
affected by translation inaccuracies than users, especially when these translation
inaccuracies are generated by MT systems. Besides, evaluators were asked to read a
document in its entirety, whereas users may have consulted only part of a document.
Since some of the CL rules were violated in sections that were not required for
users to complete a task, their impact may have gone unnoticed.
This finding limits the conclusions that can be derived from this study, since other
documents containing different rule violations may have generated different results.
The fact that results were not consistent across both documents suggest that many
parameters have the potential to mask the impact of certain CL rules. Such
parameters include the textual location of the rule violation, the amount of
terminological preparation performed prior to the MT process, and the ability of the
MT system to leave certain sections untranslated. While the impact of certain rules
was proved at the segment level, some of them failed to make an impact at the
document level.
201
7'he impact of certain CL rules was evaluated at the document level by asking users
to provide their opinions on the usefulness, comprehensibility, and acceptability of
MT documents. The most important finding of this study is that the introduction of
a set of CL rules in source documents produced different results depending on the
locale. Whereas no significant difference was noted with the answers of French
users for any of the questions, a significant difference was found in the proportion
of answers obtained from German users with regard to the comprehensibility of one
of the documents. These results clearly suggest that more studies arc required using
different documents, text types, and language pairs.
202
Chapter
203
7
Chapter 7: Conclusion
7.1. The objectives
This research had two main objectives. The first objective was to determine whether
certain MT-oriented CL rules would be more effective than others in improving the
comprehensibility of French and German MT segments. This objective was met
when comparing the results obtained from the evaluation of two sets of MT
segments. It must be stressed, however, that improvements sometimes only became
visible when performing a micro-analysis of these MT segments. In certain cases,
the use of well-defined evaluation and analysis metrics was not sufficient to
consistently determine whether a rule was effective. This analysis challenge was
explained by a number of reasons which will be summarised again in section 7.2
The second objective of this research was to investigate whether the application of
certain CL rules would impact on the reception of specific MT technical documents
from a Web user's perspective. This objective was met by conducting an online
experiment involving genuine users of Symantec products. This was the first time a
study of this type was conducted to evaluate the impact of CL rules on MT
documents. Such a study was only made possible thanks to this researcher's
privileged position at Symantec. Without the support of Symantec's Technical
Support and Localisation groups, such a study could not have involved genuine
users in a genuine setting. This unique case study suggests that the use of clearlydefined rules can significantly improve the comprehensibility of German technical
support documentation that has been machine-translated but not post-edited. For
French,
the
introduction
of CL
rules
did not
improve the usefulness,
comprehensibility, and acceptability of MT documents. The findings for German
must be interpreted very carefully since the results were not consistent across the
two documents that were used in this experiment. Additional findings of this
research are summarised in section 7.2.
204
7.2. F indings
The results of both pails of this study confirm that the comprehensibility of MT
output can be improved by the application of CL rules in certain situations to such
an extent that no comprehension problems are reported at the segment level and
document level. This finding brings empirical evidence to support the claim that
refined MT output can sometimes be useful, comprehensible, and acceptable for a
majority o f users when source content is strictly controlled. In the previous
statement, the word 'sometimes' is of paramount importance, because cases have
been identified when the impact of certain CL rules is not visible. Our data show
that machine-translated technical support documents can sometimes be useful to a
majority of users despite containing CL rule violations. This finding is consistent
with what was found in studies conducted with a different MT system or different
language pairs (Richardson: 2004, Jaeger: 2004).
In section 4.4.9 of Chapter 4, the CL rules that proved the most effective for both
French and German at the segment level were identified. Examples of such CL rules
include rules that specifically proscribe the use of conjoined verbs when they have
different valency, the use of pronouns that have no specific referent, or the use of
phrasal verbs. It is our contention that these rules could be used in a systematic way
to significantly improve the machine-translatability of Web-based technical support
documentation. These rules are a good complement to other very effective rules
which are often cited in writing guidelines. These include the use of spell-checked
content, grammatical constructions, and standard punctuation.
The analysis of the results obtained in the first part of the study enabled us to make
additional discoveries, which are limited to the two language pairs used in this study.
In certain cases, the scope of certain rules is sometimes too broad, leading to
situations where the application of the rule has no significant effect on the
comprehensibility of certain MT segments. For instance, this was the case with the
rule prescribing the use of relative pronouns. This was explained by the fact that the
MT system can properly handle allegedly complex or ambiguous structures,
sometimes keeping the ambiguity in the target language. Individual CL rules can
205
also have a negative impact on the comprehensibility of MT output, when
ambiguity is introduced by certain rewritings.
It was also found that the impact of individual CL rules is sometimes masked by
other linguistic phenomena, such as polysemy or homography, which can only be
handled using a controlled lexicon. Finally, certain individual rules are effective in
one language pair only, because the MT system's analysis and transfer rules differ
from one language pair to the next.
At the document level, results differ between the two language pairs, since the
introduction of extra CL rules significantly improved the comprehensibility of one
German MT document whereas it did not lead to any significant difference with the
French documents. These results suggest that another factor, the use of fully
updated user dictionaries, plays a significant role in ensuring the usefulness,
comprehensibility, and acceptability of MT documents. Further research is required
to determine the precise impact of this parameter with regard to the role played by
the application of CL rules.
Our data also show that regardless of the use of CL rules, an overwhelming
majority of users is likely to favour a machine-translated document instead of its
English version. These results therefore present a good opportunity for MT services
to be provided for this type of content as long as key parameters are respected (such
as the use o f customised user dictionaries and simple sentences). If these key
parameters were not respected, dissatisfied users who cannot benefit from MT
content might look for information elsewhere. In this situation, the raison d'être of a
translated document would be undermined, as mentioned by Loffler-Laurian (1996:
86):
L a tra d u c tio n e st u n o b jet fo u rn i à un c lien t qui d o it en être
s a tisfa it p o u r en co n so m m e r d ’avantage.
206
This trend was noticeable with German users. The present study found that German
users are less likely than French users to consult MT documents again when source
content is not controlled. This difference in attitude may be explained by genuine
translation issues contained in German MT documents, or by the relatively higher
linguistic standards expected by German users. Further research is required to
determine whether these differences can be reproduced with other language pairs
based on the methodological framework used in this study.
7.3. Review of the methodology used
Due to the lack of empirical studies in the field of CL and MT, it was decided to
combine two distinct experiments to evaluate the impact of CL rules at the segment
and document levels. If rules had only been evaluated at the segment level, it would
have been difficult to draw any conclusions on their overall impact. A CL rule may
prove very effective in improving the comprehensibility of individual MT segments,
but its impact may be limited at the document level if the rule is rarely violated in
original documents. Likewise, if rules had only been evaluated as part of a rule set
at the document level, it would have been impossible to determine their individual
contribution. This two-phase approach, despite certain weaknesses reviewed below,
is an original contribution to the methodology of CL evaluation mainly due to the
number of MT-oriented rules being evaluated. The test suite used in the first part of
this study could therefore be used as a starting point for future studies. As stated in
sections 4.4.5 and 4.4.6 of Chapter 4, the wide scope of certain rules coupled with
the number of diverse reformulations associated with specific rules prompts for
more investigation into the impact of sub-rules. This CL evaluation framework
could be easily re-used due to its modularity. In this study, rules have been
evaluated using a large number of examples, focusing on the output produced by
one MT system into two languages. The output produced by other MT systems in
other target languages could be evaluated in a similar fashion. Similarly, the impact
of these rules on stylistic characteristics of the source text could also be investigated.
During the first evaluation phase, the size of the test suite used was both a strength
and a weakness. To our knowledge, a test suite of this size had never been
employed in an empirical study of CL rules using human evaluators. In order to
make this study as thorough as possible and make a breakthrough in the field of CL
207
evaluation, a large number of rule violations was collected. In order to evaluate the
effectiveness of each of these MT-oriented rules, diverse linguistic patterns were
collected to avoid making hasty conclusions. From an evaluation perspective,
mixing examples from sets A and B at random would have presented several
advantages. First, it would have prevented evaluators from sensing that certain
segments had undergone some modifications and were part of a specific set of items,
be it controlled or uncontrolled. This would have ensured that they did not feel
obliged to give a better score to one of the sets. Second, it may have removed
potential reliability issues with regard to the scores attributed to the set containing
MT output B. Since evaluators were asked to score both sets separately, starting
with MT output A, their attitude towards MT output may have changed after
completing the evaluation of the first set. As mentioned by Krings (2001: 11), their
internal criteria may have changed during the evaluation of this first set, so they
may have started the evaluation of MT output B with different eyes, being
accustomed to some o f the output produced by the MT system. Mixing the items of
both sets would have prevented this situation. Using a random mix of items is the
strategy that was used in the second part of the study when asking a group of
evaluators to rate two versions of two machine-translated documents. This strategy,
which was described in section 6.2.1 of Chapter 6, seems appropriate for future
studies in the field of CL evaluation, be it at the segment or document level.
Besides, due to the final size of the test suite, the evaluation process proved
cumbersome for evaluators, as shown by certain scoring inconsistencies. During the
data analysis, these scoring inconsistencies were removed by using a combination
of median scores and integrity checks. Asking unmotivated users to conduct the
same evaluation process at the segment level would not have been possible. This
was confirmed by the lack of response obtained in the second part of the study
when users massively failed to respond to five short questions.
One o f the weaknesses of the online experiment performed in the second phase of
this study was indeed the low response rates it generated. Since response rates were
below 4%, one can only speculate as to what the rest of the users thought. As
mentioned by Lassen (2003: 81), it is not possible to 'provide evidence that those
who responded are representative of those who did not respond'. To our knowledge,
208
no previous study had used such a methodological framework to evaluate certain
characteristics of translated materials. It was therefore difficult to predict that users
would be so reticent in providing feedback. Despite this weakness, asking genuine
users to give their opinions on certain characteristics of MT documents allowed us
to determine that users' answers did not always match those provided by evaluators.
This finding makes a great contribution to the methodology of CL evaluation,
showing that genuine users must be involved in empirical studies focusing on the
role of CL rules. Relying purely on translators/evaluators does not appear sufficient
to make claims on the impact of certain rules on MT output, since the evaluators'
bias towards MT as a technology is a factor that must not be neglected. Different
results may have been obtained if 'fake' users (such as translation students) had been
employed instead. Having access to genuine users, even if their answers were scarce,
was therefore a great advantage to explore new horizons. It also enabled us to
establish that the difficulties involved in the evaluation of translations by users
should not be underestimated.
Another potential weakness of this online experiment concerns the number of
response categories included in the questionnaire. The selection of dichotomous
responses was motivated by the need to avoid 'too many fine distinctions' (De Vaus,
2002: 107) that could have confused the respondents and possibly impacted the
response rates. One of the issues associated with the approach taken in the present
study, however, is that these dichotomous responses did not capture the
respondents' real position with regard to the usefulness or comprehensibility of the
documents - for instance, the documents may have been completely useless and
incomprehensible
to
some
users,
but
to
others,
somewhat
useful
and
comprehensible. In future studies, a larger set of ordered responses may be
considered once the consequences of having non-committal answers have been
taken into account. Using a three-valued scale may provide more fine-grained
answers, as long as it does not trigger 'sitting on the fence answers' (De Vaus, 2002:
106). Future studies may yield interesting results if respondents were asked to
qualify how comprehensible MT documents are (easy to understand, difficult to
understand, or impossible to understand).
209
On the other hand, one of the strengths of this study lies in the experimental
variations that were performed to generate controlled documents out of original
documents. The aim of this study was to work with genuine original materials so as
to avoid creating artificial documents that may never be produced in a real life
scenario. The original documents chosen in this study contained violations of a
limited number of rules. To some extent, this lack of rule violations was initially
perceived as a disappointment, because a thorough evaluation of 54 CL rules had
been performed in the first part of the study. The absence of rule violations in the
original documents meant that the impact of certain CL rules would not be
evaluated at the document level unless rule violations were artificially introduced. If
such an approach had been taken to turn original documents into 'uncontrolled'
documents, the translation differences between 'uncontrolled' and controlled MT
outputs may have been more significant. However, the external validity of the study
would have been undermined because such 'uncontrolled' documents may never be
produced by professional content developers.
The results obtained in this study confirmed this assumption, showing that the
impact of extra CL rules may not be significant when key CL rules are
(unconsciously) being adhered to by content developers. This statement may be
refined by hypothesising that the rules that are adhered to in a specific part of a
given document are more important than the rules that are violated in less essential
sections o f a document. Future studies in the field of machine translatability would
need to take this challenge into account. If one wanted to determine a confidence
level for the machine-translatability of a document and the subsequent usefulness or
comprehensibility of its MT output, both rule adherence and rule violations should
be taken into account. Other challenges encountered during this study and requiring
further research are discussed in section 7.4.
210
7-4 « Future challenges and research
opportunities
The findings of the present study are limited to one MT system with two language
pairs, and one text type with two documents. The results of the second part of the
study showed that the introduction of a set o f CL rules did not have any significant
impact on the usefulness, comprehensibility, and acceptability of two French MT
documents while significantly improving the comprehensibility of one German MT
document. Based on these findings, language pairs could be diversified in future
studies to determine whether results obtained with Romance languages such as
Spanish and Italian are similar to those of French. On the other hand, an Asian
language with a different word order to English, such as Japanese, and traditionally
high linguistic standards might produce results closer to those obtained for German.
Future research could also involve different text types and MT systems.
The findings of the second part of this study are limited to the perceived usefulness
of MT documents by users. In order to determine whether CL rules would have any
impact on the actual usability of these documents, a genuine usability study would
have to be conducted in a controlled environment. Such a study may provide some
explanations for the discrepancies found between the evaluators' answers and the
users' answers during the data analysis. Based on the data obtained in this study,
certain users do not seem to be affected by 'incorrectly' translated sentences. Future
studies could further investigate the translation problems that have a negative
impact on users or certain categories of users. In this study, users were not asked to
provide any examples of sentences that hampered their comprehension so as to
avoid burdening them with detailed feedback. In a controlled environment, asking
for and getting this type of feedback may be done more easily.
The results obtained in both parts of this study also show that a set of CL rules is
sometimes not sufficient to significantly improve the comprehensibility of MT
documents. If a fully automated process were to be used in a production
environment, the use o f CL rules may need to be combined with other automated
processes. In section 3.3.1.2.1, the concept of automated post-editing was discussed.
This concept has been described by Allen and Hogan (2000) and Allen (2003), but
211
few studies have explored automated post-editing in a systematic way (Knight &
Chander 1994, Elming, 2006). Yet, certain customised MT systems, such as the
SPANAM system used at the Pan American Health Organization, make use of
macros to automate some of the post-editing tasks (Aymerich, 2001: 5). More
research is therefore required to investigate whether certain PE replacements could
be completely automated using a bilingual environment.
Such automatic
replacements could be useful to avoid implementing CL rules that are rarely
effective. To some extent, a similar approach may also be used to automatically
rewrite source text to make it more machine-translatable without asking authors to
adhere to a rule that is not always effective. Such an approach has been described in
Shirai et al. (1998) and Nyberg et al. (2003: 276). However, more studies are
required to determine the improvements generated by these automatic replacements,
when they are performed prior to the MT process as a pre-processing step of after
the MT process as a post-processing step.
7.5. Final words
This study has demonstrated that despite using strictly-defined evaluation criteria
determining accurately the effectiveness of a particular CL rule is a complex
process. While it is possible to formulate some conclusions on the impact of certain
rules at the segment level, generalising these results at the rule set and document
levels is complicated by several parameters. Before embarking on the strict
implementation of a set of CL rules in a particular environment, the impact of these
rules should therefore be fully evaluated. Besides, one should not forget that the
design of specific CL rules may preclude the refinement of existing MT systems.
Certain linguistic phenomena may currently not be handled accurately by specific
MT systems in certain language pairs, so a drastic approach would be to develop
CL rules to restrict the usage of these phenomena. Such an approach is restrictive
and does not seem to encourage future investment in the research and development
of MT technology. If the CL and MT paradigm were to be used in more diverse
situations, a balance must be found between the implementation of key CL rules
and the refinement of certain components of MT systems.
212
References
213
References
Acrolinx, 2005. Acrocheck™ version 2.6. Berlin: Acrolinx.
Adams, A.H., G. Austin, and M. Taylor. 1999. 'Developing a Resource for
Multinational Writing at Xerox Corporation'. In Technical Communication, Volume
46, Number 2. pp. 249-254.
Adriaens, G. 1996. 'SECC: Using Test Structure Information to Improve Checker
Quality and Coverage'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 226-232.
Adriaens, G. and L. Macken. 1995. ‘Technological evaluation of a controlled
language application: precision, recall, and convergence tests for SECC’. In
Proceedings o f the Sixth International Conference on Theoretical and
Methodological Issues in Machine Translation (TMI95), Vol.l. Centre for
Computational Linguistics, Leuven, Belgium, pp. 123-141.
Adriaens, G. and D. Schreurs. 1992. ‘From COGRAM to ALCOGRAM: Toward a
Controlled English Grammar Checker’. In COLING 92 Proceedings o f the N th
International Conference on Computational Linguistics, Vol. Ill, Nantes, France, pp.
595-601.
Akis, J. W., and W.R. Sisson. 2002. 'Improving Translatability: A Case Study at
Sun Microsystems'. In Globalization Insider XI/4.5. 4 pp. Available at:
http://vvwvv.lisa.orc/archive domain/iiewslctters/2002/4.5/akis.html?prinlerFriendlv
=yes [Last accessed: 18/12/2003],
Allen, J. 2003. ‘Post-Editing’. In H. Somers (ed.) Computers and Translation: A
Translator’s Guide. Amsterdam: John Benjamins Publishing Company, pp. 297317.
Allen, J. 2003. ‘Controlled Translation: the Integration of Controlled Language
(CL) and Machine Translation (MT)’. Panel talk presented at the European
Association for Machine Translation and Controlled Language Applications
Workshop (EAMT/CLAW2003). 15-17 May 2003. Available at:
http://www.ctts.dcu.ie/allen.ppt [Last accessed: 04/04/2005].
Allen, J. 2001. ‘Postediting: an integrated part of a translation software program’. In
Language International, April 2001, Vol. 13, No. 2. pp. 26-29.
Allen, J. and C. Hogan. 2000. ‘Toward the Development of a Postediting Module
for Raw Machine Translation Output: a Controlled Language Perspective’. In
Proceedings o f the Third International Workshop on Controlled Language
Applications (CLAW 2000), Seattle, Washington, pp.62-71.
Almqvist, I., and A. Sagvall Hein. 1996. 'Defining ScaniaSwedish- a Controlled
Language for Truck Maintenance'. In Proceedings o f the First Controlled Language
214
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 159-165.
ALP AC. 1966. Language and Machines: Computers in Translation and Linguistics.
Report by the Automatic Language Processing Advisory Committee (ALPAC),
Publication 1416, National Academy of Sciences National Research Council,
Washington, D.C.
Amant, K.S. 2003. 'Designing effective writing-for-translation intranet sites'. In
IEEE Transactions on Professional Communication, Volume 46, Issue 1, March
2003. p p .55-62
Armistead, C.G, and G. Clark. 1992. Customer Service and Support: Implementing
Effective Strategies. London: Pitman Publishing.
Arnold, D. 'Why Translation is Difficult for Translators'. 2003. In H. Somers (ed.)
Computers and Translation: A Translator’s Guide. Amsterdam: John Benjamins
Publishing Company, pp. 119-142.
Arnold, D., L. Balkan, R. Lee Humphreys, S. Meijer, and L. Sadler. 1994. Machine
Translation: An Introductory Guide. Manchester: NCC Blackwell.
Aymerich, J. 2001. 'Generation ofNoun-Noun Compounds in the Spanish-English
Machine Translation System SPANAM'. In Proceedings o f MT Summit VIII,
Santiago de Compostela, Spain. 6 pp.
Babych, B., D. Elliott, and A. Hartley. 2004. 'Extending MT evaluation tools with
translation complexity metrics'. In COLING 2004 Proceedings o f 20th International
Conference on Computational Linguistics, University of Geneva, Switzerland,
August 2004. pp. 106-112.
Baker, K. L., A. M. Franz, P. W. Jordan, T. Mitamura, and Eric H. Nyberg. 1994.
‘Coping With Ambiguity in a Large-Scale Machine Translation System’. In
COLING 94 Proceedings o f the 15th International Conference on Computational
Linguistics, Kyoto, Japan, pp. 90-94.
Balkan, L. 1994. ‘Test Suites: Some Issues in Their Use and Design’. In Machine
Translation: Ten Years On, Proceedings o f the second international conference on
Machine Translation organised by Cranfield University and the Natural Language
Translation Specialist Group o f the British Computer Society (BCS-NLTSG). British
Computer Society: 9 pp.
Barthe, K. 1996. 'Eurocastle - A User's Experience With Prototype AECMA SE
Checkers'. In Proceedings o f the First Controlled Language Application Workshop
(CLAW 1996), Centre for Computational Linguistics, Leuven, Belgium, pp. 42-63.
Barthe, K., C. Juaneda, D. Leseigneur, J-C. Loquet, C. Morin, J. Escande, and A.
Vayrette. 1999. 'GIFAS Rationalized French: A Controlled Language for Aerospace
Documentation in French'. In Technical Communication, Volume 46, Number 2,
May 1999. pp. 220-229
215
Bernth, A. 2006. 'EasyEnglishAnalyzer: Taking Controlled Language from
Sentence to Discourse Level'. Paper presented at the Fourth Controlled Language
Application Workshop (CLAW 2006), Boston Marriott, Cambridge, MA, USA. 7
pp.
Bernth, A. 1999. ‘Controlling Input and Output of MT for Greater Acceptance’. In
Proceedings o f the 21st ASLIB Conference, Translating and the Computer. London,
1999. 10 pp. '
Bernth, A. 1998. ‘EasyEnglish: Preprocessing for M T’. In Proceedings o f the
Second International Workshop on Controlled Language Applications (CLAW
1998), Pittsburgh, Pennsylvania, pp. 30-41.
Bernth, A. and M.C. McCord. 2000. ‘The effect of Source Analysis on Translation
Confidence’ In J. White (ed.) Envisioning Machine Translation in the Information
Future: Proceedings o f the 4th Conference o f the Association fo r M T in the
Americas, AMTA 2004, Cuernavaca, Mexico. LNAI 1934, Springer-Verlag: Berlin
Heideiberg. pp. 89-99.
Bernth, A. and C. Gdaniec 2001. ‘MTranslatability’, In Machine Translation,
December 2001, Vol. 16, No. 3. Dordrecht: Kluwer Academic Publishers, pp. 175218.
Bloor, T. and M. Bloor. 2004. The Functional Analysis o f English: A Hallydayan
Approach (Second Edition). London: Arnold.
Bosnjak, M., T. Tuten, and W. Bandilla. 2001. 'Participation in Web Surveys - A
Typology. ZUMA Nachrichten 48. pp. 7-17.
Bowker, D., and Dillman, D. 2000. 'An Experimental Evaluation of Left and Right
Oriented Screens for Web Questionnaires'. Paper presented at the Annual meeting
of the American Association for Public Opinion Research, May 2000. 19 pp.
Bowker, L. and J. Pearson 2002. Working with Specialized Language: A Practical
Guide to Using Corpora. London: Routledge.
Brants, T. 2000. 'TnT - A Statistical Part-Of-Speech Tagger'. In Proceedings o f the
Sixth Applied Natural Language Processing Conference ANLP-2000, Seattle, WA.
Bredenkamp, A., B. Crysmann, and M. Petrea. ‘Building Multilingual Controlled
Language Performance Checkers’. In Proceedings o f the Third International
Workshop on Controlled Language Applications (CLAW 2000), Seattle,
Washington, pp. 83-89.
Bryman, A. and D. Cramer. 2001. Quantitative Data Analysis with SPSS release 10
fo r Windows. Hove: Routledge.
216
Burke, B. 2004. Worldwide Antivirus 2004-2008 Forecast and 2003 Vendor Shares.
IDC report 31737. Available for purchase at:
http://www.idc.com/getdoc.jsp7containerIcK31737. [Last accessed: 03/09/005],
Byrne, J. 2004. Textual Cognetics and the Role o f Iconic Linkage in Software User
Guides. PhD thesis. Dublin City University, Dublin, Ireland.
Cardey, S., P. Greenfield, and X. Wu. 2004. ‘Designing a Controlled Language for
the Machine Translation of Medical Protocols: The Case of English to Chinese’. In
R.E. Frederking and K.B. Taylor (eds.) Proceedings o f the 6th Conference o f the
Association fo r M T in the Americas, AMTA 2004, Washington DC, USA. LNAI
3265, Springer-Verlag: Berlin Heideiberg. pp. 37-47.
Clemencin, G. 1996. 'Integration of a CL-Checker in a Operational SGML
Authoring Environment'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 32-41.
Coughlin, D. 2003. ‘Correlating Automated and Human Assessments of Machine
Translation Quality’. In Proceedings o f M T Summit IX, New Orleans, USA. pp. 6370.
D'Agenais, J., and J. Carruthers. 1985. Creating effective manuals. Cincinnati:
South-Western Pub. Co.
Dabbadie, M., A. Hartley, M. King, K. J. Miller, W. M. El Hadi, A. Popescu-Belis,
F. Reeder, and M. Vanni. 2002. ‘A Hands-on Study on the Reliability and
Coherence of Evaluation Metrics’. In Proceedings o f the Workshop on Machine
Translation Evaluation, LREC 2002. Las Palmas de Gran Canaria, Spain, May 2002.
pp. 8-16.
Danks, J.H., and J. Griffin. 1997. ‘Reading and Translation’. In J.H. Danks, G.M.
Shreve, S.B. Fountain, and M.K. McBeath (eds). Cognitive Processes in
Translation and Interpreting. London: Sage Publications.
Dirven, R. and M. Verspoor. 1998. Cognitive Exploration o f Language and
Linguistics. Amsterdam: John Benjamins Publishing Company.
Doddington, G. 2002. ‘Automatic Evaluation of Machine Translation Quality Using
N-Gram Co-Occurrence Statistics’. In Proceedings o f the Second International
Conference on Human Language Technology. San Diego, CA. pp. 138-145.
Dray, S.M. 2004. ‘Usable in New York, Usable in Nairobi’. In Multilingual
Computing & Technology. Vol. 15, Issue 5. pp. 31-32.
De Beaugrande, R., and W. Dressier. 1981. Introduction to Text Linguistics. New
York: Longman.
217
De Preux, N. 2005 ‘How Much Does Controlled Language Improve Machine
Translation Results’. In Proceedings o f the 27 th ASLIB Conference, Translating
and the Computer. London, 2005. 14 pp.
De Vaus, D. 2004. Surveys in social research. London: Routledge.
Dillman, D., R. Tortora, J. Conradt, and D. Bowker. 1998. 'Influence of Plain vs.
Fancy Design on Response Rates for Web Surveys'. Paper presented at annual
meeting o f the American Statistical Association, Dallas, TX. 6 pp.
Douglas, S., and M. Hurst. 1996. 'Controlled Language Support for Perkins
Approved Clear English (PACE)'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 93-105.
Dyson, M. and Hannah, J. 1987. ‘Towards a Methodology for the Evaluation of
Machine-Assisted Translation Systems’. In Computers and Translation, 2 (3). pp.
163-176.
Eclipse Foundation. 2005. Eclipse Platform 3.1. US: Eclipse Foundation.
Elliott, D., Hartley, A. and E. Atwell. 2004. ‘A Fluency Error Categorization
Scheme to Guide Automated Machine Translation Evaluation’. In R.E. Frederking
and K.B. Taylor (eds.) Proceedings o f the 6th Conference o f the Association fo r M T
in the Americas, AMTA 2004, Washington DC, USA. LNAI 3265, Springer-Verlag:
Berlin Heideiberg. pp. 64-73.
Elming, J. 2006. ‘Transformation-based correction of rule-based MT’. In
Proceedings o f the 11th Annual Conference o f the European Association for
Machine Translation, Oslo University, Norway, pp. 219-226.
Esselink, B. 2003. ‘Localisation and Translation’. In H. Somers (ed.) Computers
and Translation: A Translator’s Guide. Amsterdam: John Benjamins Publishing
Company, pp. 67-86.
Esselink, B. 2001. ‘Web Design: Going Native’. In Language International. Vol. 13,
No. l.p p . 16-18.
European Association of Aerospace Industries, 2001. AECMA simplified English.
CD-ROM. UK: AECMA.
Evans, M., C. Wei, and J. Spyridakis. 2004. 'Using Statistical Power Analysis to
Tune-Up a Research Experiment: a Case Study'. In Proceedings oflPC C 2004,
Professional Communication Conference, 2004. pp 14-18.
Farrington, G. 1996. 'AECMA Simplified English: an Overview of the International
Aircraft Maintenance Language'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp 1-21.
218
Flanagan, M. 1994. ‘Error Classification for MT Evaluation’. In Technology
Partnerships for Crossing the Language Barrier: Proceedings o f the First
Conference o f the Association fo r Machine Translation in the Americas. Columbia,
Maryland, USA. pp. 65-72.
Fouvry, F. and L. Balkan. 1996. 'Test Suites for Controlled Language Checkers'. In
Proceedings o f the First Controlled Language Application Workshop (CLAW 1996),
Centre for Computational Linguistics, Leuven, Belgium, pp. 179-192.
Freeman, B. 2006. 'What Content Belongs in a CMS?' In Multilingual Computing &
Technology's Content Management Getting Started Guide, April/May 2006, p. 4.
Fuchs, N.E. and R. Schwitter. 1996. ‘Attempto Controlled English (ACE)’. In
Proceedings o f the First Controlled Language Application Workshop (CLAW 1996),
Centre for Computational Linguistics, Leuven, Belgium, pp. 124-136.
Fukuoka, W., Y. Kojima, and J. Spyridakis. 1999. 'Illustrations in User Manuals:
Preference and Effectiveness with Japanese and American Readers'. In Technical
Communication, Volume 46, Number 2, May 1999, pp. 167-176.
Galesic, M. 2006. 'Dropouts on the Web: Effects of Interest and Burden
Experienced During an Online Survey'. In Journal o f Official Statistics, Vol. 22 No.
2, pp. 313-328.
Gdaniec, C. 1994. ‘The Logos Trans!atability Index’. In: Technology Partnerships
fo r Crossing The Language Barrier: Proceedings o f the First Conference o f the
Association fo r Machine Translation in The Americas. Columbia, Maryland, pp.
97-105.
Gerson, S.J., and S.M. Gerson. 2000. Technical Writing: Process and Product.
Upper Saddle River: Prentice Hall.
Gillham, W.E.C. 2000. Developing a Questionnaire. London: Continuum
International Publishing Group.
Godden, K.1998. ‘Controlling The Business Environment for Controlled Language’.
In Proceedings o f the Second Controlled Language Application Workshop (CLAW
1998), Pittsburgh, Pennsylvania, pp. 185-189.
Godden, K., and L. Means. 1996. 'The Controlled Automotive Service Language
(CASL) Project'. In Proceedings o f the First Controlled Language Application
Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven, Belgium,
pp. 106-114.
Goritz, A. S. 2006. 'Incentives in Web Studies: Methodological Issues and a
Review'. In International Journal o f Internet Science, Vol. 1, No. 1, pp. 58-70.
Govyaerts, P. 1996. 'Controlled English, Curse or Blessing? A User's Perspective'.
In In Proceedings o f the First Controlled Language Application Workshop (CLAW
1996), Centre for Computational Linguistics, Leuven, Belgium, pp. 137-142.
219
Gough, N. 2005. Example-based machine translation using the marker hypothesis.
PhD Thesis. Dublin: Dublin City University.
Hajic, J., P. Homola, and V. Kubon. 2003. ‘A Simple Multilingual Machine
Translation System’. In Proceedings o f M T Summit IX, New Orleans, USA. pp.
157-164.
Halliday, M.A.K and Hasan, R. 1976. Cohesion in English. London: Longman.
Hammerich, I., and C. Harrison. 2002. Developing Online Content: The Principles
o f Writing and Editing fo r the Web. Toronto: John Wiley & Sons, Inc.
Hartley, A., and C. Paris. 2001. 'Translation, controlled languages, generation'. In E.
Steiner, C. Yallop (Eds) Translation and Multilingual Text Production, pp. 307-325.
Berlin : Mouton de Gruyter.
Hayes, P., S. Maxwell, and L. Schmandt. 1996. 'Controlled English Advantages for
Translated and Original English Documents'. In Proceedings o f the First Controlled
Language Application Workshop (CLAW 1996), Centre for Computational
Linguistics, Leuven, Belgium, pp. 84-92.
Heald, I., and R. Zajac. 1996. 'Syntactic and Semantic Problems in the Use of a
Controlled Language'. In Proceedings o f the First Controlled Language Application
Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven, Belgium,
pp. 205-215.
Hoft, N.L. 1995. International Technical Communication: How’ to export
Information about High Technology. New York: John Wiley & Sons, Inc.
Holmback, H., S. Shubert, and J. Spyridakis. 1996. ‘Issues in Conducting Empirical
Evaluations of Controlled Languages’. In Proceedings o f the First Controlled
Language Application Workshop (CLAW 1996), Centre for Computational
Linguistics, Leuven, Belgium, pp. 166-177.
Hovy, E., M. King, and A. Popescu-Belis. 2002. ‘An Introduction to MT
evaluation’. In Workbook o f the LREC 2002 Workshop on Machine Translation
Evaluation: Human Evaluators Meet Automated Metrics. Las Palmas de Gran
Canaria, Spain, pp. 1.7.
Huijsen, W.O. 1998. ‘Controlled Language: An Introduction’. In Proceedings o f the
Second Controlled Language Application Workshop (CLAW 1998), Pittsburgh,
Pennsylvania, pp. 1-15.
Hutchins, J. 2004. 'Machine translation and computer-based translation tools:
what’s available and how it’s used'. In Bravo, J. M. (ed.) A new spectrum o f
translation studies, Issue 4. Valladolid: Universidad de Valladolid. 20 pp.
220
Hutchins, J. 2003. ‘Machine Translation: General Overview’. In R. Mitkov (ed.)
The Oxford Handbook o f Computational Linguistics. New York: Oxford University
Press, pp. 501-511.
Hutchins, W. J. and H. Somers. 1992. An Introduction to Machine Translation.
London, Academic Press.
IAI, 2002. CLATfactsheet. Available at: http://www.iai.unisb.de/docs/clatfactsheet.pdf. [Last accessed: 03/09/2005],
Isabelle, P. 1987. ‘Machine Translation at the TAUM Group’. In M. King (ed.)
Machine Translation Today: The State o f the Art. Edinburgh: Edinburgh University
Press, pp. 247-277.
ITR Ltd. 2002. Blackjack 2.03. Available for purchase at
http://www.itrblackjack.com. [Last accessed: 24/01/2006].
Jackob, N, and T. Zerback. 2006. 'Improving Quality by Lowering Non-Response:
A Guideline for Online Surveys'. Paper presented at the WAPOR Seminar 'Quality
Criteria in Survey Research VI', Cadenabbia, Italy, 2006.
Jaeger, P. 2004. 'Putting Machine Translation To Work - Case Study: Language
Translation At Cisco'. Presentation given at the LISA Global Strategies Summit,
Crowne Plaza Hotel, Foster City, California, USA, June 21-24, 2004.
Jufrasky, D., and J. Martin. 2000. Speech and Language Processing: An
Introduction to Natural Language Processing, Computational Linguistics, and
Speech Recognition. Upper Saddle River, New Jersey: Prentice Hall.
Kamprath, C., E. Adolphson, T. Mitamura, and E. Nyberg. 1998. ‘Controlled
Language for Multilingual Document Production: Experience with Caterpillar
Technical English’. In Proceedings o f the Second Controlled Language Application
Workshop (CLAW 1998), Pittsburgh, Pennsylvania, pp. 51-61.
Kaplan, R. N. 2003. ‘Syntax’. In R. Mitkov (ed.) The Oxford Handbook o f
Computational Linguistics. New York: Oxford University Press, pp. 70-90.
Kennedy, G. 1998. An introduction to corpus linguistics. London: Longman.
King, M. 1996. ‘On the Notion of Validity and the Evaluation of MT systems'. In H.
Somers (ed.) Terminology, LSP and Translation: Studies in language engineering
in honour o f Juan C. Sager, pp. 189-203.
King, M., and K. Falkedal. 1990. ‘Using Test Suites in Evaluation of Machine
Translation’. In COLING-90 Proceedings o f the 13th International Conference on
Computational Linguistics, Helsinki, Finland, pp. 211-216.
Kirkman, J. 1992. Good Style: Writing fo r Science and Technology. London: E &
FN Spon.
221
Kittredge, R. 2004. ‘Sublanguages and Controlled Languages’. In R. Mitkov (ed.)
The Oxford Handbook o f Computational Linguistics. New York: Oxford University
Press, pp. 430-447.
Kittredge, R. 1982. ‘Variation and Homogeneity of Sublanguages’. In Kittredge, R.
and J. Lehrberger (eds.) Sublanguage: Studies o f Language in Restricted Semantic
Domains. New York: Walter de Gruyter. pp. 107-137.
Krings, H.P. 2001. Repairing Texts: Empirical Investigations o f Machine
Translation Post-Editing Processes. Koby, G.S (ed.). Kent, Ohio: The Kent State
University Press.
Klare, G. 1963. The Measurement o f Readability. Iowa State University: Ames.
Knight, K. and I. Chander. 1994. ‘Automated Postediting of Documents’. In
Proceedings o f the Twelfth National Conference on Artificial Intelligence (AAAI),
Seattle, Washington, pp. 779-784.
Kohl, J.R. 1999. 'Improving Translatability and Readability with Syntactic Cues'. In
TechnicalCOMMUNICATION, Vol. 46, No. 2. pp. 149-166.
Kreyszig, E. 1999. Advanced Engineering Mathematics. Singapore: John Wiley and
Sons, Inc.
Lalaude M., V. Lux, and S. Regnier-Prost. 1998. ‘Modular Controlled Language
Design’. In In Proceedings o f the Second Controlled Language Application
Workshop (CLAW 1998), Pittsburgh, Pennsylvania, pp. 103-113.
Lassen, I. 2003. Accessibility and Acceptability in Technical Manuals.
Amsterdam/Philadelphia: John Benjamins Publishing Company.
Lauscher, S. 2000. ‘Translation Quality Assessment: Where can Theory and
Practice Meet? In The Translator: Evaluation and Translation, Special Issue, Vol. 6,
No. 2. pp. 149-168.
Lehmann, S., Oepen, S., Regnier-Prost, S., Netter, K., Lux, V., Klein, J., Falkedal,
K., Fouvry, F., Estival, D., Dauphin, E., Compagnion, H., Baur, J., Balkan, L., and
Arnold, D. 1996. ‘TSNLP - Test Suites for Natural Language Processing’. In
COLING 96 Proceedings o f the 14th International Conference on Computational
Linguistics, Copenhagen, Denmark, August 1996. pp. 711-716.
Lehrberger, J. and L. Bourbeau. 1988. Machine Translation: Linguistic
Characteristics o f M T Systems and General Methodology o f Evaluation.
Amsterdam: John Benjamins Publishing Company.
Levin, R. 2005. ‘Tools for Multilingual Communication’. In Multilingual
Computing & Technology, March 2005, Vol. 16, Issue 2. pp. 45-49.
Levine, D.M., and D. F. Stephan. 2005. Even You Can Learn Statistics. Upper
Saddle River, NJ: Pearson Prentice Hall.
222
Lieske, C., C. Thielen, M. Wells, and A. Bredenkamp. 2002. 'Controlled Authoring
at SAP'. In Proceedings o f Translating and the Computer 24, ASLIB, London, 2002.
(no page numbers).
Loffler-Laurian, A.M. 1996. La Traduction Automatique. Lille: Presses
Universitaires du Septentrion.
Lux, V., and E. Dauphin. 1996. 'Corpus Studies: a Contribution to the Definition of
a Controlled Language'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 193-204.
Marlow, A. 1995. Technical Documentation. Oxford: NCC Blackwell.
Marlow, A. 2005. Quality Control fo r Technical Documentation. Titchmarsh:
Marlow Dumdell.
McCarthy, A. 2005. An Investigation into the Effect o f Controlled Language Rules
on the Machine Translation o f Multiple Language Pairs. MA thesis. Dublin City
University, Dublin, Ireland.
Microsoft Corporation. 1998. The Microsoft manual o f style fo r technical
publications. 2nd Edition. Redmond: Microsoft Press.
Mitamura, T. Nyberg, E, and J. Carbonell. 1991. 'An Efficient Interlingua
Translation System for Multi-lingual Document Production'. In Proceedings o f the
Third Machine Translation Summit.
Mitamura, T. 1999. ‘Controlled Language for Multilingual Machine Translation’. In
Proceedings o f Machine Translation Summit VII, Singapore, 13-17 September 1999.
pp. 46-52.
Mitchell P., B. Santorini, and M. Marcinkiewicz; 1993. ‘Building a Large
Annotated Corpus of English: The Penn Treebank’. In Computational Linguistics,
June 1993, Vol. 19, Number 2. pp. 313-330.
M 0ller, M.H. 2003. ‘Grammatical Metaphor, Controlled Language and Machine
Translation’. In Proceedings o f EAMT-CLAW-03, Dublin City University, Dublin,
Ireland, 15-17 May 2003. pp. 95-104.
Moore, C. 2000. 'Controlled Language at Diebold, Incorporated'. In Proceedings o f
the Third International Workshop on Controlled Language Applications (CLAW
2000), Seattle, Washington, pp.51-61
Myerson, C. 2001. ‘Beyond the hype’. In Language International, Vol. 13, No. 1.
pp. 12-14.
Nielsen, J. 2000. Designing Web Usability: the Practice o f Simplicity’. Indianapolis:
New Riders Publishing.
223
Nielsen, J., and H. Loranger. 2006. Prioritizing Web Usability. Berkeley: New
Riders Press.
Nirenburg, S. 1987. ‘Knowledge and Choices in Machine Translation’. In S.
Nirenburg (ed.) Machine Translation: Theoretical and Methodological issues.
Cambridge: CUP. pp. 1-21.
Nyberg, E., and T. Mitamura. 1996. 'Controlled Language and Knowledge-Based
Machine Translation: Principles and Practice.' In Proceedings o f the First
Controlled Language Application Workshop (CLAW 1996), Centre for
Computational Linguistics, Leuven, Belgium, pp. 74-83.
Nyberg, E., T. Mitamura, and W.O. Huijsen. 2003. ‘Controlled Language for
Authoring and Translation’. In H. Somers (ed.) Computers and Translation: A
Translator’s Guide. Amsterdam: John Benjamins Publishing Company, pp. 245281.
O'Brien, S. 2006. Machine Translatability and Post-Editing Effort: An Empirical
Study using Translog and Choice Network Analysis. PhD thesis, Dublin City
University, Ireland.
O'Brien, S. 2003. ‘Controlling Controlled English: An Analysis of Several
Controlled Language Rule Sets’. In Proceedings o f EAMT-CLAW-03, Dublin City
University, Dublin, Ireland, 15-17 May 2003. pp. 105-114.
O Broin, U. 2005. ‘Taking Care of Global Business: Translatability and localization
of e-business applications’. In Multilingual Computing & Technology. Vol. 16,
Issue 5. pp. 35-39.
Ogden, C.K. 1930. Basic English: A General Introduction with Rules and Grammar.
London: Paul Treber.
O’Hagan, M. and D. Ashworth 2002. Translation-Mediated Communication in a
Digital World: Facing the Challenges o f Globalization and Localization. Clevedon:
Multilingual Matters.
Olohan, M. 2004. Introducing Corpora in Translation Studies. New York:
Routledge.
Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2002. ‘BLEU: A Method for
Automatic Evaluation of Machine Translation’. In Proceedings o f the 40th Annual
Meeting o f ACL, Philadelphia, PA. pp. 311-318.
Petrits, A. 2001. ‘EC Systran: The Commission’s Machine Translation System’.
Brussels: Translation Service, Directorate of Resources and Language Support.
Available at:
http://europa.eu.int/comm/translation/reading/articles/tools_and_workflow_en.htm
[Last accessed: 01/03/2006], 32 pp.
224
Pool, J. 2006. 'Can Controlled Languages Scale to the Web?' Paper presented at the
Fourth Controlled Language Application Workshop (CLAW 2006), Boston
Marriott, Cambridge, MA, USA. 12 pp.
Popescu-Belis, A., M. King, and H. Benantar. 2002. ‘Towards a Corpus of
Corrected Human Translations.' In Workbook o f the LREC 2002 Workshop on
Machine Translation Evaluation: Human Evaluators Meet Automated Metrics. Las
Palmas de Gran Canaria, Spain, pp. 17-21.
Popescu-Belis, A., P. Estrella, M. King, and N. Underwood. 2005. ‘Towards
Automatic Generation of Evaluation Plans for Context-based MT evaluation’.
ISSCO Working Paper 64, August 2005. Available at:
http://www.issco.iiniiie.ch/publicalions/workina-papers/papers/femti-isscowp64.pdf [Last accessed: 22/06/2006],
Power, R., D. Scott, and A. Hartley. 2003. 'Multilingual Generation of Controlled
Languages'. In Proceedings ofEAMT-CLAW-03, Dublin City University, Dublin,
Ireland, 15-17 May 2003. pp. 115-123.
Price, J., and H. Korman. 1993. How to Communicate Technical Information: A
Handbook o f Software and Hardware Documentation. Redwood City: The
Benjamin/Cummings Publishing Company, Inc.
Pullman, B. 2006. 'Size Really Doesn't Matter'. In LISA Globalization Insider,
August 2006. Available at:
http://vvww.lisa.ora/ulobalizationinsider/2006/08/size really doe.html [Last
accessed: 23/09/2006]
Pym, A. 2004. The Moving Text: Localization, Translation, and Distribution.
Amsterdam: John Benjamins Publishing Company.
Pym, P. J. 1990. ‘Pre-editing and the use of simplified writing for MT: an
engineer’s experience o f operating an MT system’. In Pamela Mayorcas (ed.),
Translating and the Computer 10: The Translation Environment 10 Years on,
London: ASLIB. pp. 80-96.
Quirk, R., Greenbaum, S., Leech, G., and J. Svartvik. 1985. A Comprehensive
Grammar o f the English Language ’. London: Longman.
Raman, M, and S. Sharma. 2004. Technical Communication: Principles and
Practice. Oxford: Oxford University Press.
Ramirez Polo, L., and J. Haller. 2005. ‘Controlled Language and the
Implementation of Machine Translation for Technical Documentation.’ In
Proceedings o f the 27 th ASLIB Conference, Translating and the Computer. London,
2005. 8 pp.
Reeder, F. 2004. ‘Investigation of Intelligibility Judgments’. In R.E. Frederking and
K.B. Taylor (eds.) Proceedings o f the 6th Conference o f the Association for M T in
the Americas, AMTA 2004, Washington DC, USA. LNAI 3265, Springer-Verlag:
Berlin Heideiberg. pp. 227-235.
225
Reid-Smith, E. 2000. E-Loyalty: How to keep customers coming back to your
website. New York: Harper Collins Publishers.
Reips, U.D. 2002. 'Standards for Internet-based experimenting'. In Experimental
Psychology, Volume 49, Issue 4. pp. 243-256.
Reuther, U. 2003. ‘Two in One: Can it work? Readability and Translatability by
Means of Controlled Language’. In Proceedings o f EAMT-CLAW-03, Dublin City
University, Dublin, Ireland, 15-17 May 2003. pp. 124-132.
Rice, C. 1997. Understanding Customers. Oxford: Butterworth-Heinemann.
Richardson, S., Dolan, W., Menezes, A., Pinkham, J. 2001. Achieving commercial
quality translation with example-based methods. In Proceedings o f MT Summit VIII,
Santiago de Compostela, Spain, pp. 293-298.
Richardson, S.D. 2004. ‘Machine Translation of Online Product Support Articles
Using a Data-Driven MT System’. In R.E. Frederking and K.B. Taylor (eds.)
Proceedings o f the 6th Conference o f the Association fo r M T in the Americas, AMTA
2004, Washington DC, USA. LNAI 3265, Springer-Verlag: Berlin Heideiberg. pp.
246-251.
Rochford. H. 2005. An Investigation into the Impact o f Controlled Language across
Different Machine Translation Systems: Translating Controlled English into
German. MA thesis. Dublin City University, Dublin, Ireland.
Ross, C. 2004. Chinese Grammar. New York: McGraw-Hill.
Roturier, J. 2004. ‘Assessing Controlled Language rules: Can they improve
performance o f commercial Machine Translation systems?’ In Proceedings o f the
26th ASLIB Conference, Translating and the Computer. London, 2004. (no page
numbers).
Roturier, J., S. Kramer, and H. Diichting. 2005. 'Machine Translation: The
Translator's Choice'. Presentation given at the 10th Annual Internationalisation and
Localisation Conference, LRC X, The Global Initiative fo r Local Computing,
Limerick, Ireland.
Rychtyckyj, N. 2006. 'Standard Language at Ford Motor Company: A Case Study
in Controlled Language Development and Deployment'. Paper presented at the
Fourth Controlled Language Application Workshop (CLAW 2006), Boston
Marriott, Cambridge, MA, USA. 10 pp.
Rychtyckyj, N. 2002. ‘An Assessment of Machine Translation for Vehicle
Assembly Process Planning at Ford Motor Company’. In S. Richardson (ed.)
Machine translation: from Research to Real Users, Proceedings o f the 5th
Conference o f the Association fo r Machine Translation in the Americas, AMTA
2002, Tiburon, CA, USA, October 8-12. LNAI 2499, Springer-Verlag: Berlin
Heideiberg. pp. 207-215.
226
Sager, J.C. 1994. Language Engineering and Translation: Consequences o f
Automation. Amsterdam: John Benjamins Publishing Company.
Schachtl, S. 1996. 'Requirements for Controlled German in Industrial Applications',
In Proceedings o f the First Controlled Language Application Workshop (CLAW
1996), Centre for Computational Linguistics, Leuven, Belgium, pp. 143-149.
Schäfer, F. 2003. ‘MT Post-Editing: How to Shed Light on the ‘Unknown Task’.
Experiences made at SAP’. In Proceedings o f EAMT-CLAW-03, Dublin City
University, Dublin, Ireland, 15-17 May 2003. pp. 133-140.
Schmidt-Wigge, A. 1998. 'Grammar and Style Checking in German'. In
Proceedings o f the Second Controlled Language Application Workshop (CLAW
1998), Pittsburgh, Pennsylvania, pp. 76-86.
Schwertel, U. 2000. 'Controlling Plural Ambiguities in Attempto Controlled English
(ACE)'. In Proceedings o f the Third International Workshop on Controlled
Language Applications (CLAW 2000), Seattle, Washington, pp. 105-119.
Scott, M. 2004. WordSmith Tools 4.0. Oxford: Oxford University Press.
Senellart, J., P. Dienes, and T. Väradi. 2001. ‘New Generation Systran Translation
System’. In Proceedings o f M T Summit VIII, Santiago de Compostela, Spain. 6 pp.
Senez, D. 1998. ‘The Machine Translation Help Desk and the Post-Editing Service’.
In Terminologie & Traduction 1-1998, OPOCE, Brussels: European Commission,
pp. 289-295.
Shackel, B. 1990. 'Human Factors and Usability'. In Preece, J. & L. Keller (eds.)
Human-Computer Interaction. Hemel Hempstead: Prentice Hall International, pp.
27-41.
Shirai, S., S. Ikehara, A. Yokoo, and Y. Ooyama. 1998. 'Automatic Rewriting
Method for Internal Expressions in Japanese to English MT and Its Effects'. In
Proceedings o f the Second Controlled Language Application Workshop (CLAW
1998), Pittsburgh, Pennsylvania, pp. 62-75.
Shubert, S.K, Spyridakis, J.H, Holmback, H.K, and Coney, M.B. 1995. ‘The
Comprehensibility of Simplified English in Procedures’. In Proceedings o f the
Professional Communication Conference:. 'Smooth sailing to the Future', IEEE
International 27-29 Sept. 1995. p. 171-173.
Slatin, J.M. and S. Rush, 2003. Maximum Accessibility: Making your Web Site
More Usable fo r Everyone ’. Boston: Addison-Wesley.
Smart. 2005. Smart Communications. Available at:
http://www.smartny.com/download.htm. [Last accessed: 03/09/2005],
227
Smart, J. 2006. 'SMART Controlled English'. Paper presented at the Fourth
Controlled Language Application Workshop (CLAW 2006), Boston Marriott,
Cambridge, MA, USA. 11 pp.
Society of Automotive Engineering (SAE). 2001. Surface Vehicle Recommended
Practice. Available at: www. 1isa.ore/useful/2001 /J245OPractice.pdf. [Last accessed:
18/11/2004], 11 pages.
Somers, H. 2003. ‘'Machine Translation: Latest Developments’. In R. Mitkov (ed.)
The Oxford Handbook o f Computational Linguistics. New York: Oxford University
Press, pp. 512-528.
Spyridakis, J. 2000. 'Guidelines for Authoring Comprehensible Web Pages and
Evaluating Their Success'. In Technical Communication, Volume 47 (3). pp. 301310.
Spyridakis, J.H., H. Holmback, and S.K Shubert. 1997. 'Measuring the
translatability of Simplified English in procedural documents'. In Transactions on
Professional Communication, IEEE Volume 40, Issue 1, March 1997. pp. 4 - 1 2
Spyridakis, J., C. Wei, J. Barrick, and E. Cuddihy. 2005. 'Internet-Based Research:
Providing a Foundation for Web-Design Guidelines'. In IEEE Transactions on
Professional Communication, Volume 48, No. 3, September 2005. pp. 242-260.
Sun Microsystems. 2003. Read ME First!: A Style Guide fo r the Computer Industry.
Mountain View: Sun Technical Publications.
Systran, 2005. SYSTRAN5.0 User’s Guide. Paris: Systran.
Torrejón, E. and C. Rico, 2002. A New Teaching Scenario Tailor-made for the
Translation Industry’. In Proceedings o f the BCS-EAMT Workshop: Teaching
Machine Translation, UM1ST, Manchester, UK, November 14-15, 2002 .
Tucker, A.B. 1987. ‘Current Strategies in MT research and development'. In S.
Nirenburg (ed.) Machine Translation: Theoretical and Methodological issues.
Cambridge: CUP. pp. 22-41.
Turian, J., L. Shen, and D. Melamed. 2003. ‘Evaluation of Machine Translation and
its Evaluation’. In Proceedings o f M T Summit IX, New Orleans, USA. 8 pp.
Underwood, N.L. and B. Jongejan. 2001. ‘Translatability Checker: A Tool to Help
Decide Whether to Use MT’. In Proceedings o f M T Summit VIII, Santiago de
Compostela, Spain, 18-22 September 2001. pp. 363-368.
van der Eijck, P., M. De König, and G. Van der Steen. 1996. 'Controlled Language
Correction and Translation'. In Proceedings o f the First Controlled Language
Application Workshop (CLAW 1996), Centre for Computational Linguistics, Leuven,
Belgium, pp. 64-73.
228
Van der Meer, J. 2006. 'The Emergence of FAUT: Fully Automatic Useful
Translation'. Keynote Address at the 11th Amiual Conference of the European
Association for Machine Translation, Oslo University, Norway.
Van der Meer, J. 2003. 'At Last Translation Automation Becomes a Reality: An
Anthology of the Translation Market'. Proceedings o f EAMT-CLA W-03, Dublin
City University, Dublin, Ireland, 15-17 May 2003. pp. 180-184.
Van Slype, G. 1979. ‘Critical Study of Methods for Evaluating the Quality of MT.
Technical Report BR19142, Bureau Marcel van Dijk/European Commission (DG
XIII), Brussels. Available at: http://www.issco.unige.ch/proiects/isle/van-slvpe.pdf.
[Last accessed: 23/02/2006].
Vassiliou, M., S. Markantonatou, Y. Maistro, and V. Karkaletsis. 2003. 'Evaluating
Specifications for Controlled Greek'. In Proceedings ofEAMT-CLAW-03, Dublin
City University, Dublin, Ireland, 15-17 May 2003. pp. 185-191.
Warburton, K. 2003. 'Terminology for Machine Translation'. Presentation given at
the LISA Forum, Washington DC, 2003.
Warburton, K. 2001. ‘Globalization and Terminology Management’. In S.E. Wright
and G. Budhin (eds.) Handbook o f Terminology Management, Vol.2. pp. 677-696.
Amsterdam: John Benjamins Publishing.
W3C, 1999. ‘Web Content Accessibility Guidelines 1.0’. Available from:
http://www.w3.org/TR/WCAG10. [Last accessed: 18/08/2005].
White, J. 2003. ‘H ow to Evaluate Machine Translation’. H. Somers (ed.) Computers
and Translation: A Translator’s Guide. Amsterdam: John Benjamins Publishing
Company, pp. 211-244.
White, J., T. O’Connell, and F. O’Mara. 1994. ‘The ARPA MT Evaluation
Methodologies: Evolution, Lessons/ and Future Approaches’. In Technology
Partnerships fo r Crossing the Language Barrier, Proceedings o f the First
Conference o f the Association fo r Machine Translation in the Americas, Columbia,
Maryland, pp. 193-205.
Wojcik, R.H., J.E. Hoard, and K. Holzhauser. 1990. ‘The Boeing Simplified
English Checker’. In Proceedings o f the International Conference, Human Machine
Interaction and Artificial Intelligence in Aeronautics and Space ’. Toulouse, France,
pp. 43-57.
Wojcik, R. and H. Holmback. 1996. 'Getting a Controlled Language Off the Ground
at Boeing'. In Proceedings o f the First Controlled Language Application Workshop
(CLAW 1996), Centre for Computational Linguistics, Leuven, Belgium, pp. 114123.
Wojcik, R., H. Holmback, and J. Hoard. 1998. ‘Boeing Technical English: An
Extension of AECMA SE beyond the Aircraft Maintenance Domain’. In
229
Proceedings o f the Second Controlled Language Application Workshop (CLA W
1998), Pittsburgh, Pennsylvania, pp. 114-123.
Yang, J. and E. Lange. 2003. ‘Going live on the Internet’. In H. Somers (ed.)
Computers and Translation: A Translator 's Guide. Amsterdam: John Benjamins
Publishing Company, pp. 191-210.
Yunker, J. 2003. Beyond Borders: Web Globalization Strategies. Indianapolis: New
Riders Publishing.
230