WHAT IS TQA?

TQA is a type of evaluation, but what is evaluation? Michael Scriven, a leading

evaluation researcher, defines it a follows: Evaluationis taken to mean
the determination of merit, worth, or significance (2007: 1). This definition
itself presents a difficulty: How do we define value or worth, be it moral,
aesthetic or utilitarian? By extension, evaluation involves asking a question
that has challenged thinkers from time immemorial: Is a particular thing
good? Just like evaluation in the broad sense, TQA can be quantitative or
qualitative: it can be based on mathematical/statistical measurement (as in
the case of most academic instruments) or on reader response, interviews
and questionnaires (e.g. Nida). TQA can be diagnostic (determining areas for
improvement at the outset of a course of study), formative (measuring
progress and giving feedback during a course of study) or summative
(measuring the results of learning).

Quality in translation is certainly one of the most debated subjects in the field.
The strong interest it continues to generate among different groups, from
researchers and translation organisations to practitioners and translation
teachers, has made it a field of inquiry on its own, called translation quality
assessment (TQA). This interest is motivated by both academic and
economic/professional reasons: the need to evaluate students work and the
translation providers need to ensure a quality product). What makes a good
translation? What are the standards that have to be met for a translation to
be excellent, good or simply acceptable? Is there a universally
acceptable model of evaluation? In the absence of any precise answer to the
above questions, one can imagine an impetuous wish to develop an
evaluation system which would solve the problem of subjectivity by providing
a standard specification of what an acceptable translation should or should
not contain. But, as it is extensively recognised (Pym 1992, Sager1989),
there is still no universally accepted evaluation model in the translation world:
there are no generally accepted objective criteria for evaluating the quality of
both translations and interpreting performance. Even the latest national and
international standards in this area DIN 2345 and the ISO 9000 series
do not regulate the evaluation of translation quality in a particular context.
[] The result is assessment chaos. (Institut fr Angewandte Linguistik und
Translatologie, 1999, in Williams, 2001: 327). The reason why no single
standard will suffice is that quality is context dependent. This is what Sager
(1989) says when he says there are no absolute standards of translation
quality, but only more or less appropriate translations for the purpose for
which they are intended. For many types of texts, both vocative and
informative, an important element of their appropriateness or fitness for
purpose will be extrinsic whether they effectively usable by their
consumers/readers in pursuit of their purpose. Since the establishment of
such extrinsic standards of translation quality is elusive, a common tendency
is to take a narrower view, focusing on intrinsic characteristics of translated
texts and on errors committed in translation as a way of measuring quality.

PROBLEMS AND ISSUES IN DESIGNING AND APPLYING TQA

Why is it so difficult to establish and apply a TQA model? There are many reasons. I
consider the following ones to be the most important, and they inevitably entail problems
of validity or reliability.
a) The evaluator: Does the evaluator have the linguistic or subject-field knowledge
required? The client, whose knowledge may be limited, inevitably evaluates the finished
product too. Indeed, the clients assessment may be the only one. Further, a number of
translation researchers, including Hnig and other functionalists, Dyson (1994) and
Kingscott (1996) have implicitly or explicitly given precedence to the readers response
or requirements, not the translators definition of an adequate translation, as the yardstick
for gauging quality.

b) Level of target language rigour: Elegant style is considered essential by some

evaluators, but not by others. Some evaluators consider typos and spelling and
punctuation errors to be peccadillos and ignore them in their overall assessment, while M.
Williams / Translation Quality Assesment Mutatis Mutandis. Vol 2, No 1. 2009. pp. 3 - 23
6 others will regard them as serious because they are precisely the errors that the
client/end user will detect.
c) Seriousness of errors of transfer: The same inconsistency is apparent in the assessment
of level of accuracy. Some evaluators will ignore minor shifts in meaning if the core
message is preserved in the translation, while others will insist on total "fidelity," even if
an omission of a concept at one point is offset by its inclusion elsewhere in the text.
Reasons (b) and (c) underlie the frequent complaints about evaluator subjectivity.
d) Sampling versus full-text analysis: TQA has traditionally been based on intensive error
detection and analysis and has therefore required a considerable investment in human
resources. It takes time. One means of obviating the problem has been sampling the
analysis of samples of translations instead of whole texts. Yet this approach has
shortcomings. First, the evaluator may not take into account any "compensatory" efforts
that the translator has made in unsampled parts of the text. Second, the evaluator may not
have taken into account the co-text in order to grasp the meaning of the text as a whole.
Third, as Daniel Gouadec has pointed out, "There is always a risk that the most serious
errors may lie outside the samples. This is especially true of the work of established
translators, who are capable of dramatic, uncontrolled deviations from the meaning of the
source text (1989: 56).
e) Quantification of quality: Microtextual analysis of samples has been used extensively
not only because it saves time but also because it provides error counts as a justification
for a negative assessment. Translation services and teachers of translation alike have
developed TQA grids with several quality levels, or grades, based on the number of errors
in a text of fixed length. It is felt that quantification lends objectivity and defensibility to
the assessment. The problem lies with the borderline cases. Assuming that, in order to be
user-friendly, such a grid does not allow for many levels of seriousness of error, it is quite
possible for a translation containing one more error than the maximum allowed to be as
good as, if not better than, another translation that contains exactly the maximum number
of errors allowed and yet be rated unsatisfactory.
f) Levels of seriousness of error: One way to circumvent the drawbacks of quantification
is to grade errors by seriousness: major, minor, weak point, etc. The problem then is to
seek a consensus on what constitutes a major, as opposed to a minor, error. For example,
an error in translating numerals may be considered very serious by some, particularly in
financial, scientific or technical material, yet others will claim that the client or end user
will recognize the slip-up and automatically correct it in the process of reading.
g) Multiple levels of assessment: Many authorities, including Nord (1991) and House
(1997), identify a number of parameters against which the quality of a translation M.
Williams / Translation Quality Assesment Mutatis Mutandis. Vol 2, No 1. 2009. pp. 3 - 23

7 should be assessed: accuracy, target language quality, format (appearance of text),

register, situationality, etc. The problem is this: Assuming you can make a fair assessment
against each parameter, how do you then generate an overall quality rating for the
translation?
h) TQA purpose/function: The required characteristics of a TQA tool built for formative
assessment in a university context may differ significantly from one developed for
predelivery quality control by a translation supplier. According to Hatim and Mason,
"Even within what has been published on the subject of evaluation, one
must ],distinguish between the activities of assessing the quality of translations
[ translation criticism and translation quality control on the one hand and those of
assessing performance on the other" (1997: 199).
Professional Approaches to Translation Evaluation
The 70s saw a development in the field, in both practice and theory. A growing
emphasis started to be placed on the creation of explicit and applicable correction scales
and, as a result, on the creation of translation error typologies. Following this, the idea of
a translation acceptability threshold based on a certain number of errors was introduced.
The broader motivation was to reduce such factors as time, money, human effort and
subjectivity and to introduce a more systematic type of analysis. The commitment to
deliver error-free translations to clients, the enormous amount of materials to be
translated, and the growing competition between translation providers resulted in an
increasing interest in quality assurance. The first step towards an innovative TQA model
which had at heart the concept of categorisation of errors, was taken in Canada in the 70s
and resulted in the Canadian Language Quality Measurement System (Sical), Canadian
governments Translation Bureaus property. This system, in its successive versions was
developed by Gouadec (Williams 2001) and was based on an error scheme which, on the
one hand, made the distinction between transfer and language error and, on the other
hand, labelled the error as being major or minor. The complexity of the scale is such that
it allows 39 the identification of 675 error types (300 lexical and 375 syntactic) (Melis
& Hurtado 2001: 274). In judging the acceptability of one translation, the major errors
were the one that counted. A major error was considered when, translating an essential
element from the ST, the translator would fail to render the exact meaning of the original,
create confusion related to meaning or use incorrect or obvious inadequate language. As a
result, the threshold for acceptability was not situated too high and, presumably,
translations of a questionable quality could have still be judged as acceptable. In theory,
then, a fully acceptable translation of 400 words could contain as many as 12 errors of
transfer, provided no major error was detected. However, the designers of Sical III
predicated the lowering of the tolerance level on the statistical probability that a
translation with 12 such errors would also contain at least one major error. (Williams,
2001: 330) It was clear that Sical was conducting a sample analysis at the word and
sentence level, not on the text as a whole. The fact the Sical deals only with syntactic and
semantic aspects means it overlooks any phenomena that occur at the level of sentence
relations. This resulted in criticism over the acceptability of the content of a translation as
a whole and over the imprecision of the specific number of errors and their type

(Williams, 2001:331). Moreover, the large number of error types made this model hard to
use. However, it proved to be popular, since numerous other organisations and agencies
in Canada (the Ontario government translation services, Bell Canada) opted for a
customised version of Sical. The search for workable evaluation schemes based on error
classification has continued, with many following the Sical model and listing a number of
error categories with or without a certain score attached to each and every one of them. In
that category fall schemes developed and adopted by big translation organisations such as
ATA (American Translators Association) whose scheme include 22 errors types ranging
from terminology and register to accents and diacritical marks. The categories require the
evaluator to spot the translation errors, then to assign 1, 2, 4, 8, or 16 error points for each
error. A passage (of usually 225- 275 words) with a final score of 18 or higher is marked
Fail. This tendency to assign a weighting on a pre-defined scale to every translation error,
rather than simply mark it as minor or major, rapidly gained popularity as it was
considered a step forward in the development of translation quality evaluation models.
But, while acknowledging that such a scale is more refined, we cannot implicitly accept
its objectiveness. No meta rules are stressed as to how an evaluator should apply these
scores, that is what constitutes a 1-point error versus a 16-point one. By making it
available on their web site, ATA gives the translators an idea about what types of errors
might be allowed in a translation that meets ATA standards.

In an attempt to further minimise the time and effort spent by the evaluator in objectively
grading a translation, some companies used the computer to manage mathematical
operations and manual processing in the allocation of translation errors. SAE J 2450 is a
quality metrics developed by SAE (Society of Automotive Engineers) in collaboration
with GM (General Motors). The aim was to establish a standard quality metric for the
automotive industry that could be used to provide an objective measure of linguistic
quality for automotive service information regardless of language or process. The metric
became an SAE Recommended Practice in October 2001. The model is based on seven
error categories focussing on content problems that might affect the overall understanding
of the content, rather than on style see Figure 2. These categories prompt the evaluator
or the translator to classify them as major or minor, with a numeric score and severity
level (serious /minor) attached to each error. According to its relevance in the source text
(ST), each error has a certain weight; the final score is obtained by adding up the scores
of the errors and dividing the result by the number of words in the text.)
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.126.3654&rep=rep1&type=pdf