Translation Quality Assessment

The term “quality” has been the topic of discussion in translation studies for several years, and it has been challenging to define, but there is a long history of different approaches. For the translation of written texts, there is the proposal of ^{Reiss (2000)}, for whom the text operates at a communication level, and therefore, the translation should aim at being functionally equivalent to the source. ^{House (2015)}, who initially made the proposal in 1971 but presented a reviewed version in 2015, focuses the analysis on the register and genre of both texts. Also, ^{Nord (1997)}, with the functionalist approach, focuses on the target audience. ^{O’Brien (2012)} analyzed eleven quality assessment models and concluded that most of them were based on an error typology. Moreover, organizations such as the International Organization for Standardization and the European Committee for Standardization have drafted norms for the translation process where the qualities and competencies necessary for translating are specified (ISO 17100:2015, EN 15038). However, all these models are designed for written text and do not consider the technical parameters of audiovisual translation. Therefore, they are not suitable for assessing subtitling or other modalities.

Audiovisual translation entails technical restrictions specific to each modality that force translators to modify the texts only to comply with the spatial and temporal restraints imposed by the media they use. Thus, what might be considered an error in the translation of a scientific article might not be an error in a documentary’s interlingual subtitles. For example, in dubbing, the restrictions are related to lip synchronization and isochrony (^{Chaume, 2005}). Likewise, in audio description, the limitation is the time since it is only possible to describe when there is no dialogue (^{Orero & Vilaró, 2014}). Furthermore, for the different subtitling modalities, there is a limit on reading speed and characters per line, among others (^{Díaz-Cintas & Remael, 2021}). All these elements particular to each translation modality create the necessity of using a quality assessment method designed specifically for each modality.

Subtitles Quality Assessment

Subtitling is part of a superordinate concept: timed text, divided into several categories, subtitling, surtitling, subtitles for the deaf and hard of hearing, live subtitles, and cybersubtitling (^{Díaz-Cintas & Remael, 2021}). Each of these types of subtitles has its own set of limitations, and the guidelines are usually established by the company requesting the service from the translators, making the definition of quality in subtitling more complex.

Defining quality is not the only challenge; designing a model for measuring quality can be even more complicated. In interlingual subtitling, this term can be addressed differently depending on the agents involved in the subtitling process, subtitler, quality controller (QCer), language service provider, researchers, and others (^{Díaz-Cintas, et al., 2020}). Nonetheless, quality is measured every day in translation. In the case of interlingual subtitling, ^{Nikolić (2021)} describes the quality assurance processes followed in different companies -where QCers usually check the technical and linguistic aspects of the subtitles against the client’s guidelines- and presents the results of a survey which indicate that the stakeholders in the quality control process do not have significantly different opinions on what are quality subtitles. This is somehow different from what ^{Remael and Robert (2016)} found in their survey study on subtitle quality assurance and quality control, which suggested that clients seem to be more focused on the technical requirements than the linguistic ones, and subtitlers had a broader approach and considered every aspect equally (^{Remael and Robert, 2016}).

Guidelines tend to be a quality assessment tool in subtitling; but each company usually creates its own guidelines, and different translators’ associations have also proposed their guidelines. ^{Carroll and Ivarson (1998)} proposed the Code of Good Subtitling Practices a couple of decades ago, a short and simple list of principles for subtitle spotting and translation supported by the European Association for Studies in Screen Translation. This association also provides a list of guidelines for each audiovisual translation modality proposed by international translators’ associations such as the Association des Traducteurs/Adaptateurs de l’Audiovisuel (ATAA) or the Asociación de Traducción y Adaptación Audiovisual de España (ATRAE). However, these guidelines are focused on the technical limits and not on assessing the quality of a finalized product. They provide instructions regarding technical restrictions such as character limit, the use of italics and hyphens, and how to treat songs. There is also the Multidimensional Quality Framework (MQM) designed to assess human and machine translation, and it includes over 100 categories for issue types that can be selected depending on the specific type of text to be assessed (^{Lommel et al., 2014}). Nevertheless, it does not include categories related to audiovisual translation.

Within the audiovisual translation models, there is the NER model. ^{Martínez and Romero-Fresco (2015)} proposed this model for the quality assessment of live subtitles for the deaf and hard of hearing. It includes three severity labels for the errors (minor, standard, and serious) and measures accuracy by subtracting the total number of errors from the total number of words in the respoken subtitles. Then, the result is divided by the total number of words, and this is multiplied by 100 to obtain a number in percentage values. Based on this model, ^{Romero-Fresco and Pöchhacker (2017)} developed the NTR model to assess interlingual live subtitles. The NER model is also the basis for the FAR model proposed by ^{Pedersen (2017)}.

The FAR Model

In 2017, Pedersen proposed this model for assessing translated subtitles, an error-based analysis where the errors are included and classified depending on their impact on the viewer.

FAR stands for the three main areas assessed in the model: functional equivalence, acceptability, and readability. The functional equivalence area includes two types of errors, semantic and stylistic. These errors are related to the transmission of the original meaning or message; and for ^{Pedersen (2017)}, the adequate form of assessing this is pragmatic equivalence. Acceptability assesses the adherence of the subtitles to the target language norms and is divided into grammar, spelling, and idiomaticity errors. Finally, readability evaluates how easy to read the subtitles are, and it is essentially related to the technical parameters of subtitling. It is divided into the following three categories: segmentation and spotting, punctuation and graphics, and reading speed and line length.

This model uses error labels and a penalty point system by classifying the errors into minor, standard, or serious, with scores ranging from 0.25 to 2 points. It also proposes an approval rate by dividing the error score by the total number of subtitles.

The FAR model was selected to carry out this research due to the broad scope of the error categories and the penalty point system that allows the comparison of the results between the categories of one set of subtitles and several sets, which was the purpose of this study. However, some modifications were made to increase objectivity when classifying the errors and speed up the process in general.

Model Adaptation

The assessment of these subtitles was made to obtain a general perspective of the English subtitle quality in Colombian films with international exposure. However, given the study’s limits, and since the objective was to give an appreciation of a finalized product instead of a deep analysis of the translation errors, the adaptation of the FAR model in this study modified the use of the minor, standard, and serious labels since the author believes classifying each error within these three categories could entail a high degree of subjectivity unless a group of professional translators can reach an agreement on the severity.

The penalty points of the errors in the FAR model depend on their impact on the contract of illusion with the viewer. For ^{Pedersen (2017)}, the contract of illusion is an agreement the viewer makes when watching subtitled films where they pretend the subtitles are the exact representation of what is being said on screen. However, measuring that impact is not easy. There are some studies on the subject, such as that of ^{Deckert (2021)}, where the impact of spelling errors in interlingual subtitles was assessed, and it was found that the viewers did not report a negative impact on their enjoyment, comprehension, or cognitive effort. Furthermore, Kruger et al. (2022) conducted an eye-tracking experiment to assess the impact of high reading speeds and concluded that comprehension declined as the reading speed increased.

More research is necessary to reach a consensus regarding the impact of specific errors on the audience. Nonetheless, as presented in the models of ^{Pedersen (2017)}, ^{Martinez and Romero-Fresco (2015)}, and other quality assessment systems, not all errors have the same impact; missing a comma or adding a capital letter will not affect the viewer in the same way a semantic error would, but both should be avoided.

After carefully considering the previously mentioned issues, the author decided that it would be more suitable for the study to apply the severity labels to the categories instead of the errors as follows: minor for readability, standard for acceptability, and serious for functional equivalence. This modification recognizes that not all errors have the same impact while decreasing the subjectivity in the error classification. Therefore, the scores are assigned equally to all errors in the category: all readability errors will have a score of 0.25, all acceptability errors 0.5, and all functional equivalence errors 1.

The approval rate will also be obtained as proposed by the FAR model although, as expressed by ^{Pedersen (2018)}, there is no reference range for this type of measure in interlingual subtitling. The NER model (^{Martínez & Romero-Fresco, 2015}) considers an approval rate of over 98% acceptable, but since the calculation is based on words and not subtitles as the FAR model, they cannot be compared. Therefore, this measure will only be used to compare the films and support the qualitative data.

La sirga (Vega, 2012)

This film is a drama narrating the story of Alicia, who has fled the armed conflict in her village to go live with his uncle and help him build a hostel in the Andes. It was part of the Director’s Fortnight in Cannes 2012 and won the Best Debut Film award at the Havana Film Festival, among other international and national awards.

Table 1 shows that the functional equivalence category had the highest score, which is of concern since this category has the most severe errors, those that might affect the general cohesion and understanding of the film. In addition, it appears the translator had issues understanding some local phrases and expressions; for example, at 00:36:04, three characters are talking about a friend of theirs who is not present to help them with something.

In the subtitles in Table 2, we can see a semantic error with the phrase “A strange child.” In the original audio, “Y que anda raro” is a local expression that, in this context, means that the person is not behaving as usual, something is wrong with them, but it is related to the paranormal event. Instead of referring to this situation with the ill character, the translation repeats the fragment about the strange child. It was considered a functional equivalence error since the original sentence intends to explain why the character is not there to help the others, and omitting this information leaves the absence of the character unexplained, affecting the coherence of the dialogues especially since the next subtitle also has a functional equivalence error. The Spanish sentence “Sí, está enduendado” means to be haunted by an elf, but the English translation repeats the information about the child instead of referring to the character. Since the scene finishes there, the viewer might not understand the dialogue entirely.

The acceptability category had the highest number of errors, which were related to the spelling of proper names and some missing verbs and articles (such as the missing article in the sentence “I forgot shovels”), the name Óscar was written without the accent, there is a missing verb in “The only one who does don Julian,” and the name of the lagoon is La Cocha, thus, the article has to be capitalized, which was not the case in the entire film.

This film does not present a challenge regarding technical aspects of subtitling since the dialogues are slow. Table 3 presents some statistics regarding the technical aspects of the file, and it can be said that this film is within the average guidelines of the industry by not exceeding 40 characters per line and maintaining an average reading speed of 12 cps. Furthermore, the minimum time on screen of a subtitle is usually 1 second (^{Díaz-Cintas & Remael, 2021}), and at least one subtitle was below this measure. However, no penalizations were made for those subtitles since this could be within the guidelines provided to the translators and some companies consider 20 frames acceptable as the minimum duration (for example, Netflix).

La playa D. C. (Arango, 2012)

This film narrates the story of Tomás, an Afrodescendant young man who ran away from the armed conflict on the Pacific coast with his family and arrived in Bogotá, where life is not easy either. It was part of the Cannes Film Festival’s Un Certain Regard, and it won the Best Director award at the Santiago International Film Festival (SANFIC), among other international and national awards.

This film also had many errors in the functional equivalence category and had the lowest approval rate of all films; most issues were related to the translation of cultural references (see Table 4). For example, at 00:36:11, one of the characters is telling the story of a man who was fighting with his wife over a secret she discovered (Table 5).

The word “pelar” means “to peel,” but it is also commonly used in slang to replace “kill” (Table 5). However, in this case, it means that the woman was extremely angry with the man and almost got physical with him. The English translation is FAR from the original meaning and creates a different interaction between the characters; something similar to what happened at 00:37:59, where a character gives his opinion on his haircut.

The literal translation for “violento” is “violent,” but here it is slang for “really good” (Table 6). The man thinks his haircut is amazing; however, the translation conveys the opposite. This film, in particular, had a high number of spelling mistakes and punctuation errors, such as the following subtitles: “What’s uo?,” “Go on, tell him waht we decided.”, “You’ve got talent or ypu don’t.”, “Unbelieable!”

This film is also within the industry standards (Table 7), but the minimum duration on screen here is low, with only 16 frames in several subtitles.

Tierra en la lengua (Mendoza, 2014)

This is the story of a man raised in the violence of the Colombian countryside who one day makes a trip with two of his grandchildren to convince them to kill him before a disease does. It won a special mention by the jury and the Best Film award by the Youth Jury in the Pesaro Film Festival, among other international and national awards.

This film had the highest number of errors in total and the lowest approval rate for functional equivalence (Table 8). The approval rate for readability was the lowest of all films, but given the low score for these errors, the total approval rate of the film was not significantly affected; most errors were due to poor segmentation. Something probably related to the result in functional equivalence is the fact that it contains a significantly higher number of cultural references, which is where the translator failed to convey the accurate meaning. For example, at 00:00:45, a woman is describing her feelings for her husband (Table 9).

The original phrase “que no le temblara la mano” is an expression making reference to a determined, confident man who does not hesitates when making decisions. Unfortunately, the literal translation used in this case does not render the original meaning adequately, affecting the understanding of the sentence. The same issue happens at 00:17:55 when a woman is in a difficult yoga position, and her grandfather sees her and says, “Uy, mamacita, ¿se torció?” In Colombia, it is a local belief that if you have been in the heat or exercising, you should not open the freezer or take a cold shower because your face or body might get paralyzed due to the sudden temperature change; this is called “torcerse.” The grandfather’s comment in the scene is related to this local belief, and it is also a joke due to the intonation of his voice, a joke lost in the English translation of “Honey, you’re all bent out of shape!”

There were also ten missing subtitles in the file, two of which were on-screen text. Regarding readability, the score was affected by poor segmentation, as exemplified in Table 10.

Table 11 presents the technical statistics. The technical aspect of this film follows the tendency of the previous two. It is within the industry parameters regarding line length and reading speed, but the minimum duration on screen is too low (14f). Moreover, when observing the waveform in the subtitling program, it was noticeable that the spotting was not very precise; the out time of many subtitles is over 20 frames late, which would be the reason so many subtitles have reading speeds under 5 cps, and in some occasions, the audio does not match the subtitle on screen due to the late timing. Figure 1 shows the soundwave with the subtitles of a scene. We can see that the boxes do not exactly match the audio. The first subtitle appears six frames after the audio starts. Moreover, subtitles 1, 2, 3, and 4 end more than 6-7 frames after the audio, which affects the synchrony of the subtitles. These issues are not related to reading speed since it is low in all of them.

Figure 1 Soundwave of Tierra en la lengua 

La tierra y la sombra (Acevedo, 2015)

This film is a drama narrating a fragment of an old man’s life who returns after 17 years to a home he abandoned, given that his only son is ill. He quickly discovers that sugarcane monocrops have destroyed all the farms in the area. It won the Camera d’Or at Cannes and the Le Grand Rail D’Or and Cannes SACD awards during Critics Week, among other international and national awards.

The results in Table 12 indicate that, although the highest number of errors were in the readability category, the category with the highest score was also functional equivalence, with a score of 9. For example, at 01:04:20, one character admires the personality of the other character’s wife (Table 13).

While in the Royal Spanish Academy’s dictionary, the first definition for “bravo” is someone brave, in Colombia, this word is usually connected to being upset or bad-tempered, which is the case in this scene. Another linguistic error was found at 00:04:42, where one character is saying to the other that he arrived really fast from the other city. He says, “Le rindió” in Spanish, which means he arrived faster than expected. However, the English translation says, “You must be exhausted,” which is entirely unrelated to the original expression.

Two subtitles appeared 10 seconds before the actual start of the audio, and the dialogue happening at the moment was lost. This error would severely affect the viewer since it does not have coherence with what was said before. Therefore, the viewer cannot understand that part of the scene. Regarding readability, there were several missing commas and poor segmentation (Table 14).

Overall, the spotting of the subtitles can be significantly improved. The out time of many subtitles was unnecessarily late compared to the audio, and the in time was a couple of frames late sometimes, making the subtitles appear a little bit later than the audio. However, these issues were not included in the score. This film is also within the technical limits of the industry, except for the minimum duration on screen, which is again too low.

Niña errante (Mendoza, 2018)

This film tells the story of Angelita, a young girl who meets her three half-sisters when their father dies. They make a trip to leave Angelita with an aunt while they grieve their father and get to know each other. It won the Golden Wolf award at the Tallin Black Nights Film Festival and the Best Fiction Film award at the Colombian Film Festival New York, among other international and national awards.

This was the film with the highest total approval rate (Table 15). However, there are several errors related to local expressions or cultural elements, which is why the functional equivalence category has the lowest approval rate. One crucial but easily avoidable error was in the pseudonym of a poet. There is a scene where all the girls are listening to a recording from their father, which transcribed says, “Acordáte¹ de equis quini,” literally translated would be “Remember X five hundred.” However, it was translated as, “Remember the poet, ‘X Quint’...?”, which is incorrect because X Quint is not the name. X-504 is the correct pseudonym of a Colombian poet named Jaime Jaramillo Escobar and what should be included in the subtitle. The problem may come from the fact that the actor does not say the complete name “ex five hundred and four”; he only says “equis quini,” which is “ex five hund.” Nevertheless, the translator did not do further research to confirm the name, even though the next lines were a poem written by X-504. A quick search on the internet with the poem’s verses would have clarified who the author was (Table 16).

On the technical side, there were several subtitles with more than 45 characters per line, even one with 49 and another with 53; these were included as errors. Something particular about this film was the use of hyphens. The others used two hyphens at the beginning of the line to indicate that two characters were speaking, which is the standard. However, in this film, only the second line contained a hyphen, but it was consistent throughout the film. Therefore, there were no penalizations since the guidelines could have instructed it.