Readability Transfer Capabilities of Neural Machine Translation Services

Abstract

Neural Machine Translation (NMT) services demonstrate high semantic accuracy, but their ability to convey the readability of the source text is understudied. This study, therefore, provides a comprehensive evaluation of readability transfer in four leading NMT services: Amazon Translate, Azure Translator, DeepL, and Google Cloud Translation, across the English-German, English-Turkish, and German-Turkish language pairs. For this analysis, translations from various genres were assessed using a combination of language-specific readability formulas and textual metrics. Results revealed a significant directional asymmetry: readability decreased when translating from English to German or Turkish, but increased from German or Turkish to English. Statistically insignificant but consistent differences were found among the four NMT services in readability scores, with target language and source text properties having a greater influence. The findings reveal that readability is not inherently preserved in NMT and is significantly influenced by the characteristics of the target language and the nature of the source text. This highlights the critical importance of considering readability metrics alongside semantic accuracy when evaluating machine translation, especially for applications that require high accessibility or target a specific level of accessibility, suggesting potential requirements for readability-focused post-editing.