AWEJ for Translation & Literary Studies, Volume 8. Number 4. October 2024 Pp.41-53
Center for Translation and Intercultural Studies
School of Arts, Languages and Cultures
University of Manchester, Manchester, UK
This research aims to offer deeper insights into the translation quality of large language models and neural machine translation systems. It compares the performance of two state-of-the-art models, ChatGPT and DeepL, to explore which system achieves higher translation accuracy. This research employs a highly focused and specialized method to evaluate translation performance, enabling the identification of nuanced differences between advanced AI/machine models. It focuses on implied subjects in contemporary Chinese writer Qiuyu Yu’s Iraq travelogue and examines how the two models handle these implied subjects when translating the text into English. The analysis shows that both ChatGPT and DeepL exhibit high-level intelligence when dealing with the implied subjects. They can flexibly adjust sentence structures to maintain the implicitness or to make the implied subjects explicit. The analysis further reveals that ChatGPT performs better in accurately inferring the implied subjects, which thereby reflects ChatGPT’s superior contextual understanding capability. The comparison of the source text and target texts also shows that ChatGPT’s translation is more succinct, nuanced, and easier to read. DeepL’s translation, by comparison, prioritizes reducing potential misunderstanding. This research tentatively concludes that, though not specialized for translation, ChatGPT translates with higher accuracy and demonstrates more sophisticated intelligence compared to DeepL when translating literary texts.
Sun, R. (2024). Evaluating the Translation Accuracy of ChatGPT and DeepL Through the Lens of Implied Subjects. Arab World English Journal for Translation & Literary Studies 8 (4), 41-53.
AlAfnan, M. A. (2024). Large Language Models as Computational Linguistics Tools: A Comparative Analysis of ChatGPT and Google Machine Translations. Journal of Artificial Intelligence and Technology (Early Access). https://doi.org/10.37965/jait.2024.0549
Alkhawaja, L. (2024). Unveiling the New Frontier: ChatGPT-3 Powered Translation for Arabic-English Language Pairs. Theory and Practice in Language Studies, 14(2), 347–357. https://doi.org/10.17507/tpls.1402.05
Cai, L. (2024). How Does ChatGPT Compare with Conventional Neural Machine Translation Systems in Performing a Chinese to English Translation Task? Journal of Translation Studies, 4(1), 25–45. https://doi.org/10.3726/JTS012024.02
Castilho, S., Mallon, C. Q., Meister, R., & Yue, S. (2023). Do Online Machine Translation Systems Care for Context? What About a GPT Model? In M. Nurminen et al. (Eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation (12-15 June 2023, Tampere, Finland) (pp. 393–417). European Association for Machine Translation. https://aclanthology.org/2023.eamt-1.39
DeepL. (n.d.). Why DeepL? Retrieved 30 September 2024, from https://www.deepl.com/en/whydeepl
Gao, Y., Wang, R., & Hou, F. (2023). How to Design Translation Prompts for ChatGPT: An Empirical Study. arXiv (2304.02182). https://doi.org/10.48550/arXiv.2304.02182
Haque, S., Eberhart, Z., Bansal, A., & McMillan, C. (2022). Semantic Similarity Metrics for Evaluating Source Code Summarization. Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension (16-17 May 2022, New York, US) (pp. 36–47). Association for Computing Machinery. https://doi.org/10.1145/3524610.3527909
Hendy, A. et al. (2023). How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation. arXiv (2302.09210). https://doi.org/10.48550/arXiv.2302.09210
Jiao, W. et al. (2023). Is ChatGPT a Good Translator? Yes with GPT-4 as The Engine. arXiv (2301.08745). https://doi.org/10.48550/arXiv.2301.08745
Kamath, U., Keenan, K., Somers, G., & Sorenson, S. (2024). Large Language Models: A Deep Dive. Springer. https://doi.org/10.1007/978-3-031-65647-7
Manakhimova, S. et al. (2023). Linguistically Motivated Evaluation of the 2023 State-of-the-Art Machine Translation: Can ChatGPT Outperform NMT? In P. Koehn, B. Haddow, T. Kocmi, & C. Monz (Eds.), Proceedings of the Eighth Conference on Machine Translation (6-7 December 2023, Singapore) (pp. 224–245). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.23
OpenAI. (2024, May 13). Hello GPT-4o. https://openai.com/index/hello-gpt-4o
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A Method for Automatic Evaluation of Machine Translation. In P. Isabelle, E. Charniak, & D. Lin (Eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (6-12 July 2002, Philadephia, US) (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135
Peng, K. et al. (2023). Towards Making the Most of ChatGPT for Machine Translation. arXiv (2303.13780). https://doi.org/10.48550/arXiv.2303.13780
Popović, M. (2015). ChrF: Character N-gram F-score for Automatic MT Evaluation. In O. Bojar et al. (Eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation (17-18 September 2015, Lisbon, Portugal) (pp. 392–395). Association for Computational Linguistics. https://doi.org/10.18653/v1/W15-3049
Popović, M. (2017). ChrF++: Words Helping Character N-grams. In O. Bojar et al. (Eds.), Proceedings of the Second Conference on Machine Translation (7-8 September 2017, Copenhagen, Denmark) (pp. 612–618). Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-4770
Raunak, V., Menezes, A., Post, M., & Awadalla, H. H. (2023). Do GPTs Produce Less Literal Translations? arXiv (2305.16806). https://doi.org/10.48550/arXiv.2305.16806
Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A Neural Framework for MT Evaluation. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (16-20 November 2020, Online) (pp. 2685–2702). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.213
Schioppa, A., Vilar, D., Sokolov, A., & Filippova, K. (2021). Controlling Machine Translation for Multiple Attributes with Additive Interventions. In M.-F. Moens, X. Huang, L. Specia, & S. W. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (7-11 November 2021, Online and Punta Cana, Dominican Republic) (pp. 6676–6696). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.535
Segonne, V., & Mickus, T. (2023). “Definition Modeling: To Model Definitions.” Generating Definitions with Little to No Semantics. In M. Amblard & E. Breitholtz (Eds.), Proceedings of the 15th International Conference on Computational Semantics (20-23 June 2023, Nancy, France) (pp. 258–266). Association for Computational Linguistics. https://aclanthology.org/2023.iwcs-1.27
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A Study of Translation Edit Rate with Targeted Human Annotation. Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers (8-12 August 2006, Massachusetts, US) (pp. 223–231). Association for Machine Translation in the Americas. https://aclanthology.org/2006.amta-papers.25
Son, J., & Kim, B. (2023). Translation Performance from the User’s Perspective of Large Language Models and Neural Machine Translation Systems. Information, 14, 574. https://doi.org/10.3390/info14100574
Yamada, M. (2024). Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability. arXiv (2308.01391). https://doi.org/10.48550/arxiv.2308.01391
Zayed, A. B. (2024). Evaluating the Fidelity and Accuracy of ChatGPT 4 and Google Translate in Translating Legal English Documents into Arabic – and Vice Versa. Journal of the Faculty of Languages, University of Tripoli, Libya. 1(29), 64-87. https://uot.edu.ly/journals/index.php/flj/article/view/1063
Zhang, H. (2013). A Comparative Case Study of the “Subject” in Chinese and Its Parallel English Translation. Quarterly Journal of Chinese Studies, 1(3), 104–120. https://www.proquest.com/scholarly-journals/comparative-case-study-subject-chinese-parallel/docview/1412848586/se-2
Zhang, Z., Liu, S., Li, M., Zhou, M., & Chen, E. (2018). Joint Training for Neural Machine Translation Models with Monolingual Data. arXiv (1803.00353). https://doi.org/10.48550/arxiv.1803.00353
Rui Sun recently passed her PhD viva in Translation and Intercultural Studies at the University of Manchester in September 2024. Her research interests include corpus-based analyses of translator style, features of translation, and AI/machine translation. Her PhD project investigates translator style through explicitation and implicitation in the translations of reporting verbs using a corpus-based method. ORCID ID: https://orcid.org/0000-0003-3552-9601