AWEJ for Translation & Literary Studies, Volume 10. Number 2. May 2026 Pp.43-58
School of Languages, Literacies and Translation
Universiti Sains Malaysia, Penang, Malaysia
School of Languages, Literacies and Translation
Universiti Sains Malaysia, Penang, Malaysia
&
Shanxi Vocational College of Information Technology
Shanxi, China
While neural machine translation has transformed cross-lingual communication, its efficacy in specialized, high-stakes domains such as legal translation remains underexplored. This study addresses the central question of how different neural machine translation paradigms perform under the extreme syntactic and terminological constraints of customs law. The primary aim is to quantitatively evaluate three prominent systems—Google Translate, Systran, and Tarjama Translator—for translating English customs legal texts into Arabic. A 1,000-segment domain-specific sample drawn from the Revised Kyoto Convention was assessed using the cross-lingual, optimized metric for translation evaluation in reference-based mode. Because each source segment was evaluated across all three systems, statistical inference was conducted using a within-segment repeated-measures analysis with multiplicity-adjusted post hoc pairwise comparisons. Results reveal a clear performance hierarchy: Google Translate achieved the highest mean score (87.70), followed closely by Tarjama Translator (87.51), while Systran scored significantly lower (84.48). Stratified analysis by sentence length demonstrated consistent performance degradation across all systems as syntactic complexity increased, with Systran exhibiting the steepest decline on long sentences. These findings establish an operational baseline for tool selection and inform policymakers regarding neural machine translation readiness for cross-border customs communication. It is recommended that practitioners incorporate complexity-aware risk controls and human-in-the-loop verification when deploying these systems in legal workflows to mitigate the risks posed by structurally flawed translations.
Damseh, K., & Ghassemiazghandi, M. (2026). A Quantitative Evaluation of Neural Machine Translation Systems for English-Arabic Customs Legal Texts. Arab World English Journal for Translation & Literary Studies 10 (2):43-58.
Alhassan, N. (2023, January 30). How Tarjama went from a home office to one of MENA’s translation powerhouses. Majarra. https://share.google/f4iRA9BCpOJ5VyQKe.
Alkatheery, E. R. (2023). Google Translate Errors in Legal Texts: Machine Translation Quality Assessment. Arab World English Journal for Translation & Literary Studies, 7(1), 208–219. https://doi.org/10.24093/awejtls/vol7no1.16.
Altarabin, Mahmoud. “Translating English legal lexical features into Arabic: Challenges and possibilities.” Arab World English Journal for Translation & Literary Studies 2, no. 2 (2018). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3187002.
Amrhein, C., & Sennrich, R. (2022). Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET. Association for Computational Linguistics, 1125–1141. https://doi.org/10.18653/v1/2022.aacl-main.83.
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., & Xie, X. (2024). A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3), 1–45. https://doi.org/10.1145/3641289.
Evtikhiev, M., Bogomolov, E., Sokolov, Y., & Bryksin, T. (2023). Out of the BLEU: How should we assess quality of the Code Generation models? Journal of Systems and Software, 203, 111741. https://doi.org/10.1016/j.jss.2023.111741.
Freitag, M., Rei, R., Mathur, N., Lo, C., Stewart, C., Avramidis, E., Kocmi, T., Foster, G., Lavie, A., & Martins, A. F. T. (2022). Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust. Association for Computational Linguistics, 46–68. https://doi.org/10.18653/v1/2022.wmt-1.2.
Gamal, D., Alfonse, M., Jiménez-Zafra, S. M., & Aref, M. (2022, May). Survey of Arabic machine translation, methodologies, progress, and challenges. In 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC) (pp. 378-383). IEEE. https://doi.org/10.1109/miucc55081.2022.9781776.
Ghassemiazghandi, M. (2025). Translation of Humour in Subtitles of Animation from English into Persian. Arab World English Journal for Translation and Literary Studies, 9(1), 17–42. https://doi.org/10.24093/awejtls/vol9no1.2.
Habib, R., Alkhawaja, L., Khoury, O., & Al-Sayyed, S. (2025). Six NMT systems, one language pair: Which best translates Arabic-English? World Journal of English Language, 16(1), 1. https://doi.org/10.5430/wjel.v16n1p1.
Intento. (2024, May 14). The state of Machine Translation 2024. https://inten.to/machine-translation-report-2024/.
Jon, J., & Bojar, O. (2023). Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation. Association for Computational Linguistics, 2191–2212. https://doi.org/10.18653/v1/2023.acl-long.122.
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Karpinska, M., Koehn, P., Marie, B., Monz, C., Murray, K., Nagata, M., Popel, M., Popović, M., . . . Zouhar, V. (2024). Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet. In Proceedings of the Ninth Conference on Machine Translation, 1–46. https://doi.org/10.18653/v1/2024.wmt-1.1.
Lee, S., Lee, J., Moon, H., Park, C., Seo, J., Eo, S., Koo, S., & Lim, H. (2023). A survey on Evaluation Metrics for Machine Translation. Mathematics, 11(4), 1006. https://doi.org/10.3390/math11041006.
Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., Zhao, T., Panalkar, A., Mehta, D., Pasquali, S., Cheng, W., Wang, H., Liu, Y., Chen, Z., Chen, H., . . . Zhao, L. (2025). Domain specialization as the key to make large language models disruptive: A Comprehensive survey. ACM Computing Surveys, 58(3), 1–39. https://doi.org/10.1145/3764579.
Liu, S., & Zhu, W. (2023). An analysis of the evaluation of the translation quality of neural machine translation application systems. Applied Artificial Intelligence, 37(1). https://doi.org/10.1080/08839514.2023.2214460.
Maruf, S., Saleh, F., & Haffari, G. (2021). A survey on document-level neural machine translation: Methods and evaluation. ACM Computing Surveys (CSUR), 54(2), 1-36. https://dl.acm.org/doi/pdf/10.1145/3441691.
Rivera-Trigueros, I. (2022). Machine translation systems and quality assessment: a systematic review. Language Resources and Evaluation, 56(2), 593-619. https://doi.org/10.1007/s10579-021-09537-5.
Sammarraie, H. S. M. A., & Ghassemiazghandi, M. (2024). Enhancing Post-Editing machine translation skills among Iraqi undergraduate students. Arab World English Journal for Translation and Literary Studies, 8(2), 165–182. https://doi.org/10.24093/awejtls/vol8no2.12.
Saunders, D. (2022). Domain Adaptation and Multi-Domain adaptation for Neural Machine Translation: A survey. Journal of Artificial Intelligence Research, 75, 351–424. https://doi.org/10.1613/jair.1.13566.
SYSTRAN. (2024). SYSTRAN is the pioneer in machine translation technology. SYSTRAN by ChapsVision. https://www.systransoft.com/systran/translation-technology/systran-50-years-of-mt-innovation/.
Tien, H. N., & Minh, H. N. T. (2019). Long Sentence Preprocessing in Neural Machine Translation. In 2019 IEEE-RIVF International Conference on Computing and Communication Technologies (RIVF) (pp. 1-6). IEEE. https://doi.org/10.1109/rivf.2019.8713737.
Vanroy, B., Tezcan, A., & Macken, L. (2023). MATEO: MT evaluation Online. In the 24th Annual Conference of the European Association for Machine Translation (EAMT 2023), 499–500. https://aclanthology.org/2023.eamt-1.
Wang, M., Pham, V. T., Moghimifar, F., & Vu, T. (2025). Proverbs Run in Pairs: Evaluating Proverb Translation Capability of Large Language Model. Association for Computational Linguistics, 1646–1662. https://doi.org/10.18653/v1/2025.findings-acl.83.
Zakraoui, J., Saleh, M., Al-Maadeed, S., & Alja’am, J. M. (2021). Arabic Machine Translation: A survey with challenges and future directions. IEEE Access, 9, 161445–161468. https://doi.org/10.1109/access.2021.3132488.
Zhang, Q., Fang, C., Ma, Y., Sun, W., & Chen, Z. (2023). A survey of learning-based automated program repair. ACM Transactions on Software Engineering and Methodology, 33(2), 1–69. https://doi.org/10.1145/3631974.
Zhang, Z., Abdullah, S. N. S., Abdullah, M. A. R., & Zhou, L. (2025). Google Translate or ChatGPT-4? A Multi-Metric Evaluation of Chinese-to-English Technical Translation. Forum for Linguistic Studies, 7(9). https://doi.org/10.30564/fls.v7i9.11014.
Zhao, Y., & Wang, Q. (2025). Improvement of English-Chinese bilingual learning by integrating semantic analysis and neural machine translation. Scientific Reports, 15(1), 27319. https://doi.org/10.1038/s41598-025-12614-2.
Zhao, Y., Zhang, Q., & Zhou, S. (2025). Orthogonal position representations for transformer in neural machine translation. PLoS ONE, 20(10), e0334443. https://doi.org/10.1371/journal.pone.0334443.
Karam Damseh is a Ph.D. candidate in the School of Languages, Literacies, and Translation at Universiti Sains Malaysia. His research focuses on machine translation evaluation, with particular interest in the interface between computational linguistics and legal compliance. ORCID ID: https://orcid.org/0009-0009-8178-6634
Mozhgan Ghassemiazghandi, Ph.D., is a Senior Lecturer at the School of Languages, Literacies and Translation, Universiti Sains Malaysia. Her research focuses on translation technology, machine translation, and audiovisual translation.
ORCID ID: https://orcid.org/0000-0002-3038-3124