Cross-Lingual NLU: Mitigating Language-Specific Impact in Embeddings Leveraging Adversarial Learning
Low-resource languages and computational expenses pose significant challenges in the domain of large language models (LLMs). Currently, researchers are actively involved in various efforts to tackle these challenges. Cross-lingual natural language processing (NLP) remains one of the most promising strategies to address these issues. In this paper, we introduce a novel approach that utilizes adversarial techniques to mitigate the impact of language-specific information in contextual embeddings generated by large multilingual language models, with potential applications in cross-lingual tasks. The study encompasses five different languages, including both Latin and non-Latin ones, in the context of two fundamental tasks in natural language understanding: intent detection and slot filling. The results primarily show that our current approach excels in zero-shot scenarios for Latin languages like Spanish. However, it encounters limitations when applied to languages distant from English, such as Thai and Persian. This highlights that while our approach effectively reduces the effect of language-specific information on the core meaning, it performs better for Latin languages that share language-specific nuances with English, as certain characteristics persist in the overall meaning within embeddings.
The table below presents a sample of our results for the Spanish language in a zero-shot scenario, reporting accuracy and F1 score. These results highlight the significant superiority of the proposed method over baseline approaches. The table shows the mean values of the micro-average across five runs, demonstrating that the variance values approach zero, which indicates the model's high stability.
| Model | ID (Acc.) | SF (F1) |
|---|---|---|
| CL. XLU embd. | 36.94 | 17.50 |
| CL. CoVe | 37.13 | 5.35 |
| CL. multi CoVe | 53.34 | 22.50 |
| CL. multi CoVe w/ auto | 53.89 | 19.25 |
| Zero-shot SLU | 46.64 | 15.41 |
| Ours (variance) | 68.74† (7e-5) | 44.45† (2e-3) |
Results for Spanish on the Facebook multilingual dataset utilizing Spanish auxiliary data.
†: Significant results with p-value < 1-e5.
If you use this work, please cite our paper as follows:
@article{Cross-Lingual_NLU2024,
author = {Saedeh Tahery and Sahar Kianian and Saeed Farzi},
title = {Cross-Lingual NLU: Mitigating Language-Specific Impact in Embeddings Leveraging Adversarial Learning},
conference = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
year = {2024},
url = {https://aclanthology.org/2024.lrec-main.370/}
}