MPLM-Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models
MCML Authors
Zheyu Zhang
Hinrich Schütze
Prof. Dr.
Principal Investigator
Abstract
Zheyu Zhang
Hinrich Schütze
Prof. Dr.
Principal Investigator
Abstract
Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to measure language similarity, and subsequently use the similarity results to select source languages for boosting cross-lingual transfer. To investigate this, we propose mPLM-Sim, a language similarity measure that induces the similarities across languages from mPLMs using multi-parallel corpora. Our study shows that mPLM-Sim exhibits moderately high correlations with linguistic similarity measures, such as lexicostatistics, genealogical language family, and geographical sprachbund. We also conduct a case study on languages with low correlation and observe that mPLM-Sim yields more accurate similarity results. Additionally, we find that similarity results vary across different mPLMs and different layers within an mPLM. We further investigate whether mPLM-Sim is effective for zero-shot cross-lingual transfer by conducting experiments on both low-level syntactic tasks and high-level semantic tasks. The experimental results demonstrate that mPLM-Sim is capable of selecting better source languages than linguistic measures, resulting in a 1%-2% improvement in zero-shot cross-lingual transfer performance.
inproceedings
Findings @EACL 2024
Findings of the 18th Conference of the European Chapter of the Association for Computational Linguistics. St. Julians, Malta, Mar 17-22, 2024.
Authors
P. Lin • C. Hu • Z. Zhang • A. Martins • H. SchützeLinks
URLResearch Areas
BibTeXKey: LHZ+24