Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders

MCML Authors

Yihong Liu

Dr.

→ Group Hinrich Schütze
 Computational Linguistics

Hinrich Schütze

Prof. Dr.

Core PI

Computational Linguistics

Abstract

Cross-lingual alignment is often assumed to improve cross-lingual transfer by bringing representations of different languages closer together. However, improvements in representational alignment do not consistently translate into better downstream performance. We investigate this disconnect using XLM-R models explicitly aligned across four language pairs with token-level, sentence-level, and masked-language-modeling objectives. We evaluate their zero-shot transfer on a token-level task (part-of-speech tagging) and a sentence-level task (sentence classification), and analyze both representational changes and the gradients induced by the alignment and downstream objectives. We find that embedding-based alignment metrics do not reliably indicate whether alignment will improve or degrade downstream performance. Moreover, alignment and downstream-task gradients are often nearly orthogonal, particularly when the alignment objective and downstream task operate at different representational levels. These findings suggest that representation alignment alone is insufficient for assessing cross-lingual transfer, and that the compatibility between alignment and downstream objectives should be considered when designing or evaluating alignment methods.

inproceedings VLS26

---

MRL @EMNLP 2026

6th Workshop on Multilingual Representation Learning at the Conference on Empirical Methods in Natural Language Processing. Budapest, Hungary, Oct 24-29, 2026.

---

Authors

Y. Veitsman • Y. Liu • H. Schütze

---

Links

DOI

---

Research Area

 B2 | Natural Language Processing