The MaLA-LM Journey: Scaling Multilingual Large Language Models to 500+ Languages

The rapid evolution of Large Language Models (LLMs) has transformed natural language processing. Yet, the benefits of these models remain concentrated in a handful of high-resource languages. The MaLA-LM initiative, a collaborative research team, is breaking these barriers by developing data-driven methods to scale multilingual language models to over 500 languages.

Historically, many human languages have been left behind in the AI revolution. Building state-of-the-art multilingual models requires navigating significant hurdles. Finding high-quality data for rare languages is incredibly difficult, and adapting a model to understand completely new grammatical structures takes delicate work. The MaLA-LM project directly tackles these bottlenecks. Our goal is to create pathways for cross-lingual transfer that allows models to generalize from high-resource languages to minority or underrepresented ones.

The Landscape of Prior Endeavors

The quest to build massively multilingual models has seen several foundational milestones. In late 2022, projects like SERENGETI [1] pioneered African language representation by training an encoder model covering 517 languages, taking a powerful step toward bridging the digital language divide.

This was followed by Glot500 [2] in May 2023, which constructed massive corpora and continually pretrained the XLM-R encoder on over 500 languages. In September 2023, MADLAD-400 [3] scaled decoder architectures using the modified UL2 framework [4], prioritizing data quality via large, audited document-level datasets.

Despite these advances, scaling to hundreds of languages triggers the "curse of multilinguality" [5]. As model capacity is shared across more languages, performance on individual languages can degrade. Mitigating this degradation requires careful representation alignment, specialized tokenization, targeted data curation, and advanced training techniques.

The MaLA-LM Journey

Tracing its trajectory back to 2023, I started the MaLA-LM project, which stands for massive language adaptation of language models. With foundational support from Jörg Tiedemann at the University of Helsinki, I embarked on this journey, collaborating with researchers from LMU Munich (Peiqin Lin and Hinrich Schütze) to develop the first iteration of the MaLA-LM model series: MaLA-500 [6]. Our first MaLA-500 model was trained on the Glot500-c dataset and adopted a continual pretraining strategy with a vocabulary expansion mechanism to accommodate the linguistic diversity of 534 languages. The model demonstrated good results in cross-lingual transfer, particularly for low-resource languages, as evaluated on intrinsic and text classification tasks. We did not frame the approach itself as novel; rather, we saw the contribution as showing that continual pretraining with vocabulary expansion could be effective for low-resource languages and as encouraging future research in this area. However, the work turned out to be premature. We later evaluated the model on more tasks and found that continual pretraining with vocabulary expansion was not effective for low-resource languages, especially for text generation tasks.

Later, we rethought the continual pretraining strategies, including training corpus, data mixes, and training methods, and more collaborators joined the project. In 2024, we constructed and released the MaLA corpus [14] which is a broad-coverage pretraining dataset containing clean documents spanning 500+ underrepresented languages, sourced, normalized, and filtered from various public repositories. In model training, we moved away from vocabulary expansion. While conceptually appealing, it proved to be ineffective, especially for low-resource languages where there simply aren't enough training examples to adequately optimize the new embeddings. This observation aligns with findings from the LLaMAX project [16], which demonstrated through experiments on over 100 languages that simply expanding the vocabulary does not reliably translate to better performance across diverse linguistic horizons. Perhaps future work can explore vocabulary expansion for low-resource settings in a more nuanced and systematic way.

Building on these insights, we developed the EMMA-500 model [7], training it on the MaLA corpus using a diverse mix of data including papers, code, and instructional content. We evaluated across 15 benchmarks (including translation, classification, commonsense reasoning, natural language inference, summarization, math, and code generation), and EMMA-500 model continue-trained from LLaMA 2 demonstrated improved performance, especially for low-resource languages, comparing with LLaMA-2-based CPT models and a few strong baselines, and remarkably outperformed many strong models on machine translation task. The EMMA-500 model demonstrated that a more carefully curated corpus and diverse data mix can improve performance.

In the quest of improving continual training our EMMA-500 model, we considered the inclusion of parallel data. However, it was not until late 2024 and early 2025 that we were able to collect a large-scale parallel corpus for 500+ languages with good quality (actually, the quality was still not ideal). We then released the EMMA-500 Gen 2 model series, which is continually pretrained on up to 671B tokens and shows that including parallel data can improve performance, especially for low-resource languages, and published our findings in a new preprint [11] alongside our related work on data mixing and multilingual continual pretraining [12] on a selective set of languages.

Key Milestones

After hard work in a couple of years, the project has achieved several major breakthroughs in expanding linguistic coverage:

• MaLA-500: Introduced in early 2024, MaLA-500 [6] represents a massively multilingual adaptation of large language models, demonstrating our first effort on language adaptation strategies for 534 languages.
• MaLA Corpus: A broad-coverage pretraining dataset containing clean documents spanning 500+ underrepresented languages [14].
• EMMA-500: Released in 2024 fall, EMMA-500 [7] (Enhancing Massively Multilingual Adaptation) further optimizes continual pretraining regimes using curated multilingual data and data mixes, establishing improved performance for low-resource languages.
• EMMA-500 Gen 2: Published in June 2025 [11], this work introduces a new bilingual translation corpus named MaLA translation corpus and the EMMA-500 Llama 3 suite of models. Continually pretrained on up to 671B tokens, the work shows that including parallel data enhances performance, especially for low-resource languages [15].

Reflections on the Journey

In addition to corpus and model development, the MaLA-LM team developed the GlotEval framework [13], a comprehensive evaluation suite for massively multilingual LLMs. GlotEval provides a standardized benchmark for assessing model performance across diverse languages and tasks, enabling researchers to identify specific strengths and blind spots. On a personal note, evaluating multilingual LLMs is an incredibly complex frontier, and there is simply no one-size-fits-all metric. It is a bit disheartening that I currently lack the dedicated resources and personnel to continuously maintain and expand GlotEval on my own. I am highly open to have open-source community step in, build upon this foundation, and collectively evolve it into an even more robust and dynamic tool for the field. And I hope I could find the resource to bring this effort back to active development in the future.

This need for community-driven evaluation brings us to one of the hardest realities of the field: effectively benchmarking a model across hundreds of languages is fraught with difficulties. The sheer scarcity of high-quality, culturally accurate evaluation datasets for low-resource languages, combined with immense linguistic diversity, makes it frustratingly difficult to draw definitive conclusions about how well a model actually understands a given language. It prompts a critical question: is a model that covers 500 languages truly "better" than one that perfectly masters 100 or 10?

The answer is far from straightforward. While broad linguistic coverage is a noble and necessary goal, it must be carefully balanced against the depth of comprehension and the dreaded "curse of multilinguality," where adding more languages can dilute the model's overall accuracy. Ultimately, evaluation only matters when tied to a specific context; the intended real-world use case should dictate how we measure success. A model designed for rapid, cross-lingual translation requires vastly different evaluation criteria than one intended for creative writing in an indigenous dialect.

So, is a 500-language model genuinely useful today? My answer is a resounding "no", but with a caveat: it has the potential to be. While the potential to democratize information access and facilitate global communication is immense, a model's true utility is dictated not just by the raw number of languages it covers, but by its practical quality and ability to generalize effectively. If a model supports a language but performs too poorly to be relied upon for critical tasks, its real-world value is limited. Optimizing performance for the target language is essential for actual adoption. Scaling to 500 languages remains a profound and ongoing research challenge. Moving forward, it is crucial that we maintain a clear understanding of the trade-offs involved in data collection and model design, ensuring that as we expand our linguistic horizons and preserve the linguistic heritage of underrepresented languages, we never lose sight of quality and practical accessibility.

The recognition of this work was not always immediately reflected in the peer-review process—an experience that is becoming increasingly common for many strong papers in today's era of lottery-style review. Our early drafts faced multiple rejections, often with feedback questioning the novelty of the approach. I understand that our work didn't propose new methods, and it's never the claim of our work. But I have believed that our core contributions—scaling these capabilities to hundreds of underrepresented languages in a robust, reproducible way—are deeply significant for the research community. The inherent randomness and subjectivity of modern peer review are likely to remain the norm as submission volumes grow. So trust your ideas, stay resilient, and do not let rejections deter you from important work. After several challenging rounds of review, our findings were finally accepted to COLM 2026 and ACL 2026 Findings. I am grateful to the reviewers who saw the potential in our vision.

I deeply appreciate the support of my colleagues—some of us used our free time to work on this project. MaLA-LM is a long-term commitment. I am incredibly glad it has found recognition, and I will personally continue striving to make high-quality multilingual LLMs accessible for everyone, especially for the communities speaking low-resource languages.

The Quest for Linguistic Universality

Concurrent initiatives across the broader AI ecosystem underscore the collective drive toward universal linguistic representation. Hugging Face's FineWeb2 [8] introduces a unified pipeline to adapt pretraining data processing to 1000+ language.

Similarly, the GlotCC project [9] presents a broad-coverage CommonCrawl processing pipeline tailored for minority languages, while DCAD-2000 [10] scales data cleaning up to 2,000+ languages using anomaly detection. Together, these open-source resources are rewriting the rules of multilingual AI representation.

FineOPUS

As an open-source initiative, the MaLA-LM project continues to contribute to this ecosystem by providing datasets, models, and evaluation tools that empower researchers to build more inclusive AI systems. A prime example of this ongoing work is FineOPUS. FineOPUS is a collaborative effort involving researchers from the ELLIS Institute Finland, TurkuNLP, Helsinki-NLP, and collaborative institutes, designed to act as a "digital refinery" for the massive OPUS collection of parallel texts.

While the open-source OPUS repository is a foundational resource for machine translation, its web-scraped data inherently contains noise, semantic misalignments, and language contamination. Rather than accepting this noisy data as a given, FineOPUS applies a rigorous, transparent curation pipeline. By systematically filtering and cleaning the data—with a special emphasis on preserving and enhancing low-resource languages—the project significantly elevates the reliability of parallel corpora.

FineOPUS is currently under active development. Once complete, our team is deeply committed to open science and will proudly release three key assets to the community: the curated FineOPUS Dataset, the reproducible open-source FineOPUS Pipeline, and a comprehensive Technical Report detailing our methodologies and evaluations. Through these contributions, we aim to lay a stronger, cleaner foundation for the next generation of resilient multilingual language technologies.

Join the Journey

The MaLA-LM project operates on a foundation of open collaboration and community engagement. If you are interested in contributing, analyzing dataset properties, or scaling models for underrepresented languages, we invite you to join our community Discord server:

Join our Discord Server 🤗 Hugging Face GitHub

References

1. I. Adebara, A. Elmadany, M. Abdul-Mageed, and A. A. Inciarte. 2023. SERENGETI: Massively Multilingual Language Models for Africa. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1498–1537, Toronto, Canada. Association for Computational Linguistics.
2. A. Imani, P. Lin, A. H. Kargaran, S. Severini, M. J. Sabet, N. Kassner, C. Ma, H. Schmid, A. Martins, F. Yvon, and H. Schütze. 2023. Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1082–1117, Toronto, Canada. Association for Computational Linguistics.
3. S. Kudugunta, I. R. Caswell, B. Zhang, X. Garcia, D. Xin, A. Kusupati, R. Stella, A. Bapna, and O. Firat. 2023. MADLAD-400: A Multilingual And Document-Level Large Audited Dataset. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
4. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, D. Zhou, N. Houlsby, and D. Metzler. 2023. UL2: Unifying Language Learning Paradigms. In The Eleventh International Conference on Learning Representations.
5. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
6. P. Lin, S. Ji, J. Tiedemann, A.F. Martins, and H. Schütze. 2024. MaLA-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303.
7. S. Ji, Z. Li, I. Paul, J. Paavola, P. Lin, P. Chen, D. O'Brien, H. Luo, H. Schütze, J. Tiedemann, and B. Haddow. 2024. EMMA-500: Enhancing massively multilingual adaptation of large language models. arXiv preprint arXiv:2409.17892.
8. G. Penedo, H. Kydlíček, V. Sabolčec, B. Messmer, N. Foroutan, A. H. Kargaran, C. Raffel, M. Jaggi, L. Von Werra, and T. Wolf. 2025. FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language. arXiv preprint arXiv:2506.20920.
9. A.H. Kargaran, F. Yvon, and H. Schütze. 2024. GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages. arXiv preprint arXiv:2410.23825.
10. Y. Shen, W. Lai, S. Wang, X. Zhang, K. Luo, A. Fraser, and M. Sun. 2025. DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection. arXiv preprint arXiv:2502.11546.
11. S. Ji, Z. Li, J. Paavola, H. Luo, and J. Tiedemann. 2025. Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data. arXiv preprint arXiv:2506.00469.
12. Z. Li, S. Ji, H. Luo, and J. Tiedemann. 2025. Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources. In Conference on Language Modeling (COLM).
13. H. Luo, Z. Li, J. Attieh, S. Devkota, O. de Gibert, X. Huang, S. Ji, P. Lin, B. S. P. V. Mantina, A. Sreenidhi, R. Vázquez, M. Wang, S. Yusofi, F. Yuan, and J. Tiedemann. 2025. GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 602–614.
14. S. Ji, Z. Li, J. Paavola, P. Lin, P. Chen, D. O'Brien, H. Luo, H. Schütze, J. Tiedemann, and B. Haddow. 2026. MaLA: A Corpus and Data Mix for Massive Language Adaptation of Large Language Models. In Conference on Language Modeling (COLM 2026).
15. S. Ji, Z. Li, J. Paavola, H. Luo, and J. Tiedemann. 2026. Data-Centric Continual Pre-training for 500+ Languages: A New Bilingual Translation Corpus and Multilingual Models. In Findings of the Association for Computational Linguistics: ACL 2026, pages 18776–18807, San Diego, California, United States. Association for Computational Linguistics.
16. Y. Lu, W. Zhu, L. Li, Y. Qiao, and F. Yuan. 2024. Llamax: Scaling linguistic horizons of llm by enhancing translation capabilities beyond 100 languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10748–10772.