Understanding the effects of language-specific class imbalance in multilingual fine-tuning
Informazioni aggiuntive
Autori
Jung V.,
van der Plas L.
Tipo
Contributo in atti di convegno
Anno
2024
Lingua
Inglese
Sommario
"We study the effect of one type of imbalance often present in real-life multilingual classification datasets: an uneven distribution of labels across languages. We show evidence that fine-tuning a transformer-based Large Language Model (LLM) on a dataset with this imbalance leads to worse performance, a more pronounced separation of languages in the latent space, and the promotion of uninformative features. We modify the traditional class weighing approach to imbalance by calculating class weights separately for each language and show that this helps mitigate those detrimental effects. These results create awareness of the negative effects of language-specific class imbalance in multilingual fine-tuning and the way in which the model learns to rely on the separation of languages to perform the task.
Titolo atti di convegno
Proceedings of Findings of the Association for Computational Linguistics
Number ( Month )
March
Editore
"Association for Computational Linguistics"
Nome convegno
EACL 2024
Luogo convegno
St. Julian's, Malta
Data convegno
March 2024
Pagine (o numero dell’articolo)
"2368-2376"
