Speech Emotion Recognition with Limited Data: A Layer-Wise Analysis on a Novel Low-Resource Multilingual Corpus
28th International Conference on Digital Signal Processing and its Applications, DSPA 2026, Moscow, Rusya, 25 - 27 Mart 2026, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/dspa69176.2026.11476732
- Basıldığı Şehir: Moscow
- Basıldığı Ülke: Rusya
- Anahtar Kelimeler: Layer-Wise Analysis, Low-Resource Languages, Self-Supervised Learning, Speech Emotion Recognition
- Ondokuz Mayıs Üniversitesi Adresli: Evet
Özet
Speech emotion recognition (SER) plays a very significant role in creating natural human-computer interfaces (HCI). Recent progress in self-supervised learning (SSL) has contributed to the broader use of pre-trained models instead of traditional spectral or acoustic features in SER. However, there is still limited research on which layers of pre-trained models are optimal for emotion recognition in cross-lingual settings. Another significant challenge is the lack or total absence of emotional speech datasets for underrepresented languages. In this paper, we present a layer-by-layer analysis of three state-of-the-art self-supervised learning models: XLSR-53, HuBERT, and WavLM, tested on a balanced multilingual corpus (Russian, Spanish, Turkish, Dari) created for this study. Contrary to the common practice of using the output of the last layer, we show that the optimal features for emotion recognition are located in the middle layers (layers 7-15). The XLSR-53 model achieved the highest accuracy of 74.7%, proving that extensive pre-training on different languages is crucial for capturing universal prosodic patterns. We also found that although XLSR-53 performs well in the intermediate layers, its accuracy falls sharply to 39.8% in the last layers, indicating a shift in the model's focus from paralinguistics to linguistics. The WavLM model showed more robust results across all layers due to its noise reduction pre-training task.