Multilingual Speech Emotion Recognition with Self-Supervised Representations and Prototypical Networks
8th International Congress on Human-Computer Interaction, Optimization and Robotic Applications, ICHORA 2026, Ankara, Türkiye, 21 - 23 Mayıs 2026, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/ichora69329.2026.11537119
- Basıldığı Şehir: Ankara
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: Affective Computing, Low-Resource Languages, Prototypical Networks, Self-Supervised Learning, Speech Emotion Recognition
- Ondokuz Mayıs Üniversitesi Adresli: Evet
Özet
It is challenging to recognize emotions from a speaker's voice, especially when people speak different languages. Most Speech Emotion Recognition (SER) systems require massive amounts of data, making them difficult to use for languages with limited resources. While modern self-supervised models can capture sound well, fine-tuning them on new languages often fails because emotions are expressed differently across cultures. To solve this, we propose a hybrid few-shot learning method that combines the intermediate representations (Layer 9) of a frozen multilingual model (XLSR-53) with Prototypical Networks. We compared traditional acoustic features (eGeMAPS) and deep-learned features against the Support Vector Machine classifier and Prototypical Networks. We tested this on a novel 5-class multilingual dataset (Russian, Turkish, Spanish, Dari) using Leave-One-Speaker-Out cross-validation. Our XLSR-53+ Prototypical Networks method achieved an accuracy of 87.35%, significantly outperforming the traditional eGeMAPS+SVM baseline $(\text{4 2. 0 0 \%) })$. Our results establish that using prototypical networks with the intermediate layers of multilingual transformers is a highly efficient way to recognize speech emotions across different languages using very little data.