No Sample Left Behind: Towards a Comprehensive Evaluation of Speech Emotion Recognition Systems

AGUSTÍN GRAVANO; LUCIANA FERRER; PABLO RIERA; GAUDER, LARA

Viena

Workshop; Speech, Music and Mind Workshop 2019; 2019

International Speech Communication Association

Human agreement for the task of labeling speech utterances with emotion information is usually low, especially for natural speech, where emotions could be ambiguous or subtle. For this reason, datasets of emotional speech are generally labeledby several human annotators. The common practice in speech emotion recognition (SER) literature is to summarize the multiple labels provided by the annotators for a sample into a single one by choosing the majority label. The problem with this approach is that a significant proportion of samples may not be assigned a majority label. These samples are usually ignored for system evaluation, along with any samples initially labeled by the annotators as being from emotions other than the emotions of interest for the specific dataset. This implies that the estimation of emotion recognition performance is incomplete. We do not know how the system will behave when presented with those ambiguous samples, which will certainly appear in practice. In this paper, we analyze the effects that these samples have in system performance and propose different ways to use the multiple labels available from the annotators during evaluation and to assess system performance without discarding any samples.