No Sample Left Behind: Towards a Comprehensive Evaluation of Speech Emotion Recognition Systems

PABLO RIERA; LUCIANA FERRER; AGUSTIN GRAVANO; LARA GAUDER

Vienna

Workshop; Workshop on Speech, Music and Mind 2019; 2019

ISCA

Human agreement for the task of labeling speech utterances with emotion information is usually low, especially for natural speech, where emotions could be ambiguous or subtle. For this reason, datasets of emotional speech are generally labeled by several human annotators. The common practice in speech emotion recognition (SER) literature is to summarize the multiple labels provided by the annotators for a sample into a single one by choosing the majority label. The problem with this approach is that a significant proportion of samples may not be assigned a majority label. These samples are usually ignored for system evaluation, along with any samples initially labeled by the annotators as being from emotions other than the emotions of interest for the specific dataset. This implies that the estimation of emotion recognition performance is incomplete. We do not know how the system will behave when presented with those ambiguous samples, which will certainly appear inpractice. In this paper, we analyze the effects that these samples have in system performance and propose different ways to use the multiple labels available from the annotators during evaluation and to assess system performance without discarding any samples.