Effective questions in referential visual dialog

MAURICIO MAZUECOS; ALBERTO TESTONI; RAFFAELLA BERNARDI; LUCIANA BENOTTI

Workshop; ACL Workshop on Widening Natural Language Processing; 2020

An interesting challenge for situated dialogue systems is referential visual dialogue: by asking questions, the system has to identify the referent to which the user refers to. Task success is the standard metric used to evaluate these systems. However, it does not consider how effective each question is, that is how much each question contributes to the goal. We propose a new metric, that measures question effectiveness. As a preliminary study, we report the new metric for state of the art publicly available models on GuessWhat?!. Surprisingly, successful dialogues do not have a higher percentage of effective questions than failed dialogues. This suggests that a system with high task success is not necessarily one that generates good questions.