On the role of effective and referring questions in GuessWhat?!

MAURICIO MAZUECOS; ALBERTO TESTONI; RAFFAELLA BERNARDI; LUCIANA BENOTTI

Workshop; ACL Workshop on Advances in Language and Vision Research; 2020

Task success is the standard metric used toevaluate referential visual dialogue systems.In this paper we propose two new metrics thatevaluate how each question contributes to thegoal. First, we measure how effective eachquestion is by evaluating whether the questiondiscards objects that are not the referent. Second, we define referring questions as thosethat univocally identify one object in the image. We report the new metrics for human dialogues and for state of the art publicly available models on GuessWhat?!. Regarding ourfirst metric, we find that successful dialoguesdo not have a higher percentage of effectivequestions for most models. With respect to thesecond metric, humans make questions at theend of the dialogue that are referring, confirming their guess before guessing. Human dialogues that use this strategy have a higher tasksuccess but models do not seem to learn it