Using LSTM-based Language Models and human Eye Movements metrics to understand next-word predictions

UMFURER, ALFREDO ; KAMIENKOWSKI, JUAN E.; BIANCHI, BRUNO

Buenos Aires

Congreso; Jornada Argentina de Informática; 2022

SADIO

Modern Natural Language Processing (NLP) models canachieve great results resolving different types of linguistic tasks. Thisis possible thanks to a high volume of internal parametersthat are optimized during the training phase. They allow to model high-level linguisticproperties. For example, LSTM-based language models have the abilityto find long-term dependencies between words on a text, and use them tomake predictions about upcoming words. Nevertheless, their complexitymakes it hard to understand which features they use to generate predictions.The neurolinguistic field faces a similar issue when studying how ourbrain processes language. For example, every adult reader has the abilityto understand long texts and to make predictions of upcoming words.Nevertheless, our understanding on how these predictions are driven islimited. During the last decades, the study of eye movements duringreading have shed some light on this topic, finding a relation betweenthe time spent on a word (gaze duration) and its processing cost.Here, we aim to understand how LSTM-based models predict futurewords and these predictions relate with human predictions, fitting statistical models commonly used in the neurolinguistic field with gaze duration as the dependent variable. We found that an AWD-LSTM LanguageModel can partially model eye movements, with high overlap with bothhuman-Predictability and lexical frequency. Interestingly, this last overlap is seen to depend on the training corpus, being lower when the modelis fine-tuned with a corpus similar to the one used for testing.