Analysis of Semantic Bias in ChatGPT: Generating Word Definitions by Extension

TOTARO ARIEL; LAURINO JULIETA; KACZER LAURA; KAMIENKOWSKI JUAN; BIANCHI BRUNO

Buenos Aires

Congreso; Congreso de la Sociedad Argentina de Neurociencias; 2024

The encoding of the word meaning in our brain is one of the main unknowns in the study of language as a cognitive ability. In this regard, words with more than one meaning, homonyms and polysemous words (e.g., “banco”), provide us with great possibilities to further the understanding of this issue. Behaviouraly, state-of-the-art Language Models (LM) can disambiguate word meanings similar to the human brain. This similarity between humans and LM prompts the question of whether they process language in a similar manner. Therefore, understanding how these models work can help us better understand the brain. In a preliminary analysis, meaning assignment to ambiguous words in LM was studied neurally, using a corpus of biasing contexts and ambiguous sentences. This corpus consisted of 48 ambiguous words, with at least 2 meanings each. Meanings were defined as a unique word (e.g., for “banco”, “ecnonomía” and “mobiliario”). The meaning assignment was determined by comparing the distance between the embeddings (i.e., the vectorized representation of words) of the target-word and its meaning-word. Despite promising results, it was noted that the measurement of word meaning using only one meaning-word is noisy. In the present work, we aimed to define the ambiguous words’ meaning as lists of words, and to validate them through an online experiment. Increasing the precision of how we measure meaning assignment with LMs will help us better understand how the brain performs this task.