Predicting when a programming language learner needs help via neural code snapshots at word-level

MARCO MORESI; MARCOS GÓMEZ; LUCIANA BENOTTI

Conferencia; Latin American Meeting In Artificial Intelligence (Khipu); 2019

In this paper we propose to model how fluent a beginner programmer is in a new programming language combining word embeddings and long-short term memory (LSTM) recurrent neural networks. We base our model on previous work predicting how fluent people are when learning a second natural language. In particular, given a piece of code by the programmer we predict whether she/he will be able to successfully solve a given programming exercise without help. Previous work has used abstract syntax trees for this same task. However, since we deal with beginner programmers, almost 40% of their code contains parsing errors. Hence a complete abstract syntax tree cannot be built. Our model obtains a .84 F1 score for the task at hand on a dataset of 240 thousand fragments of code produced by 4 thousand students. On the same dataset a baseline built over pedagogical theories using feature engineering on the students and the programming exercises achieves a .70 F1 score.