Demographically-Informed Prediction Discrepancy Index (DIPDI): Early Warnings for Biases in Unlabeled Populations

LUCAS MANSILLA; ESTANISLAO CLAUCICH; RODRIGO ECHEVESTE; DIEGO H MILONE; ENZO FERRANTE

Montevideo

Conferencia; Latin American Meeting In Artificial Intelligence, Khipu 2023; 2023

UDELAR

An ever-growing body of work has shown that machine learning systems can be systematically biased against certain sub-populations. Data imbalance and under-representation in the training datasets have been identified as potential causes behind this phenomenon. However, understanding whether data imbalance may result in biases for a given task and model class is not simple. A typical approach to answering this question is to perform counterfactual experiments in a controlled scenario, where several models are trained with different imbalance ratios and then evaluated on the target population. However, in the absence of ground-truth annotations at deployment for a new target population, most fairness metrics cannot be computed. In this work, we explore an alternative method based on the output discrepancy of pools of models trained on different demographic groups. Our hypothesis is that the output consistency between models may serve as a proxy to anticipate biases. We formulate the Demographically-Informed Prediction Discrepancy Index (DIPDI) and validate our hypothesis using both synthetic and real-world datasets. Our work sheds light on the relationship between model output discrepancy and demographic biases, and provides a means to anticipate fairness issues in the absence of ground-truth annotations.