A learning algorithm with compression-based regularization

MATÍAS VERA; LEONARDO REY VEGA; PABLO PIANTANIDA

Calgary

Conferencia; IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); 2018

IEEE

This paper investigates, from information theoretic principles,a learning problem based on the principle that any regularityin a given dataset can be exploited to extract compact features from data, in order to build meaningful representationsof a relevant content. We begin by introducing the fundamental tradeoff between the average risk and the model complexity. Interestingly, our formulation allows an information theoretic formulation of the multi-task learning (MTL) problem.Then, we present an iterative algorithm for computing the optimal tradeoffs. Remarkably, empirical results illustrate thatthere exists an optimal information rate minimizing the excessrisk which depends on the nature and the amount of availabletraining data. An application to hierarchical text categorization is also investigated, extending previous works.