The Role of the Information Bottleneck in Representation Learning

MATÍAS ALEJANDRO VERA; PABLO PIANTANIDA; LEONARDO REY VEGA

Colorado

Simposio; IEEE International Symposium on Information Theory (ISIT); 2018

A grand challenge in representation and deep learning is the development of computational algorithms that learn the different explanatory factors of variation behind the high-dimensional data. Models are usually generated to optimize performance on training data when the real objective is to generalize for other (unseen) data. In this work we obtain an upper bound of the generalization gap and prove that, when this bound and the empirical risk function are minimized, the problem is equivalent to the Information Bottleneck. We also show a deep learning example to justify Dropout, a state of art regularization method.