Self-stabilizing Plasticity Rules derived from the Stationarity Principle of Statistical Learning

ECHEVESTE, RODRIGO

Otro; ELSC Retreat; 2016

Objective functions provide a useful framework for the formulation of guiding principles in dynamical systems. In the case of information processing systems, such as neural networks, these guiding principles can be formulated in terms of information theoretical measures with respect to the input and output probability distributions. The stationarity principle requires that, for a stationary input distribution, once learning of the relevant input features has been completed, the output probability distribution should also be stationary. Using the concept of the Fisher Information, we formulate this principle[1], and derive synaptic plasticity rules from it. The resulting learning rules are both Hebbian and self-limiting, avoiding unbounded weight growth. Furthermore, we show that a neuron operating under this learning rules, has a preference for non-Gaussian input directions, making it a suitable tool for independent component analysis (ICA).As an application, we study the non-linear bars problem[2], in which neurons are presented with a set of images, each a non-linear superposition of horizontal and vertical bars. We show that, with these rules, the neurons are able to learn single bars or points (the independent components of the input), even when these are never presented in isolation.