language: en

Generative Models for Discovering Sparse Distributed Representations

Geoffrey E. Hinton and Zoubin Ghahramani
Department of Computer Science University of Toronto
Toronto, Ontario, M5S 1A4, Canada
hinton@cs.toronto.edu, zoubin@cs.toronto.edu May 9, 1997

[A modified version to appear in]

[Philosophical Transactions of the Royal Society B, 1997.]

Abstract We describe a hierarchical, generative model that can be viewed as a non-linear generalization of factor analysis and can be implemented in a neural network. The model uses bottom-up, top-down and lateral connections to perform Bayesian perceptual inference correctly. Once perceptual inference has been performed the connection strengths can be updated using a very simple learning rule that only requires locally available information. We demon strate that the network learns to extract sparse, distributed, hierarchical representations.

1 Introduction