language: en

What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation

Steve Lawrence 1;2 , C. Lee Giles 1 , Ah Chung Tsoi 2
flawrence,actg@elec.uq.edu.au, giles@research.nj.nec.com
1 NEC Research Institute,
4 Independence Way, Princeton, NJ 08540
2 Department of Electrical and Computer Engineering University of Queensland,
St. Lucia 4072, Australia
Technical Report UMIACS-TR-96-22 and CS-TR-3617
Institute for Advanced Computer Studies University of Maryland
College Park, MD 20742
June 1996

(Revised August 1996)

Abstract One of the most important aspects of any machine learning paradigm is how it scales according to problem size and complexity. Using a task with known optimal training error, and a pre-specified maximum number of training updates, we investigate the convergence of the backpropagation algorithm with respect to a) the complexity of the required function approximation, b) the size of the network in relation to the size required for an optimal solution, and c) the degree of noise in the training data. In general, for a) the solution found is worse when the function to be approximated is more complex, for b) oversized networks can result in lower training and generalization error in certain cases, and for c) the use of committee or ensemble techniques can be more beneficial as the level of noise in the training data is increased. For the experiments we performed, we do not obtain the optimal solution in any case. We further support the observation that larger networks can produce better training and generalization error using a face recognition example where a network with many more parameters than training points generalizes better than smaller networks.

Keywords: Local Minima, Generalization, Committees, Ensembles, Convergence, Backpropagation, Smoothness, Network Size, Problem Complexity, Function Approximation, Curse of Dimensionality.

[Also with the]

Institute for Advanced Computer Studies, University of Maryland,
College Park, MD 20742.