language: en

A Delay Damage Model Selection Algorithm for NARX Neural Networks

Tsungnan Lin 1;2y ,C. Lee Giles 1;3 , Bill G. Horne 1 , S.Y. Kung 2
1 NEC Research Institute,
4 Independence Way, Princeton, NJ 08540
2 Department of Electrical Engineering, Princeton University,
Princeton, NJ 08540
3 UMIACS, University of Maryland,
College Park, MD 20742

Abstract Recurrent neural networks have become popular models for system identification and time series prediction. NARX (Nonlinear AutoRegressive models with eXogenous inputs) neural network models are a popular subclass of recurrent networks and have been used in many applications. Though embedded memory can be found in all recurrent network models, it is particularly prominent in NARX models. We show that using intelligent memory order selection through pruning and good initial heuristics significantly improves the generalization and predictive performance of these nonlinear systems on problems as diverse as grammatical inference and time series prediction.

Keywords: Recurrent neural networks, tapped-delay lines, long-term dependencies, time series, automata, memory, temporal sequences, gradient descent training, latching, NARX networks, auto-regressive, pruning, embedding theory.

[Published in IEEE Transactions on Signal Processing, "Special Issue on Neural Networks," vol. 45, no. 11, p. 2719-2730, 1997. Copyright IEEE.]

[Current address:]

Epson Palo Alto Laboratory,
3145 Porter Drive, Suite 104, Palo Alto, CA 94304