Browse Prior Art Database

FAST RANDOM TRAINING METHOD FOR RECURRENT BACKPROPAGATION NEURAL NETWORKS

IP.com Disclosure Number: IPCOM000007108D
Original Publication Date: 1994-Feb-01
Included in the Prior Art Database: 2002-Feb-26
Document File: 2 page(s) / 131K

Publishing Venue

Motorola

Related People

Orhan Karaali: AUTHOR

Abstract

The following techniques make it possible to train recurrent backpropagation networks in. random mode. They also speed up the training of recurrent neural nets in a random mode by as much as a factor of two. Random mode training of recurrent nets is achieved by keeping all the past neural network out- put states in the recurrent buffer. Networks trained in random mode converge to lower error rates in comparison to sequential mode trained networks.

This text was extracted from a PDF file.
At least one non-text object (such as an image or picture) has been suppressed.
This is the abbreviated version, containing approximately 50% of the total text.

Page 1 of 2

0 M MO-LA

Technical Developments Volume 21 February 1994

FAST RANDOM TRAINING METHOD FOR RECURRENT BACKPROPAGATION NEURAL NETWORKS

by Orhan Karaali

  The following techniques make it possible to train recurrent backpropagation networks in. random mode. They also speed up the training of recurrent neural nets in a random mode by as much as a factor of two. Random mode training of recurrent nets is achieved by keeping all the past neural network out- put states in the recurrent buffer. Networks trained in random mode converge to lower error rates in comparison to sequential mode trained networks.

  Preservation of trends in the recurrent buffer by updating the recurrent buffer only during sequen- tial training results in much faster random training. To update the recurrent buffer, extra sequential pas- ses are inserted between the random training pas- ses. The recurrent buffer is not updated during ran- dom training passes.

able to train recurrent nets in random mode in our simulator, we implemented a data structure called the "recurrent buffer? The recurrent buffer is used to store all the past output states of the neural net- work as it goes through the data file and has the same dimensions as the target vector buffer.

  Because the adjacent locations of the recurrent buffer are updated in a random fashion and the network characteristics change incrementally over time, the values stored in the adjacent locations of the recurrent buffer are unpredictable in regards to each other. Since only some of the n old locations (states) used by the network are updated in the ran- dom training, the trends present in the past output states are lost.

  As the network converges the output of the net- work and the values in the recurrent buffer start becoming more stable. The lack of relationship (trends) between the sequential buffer locations dur- ing the early training reduces the information pres- ent in the recurrent buffer and slows down the train- ing. In time series prediction the difference of past values carries information as does the magnitude of the values. In the random training of recurrent net with recurrent buffer, then information present in mag- nitude differences ofthe past values is partially lost.

I. INTRODUCTION

II. PROBLEM DESCRlPTlON

  Backpropagation neural networks are used to pre- dict the values of time series data. Commonly, neu- ral networks are trained in one of two modes: sequen- tial and random. In sequential training, all the data' vectors in the training data file are presented to the network in a sequential fashion. In the random mode, the training vectors are chosen from the training tile randomly. In general, random training improves

the network performance.

  Adding recurrent connection paths from the out- put stage of the network to the input stage of the network results in better accuracy and more stable

output values. One or more past output states can be routed back into the...