Modeling systems with internal state using evolino

Daan Wierstra, Faustino J. Gomez, Jürgen Schmidhuber
2005 Proceedings of the 2005 conference on Genetic and evolutionary computation - GECCO '05  
Existing Recurrent Neural Networks (RNNs) are limited in their ability to model dynamical systems with nonlinearities and hidden internal states. Here we use our general framework for sequence learning, EVOlution of recurrent systems with LINear Outputs (Evolino), to discover good RNN hidden node weights through evolution, while using linear regression to compute an optimal linear mapping from hidden state to output. Using the Long Short-Term Memory RNN Architecture, Evolino outperforms
more » ... outperforms previous state-of-the-art methods on several tasks: 1) context-sensitive languages, 2) multiple superimposed sine waves.
doi:10.1145/1068009.1068315 dblp:conf/gecco/WierstraGS05 fatcat:n5ddrvs7mncobnf4bd5z56mwlq