A Unified Framework for Reservoir Computing and Extreme Learning Machines based on a Single Time-delayed Neuron

S. Ortín, M. C. Soriano, L. Pesquera, D. Brunner, D. San-Martín, I. Fischer, C. R. Mirasso, J. M. Gutiérrez
2015 Scientific Reports  
In this paper we present a unified framework for extreme learning machines and reservoir computing (echo state networks), which can be physically implemented using a single nonlinear neuron subject to delayed feedback. The reservoir is built within the delay-line, employing a number of "virtual" neurons. These virtual neurons receive random projections from the input layer containing the information to be processed. One key advantage of this approach is that it can be implemented efficiently in
more » ... hardware. We show that the reservoir computing implementation, in this case optoelectronic, is also capable to realize extreme learning machines, demonstrating the unified framework for both schemes in software as well as in hardware. A number of machine learning techniques using random nonlinear projections of inputs into a high-dimensional space have been proposed during the last decade 1,2 . In case of using recurrent networks for the projection, they are sometimes referred to as reservoirs. Outputs are typically obtained as linear combinations of the reservoir node states with weights trained from data in a supervised manner, thus strongly simplifying the training process. Two of the most popular random-projection techniques are Extreme Learning Machines (ELMs) 3 and Reservoir Computing (RC) and in particular Echo State Networks (ESNs) 4 . Both concepts have been developed independently, including different terminology and notations 5 . The ELMs were introduced as a simplification of (one-layer) feedforward neural networks, suitable for prediction and classification problems. The ESNs were inspired in recurrent neural networks, suitable for time dependent data. In this paper we propose a unified framework for random-projection machines, based on ESNs and ELMs. Although the similarities between both concepts are now being recognized 1,6 , this is the first time that ELMs and ESNs are implementated on identical hardware. This fact illustrates the strong conceptual analogies between the two approaches. We present an implementation of both ELM and ESN in the same hardware where the switching between the two concepts (ESNs and ELMs) is easily obtained by activating or deactivating one physical connection. We build on a recently proposed architecture for RC consisting of a single nonlinear neuron subject to a recurrent self-feedback loop 7-11 . An advantage of the proposed architecture is that it can be easily implemented in hardware, potentially allowing for high-speed information processing. In this particular case, we choose an optoelectronic system similar to those described in 8, 9 . We would like to highlight that our approach is not bound to this particular architecture. It can be easily extended to any hardware implementations of a single neuron with a delay feedback line, including electronic and
doi:10.1038/srep14945 pmid:26446303 pmcid:PMC4597340 fatcat:k6d3mgbwwjewbe3ikd6mr3bduy