1D convolutional neural networks and applications: A survey

Serkan Kiranyaz, Onur Avci, Osama Abdeljaber, Turker Ince, Moncef Gabbouj, Daniel J. Inman
2021 Mechanical systems and signal processing  
j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l oc a t e / y m s s p processing the electrical signal based on the three individual operations [1, 2] : 1) reception of the other neurons outputs through the synaptic connections in Dendrites, 2) the integration (or pooling) of the processed output signals in the soma at the nucleus of the cell, and, 3) the activation of the final signal at the first part of the Axon or the so-called Axon hillock: if the pooled potentials
more » ... a certain limit, it "activates" a series of pulses (action potentials). As shown in Fig. 1(b) , each terminal button is connected to other neurons across a small gap called a synapse. During the 1940s the first "artificial neuron" model was proposed by McCulloch-Pitts [3], which has thereafter been used in various feed-forward ANNs such as Multi-Layer Perceptrons (MLPs). As expressed in Eq. (1), in this popular model the artificial neuron performs a linear transformation through a weighted summation by the scalar weights. So, the basic operations performed in a biological neuron, that operate the individual synaptic connections with specific neurochemical operations and the integration in the cell's soma are modeled as the linear transformation (linear weighted sum) followed by a possibly nonlinear thresholding function, f(.), which is called activation function. The concept of "Perceptron", was proposed by Frank Rosenblatt in his seminal work [4] . When used in all neurons of a MLP, this linear model is a basic model of the biological neurons leading to well-known variations in learning and generalization performances for various problems [4] [5] [6] [7] [8] . In the literature, there have been some attempts to change MLPs by modifying the neuron model and/or the conventional Back Propagation (BP) algorithm [9] [10] [11] , or the MLP configuration [12] [13] [14] or even the way to update the network parameters (weights and biases) [15] . The most promising variant is called Generalized Operational Perceptrons [7, 8] , which is a heterogeneous network with non-linear operators and has thus exhibited significantly superior performance than MLPs; however, this is still the most common network model that has inspired the modern-age ANNs that are being used today. Starting from the 1959, Hubel and Wiesel have established the foundations of the visual neuroscience through the study of the visual cortical system of cats. Their collaboration has lasted more than 25 years during which they have described the major responsive properties of the visual cortical neurons, the concept of receptive field, the functional properties of the visual cortex and the role of the visual experience in shaping the cortical architecture, in a series of articles published in The Journal of Physiology [16] [17] [18] [19] [20] . They are the pioneers who found the hierarchical processing mechanism of information in the visual cortical pathway, which eventually led to the Nobel Prize in Physiology or Medicine in 1981. With these advances in neurocognitive science, Fukushima and Miyake [21] in 1982 proposed the predecessor of Convolutional Neural Networks (CNNs), at the time called as "Neocognitron" which is a self-organized, hierarchical network and has the capability to recognize stimulus patterns based on the differences in their appearances (e.g., shapes). This was the first network, which has the unique ability of a biological mammalian visual system, that is, the assessment of similar objects to be assigned to the same object category independent from their position and certain morphological variations. However, in an attempt to maximize the learning performance, the crucial need of a supervised method to train (or adapt) the network for the learning task in hand became imminent. The ground-breaking invention of the Back-Propagation (BP) by Rumelhart and Hinton in 1986 [22] became a major cornerstone of the Machine Learning (ML) era. BP incrementally optimizes the network parameters, i.e., weights and biases, in an iterative manner using the gradient descent optimization technique. These two accomplishments have started a new wave of approaches that eventually created the first naïve CNN models but it was the seminal work of Yann LeCun in 1990 who formulated the BP to train the first CNN [23], the so-called "LeNet". This CNN ancestor became mature in 1998 and its superior classification power was demonstrated in [24] over the benchmark MNIST handwritten number database [25] . This success has begun the era of CNNs and brought a new hope to otherwise "idle" world of ML during the 1980s and early 90s. CNNs have been used in many applications during the 90s and the first decade of the 21st century but soon they fell out of fashion especially with the emergence of new generation ML paradigms such as Support Vector Machines (SVMs) and Bayesian Networks (BNs). There are two main reasons for this. First, small or medium size databases were insufficient to train a deep CNN with a superior generalization capability. Then of Nucleus Axon Dendrites Soma Terminal Buttons Axon Terminal Button Dendrites Neurotransmitters
doi:10.1016/j.ymssp.2020.107398 fatcat:2jnawwvaanbmhci2roibiyqmlm