Filters








268 Hits in 4.6 sec

Deep, Skinny Neural Networks are not Universal Approximators [article]

Jesse Johnson
2018 arXiv   pre-print
In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate.  ...  This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions.  ...  The main result of this paper states that deep, skinny neural networks can only approximate functions with unbounded level components.  ... 
arXiv:1810.00393v1 fatcat:vyl4banb4jc5hojcq75vw2zrty

Topological Deep Learning: Classification Neural Networks [article]

Mustafa Hajij, Kyle Istvan
2021 arXiv   pre-print
Using this topological framework, we show when the classification problem is possible or not possible in the context of neural networks.  ...  Finally, we demonstrate how our topological setting immediately illuminates aspects of this problem that are not as readily apparent using traditional tools.  ...  This is also related to the work [17] which shows that skinny neural networks are not universal approximators.  ... 
arXiv:2102.08354v1 fatcat:5hw2yfasl5debpxhkqlhmylggu

Loss Surface Modality of Feed-Forward Neural Network Architectures

Anna Sergeevna Bosman, Andries Petrus Engelbrecht, Marde Helbig
2020 2020 International Joint Conference on Neural Networks (IJCNN)  
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm.  ...  This study employs fitness landscape analysis to study the modality of neural network loss surfaces under various feed-forward architecture settings.  ...  This observation correlates with [11] , where deep and "skinny" NNs were shown to not exhibit the universal approximator properties.  ... 
doi:10.1109/ijcnn48605.2020.9206727 dblp:conf/ijcnn/BosmanEH20 fatcat:sp7hpquuizcclieckzlsjgtfiu

Loss Surface Modality of Feed-Forward Neural Network Architectures [article]

Anna Sergeevna Bosman, Andries Engelbrecht, Mardé Helbig
2020 arXiv   pre-print
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm.  ...  This study employs fitness landscape analysis to study the modality of neural network loss surfaces under various feed-forward architecture settings.  ...  This observation correlates with [11] , where deep and "skinny" NNs were shown to not exhibit the universal approximator properties.  ... 
arXiv:1905.10268v2 fatcat:i6uvistfovfcxhw57j6ua4gfkm

Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization [article]

Zachary Susskind, Bryce Arden, Lizy K. John, Patrick Stockton, Eugene B. John
2021 arXiv   pre-print
However, the neural aspect of computation dominates the symbolic part in cases where they are clearly separable.  ...  Due to the recency of the field's emergence and relative sparsity of published results, the performance characteristics of these models are not well understood.  ...  INTRODUCTION Conventional neural networks, based on Deep Learning (DL), have proven to be effective in solving problems in many domains.  ... 
arXiv:2109.06133v1 fatcat:mnoc4hv26rdk5mi4pls4hysqyu

Efficient Modeling of Morphing Wing Flight Using Neural Networks and Cubature Rules [article]

Paul Ghanem, Yunus Bicer, Deniz Erdogmus, Alireza Ramezani
2021 arXiv   pre-print
Fluidic locomotion of flapping Micro Aerial Vehicles (MAVs) can be very complex, particularly when the rules from insect flight dynamics (fast flapping dynamics and light wings) are not applicable.  ...  In other applications related to robot locomotion, often neural networks are trained around a local attractor with no possibility of updating neural networks weights in an online fashion.  ...  Note that the neural network weights are the state vector and the vehicle configuration x is the input in Eq. 10.  ... 
arXiv:2110.01057v1 fatcat:uiudc5pxqnduvpgdwxszggiiiy

A multiscale neural network based on hierarchical matrices [article]

Yuwei Fan and Lin Lin and Lexing Ying and Leonardo Zepeda-Nunez
2019 arXiv   pre-print
This network generalizes the latter to the nonlinear case by introducing a local deep neural network at each spatial scale.  ...  In this work we introduce a new multiscale artificial neural network based on the structure of H-matrices.  ...  In addition, we note that deep neural networks with related multi-scale structures [53, 50] have been proposed mainly for applications such as image processing, however, we are not aware of any applications  ... 
arXiv:1807.01883v4 fatcat:rchufifaajamtkwaxec3dkbsci

Merging Two Cultures: Deep and Statistical Learning [article]

Anindya Bhadra, Jyotishka Datta, Nick Polson, Vadim Sokolov, Jianeng Xu
2021 arXiv   pre-print
We clarify the duality between shallow and wide models such as PCA, PPR, RRR and deep but skinny architectures such as autoencoders, MLPs, CNN, and LSTM.  ...  The connection with data transformations is of practical importance for finding good network architectures.  ...  Deep Learners are based on superposition of univariate affine functions [Polson and Sokolov, 2017] and are universal approximators.  ... 
arXiv:2110.11561v1 fatcat:twkrsiyrora7bnhiu54uw5v2zm

A Topological Framework for Deep Learning [article]

Mustafa Hajij, Kyle Istvan
2021 arXiv   pre-print
To demonstrate these results, we provide example datasets and show how they are acted upon by neural nets from this topological perspective.  ...  Finally, we show how the architecture of a neural network cannot be chosen independently from the shape of the underlying data.  ...  This is also related to the work [13] which shows that skinny neural networks are not universal approximators.  ... 
arXiv:2008.13697v13 fatcat:yp4hxaq23jbhxh553nj7v2655e

Learning to Detect [article]

Neev Samuel, Tzvi Diskin, Ami Wiesel
2018 arXiv   pre-print
In this paper we consider Multiple-Input-Multiple-Output (MIMO) detection using deep neural networks.  ...  We introduce two different deep architectures: a standard fully connected multi-layer network, and a Detection Network (DetNet) which is specifically designed for the task.  ...  In addition, we thank Amir Globerson and Yoav Wald for their ideas and help with the soft output networks.  ... 
arXiv:1805.07631v1 fatcat:xl2j5p7z2ffsvlr2265clzbqay

Transparency order versus confusion coefficient: a case study of NIST lightweight cryptography S-Boxes

Huizhong Li, Guang Yang, Jingdian Ming, Yongbin Zhou, Chengbin Jin
2021 Cybersecurity  
Unfortunately, all of them are invalid in this scenario.  ...  In order to measure side-channel resistance of S-Boxes, three theoretical metrics are proposed and they are reVisited transparency order (VTO), confusion coefficients variance (CCV), and minimum confusion  ...  Acknowledgements Not applicable.  ... 
doi:10.1186/s42400-021-00099-1 fatcat:ulv3gpzjxfgrjlhrxa764n5n5q

Politicians-based Deep Learning Models for Detecting News, Authors and Media Political Ideology

Khudran M. Alzhrani
2022 International Journal of Advanced Computer Science and Applications  
Constructing deep neural network models based on politicians' personalization improved the performance of political ideology detection models.  ...  Also, deep networks models could predict news articles' politician personalization with a high F1 score.  ...  ACKNOWLEDGMENT The authors would like to thank the Deanship of Scientific Research at Umm Al-Qura University for supporting this work by Grant Code: (22UQU4340018DSR01).  ... 
doi:10.14569/ijacsa.2022.0130286 fatcat:iysgh44awvcwjmmyx326lurotq

One of a Kind

Xue Geng, Hanwang Zhang, Zheng Song, Yang Yang, Huanbo Luan, Tat-Seng Chua
2014 Proceedings of the ACM International Conference on Multimedia - MM '14  
In particular, we propose a new deep learning strategy called multi-task convolutional neural network (mtCNN) to learn profile models and profile-related visual features simultaneously.  ...  , and b) content-centric social network.  ...  Implementation Details The underlying deep architecture we adopted in Section 3.2 is the deep convolutional neural network architecture proposed by Krizhevsky et al. [13] .  ... 
doi:10.1145/2647868.2654950 dblp:conf/mm/GengZSYLC14 fatcat:drvzd5jsojdp7jqzbrmcfdqxgu

Git Re-Basin: Merging Models modulo Permutation Symmetries [article]

Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa
2022 arXiv   pre-print
We argue that neural network loss landscapes contain (nearly) a single basin, after accounting for all possible permutation symmetries of hidden units.  ...  Despite non-convex optimization being NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice.  ...  Therefore we conjecture that the permutation symmetry hypothesis is a necessary piece, though not a complete picture, of whatever fundamental invariances are at play in neural network training dynamics  ... 
arXiv:2209.04836v1 fatcat:fwbixwmc2vd3feq6s5nfzrlibe

Deep Rewiring: Training very sparse deep networks [article]

Guillaume Bellec, David Kappel, Wolfgang Maass, Robert Legenstein
2018 arXiv   pre-print
We present an algorithm, DEEP R, that enables us to train directly a sparsely connected neural network.  ...  DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded.  ...  Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05423, 2016.  ... 
arXiv:1711.05136v5 fatcat:dkyhr7g6hna3ve7o7bagat2nva
« Previous Showing results 1 — 15 out of 268 results