Filters








299 Hits in 11.4 sec

Pre-Wiring and Pre-Training: What Does a Neural Network Need to Learn Truly General Identity Rules?

Raquel G. Alhama, Willem Zuidema
2018 The Journal of Artificial Intelligence Research  
We implement such techniques in an Echo State Network (ESN), and we show that only when combining both techniques the ESN is able to learn truly general identity rules.  ...  We argue that, in order to simulate human-like learning of grammatical rules, a neural network model should not be used as a tabula rasa, but rather, the initial wiring of the neural connections and the  ...  We are grateful to Dieuwke Hupkes and Michael Repplinger for their comments on a previous version of this paper.  ... 
doi:10.1613/jair.1.11197 fatcat:uze2q46bwvgypdq3yuteol72by

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization [article]

Jingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. Liu
2020 arXiv   pre-print
In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary.  ...  In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective.  ...  , and Samy Bengio for reviewing the manuscript.  ... 
arXiv:1912.08777v3 fatcat:sniaydtda5a35pgyugitvcjtpy

Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [article]

Alexander Ororbia, Ankur Mali, Daniel Kifer, C. Lee Giles
2020 arXiv   pre-print
In this paper, we propose a gradient-free learning procedure, recursive local representation alignment, for training large-scale neural architectures.  ...  Experiments with residual networks on CIFAR-10 and the large benchmark, ImageNet, show that our algorithm generalizes as well as backprop while converging sooner due to weight updates that are parallelizable  ...  Earlier approaches that employed local learning included the layer-wise training procedures that were once used to pre-train networks [57, 10, 29, 45 ].  ... 
arXiv:2002.03911v3 fatcat:2k326rdnnjhutfmthzqrvrsxui

Robust-fit to nature: an evolutionary perspective on biological (and artificial) neural networks [article]

Uri Hasson, Samuel A. Nastase, Ariel Goldstein
2019 bioRxiv   pre-print
Does the mammalian brain use similar brute-force fitting processes to learn how to perceive and act upon the world?  ...  Evolution is a blind fitting process by which organisms, over generations, adapt to the niches of an ever-changing environment.  ...  Our affordances are constrained by our bodies and brains, and there is an intimate relationship between how our bodies and neural networks are wired and what we can learn.  ... 
doi:10.1101/764258 fatcat:r2f7qzq5aneppfirr6bho76m6y

Building machines that learn and think like people

Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman
2016 Behavioral and Brain Sciences  
We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn and how they learn it  ...  in intuitive theories of physics and psychology to support and enrich the knowledge that is learned; and (3) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge  ...  Tom Schaul was very helpful in answering questions regarding the DQN learning curves and Frostbite scoring.  ... 
doi:10.1017/s0140525x16001837 pmid:27881212 fatcat:3fjriprksbhaxpqdcydrhmcjqm

Deep Convolution Neural Networks in Computer Vision: a Review

Hyeon-Joong Yoo
2015 IEIE Transactions on Smart Processing and Computing  
Since there exist various deep learning techniques, this review paper is focusing on techniques directly related to DCNNs, especially those needed to understand the architecture and techniques employed  ...  Among them, GoogLeNet network which is a radically redesigned DCNN based on the Hebbian principle and scale invariance set the new state of the art for classification and detection in the ILSVRC 2014.  ...  Dongsun Park at Chonbuk National University, Korea, greatly helped to improve an earlier version of this manuscript.  ... 
doi:10.5573/ieiespc.2015.4.1.035 fatcat:catvathw5vdppbudzrsixbwtl4

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

András Szilágyi, István Zachar, Anna Fedor, Harold P. de Vladar, Eörs Szathmáry
2016 F1000Research  
transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors. : Attractor dynamics of recurrent neural networks can be  ...  Since all the suggested components are present in advanced nervous systems, we hypothesize that the brain could implement a truly evolutionary combinatorial search system, capable of generating novel variants  ...  Acknowledgements We are thankful to Luc Steels, Chrisantha Fernando, Mauro Santos, and Thomas Filk for useful comments and discussions.  ... 
doi:10.12688/f1000research.9630.1 pmid:27990266 pmcid:PMC5130073 fatcat:vdwexjdssnfhjl26mwu7vnisvu

Building Machines That Learn and Think Like People [article]

Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman
2016 arXiv   pre-print
We review progress in cognitive science suggesting that truly human-like learning and thinking machines will have to reach beyond current engineering trends in both what they learn, and how they learn  ...  in intuitive theories of physics and psychology, to support and enrich the knowledge that is learned; and (c) harness compositionality and learning-to-learn to rapidly acquire and generalize knowledge  ...  Tom Schaul was very helpful in answering questions regarding the DQN learning curves and Frostbite scoring.  ... 
arXiv:1604.00289v3 fatcat:ph2rrwk2znb4dpb5nvcg54x2xi

Toward an Integration of Deep Learning and Neuroscience

Adam H. Marblestone, Greg Wayne, Konrad P. Kording
2016 Frontiers in Computational Neuroscience  
Such a heterogeneously optimized system, enabled by a series of interacting cost functions, serves to make learning data-efficient and precisely targeted to the needs of the organism.  ...  In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and relatively  ...  AUTHOR CONTRIBUTION All authors contributed ideas and co-wrote the paper. ACKNOWLEDGMENTS We thank Ken Hayworth for key discussions that led to this paper. We thank Ed Boyden, Chris Eliasmith, Gary  ... 
doi:10.3389/fncom.2016.00094 pmid:27683554 pmcid:PMC5021692 fatcat:yikwc4h5yvfj7gwzlimtw5n6ai

Towards an integration of deep learning and neuroscience [article]

Adam Henry Marblestone, Greg Wayne, Konrad P Kording
2016 bioRxiv   pre-print
In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and relatively  ...  Such a heterogeneously optimized system, enabled by a series of interacting cost functions, serves to make learning data-efficient and precisely targeted to the needs of the organism.  ...  Acknowledgments We thank Ken Hayworth for key discussions that led to this paper.  ... 
doi:10.1101/058545 fatcat:4ryejpe2tnf7dgoaqhoastoiya

Breeding novel solutions in the brain: a model of Darwinian neurodynamics

András Szilágyi, István Zachar, Anna Fedor, Harold P. de Vladar, Eörs Szathmáry
2017 F1000Research  
Churchill AW, Sigtia S, Fernando C: Learning to generate genotypes with neural networks. Evolutionary Computation. 2015. Reference Source 66.  ...  transmission of candidate solutions as messages between networks, and, (iii) spontaneously generated, untrained patterns in spurious attractors. : Attractor dynamics of recurrent neural networks can be  ...  Acknowledgements We are thankful to Luc Steels, Chrisantha Fernando, Mauro Santos, and Thomas Filk for useful comments and discussions.  ... 
doi:10.12688/f1000research.9630.2 pmid:27990266 pmcid:PMC5130073 fatcat:527hajvilnft5lfxtijia3dcba

The Virtual-Environment-Foraging Task enables rapid training and single-trial metrics of rule acquisition and reversal in head-fixed mice

Martha N. Havenith, Peter M. Zijderveld, Sabrina van Heukelum, Shaghayegh Abghari, Paul Tiesinga, Jeffrey C. Glennon
2019 Scientific Reports  
Based on these metrics, we show that mice can predict new task rules long before they are able to execute them, and that this delay varies across animals.  ...  By generating multiple non-binary performance metrics per trial, it provides single-trial estimates not only of response accuracy and speed, but also of underlying processes like choice certainty and alertness  ...  We would also like to thank the reviewers of this manuscript for their helpful and insightful suggestions.  ... 
doi:10.1038/s41598-019-41250-w pmid:30886236 pmcid:PMC6423024 fatcat:ippczklitvf6lmtwu4n3slqx7q

Visually Grounded Models of Spoken Language: A Survey of Datasets, Architectures and Evaluation Techniques

Grzegorz Chrupała
2022 The Journal of Artificial Intelligence Research  
Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and  ...  The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas.  ...  Acknowledgements This survey has been greatly improved thanks to suggestions, corrections and other feedback from David Harwath, William Havard, Lieke Gelderloos, Bertrand Higy, Afra Alishahi, and two  ... 
doi:10.1613/jair.1.12967 fatcat:zib2mr5wkjdmteyrgac6gxekli

Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques [article]

Grzegorz Chrupała
2021 arXiv   pre-print
Several fields have made important contributions to this approach to modeling or mimicking the process of learning language: Machine Learning, Natural Language and Speech Processing, Computer Vision and  ...  The current paper brings together these contributions in order to provide a useful introduction and overview for practitioners in all these areas.  ...  Acknowledgements This survey has been greatly improved thanks to suggestions, corrections and other feedback from David Harwath, William Havard, Lieke Gelderloos, Bertrand Higy, Afra Alishahi, and two  ... 
arXiv:2104.13225v3 fatcat:edodewkhljbqtpcrm2knd2zw7i

Train-by-Reconnect: Decoupling Locations of Weights from their Values [article]

Yushi Qiu, Reiji Suda
2020 arXiv   pre-print
What makes untrained deep neural networks (DNNs) different from the trained performant ones?  ...  to that of a well-trained fully initialized network; when the initial weights share a single value, our method finds weight agnostic neural network with far better-than-chance accuracy.  ...  In addtion, we would like to thank Vorapong Suppakitpaisarn, Farley Oliveira, and Khoa Tran for helpful comments on a preliminary version of this paper.  ... 
arXiv:2003.02570v6 fatcat:pcuo4egw3rewjguhcjim3oefsu
« Previous Showing results 1 — 15 out of 299 results