22 Hits in 6.1 sec

Backpropagation in the simply typed lambda-calculus with linear negation

Aloïs Brunel, Damiano Mazza, Michele Pagani
2019 Proceedings of the ACM on Programming Languages (PACMPL)  
In this paper, we extend the backpropagation algorithm to a paradigmatic example of such a programming language: we define a compositional program transformation from the simply-typed lambda-calculus to  ...  itself augmented with a notion of linear negation, and prove that this computes the gradient of the source program with the same efficiency as first-order backpropagation.  ...  on exactly what is required to express the backpropagation algorithm in the λ-calculus, avoiding a full linear logic typing assignment and just tracking the linearity of a single variable of type R in  ... 
doi:10.1145/3371132 fatcat:aecr5el4cjgbjbso3o2x6buipa

Dual-Numbers Reverse AD, Efficiently [article]

Tom Smeding, Matthijs Vákár
2022 arXiv   pre-print
language with resource-linear types, such as Haskell.  ...  idea: by pairing each scalar value with a backpropagator function.  ...  on the lambda calculus, it is tricky to make efficient.  ... 
arXiv:2205.11368v1 fatcat:e4ix6vxk7jeypgcax3zl6pnjsu

The simple essence of automatic differentiation [article]

Conal Elliott
2018 arXiv   pre-print
They are inherently parallel-friendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an AD-agnostic  ...  In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation.  ...  b) for real values has the expected type: R × R R. 12 To make the linearity more apparent, and to prepare for variations later in this paper, let's now rephrase D mulC without using lambda directly.  ... 
arXiv:1804.00746v4 fatcat:h4do54mnjffxxje4vhkhrg4ecm

A Differential-form Pullback Programming Language for Higher-order Reverse-mode Automatic Differentiation [article]

Carol Mak, Luke Ong
2020 arXiv   pre-print
We justify our reduction strategy by interpreting our language in any differential λ-category that satisfies the Hahn-Banach Separation Theorem, and show that the reduction strategy precisely captures  ...  Building on the observation that reverse-mode automatic differentiation (AD) – a generalisation of backpropagation – can naturally be expressed as pullbacks of differential 1-forms, we design a simple  ...  [10] study a simply-typed λ-calculus augmented with a notion of linear negation type. Though our dual type may resemble their linear negation, they are actually quite different.  ... 
arXiv:2002.08241v1 fatcat:6ygchctcwnbktczd66izoop6ym

Semantic Compositionality through Recursive Matrix-Vector Spaces

Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng
2012 Conference on Empirical Methods in Natural Language Processing  
This matrix-vector RNN can learn the meaning of operators in propositional logic and natural language.  ...  Our model assigns a vector and a matrix to every node in a parse tree: the vector captures the inherent meaning of the constituent, while the matrix captures how it changes the meaning of neighboring words  ...  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the view of DARPA, AFRL, or the US government.  ... 
dblp:conf/emnlp/SocherHMN12 fatcat:ecjt2eamhfb3fbnjttxumge6me

Change actions: from incremental computation to discrete derivatives [article]

Mario Alvarez-Picallo
2020 arXiv   pre-print
of Ehrhard and Regnier's differential lambda-calculus.  ...  Second, to establish and clarify the connection between derivatives in the incremental sense and derivatives in the analytic sense, that is to say, to provide a common definition of derivative of which  ...  In what follows we will consider a fixed difference λ-category C, and proceed to define interpretations for the types, contexts and terms of the simply-typed λ ε -calculus.  ... 
arXiv:2002.05256v2 fatcat:o6zzymbz5rcgbgamuqve2fkyou

Reverse AD at Higher Types: Pure, Principled and Denotationally Correct

Matthijs Vákár
2021 arXiv   pre-print
We give a semantic proof of correctness of the transformations. In their most elegant formulation, the transformations generate code with linear types.  ...  To do so, we make use of abstract data types to represent the required linear types, e.g. through the use of a basic module system.  ...  [9] defines an AD algorithm in a simply typed λ-calculus with linear negation (essentially, the continuation-based AD of [21]) and proves it correct using operational techniques.  ... 
arXiv:2007.05283v2 fatcat:x4xqca5us5gzzd3e6e6ov6npwu

Differentiation in logical form

Abbas Edalat, Mehrdad Maleki
2017 2017 32nd Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)  
Approximable mappings of the latter type form a bounded complete domain isomorphic with the function space of Scott continuous functions of a real variable into the domain of non-empty compact and convex  ...  Finally, we derive a calculus for Lipschitzian derivative of approximable mapping for some basic constructors and show that it is dual to the calculus satisfied by the Clarke subgradient.  ...  In [14] , Ehrhard and Regnier have introduced the differential lambda calculus which syntactically models the derivative operation on power series in a typed lambda calculus or a full linear logic.  ... 
doi:10.1109/lics.2017.8005143 dblp:conf/lics/EdalatM17 fatcat:d7ngtohvqrgr7pjhc3n5tfpq7a

Categorical semantics of a simple differential programming language

Geoffrey Cruttwell, Jonathan Gallagher, Dorette Pronk
2021 Electronic Proceedings in Theoretical Computer Science  
With the increased interest in machine learning, and deep learning in particular, the use of automatic differentiation has become more wide-spread in computation.  ...  There have been two recent developments to provide the theoretical support for this types of structure. One approach, due to Abadi and Plotkin, provides a simple differential programming language.  ...  While Smooth is the canonical example, there are many others, including examples from infinite dimensional vector spaces, synthetic differential geometry, algebraic geometry, differential lambda calculus  ... 
doi:10.4204/eptcs.333.20 fatcat:chxoorwi6vgfxpfjmu6tu52zq4

Physics-based Deep Learning [article]

Nils Thuerey and Philipp Holl and Maximilian Mueller and Patrick Schnell and Felix Trost and Kiwon Um
2022 arXiv   pre-print
As much as possible, all topics come with hands-on code examples in the form of Jupyter notebooks to quickly get started.  ...  This digital book contains a practical and comprehensive introduction of everything related to deep learning in the context of physical simulations.  ...  Fundamental theorem of calculus To more clearly illustrate the advantages in non-linear settings, we apply the fundamental theorem of calculus to rewrite the ratio ∆𝑥 PG /∆𝑦 from above.  ... 
arXiv:2109.05237v3 fatcat:pz7ot63dlbdkriihkwloefk3im

Adversarial Attacks on Face Detectors Using Neural Net Based Constrained Optimization

Avishek Joey Bose, Parham Aarabi
2018 2018 IEEE 20th International Workshop on Multimedia Signal Processing (MMSP)  
Adversarial Attacks on Face Detectors using Neural Net based Constrained Optimization Adversarial attacks involve adding, small, often imperceptible, perturbations to inputs with the goal of getting a  ...  Unlike in many attack strategies we show that the same trained generator is capable of attacking new images without explicitly optimizing on them.  ...  The most common type of neural network architecture is the feedforward which is simply a collection of layers.  ... 
doi:10.1109/mmsp.2018.8547128 dblp:conf/mmsp/BoseA18 fatcat:lochow2euzbyvg2l3eonjgo43u

Grammatical Assimilation [chapter]

Ted Briscoe
2003 Language Evolution  
Acknowledgements I am grateful to the editors for providing helpful feedback, both from them and from their students on the first draft, which helped me improve this one.  ...  The details of how this mapping is actually realized formally are not important to the argument, but either a rule-to-rule semantics based on the typed lambda calculus or a unification-based analogue would  ...  Yamauchi, however, simply does not report whether decorrelation affects the ability of the evolving population to match phenotypes via learning.  ... 
doi:10.1093/acprof:oso/9780199244843.003.0016 fatcat:npvxotskwnbprkla5zlo2xumvi

Artificial intelligence and molecular biology

1994 ChoiceReviews  
The depict slot is not illustrated in the frame in Figure 2 since this function is constant for all images.  ...  ., for providing DLB with support for a sabbatical during which this work was initiated.  ...  Nodes and arcs are the typed and weighted data objects. Typed nodes in our graph represent the twenty amino acid types found in protein sequences.  ... 
doi:10.5860/choice.31-3826 fatcat:5jmryy7mkbcrvifz6ahlbbdceu

Applications of Deep Neural Networks with Keras [article]

Jeff Heaton
2022 arXiv   pre-print
Deep learning allows a neural network to learn hierarchies of information in a way that is like the function of the human brain.  ...  Focus is primarily upon the application of deep learning to problems, with some introduction to mathematical foundations.  ...  Momentum backpropagation has two training parameters: learning rate (η, eta) and momentum (λ, lambda).  ... 
arXiv:2009.05673v5 fatcat:h3jghqylwrbfvfglmwutlfpmay

Sentiment Analysis [chapter]

2017 Encyclopedia of Machine Learning and Data Mining  
with the backpropagation algorithm as usual.  ...  Representation languages range from more expressive, such as first-order logic and lambda calculus, to more simple database query languages designed specifically for a small domain, like Geoquery (Zelle  ...  The term should not be confused with a planning problem such as BOXWORLD or BLOCKSWORLD.).  ... 
doi:10.1007/978-1-4899-7687-1_100512 fatcat:ce4yyqo2czftzcx2kbauglh3fu
« Previous Showing results 1 — 15 out of 22 results