Backprop as Functor: A compositional perspective on supervised learning [article]

Brendan Fong, David I. Spivak, Rémy Tuyéras
2019 arXiv   pre-print
A supervised learning algorithm searches over a set of functions A → B parametrised by a space P to find the best approximation to some ideal function f A → B. It does this by taking examples (a,f(a)) ∈ A× B, and updating the parameter according to some rule. We define a category where these update rules may be composed, and show that gradient descent---with respect to a fixed step size and an error function satisfying a certain property---defines a monoidal functor from a category of
more » ... ed functions to this category of update rules. This provides a structural perspective on backpropagation, as well as a broad generalisation of neural networks.
arXiv:1711.10455v3 fatcat:m2xi2jtrgrfo7adviem23zopxq