Approximating Continuous Functions by ReLU Nets of Minimal Width [article]

Boris Hanin, Mark Sellke
2018 arXiv   pre-print
This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed d_in≥ 1, what is the minimal width w so that neural nets with ReLU activations, input dimension d_in, hidden layer widths at most w, and arbitrary depth can approximate any continuous, real-valued function of d_in variables arbitrarily well? It turns out that this minimal width is exactly equal to d_in+1. That is, if all the
more » ... n layer widths are bounded by d_in, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions, and, on the other hand, any continuous function on the d_in-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly d_in+1. Our construction in fact shows that any continuous function f:[0,1]^d_in→ R^d_out can be approximated by a net of width d_in+d_out. We obtain quantitative depth estimates for such an approximation in terms of the modulus of continuity of f.
arXiv:1710.11278v2 fatcat:aoluxiwi5jgzzc7w5fv42jpqj4