Neural networks and rational functions [article]

Matus Telgarsky
2017 arXiv   pre-print
Neural networks and rational functions efficiently approximate each other. In more detail, it is shown here that for any ReLU network, there exists a rational function of degree O(polylog(1/ϵ)) which is ϵ-close, and similarly for any rational function there exists a ReLU network of size O(polylog(1/ϵ)) which is ϵ-close. By contrast, polynomials need degree Ω(poly(1/ϵ)) to approximate even a single ReLU. When converting a ReLU network to a rational function as above, the hidden constants depend
more » ... xponentially on the number of layers, which is shown to be tight; in other words, a compositional representation can be beneficial even for rational functions.
arXiv:1706.03301v1 fatcat:sinwakabengojg5h6alxggu7uq