A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf
.
DizzyRNN: Reparameterizing Recurrent Neural Networks for Norm-Preserving Backpropagation
[article]
2016
arXiv
pre-print
The vanishing and exploding gradient problems are well-studied obstacles that make it difficult for recurrent neural networks to learn long-term time dependencies. We propose a reparameterization of standard recurrent neural networks to update linear transformations in a provably norm-preserving way through Givens rotations. Additionally, we use the absolute value function as an element-wise non-linearity to preserve the norm of backpropagated signals over the entire network. We show that this
arXiv:1612.04035v1
fatcat:istwzreyhjhovkaoxsh66etf3a