Research Project Title:
Long-Memory Recurrent Models through Regularization
abstract:Recurrent Neural Nets (RNNs) are increasingly popular tools for a variety of machine learning applications. However, past research has exposed numerous drawbacks to setting up and training RNNs using the standard back-propagation algorithm using gradient descent. In particular, the exploding and vanishing gradient problems make RNNs hard to train. It has been hypothesized that because of these problems RNNs do not work better in practice than much simpler feed-forward neural networks, and that feed-forward networks are sufficient for modelling tasks. We plan to outline a method by which a RNN can be regularized in order to mitigate the exploding and vanishing gradient problems. We further plan to empirically show that these RNNs can solve benchmark problems that cannot be solved by a feed-forward network of comparable size. and that these RNNs are able to maintain longer-term memory than un-regularized models.