elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Constructing Novel Recurrent Neural Network Architectures Using Hamiltonian Dynamics

Rusch, Konstantin (2019) Constructing Novel Recurrent Neural Network Architectures Using Hamiltonian Dynamics. Master's, The University of Edinburgh.

[img] PDF - Registered users only
1MB

Abstract

Recurrent neural networks (RNNs) have gained a great deal of attention in solving sequential learning problems. The learning of long-term dependencies, however, is still an open problem caused by the vanishing or exploding gradient. This problem corresponds to the loss of information in the sequential learning data. The goal is therefore to construct RNNs which can preserve the information at least for very long times. In this thesis we will provide the connection between RNNs and Hamiltonian dynamics which is the natural tool in theoretical physics when it comes to preservation characteristics. We will construct an RNN based on a specific Hamiltonian system and call it Hamiltonian recurrent neural network (HRNN). We will derive a sensitivity analysis for the HRNN and show that we can directly control the hidden states gradient which is causing the vanishing or exploding gradient problem. We demonstrate the superiority of the HRNN compared to other state-of-the-art RNNs by providing several benchmark experiments. Finally, we present a real-world application for the HRNN by forecasting the future trends of a stock based on historical financial data.

Item URL in elib:https://elib.dlr.de/132366/
Document Type:Thesis (Master's)
Additional Information:Die Arbeit als Volltext soll noch nicht öffentlich zugänglich gemacht werden, da gerade ein Paper eingereicht wurde
Title:Constructing Novel Recurrent Neural Network Architectures Using Hamiltonian Dynamics
Authors:
AuthorsInstitution or Email of AuthorsAuthors ORCID iD
Rusch, KonstantinKonstantin.Rusch (at) dlr.deUNSPECIFIED
Date:August 2019
Refereed publication:No
Open Access:No
Gold Open Access:No
In SCOPUS:No
In ISI Web of Science:No
Number of Pages:42
Status:Published
Keywords:Machine learning, recurrent neural networks, vanishing gradients, dynamical systems, time series
Institution:The University of Edinburgh
Department:The School of Mathematics
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Space Technology
DLR - Research area:Raumfahrt
DLR - Program:R SY - Technik für Raumfahrtsysteme
DLR - Research theme (Project):R - Vorhaben SISTEC
Location: Köln-Porz
Institutes and Institutions:Institut of Simulation and Software Technology
Deposited By: Siggel, Dr. Martin
Deposited On:10 Dec 2019 11:12
Last Modified:10 Dec 2019 11:12

Repository Staff Only: item control page

Browse
Search
Help & Contact
Information
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.