Exploration of stochastic quasi-Newton optimization for physics-informed neural networks

Abstract

Partial differential equations play a central role in modeling physical systems. However, classical numerical methods such as finite elements or finite volumes are limited due to their computational complexity. Recently, machine learning methods and in particular physics-informed neural networks have emerged as promising alternatives, embedding physical laws directly into the training process. Training physics-informed neural networks requires solving a challenging optimization problem. While first-order optimizers such as stochastic gradient descent are standard in deep learning, second-order methods like quasi-Newton algorithms can potentially offer faster convergence and improved accuracy. However, deterministic quasi-Newton methods are not directly suitable for stochastic optimization, which is typically used in neural network training. This motivates the investigation of stochastic quasi-Newton methods as potential optimizers for physics-informed neural networks. This thesis investigates a stochastic quasi-Newton method for training physics-informed neural networks using three benchmark problems. These investigations study the accuracy as well as the training time and memory costs of this method, evaluating the general applicability of advanced second-order optimization algorithms for physics-informed neural networks.