A slighly deeper dive into minimal reservoir computing

Kurzfassung

Reservoir computers (RC) are machine learning algorithms for predicting complex, nonlinear systems. The traditional setup of RC uses random matrices to define the underlying recurrent neural network and the input layer that transforms the input data to input signals of each node in the reservoir. Thus, a large number of random weights are to be chosen via a set of optimized hyperparameters (e.g. spectral radius of the reservoir). Here, we show that a few simple modifications to the traditional RC-architecture, which eliminate randomness and minimize computational resources, lead to significant and robust improvements in short- and long-term predictive performance compared to similar models, while requiring minimal amount of training data. Specifically, we first introduce block-diagonal reservoirs, which implies that a reservoir can be composed of multiple smaller reservoirs, each with its own dynamics. Furthermore, we take out the randomness of the reservoir by using matrices of ones for the individual blocks. This breaks with the widespread interpretation of the reservoir as a single network [1]. In a further step, we also omit the nonlinear activation function in the nodes of the reservoir. The nonlinearity is only introduced by also taking higher powers of the reservoir response. Thus, this new architecture opens new avenues to explainable and interpretable reservoir computing. For certain parameterizations, we find that the predictions are accurate for more than 10 Lyapunov times and that ordinary least squares regression directly on the embedded data can predict the long-term climate of chaotic systems [2]. We benchmark our results against those obtained with normal reservoir computing, next generation reservoir computing (NG RC) and SINDy. We find better and more stable predictions for minimal reservoir computing. Finally, we discovered that for certain choices of hyperparameters the prediction fails. With only a few parameters, the phase transition between various parameterizations can be analyzed to comprehend the reasons behind the success of a prediction. We find that a crucial parameter is the size of the boxes in the block-diagonal reservoir [3]. Detailed analyses of the reconstructed, underlying equations give further insights into structural prerequisites for successful predictions. [1] H. Ma et al., Chaos, 33, 063130 (2023) [2] H. Ma et al., Sci. Rep., 13, 12970 (2023) [3] H. Ma, PhD thesis, LMU (2024)