Assessing and improving the prediction of dynamical systems with echo state networks

Abstract

The prediction of complex nonlinear dynamical systems with the help of machine learning techniques has become increasingly popular. In particular, the so-called echo state networks (ESN) turned out to be a very promising approach especially for the reproduction of the long-term properties of the system. While it was demonstrated that chaotic attractors of nonlinear systems can well be reproduced a thorough statistical analysis was so far missing. Here, we statistically analyze the quality of prediction of ESNs - both the exact short term prediction as well as the reproduction of the long-term properties of the system as estimated by the correlation dimension and largest Lyapunov exponent. Using the Lorenz and the Roessler system we find significant variations of the results for different realizations of random numbers. Thus, special care must be taken in selecting the good predictions: Those realizations, which deliver better short-term prediction also tend to better resemble the long-term climate of the system. The heart of ESN is a network of nodes that is fed with input data and is connected with an output layer. So far mostly random Erdös-Renyi networks are used. However, there is a variety of conceivable other network topologies that may have an influence on the results. We find that the long term prediction is worse for the scale-free network, where the differences between the network types are more pronounced in the Rössler system. Random and scale-free networks perform similar in both cases with slight advantages for the small world network. Finally, we demonstrate that a controlled removal of nodes which are associated with the smallest or largest output weights does not decrease the overall prediction quality. This result suggests that long term predictions can equally well be performed with a well-defined minimal network. Studying and interpreting these results in detail will give new insights about the essential requirements for chaotic or (more general for) complex behavior in nonlinear dynamical systems.