Exploring the impact of noise on quantum DDPG in portfolio allocation

Abstract

In the era of Noisy Intermediate-Scale Quantum devices, Variational Quantum Circuits in Quantum Machine Learning are gaining attention, advancing towards practical quantum computing applications on NISQ devices. Reinforcement Learning (RL), known for its humanlike, trial-and-error learning, excels in adapting to changing financial environments. In particular, classical algorithms such as Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) show promise, while emerging quantum neural networks offer potential for improved function approximation, better generalization capabilities and reduced parameters. In light of these advancements, we introduce a quantum-enhanced version of the DDPG agent (QDDPG), aiming to leverage these quantum capabilities for more efficient financial decisionmaking processes. The quantum part of the model utilizes a parameterized quantum policy, which consists of a sequential reuploading variational circuit with trainable input scaling parameters. Our objective is to explore the practicality and potential benefits of QRL in finance, aiming to realize viable quantum computing applications on NISQ devices. The QDDPG model surpasses traditional analytic strategies in portfolio allocation, achieving higher returns with fewer parameters and a favourable Sharpe ratio. In a simple configuration using a RAW-parameterized quantum policy, the model achieves an average Sharpe ratio of 1.20 across 15 trained agents, compared to 0.96 for the minimum-variance strategy and 0.81 for the buy-and-hold baseline. Furthermore we investigate the impact of quantum noise, a significant challenge for NISQ devices. We analyze various types of noise, such as general depolarization, amplitude damping, phase damping, measurement noise, and shot noise, which uniquely affect quantum computations. Our study quantifies how these noise types influence the performance, reliability, and robustness of the QRL model. We study new techniques to harness quantum noise as hyperparameter to control the exploration-exploitation balance of the agent, using noise as regularization method and to stabilize the training of the agents. This comprehensive analysis aims to not only highlight the challenges posed by quantum noise but also to explore innovative methods to mitigate these effects and enhance the applicability of quantum algorithms in real-world applications.