Integration and use of an augmented reality display in a maritime helicopter simulator

Abstract

Operating a helicopter in offshore wind parks with degraded visual environments from clouds or fog can endanger crew and material due to the presence of unseen obstacles. Utilizing on-board sensors such as LIDAR or radar, one can sense and record obstacles that could be potentially dangerous. One major challenge is to display the resulting raw sensor data in a way that the crew, especially the pilot, can make use of it without distracting them from their actual task. Augmented reality and mixed reality applications play an important role here. By displaying the data in a see-through helmet-mounted display (HMD), the pilot can be made aware of obstacles that are currently obscured by degraded visual conditions or even parts of the helicopter. This can be accomplished in one HMD. No attention sharing between the outside view and a head-down instrument is necessary. The German Aerospace Center (DLR) is continuously aiming at testing and evaluating both flight-proof and consumer grade HMDs. One particular widely known system is the Microsoft HoloLens. DLR will integrate this low-cost HMD into their experimental helicopter. For this, as a first step, a Microsoft HoloLens was integrated into DLR's Air Vehicle Simulator (AVES). The integration process is detailed. The simulation capabilities are described, especially for conformal, open-loop LIDAR sensor data. Furthermore, first concepts of the display format are shown, and strengths and drawbacks of the HoloLens in a cockpit environment are discussed.