Mixed and Virtual Reality Play a Key Role in Pilot Training and Simulation Solutions of the Future

Substantial changes are afoot in the world of aviation. Even before the pandemic, both the aviation industry and government organizations were under increasing cost and operational tempo pressures. The pandemic exacerbated the challenges further. While the world is rapidly returning to a new normal, the challenges facing aviation have not gone away.

Studies suggest that by 2029, there will be a shortage of 60,000 pilots globally. This will cause massive challenges unless measures are taken to grow the qualified pilot ranks. With civilian airlines recruiting heavily from the military, defense organizations face growing pilot shortages which could affect national defense.

For aviation to flourish, pilot training must become drastically more productive, scalable and cost-effective. As a result, the aviation training community is looking to emerging virtual and mixed reality technologies to enable this shift in less than a decade to meet the growing demand.

Head-Mounted Displays Are Gaining Traction

Virtual simulation has long been successfully used to train both military and civilian pilots. Traditional simulators typically use large, collimated displays and domes with arrays of projectors to present the virtual environment. Where these displays are extremely realistic, they are expensive to acquire and operate, difficult to maintain, and require trainees to travel to a fixed location as they cannot be moved.

The prospect of using head-mounted displays (HMDs) to eliminate the need for collimated displays and domes has long been envisioned but has not come to fruition due to deficiencies in this technology. Previously, HMDs have lacked the resolution, field of view, and low latency needed to produce adequate image quality, causing cue mismatch that often leads to simulation or cyber sickness.

Recent developments in HMD technology now show a major shift is underway. For example, the US Navy is completely reworking their flight training curriculum to leverage virtual and mixed reality devices to support basic pilot training. On the civil aviation side, the European Union Aviation Safety Agency (EASA) has, for the first time, officially qualified a virtual reality based training solution for pilot training. Developed and built by VRM Switzerland, the qualified helicopter pilot simulator uses Varjo’s latest human-eye resolution virtual reality headsets.

With VRM Switzerland’s training solution utilizing Varjo headsets, pilots can now have VR flight simulation time credited to their training. Image credit: VRM Switzerland

The capabilities, credibility, and efficacy of high-end virtual reality training solutions have begun to increase massively – and we have only scratched the surface of the capabilities of extended reality simulation technologies.

Where Does Extended Reality Training Fit in the Pilot Training Curriculum?

The main benefits of using HMDs include cost efficiencies; improved utilization of existing full-mission simulators; greater portability of devices to enable training at the point of need while saving travel costs; increased availability of training tools that empower trainees to achieve more reps and sets; more engaging training that encourages task repetition until mastery is achieved, and all-around better scalability.

Today’s pilot training curriculum is a well understood progression of coursework delivered via online and/or instructor-led training, typically with some simulator training progressing to flights in a real aircraft. Follow-on recertification is usually performed in high-fidelity simulators. Mixed reality (XR) devices should be thought of as high-context training tools which can be inserted into the curriculum to supplement and, in some cases, replace the existing coursework and exercises. The goal is to maximize student engagement while presenting material in a context and setting that closely simulates the true operational environment.

Special purpose, low-cost XR-based devices can be built to support a wide variety of core flight training tasks, including:

XR-based devices enable students to immediately grasp the point of each lesson by creating a training context that mirrors a real aircraft where cause and effect relationships are clearly demonstrated. And, because the devices are low cost compared to traditional simulators, they can be made available to students as needed to learn at their own pace. When training organizations successfully implement these solutions, they will train more pilots faster with greater training flexibility and at a lower cost than ever before.

Not All HMDs Are Equally Suited to the Task

For headset-based simulators to be highly effective, the headset needs to fulfill demanding criteria. These include eye tracking for trainee assessment; sufficient visual quality, resolution and immersion; elimination of simulator sickness; the capability for extended use; and, in the case of mixed reality, sufficiently low latency. Consumer-grade headsets simply do not deliver the visual clarity and other capabilities required in these demanding training scenarios.

Professional-grade headsets, such as Varjo XR-3 and VR-3, have made massive strides in addressing simulation fatigue and cyber sickness issues. With features such as high-quality ergonomics, rapid refresh rates, exceptional visual quality and resolution, automatic interpupillary-distance (IPD) adjustment, and low-latency video pass-through, these headsets can be used for hours on end without adverse effects.

A critical component in the accurate measurement of training effectiveness is eye tracking. This enables a trainer to effectively follow and evaluate the trainee's gaze while in the simulation. Varjo headsets feature industry-leading eye tracking at up to 200 Hz and sub-degree accuracy, which enables capture of even the most minute movements of the trainees' eyes, such as saccade velocities and rapid glances. The trainer can observe on a separate screen exactly where the trainee looks and accurately personalize training further if behavioral changes are required. The data is also collected so that it can be analyzed later or even combined with data from other sensors, such as heart rate monitoring.

Varjo XR-3. Image credit: Varjo

Learn More About the Future of Pilot Training Solutions

In our recently published whitepaper, we explore the use cases and benefits of virtual and mixed reality across the pilot training and simulation value chain. You will hear from world-class professionals and organizations who are already pioneering the new reality of immersive training. You will also discover several benefits to using virtual and mixed reality (collectively known as XR) as a part of the training process compared to only using traditional tools and curricula.

Download the Varjo whitepaper to discover why these technologies are not flights of fancy but the future of pilot training that industry pioneers are already adopting.