Enhanced Pilot Training Via VR

The application of virtual reality technology for flight training is gaining momentum, and training providers are developing new tools that may provide more effective training. Chuck Weirauch investigates.

With several game-based, off-the-shelf flight simulator providers already featuring virtual reality (VR) headset capabilities, and Microsoft stating that it is planning to do so for its new Flight Simulator 2020 to be released this year, the market potential for the technology has already taken off. Commercial flight training device providers have already begun to develop flight simulators that employ VR HMDs, but, like the game-based PC flight sims, these devices in most part do not require the high level of fidelity that is required of a full-flight simulator, and employ a lower resolution HMD. But the main problem with the lower-resolution (and often lower-cost) VR headsets is that they display a screen-door effect that can lead to visual disorientation.

Still in all, the purpose of employing VR technology for flight training is that some research indicates it could drastically reduce the cost of that training. With that in mind, several flight schools have integrated PC flight sims into their curricula.

UND: Teaching Basics

The University of North Dakota’s John D. Odegard School of Aerospace Science has gone one step beyond by recently opening its new Virtual Reality Lab, where students can view a 360-degree full-cockpit view of a simulated Piper Archer training aircraft through their HTC Vive VR headsets. Everything viewed in the headset is realistic, since the photorealistic images they are seeing via the HUD are from actual aircraft simulators located in the School’s Flight Operations Center. The students also have yolk, throttle and rudder pedal devices at their training stations.

According to the VR Lab’s creator, UND multimedia specialist Neil Nowatzki, the lab setup is primarily for new ab initio students who are just beginning their flight courses at UND. The VR training is being used to learn basic stick and rudder skills, as well as developing and improving students’ situational awareness.

“We’re focusing on teaching the basics of how to fly an airplane,” Nowatzki said. “It’s ‘eyes outside’ in the lab. Students should be able to look out the window and, just by where the nose is, they should know whether they are level or not — are they climbing or descending?”

As well as providing basic piloting skills while not taking up simulator time in training devices that students will be flying later in their curriculum, UND intends to use the lab, which has mobile stations, as a part of its outreach efforts to encourage more people to pursue a commercial pilot career.

UND Associate Dean of Aerospace Beth Bjerke hopes that further development of VR technology will provide a relatively low-cost training capability that will reduce the amount of training now required in the actual aircraft, as well as lowering the costs of overall aviation operations. Bjerke also said that UND wants to partner with other institutions to convince the FAA to allow more use of VR technology for flight training.

“Human Eye Resolution”

One of the newest VR HMDs that was designed to solve the resolution problem is the Varjo VR2 Pro VR headset, which features what its Finnish developer labels “human-eye resolution,” along with an integrated 20/20 eye tracker and integrated hand-tracking by Ultraleap. The VR2 features a pixel-per-degree (PPI) count of 60, which is where it gets its claim of “human eye resolution,” or in other terms, providing 20/20 visual acuity for the user by displaying pixels so small that they are invisible to parts of the human eye.

According to Varjo Marketing and Communications Manager Annaleena Kuronen, VR2 Pro supports Prepar3D and a recent Bohemia Interactive helicopter flight simulation.

Varjo’s XR1 Developer Edition pass-through mixed reality solution has the capability for providing photorealistic virtual scenes with real-world visibility, Kuronen added. This means that training product developers can use physical cockpits in high-resolution virtual scenarios in order to allow the seamless mixing of synthetic content with reality, she explained. Integrating a Vital image generator with the XR1, FlightSafety International has demonstrated a mixed-reality flight trainer that provides a simulated photorealistic out-the-window and instrumentation view within an actual aircraft cockpit.

Military Embracing VR

The value of employing VR for flight training has been demonstrated by the US Air Force via its SAIC-supported Pilot Training Next program, which has successfully employed PC-based VR and mixed-reality flight trainers to help reduce the amount of time that it takes to put its pilots through initial training.

The CAE Sprint Virtual Reality (VR) trainer is an integral part of the CAE TRAX Academy that delivers self-paced training in a high-fidelity virtual environment Image credit: CAE.

CAE has responded with its Sprint Virtual Reality Trainer and its CAE TRAX Academy curriculum. The CAE variant employs the same software as the T6 Texan II full-flight simulator in the Sprint (Self-Paced Real-time Insights for Next-generation Training). The Canadian training services integrator also added a joystick that simulates G-forces plus haptics to better provide better sensory feedback to mimic aircraft performance cues to the student pilots. It also incorporated the Varjo VR2 HMD, which Bohemia Interactive and FSI are employing as well.

The TRAX Academy courseware allows student pilots to perform at their own pace and features mobile capabilities. CAE hopes to provide its new, advanced product to the US Air Force and other international military services.

Airbus: Addressing Demand

In December 2019, Airbus announced its newest and perhaps most significant products in the XR technologies category, the Airbus Virtual Flight Trainer, which puts pilots into a virtual cockpit of an Airbus A320. This trainer could be the vanguard of such VR-based flight trainers, since it is to be used in Airbus training centers starting in the second half of this year, “with the aim to make it an integral part of the Airbus type rating curriculum."

Other missives indicate that eventually such trainers will be based on other Airbus aircraft types. The goal of the move toward a VR-based trainer is to help meet the demand of providing adequate training for the increasing number of pilots that will be needed to crew Airbus-predicted number of commercial aircraft in the post-Covid future.

“We estimate there will be 37,400 aircraft joining the global fleet,” Captain Shane Carroll, designer of Airbus’ VR flight training solution, pointed out. “That roughly means a doubling of the existing fleet and of the training demand, so there is a clear need to address this. In addition, there is a new generation of pilots coming through that won’t accept a textbook, chalk and a blackboard as sources of their training. VR essentially offers us opportunities to address the training need in that we can get pilots in a cockpit earlier and more often, as it is more accessible than a full-flight simulator.”

Rather than replacing FFS training, the VR flight simulator will complement the type of procedures training that is normally done in a full-motion sim. The VR simulator consists of consumer off-the shelf (COTS) equipment, including a VR headset, personal computer, a pair of hand controllers and non-proprietary software. The VR HUD provides pilots an immersive, virtual cockpit environment where they can practice procedures individually, or as members of a crew, and can do so remotely with or without an instructor.

Simulator Aboard?

What if students and commercial pilots could train for difficult maneuvers and procedures while in flight, with a flight simulator capability actually built into the aircraft? While this was not the original goal behind the Fused Reality Small Business Innovation Research (SBIR) contract between NASA Armstrong Flight Research Center, Systems Technology Inc. of Hawthorne, California, the US Air Force and the US Naval Air Command, it is one of the potential applications, according to Dave Klyde, Vice President and Technical Director for STI’s Engineering Services. He described Fused Reality as a blending of XR technologies, with a VR HMD as the primary tool.

The SBIR was initially designed and conducted to evaluate aircraft handling capabilities. NASA test pilots wearing an early version of a VR headset equipped with a camera were able to practice simulated landings on an out-the-window virtual runway they viewed through the VR HUD while at a safe aircraft altitude, as well as formation flying with other aircraft, aerial refueling and other challenging aerial maneuvers.

STI recently was recently awarded a contract by what Klyde described as a “commercial aviation company” to further develop the Fused Reality technology.

While Klyde emphasized that the product is now only being further refined to evaluate aircraft performance, he envisions pilot training applications as well. One such application would be for general aviation pilots, who while wearing the Fusion Reality HUD, could practice virtual “touch-and-go” landings on a virtual runway while at altitude, he said. The aircraft instruments can also be viewed via the HMD, which features hand-tracking technology as well. The redesigned and updated Fused Reality system is now dubbed Fused Reality Flight.

Klyde announced Fused Reality Flight at the 2020 American Institute of Aeronautics and Astronautics (AIAA) SciTech Forum held in Orlando this January. He described it as providing “an in-flight experience that allows for virtual objects, including other aircraft, to be placed out-of-the-cockpit-window as part of combined virtual and real-world video scene that is viewed through a head-mounted display.” The revised system features an Oculus Rift HMD.

HMD for HUD

Along with other training device providers, Collins Aerospace is also looking toward VR and other XR technologies to help reduce the costs of flight training and make it more effective. The company is the world’s leading provider of head-up guidance systems for aircraft. Its most recent XR-based trainer is the Head-Up Display (HUD) virtual reality training device, which is to be provided to the FAA for human factors research on advanced cockpit technology. The trainer currently also employs the Oculus Rift HMD.

According to Marc Cronan, Collins Aerospace Senior Manager for Commercial Programs, this flight training device is not just for the FAA but is being evaluated by airlines that feature the company’s HUD systems in their aircraft.

“We are talking to a number of airlines right now that are using our HUD systems,” Cronan reported. “They are looking to better understand how it would fit into their training program. But one initial conversation is with a large airline that has a number of aircraft operating with our HUD systems. They feel that it takes anywhere from 300 to 350 hours before crews are really comfortable using the HUD system and flying with it. They see this device as a means to reduce the amount of time it takes to train their crews in adapting to the use of the HUD in the airplane. You can train far more cycles with it without using the full-flight simulator.”

Collins is also considering the expansion of the Virtual Reality HUD trainer for aircraft types such as the Airbus A320 and Boeing 737 MAX, where the company’s HUDs have a presence in the cockpit. He sees that as another expansion of the firm’s efforts to increase its focus on applying VR and XR technologies to flight training. The next step being considered is adding cockpit head-down displays to the virtual environment viewed through the HUD Trainer’s headset. Collins also has developed a flat panel trainer-like procedures trainer that employs a VR headset.

Along with the potential training time-and-cost savings being associated with the application of VR and XR technologies for training, Cronan considers one of the biggest benefits of doing so is providing a more immersive environment for the student. This latter factor is another reason why Collins is working to expand applications of the technologies.

“You can use the HUD VR trainer without the virtual environment and get a 2D representation of the cockpit,” Cronan said. “But when you put the headset on, you are suddenly transported to the cockpit itself. It almost shocks the system and creates such an immersive environment. It is tracking your head movement, so wherever you look, it’s like you are actually sitting in the cockpit.”

An XR Future

While the XR technologies arena is still much in its infancy in terms of its application to training, the company is working to find its way as far as developing how best to incorporate this new tech.

“We have a number of XR activities around the business unit, including Coalescence that is orientated toward warfighter training applications,” Cronan described. “But in some cases, we may incorporate the Coalescence mixed reality technology into a virtual flight training device. As the technology advances, we perhaps will find more applications for the technology than we realize now.”

Cronan also sees the potential of adding virtual instructors to XR flight and maintenance training environments to add even more fidelity to those solutions.

“I will say that I hate to discount any concept, because as you look back in the day, some people never envisioned that a full-flight simulator could be a reality,” Cronan summarized. “Synthetic vision and enhanced vision combined can effectively let pilots see what exists out in front of the aircraft in an actual zero-visibility environment. So, you could foresee a windowless cockpit, with a scenario where you could have pilots wearing virtual reality headsets flying the airplane. I think that you are just going to see this technology explode in the future as time progresses.”

Published in CAT issue 2/2020