CAT Leader Forum - XR Emergence Workshop Report

Impact of Emerging VR/AR/MR/XR Immersive Technologies on Commercial Pilot Training.

During the virtual Heads of Training meeting, the group viewed the use of emerging VR/AR/MR/XR immersive technologies as the new normal in pilot training within the next decade. These are already used to good effect in Cabin Crew and Maintenance Technician training, and the military are also already using the “new realities” in multiple fields.

So what are the opportunities and limitations for further developing their use in civil Pilot training?

A group of training and technology experts met (virtually) and concluded that VR hardware technology is no longer a limiting factor; however, to properly incorporate these new technologies, the entire training philosophy must be reviewed.

First, let us agree on some definitions.

Definitions

Virtual Reality (VR) is an experience taking place within simulated and immersive environments that can be similar to or completely different from the real world.

Applications of virtual reality can include entertainment ( such as gaming) and educational purposes ( such as medical or military training).

--

Augmented Reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.

The overlaid sensory information can be constructive (additive to the natural environment) or destructive (masking the natural environment).

Augmented reality alters one's ongoing perception of a real-world environment, whereas  virtual reality completely replaces the user's real-world environment with a simulated one.

--

Mixed Reality (MR) is the merging of real and  virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real-time.

Mixed reality takes place not only in the physical world or the virtual world, but is a hybrid of reality and virtual reality, encompassing both augmented reality and augmented virtuality via immersive technology.

--

eXtended Reality (XR) is a catch-all term referring to real-and-virtual combined environments, where the 'X' represents any current or future spatial computing technologies.

“I would not say it’s something for tomorrow... it’s something for today.” – Fabi Riesen, VRM Switzerland

XR Technologies

Technology has evolved exponentially in recent years. A few years ago, VR/AR/MR equipment was not usable for daily training activities. The resolution was low, the goggles too heavy and cumbersome. Motion sickness initiated very quickly with 30% of the trainees.

Today the technology of the better providers is stunning. Definition is at human eye level, eye-tracking built-in, cameras for mixed reality built-in. Motion sickness is now reduced to a small percentage of users.

One could say the use of immersive training for a single user is no longer hampered by technology.

The military is already using XR for a year or more, and Maintenance is undoubtedly ahead of the Pilot training. Cabin Crew is using it primarily for safety training and emergency equipment location.

Two areas need to be enhanced to develop the use in pilot training: multi-user and mixed reality.

Most commercial Pilot training happens in crew since the procedures in airliners are based on a crew concept. For example, when training an on-ground engine fire, the actions of Captain and First Officer are separate and synchronised, so one pilot cannot perform all actions as this would be negative training.

When training procedures in a virtual cockpit, it is essential to do that as a crew, which is a limitation that needs resolving.

There are solutions where an avatar will act as a second pilot, but ideally the two human pilots should work together in the virtual cockpit.

The other issue is the "feel". Manufacturers design cockpits with ergonomics in mind, and flap handles, gear handles, FCU (flight control unit) knobs such as HDG (heading), Speed -- all have a distinct form and feel. A pilot puts his fingers on the HDG knob without looking and feels he has the correct knob.

By default, that is not possible in VR. That is the reason why most current solutions are built on mixed reality, part actual hardware and part VR.

Such solutions are already in use in the military: gunners lying on the floor with a physical gun whilst ‘flying’ in VR in a simulated gunship or helicopter where everything is virtual, except the gun. Apart from losing part of the immersion, the downside with a mixed-reality solution is geometric distortion, resulting from the fact that the cameras which are filming the cockpit and hands are not at the same position as the eyes. In addition, the camera stream from the best-available MR headset does not meet the required resolution and lighting and contrast also fail qualification criteria.

By today’s regulatory requirements, a simulator built on mixed reality does not meet the current qualification standard in civil aviation such as the CS-FSTD(H) from EASA.

The world’s first certified simulator using a VR concept is built with VR only, a haptic VR cockpit and a full body ‘pose tracking’ which is animating the body of the pilot as an avatar in real time. (EASA-qualified R22 from VRM Switzerland, using Varjo-VR-3 HMD.) VRM Switzerland’s new helicopter simulator with minimum hardware is said to enable training for takeoff and hover far better than the standard full-flight simulator.

Image credit: VRM Switzerland

The USAF is trialling Undergraduate Pilot and Instructor Pilot basic training in a mixed reality ITD (immersive training device) for the Texan II aircraft. The seat, pedals, stick and throttle are real, and the rest is virtual.

So solutions are rapidly emerging for fixed- and rotary-wing pilot training, and it is only a matter of months, not years before we have what we need.

The conclusion is that technology is not hampering progress in the use of virtual or mixed reality.

“We must take care that XR is used appropriately… for its strengths and not just because it exists.” – Capt. Shane Carroll, Airbus

Training: Why, What, Who, How?

In the past, in aeroplanes and training devices, engineers create new designs and capabilities, then the pilot community asks, “What do we do with this?”

It’s the same today. Virtual reality technology is being developed, and applied to a multitude of applications. Organisations are now proposing training solutions for commercial aviation based on VR.

Solutions for what?

We have iPads for ground school, FTDs for procedure training, and Level D FFSs for the flight part. We have been doing fine for decades.

What problem do we have that VR is going to solve?

Whatever we do, the quality must be equal or higher for a lower or similar cost.

What benefits are enabled by using these new technologies, and how can we, as a training community, not only use what is available but also drive future development?

Image credit: USAF / Airman 1st Class Tyler McQuiston.

Accessibility

For many possible training applications, a helmet-mounted display, a computer and good internet connection are all you need. Access to training is no longer limited to a training centre; certain tasks can be done at home. Certainly, a tablet solution works also, but the quality of training using a VR solution versus an iPad is significantly better. Testimony of crews indicates the superior realism of the VR solutions augments the training quality and the motivation, as the experience is stimulating.

For airlines in more remote areas or airlines with multiple bases, crewmembers can do part of the training at home or in a small office at the base. One airline explained that they provide VR HMDs for the crew to review at home.

“We have had great success in demonstrating to our Cabin Crew the emergency landing on water… some reported the realism was so high they were afraid of falling into the water.” – Nicolai Rasmussen, Sunclass Airlines

Effectiveness

For certain types of training, the effectiveness, or transferability, is measurably better with VR.

Cockpit setups and procedure drills, for example. Formerly, we trained with a paper or cardboard cockpit to learn the flows. But that was limited to touch exercises with little realism.

More recently, these flows are trained via a screen, but that takes away the location and movements from the learning process.

With a VR solution, you have the best of these two: the accurate dimension and location of everything around you, and the possibility of creating scenarios, thus rendering the training much more effective.

The same applies to emergency equipment location, be it in a cabin or galley.

With the newest high-definition HMDs, new possibilities are emerging. The new headsets provide a 3D perception that is better than the classic visuals in simulators.

One application is the VRM Switzerland helicopter simulator in which depth perception for near-ground is excellent for training exercises such as hovering.

Cost-Benefit

The new technology is not free: both hardware and content creation have costs, though certainly not at the level of high-end FFSs.  But Training Managers must answer the question: what is the business case?

There are many different ways and reasons to use this technology.

Doing aeroplane familiarisation for Cabin Crew (or Pilots) can be a logistical nightmare. It is seldom possible to find a fully functional aircraft sitting on the ground to have a group of Cabin Crew become familiar with their new aircraft. Most of the time, the whole group must come at night to spend some time in a cold aircraft… not the best environment for learning. And also expensive since the entire group loses two working days for a few hours of night duty.

The other solution, often used for pilots, is accessing an aeroplane under maintenance in a hangar, but that creates safety issues.

One might argue that the first aeroplane visit of a newly hired crewmember touching the real thing has a Human Factors advantage. This first aircraft visit is much less about any technical training per se, but more about the motivational aspect of seeing your first aircraft for the first time. It is exciting and brings to life everything that has been learned thus far.

Beyond that, VR solutions are certainly worth considering, especially for experienced crews or as refresher.

Another option is using VR to reduce the usage of FTDs. One company uses VR in the briefing room to substitute the first hour of an FFS session, usually devoted to setups, entering data in the FMS, pre-flight briefings, etc. Then they enter the sim, which the instructor has set up ready for take-off. The four-hour session is now a three-hour session.

It may need to be validated in a more extensive study from a training perspective since some pilots may have difficulty getting in the mindset once they enter the sim. From a financial standpoint, the business case of buying the hardware and software versus the cost-saving needs to be evaluated. This may work especially if the number of crews is large enough. It is certainly a concept that is worth exploring.

The use of VR as a debriefing tool also seems to be very powerful.

A simple business case would be to replace actual aeroplane hours with VR time. In this case, it may be necessary to use some form of mixed reality. As mentioned, the USAF training devices are mixed reality. Although more time is needed to validate the project thoroughly, it seems to work well. Students train the missions before flying the same task in the aircraft. Less time is required in the aeroplane to get to proficiency, giving an overall reduction of flight hours without losing quality.

The cost of the hardware, probably not the main cost, and the content development cost are fixed costs. Still, benefits are dependent on numbers (number of crews, hours of simulator or aeroplane), so the size of the operator might be a factor in the introduction of this technology.

Image credit: Airbus.

Training Syllabi

Numerous VR/AR/MR applications are in use or envisioned soon. But everyone in the subject expert group agreed that the use in actual training is not as simple as replacing an hour of training on an FTD with an hour of VR.

We are not talking about a simple evolution of an existing tool, like improving the visuals or motion system. It is not replacing a PowerPoint-based course with a tablet application.

We are looking at a fundamental change that needs rethinking of the whole course syllabus.

The effective use of this technology can only work in blended training, re-designed from the ground up.

This re-design needs to take into account the capabilities of the tools (existing and VR), tasks to be trained, and other aspects such as Human Factors. Break down the existing training program and restructure it differently, taking advantage of each available tool.

We must move away from using technology because it exists and use the best technology for the task at hand. VR is not there to replace FTDs or FFSs. This new technology can be used effectively in an adapted program to prepare better the candidates for the parts that require these devices, and use the time saved in the FFS for more suitable learning tasks that need to be done in the full-flight.

“Right now a lot of research goes into mixed-reality and crew training scenarios.” – Seppo Aaltonen, Varjo

Training Environment

The word training, in commercial aviation, covers quite different target populations and target outcomes. Training a cadet in ab initio is very different from a recurrent training of a senior Captain.

The tools we are using for ab initio, ground school, type rating, and recurrent training can differ.

So the questions are: can the VR suite be of use in all these training environments?

Which type of training can use which specific kind of tool? What are we still missing today?

As long as you use the tool (whatever tool) for what it is meant and in an integrated way, then the answer is probably yes. But there is still some work to do for the equipment providers for some applications.

Ab Initio

We will probably see a larger ATO starting to use this in the civil ab initio realm, reducing real aeroplane time and getting better results in the visual part of the flight syllabus.

Type Rating

The obvious use is ground school, but also the cockpit setups and procedure flows. By using a VR solution, the crewmembers can practice the setups and flows to the point that this no longer needs to be done in the simulator, or that the confidence is higher. But we could also imagine using the VR tool for the walkaround prior to a LOFT scenario in the sim.

It is hard to say how much time could be saved in an FFS via VR; it may be more an increase in quality than cost savings.

We have learned from military applications to date that the effectiveness of the training in the sim or aeroplane is higher after preparation with VR.

Recurrent

For recurrent training, it is slightly different. For Cabin Crew, for example, VR is an ideal tool for refresher training of emergency equipment location and procedures. The psychological advantage of the actual aeroplane for the first time is no longer valid for refresher training, and this way hundreds, or thousands, of Cabin Crew can go through recurrent training without ever blocking an aeroplane.

For Pilot recurrent, it is less obvious, except in the present pandemic situation where pilots are coming back from a long absence of flying. They can use the VR tools for a refresher in preparation for the sim sessions. They can get back up to speed with the scans and flows and be ready to do real training when they enter the sim.

To be used as an actual type rating or recurrent training tool for pilots, there likely needs to be some kind of mixed reality. The feel of certain knobs and handles is an essential element.

But the most important missing element currently  is the linking of the two pilots in the virtual world.

Data Collection

One element where the VR tools can also be helpful is data collection. It is possible to collect data of where the students are looking, so instructors can rectify faulty scan patterns, not based on assumptions but on actual data.

Instructors

As we all know, good training does not come from tools. It comes from efficient syllabi and foremost from quality instructors.

What kind of instruction is needed to optimise the effectiveness of VR tools?

Having an instructor sitting on a chair next to a pilot, or crew, working in a virtual world will not work very well.

Are we going to have an avatar instructor?  Maybe we can learn from the gaming industry. Perhaps the software can be instructing the pilots on how to do specific procedures while the instructor is in a supervising role.

Maintenance training is ahead on this, and experience shows that it takes time for all involved to change the method of instruction. The Maintenance trainees work together to solve the problems, and the instructors are more facilitating this “discovery” instead of classic teaching. In the beginning, there was some resistance, but as the positive results became apparent, the instructors were won over.

We can imagine an instructor giving a difference course with six students, all in the virtual cockpits. In their virtual cockpit, the instructor sits with them and walks them through the different panels and explains the differences. It is no longer a classroom instruction but a simulation instruction, thereby changing the role of the instructor.

Once again, it is not the technology that will be an issue, but the change management. And as everybody is discovering these new tools and experimenting with them, we will need to be very careful on the human aspects and implications.

“We can have between 9 and 21 senses that the body is constantly processing.” – Helen Heenan, KURA Human Factors

Human Factors

We have seen earlier that the technology has improved dramatically and the initial issues with nausea or vertigo are now nearly eliminated. Yet there is still room for study on how the human body, the human brain, handles these different forms of virtual reality. How long can one take instruction under these circumstances? Can our brain take two, three hours of virtual reality training? What is the retention after a prolonged use time? What is the optimum time?  Can we have one session per day or multiple shorter ones? Can we have daily sessions?

Further studies need to provide answers to these questions.

But there are other Human Factors to be considered:

Tactile feedback. Current cockpits have a lot of tactile information. Over time, a pilot will engrave in memory the feel of different knobs, pushbuttons, or levers. During training, this tactile feedback must be part of the instructional targets. Manufacturers and operators must acknowledge these issues as part of the analysis when deciding which type of device is necessary for training.

Motion. There has been, and still is, a debate about the need and usefulness of motion in simulators. Most of it is a cost-benefit discussion.

With the use of VR devices, the possibility exists to mount these on small, relatively inexpensive motion systems, which will help create a more realistic immersion for the student. Such a motion base has been successfully implemented in single-pilot applications like a helicopter.

Communication is a vital element of all human activity, and certainly in training.

But communication is not only words spoken. There is a massive amount of learning that goes on subconsciously. More than 30% of communication is the intonation, and more than half is the body language. Most interestingly, the correlation between words, intonation and body language is the most important factor for effectiveness. If the words, the intonation and body language do not express the same, there will be confusion and miscommunication. Even if we get the crew talking together in a virtual world, what will be the body language of the avatar representing the other pilot?

Solutions like HoloLens could be part of the answer.

There are any open questions the industry must consider when getting deeper into these technologies for pilot training. In due course, the challenges can certainly be overcome.

“What tests do we need to run that are different from traditional simulation”? – Mark Dransfield, Simops

Regulation

In both Europe and North America, there is a willingness to facilitate the use of new technology in training.

One of the aims is to make the requirements for simulation technology-agnostic. Whether a solution using traditional simulation or new technologies, the training requirements are the same.

One of the difficulties seems to be the wide variety of quality of the proposed devices. Some may be good enough as entertainment, but not for professional training. Should there be a quality label of some sort to assist the various authorities in their approval work?

There seems to be enthusiasm among regulators to help the industry adopt some of this new technology and making the regulation agile enough to cope with it.

Conclusions & Recommendations

Workshop subject experts included:

Mark Dransfield, Simulation consultant

Christian Marty, VR simulation developer

Julien Colinos, EASA

Ernst Pontier, Major airline

Chris Ranganathan, Simulator manufacturer

Fabi Riesen, VR simulation developer

Seppo Aaltonen, Headset developer

Joris Field, Laboratory

Shane Carroll, Aircraft OEM

Nicolai Rasmussen, Regional airline

Helen Heenan, Human factors consultant

Tanja Harter, European Cockpit Association

Some Links

First regulatory-qualified VR-based flight training device

An example demonstrating the 3D difference in an FFS with a Dome Projection versus a VR Simulator:

Examples demonstrating “close to ground flying” resulting from a correct 3D perception:

USAF Immersive Training Devices

Military VR/Motion flight training platform

VR/AR Device Showcase

A Market-Drive Informational Webinar Series

The CAT Leader Forum emerged from an annual Heads of Training (HoT) meeting which Halldale typically conducts with EASA at the European Aviation Training Symposium – EATS. (This year’s EATS will be 2-3 November in Berlin - https://www.halldale.com/events/69-eats-2021-european-airline-training-symposium)

As there was no live EATS in 2020 because of pandemic restrictions, EASA and Halldale agreed to conduct a virtual HoT meeting.

From that meeting emerged five “HoT” topics facing the airline training community in Europe which warranted deeper discussion.

•           Skills Decay

•           XR Technology and Training

•           UPRT

•           Big Data

•           EBT/Competency-Based Training & Assessment

Led by Capt. Jacques Drappier, JDR Consult, former Vice President of Flight Operations and Training Services for Airbus, and Chair of EATS, Halldale is conducting intensive workshops with about a dozen leading subject experts on each of the five topics.

From the workshop discussions, a subset of subject experts are then presenting webinars summarizing the group’s findings, as well as a detailed text report and executive summary.