MTX-Realities

It has been dubbed the “Fourth Industrial Revolution.” US Editor Chuck Weirauch reports how mixed reality applications are poised to revolutionize aviation operations and training.

In an Oliver Wyman survey of companies involved in maintenance, repair and overhaul (MRO) operations, approximately 68% of respondents replied that they would be making investments in wearable or handheld devices in the 2018-2020 timeframe. A 2019 report by Internet of Things (IoT) and augmented reality (AR) specialist company PTC reported 86%of industrial enterprises are expected to be moving AR pilot programs into production within the next 12 months.

Still, it’s important to remember that AR is an emerging technology. Except for a few early adopters who have plunged full-bore to incorporate its use into their operations, most firms seem to be at the pilot program stage to determine if and when investments in the technology might pay off. AR hardware is still prohibitively expensive for some, and still is limited in some aspects, such as image resolution, field of view, battery power, durability and other design factors. But experts in the field expect these limitations to be overcome in the near future.

AR for Aviation Ops

One example of the kind of impact AR technology could have on aviation is a recent partnership between Lenovo Research and the Commercial Aircraft Corporation of China (COMEC). The two companies will create AR applications as a major tool in the development and manufacture of commercial airliners. The primary goal is to “improve the quality and efficiency of China’s commercial aircraft manufacturing while cutting production costs.” More recently, Lenovo announced that China’s Suparna Airlines Group will be employing Lenovo’s daystAR application in its airline maintenance process.

Boeing and Airbus are also early adopters of AR, employing it in the aircraft manufacturing process, as well as operations and training. In one case study, Boeing reported that aircraft wiring productivity increased by 25% while errors were reduced to practically zero. More recently, the aircraft manufacturer has been using AR to promote its new 777X airliner to customers, and testing its application to familiarize technicians with wiring in the KC46 tanker.

Airbus, a member of the Microsoft Mixed Reality Partner Program, has reportedly created Microsoft HoloLens apps to train engineers and cabin crews, along with Japan Airlines (JAL) and JAL Engineering. JAL is also a member of the Microsoft program. This effort is a part of Airbus customer support for its A350 XWB airliners. JAL took delivery of the first of 31 A350 XWBs in June.

Koji Hayamizu, Senior Director of JAL’s Products and Service Administration, stated that employees’ practicing maintenance procedures with the HoloLens helps change their “intellectual memory to muscle memory,” and he believes that the application of AR can contribute to safety.

Airbus plans to offer HoloLens-based training applications for some selected training scenarios to airlines. The app is part of Airbus’s greater plan to use mixed-reality technology for training; the aircraft manufacturer has collected more than 250 different use cases for the HoloLens, ranging from in-flight entertainment to communication.

United Technologies Research Center and Pratt & Whitney’s customer training division are experimenting with virtual reality (VR) engine maintenance training for airline mechanics. Bruce Hall, General Manager of the division at the time, said the company was testing the use of AR headsets and hand sensor controls that would allow mechanics to virtually walk inside a GTF engine to examine parts and view a running engine in motion.

What’s the Fuss?

Since AR technology has been employed in commercial and military aircraft for many years now, it may seem difficult to understand just why it has become such an exciting development. The concept is the same as found in aircraft cockpit head-up displays and jet fighter helmet-mounted displays; projecting computer-generated flight performance data onto a transparent screen in front of the pilot while still allowing that pilot to view the real world.

But in the case of an aviation maintenance technician, for example, their wearable head-mounted smart glasses, such as a HoloLens, and mobile devices, such as phones and tablets, can project maintenance procedures manuals, simulated 3D aircraft components and other data onto their viewscreens while they have a clear view of the actual component that they need to assemble, inspect or repair. This technology is applicable to most industries that need to perform MRO operations.

But AR advocates say that the biggest advantage of employing the technology is being able to collaborate remotely with experts to help resolve a problem. Technicians can wirelessly share the visual image and computer data of the component to be inspected or repaired with those experts at remote locations via the AR and mobile devices.

In a collaborative effort, the networked team members can then work together to resolve the problem or issue at hand. If you have ever had a computer expert remotely access your computer to resolve a problem, you can have a basic idea of how this concept works. Major industries are saving considerable time and money by not having to send expert teams to resolve difficult problems on-site, such as identifying equipment malfunctions, through this so-called “see-what-I-see” concept.

An MRO Perspective

Aircraft OEMs and Aviation MROs are most likely to be the first beneficiaries adopting AR technology into their operations and training programs. AAR’s digital services expert Matt Kammerait said that the MRO giant is working on an initial AR-based training program to supplement current training programs, but for now is concentrating more on pilot efforts to incorporate the technology into operational applications

“We have been looking at AR for the past couple of years,” Kammerait reported. “We are targeting a lot of operations use first, particularly in addressing the broad labor shortage issue affecting the industry in terms of getting more out of our most-qualified people and multiplying their effectiveness at the point of use. We view AR and VR as a part of a continuum that starts with training, and then makes its way into operations. There, we make more usage of virtual environments to get people more acquainted with things that they will see on the floor. Then they will progress to where that digital content kind of recedes into the background, as people become more and more capable, and just access it when they need additional systems knowledge or when they need to connect to the content.”

During the pilot program, AAR expects to find that using AR headsets, phones and tablets to connect technicians with on-site engineers and off-site subject matter experts remotely in a collaborative manner to help resolve issues will pay off. The MRO is targeting about a 30% reduction in requests from technicians for engineering support, and then the faster resulting resolution of problems. Some of the savings would come from the sharing of digital images and other data via the devices, rather than technicians traditionally queuing up for engineering support and submitting paper-based request forms. Some resolution scenarios can be captured digitally and then referred to when a similar situation arises to save time and money.

“In terms of the training, though, I think that one of the primary challenges is creating the content for the training experience, where the operational use cases connect end users with experts or users with each other, such as in remote collaboration scenarios,” Kammerait said. “We ultimately ended up sequencing those scenarios first, and those are the furthest along because they are involved with content opportunity to capture the questions that people have, and the interactions they have, in order to resolve those questions.”

Although there is considerable promise for adopting AR technology, there are still multiple issues for MROs, especially in the hardware area, Kammerait noted, “Most people in the industry are at the stage where we are at, and the hardware implementation is difficult for everybody,” he reported. “But everybody in the industry is investing in the technology one way or another. Everyone is sort of chipping away at it. There is really a big opportunity for the industry as whole to work together in AR research, with everyone having access and benefiting from the resulting efficiencies. The thing that I would like to see is having more collaboration between the players, and sharing what we are learning and the content being developed. In this way, it winds up benefitting everybody.”

AR for MTX Training

One of the latest offerings in an AR-based aviation maintenance training program was announced in June by a partnership of KLM Engineering & Maintenance and the Royal Netherlands Aerospace Centre (NLR), in affiliation with KLM Royal Dutch Airlines. Dubbed NUVEON, this new product “will integrate AR throughout the MRO chain to improve everyday performance in maintenance.” The NUVEON program will be based on HoloLens technology, and “will be offered as a service to MRO-related companies that seek to improve and make their aircraft and aircraft components more efficient and more effective.”

Both partners are currently developing several maintenance training applications based on HoloLens. Those applications can be employed to familiarize technicians with MRO operations and aircraft components. Training will then be conducted on 3D-projected virtual components.

Last year, Quantum3D announced they had developed a free augmented reality maintenance app for the iPhone and iPad. The HoloLens can be connected to such mobile devices, with data projected onto phones and tablets for easy reference by maintenance technicians.

A Quantum3D demonstration video of the product shows an aircraft mechanic being trained on the procedures for repairing a helicopter jet turbine engine, with a CAD model of the engine floating in front the technician. A digital maintenance procedures manual provides step-by-step instructions for a critical repair operation. The free app can be downloaded from the Apple App Store at Quantum3D Apple App.

GE Aviation completed a six-month pilot program in which its maintenance technicians used an AR headset and related software to follow maintenance procedures projected to the smart glasses to properly torque and tighten B-nuts on an aircraft engine. The results of the study showed an 8-12% improvement in the efficiency of the operation. The majority of the technicians involved in the study said that they preferred the use of the technology over traditional methods, and 85% believed that the AR approach would reduce errors during the procedure

In 2016, Lufthansa Technik introduced a laser-based AR interiors installation system to support completions and refurbishments of private aircraft cabins. The mobile app provides a more precise tool for component positioning and alignment during installation. Since then, the aviation MRO has introduced its cloud-based AVIATAR portal, which provides web-based digital products and services based on emerging immersive technologies that include those involved with predictive maintenance and data analytics for airline MROs as a part of its “Digital Fleet Solutions.” One of several apps offered is Engine Health Management, which features a virtual replica of an aircraft engine.

A Closer Look

AVT Simulation provides AR-based training tools for military rotary aircraft maintenance, though not yet for fixed-wing commercial aircraft. Nevertheless, company management feels that these tools can be effectively employed for commercial aircraft maintenance training as well.

Kyle Crooks, the Orlando-based firm’s President, and Kevin Vizzarri, AVT’s VP of Business Development, provided CAT with an overview of how the firm applies such immersive technologies to its own training products.

AVT makes use of Unity gaming and Azure software development packages to create simulated aircraft components that are viewed with Magic Leap AR headsets and iPhones and iPad tablets by students during their training programs.

AVT makes use of Unity gaming and Azure software development packages to create simulated aircraft components that are viewed with Magic Leap AR headsets and iPhones and iPad tablets. Image credit: AVT.

Students wearing the AR headsets in the classroom can view both the actual aircraft part and the simulated image of it at the same time, Crooks described. The students can first take the simulated component apart – virtually. Then they can review the step-by-step maintenance procedures via a digital copy of the maintenance manual projected onto the AR headset glass.

The students can then use virtual tools to perform the procedure, practicing the “muscle memory” to properly perform the task on the actual hardware, Vizzarri explained. Instructors can put on their own networked AR headsets to view student performance and provide instructions to the students at the same time. The AR headsets are useful in the classroom, but not on the flight line.

“The AR headset would be more for use in the classroom environment, as you probably don’t want people on the flight line with AR goggles,” Crooks pointed out. “There are still some safety concerns for that. But for the classroom, that use makes perfect sense. You put it on – you see the aircraft in front of you, and instead of being on your phone, you can walk through the aircraft as if it is a full one-to-one scale and learn the maintenance task that way. It’s as close as you could be to actually being on the aircraft. The iPhone and iPad apps are more for qualified mechanics who only need a refresher on the maintenance task. Common sense implies that you can train more people quicker for less cost this way than with traditional methods.”

Published in CAT issue 5/2019