The MS&T Interview

FlightSmart: FlightSafety and IBM Launch AI-Based Pilot Training Performance Analysis and Predictive Tool

FlightSmart applies artificialintelligence and machine learning algorithms to evaluate a pilot’s ability toperform critical tasks and maneuvers during all phases of flight. Any identifieddeficiencies result in a remedial training action path, personalized to thepilot, to increase proficiency.

YOU ARE HERE. FlightSafety’s new FlightSmart data analysis tool enables AI-based identification of student pilot deficiencies and prediction of remedial training actions.

FlightSafety believes the tool’s capabilityto automatically predict student performance and identify corrective action is akey differentiator, beyond any other simulator data-driven product currently onthe market.

The launch customer for FlightSmart isthe US Air Force Air Education and Training Command (AETC) for implementationon T6A initial and operational flight training devices.

MS&TEditor in Chief Rick Adams spoke with Matt Littrell and Bert Sawyer of FlightSafetyInternational. Littrell is Product Director, AI and Adaptive Learning; Sawyeris Director of Strategic Management.

MS&T: Let’sstart with the elevator speech.

Sawyer: What we're ultimately tryingto do with the FlightSmart product is take training to the next level in termsof taking the subjectivity out of the evaluation of student performance andbringing objectivity to that.

The simulators, of course, are giantcomputers in essence. And we're able to capture a tremendous amount of data outof them. As we collect that data, we utilize the algorithms of FlightSmart toanalyze and grade the performance of the student a couple of different ways.

We do it on a basic level of simplyevaluating against a known standard. In most cases, that's going to be theapplicable SOPs. It could be whatever standard is appropriate for that customer.

But we've taken a step further. Ratherthan just a binary, did you pass or fail, we apply a grading scale so we canget some granularity. Did you do better than or worse than the criteria?

We also look at another element – evaluatingthat performance against a “gold standard.”

MS&T:What parameters are used to determine the gold standard of performance?

Littrell: When we first set up FlightSmartfor a particular aircraft, we gather a number of data points from what we wouldterm “best of the best” pilots. Generally those are instructors, could be linepilots. They fly a whole slew of runs, covering all the training tasks thatwe're looking at. On average, in a simulator, we're collecting about twothousand parameters. As we work through the data, we're actually training thesystem to do the initial parsing of the data and then the human will interveneand throw out the “corner cases.” Ultimately, we end up with a baseline.

We could have the simulator fly aperfect maneuver, but there is no such thing as a perfect maneuver or trainingtask. There's no human that can fly a perfect scenario. We intentionally usehumans so we could capture the human element.

MS&T: Howdoes FlightSmart evaluate the student against the standards?

Littrell:  We evaluate the performance of the studentagainst the baseline in two different ways. One, we measure the physicalcloseness of the student plot to the baseline plot.

But then we also look at how smoothwere they. We can look at the frequency and the amplitude of that plot linewhere they were very smooth and precise, where they were over-controlling, wherethey under-controlling.

Ultimately that leads us to identifywhat we're calling “personas.” It could be somebody that's timid on thecontrols; they're continually late to respond with control inputs to thedeviations. Or maybe they're aggressive; they're just manhandling that airplaneconstantly.

Think about the binary grading: passor fail? Let's say the standard is plus or minus 100 feet on altitude. Theycould have been bouncing off the walls between 99 feet above and 99 feet belowand, per the standard, technically they passed. The standard binary gradingdoesn't take into account their precision. And what we're accomplishing withthis element of the evaluation against the baseline is determining were theybouncing off the walls constantly or were they very smooth and precise?

That helps feed into the overallscore, but it also helps us down the line as we get to the predictive analyticsand ultimately the adaptive learning so we can tailor the remedial learning. Ie,this is the ideal path or method to train this task for this “persona,”somebody that's aggressive on the controls. Here's the best path or it providesthat information to the instructor to help them understand and tailor thetraining appropriately to that persona style and really enhance overall thatperformance.

So we have this objective measurementof training tasks that each student would go through. We're looking at thecompetencies of that student against those tasks. We have signs that thealgorithms are using to decide what is the performance of the student. Is heproficient at this task or does he need some remediation?

It's all that root cause. Evaluationof his performance comes back to some prediction of the success of thatstudent.

TRAINING MANAGER DASHBOARD. FlightSmart is intended to supplement an instructor’s skills and make them more efficient, not replace them. An instructor can log in to the dashboard from a computer, iPad or iPhone.

MS&T: Whatare some of the parameters that you're measuring to get to this best-pilotprofile?

Littrell: Let's say that we want tolook at steep turns, V1 cuts and rejected takeoffs. So the (best of the best)pilots fly a whole series of those training tasks. And repetition so we can getsome good data on those. And, again, throw out those corner cases. We create anaverage of all those.

With big data, there's a learningelement or a self-learning element to that. Over time, as we collect more data,we'll be able to continue to refine that baseline and make it even morerepresentative of that top pilot population.

Sawyer: We also have a shaded areathat represents the 25th and the 75th percentile of that gold standardpopulation of pilots. You can see look at that the student plotline, and ifyou're within that shaded area you know you're within that 25th and 75thpercentile of the gold standard.

MS&T: Explain,please, the difference between the diagnostic and predictive analysis.

Sawyer: The diagnostic is the why didyou deviate? We may be focused primarily on seven parameters and identified bythe SMEs on the aircraft as being the most important ones to focus on. Butrather than just limiting our vision to just those seven, once we see thatthere has been a deviation, the machine learning will then go look at all twothousand of those parameters. And in essence, create an error chain todetermine what ultimately led to you deviating from that standard.

So let's say you deviated on youraltitude. It may recognize that two minutes and 36 seconds ago you bumped therudder pedal, which destabilised you, which led to this, this and this andultimately led to that deviation. And it can really identify causal factorsthat the instructor may not be able to get to on their own or, if they ever doit, be a fairly significant period of time for them to actually get there. Andthat's one of the tremendous values – giving the instructors insights intothose things that they would not necessarily otherwise have visibility into,either physically can't see in the simulator or cannot make all the connectionsto determine that it is causal.

Once we've figured out that that rootcause, why did they do it, then we can absolutely provide remediation. It maybe as simple as reduce the amount of control input in this particularsituation, and you'll do better.

Take something as simple as a glideslope. You have a tolerance of plus/minus 100 feet, but a student is outside oftolerance. It doesn't say why he did that. (The instructor) doesn't know thatthat the guy was on the throttles, that he was putting too much force on thestick. And that's where the machine learning tells the instructor, this is the reason the guy got into that situation.This is why he was outside of thetolerance.

So with all those connections, basedon historical data, you should go spend 25 minutes on the flat-panel trainerpracticing this specific task and you need to go read these two paragraphs inyour training manual and X, Y or Z, because, based on the data, we can provethat students that do that will have a high probability of being successful onthis training task based on your specific situation.

It really takes the art out of it andapplies science to figuring out how to improve their performance.

The vision of FlightSmart isconnecting the entire training ecosystem, whether it's the simulator or theaircraft or mixed reality or computer-based training. Whatever it is, any typeof training the students are doing, we're connecting together.

MS&T:  Do you see the potential to analyze whether apilot is competent in a certain training task or training requirement?

Littrell: That's a challenging one.  think everybody would love to figure out. Ifirmly believe that we will be able to get there. It's a matter of finding theright combination of technology and algorithms to figure out how to do it. Butthe fact is, those are the soft skills, not something that there's datadirectly tied to, that makes a lot more challenging to do.

Sawyer: In one example, duringrejected takeoff training, FlightSmart showed the instructor that the co-pilotwas compensating for the pilot on the controls. The instructor couldn't tellthat; he couldn't see that from where he sat. That's where I believe we'rehelping these instructors see more about the competency of pilots than what wehave in the past.

MS&T: WithFlightSmart, how does the instructor's role change?

Littrell: Whenever I talk toinstructors, the very first question is always, “Are you trying to replace me?”And the answer is no. They're already great at what they do. Our goal is toprovide them with additional tools that make them even better at what they doand more efficient.

From the beginning, our goal has beenthat this cannot significantly increase the burden on the instructor. One ofwhich is the automatic identification of training tasks rather than theinstructor sitting in the simulator having to select, okay, now I'm going to dohave them do a steep turn. The algorithms automatically identify those withinthe data.

The impact on the instructors, how wesee their lives changing is reviewing the results on the dashboard. Seeing theinsights that are generated, the recommended remediation and taking thatknowledge, combining that with their knowledge and experience, and being ableto more quickly and more effectively communicate to the student how to improvetheir performance.

MS&T: Sowe're not talking about it being a robot instructor.

Littrell: No, not noticeably. We'vehad requests from a customer to integrate a coach, especially in freeplay typeenvironments. If a student gets into an FTD and is practicing on their ownwithout an instructor, we are looking at adding functionality to serve as, inessence, a robotic instructor to help coach them through that training. But ina live instructor training session, it's not designed to replace thatinstructor – it’s designed to provide additional information to the instructor.

You need more pilots through thesystem faster and with the pilot shortage that means we have an instructorshortage. That’s within the DoD and in any commercial area. This tool is thereto help supplement the instructor. It makes them a learning manager where theycan manage more.

Sawyer: One of the reasons we pickedthe DoD is they have better control over the regulatory requirements and theirsyllabus. And they can decide how fast the student progresses in the system.

MS&T:What’s the status of the T6A launch program?

Littrell: We're in the implementationphase right now, which includes modifying the simulators. Those are 20-plus-year-oldsimulators. So there's a little bit of work we have to do for them to enablethe data capture faster.

This spring, we will have thetraining developed. We'll have an initial implementation on the base with theinstructors utilizing the tool in their training. Late spring, we'll be gettingsome really good insights into the improvements that it's making to theirtraining.

STUDENT DASHBOARD. FlightSmart is being rolled out initially for pilots but will be expanded in the future for aircraft maintenance technicians, operators of unmanned systems, and others.

MS&T: Howabout on the commercial side? Do you anticipate a customer launch there?

Sawyer: Internally, we have thisinstalled on several of our devices and we've done some test cases using ourcurrent customer base and our instructors. We’re looking at instructorutilization aircraft, platform types, putting a strategy together on how we'regoing to roll this out.

I would expect that by next yearwe'll have this in learning centers and available, but it's going to be basedon whatever the customer base is willing to sign up for.

We've also had some very goodconversations with, we'll say, a large university that has tremendous interestin what we're doing. Their vision is to go full-on adaptive learning andFlightSmart does a wonderful job of creating an environment that allows forthat and provides the tools that are necessary to accomplish it. It's a robustmarket opportunity with primary training. Even with the skill level of thestudents, there's a lot of opportunity to use that data to shape their trainingexperience and tailor it to their individual needs.

MS&T:How would you compare this with some of the other data-related products thatare on the market or are announced as coming to market?

Sawyer: I would say one of the keydifferentials is the automatic identification of the training tasks (to beremediated). I've not seen any others on the market that have that capability. That’skey to reducing the burden on the instructors. They have enough on their platesalready.

Another is the use of the machinelearning / AI and really getting into the diagnostic and predictivecapabilities. Some of (the competitor products) that I've seen, all theyprovide is an evaluation of how the student performed against the standard;that's the extent of how far it goes. It doesn't give any detail on why, thaterror chain or the root cause analysis as to exactly why they deviated. Andfrom there, what specifically should they do in the future to ensure success?Everything that I've seen on the market stops. You deviated from your altitude,and then it's up to the instructor to apply their traditional instructionaltechniques to deduce why that was and what they need to do differently in thefuture.

I would add to that, a lot of this isscalability. When you look at how we we've implemented these algorithms, thisis capable of scaling from platform to platform.

MS&T: Whatdoes what does IBM bring to the team on this?

Littrell: They have 300 and some odd veryhigh-level data scientists available in a pool that we can drive as needed.They have tremendous amounts of experience in adaptive learning in severaldifferent industries. (They are) one of the pioneers on data privacy, (healthcare)HIPPA and so forth. So they bring a very well-rounded experience of capabilityto us to help augment where we may not be as strong or they help us see outsidethe box. They bring in that perspective from other industries, which has beenvery helpful, different ways to look at the data or manipulate it to get towhere we're going.

Sawyer: We wanted to be able to scalethis, and we can lean back into IBM and rapidly build these algorithms as we gofrom aircraft to aircraft. They give us the ability to scale, which is reallyvery important as we roll this out into the training ecosystem.

We did a lot of analysis, as we weregoing to launch with the Department of Defense being the first customer. IBM isranked third with the DoD in regard to “validated algorithms.” The definitionis that it requires no human interaction from the decision tree to happen andfor changes to be imposed, which is huge because that's just what we're doing.We want the system to be able to provide prescriptive remediation and thenmonitor the student throughout.

MS&T: Whencollecting all of this data from aircraft and simulators and so forth, sometimesthe pilots or the unions get uptight. What sort of privacy controls are builtinto the system?

Littrell: That's definitely been achallenge. On the commercial side, with the pilot unions especially.Thankfully, they're already at least somewhat accustomed to it with FOQA wherethey've been collecting local data for 20-plus years.

With the Air Force, they're not somuch concerned about the data privacy and they've requested that we identifythe data all the way down to the student or the individual level, which reallyis the ideal level, because now we can get a history for you as the pilot andreally provide the predictive abilities in the tailored training based on yourhistory.

Other customers don't feelcomfortable due to privacy concerns, especially when it comes to the unions. Insome cases, we will identify solely to the customer level. We know that it is apilot that flies for XY airline, but we don't know that specific individual’sname, just a pilot with that airline.

And the final level is it'scompletely anonymous; it’s pretty limited what you can do with that data.

This goes back to IBM's experiencewith data privacy; they've provided a lot of assistance to help with ourknowledge of data privacy, on how best to structure it to protect thatinformation. And ultimately, at the end of the day, the identified data residesin the possession of the customer.

ICAO / AIRBUS / BOEING competency-based training requires soft-skill evaluation, which is more challenging for data-driven systems. But FlightSmart can assist instructors with CBT Assessment.

The only data that is transmitted tous outside of a development effort is purely de-identified. It's what's called“pseudo anonymized.” There's a random ID that's attached to it; we don't havethe decoder ring to identify exactly who that is. There are a number ofdifferent tools that we employ to tailor that privacy.

MS&T: Wouldyou also anticipate aggregating some of the data, say, for a customer or an aircrafttype, so that you could tell this customer that your group of pilots that gothrough our training have these tendencies?

Littrell: Absolutely, there'stremendous value at that level. If we implement it at an airline, they do theirown training. But really, this needs standardization. We can provide analyticsor metrics on performance of that population either at an airplane level or ata total population level. You know where are they struggling, where they excel.The population is doing very well in this area, but not so well over here.Maybe we can fine tune the training program to focus less on this area andfocus more on that area.

Also, the airplanes are continuallyevolving either through service bulletins or avionics upgrades and so forth. Historically,the industry's not done the best job of validating training that they developedto support those changes. We can do pre- and post-implementation comparisons ofstudent performance relating to those areas to help identify the best trainingthat was developed to support this change.

You can look at instructorstandardization. How well are the students under this instructor performingcompared to this instructor and ensuring that everybody's on a level playingfield. The students with this instructor historically doing not as well as theyare with these other instructors. What remediation do we provide thatinstructor to get them back up to the level of everybody else?