A Dive into the Data Lake of Human Performance

By Peter Neubauer.

Peter is Principal Software Engineer, Engineering Manager for Air Force Programs, Aptima, Inc., which specializes in military training simulation systems, human performance assessment, data management, and analytics to drive more effective and efficient training.

New Training Paradigms Depend on Data… and Lots of It. The US Air Force move to ‘Proficiency-Based Training’ to unlock the enormous potential of data to improve training and readiness—and the challenges posed.

The digital transformation unfolding across the US military is modernizing not just operations, but also training. New blended- and mixed-reality training technologies are being adopted to prepare for an increasingly complex and lethal mission environment.

But there’s a transformation in training doctrine too, with a shift from rote one-size-fits-all training to a personalized, data-driven approach focused on streamlining individualized training at scale to enhance readiness.

The US Air Force (USAF) is leading the way with numerous initiatives to leverage data, which it refers to as ‘Proficiency-Based Training.’ In Proficiency-Based Training (PBT), warfighter performance is measured, tracked, and compared against learning objectives. Gaps are then targeted for more efficient and effective training.

With its mission to be superior to any opposing force, the USAF’s shift to data-driven training is central to its goal of measuring and maintaining a qualitative edge.

Traditionally, fighter pilot training is managed by completing a set program of ‘checkboxes’ over a set period of time. But in a domain that is highly complicated and variable, that approach is very limited in its ability to capture (1) how well skills are learned, (2) how frequent maintenance training is actually needed, and (3) what training gaps exist.

Without data, instructors and learners are limited to relying on general-purpose curriculum and educated guesses that can lead to over-training in some areas and under-training in others.

With data, learners advance based on mastering the competencies and skills linked to their job or mission. Instructors have a proficiency record for each new trainee, making logistics easier and instructor expertise more potent. Deficiencies can be addressed through more tailored, individualized learning to close the readiness loop. And ultimately, commanders gain better insight into force and unit-level readiness.

Air Combat Command

Future Training Concept

Turning 0’s and 1’s into Actionable Information

While a data-centered approach may sound simple in concept, building a complete and accurate picture of warfighter competencies presents unique challenges as training environments become more sophisticated and distributed.

Most of the services are adopting new blended-/ mixed-reality technologies and live-virtual-constructive (LVC) training. These more realistic training and rehearsal environments produce a wealth of multimodal data, including system-generated data from simulators, radio communications, electronic chat, video, physiological measures, expert observations, and more. These data, in isolation, do not adequately represent proficiency. They must be mapped to knowledge constructs to provide actionable training insight.

Today, characterizing human performance is no longer a matter of simply collecting more data and determining its usefulness later. Rather, it’s evolved to fusing together more and more select data from diverse sources into what are referred to as ‘data lakes.’ A combined data set gives insights greater than possible with any one data source.

Solving the Data Integration Challenge

Data lakes represent a scale and variety of ever-larger amounts data. Because of this, they require strategies for their use in the high-volume distributed storage and processing of the learning enterprise, lest they turn into ‘data swamps,’ unusable islands and stovepipes of disconnected and disorganized information.

The Air Force Research Laboratory (AFRL) is at the forefront of human performance advances and leading the charge to solve the data challenge. A major push has been defining a common model for how human performance data is captured, stored, retrieved, and packaged in a way that can be applied to a variety of applications and high-velocity analytics, from machine learning and mathematical modeling to longitudinal trend analysis of training and readiness. The model accommodates different types of data (e.g., objective, subjective, and physiological) from disparate training environments.

By establishing an open standards-based framework, AFRL is codifying the past 20 years of training data that it continues to harvest, while paving the way for interoperability where training in the future will be increasingly driven by AI and machine learning.

Extracting Meaning from Data

To make sense of these data lakes for use in Proficiency-Based Training, AFRL has developed tools to build accurate pictures of warfighter proficiency and readiness. These include the Performance Evaluation & Tracking System (PETS™), which ‘listens’ to, collects, and analyzes data from simulated and live operations; the LVC Network Control Suite (LNCS™), a data visualization suite that produces a common operating picture of individual and mission performance; SPOTLITE®, a tailored automated grade sheet for collecting expert assessments; and SimMD™, a platform for systematically gathering the capabilities of the training environments themselves.

PETS, LNCS, and SPOTLITE provide trainees with improved debrief and feedback for more efficient and targeted learning. And for instructors and decision-makers, they provide quantifiable performance and readiness outcomes at the unit, wing, and wider Air Force levels.

These AFRL tools inform training decisions at all levels, from who needs what training, to where they can get that training most efficiently and effectively, to how that training will be delivered. PETS and LNCS act to ‘filter out’ unnecessary information to make training, and planning for training, easier, customizable, and adaptive. Reducing or eliminating overtraining and undertraining produces mission-ready warfighters faster and allows them to stay on the leading edge longer.

Using Data and AI to Close the Skills Gap

The move to new adaptive learning systems that tailor learning to the individual promise a more responsive and engaged experience than traditional knowledge management systems that only track the status or completion of static one-size fits-all courseware.

In the data-driven approach, AI and machine learning personalize training paths and provide real-time feedback and assessment based on the trainee’s current skills and performance. Algorithms precisely digest trainee knowledge and proficiency so as to recommend what learning should occur next, tailoring the sequence, the difficulty, and the type of content to accelerate time to proficiency and improve learning outcomes.

Algorithms also more accurately predict when maintenance training will be needed, and where it can be most efficiently and effectively obtained by operational warfighters, to enable a leaner, just-in-time training program.

While AI-driven adaptive learning systems are coming into their own, they will require putting these data lakes to use in meaningful ways to benefit learners and instructors. AFRL has infused a high level of scientific rigor in its development of tools and approaches to ensure that the measurements, algorithms, analytics, and interfaces provide high-quality information for trainees, instructors, and decision-makers.

Standardizing and validating these approaches helps ensure the technology is accurately assessing, predicting, and prescribing learning. Further, being government-owned not only protects the technology from the vagaries of unverified measures or unqualified claims, but also ensures it is free of vendor lock and is compliant with DoD modeling and simulation standards.

Ultimately, such data strategies in the Air Force, Navy, and elsewhere will enable commanders at a post, on a ship, or at HQ to have dashboard insight into the proficiency of their learners, with the ability to drill down to identify and remediate gaps.

In fact, the role of data is so central to the Department of Defense’s digital modernization strategy that DoD identifies it as a strategic asset. Common practices and open standards for collecting, organizing, securing, and accessing these data sets are critical to realizing the training and readiness advantages.

The Data Era Ahead

Today’s increasingly data-rich training and operational environments hold tremendous potential to enable more realistic and frequent training. However, they will require advanced human performance measurement capabilities that can leverage more and more diverse data to provide insight and informed decision-making at all levels to close the readiness loop.

Proficiency-Based Training represents a new data-driven era, and its promise is here today, deployed within the USAF. AFRL is fostering collaboration among government, industry, and academia, and its leadership with open architectures and data standards is critical to the innovation to fully deliver this potential to warfighters.

ABOUT THE AUTHOR

Peter Neubauer is Principal Software Engineer, Engineering Manager for Air Force Programs, Aptima, Inc., which specializes in military training simulation systems, human performance assessment, data management, and analytics to drive more effective and efficient training. Neubauer provides technical and project leadership for Proficiency-Based Training and works closely with US Air Force pilots, research and engineering teams to identify scientifically sound and technically sustainable solutions to emerging problems. Neubauer holds an MS in embedded systems and a BSE in computer systems engineering from Arizona State University.

The views expressed are those of the author and do not reflect the official guidance or position of the United States Government, the Department of Defense or of the United States Air Force.

The content or appearance of hyperlinks does not reflect an official DoD, Air Force, Air Force Research Laboratory position or endorsement of the external websites, or the information, products, or services contained therein.