Measuring and Evaluating – With Greater Purpose

Group editor Marty Kauchak reports on the military services’ quest – with industry’s support – to raise the performance bar on their training measurement and evaluation capabilities.

Defense departments have been measuring and evaluating training events since well before this contemporary era. The major news and developments in this learning space find military services sprinting toward more rigorous, technology-based practices and methodologies to certify individuals, units and staffs as mission ready. Of particular significance, training organizations and their industry partners are increasingly using technology enablers familiar to MS&T readers, including artificial intelligence (AI), machine learning (ML) and others, to more quickly, accurately and comprehensively, measure and evaluate training audiences.

Beyond “Check the Block”

Captain Tim Hill, commanding officer of the Naval Air Warfare Center Training Systems Division (NAWCTSD), provided one service perspective on these trends, first calling attention to a “great deal” of senior level interest in better understanding the effectiveness of any training/training system. “We have always measured performance of the individuals/teams that are completing training periods in some way. But the manner and degree to which that is done is always an individual decision for each program or capability,” the Orlando-based community leader added.

The NAWCTSD commander continued his service overview, pointing out in the current/future environment, his command is seeking to harness emerging technologies, such as AI and ML, to measure performance in new and better ways. He explained, “We have an opportunity to measure performance at a much deeper level by collecting data that is resident in our learning management and training systems and analyzing it in new ways with this technology. This will allow us to measure individual or team performance, for a given event and collect trend data for these individuals and teams.”

Further the collected data can be compared to standards to rate the proficiency of those individuals and teams; but this will also allow subject matter experts to begin examining trends across the entire population of individuals and teams in the Navy, so that the service can better understand proficiency across the entire fleet.

“This has the potential to allow us to move from a ‘check the block’ training mentality, where we assume operators are skilled because they completed a training event, to an environment where we’re examining true warfighting proficiency,” Hill emphasized.

Beyond the US Navy’s quest to better understand fleet-wide proficiency and gain other outcomes, more quick-paced developments are occurring.

Also Trending

Cervus Defence’s rapidly expanding client portfolio includes military services in the UK, US, the Netherlands and the Middle East. Alan Roan, the UK-based company’s managing director, reflected on this diverse customer list and noted the main, current, common requirement among his varied customers was descriptive analytics. “A lot of our customers are seeking a better understanding of what they have just done. From a training activity, they want to capture the data and very quickly turn the data around, get it analyzed, and presented back to the trainee through an after action review or part of the learning experience.”

The veteran owned data-analytics company is moving beyond this common datum point, using AI and some of the data science tools, to reach a higher plateau of predictive and prescriptive analytics. This journey allows leaders to take and compare existing data sets against a recently completed event. Roan explained “You can start forecasting where those individuals and units are in terms of performance, and even what might happen if they do different type activities.”

Indeed, one Cervus customer, the US Marine Corps, is starting to explore how to use analytics to forecast where its readiness gaps are situated. “On the workforce analytics side, it can be used to see where Marines aren’t ready to deploy and do their job, and they can start to consider training systems to give them a more efficient use of training.”

Another rapidly emerging trend finds military customers starting to aggregate data. Whereas, heretofore, one training system delivers analytics for that specific system, multiple training systems, from different simulation environments, are delivering data in a common format into one repository. One prime example is travel and related cost-avoidance data for exercise participants. Roan noted, the cost data applied to the training data can provide a much more accurate cost for training readout. “And you can apply other data sources – workforce analytics, engineering analytics and others, on top of those – you are combining these and getting an enterprise approach to data. This is a much more holistic use of these data sets, by joining them together,” he added.

A third emerging defense department requirement trend moves beyond traditional military performance, toward performance for individuals, teams and tasks. “These are often psychological measures,” Roan observed, and noted that the UK military, in particular, has a sharpening focus in some of its training programs in team performance, relationship measurements, how to make teams perform better on specific tasks, and other focal points.

Below these overarching perspectives, other projects and activities are being funded and supported in military services’ programs of record to advance the state of training evaluation and measurement.

4C Strategies consultants support military exercises and evaluations worldwide with the company's Exonaut Readiness Management platform.
Image credit: 4C Strategies.

Meeting Requirements

Providing one user case in the maritime training domain, Capt. Hill, further recalled his command is “aggressively searching” for opportunities to embed performance measurement in training and training systems it is developing. To point, “The Sailor 2025 Ready Relevant Learning” initiative is a great example of where we are building the infrastructure for wider capability that will come to fruition as the associated technology matures. That initiative is a very structured approach to modernizing all accession training for the Navy and will build in the ability to measure career-long learning/proficiency. We also look for opportunities to include similar efforts in new developments and upgrades across our portfolio.”

Industry is also offering enhanced and new products to meet this and other service expectations and requirements.

One instance of rapid evolution can be gleaned in 4C Strategies’ Exonaut® Training & Readiness Management Platform. Martin Rusner, the company’s Head of Military Products, noted the platform’s development has taken place in partnership with some of the world’s leading defense forces to respond to an evolving battlespace and more sophisticated training needs. Rusner then presented the business case to invest in Exonaut, noting the platform relied primarily on capturing human observations to support and ensure a degree of objectivity to formal training assessments made by commanders and external assessors.

“In most training environments these observations will continue to be important to add context and quality. However, today we also recognize the need for automated data collection and AI-supported assessment, and training recommendations to simplify and further enhance the training assessment process as well as the complete training management cycle,” he added.

Describing Exonaut’s technology underpinnings, Rusner said, in part, that to enable automated data collection, the platform runs in a fully containerized environment (either on site or in the cloud or as hybrid), and allows evaluators to get close to the point where they need to interact with the data, often in an “edge node.” As such, Exonaut can be integrated with numerous performance data sources in an edge system of systems environment, including: data feeds from real-time monitoring sensors through the Exonaut Integration Engine, linked to training objectives/tasks (biometric data, movements, consumption, etc.); data feeds from simulation and mission command systems using, for example, geofencing and automated triggers based on training objectives/tasks associated with an order of battle; and UAS feeds when training in live environments to enrich both situational awareness and AARs.

Turning to methodology, Rusner explained 4C’s evaluation approach, is to allow for multi-tiered evaluations and assessments. “Step one means that observations and data are continuously captured and correctly contextualized and associated with the training audience’s training tasks/training objectives. In step two, observations and captured data supports both the production and performance/delivery of improved AARs and the production of a formal assessment.” He added, “In this way you ensure the alignment of the training feedback loop with troops and the formal assessment made by commanders or external assessors, which in turn drives step three, which is future training adjustments as well as decision support to senior command.”

Throughout each step in 4C’s evaluation approach, visualization and analytics, with multiple layers of information that can be turned on and off in a user-friendly format, are instrumental. This needs to be achieved while keeping sensitive training and readiness data secure. Exonaut’s architecture and security framework has allowed it to be accredited to manage classified data by several armed forces, including NATO. Furthermore, with a holistic training management approach and Exonaut integrated with content, including simulation-generated data, added value can be gained by allowing evaluators capturing observations to use Exonaut mobile applications to automatically generate bookmarks for use in AARs.

4C continues to expand automatic measurement and assessment capabilities beyond individual training – directly linking them to tasks and objectives for increasingly important multi-echelon and/or complex collective training events. Rusner emphasized that while Exonaut currently supports automated readiness report algorithm recommendations, using more sophisticated AI and ML brings it one step further along in our effort to automate recommendations for commanders. In the end, warfighting is an art and Exonaut can support the commanders with some of the science behind it.

4C Strategies supports several automated readiness report algorithm models, among others, in the US, the UK and Sweden, as well as NATO.

One significant 4C user case is the company’s contribution to the US Army Synthetic Training Environment (STE) and its Training Management Tool (TMT). Rusner pointed out Exonaut is a core component in the TMT, which is currently under development by VT MAK. As noteworthy, 4C has, during the last 18 months, been awarded several Other Transaction Authority contracts within the STE program.

In a second industry advancement, FlightSafety International (FSI) received a contract in November 2019 for FlightSmart from the United States Air Force Air Education and Training Command (AETC), for implementation of FlightSmart at Columbus AFB, Mississippi on 16 T-6A training devices, including Initial and Operational Flight Trainers. Chris Starr, senior product manager, at FSI, told MS&T that under the contract, the service requires after-action performance feedback through a web-based interface. FSI brings to this recent contract its competencies in measuring training performance and evaluating outcomes, in multiple US Air Force programs, including those at Air Combat Command, Air Force Special Operations Command and Air Mobility Command.

Starr explained that FlightSmart will help users to understand why students may, or may not, be struggling, with a sampling of the most important training tasks in a syllabus. He added, “Furthermore, it can provide: insights into a student’s strengths and weaknesses; alerts for performance deviation(s); and individual- and population-level insights – all while incorporating machine learning to predict probability of student success. FlightSmart can reduce student attrition and increase training throughput.” Of added note, FlightSmart was also designed with the instructor in mind, specifically enabling “instructors to proactively address any deficiencies by optimizing the training curriculum while focusing on areas that need improvement, as opposed to repetitive actions rooted in a fixed syllabus.”

Beyond this Air Force use case, other requirements FSI has received from government and commercial customers in this learning space, include the ability to: evaluate the performance of the entire crew (including sensor operators and combat systems officers, and others); reduce requirements for additional training necessary to achieve successful course completion; evaluate total training program efficiency; support the transition to Evidence-Based Training (EBT) and Competency-Based Training and Assessment (CBTA), and the efficacy of their safety management system; and change management processes.

Analytics and Data Storage as a Service

Driven by military end users’ persistent demand, the pace of community-wide practices and requirements, and their enabling technologies, will continue to evolve in the next 24 or so months.

At the top of Cervus’s Roan’s list of evolutionary trends was continued aggregation, whereby more training data sets will be connected to other sources of data. Outcomes from this effort will allow training organizations to measure mission stress and physical loading during a training event. “But this will go much further than that – toward the ‘internet of things’ as military systems become more connected. And when you are doing training or are in operations, we’re going to capture all of this data and we will need to figure out formats by which we can extract, align and store it so we can start doing analysis against it.”

Other concurrent technology enablers to bring Roan’s vision to reality will include increased, remote computing power, quantum computing and cloud-based approaches to allow storage, analyze and process data. Roan concluded, “These will lead to a change in the market – a disruption – reducing costs a lot and giving increased access to smaller companies so they can start putting and building their applications on these cloud-based systems. Things will also go to more of a service model, with analytics, storage or other capabilities as a service.”

At the product level, FSI’s Starr first revealed his product team is actively collecting data within FlightSafety International training centers to train and mature its algorithms. And beyond this activity, other enhancements are planned to keep pace with the rapid movement of technology changes. “We are seeing advances in objective training outputs daily and discovering new ways to best communicate those outputs to our customers,” Starr noted. For instance, mediums by which pilots train are continuing to evolve and FSI is adapting to those needs. “Whether it’s digital, interactive courseware, or partial task trainers that harness advancements in AR, VR and/or MR, we’re posturing our FlightSmart technology to ingest data across all categories of learning to affect the entire training ecosystem.”

An additional, key focus area for FSI during the next 12-24 months is to expand the FlightSmart product over multiple airframes, for both commercial and government customers. Starr concluded, “We expect the feedback that we receive from our customers, CBTA and EBT will become further entrenched in aviation training and simulation.”