Simulating the World, Faster

MS&T’s Special Correspondent Andy Fawkes reports on some of the latest developments in the processing and exploitation of real-world terrain data.

The Challenge

Updating simulation and operational systems rapidly with the latest terrain data is vital in support of multi-domain operations.  In the past this process has often been expensive and labor intensive.  However, with the proliferation of commercial and government sourced data, and more capable networks and AI-driven software, updating systems can now be significantly quicker and less manpower intensive.

Recognizing customer demand and technology trends, Unity has established its Geospatial Group, led by Sébastien Lavier, with the goal "to make it as easy as possible to automatically generate, stream and render large, complex synthetic environments based on any kind of geospatial data." Lavier told MS&T "geospatial is front and center as a challenge for most of our customers" and that his team was working to bridge the world of GIS and real-time 3D and “democratize” the exploitation of geospatial technology and data for its clients.

Parallels

Unity is tackling the geospatial challenge for its customers across sectors. Lavier noted that "there is a lot of interest today in Digital Twins and the Metaverse, but geospatial remains front and center as a challenge for most of our customers." One example of such a customer is the architectural, engineering and construction (AEC) sector who need to achieve "rapid reality capture", as Lavier put it, of the latest 3D state of a construction site. Such data can then be exploited within Unity for multiple uses including safety training and construction decision support. This rapid creation of simulated 3D environments based on the latest real-world data is very similar to the military customer requirement to update its training and mission support simulation systems. "It's essentially the same problems that we're trying to solve" Lavier told us, to "capture their environments and then use this environment to build scenarios on top of it."

An example of Unity’s geospatial capabilities: round-Earth (WGS-84) streaming and visualization using 3D tiles in Unity.

A Soldier and Backpack

With its very large user base, ease of use, deployability across multiple hardware platform types and new geospatial features, Unity is in a strong position to support clients who require an agile geospatial capability. One such client is the University of Southern California’s Institute for Creative Technologies’ (ICT). With Unity as its simulation core, ICT’s Rapid Integration & Development Environment (RIDE) prototypes ideas and techniques at pace in support of DoD simulation initiatives, including the US Army’s One World Terrain (OWT) project.

ICT’s Director of Modeling & Simulation Kyle McCullough spoke to MS&T about their terrain efforts. ICT has been conducting geospatial research for over 5 years and although a key use case remains training it has expanded into what McCullough termed "reality capture" for mission preparation. Developed with Lockheed Martin for the US Marines, the Tactical Decision Kit was one use case concept described by McCullough. Here a single soldier equipped with a backpack and a drone, laptop and tablet captures an area and together with a "fully automated pipeline" the terrain data is processed automatically and loaded in Unity simulation software for subsequent rapid decision support and line of sight analysis.

A Fully Automated Pipeline

The concept of a fully automated pipeline for terrain data is a key challenge that ICT is working on for OWT and other DoD stakeholders and involves the collection of terrain data through to processing, storage, distribution, and its runtime use. "To support a usable application without any expertise or training" is the goal McCullough told us, and requires advances in technology such as in machine learning and simulation technology.

To research such a pipeline, ICT have developed STPLS (Semantic Terrain Points, Labelling System) which takes raw output from the photogrammetric process and generates semantically classified data such as individual buildings and roads, and data model information such as whether a building is made of concrete. McCullough explained that this data is loaded into Unity and STPLES "can start to define everything within that space" and even support AI units navigating the space. Further, as weather or blast events are triggered, the data model is dynamically updated, and the simulation and AI behavior changed accordingly.

ICT is developing technology to automatically classify terrain data for use within simulation systems.  Here the roads and buildings at the USC Wrigley Marine Science Center have been detected and used for line of sight analysis and routing AI-controlled entities in Unity. Image credit: ICT.

McCullough told us that key to such research was Unity as it brings ease of use and rapid development to ICT. We benefit from "years of work" using the same engine and "as the foundational engine it gets us 90% of the way there and then we build a 10% layer that has the DoD simulation functionality on top and we're able to very quickly generate demonstrations like this," McCullough said.

Training the Machine

Whether it be a photogrammetry or LIDAR scan, humans are good at "figuring out where the trees are, where the buildings are, where the roads are," whilst computers currently are not, Lavier told MS&T. The creation of simulation databases hence still requires humans to classify some terrain objects such as a window’s glass pane. However, with the latest developments in machine learning this requirement can be reduced or eliminated. The data used for the machine learning is generated in Unity simulation software and the so-called synthetic data is used to train the classification system and hence automate the process.

The prospects of such an automated process are not far off as McCullough told us "if we were still looking at the classical method of going out flying a drone, hand annotating, collecting, training, I think we would be potentially 5-10 years off." However, "with synthetic training data, which we've shown to be as effective as the real-world data, I think we're only a couple years away from having a fairly robust solution," McCullough explained.

Beyond automating terrain data processing, ICT is looking to the other data necessary to support multi-domain training and operations. According to McCullough this "includes weather data, subterranean data, the electromagnetic spectrum, you need all of these other such non-physical terrain features … these are really big challenges."

This article is sponsored by Unity Technologies (www.unity.com/solutions/government-aerospace) in support of its partners.  Contact Unity's sales team to learn more about Unity's dedicated business and technical support for government and contractors. See Unity's solutions in-person at  I/ITSEC 2021, including a geospatial large-scale disaster response simulation demo created in partnership with Amazon Web Services (AWS) and Lockheed Martin Skunk Works®. Learn more about the Institute for Creative Technologies at  ict.usc.edu.