US Army Researchers Envision Cross-Reality Future

Army researchers laid out a plan that uses cross-reality technology to establish a common operating picture for soldiers. The system uses cross-reality, a form of mixed reality that interacts with the physical world, to ensure access to the right information at any time.

Dr. Mark Dennison, an Army research psychologist at the U.S. Army Combat Capabilities Development Command’s Army Research Laboratory, and Dr. Christopher Reardon, an Army computer scientist at the laboratory, partnered with Stormfish Scientific Corporation to design a common operating picture framework that keeps decision-makers at all levels properly informed. The team recently published their research at the Virtual, Augmented, and Mixed Reality (XR) for MDO Conference at SPIE, a technical society dedicated to advancing the scientific research and engineering applications of optics and photonics.

“A common operating picture is the concept of one interface or system that can incorporate all the information that occurs from a particular operation or mission,” Dennison said. “It should be something  that every soldier or commander can look at and get their perspective on everything that is happening — where are my friendly forces, where is the enemy, what are our planned movements and so on.”

The advancement of technologies that enhance situational awareness prompted the Army to improve the communication network that facilitates the exchange of information; however, soldiers require different types of information at different times.

“The problem is that the idea of a common operating picture that fits all needs, in all settings, doesn’t really make sense,” Dennison said. “The amount of autonomy that soldiers will have in future conflicts is only going to increase. Since individual soldiers have more power to make decisions, the information they need is going to be unique for their situation, essentially an uncommon operating picture.”

As a result, Dennison and his colleagues suggested the idea of a user-defined operating picture, where each individual’s common operating picture differs slightly depending on their preference, environment, role and mission.

In order to customize and enhance how each command level receives and communicates information, the team integrated cross-reality technology into their design, which would present the relevant data to users in an easy-to-understand format without overwhelming them.

“The difference between cross-reality and, say, mixed reality is that interactions with virtual objects in cross-reality will actually affect things in reality,” Dennison said. “I might have a virtual hologram of a light switch, and if I touch it, it will turn on an actual device in the room. It’s this notion of what you’re doing in one reality or domain crossing over to the other and vice versa.”

Cross-reality technology has only recently begun to emerge in research as another avenue for information presentation and manipulation, but Dennison emphasized the importance of how cross-reality concentrates on the link between the cyber domain and the physical domain.

“We believe that cross-reality will enable soldiers at tactical echelons, as well as at higher levels, to interact with information in ways that are very different than the traditional approach,” he said.

According to the researchers’ investigation, virtual reality, augmented reality and cross-reality will all play an important role in the Army’s future information network, where each will assist soldiers in various ways depending on their role and position.

“For example, we are unlikely to see virtual reality being used at the tactical edge, because it completely blocks your eyes, which is a non-starter when you’re in combat,” Dennison said. “But a mixed reality platform like the Army’s Integrated Visual Augmentation System makes total sense to have down at the tactical edge because you can see through the holographic display.”

While vision-occluding virtual reality headsets may never be incorporated into gear for close-quarters combat, Dennison explained that virtual reality may play a more prominent role in a tactical operations center or a command post since commanders and support staff can afford to completely immerse themselves in the data.

Overall, the researchers conceptualized four interrelated factors that they believe the Army should consider when implementing cross-reality:

“These ideas are grounded in a new field of study called immersive analytics that investigates how technology supports analysis and decision-making,” Dennison said. “We developed these four factors as a kind of concept roadmap to consider as we move forward in this space theoretically.”

At the moment, the team’s research is in its early stages. Army researchers created a prototype system to explore these concepts and experiment with a cross-reality common operating environment. As part of the theoretical work, they have also conducted limited empirical experimentation with soldiers to obtain their feedback.

“From a pragmatic perspective, the first big problem to solve in creating a common operating picture is interoperability, which means we need to find a way to get different devices to talk to each other,” Dennison said.

The researchers plan to present elements of their work and prototype system at upcoming technical demonstrations with the Army’s Network Cross Functional Team later this year.