How Relevant is the Metaverse to Military Simulation & Training?

The metaverse is in the news everywhere, but how relevant is it to military simulation and training? MS&T’s Special Correspondent Andy Fawkes reports.

Over the last year we have seen an explosion of interest in the metaverse, not just in the tech community, but in the wider media. Social media giants such as Facebook have changed their name (to Meta), and business consultants such as McKinsey claim “the metaverse is too big for companies to ignore”. Accenture state that “it will revolutionize nearly all aspects of life and business in the next decade”. Like any grand vision it will have its enthusiasts and detractors but the metaverse is an important topic right now and many of its themes and technologies chime with military simulation and training.

Inevitably the first question asked is what exactly is the metaverse? The term originates from Neal Stephenson’s 1992 science fiction book Snow Crash and is an imaginary “computer-generated universe” with many participants, typically (but not necessarily) accessed by “goggles” providing 3D imagery and sounds. Over the last 30 years the word metaverse has been used principally by technologists as inspiration for the future of computing but now with its broadening appeal (and sometimes repulsion) its meaning is evolving depending on the perspective of the person or organisation. This is not that unusual, as defining related terms such as “digital twins” and “cyber” is still ongoing.

The definition or description of the metaverse appears to be in two schools, one which sees it as a “massively scaled and interoperable network of real-time rendered 3D virtual worlds”, such as defined by the influential metaverse writer and commentator Matthew Ball in his 2022 book, The Metaverse And How It Will Revolutionize Everything. Others see it as virtual worlds but also with more of an extension into the real world. For example, at Microsoft Ignite November 2021, its annual conference for developers and IT professionals, CEO Satya Nadella said “the metaverse enables us to embed computing into the real world and to embed the real world into computing bringing real presence to any digital space”.

A Military Metaverse?

As for a “military metaverse”, there have been a number of recent articles attempting to describe it and discuss the opportunities and challenges ahead. In the absence of, say a NATO definition, we might suggest that it is the enterprise-wide persistent networking and integration of live, virtual and constructive simulations which is a long-standing vision. Whatever a military metaverse is or is not, and whether current-day metaverse vision(s) will be fully realised, we can nevertheless report on metaverse related trends and enablers.

Immersion

Militaries such as the British Army and USAF are seeking more immersive training environments to improve the feeling of presence for the trainee(s) and ultimately improved learning efficiency. eXtended Reality (XR) devices can be a powerful way of immersing trainees, particularly virtual reality (VR) devices which can completely immerse trainees, at least in terms of their visual experience. This is an area of technology that is advancing very swiftly, and it is challenging to report on let alone make hardware choices. As computing power, sensors, networking, AI and visuals all improve, new headsets are emerging and at the time of writing the Meta Quest Pro and Pico 4 Enterprise have been announced, aimed at enterprise and professional users. Both include face- and eye-tracking, with HTC offering these features as extras. Such capabilities are touted to improve user experience and support more realistic avatars but can raise privacy concerns. In the AR space, Magic Leap also recently launched their second-generation device aimed at the professional market. Efforts to bring AR into the military market have hit the headlines through the US Army IVAS capability based on Microsoft HoloLens technology. It was reported that 5,000 units were ordered in September 2022 but there have been negative headlines of poor user experience.

XR Required?

Gaming illustrates that wearing headsets is not always necessary to interact with more 3D and interactive digital worlds. Some companies are focusing on ease and ubiquity of access to the metaverse which for defence can be a key advantage with its diversity in IT hardware. Exploiting existing internet JavaScript technologies and skills, web-based Croquet OS supports 3D apps for any device. Croquet founder and 3D and XR pioneer David A. Smith wrote recently, a “web-based metaverse enables any world to be connected to any other world on the web, no matter who developed it or on which web server it is hosted”. Croquet also provides “portals” that allow for users to easily move between virtual worlds and Smith claims “portals will transform the metaverse in the same way that links transformed the web”.

Persistence

Matthew Ball is among those metaverse commentators that see it as “persistent”, so in Ball’s words “it never “resets” or “pauses” or “ends”, it just continues indefinitely”. This concept can lead to the impression that we will all be living our lives as avatars 24/7 but even Snow Crash’s protagonist Hiro spends time also in the real world. Perhaps what is more relevant to any military metaverse is the potential for the persistency of data such as terrain, together with accessibility to a persistent network of live and virtual worlds that the warfighter can exploit 24/7 for training and experimentation.

Blending Real and Digital

In his special address at SIGGRAPH 2022, NVIDIA CEO Jensen Huang stated “the metaverse is the next evolution of the internet… where humans will portal into a virtual world with XR devices while AIs will portal out to our world as physical robots”. Although Snow Crash had robots in both the metaverse and the real world, Huang’s claim seems to extend Stephenson’s original vision with today’s metaverse controlling real-world robots. Nevertheless, this real world and metaverse interaction should have utility for the military. Already researchers are training robots in simulation ahead of real-world situations never encountered before. For the training of humans, a proliferation of real-world biometric sensors connected to the metaverse should enhance the training experience and associated analytics.

Digital Twins

In a recent blog post, information and communications technology giant Fujitsu foresees the convergence of digital twins and the metaverse with humans operating machinery remotely and making consequence-free errors while experimenting with new designs and settings. “Anyone can realize their potential in a virtual space, for example, by using remote robotics to see the physical world or by using technology to pursue advanced education”, Fujitsu claim. Similarly in a military metaverse, military personnel might explore new concepts collaboratively with designers and analysts and remote operate equipment, including maybe one day, warships.

It is not only physical entities that will have digital twins. A key feature of the metaverse will be avatars which are graphical representation of a user, sometimes actual, sometimes fictional. As the metaverse and associated standards develop these avatars will move from one virtual world to the next together with its assets such as clothing. In a military context, some foresee avatars or digital twins of personnel being transferable from one simulation to the next providing a digital record of performance that can be exploited in training and analytics.

Artificial Intelligence

Improved AI and machine learning are key capabilities of the metaverse, and we are seeing significant advancements. As examples, Meta researchers are using ML to predict the movement of a human’s whole avatar based only on the movement and position of the XR HMD. Companies such as VRAI, Varjo and MVRsimulation (formerly MetaVR) are also working to exploit data generated by XR technologies to improve military training evaluation. Training evaluation will always have a degree of human input, but the ability to automatically capture and analyse a whole range of data will likely only become more powerful on the back of wider metaverse-related AI advancements.

The Cesium for Unreal plugin supports Epic Games' vision for an open Metaverse. Image credit: Cesium.

Improved Connectivity

Although not highly visible, networking will be a key enabler of the metaverse and its core features of bandwidth, latency and reliability. MS&T recently wrote about Cloud developments and as networking improves the ability to place the computing power wherever it is best placed will be enhanced. This might be minimising XR HMD processing by conducting it in the Cloud over low-latency 5G, or streaming up-to-date data such as terrain and weather from the Cloud through to virtual worlds and games. Defence can similarly benefit from enhanced networking for its simulation systems, but this will only happen if emphasis is placed on advancing its network capabilities and solving security issues.

Interoperability

The current thinking of influential metaverse protagonists is that standards will be required to permit avatars and digital assets to move between virtual worlds. As illustration, Mark Zuckerberg said in an interview, “you have your avatar and your digital goods, and you want to be able to teleport anywhere… you don’t want to just be stuck within one company’s stuff”. Such thinking has inspired the launch of the Metaverse Standards Forum in June 2022 which by August had over 1,500 members large and small. It plans to coordinate and foster standards leaving it to existing organisations to develop standards. With such effort being put into standards, defence will have to at least monitor such developments and exploit as necessary for its simulation systems.

Challenges

Inevitably, with such bold visions there are many hurdles to be overcome before the full potential of the metaverse can be realised, if ever, and no doubt new ones will emerge over time. One example is culture change. The Covid pandemic made video calling a mainstream activity; the technology was there before but the pandemic forced culture change. Whether a similar scale event is necessary to force us more into 3D worlds only time will tell. However, it is worth considering metaverse-like experiences such as Minecraft and Roblox are already mainstream activities for young people.

Another area of concern in many quarters is data privacy. XR technologies for example can generate considerable volumes and diversity of data that researchers have shown can be exploited directly or inferred, providing key personal data attributes such as gender, body dimensions and fitness. Of course, in a military context this could be valuable for training but there appears to be a long journey ahead in terms of the people understanding what data they are giving away in the metaverse.

A More Three-Dimensional World Ahead

MS&T endeavoured in 2020 to bring the metaverse to the attention of readers in the article Has the Military Been Building the Metaverse? Our article suggested that the military had already been developing an early form of the metaverse, or “proto-metaverse”.

What we did not foresee was the explosion of interest and investment in this topic since and there is now an almost overwhelming amount of metaverse news. But, like in 2020, the metaverse and a future military metaverse appears highly relevant to simulation and training with many opportunities, and some challenges ahead. The real world is three-dimensional, and increasingly the digital world will be too.