The worlds of warfare and virtual reality are an appealing combination. Millions of fans of Call of Duty would no doubt jump from their sofas at the chance to immerse themselves in a 3D version of the game, given the right kit.

But the serious side of this combination is appealing too. Using virtual reality to train soldiers and other military personnel is a much safer (and cheaper) option for governments and their defence organisations. So what does the future hold for virtual reality technologies and their military applications?

Every year, the world’s defence simulation industries come together at international events to show off their latest products, each vying for the attention of the browsing military supremos. ITEC and DSEI are huge exhibitions where defence procurers queue side by side with developers and high-tech enthusiasts, hopeful of experiencing the latest (alleged) advances in virtual reality military simulation.

It is hard not to be carried away by the marketing hype produced by defence simulation companies. But standing back and reviewing exactly what is on show, I often find myself questioning how much of this high-tech pizzazz will ever deliver real benefit to members of our armed forces. I suspect that in the unforgiving world of the military, the stark answer is very little.

The vast majority of products on show are, in truth, unsuited to the realities of military training or live missions. The one essential ingredient that always seems to be missing from these so-called advanced simulations is clear evidence that real human factors have played a significant role in their development. Instead it soon becomes clear that human elements have been ignored in favour of staging visually appealing demonstrations that do little more than show off the latest “must-have” hardware and software.

But not all the reality can be virtual. I was directly involved in two projects which demonstrated how the blending of the virtual with the real – “mixed reality” – could deliver a far more believable training setup than one relying on virtual reality technology alone.

The applications in these two projects involved a new close-range weapons trainer for the Royal Navy, and the development of aircraft situational awareness and communication skills for helicopter marshals in the RAF.

Both projects exploited significant real or physical objects in the quest to make the virtual features of the simulation more believable. Inert but real weapons were used in the naval project and a wooden helicopter door frame mock-up for the RAF one. Both demonstrated improvements in trainee performance, as well as significant financial savings for the defence sponsors in estate costs, ammunition and the need to tie up real vessel or aircraft assets.

A more recent example was the development of a low-cost simulator supporting the training of remote operation of the UK’s latest unmanned bomb disposal robot, the Cutlass. The need for a simulation-based trainer for Cutlass was obvious, especially since the prohibitive expense of repair meant that real robots could only be made available to trainees on limited occasions under strict supervision.

image-20160601-3253-1773c9y — The Cutlass and its virtual training console. Author, Author provided

There were many, myself included, who felt that the Cutlass operator’s console – in particular, the cluttered, multi-menu, screen-based interface which featured small windows relaying images from the vehicle’s cameras, was overly complex and could cause problems. Ditto the six joysticks governing vehicle driving and multi-axis control.

Initially, it was believed the complex skills necessary to control the Cutlass vehicle could be delivered using a simple, software-only virtual environment. One US organisation even produced such a simulation using Second Life, the online virtual world. However, from a human perspective, it became very clear that such a low-fidelity solution would be doomed to failure, as there was no way of guaranteeing transfer of system specific knowledge and skills from the virtual to the real.

Let’s get real

So the decision was taken to use a mixed reality solution and produce a low-cost physical replica of the operator’s console to accompany the high-fidelity graphical environment in which the virtual Cutlass could be deployed.

The fundamental human-centric lessons learned during these projects today form the basis of the latest virtual, augmented and mixed reality demonstrators under development by the Human Interface Technologies Team at the University of Birmingham, including an advanced command-and-control station, and a brand new mixed reality training system for future defence medics which will benefit from the interplay between real-world objects and virtual reality reconstructions of the interior of a Chinook helicopter.

image-20160601-1951-2n4il0 — Mixed reality training.

Without such an approach they stand very little chance of being successfully exploited in actual armed conflicts. And with today’s emergence of highly speculative interactive technologies, those same principles must be applied rigorously in all projects seeking to exploit virtual, augmented and mixed reality. Especially in matters of war, when life and death situations are all too real.