The blower was broken, and I had to fix it. I could see the faulty part, highlighted in red, and was able to see how to repair it. The blower, in this instance, was a massive chunk of machinery in a wastewater treatment plant. And I was wearing a quite remarkable augmented reality (AR) headset from Microsoft called the HoloLens 2. It showed me, through a hologram over the physical hardware, what I needed to do.
It was a demonstration at MWC Barcelona in February to show off the HoloLens 2, which was launched by CEO Satya Nadella at the annual conference for the wireless industry. “This new medium is just the beginning of experiencing what’s possible when you connect the digital world to the physical world, to transform how we work, learn and play,” he said.
This is part of Microsoft’s new vision as a cloud-orientated company, a strategy it calls intelligent cloud and intelligent edge. The “edge” is whatever device is at the periphery of the cloud and connected to all those servers offering a range of services. “Computing is no longer confined to a device,” Nadella said, arguing that a wide range of inputs will replace keyboards and mice, from touch to speech and gestures.
The HoloLens is a powerful example of this, as multiple demonstrations of its capabilities showed. The original model was launched four years ago and was used by NASA for its Mars Rover mission. Because of the eight-minute delay in communications from earth to the red planet, controllers needed to map, plan and execute the path the rover would take, former NASA Administrator Charles Bolden told me last year.
I had seen the tourist version of this during a trip to Seattle earlier, which depicted images of Mars so you could see the surface of the planet. But NASA used it for planning purposes, Bolden explained, using imagery captured by the rover’s cameras to rehearse its route and make sure it didn’t run into obstacles or get stuck in crevices.
This year’s model, which cost US$3,500, is lighter, balances its weight better and has a clever mount at the back of your head. The entire computer that runs it is encased in this lightweight frame and gives you what Microsoft calls “mixed reality”.
Microsoft demonstrated numerous applications where using the headset has been helpful in real-world scenarios. Instead of reading a paper-based manual, the HoloLens allows a technician assembling the complicated interior of a sleeper truck to see a holographic overlay of where individual parts are placed. Or, when a building is being constructed, architects and engineers can see where real-world problems might occur when conduits were moved, and other pipes would not have fitted into the ceiling space.
Quoting Mark Weiser, the chief technologist at the renowned Xerox PARC research institute, Nadella said: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
The HoloLens 2 has much more accurate tracking of your hands – and uses eye gestures like looking down to scroll through pages of text or voice recognition when you give it verbal instructions – and even gives you haptic feedback when you interact with holograms. I was amazed at this when I experienced it. It really is a game changer.
“For the first time, you’re going to feel what it feels like to touch a hologram,” said Alex Kipman, a technical fellow in Microsoft’s Cloud and AI group and one of the original HoloLens inventors, “to interact with a hologram and to play with it, almost where you forget that this is a piece of digital content you’re looking at as opposed to it just existing in the real world”.
Along with the HoloLens 2, Microsoft unveiled another upgraded device that is infused with the same cloud intelligence, the Azure Kinect. The first Kinect was a motion tracking bar that rested on top of the Xbox gaming console and was used for tracking a player’s activity. Instead of using a handheld controller, you used your body and the console monitored your movements. From games, it started to find other useful applications, including in operating theatres where surgeons could use gestures to page through textbooks while in surgery – without having to physically touch anything and therefore contaminate themselves.
The Azure Kinect is a cloud-connected camera and sensor device that can track when patients might fall out of bed and a range of other useful things.
As Nadella said: “When you change the way the way you see the world, you change the world you see”.
This article first appeared on Financial Mail on 21 March 2019