Microsoft has unveiled an impressive new mixed-reality platform called Microsoft Mesh that lets people “share” the same hologram using its HoloLens 2 headset.
The augmented reality (AR) headset is a technological wonder in of itself: A lightweight frame that fits comfortable over your eyes and has the entire computer built in (mostly in a largeish curved case on the back of your head).
The lenses depict AR images, or holograms, that are amazingly good, while sensors in the eye sockets track your eye movement – so, for instance, if you’re reading a page (of instructions or just email) as your eyes move down, it scrolls the text up so you can seamlessly read it. Other sensors track your hands as they move – and allow you to move objects in AR or play the piano, in one example. And the headset does this without additional controllers in your hands or clumsy tagged gloves.
It’s an amazing experience I discovered on a trip to Microsoft HQ in Seattle a few years ago, and at the launch of the HoloLens 2 at MWC Barcelona in 2019.
I’ve been a tourist on Mars and fixed a huge industrial metal machine (where the broken part was visible over the actual physical machine, and I could see where to adjust it and screw it back into place). The use cases are quite fantastic. As the son of an architect – Hi Ma – I know how much goes into conceptualising a building, or a factory. One current commercial use is for construction where architects and engineers can plan buildings in a remarkable new way. Microsoft calls it mixed reality, which is an apt description of being able to see the real world, but with an overlay of useful information, like visual instructions on how to fix a broken machine or how to put together the cab of a transport truck.
It’s was also used by surgeons last month to demonstrate to their colleagues around the world their techniques and procedures. Visualise a patient lying on an operating table, with a surgeon (wearing a HoloLens 2) seeing bones, blood vessels and organs overlaid on their body.
South African surgeon Professor Stephen Roche – from Groote Schuur Hospital and the University of Cape Town – was one of 15 surgeons from 13 different countries who performed 13 mixed reality orthopaedic operations.
I had to explain just what a wonder the HoloLens 2 is because its infinitely more powerful when it’s part of a network – and that network can create a shared, group hologram nogal.
Microsoft Mesh is a brilliant way to tie all of the HoloLens’s individual functions into a remarkable system that “allows people in different physical locations to join collaborative and shared holographic experiences”.
Everyone is in the same hologram, and can able to interact with each other.
“This has been the dream for mixed reality, the idea from the very beginning,” said Microsoft Technical Fellow Alex Kipman. “You can actually feel like you’re in the same place with someone sharing content or you can teleport from different mixed reality devices and be present with people even when you’re not physically together.”
Microsoft launched this concept by staging its annual Ignite conference, held virtually because of Covid, using Microsoft Mesh for the presenters. Kipman appeared as a “fully realized holoportation of himself,” as Microsoft later described his holographic version.
Kipman was joined by Hollywood filmmaker turned ocean explorer James Cameron and John Hanke, the CEO AR company Niantic (which makes that runaway AR successful game, Pokémon GO) to show off how they were using mixed reality to combine shared experiences. The prototype of the Pokémon GO game was very cool.
Mesh is possible because Microsoft has grown Azure into a cloud computing behemoth which makes sexy applications like this.
“In these collaborative experiences, the content is not inside my device or inside my application. The holographic content is in the cloud, and I just need the special lenses that allow me to see it,” says Kipman.
“This is why we’ve been so passionate about mixed reality as the next big medium for collaborative computing. It’s magical when two people see the same hologram.”
I can’t wait to try it out.
This article first appeared in the Daily Maverick.
