Varjo’s Innovative Mixed Reality Merges the Real and Virtual

Varjo continues to develop innovative Mixed Reality experiences with their human-eye resolution headsets. We’ve demoed their headsets before – at the AWE last spring – and were blown away by the realism. Varjo’s latest development is an innovative Mixed Reality green screen experience that incorporates hand tracking. As Jamie Feltman put it in UpLoadVR,

Time for another slightly cruel glimpse of the future of VR packaged into Varjo’s ever-impressive enterprise-level VR tech.

This is the direction VR and Mixed Reality is headed in the future. The technology is impressive, but what’s truly important are the ways these developments will impact education, work, and our lived experience. As always, the critical issues are the sociological, psychological, and human dimensions.

Varjo’s Innovative Mixed Reality Demo

Innovative Mixed Reality experiences from Varjo reveal the future of XR

UploadVR describes Varjo’s XR-1 headset which combines the real and virtual worlds through chroma key integration.

. . . when you look at a green screen with the headset on, you’ll see a virtual world. At the start of my demo in London last week, I gazed on in amazement at Google Earth running on the green screen. It was fully 3D, just like you’d expect, stretching beyond the walls of the room, literally like a window into another reality. But, when I turned my head away from the screen, I saw the real world again. Not only that but my full, human hands could be seen clear as day in the virtual environment and anyone else in the room could step into the screen and appear in my virtual world. It was pretty mind-blowing to behold.

As of now, your hands can’t interact with the virtual world, but that could change in the future.

What’s striking is that Varjo has brought the real and virtual worlds into the same physical space. Oculus Quest has done something similar by visibly positioning the virtual space of the Guardian in physical space. In essence, you can step out of a VR experience and interact with the real world. But Varjo’s solution is light years ahead of the popular Oculus HMD. You have photo-realistic experiences that can be placed anywhere by simply moving a green screen.

Moreover, you can combine virtual experiences with real-world objects, which would revolutionize training. Varjo demonstrated a flight simulator with the virtual world incorporating real-world controls.

Here’s a short video on how Varjo’s chroma keying works and the way it could transform our current learning experiences in virtual reality.

As Varjo describes it in their blog,

With XR-1, we’ve solved the ‘hard AR’ problem, which exists on optical see-through devices like HoloLens or Magic Leap that always portray virtual elements as hazy and holographic and can’t display shadows or opaque content.

The Magic Leap One and HoloLens 2 may be breakthrough devices, but the fuzzy holographic experiences are only an interim step to merging the real and virtual worlds.

A SciFi Like Future

At times, Varjo’s innovative Mixed Reality experiences feel like science fiction. Varjo’s Roope Rainisto showed some of the creative ways you could blur the boundaries between the real and the virtual.

Rainisto picks up the green screen on the floor, wraps it around himself and then suddenly disappears inside of Google Earth. He runs to the other side of the room, away from the rest of the screen, carrying a window into the world with him. He has literally become Google Earth, offering glimpses of the 360 view of the Vatican all around me before dropping his makeshift invisibility cloak and reappearing again.

Did Varjo just reveal the future of fashion a couple of decades from now? Forget YouTube influencers, we will have people walking around as virtual experiences.

With options like this, the possibilities are endless. Green screened classroom and office walls will transform our learning and work environments. Walls would no longer limit our experience but would become a basis for our engagement with the world. The whole notion of working and learning remotely would eventually disappear. Your sense of presence in the virtual space will be no less than what physical reality offers.

This is why the current Social VR  solutions to the Coronavirus crisis are disappointing. The potential for virtual presence is in our platforms, but it’s nowhere close to where it needs to be.

The promise of VR and Mixed Reality is not in Second Life type experiences – but a sense of presence no different than reality. Varjo is proving it can be done now – at considerable cost. In five years, this technology will be entering the consumer market.

It’s utterly fascinating when you consider the longterm. Varjo’s innovative Mixed Reality experiments will undermine the primacy of physical location. Technology has been chipping away at this for the past few centuries (the railroads, telegraph, phones, radio, TV, the Web, etc.). High-end virtual reality takes it a step further, shoving you into a constructed virtual environment. But the blending of the real and virtual in Mixed Reality will ultimately eviscerate physical space. Real and virtual will blend seamlessly together, and we’ll transition from one to another as quickly as we now move through social media. 

This is still years away from the consumer market. Varjo’s headset runs nearly $10,000, limiting the device and the experiences to high-end enterprise settings.

But demo Varjo’s innovative Mixed Reality experiences if you can. It’s a deep dive into our lived experience in the future.