The Finnish company Varjo has delivered some remarkable virtual and mixed reality projects with their human-eye resolution VR/XR headset. We were blown away by their photorealistic XR demo at Augmented World Expo back in 2019. The Matrix-like experience had you sitting in a virtual version of a Volvo XC60 and then stepping out into a breathtakingly realistic scene in Venice, Italy. Now, Varjo’s ready to take this a step further with their Reality Cloud, a platform that can recreate physical spaces in virtual environments anywhere in real-time.

If they pull this off, it’s a huge step toward the metaverse envisioned in novels such as Snow Crash and Ready Player One.

Varjo’s New Reality Cloud Platform

Here’s the description from RoadtoVR:

. . . the company plans to leverage the wide field-of-view depth sensors on its Varjo XR-3 headset to allow users to easily capture their surroundings and use it as the basis for virtual meetings. What’s more, beyond just a static capture, the Varjo Reality Cloud platform will continuously update the portion of the environment that’s in the headset’s view while the meeting is happening. That means if there’s something real and relevant in the host’s environment—like a book, product, or even another person—the virtual viewers will be able to see that thing moving and updating in real-time (as long as the host is actively looking at it).

The idea melds well with XR-3’s existing high-quality passthrough capabilities. During normal use of the headset it’s easy to toggle on the headset’s pass-through view to see the environment around you, which means if you were sharing your local environment through the Varjo Reality Cloud, it could seem like others in the meeting were standing right in the same room as you.

What’s remarkable in Varjo’s Reality Cloud is that you’re not standing in a 360 video reproduction but the scanned digital environment of a space in the real world. It’s as if you’ve been immediately teleported into another environment. While the platform is still in demo mode, it’s a signpost to just how real how XR experiences will become – and how easy we will blur the lines between reality and our virtual worlds in the future.

Varjo’s Demo of Teleportation in the Future

Right now, Reality Cloud is only a demo and not ready for prime-time. Varjo’s video only offers a highly pixelated version, though the company promises it will quickly improve in quality. Here’s their short version of the demo (or you can opt for a longer 20-minute version).

This would be a game-changing development for social VR and our virtual worlds, which right now fall on the cartoonish side of the spectrum.

Making the Reality Cloud Real

Delivering Varjo’s Reality Cloud in real life will be no small achievement. The processing required to render a real space into its virtual counterpart in real-time means doing this in the cloud, not on your local headset. That may solve the processor demands but raises new ones for bandwidth. Varjo has promised a new foveated compression algorithm that will dramatically cut the amount of data being pushed to the headset – though don’t expect this to initially run on standalone wireless HMDs like the Oculus Quest. If we have a moment of skepticism about the entire project, it would be here in getting the data transfer down to a manageable load. If the COIVD-19 pandemic taught us anything, it’s that staying connected largely depends on bandwidth. 

One remarkable aspect of Varjo’s project is that Reality Cloud is expected to run on a wide range of devices, not just Varjo’s own high-resolution headsets. It will also function on mobile devices and flat screens, opening the experience to a far broader audience.

Is it a Paradigm Shift?

Surprisingly, RoadtoVR’s Ben Lang is not convinced of the value of Varjo’s Reality Cloud over our current synthetic virtual spaces. He sees only limited usefulness without being able to interact with objects in the remote virtual environment. But we’ll take a more open view. Having the option to interact with another person in a virtual space that is the digital twin of their live environment would be a deeply engaging experience. They wouldn’t be in a synthetic VR space that may or may not relate to what they are talking about. A doctor could walk students through the equipment in an operating room; a supervisor could use it to train customer service reps. It wouldn’t replace VR simulations where you are learning by doing. Instead, it would offer new options for engaging with instructors and guest speakers in education and workforce training. 

And in the end, it puts us one step closer to a Matrix-like world – which is precisely where we are headed in our XR future.