If you happen to be near Copenhagen later this month, drop by the Unite Copenhagen conference to see Varjo’s incredible mixed reality demo of a Volvo. Working with the auto manufacturer and Unity, Varjo has created an amazing experience – at once VR and Mixed Reality. An experience of the future, with implications for every aspect of our lives.

We had an opportunity to see an earlier version of this and try out the Varjo HMD at this summer’s Augmented World Expo in Santa Clara. Varjo makes what you might call industrial-strength VR headsets. No blurry scenes here; these are photorealistic graphics. Compile them into an immersive experience, and the results are like something you’ve never seen or experienced before. Seriously, seeing the Volvo before you is one thing. Getting inside it (virtually) and looking over the dashboard at the road ahead of you is something you’ll only see in this mixed reality demo.

Varjo has created a wicked HMD in the VR-1, targeting high-end design applications.

According to their description,

VR-1 is the world’s only human-eye resolution headset. Designed for use in complex and demanding industries, it comes with the world’s most advanced integrated eye tracking technology and is compatible with the most popular 3D software tools.

As for the specs, you’ll find,

[A] Bionic Display™ with resolution at over 20/20 vision (over 60 PPD / 3000 PPI), flicker-free screen refresh at 60/90 Hz. Combining two 1920×1080 low persistence micro-OLEDs and two 1440×1600 low persistence AMOLEDs.

And as you might expect, this will cost you a little – $5,995 to be exact. The full replication of reality doesn’t come cheap in the virtual world.

Varjo’s Photorealistic Mixed Reality Demo

There’s a good account of this project and the collaboration between Varjo, Volvo, and Unity from Nick Davis on Unity’s blog.

Mixed reality means blending virtual content with the real world. So far mixed reality has been accomplished with optical see-through, where the user sees digital objects augmented on top of reality through a pair of glasses. This is fine for portraying infographics or playing games, but for realistic scenes, it offers little value. Optical see-through devices can’t display black or opaque content on top of the real world. Everything appears hazy and holographic.

We at Varjo wanted to get rid of this limitation and be able to render photorealistic, opaque content – where it is impossible to distinguish between what is real and what is virtual. Our mission was to make photorealistic mixed reality possible with video pass-through. Video pass-through means using cameras to digitize the world in real-time, and then showing the combined result of real mixed with virtual to the user.

The pass-through video required an upgraded XR-1 HMD with dual cameras that could work with the mixed reality demo developed in the Unity platform.

The fact that Unity is easy to integrate and extend with C++ libraries, such as our own Varjo plug-in, made it possible for us to extend our plug-in to support mixed reality. By simply defining the empty background in a VR scene to be replaced by the video-pass-through signal, we were quickly able to see virtual objects in a real environment.

Here’s how the Volvo appears when you move from optical pass through to the Varjo XR-1 mixed reality experience.

The Varjo Mixed Reality Experience

Here’s Varjo’s description of the experience.

A beautiful Volvo XC60 builds up in front of you. It first appears as a stylized transparent blue wireframe. The virtual car is anchored to the real floor in the room around you and oriented so that the chair in the booth is aligned with the driver’s seat of the virtual car. The viewer can take a seat in the real chair and is still able to see the real surroundings through the wireframe.

The car now turns into a solid model, and the surfaces goes from transparent to opaque. The virtual car casts shadows on the floor of the real world, and looking on the car’s surface, it is possible to see that the real world is reflected in the car’s surfaces. The reflections come from an HDR cube map that was taken during setup on the exact spot of the car. The same cube map is also used for ambient lighting.

This is the first time the viewer sees opaque mixed reality, and the effect is stunning. You can still see the real world and your colleagues through windscreens.

While we didn’t have the full mixed reality demo at AWE 2019, the effects were similar, especially the reflections on the surface of the Volvo.

Needless to add, as good as the Varjo HMD is, there’s a significant amount of pre-processing to render the graphics. In the end, the Volvo model is comprised of 7 million polygons and 150 4K textures. Child’s play in the not-too-distant future, but serious stuff now. This video gives you a sense (in 2D) of the Varjo-Volvo experience.

 

At the end of Varjo’s mixed reality demo, you step out of the car and the environment around you morphs into an alleyway in Venice. You see the shadows of the Volvo on the paving stones and the reflections of the ancient buildings in the glimmering surface of the car. It’s a striking moment, and Varjo’s description doesn’t feel like an exaggeration.

It is a Matrix-like moment to see the surrounding reality disappear and replaced by a virtual scene – and then traveling back [again].

The Future

At Unite Copenhagen, this will be just a fantastic mixed reality demo. But it already has immediate applications. Volvo is using high-end VR and mixed reality experiences as part of their design process. They are putting designers and others on real roads and driving while wearing the Varjo HMD.

Here’s Volvo talking about the advantages to the design process when you combine the real world with the virtual, making the two indistinguishable from each other.

When we stepped out of the experience at AWE, all I could think of was that much of the work now done through CAD programs would be transformed. Paradoxically, the virtual will end up making the design process real. No longer staring at a screen of graphics, you’ll do the work in real-life.

Or what we’ll call “real-life” in the future. It will be a fun project trying to define that term a decade from now.

Long term, we know what happens. The high-end tech of today is available to a large segment of the public tomorrow. It won’t be a mixed reality demo or an advanced design tool, but part of our everyday experience of the world.

And that’s where the Varjo-Volvo-Unity project becomes so fascinating. It obliterates the boundaries between the real and the virtual. The possibilities across every area of society are mindboggling. As are the ethical challenges.

One thing is clear. When you step into that mixed reality Volvo and gaze through that virtual windshield, you know precisely where this immersive tech road is headed.