Microsoft’s DreamWalker – an Early Version of the Matrix?

No surprise that we’re huge fans of The Matrix around here. And we’re waiting impatiently for the next version (which goes into production next year). In the meantime, we’ll focus on Microsoft’s DreamWalker project, a fascinating development that may turn our lived experience into a Matrix-like world. Who needs the movie when you can have the real virtual experience?

Here’s the description from TechRadar.

Using virtual reality, Microsoft is seeking to change the idea of the daily commute by immersing VR users in a virtual world as they walk through real-world environments on their way to work, school or a café on foot.

In their recently published paper, researchers Jackie Yang, Eyal Ofek, Andy Wilson and Christian Holz entertain the idea of “a future in which people spend considerably more time in virtual reality, even during moments when they walk between locations in the real world.” To do so, they created a VR system that allows its users to navigate the real world on foot while simultaneously seeing themselves strolling around in a place of their choosing in the VR world.

They named this VR system DreamWalker.

The paper is a little technical but worth diving into if you have an interest in this.

Microsoft’s DreamWalker and Mixed Reality

This project fascinates us on many levels, including the link to Microsoft’s Mixed Reality (MR) platform. Up to now, MR has primarily been confined to HoloLens with parallel (but different) platforms for devices like the Magic Leap One. Those Mixed Reality headsets from Samsung and other vendors have been nothing more than standard VR headsets with tracking cameras to help orient you in VR experiences. Calling them Mixed Reality seemed to be an exercise in confusion.

But with Microsoft’s DreamWalker, the relationship between Mixed Reality and VR suddenly makes sense. MR is not just adding digital objects to the space around you. It fills your world with a virtual environment that interplays with your real environment. You’re in VR, but it’s VR that syncs to the real world.

Suddenly, a Matrix-like lived environment becomes a possibility along with all the ethical questions and challenges that will arise. We’ll come back to that point a little later.

How DreamWalker Works

Microsoft's DreamWalker VR Hardware is bulky now but will shrink down.

Microsoft’s DreamWalker is far more complicated than strapping on your favorite HMD and heading out the door. It takes a backpack, HMD, GPS phone, and two RGB depth cameras. You look like you’re setting out to climb Mount Everest, not commuting to work. But Ivan Sutherland’s first VR headset was so large it had to be hung from the ceiling. Soon enough, this hardware will shrink down.

Microsoft’s blog provides more details on how the platform works.

During the walk, DreamWalker monitors users’ environments using a variety of sensing technology— inside-out tracking via a Windows Mixed Reality-provided relative position trace, a dual band GPS sensor, and two RGB depth cameras—to detect obstacles discovered along the way.

Discovered obstacles that may move or appear in users’ paths are managed by introducing moving virtual obstacles, or characters, such as pedestrians walking near users, to block them from any potential danger. Other options for controlling users’ paths may include pets and dynamic events such as vehicles being parked, moving carts and more, limited only by the imagination of the experience creator. The real-time environment detection technology provides guidance for the creation of this additional virtual content with great care taken to generate these virtual obstacles outside users’ field of view to avoid any unnatural “popping up” of objects in the virtual world.

One fascinating feature is “Mise-Unseen,” an obvious play off French theater term mise en scène. The feature supports covert changes in a virtual environment based on eye-tracking. Look one way and the scene changes in another area where you’re not looking. That blocks the artificiality of objects suddenly popping up in front of you, which kills an immersive experience as quickly as a cable around your ankle.

Since you could be using Microsoft’s DreamWalker in a dense urban environment with moving objects from the real world, Mise-Unseen would be critical to the virtual experience. The constantly changing real world is removed without making it appear as if the virtual is concealing it.

Here, the virtual not only conceals reality, but it also hides the process of concealment.

DreamWalker Could Transform our Lived Experience

Here’s how Microsoft sees your future strolls through the city.

During the walk, DreamWalker monitors users’ environments using a variety of sensing technology— inside-out tracking via a Windows Mixed Reality-provided relative position trace, a dual band GPS sensor, and two RGB depth cameras—to detect obstacles discovered along the way.

Discovered obstacles that may move or appear in users’ paths are managed by introducing moving virtual obstacles, or characters, such as pedestrians walking near users, to block them from any potential danger. Other options for controlling users’ paths may include pets and dynamic events such as vehicles being parked, moving carts and more, limited only by the imagination of the experience creator. The real-time environment detection technology provides guidance for the creation of this additional virtual content with great care taken to generate these virtual obstacles outside users’ field of view to avoid any unnatural “popping up” of objects in the virtual world.

With Microsoft’s DreamWalker VR platform, you could walk down the street of any town and feel like you were in the middle of Times Square (though I’m not clear why one would want to do that). More calming scenarios beckon. Your sidewalk morphs into a path leading you along a tranquil river in the depths of a lush forest. Facing a bad day at work? Maybe a Medieval Village and sword fights will help you focus. Leaving that bad day at work? Cap it off with a stroll through Death Valley on the commute home.

The virtual begins to replace the real (this process has been ongoing for some time). How you use it will make for some interesting choices. Today, we design our homes to help us feel grounded. Tomorrow, we’ll design the world outside our door. To some extent, we already do the latter as a community. Soon, we’ll do it individually, and change the experiences we’re running through streaming TV channels.

Learning, Training, and the Arts

The possibilities for learning are mind-boggling. So, too, are the opportunities for enterprise training. Immersive experiences could take place anywhere, not just in a lab or training facility. The only critical feature that is missing is Social VR. If DreamWalker can handle our constantly changing real-world environment, avatars shouldn’t be a significant hurdle.

As for entertainment and the arts, the visual arts are moving outside of museums into our public spaces. Take a look at Nancy Baker Cahill’s remarkable work and last summer’s AR art exhibition in Boston. In a full Mixed Reality environment, the artist’s canvas will be everywhere, encompassing the entire world.

Urban graffiti artists a few decades back were harbingers of our future.

The Ethical Challenges

We haven’t begun to fathom the ethical challenges of something like Microsoft’s DreamWalker. Think about it for a moment.

People get concerned about the echo chamber of social media. What happens when that chamber is no longer text and video streaming across our screens, but the environment around us? Right now, we might look away from the homeless on the corner. Perhaps we judge them based on our social media consumption. With a Matrix-like platform, they might completely “disappear.”

What happens when things go wrong at the intersection of the real and the virtual? You can see an early glimpse of this in Tesla’s Smart Summon feature. The newly released feature lets your car drive itself to you from 200 feet away. It’s already created chaotic scenes in parking lots. If there’s an accident, is the “driver” (the “summoner”?), the other car, or the manufacturer responsible?

Who’s responsible if you stepped into traffic while using a platform like Microsoft’s DreamWalker? The hardware manufacturer? The programmers who worked on the content? Or you for putting it on in the first place?

In the long run, it will probably become the manufacturer’s responsibility as it eventually will for Tesla’s Smart Summon. But our legal frameworks always lag behind the technology. Not by months, but by years.

Microsoft’s DreamWalker is not ready for prime time just yet, but it’s also not that far away. It will be here sooner than we realize – before we fully grasp its implications. Are we ready to embrace a world where the boundary between the real and the virtual disappear?

That’s a question we can only answer when we step inside the coming era of spatial computing.