If mixed reality is to transform education, Magic Leap One is only part of the opening act. It will be exciting to finally test the new device, and its debut will likely push Microsoft and other providers to release their next generation devices as soon as possible. Indeed, we may soon hear more about the rumored Augmented Reality devices from Apple and Google both of which may include mixed reality elements.

To start creating for mixed reality calls for a new set of opportunities and design considerations. Think about the times you learned about the force of gravity and law of physics or the way Renaissance masters used perspective in art and architecture. For some of us this may have been through the pages of the book, but others of us may have taken a trip outside to experiment with gravity or study Renaissance masterpieces in person.

Now think about all the other things you wished you had learned this way and imagine a dynamic digital display that transforms your environment and even your living room or classroom into an immersive learning lab. It is learning within a highly dynamic and visual context infused with spatial audio cues reacting to your gaze, gestures, gait, voice and even your heartbeat, all referenced with your geo-location in the world. Unlike what happens with VR, where our brain is tricked into believing the world and the objects in it are real, MR recognizes and builds a map of your actual environment.

For instance, when you throw a digital tennis ball, it will bounce back depending on how far away you’re standing from the nearest physical wall. If you shrink or enlarge the ball, it will make the ball bounce back on a different trajectory suitable to its new scale. The tennis ball will come back if it hits the physical wall but will continue to roll if it goes through the door.

This will open a whole new world for experimentation across many fields of study. The interplay of sensors, machine learning and visuals, when done right, may feel like magic, as objects react to the users’ actions and respect the physics of the real world.

With Microsoft’s current Hololens, we’ve seen innovative learning experiences in healthcare, engineering, art and other fields. Mixed reality that includes vivid graphics and a broad spectrum of gestural interaction could lead to the creation of more deeply immersive interactive experiences.

The ability to manipulate data in 3D spaces makes it easier and more efficient to detect patterns and gain insights. For instance, think about an urban simulation where students will be able to observe the flow of traffic, weather, people and their impact on cities in a realistic 3D environment. In mixed reality, students may be able to look at urban systems that could be compressed down to a tabletop size. Students could explore individual elements of the scene from multiple angles and then bring the environment closer by expanding the size to virtually step into it at street-level. And with the gesture of a hand, they could plunge deeper into the model, moving directly inside one of the buildings and possibly meeting a guide or their peers there.

In mixed reality, virtual characters will be contextually aware, able to sit on the real chair next to you and know when you are talking to them, or to somebody else in the room. That might let users join a musical performance with a virtual symphony and play along. Collaboration comes naturally in shared physical spaces, and I expect mixed reality to offer us compelling new ways to connect and create together.

How do we design for mixed reality experiences?

As with other technologies, we will first draw upon what we already know from making earlier educational games, simulations, and videos. They will feel like mini simulations with branching learning paths and narratives.

But the full potential will emerge only as we design a new language for mixed reality. We should not simply rely on existing instructional-design methods but look into bringing user experience designers, artists, creatives and students into the development process. We will need a new visual and gestural language that will work in mixed reality. Natural user interfaces and sensors will make leveraging spatial computing ubiquitous. Sensors will collect data that will provide us with new ways to assess learning. In the future, MR will open portals to immersive stories in your living room.

In mixed reality, we will work with fewer pixels than in VR, and digital objects will need to respect the physics and in some cases social aspects of the world. In mixed reality, designing the interactions is as important if not more than designing the visual graphics. Fundamentally, if VR is about transporting you to a magical world, MR is about infusing your world full of magic.

What does the future hold?

There is still a dose of skepticism about what Magic Leap will accomplish with its device. The initial release of Magic Leap One will be a luxury item for many education institutions with little accommodation for people with disabilities or digital equity. Yet in the future, immersive technologies may hold a greater potential to solve these issues compared to our current technology tools.

Further down the line, as Moore’s law drives the technology’s power up and prices down, expect the fidelity and interactivity of these experiences to only get better in the next three years. It is likely that we will see mixed reality glasses at a variety of price points that will eventually replace our phones, screens and desktops.

Mixed reality is about breaking the boundaries between our physical and virtual worlds. It will offer us a canvas to imagine new modes of learning and new ways to collaborate with others beyond the frame of the screen. Ultimately discovering the potential of mixed reality does not depend on any single device, but on the journey we take as humans in understanding how we can learn, connect and enrich our lives.

_______________

Designing Mixed Reality Experiences is an abbreviated version of my article Mixed Reality Will Transform Learning (and Magic Leap Joins Act One)  The piece originally appeared on EdSurge on July 19, and here is the link to the original article.