If you caught the Lady Gaga as David Bowie performance at the 2016 Grammys you saw a glimpse of our augmented reality future. Lady Gaga’s use of Intel’s Living Canvas technology was not about wearables – devices that we put on our bodies. Instead, it was augmented reality as digital projection, treating the body as a canvas. Intel showed how digital projection can be used to turn us into something other than we are – in real time. It’s a fascinating development for entertainment and media – and possibly for education.

Lady Gaga as David Bowie

The one question in last night’s performance – how did she pull it off so quickly? David Bowie only passed away a month ago. Paul Trapp, Intel’s director of technology discussed the project with Vanity Fair. Gaga had already been working with Intel to push the boundaries of fashion for her current performances. When the opportunity came to do a Bowie set, they suggested creating a digital skin to capture Bowie’s look:

Bowie had tons of iconic looks throughout his very rich career,” Tapp said, “so Lady Gaga was curious how we could pay an homage to that. We introduced ‘living canvas’ technology to her, which allows her to basically have what we call digital skin—which has been used in tech art installations, but never before for a live performance.” The digital skin is projected onto Gaga as she performs so that it looks like her “makeup” is changing instantaneously throughout her six-song performance.

To create the makeup animation—essentially C.G.I. done live—Intel’s team created a 3-D scan of Gaga’s face by capturing her in about 12 different poses with complex scanning technology. The team then created 3-D motion graphics based on Gaga’s facial measurements and the Bowie looks she wanted to replicate.

Projected on to her face with real-time tracking, Gaga morphed into a series of personas created by Bowie.

Intel’s Living Canvas Technology

Intel’s Living Canvas has been around for a while, but it’s always been used for special effects after a scene was shot. This is the first time it was done live. According to Teresa Herd, ” . . . it’s literally changing and morphing right in front of your eyes.” Even more, the visuals were not entirely pre-rendered content – it was responding to her movements on the stage.

Our Augmented Reality Future in Media and Education

Technology like this will have a major impact on entertainment and become commonplace once the cost and complexity drops. And there will undoubtedly be a backlash, just as there was when a young Bob Dylan switched to an electric guitar at the Newport Folk Festival in July 1965. But artists will use this just as they became adept at using fashion to push the boundaries of music and performance (the debt Gaga owes to the early work of Madonna).

Here’s the intriguing question: does it have a use outside of entertainment? In education? Clearly not now, but what about the future? We already role play in foreign language, counseling and other courses. Why not use a technology that gives students and teachers other personas? And it has tremendous potential for art and performance projects.

The best undergraduate philosophy class I ever had was with a professor who acted out everything he taught – one week Socrates, another Hegel, and another Sartre. You walked out of the room completely immersed in the ideas of the person he was that week (yes, the Existentialism week was a tough one). The ideas swirled around in your head – until next week came along. He didn’t lecture, he did something like performance art while pulling passages from texts and video. It was absolutely breathtaking. No one ever dozed off. Few ever missed a class. And there were always students from other departments hoping to snag a seat. What he gave us was the experience of being someone else – he brought them to life for us.

The point with AR and VR is not the technology, but the experiences it brings into our lives. They will be media experiences and learning experiences and we will be profoundly impacted by them.

Whatever you think of Lady Gaga’s performance last night (and it got decidedly mixed reviews) – Intel’s Living Canvas is a peek into our augmented reality future. As Bowie said,

Tomorrow belongs to those who can hear it coming.