The recent VR news out of South Korea is remarkable – a mother reuniting with her deceased child in VR. You knew we’d have to deal with the dead in VR – or at least the virtual recreation of them. But we thought this was perhaps five years down the road, not today.

Welcome to the future. Or as VR Scout put it,

We’ve seen VR technology used in a variety of incredible ways, but nothing quite as extraordinary as bringing the dead back to life.

Extraordinary technology leads to exceptional experiences. It’s why we’ve thought all along that AR and VR developments represent a sea-change in what technology will do for us.

And to us.

Experiencing the dead in VR opens up a Pandora’s box of both personal and ethical questions. In the right hands, it could undoubtedly play a role in helping grieving families cope with the loss of a loved one. But there are other, cynical ways this could be used – to prey on the grief of others.

There’s an entire cottage industry built around the latter, but it lurks more or less in the shadows — time for another technology paradigm shift. Not only can we call up a car and driver whenever we need one. We’ll be able to call up our deceased relatives when we have the urge to interact with them.

It’s at once fascinating and unsettling.

Reuniting with the Dead in VR

Where to begin? The story of the deceased child in VR comes from Munhwa Broadcasting Corporation, a leading South Korean television network. They released a clip from their special documentary, I Met You, which focused on a family’s loss of their seven-year-old daughter, Nayeon. Toward the end of the program, the mother is reunited with her deceased daughter in virtual reality.

VR Scout describes the project.

. . . the company used VR and haptic feedback technology to reunite Jang Ji-sung with her seven-year-old daughter, Nayeon, who passed away as a result of a rare incurable disease back in 2016. And although the interactions are simple and the experience itself is brief, the virtual reunion clearly has a significant emotional impact on the grieving mother of four.

You’re not going to gain a lot from watching the full nine-minute video with its poorly generated auto-subtitles. But it’s worth giving it a few minutes. The VR experience recreates several personal moments that the mother had with her daughter. There’s a park they often visited, some of the young child’s favorite treats, and a tender moment where the daughter goes to bed.

It doesn’t take long to see that the mother’s grief is real and that it was a profoundly emotional experience for her. According to Aju Business Daily, the child’s mother Jang Ji-sung said of the experience,

Maybe it’s a real paradise . . . I met Nayeon, who called me with a smile, for a very short time, but it’s a very happy time. I think I’ve had the dream I’ve always wanted.

It was an emotional experience for the entire family. Aju Business Daily continues.

As a white butterfly flew and sat in one place, the sound of Nayeon’s song was heard. Jang burst into tears when her daughter ran with the cry of ‘Mom’ and said, ‘Where have you been, Mom? Did you think about me?’

Jang responded with a doleful voice, ‘I do all the time.’ As her daughter said, ‘I missed mom a lot,’ Jang replied, ‘I missed you, too.’ The mother was cautious to reach out to touch her daughter before Nayeon insisted, ‘Mom, please hold my hand.’ Jang finally held her daughter’s hand in her’s with tears streaming down her face. Nayeon’s father, brother and sister watched the encounter they’ve dreamed of at the side of the virtual stage, also crying.

I’m not sure I could handle an immersive experience like this.

Is it Exploitive?

There are multiple ways this could be seen as exploitive. Obviously, the young child had no say in the entire project. The dead never have much influence on how their images and presence are used. Culture plays the primary role here – though that could change when accessing the dead in VR creates new ways to interact with them. That’s another paradigm that could fall by the wayside as technology advances.

And then there’s the issue of how this was done for network television. As Luke Plunkett said in Kotatu, there might be value in the immersive experience if it was behind closed doors. Putting it on a national broadcast is another story. Though given what have on TV today, it’s not much of a stretch to see this coming.

The Technical Details

Road to VR describes the production process, which is surprisingly not all that complicated. You won’t do this at home (just yet) but a well-equipped mo-cap studio and eight months of work was enough. Using the deceased daughter’s younger sister, a 3D avatar was created. Integrating AI and voice recognition gave the virtual character of Nayeon a degree of responsiveness.

Developed by South Korean startup Vive Studios (no relation to HTC’s Vive Studios), the virtual Nayeon was created over the course of eight months using a variety of techniques. Motion capture not only recorded an adult actor’s movements, but also facial expressions, some of which were acted out based on video and photos of the real-world Nayeon.

High resolution photographs were taken in a 3D capture technique called photogrammetry. Nayeon’s little sister, a spitting image of her older sibling, was used as the basis of the character model.

Although not apparent in the video above, the studio also added a degree of liveliness to the character by integrating voice recognition and a basic AI, which would let the pair have a basic conversation. Responses were created based off of family interviews and videos.

The mother didn’t need anything more than an HTC Vive Pro HMD, the usual Vive trackers, haptic feedback gloves, and a wireless adapter. That and a whole of lot courage to go through the experience. 

We’re not at the point where you can pull this off with a standalone VR headset and hand tracking. That will come soon enough.

Future Implications

You can see projects like this evolving in several ways. Appearances by the dead have been with us for the past few years as Road to VR notes.

. . . a bevy of examples come to mind: Fred Astaire dancing with a Dirt Devil, Tupac holograms, and more recently the litany of deepfakes that make you question whether famous actors are still alive or not.

Where having the dead in VR gets interesting is when you can do this with off-the-shelf resources. A generation is coming of age where every aspect of their lives is recorded and documented. In a decade or two, it won’t be necessary to find a younger sibling to recreate a virtual avatar of a deceased child, or anyone for that matter. Facebook is already moving in this direction with their expressive avatars and their slightly creepy VR Memories.

We’ll create the resources for us to return as virtual avatars in simply living our lives. It will be an easy matter to pull up the dead in VR from the digital trail we leave behind. In the end, technology won’t be the major challenge. Instead, it will be the human questions of how this gets integrated into the lives of the living. And of course, whether or not there will be the possibility of opting out. Today, we worry about privacy and consent. Tomorrow, we’ll worry about the use of our virtual presence after we’ve passed away.

This could lead to some awkward and strange moments in the future. Will deceased employees remain in the workplace after they die? As sources of expertise? Will dead CEOs sit in their offices as virtual avatars (keeping their exorbitant salaries, of course)? Think elderly faculty don’t retire soon enough now to make way for younger scholars? They could literally be around forever in higher education.

There are tremendous benefits in bringing the past back to life on both personal and societal levels. Indeed, we are moving so fast that even current events quickly become history. But paradoxically, our most innovative and future-oriented technologies could result in imprisoning us in the past. Will we become frozen in place as the dead walk virtually among us?

Where this goes is anybody’s guess. But this project to bring back the dead in VR just brought us one massive step closer to the future.