Apple’s AR development got another boost from their acquisition of the Danish company Spektral which developed an algorithm for cutting figures out of images and video. There are some consumer knock-offs around but nothing close to what Spektral has achieved. In effect, it’s doing digital green-screening. And doing it live and on your Smartphone. At 60FPS.
The augmented reality potential is obvious. So are the ethical implications of dropping someone into an environment they’ve never physically stepped foot in. Sort of like Deepfakes for immersive environments.
A quick video illustrates the power of the software.
Here’s how Spektral describes their solution.
Our pioneering and unique technology is based on state-of-the-art machine learning and computer vision techniques. Combining deep neural networks and spectral graph theory with the computing power of modern GPUs, our engine can process images and video from the camera in real-time (60 fps) directly on the device. [Patently Apple]
Spektral’s software seems an obvious fit for Apple which continues to push the boundaries in Smartphone-based AR.
Spektral will boost Apple’s AR development
The acquisition could help Apple in multiple ways – from more traditional applications like Photobooth to their forthcoming AR platform. It turns out that they picked up the company a year ago with the news just coming out. Coincidently (or not), that’s one day after Google showed off its Pixel 3 phone with new AR stickers.
Here’s how Tech Crunch sees the acquisition helping Cupertino’s emerging AR technologies.
The startup initially pitched its product as a cheaper and more efficient “green screen” technology, to remove primary images from their plain (typically green) or standard-pattern backgrounds, with an early iteration of the product built by training the system on over 100,000 professional cutouts.
. . . but what’s most notable here is what Spektral might contribute to Apple’s imaging business. That could be for applications that Apple has yet to launch, but also to improve the quality of those that are already in the market, from legacy products like PhotoBooth through to ARKit, the company’s platform for mobile development.
It could find use in Apple’s camera, Messages, or Clips, giving them an advantage over platforms like Snap. But it’s in Apple’s augmented reality development that the software seems most promising.
[The software] can also be effective in occluding AR environments behind figures to make digital AR content appear to interact with the position of humans.
Spektral’s segmentation technology is also able to run on mobile phones, making it potentially a quicker and more efficient way of processing AR images directly on devices. And one of its main selling points is how it copes with very fine details, such as hair.
In the image below, you can see how well Spektral’s imaging algorithm handles details like hair (compare the middle image to the one on the right).
Remarkably, these renderings can be done live on a mobile device, which is where the augmented reality potential lays.
We see Apple waiting until 2020 to release a product. It’s not just the challenge of having the right technology. There are also the UXI design issues (Voice? Pointing device? Tapping?) and battery life that are critical to a successful AR headset.
Spektral is just one more move in Apple’s massive AR project. Just don’t get your hopes up too soon. Apple only delivers when they’re assured of success.
Emory Craig is a writer, speaker, and VR consultant with extensive experience in art, new media, and higher education. He speaks at global conferences on innovation, education, and ethical technology in the future. He has published widely and worked with the US Agency for International Development, the United Nations, and the Organization for Economic Co-operation and Development (OECD). Living at the intersection of learning, games, and immersive storytelling, he is fascinated by AI-based avatars, digital twins, and the ethical implications of blurring the boundaries between the real and the virtual.