4/10 - Cut & paste your surroundings to Photoshop
— Cyril Diagne (@cyrildiagne) May 3, 2020
Code: https://t.co/cVddH3u3ik
Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF
Cyril Diagne, an artist, designer and programmer currently in residency at Google Arts, created this utterly awesome augmented reality app which seamlessly captures objects in the real world... and then "pastes" them as 3D files directly into Photoshop. (Watch the video above, you'll immediately see what I mean.) I've seen similar AR apps before, but nothing quite as seamless or as magical.
Perhaps as amazing: Cyril also made the code publicly available on GitHub here.
"The code was given away because it's a research prototype," Cyril tells me. "I don't think it giving that code away necessarily take away commercial potential. There is much more that can be done with that approach."
His goal is to make AR reality capture as intuitive on a smartphone as it is in Magic Leap -- except without needing an expensive HMD.
"Also the phone has such a perfect form factor it's hard to beat," he adds. "[T]here is no UI, there's no need to move the thumb or care about how we're holding the phone, which happens to be quite pleasant and intuitive... Because we're able to delegate some of the cognitive tasks (like deciding which object is the one being pointed at in an image, selecting and removing the background, .etc) we can reduce the number of steps required to perform a task."
As for how he did it, he goes into great technical detail on Twitter, which I've excerpted for the technically minded below:
"The secret sauce here is BASNet (Qin et al, CVPR 2019) for salient object detection and background removal. The accuracy and range of this model are stunning and there are many nice use cases so I packaged it as a micro-service / docker image.
And again, the OpenCV SIFT trick to find where the phone is pointing at the screen. I also packaged it as a small python library. Send a camera image + a screenshot and you get accurate x, y screen coordinates!
"I wrote the mobile app using Expo as I wanted to try the ReactNative platform. A few rough edges with Android support but the dev workflow is impressively smooth. Could become an ideal Magic Leap interaction design research platform if it could run #TFLite models without ejecting.
"Right now, latency is about ~2.5s for cut and ~4s for paste. There are tons of ways to speed up the whole flow but that weekend just went too fast!"
Hat tip: Kottke's blog.
Comments