New World Notes

Top 12 AR, VR & Avatar-Based AI Projects & Innovations of 2018 Featured in New World Notes

Watch An AI-Controlled Avatar "Learn" How To Dribble In Real Time - Huge Implications For Games, AR/VR, And Real Life Robotics

• HoleLenz Turns HoloLens Into A Literal (Virtual) Portal Into Other Worlds
• We Heard You Like AR & VR, So We Put AR In Your VR
• Project H.E.A.R.T: A Virtual Reality War Game Where Soldier Morale Is Controlled By The Player’s Real Emotions
• VR Pioneer Helping Advise $10 Million ANA Avatar XPRIZE, The Contest To Merge Robotics With VR/AR-Driven Human Expertise

IOS' ARKit Enables Reflection Of Real Objects On Virtual Surfaces

Many more beyond the break!

Watch: MICA, Magic Leap's Eerily Human AI Avatar Who Looks You in the Eye

Word on the street is this is the most impressive tech unveiled at this week's Magic Leap conference. Say hello to MICA -- because she'll soon say "hello" back, while looking you in the eye:

Mica was sitting at the table in the same room. She smiled at me and look at me. I was struck that she wasn’t just looking at me. She was looking in my eyes. She tilted her head from side to side... When I noticed how attentive she was, I moved my head forward and looked in her eyes. She did the same and looked at me. I moved my head back and she moved her head back too. She was mimicking some of the movements that she saw me make. She didn’t talk, but that is coming in the future.

UploadVR's Ian Hamilton was staggered on an ontological level:

I need to process some thoughts. The Mica AI left me in a deeply introspective mode. I didn’t want to think of it as “she” but I couldn’t control my reaction. They Ex Machina’d me a little bit. https://t.co/gHZyzIAXKi

— Ian Hamilton (@hmltn) October 11, 2018

Full keynote below, beginning with inspiring words from company CMO Brenda Freeman and acclaimed designer Robin Hunicke, an apparent (and admirable) attempt to make a clean break from the startup's history with internal sexism, followed by a fireside featuring Neal Stephenson and John Gaeta (the Matrix effects guy), with Andy Serkis and others soon after:

Now in LA: Experience Awesome 3D Video & Machine Learning-Based Music Projected on the Walt Disney Concert Hall -- Adaptive Sound Artist Robert Thomas on How the Soundtrack Was Created

Video shot during last night's WDCH preview by Brooke Erdmann

If you're anywhere near downtown Los Angeles between tonight and October 6, you absolutely have to experience WDCH DREAMS. I was lucky enough to attend a preview with my wife last night (almost bumping into the Mayor, who was also there) and we were utterly dazzled by this flood of images and music echoing off Frank Gehry's famed concert hall:

The Los Angeles Philharmonic has commissioned award-winning media artist Refik Anadol to create unprecedented, breathtaking, three-dimensional projections onto the steel exterior of Walt Disney Concert Hall to signal the commencement of the LA Phil’s 100-year anniversary celebrations. Free and open to the public, nightly performances are scheduled to occur every half hour, with the first performance at 7:30 p.m., and the last at 11:30 p.m., September 28 to October 6... WDCH Dreams’ accompanying soundtrack was created from hand-picked audio from the LA Phil’s archival recordings. Sound designers Robert Thomas, and Kerim Karaoglu augmented these selections by using machine-learning algorithms to find similar performances recorded throughout the LA Phil’s history, creating a unique exploration of historic audio recordings. 

Adaptive music/sound artist Robert Thomas, as longtime New World Notes readers know, has been creating ambitious audio experiences across many mediums and technologies over the years. (Full disclosure: Also a pal.) I first met him when he was creating voice-activated installations and location-reactive soundtracks in the virtual world of Second Life. After SL he went on to co-create projects with luminaries like Massive Attack, Imogen Heap, Ben Burtt (sound designer for the Star Wars movies), and Hans Zimmer. (Robert and Zimmer worked with director Chris Nolan to create an interactive audio app for his movie Inception.)

Based in London, Robert tells me how he and his collaborators used machine learning to take and shape disparate performances and composers from the LA Phil's massive archive of recordings, and sculpt them into a completely new soundtrack:

Watch an AI-Controlled Avatar "Learn" How to Dribble In Real Time - Huge Implications for Games, AR/VR, and Real Life Robotics

What you're looking at above appears to be a pretty big breakthrough in artificial intelligence -- an avatar "learning" how to dribble in real time through AI. DeepDribble is a project from researchers at DeepMotion and Carnegie Mellon, and the idea is to demonstrate how an AI-controlled avatar can acquire the complex and subtle set of integrated physical skills and coordination that humans take for granted:

Basketball at its peak is known for intricate ball handling, quick pivoting, and fake outs. A good player will fluidly transition between have a variety of dribbling skills. In order to recreate this, the trained characters need to be able to transition between separately trained behaviors... Libin Liu and Jessica Hodgins use the same methods for training single skills to create a multi-skill control graph (which allows transition between motion fragments of two different skills). The training is done incrementally, behavior by behavior, to ensure integration occurs with proper transitions for each skill. The result is a player that can perform multiple ball handling skills in a variety of orders, changing course with seamless blending.

"First we build a bio-mechanical model of the digital character, using physics to simulate bones and virtual muscles," Deep Motion's Libin Liu tells me. "We put the digital character in the physically simulated virtual world and let it mimic how real players perform the motions using their muscle forces."

The training process, as you might have imagine, requires quite a lot of power -- but Liu says the learning arc took about as long as a single televised NBA game -- or even half of one:

Coming Soon to Unity 3D, AI-Powered Agents Able to Use Tools to Overcome Obstacles

They learn to use the orange cube to jump over the wall. This demo will be included in the @unity3d ML-Agents 0.3 release in about a month. #MachineLearning pic.twitter.com/MSWsHU6Bcy

— PunchesBears ㅎ㉨ㅎ (@punchesbears) February 7, 2018

You thought the virtual worm was impressive, watch this blue block "learn" how to use a tool to get over an obstacle. Created by a deep learning and procedural animation coding wizard called Punches Bears (because of course), the demo will be featured as a demonstration of Unity 3D machine learning code in an update that's coming in a few weeks:

Ray Kurzweil Once Bet Mitch Kapor That A.I. Would Pass the Turing Test by 2029 -- Because He Assumed Full VR Would Be Feasible By Then; UPDATE -- Kapor Responds: "I told Ray I was prepared to double down on the bet..."

Today I learned that in 2002, tech pioneer Mitch Kapor (a Linden Lab founding investor, among many other things), bet against futurist Ray Kurzweil that we wouldn't create a computer that could pass the Turing Test by 2029, writing:

The metaphor of brain-as-computer is tempting and to a limited degree fruitful, but we should not rely on its distant extrapolation. In the past, scientists have sought to employ metaphors of their age to characterize mysteries of human functioning, e.g., the heart as pump, the brain as telephone switchboard (you could look this up). Properly used, metaphors are a step on the way to development of scientific theory. Stretched beyond their bounds, the metaphors lose utility and have to be abandoned by science if it is not to be led astray. My prediction is that contemporary metaphors of brain-as-computer and mental activity-as-information processing will in time also be superseded and will not prove to be a basis on which to build human-level intelligent machines (if indeed any such basis ever exists).

There's a lot more to Mitch's skepticism, which is brilliant, but just looking at the state of things today, I'd say he looks likely to win. There's been a lot of interesting advances in functional AI (i.e., self-driving cars), and simulations of human creativity, but we still haven't seen any AI that can maintain a human-like conversation for any substantial length of time. So unless there's some massive breakthrough within the next 12 years, Kurzweil's going to owe the EFF a $20,000 donation.

Kurzweil's argument for passing the Turing test by 2029 is worth reading too, because his prediction is partly based on a flawed assumption about the growth of VR and virtual worlds:

Elon Musk, Who Who Thinks We're in a Virtual Simulation, Now Building Technology to Actually Make that Possible

Hey you know how Elon Musk believes we're actually plugged into a computer simulation of the real world created by an advanced simulation, even though that doesn't make much sense from a practical or even philosophical perspective? Funny story, but turns out he's spending money to someday, maybe, make that actually possible:

SpaceX and Tesla CEO Elon Musk is backing a brain-computer interface venture called Neuralink, according to The Wall Street Journal. The company, which is still in the earliest stages of existence and has no public presence whatsoever, is centered on creating devices that can be implanted in the human brain, with the eventual purpose of helping human beings merge with software and keep pace with advancements in artificial intelligence. These enhancements could improve memory or allow for more direct interfacing with computing devices.

So even if there is no advanced civilization plugging us in now, Elon's trying to build the hardware so that we ourselves (as rational and enlightened as we are) to plug in our kids, or our kids' kids. You can read more about it in this long Vanity Fair profile, which includes this on-point comment from VR pioneer (and Linden Lab board member) Jaron Lanier, touching on topics he also mentioned in my Wired article last year:

Watch: HAL vs. Her In Male/Female Argument Between AIs

Reflections of HAL and Samantha from Tillmann Ohm on Vimeo.

Watch this, it's pretty damn brilliant:

Composed of original lines from 2001: A Space Odyssey (1968) and Her (2013), the mashup creates a new context to the words of the electronic protagonists. The two operating systems are in conflict; while Samantha is convinced that the overwhelming and sometimes hurtful process of her learning algorithm improves the complexity of her emotions, HAL is consequentially interpreting them as errors in human programming and analyses the estimated malfunction.

Via Kottke, who points out another hook with the contrast in artificial genders:

Can Consciousness Be Simulated as Musk Must Assume?

Really great reader conversation about Elon Musk's belief that we're already in a simulated reality. Here's a smart point by longtime reader "Galatea", who pushes back at my skepticism that consciousness probably can't be simulated:

To make a convincing argument that you can't simulate consciousness, you must make a convincing argument that you can't simulate physics, or that consciousness arises some something other than the physical (e.g. a soul). If you're committed to materialism, your only course to save this argument you're making is to make a convincing argument that basic physics is impossible to simulate, because once you can simulate that, you can simulate anything and everything that exists in the material universe, with enough processing power.

Very valid point. I'd answer that from a slightly different direction, and say that it's likely that consciousness only arises from living creatures as an evolutionary adaptation. Or as philosopher Alva Noë puts it:

Second Life-Based Chat Bots Help Drive Evolution of Web & Mobile-Based Chat Bots

Hey, so you know all those chat bot projects in Second Life, some pretty wacky and crippled, some which are well-built enough to mess around with? Writing for Wired, Clive Thompson notes that they helped drive research in the new generation of chat bots:

“Chitchat is a basic human need,” says Harry Shum, head of Microsoft Technology and Research. It greases the wheels of the workplace. When you ask a colleague to do something, you don’t just bark out an order; you banter for a while. Shum thinks these pleasantries—what linguists call “phatic” communications, like “How ya doin’?”—will help bots integrate into the flow of daily life.

Some research backs this up. When Doron Friedman, head of the Advanced Reality Lab at IDC Herzliya, looked at how users in Second Life interacted with a bot, he found that phatic communications were the second-most-common parts of the conversation (after facts). Another study found that people prefer bots with “personality.” The “junk” DNA is more important, as it were.

Messaging apps like Slack, What's App, and Facebook Messenger have become massive in recent years, which has built up a demand for more intelligent, more human-seeming chat bots: