Can Midjourney Actually Build a Holodeck Despite Technical & IP Infringement Hurdles? Reid Southen Has Doubts.

Matrix Resurrections artwork by Reid Southern (copyright Warner Brothers)

Midjourney plans to make some kind of open world "holodeck", as CEO David Holz confirmed with me earlier this month. But that actually just begs the question: Can it even feasibly do so? 

The company is already careening toward IP infringement challenges for its 2D image creator, based on reports of rampant copyright infringement, including a lawsuit led by artist Jon Lamm, alleging that Midjourney staff (including Holz) have intentionally "laundered" the work of thousands of artists into its training database without consent.

So what happens when Midjourney tries creating a platform with immersive 3D content?

Film industry concept artist David Southen, who's worked on Marvel and DC movies, not to mention the Matrix and Alien franchise, is highly doubtful.

"In terms of them pulling something like this off without IP issues?" as he puts it. "They can't even pull off what they're currently doing without IP issues, and I doubt they're being more careful with their dataset than previously, so I would say no."

Southen would know, and not just because of his unimpeachable geek artist cred. He recently wrote this report with renowned AI ethicist Gary Marcus, highlighting the existing problem:

These results provide powerful evidence that Midjourney has trained on copyrighted materials, and establish that at least some generative AI systems may produce plagiaristic outputs, even when not directly asked to do so, potentially exposing users to copyright infringement claims. Recent journalism supports the same conclusion; for example, a lawsuit has introduced a spreadsheet attributed to Midjourney containing a list of more than 4,700 artists whose work is thought to have been used in training, quite possibly without consent. For further discussion of generative AI data scraping, see Create Don’t Scrape.

How much of Midjourney’s source materials are copyrighted materials that are being used without license? We do not know for sure. 

More on this on Marcus' blog.

And unlike most 2D artists, high quality 3D files tend to be owned by folks like major game publishers and movie effects houses who won't exactly be thrilled about giving away their content for free -- and have full-time lawyers on staff to protect it.

As for the technical challenges, Reid continued:

Midjourney developers caught discussing laundering, and creating a database of Artists (who have been dehumanized to styles) to train Midjourney off of. This has been submitted into evidence for the lawsuit. Prompt engineers, your “skills” are not yourshttps://t.co/wAhsNjt5Kz pic.twitter.com/EBvySMQC0P

— Jon Lam #CreateDontScrape (@JonLamArt) December 31, 2023

"I think David Holz is talking a much larger game than he's capable of backing up. He's been shown in the past seemingly brainstorming and spitballing ideas openly on Midjourney's office hours, but nothing about the technology they've demonstrated would indicate we're anywhere close to 'holodeck' technology. Much larger corporations with far more employees and far deeper pockets have been working towards similar goals for much longer, and haven't gotten anywhere close to a 'holodeck'.

"I don't doubt that David has some sort of 'realtime' version of Midjourney's image generation in the pipeline, whether it's something akin to video game rendering, or something with VR, but calling it a 'holodeck', which is a very specific type of technology from Star Trek, is likely very misleading."

The other issue is computing cost required by generative AI.

"I don't know if Midjourney has proven to be profitable yet," as Reid puts it, "but AI companies like Midjourney need to keep raising capital in order to cover their R&D and immense server costs, and I personally think that David has a vested interest in keeping the hype train trolling at all costs."

I asked David Holz for a comment to all this yesterday and will update this post if I get a reply.

For my own part, all this talk of a holodeck misses the fact that there are already thousands if not millions of people creating relatively high quality immersive content on metaverse platforms... and actually enjoy doing it, and make money from doing so. So a holodeck sure seems like a legally and technically expensive problem where a democratic solution already exists. 

As for Reid, follow him on Twitter/X below!

So not only do they need copyrighted material for their technology to function, but energy demands are getting so high (think small country), they also need a nuclear fusion breakthrough to sustain it. The whole thing is such a joke. https://t.co/VczeiPGj34 pic.twitter.com/yzeZyiwCED

— Reid Southen (@Rahll) January 22, 2024

Please support posts like these by buying Making a Metaverse That Matters!

Comments

[Nadeja]

"misses the fact that there are already thousands if not millions of people creating relatively high quality immersive content on metaverse platforms... and actually enjoy doing it, and make money from doing so."

Most users only reuse "high quality" objects made by others. Someone can create a scenery with them. Only a few hobbyists master tools like Blender or Maya, and even fewer monetize their work. Professional software requires a lot of learning, time, and energy. If it's a job, it is worth your time and energy. But as a hobby? It's not for everyone.

Ideas are abundant, but skills, time, and energy are scarce.

The more you raise the bar, the fewer people do it.

You can see that also in Second Life. A lot of the clothing nowadays are templates from only a handful of creators (and sometimes vehicles and other stuff downloaded from websites or ripped from videogames).
What most SLers do nowadays is e.g. buying already-made houses, furniture and tress to put together a nice place. That may still be fun, but do you remember the old sandboxes? When people were creating their own houses, castles and everything else?

When creating is simpler, it is more accessible and enjoyable for a wider audience, thus you have many more people creating (and actually enjoying doing so in their free time), in turn you have *more creations*. How fun Minecraft would be if you had to learn Blender to play with it?
What if I could create the butterly castle of my dreams by just describing it with natural language, instead? A much more accessible tool and creativity for everyone. And those with the most creative ideas still shine.

[Nadeja]

As for the "much larger corporations with far more employees and far deeper pockets" I won't underestimate the smaller, but more agile companies. Who made a successful virtual world? Meta with Horizon? Linden Lab with Sansar? Or a couple of indie devs with VRChat? And who made the best image generators that you can use so far, Google, Microsoft? Midjourney 6 itself pretty much rivals DALL-3 and it's even better in some regards.

What about OpenAI vs Google and others? Eventually Microsoft invested in OpenAI and now OpenAI has a huge hardware availability.
Anything may happen, but I won't exclude that Midjourney will follow a similar path. David Holz very likely has interest in hopping onto the hype train also because of this and I'd take what CEOs tell with a pinch of salt.

Anyway a sort of real-time is here already:
https://stability.ai/news/stability-ai-sdxl-turbo
https://arstechnica.com/information-technology/2023/11/stable-diffusion-turbo-xl-accelerates-image-synthesis-with-one-step-generation/
https://medium.com/the-generator/real-time-ai-image-generation-is-finally-here-c4d66fa954f7
In a year or few years? Who knows, but these systems are progressing quickly so far, also breakthroughs can happen (as well as wars and insane presidents, though) and new hardware is coming. There are things that are more or less likely within a certain time-frame, but eventually doable.

As for the IP issues is a complex matter with gray areas. I'm not a lawyer, but I can see that a sentence could impact the development too. On the one hand you have always people training themselves by looking at other people's works (whether in a book, in your local museum, square or on the Net), but then they can develop their own unique style even though with similarities. That sounds fine to me. i.e. although Artemisia Gentileschi was influenced by Caravaggio, her works weren't a plagiarism. On the other hand, if you use my material for an investments of billions of dollars, it makes sense that someone wants a slice.

[Mu]

People who create assets create them to be used as assets. And a lot of the time those assets get paid for.

You want to use artists work to create your own? Pay up or learn to make your own stuff. That’s all there is to it. We put our energy time and love into the things we make. And unless someone gave us a contract saying “we’ll pay you a living wage and fair royalties/licensing for all of your hard work to create assets for other people to use for theirs” it’s not yours to use.

AI creators need to get over themselves. Seriously.