Chat GPT4o May Seem Impressive But It Still Can't Accurately Summarize My Writing

Everyone in tech is seemingly psyched for the latest update of Chat GPT4o -- mainly for its genuinely impressive version of the AI from the movie Her (watch below) -- but for the record, it's still having trouble with at least one obvious question: What's my definition of the Metaverse?

Ask GPT4o that, and you get this farrago of flim flam:

Wagner James Au defines the Metaverse as a collective virtual shared space that combines virtually enhanced physical reality with persistent virtual reality. He sees it as an interconnected network of 3D spaces where users can interact with each other and the environment in real-time. The Metaverse is not just one platform but a convergence of various virtual worlds and augmented reality experiences, creating a seamless digital and physical blend.

Again, what I actually wrote on this blog years ago, and in several online articles in major news sites, and in the very opening page of my latest book, is this:

The Metaverse is a vast, immersive virtual world simultaneously accessible by millions of users through VR and other devices, highly customizable avatars and powerful experience creation tools. It is integrated with the real world economy and compatible with external technology.

Instead of outputting that, however, GPT4o insists on ascribing what others have written, then duct tapes it onto what I've quoted others having said. (Along with the usual hallucinations -- WTF is even "a seamless digital and physical blend".)

"I can see where it's picking up some of this stuff," my friend Adam Frisby, who ran the GPT4o query for me, surmises. "You're talking about other people and what they're doing, and ChatGPT is basically transposing you commenting as your personal opinion."

The fundamental problem, he suggests, is happening on a deep epistemic level:

"[ChatGPT l]acks theory of mind still. It can understand concepts -- but not the idea that you can hold differing views of the same concept. It's just treating it as there is only one."

Theory of mind, of course, is what children develop at around the age of three -- knowing that what you know or think is different from others might know or think. 

As for summarizing what writers in generl think, I am quite fairly sure this isn't an isolated incident that only I'm dealing with. (Microsoft Copilot, by the way, is already better at this.)

But hey, great Chat GPT4o voice recognition feature, which will absolutely never ever be used in a sadly creepy sexual way!

Comments

[n]

Summarization is when you provide text to the model and it generates a summary. LLMs are actually useful for that and it's a common application.

What you asked for, instead, was memory retrieval. LLMs are not databases. They generate responses based on patterns they've learned during training, not by retrieving specific pieces of information from their training data (that is many times larger than the model parameters, so it cannot remember everything precisely anyway).

Expecting an LLM to output a specific text verbatim from its training data is like asking an obscure painting to an image generator and expecting it to output a perfect replica.
To expect anything like that, you need at least the hyper-parameter temperature set to zero (to make it more deterministic) and the text you look for repeated a multitude of times in the training data. The more times it is repeated in the dataset, the more is reinforced.

Then you asked for web search. The free tier of ChatGPT is getting GPT-4o and also internet search, but ChatGPT internet search a little rudimentary, compared to Copilot. Copilot set as "more Precise" ( https://postimg.cc/5Qj1fFjz ) finds and outputs your definition correctly: https://postimg.cc/qg7psWQ7

[n]

As a developer, I find GPT-4o to be a truly remarkable advancement in the field.
Neural networks allow what was once nearly impossible, such as NLP (Natural Language Processing), as well as interacting by using natural language (not just a limited set of sentences) and image recognition. You can see a humorous take on this by xkcd: https://xkcd.com/1425/

GPT-4o is not just another LLM. It's fully multi-modal.
It's also faster and more cost-effective to operate than both the original GPT-4 and GPT-4 Turbo (optimizations aside, perhaps it's also a smaller model). It's weaker in certain areas, but an improvement in others. However, the most significant part is the multi-modal capabilities. It can process and generate any combination of text, audio, and image inputs and outputs without delegating that to other neural networks (the full capabilities will be released in the next weeks).

Hence, it can do what it did in the demo, but it can do much more, see "Explorations of capabilities" here: https://openai.com/index/hello-gpt-4o/

[Winifred]

The only people who use this garbage are programmers which is why programmers think it will take over the world. They think coding is the pinnacle of human achievement and that if this thing can code, it can clearly do everything else.

What they don't realize is that this thing is a spinning jenny for coding. It's already putting record numbers of programmers out of work. Coding is now a pretty much worthless skill. You're more likely to find a job in the 21st century if you know how to use a loom.

[¯\_(ツ)_/¯]

"This new oven may seem impressive, but it still can't store and keep my fish fresh! Plus, you could burn your fingers! Ovens are bad."
Yeah, but do you also consider what it's designed for? Baking, roasting, broiling...? There are so many different possibilities to cook and process your food.

Nope, once again only testing ovens as refrigerators to dismiss them. LOL!

On the one hand, there are folks already expecting magical ovens that provide them with free food. On the other hand, there are confidently incorrect haters who cling to their prejudices and insist on parroting the pervasive and misleading arguments.
The competent devs who patiently try to explain that it's neither of the cases, and provide information, papers and examples are systematically ignored or told they don't understand, their work is called garbage.

I can understand how doctors felt when they tried to explain the basics of the immune system to anti-vaxxers.