Being able to discern its word soup as a description of what constitutes a game is more a demonstration of you understanding what a game is, not it.

And therein lies the magic of ChatGPT: it knows nothing, but the projected significance of the token arrangements it creates tugs on what we know, and that causes a profound feeling akin to interacting with something intelligent.

Lets rephrase this differently, no one can prove whether it “understands” or not, and the same for humans. No one can prove whether consciousness is an “illusion” for humans. Your brain might just use embeddings, too.

If you ask an average person if children understand what a game is, “yes” would be a common answer, but just like an AI model it may not have enough focus to play the game properly.

I believe this is the whole point behind “the turing test”.

If you instead argued it can’t experience emotion, i would agree.

You seem to be conflating solipsism with epistemology. At the end of the day, regardless of whether you're dealing with a child, a bot, or a peer, you would never use the question "describe how to play chess" as a measure of understanding if the chosen entity understands how to play chess; you would just play chess with them. Such an inquiry would only probe if one understands how to explain how to play chess, which is not the same. One should expect an LLM model of being able to regurgitate a description of playing chess. One could even expect an LLM to regurgitate common strategies in chess, enough to fool some into thinking it understands what is happening amidst a posed match. Where it all falls apart is where the rubber meets the pavement. Get an LLM to teach your child chess, with no intervention. You'll learn rather quickly which one is capable of understanding, and which one isn't.

Clearly GPT doesn't "know" things in the way we do- I'd argue that there's probably a little bit of world modeling in there, but piecemeal and crude at best. Only so much you can do with such limited iteration. However, we must acknowledge that there are evidently some tasks that don't require knowledge the way we usually think about it. For a human, being able to talk about a game stems from the same place as our ability to understand one- our knowledge. But for LLMs, you get one without the other. The fact that ChatGPT doesn't "know" what a game is isn't sufficient to explain why it can't play one. It would be in a human, but clearly ChatGPT can perform tasks without knowledge that we'd need knowledge for, since it can talk about games just fine! So why not this one? Your hypothetical perfect sentence predictor could absolutely play a totally novel game, after all. It is a worthwhile question to ask why this particular flawed predictor can do one but not the other.

I’ll give you an example to think about in human terms.

In political discussions, a lot of humans don’t really know what the fuck they’re talking about, but they do know what to say in response to certain stimulus. They know the talking points, the key phrases, the terms, they parrot these back to you when you provoke them to say it. They are basically human sentence predictors. Stuff comes out of their mouth based on what they’ve been trained to think the next word should be. They don’t pause to reflect on some gathered knowledge and then present an observation to the world.

This is basically what GPT is, but with everything. And the only way GPT plays a game, is if the state of the game can sufficiently activate some output that represents the next best move for the game being played, which isn’t really a readily available data set for all games, especially for made up games.

Thank you for that example. I've been thinking a lot about how difficult it is to persuade people to change their mind. Most people seem impervious to incorporating new facts or ideas into their internal narrative (myself included). ChatGPT has made me wonder if that is because much of what we consider "cognition" in humans is really just "human sentence predictors".

We've all been "trained" with various facts and when two people meet who have been trained on substantially different bodies of knowledge/facts/experiences it can be very difficult to find common ground.

FWIW, I asked ChatGPT to give me a short list of cognitive biases and psychological phenomena. Interesting to think how many of these are dependent on our personal "training data":

    Confirmation Bias
    Cognitive Dissonance
    Anchoring Bias
    Belief Perseverance
    Groupthink
    Ingroup Bias
    Sunk Cost Fallacy
    Motivated Reasoning

That is indeed an example, but it still isn't an explanation. Sure, humans can also generate output without understanding- I've done it myself for my undergrad, throwing words together at 2am to make a deadline. I think quite a few people have remarked that LLMs seem to write like a human that isn't paying attention, which squares with what you said.

But the question remains why this is possible! A good enough pure predictor could play novel games or devise new theorems, but GPT absolutely can't. It can however give confident explanations, as well as write passable code and occasionally even do some novel problem solving (simple, impressive only because it comes from a computer, but still there). The question of why it can do some things and not others is interesting, and can't be swept under the rug just by reiterating that it's a predictor.

Does the structure of language really do such a good job of conveying information that GPT can operate on it blindly and get results, Blindsight style? Is composing prose far easier than we expect, leaving the bulk of the model free to do a tiny amount of "reasoning" that we find unjustly impressive because of how well it's presented? Is it handicapped primarily by the fact that it can only carry out extremely short computations, and can't be trained to use chain-of-reasoning to get around the limitation? We have no idea what, if any, inherent limitations predictive models have. We have no idea why GPT-sized models are good at the things they are, and bad at the things they aren't.

How is this "illusion" different from humans knowledge of things?

(We don't know)

GPT for instance doesn’t understand how to measure utility of things for different contexts. It doesn’t know.

That gives a limitation of knowledge, and says nothing about whether there is a fundamental difference.

Your premise is the illusion of things are the same as the actual things.

Not sure what else I can really say about that. They are not the same.

No, my premise is that we don't know whether there is a distinction. I asked how they are different, and you haven't given an actual difference.