I'm not sure why "The contents of `I` is nowhere modelled by "projection" because this does not model composition, and is not relevantly discrete and bounded by logical connectives."
In practical terms, what do you think the LLM output cannot contain right now? Because the way I read it now is "LLM can't speculate". But that's trivial to disprove by asking for that happy dog on Mars speculation you have as an example - whether you want the scientific version, or child level fun, it's available and the model will give nontrivial idea connections that I could not find anywhere. (For example childlike speculation from Claude included that maybe dogs would be ok playing in spacesuits since some dogs like wearing little coats)
Similarly "And the intensions here do not compose like the extensions." is really high level. What's the actual claim with regards to LLMs?
LLMs are just a token->token mapping. They can output any set of tokens for any set of input tokens. So there is no output which isn't in the domain or codomain.
The issue is why one (prompt, answer) pair is given. If the answer is given as a "reasoning process" over salient parts of the prompt, that, e.g., involves imagining/simulation as expected, then for {(prompt', answer')} of similar imaginings we will get reliable mappings. If its cheating, then we wont.
We can, I think, say for certain that the system is not engaged in counterfactual reasoning. Eg., we can give a series of prompts (p1, p2, p3...) which require increasing complexity of the imagined scenario, and we do not find O(answering) to follow O(p-complexity-increase). Rather the search strategy is always the same, and we can just get "mildly above linear" (pseudo-)reasoning complexity with chain-of-thought.
> LLMs are just a token->token mapping. They can output any set of tokens for any set of input tokens. So there is no output which isn't in the domain or codomain.
This applies the same to humans hearing a question and responding. Tokens in, tokens out (whether words or sound). It's not unique to LLMs, so not useful for explaining differences.
> then for {(prompt', answer')} of similar imaginings we will get reliable mappings. If its cheating, then we wont.
You're not really showing that this is/isn't the case already. Also this would put people with quirky ideas and wild imagination in the "cheating" group if I understand your claim correctly. There's even a whole game around a similar concept - Dixit - describe an image in a way that as few people as possible will get it.
> we can give a series of prompts (p1, p2, p3...) which require increasing complexity of the imagined scenario, and we do not find O(answering) to follow O(p-complexity-increase). Rather the search strategy is always the same
You're describing most current implementations, not a property of LLMs. Gemini scales the thinking phase for example. Future models are likely to do the same. Another recent post implemented this too https://news.ycombinator.com/item?id=44112326
All physical systems have "intrinsic properties" and "measure properties".
Eg., coffee has some internal kinetic energy in the motion of its molecules, and it has the disposition to cause a thermometer to rise its mercury to a certain height.
There's always an open question in these cases: is the height of the mercury a "reliable stand-in" for the temperature of the system? In many case: NO. If you read-off the height too quickly, you'll report the wrong temperature.
No system's intrinsic properties is, litearlly, just its measure properties. We are not literally our behaviours. An LLM is not literally its input/output tokens.
The question arises: what is the actual intrinsic property which gives rise to the measured properties?
it's very easy to see why people believe that the LLM case is parallel to the human case, becuse in ordinary circumstances, our linguistic behvaviours are "reliable measures" of our mental states. So we apply the same perception to LLMs: so to must they generate outputs in the way we do, they must "Reason".
However, there are many much more plausible explanations of how LLMs work that do not resort to giving them mental capacities. And so what's left to those in possession of such better explanations is to try to explain to others why you cannot just put a thermometer into a xbox cd drive and think you're measuring how hot the player is.
In practical terms, what do you think the LLM output cannot contain right now? Because the way I read it now is "LLM can't speculate". But that's trivial to disprove by asking for that happy dog on Mars speculation you have as an example - whether you want the scientific version, or child level fun, it's available and the model will give nontrivial idea connections that I could not find anywhere. (For example childlike speculation from Claude included that maybe dogs would be ok playing in spacesuits since some dogs like wearing little coats)
Similarly "And the intensions here do not compose like the extensions." is really high level. What's the actual claim with regards to LLMs?