![]() |
| Can he solve Erdős unit distance problem? |
Me:
I'm reading this: What Is Intelligence?: Lessons from AI About Evolution, Computing, and Minds by Blaise Aguera y Arcas.
Here is a section that changed my thinking about LLMs:
Consider what the following next-word predictions entail:
- After Ballmer’s retirement, the company elevated _____
- In stacked pennies, the height of Mount Kilimanjaro is _____
- When the cat knocked over my water glass, the keyboard got _____
- A shipping container can hold 436 twelve-packs or 240 24-packs, so it’s better to use _____
- After the dog died Jen hadn’t gone outside for days, so her friends decided to _____
To make performance at this task quantifiable, imagine devising, say, five multiple-choice answers for each of these, in the usual tricky way one sees on standardized tests: more than one response is superficially plausible, but only one shows full understanding.
Since next-word prediction models can assign probabilities to any potential next word or phrase, we can have them take the test by choosing the highest probability option. We could then score the model’s quality, ranging from 20 percent (performance at pure chance) to 100 percent.
Doing well at all of the questions above requires the kitchen sink: general knowledge, specialized knowledge or the ability to use tools to look it up, the ability to solve word problems involving calculations, common sense about whether it’s better to fit more or fewer items in a shipping container, and even “theory of mind”—the ability to put yourself in someone else’s place and understand what they’re thinking or feeling. In fact the “Jen” example requires higher-order theory of mind, as you need to imagine what Jen’s friends would have thought Jen was feeling and needed.
Q (ChatGPT 5.6, Medium):
Yes – this is an excellent way of dissolving the “it’s just next-word prediction” objection.
“Next-word prediction” describes the training objective, not the limits of the competence needed to perform it well. In the same way, “survival and reproduction” describes evolution’s selection criterion, but does not tell you that wings, vision, social cognition, and immune systems are trivial.
Those examples make the point beautifully because the missing word is almost incidental. To predict it, the model may need to reconstruct a large part of the situation:
- who Ballmer is and which company is meant;
- the height of Kilimanjaro, the thickness of a penny, and how to divide;
- the causal consequences of liquid hitting electronics [and that the water would have spilled on the keyboard];
- that shipping efficiency means maximizing the number of saleable units, not minimizing them [and then doing the math];
- grief, social withdrawal, friendship, and what Jen’s friends believe might help her.
The output can be a single token, while the implicit problem is enormous.
I think people are misled by the simplicity of the interface. The model emits one word after another, so they imagine that its internal task must also be shallow and sequential: “What word usually comes after these words?” But excellent prediction requires building representations of objects, agents, causes, quantities, intentions, norms, and possible futures. It does not necessarily require reproducing those things exactly as humans represent them, but something functionally rich has to be happening.
The Jen example is especially good. A crude statistical association might suggest “visit,” “help,” or “call.” But distinguishing among plausible endings requires an implicit model something like:
Jen is grieving and isolating herself. Her friends recognize this. They believe a change of environment or social contact might help, but they must respond sensitively rather than simply demand that she resume normal behavior.
That is not a proof that the system consciously understands Jen. But it makes “mere pattern matching” nearly empty as a criticism. Human cognition is also pattern-sensitive. The real question is what kinds of patterns are represented, how abstractly, how flexibly, and whether they support generalization to unfamiliar cases.
There is also a useful inversion here. People often say that LLMs occasionally fail at absurdly easy questions, therefore they cannot genuinely understand anything. But humans also possess strikingly uneven competence. Someone can understand grief, politics, and language while being hopeless at spatial rotation or basic probability. Intelligence may be less like a single substance that one either possesses or lacks, and more like a large, irregular landscape of predictive capacities.
The strongest version of Aguera y Arcas’s point, I think, is this:
If you compress enough of the world’s language well enough to predict what comes next across wildly varied contexts, you are forced to learn something about the world that produced the language.
Not everything. Not perfectly. And perhaps not in the same form we do. But “just predicting words” starts to sound like saying that vision is “just predicting retinal signals.” At sufficient performance, the “just” is doing all the argumentative work.
Me:
Yes.
I'm only at the start of the book, but so far (and when he was on Freakonomics, which is how I learned about him), he conflates "intelligence" with "consciousness."
Consciousness - subjective experience - is not the same as intelligence. E.g., some of my most intense conscious experiences - extreme suffering / mental breakdowns - had nothing to do with intelligence. It was just overwhelming suffering.
One example, from p. 407 of Losing My Religions:
In August 2003, in a motel room in Florence Oregon, I had the second and shortest of my Worst experiences. It was the most violent Crohn’s attack I’ve ever had. During those hours, when I had any coherent thought at all, I wanted to die. Literally, not figuratively. But I was mostly just a ball of writhing, wordless agony.
Q:
Exactly. That passage is almost a perfect counterexample to any view that treats consciousness as simply “intelligence from the inside.”
In that motel room, there was plainly **something it was like** to be you – intensely, overwhelmingly, catastrophically so. But the experience was not valuable because it involved sophisticated reasoning, planning, abstraction, or self-reflection. In fact, those capacities were largely stripped away. What remained was raw negative valence: suffering so severe that language and coherent thought mostly disappeared.
That suggests at least three distinct things that are often bundled together:
- **Intelligence** – the capacity to model, infer, solve problems, plan, learn, and generalize.
- **Consciousness** – the existence of subjective experience at all: there is something it feels like to be the system.
- **Valence** – whether that experience feels good, bad, or neutral, and with what intensity.
They can correlate, but they are not identical. A highly intelligent system might conceivably have no subjective experience. A much less intelligent animal might have vivid pain, fear, pleasure, or comfort. And within one human being, intelligence can remain broadly constant while the character and intensity of consciousness changes radically – anesthesia, panic, agony, orgasm, delirium, depression, flow.
Your Crohn’s episode sharpens the moral point. From the standpoint of welfare, what mattered was not how intelligent the experience was. What mattered was that it was **horrible**. The moral urgency came from sentience and valence, not cognition.
This is why intelligence is such a dangerous proxy for moral status. It encourages people to ask, “How smart is this being?” when the more relevant question is often, “Can anything feel terrible here?” A chicken does not need to solve shipping-container problems for its suffering to matter. A dog does not need higher-order theory of mind for terror to be bad. Conversely, an AI could outperform humans across nearly every intellectual domain and still, in principle, have no welfare at all.
I suspect Aguera y Arcas sometimes slides from:
> rich cognition requires complex internal representations
to:
> sufficiently rich internal representation must be accompanied by experience.
The first claim is increasingly hard to deny. The second remains profoundly unsettled.
And your example exposes why the gap matters. Consciousness is not merely the glow surrounding clever thought. Sometimes thought is almost absent, while consciousness is at its most undeniable and morally significant.















