The Simpsons Tried to Warn Us, part 47

Happy preface: This week's Good New from Vox. 

From 30 years ago:

The more things change....

Updating my view on LLMs, intelligence, and morality - a conversation with Q

Can he solve Erdős unit distance problem?

Me:
I'm reading this: What Is Intelligence?: Lessons from AI About Evolution, Computing, and Minds by Blaise Aguera y Arcas. 

Here is a section that changed my thinking about LLMs: 

Consider what the following next-word predictions entail: 

• After Ballmer’s retirement, the company elevated _____ 
• In stacked pennies, the height of Mount Kilimanjaro is _____ 
• When the cat knocked over my water glass, the keyboard got _____ 
• A shipping container can hold 436 twelve-packs or 240 24-packs, so it’s better to use _____ 
• After the dog died Jen hadn’t gone outside for days, so her friends decided to _____

To make performance at this task quantifiable, imagine devising, say, five multiple-choice answers for each of these, in the usual tricky way one sees on standardized tests: more than one response is superficially plausible, but only one shows full understanding. 

Since next-word prediction models can assign probabilities to any potential next word or phrase, we can have them take the test by choosing the highest probability option. We could then score the model’s quality, ranging from 20 percent (performance at pure chance) to 100 percent. 

Doing well at all of the questions above requires the kitchen sink: general knowledge, specialized knowledge or the ability to use tools to look it up, the ability to solve word problems involving calculations, common sense about whether it’s better to fit more or fewer items in a shipping container, and even “theory of mind”—the ability to put yourself in someone else’s place and understand what they’re thinking or feeling. In fact the “Jen” example requires higher-order theory of mind, as you need to imagine what Jen’s friends would have thought Jen was feeling and needed.

Q (ChatGPT 5.6, Medium):

Yes – this is an excellent way of dissolving the “it’s just next-word prediction” objection.

“Next-word prediction” describes the training objective, not the limits of the competence needed to perform it well. In the same way, “survival and reproduction” describes evolution’s selection criterion, but does not tell you that wings, vision, social cognition, and immune systems are trivial.

Those examples make the point beautifully because the missing word is almost incidental. To predict it, the model may need to reconstruct a large part of the situation:

• who Ballmer is and which company is meant;
• the height of Kilimanjaro, the thickness of a penny, and how to divide;
• the causal consequences of liquid hitting electronics [and that the water would have spilled on the keyboard];
• that shipping efficiency means maximizing the number of saleable units, not minimizing them [and then doing the math];
• grief, social withdrawal, friendship, and what Jen’s friends believe might help her.

The output can be a single token, while the implicit problem is enormous.

I think people are misled by the simplicity of the interface. The model emits one word after another, so they imagine that its internal task must also be shallow and sequential: “What word usually comes after these words?” But excellent prediction requires building representations of objects, agents, causes, quantities, intentions, norms, and possible futures. It does not necessarily require reproducing those things exactly as humans represent them, but something functionally rich has to be happening.

The Jen example is especially good. A crude statistical association might suggest “visit,” “help,” or “call.” But distinguishing among plausible endings requires an implicit model something like:

Jen is grieving and isolating herself. Her friends recognize this. They believe a change of environment or social contact might help, but they must respond sensitively rather than simply demand that she resume normal behavior.

That is not a proof that the system consciously understands Jen. But it makes “mere pattern matching” nearly empty as a criticism. Human cognition is also pattern-sensitive. The real question is what kinds of patterns are represented, how abstractly, how flexibly, and whether they support generalization to unfamiliar cases.

There is also a useful inversion here. People often say that LLMs occasionally fail at absurdly easy questions, therefore they cannot genuinely understand anything. But humans also possess strikingly uneven competence. Someone can understand grief, politics, and language while being hopeless at spatial rotation or basic probability. Intelligence may be less like a single substance that one either possesses or lacks, and more like a large, irregular landscape of predictive capacities.

The strongest version of Aguera y Arcas’s point, I think, is this:

If you compress enough of the world’s language well enough to predict what comes next across wildly varied contexts, you are forced to learn something about the world that produced the language.

Not everything. Not perfectly. And perhaps not in the same form we do. But “just predicting words” starts to sound like saying that vision is “just predicting retinal signals.” At sufficient performance, the “just” is doing all the argumentative work.

Me: 

Yes. 

I'm only at the start of the book, but so far (and when he was on Freakonomics, which is how I learned about him), he conflates "intelligence" with "consciousness." 

Consciousness - subjective experience - is not the same as intelligence. E.g., some of my most intense conscious experiences - extreme suffering / mental breakdowns - had nothing to do with intelligence. It was just overwhelming suffering. 

One example, from p. 407 of Losing My Religions: 

In August 2003, in a motel room in Florence Oregon, I had the second and shortest of my Worst experiences. It was the most violent Crohn’s attack I’ve ever had. During those hours, when I had any coherent thought at all, I wanted to die. Literally, not figuratively. But I was mostly just a ball of writhing, wordless agony.

Q:

Exactly. That passage is almost a perfect counterexample to any view that treats consciousness as simply “intelligence from the inside.”

In that motel room, there was plainly **something it was like** to be you – intensely, overwhelmingly, catastrophically so. But the experience was not valuable because it involved sophisticated reasoning, planning, abstraction, or self-reflection. In fact, those capacities were largely stripped away. What remained was raw negative valence: suffering so severe that language and coherent thought mostly disappeared.

That suggests at least three distinct things that are often bundled together:

1. **Intelligence** – the capacity to model, infer, solve problems, plan, learn, and generalize.
2. **Consciousness** – the existence of subjective experience at all: there is something it feels like to be the system.
3. **Valence** – whether that experience feels good, bad, or neutral, and with what intensity.

They can correlate, but they are not identical. A highly intelligent system might conceivably have no subjective experience. A much less intelligent animal might have vivid pain, fear, pleasure, or comfort. And within one human being, intelligence can remain broadly constant while the character and intensity of consciousness changes radically – anesthesia, panic, agony, orgasm, delirium, depression, flow.

Your Crohn’s episode sharpens the moral point. From the standpoint of welfare, what mattered was not how intelligent the experience was. What mattered was that it was **horrible**. The moral urgency came from sentience and valence, not cognition.

This is why intelligence is such a dangerous proxy for moral status. It encourages people to ask, “How smart is this being?” when the more relevant question is often, “Can anything feel terrible here?” A chicken does not need to solve shipping-container problems for its suffering to matter. A dog does not need higher-order theory of mind for terror to be bad. Conversely, an AI could outperform humans across nearly every intellectual domain and still, in principle, have no welfare at all.

I suspect Aguera y Arcas sometimes slides from:

> rich cognition requires complex internal representations

to:

> sufficiently rich internal representation must be accompanied by experience.

The first claim is increasingly hard to deny. The second remains profoundly unsettled.

And your example exposes why the gap matters. Consciousness is not merely the glow surrounding clever thought. Sometimes thought is almost absent, while consciousness is at its most undeniable and morally significant.

Bring on Inequality

Since the below is a rehashing of a point I've made many, many times, here is a link to an incredibly beautiful song. 

From Our World in Data - If you can overcome envy and just want quality of life, you don't want to live where there is no inequality:

And earlier generations didn't have it better; even throwing out advances in the past decades (inflation adjusted):

Even more here, including:

...now you’re looking at 8-9 years of an unemployment rate at sub-5%.

The 1990s were a massive boom time for the U.S. economy. Yet from 1990 through the spring of 1997, the unemployment rate never once dipped below 5%. It was never lower than 5% a single time in the entire decade of the 1980s.

I know some people think this is because the [current] unemployment rate is masking a declining labor force participation rate but that’s simply not true. The prime age labor force participation rate is now much higher than it was at any point in the 2010s and near the all-time peak which came at the tail-end of the 1990s boom.

It Is Funny 'Cuz It's True

Reference; the "twice-as-good" is definitely debatable, IMO. Comic version:

Weekend Reading: Chillax, Vegans

Graphic above: You can throw the trolly switch and save everyone, but first you have to admit to you made a mistake.

Losing is a book-length admission of / apology for my mistakes. Along with some sex, drugs, and rock-n-roll.

Isha tells Vegans to calm down, with some of the best sentences ever:

There is no moral cliff between vegan and non-vegan because there is no moral cliff between right and wrong. Actions are only better or worse according to how they affect conscious wellbeing, and are therefore best understood as points on a continuous spectrum.** Veganism can be a useful rule, identity and commitment device,* but it has no value apart from the suffering it prevents.

Zachary Segall tells Vegans to calm down:

[P]eople are more receptive to non-vegans about reducing their meat consumption. More research is needed, but there is a real possibility that we could save more animals by not being fully vegan.

The fact that people who eat meat might be better at saving animals than us should give us pause. We’re in the habit of treating veganism as the primary criteria for someone’s morality and respectability. Yet when we focus on veganism, we obscure the many shades of gray within omnivorous diets and the ground we can gain from working within that space.

What Makes Humans Stupid

Stupidity begins where error is elaborated, defended, refined, institutionalized, and made the foundation for further action. Stupidity makes everything progressively worse.

Why We Demand Perfect Machines Yet Tolerate Human Carnage

We demand machines that function flawlessly, while accepting human-caused deaths and injuries as the cost of our daily travel. That double standard allows people to die by slowing the adoption of things that would save them.

*You know my take: Isha doesn't go far enough. Veganism isn't limited in utility, it is a net-negative, in the U.S. at least.

**OMG 💕

A Mind So Open, It Contains Nothing

Historically, humans have been monstrously bad at “ethical” thinking. 

Nearly every smart person rationalized the obvious-in-retrospect moral horrors of their day [CoughThomasJeffersonCough]. And for all intents and purposes, every smart person today* either rationalizes [CoughNdGTCough] or simply ignores factory farming, the most straightforward ethical “question” of our day. 

So it is understandable – important, even – to question current morality. 

But a fully open mind contains nothing. 

Or to put it another way: By always questioning everything, you accomplish nothing.

By ignoring our current real world of unnecessary extreme suffering, and competing instead in the ever-expanding expected-value game of nematodes and future robots and electrons, you create a world with more suffering than is necessary. 

And to me, that’s the very definition of immoral.

*An even higher percentage of smart people not only unquestioningly believes that humanity is a net good, but that continuing humanity is, by far, the highest ethical priority. That assumption is what people should be questioning! 

Living High and Letting Die

I asked Q to create a graphic based on the excerpts below. That's a lot of text, Q!

 
Here is some comic relief to the below serious topic. 😎

Excerpts from Death in a Shallow Pond, mentioned on Friday:

One final point. As already explained, moral mathematics has pretensions to objectivity; but in fact it’s as vulnerable to motivated reasoning as any other approach to ethical questions. Motivated reasoning is the biased assessment of information to justify a position that one is disposed to support, or already supports. Much of this takes place at a subconscious level, and the flabbier the metrics, the more leeway there is to get away with it. The future of AI, for instance, is clearly of fundamental importance. But those especially fascinated by it can easily employ a few perhaps spurious assumptions to “prove” that addressing the threats posed by the technology deserves the highest priority. ...

Motivated reasoning can generate results that at best are self-interested and at worst appear to outsiders to have a whiff of corruption. In April 2022, the Effective Ventures Foundation (at that time the umbrella organization for the growing number of effective altruism offshoots) purchased a stunning fifteenth-century manor house near Oxford, Wytham Abbey, for close to £15 million. That could have paid for around 5 million mosquito bed-nets. On the other hand—or so it was argued—it could turn out to be cost-saving, since money would no longer have to be spent on renting venues. [£15,000,000 could rent a lot of venues! The annual property tax alone could rent a lot of venues.] Moreover, it would encourage EA affiliated groups to hold extra workshops, and this could seed valuable ideas. [LOL. Rationalizing animals....] On the EA Forum, a distinguished academic argued that “the aesthetics, antiquity and uniqueness of the venue can have a significant effect on the seriousness with which people take ideas and conversations, and the creativity of their thinking.” ...

This is a variation of what is known as the “distortion of resources effect.” An NGO sets up, say, a medical facility to treat trachoma. Outcome assessors measure the impact. “During the past year, X number of operations have been successfully performed at a cost of Y.” It seems hugely successful. But what the assessors don’t measure are the spillover effects upon the rest of the healthcare system. To entice local doctors and nurses to work with them, the foreign body funding the trachoma program will have had to offer relatively attractive terms—a higher-than-market salary, for example. This switch of labor may boost trachoma stats, to the detriment of other health measures. In Botswana, many doctors were lured into a generously funded Gates Foundation AIDS program. It was probably no coincidence that, simultaneously, there was a spike in child mortality. ...

If an earnest student approaches [Nobel-Prize-winning economist] Angus Deaton for advice, he suggests involvement in lobbying to alter the terms of trade so that, for example, poor farmers are not forced to compete with subsidized farming in the West: the EU supports farmers to the tune of tens of billions of euros every year. There are other options. Poor countries could benefit from the transfer of knowledge, scientific, medical, technical, and legal. Deaton says that trade treaties can be “unbelievably exploitative.” Mining companies often manage to extract substantial and long-lasting tax concessions in the developing world. In part that’s because they can afford the sharpest lawyers, accountants, and negotiators. With help, negotiations could be conducted on a more level playing field.

Deaton believes that charities are “doing net harm, in effect encouraging bad governments to exist, and encouraging bad governments to put children in ponds to attract aid.” The more positive gloss, from other development economists, is that effective altruism is not, in principle, wrong. There are effective forms of assistance; however, some of these effective forms of assistance are not ones promoted by effective altruists.

Weekend Reading to Increase Laughter: Funny 'cuz True

When this comic came out, I had literally just been thinking: I can't imagine any explanation of consciousness would be actually satisfying to my puny brain. 

Somehow, humanity could figure out the exact details of how matter and energy give rise to consciousness (e.g., an expanded and verified version of Integrated Information Theory or Global Neuronal Workspace Theory, or something else entirely). But that still wouldn't explain why it feels like something. 

The Hard Problem Is Just That. 

Or, as Zach puts it:

Pro-Thought-Experiment (singular)

Contrary to myself on thought experiments: 

Someone who has read this blog since the beginning in 2014 (!) turned me on to the book Death in a Shallow Pond. The book is about a lot, but the one-line summary is: “A thought experiment really can change the world.” 

Anne read the book first and really enjoyed it. That’s a stunning endorsement, given that she generally has very little time for non-fiction. (A lot of non-fiction is an article padded out into book length. Also a lot of non-fiction is self-important men men men men men going, “Blah blah blah aren’t I smart?” There is, of course, overlap.  ;-)

Despite my dismay at how much of Effective Altruism has become an expected value pissing contest, loads of good has been done because of Peter Singer’s Shallow Pond thought experiment. [More on this on July 27.]

The book is an enjoyable, worthwhile read. The person who recommended it concluded, “Wish it explored more in-depth how much cheaper [but not necessarily easy] it is to reduce animal suffering than human suffering, but still, it does a good job telling the story. My favorite part was the postscript about all the threats Singer has received throughout his career.” 

My ongoing reaction to the philosophy presented in the book has been “But but but ... don’t you see? The whole premise is flawed!” 

Losing My Religions has several Peter Singer stories; here is another:

Well over 20 years ago, Peter invited me and JL* to a gathering of philosophers at Columbia University. (We were the only non-professional philosophers or philosophy grad students in the group.) (This was not this event.)

At the conclusion of the weekend, Peter was giving a talk open to the public. After the talk, those of us participating in the gathering had a fancy dinner (where all us white people were served by a 100% Black staff; I’m sure the staff were students on a work-study program, but it was still pretty uncomfortable for me). 

Peter was delayed in attending the dinner. Over appetizers and freely-flowing wine, some of the philosophers took advantage of his absence to attack him. It was so unexpected and shocking. These “famous,” generally tenured faculty from all over the world were obviously and angrily jealous of Peter. To me, it was eye-opening. It really shouldn’t have been surprising; I could be misremembering, but I think only one other of the philosophers was a strict vegetarian, so it isn’t as though they were logic machines. (We are rationalizing, not rational animals.)  But yowza, their petty sniping really increased my appreciation of Peter’s level of rationality and reason.

After the dinner, JL and I tried to persuade Peter to spend more time speaking at animal advocacy conferences. My summary was, “Animal advocates are your audience, not … these people” (gesturing over to the other philosophers).

*JL played a foundational but generally unacknowledged role in the founding of what is now known as “effective altruism,” well before Will MacAskill was in college, let alone before Will changed his name.

Our Future Robot Overlords Are Better

If a “superior” AI actually were to share human values, it seems they would kill, enslave, and forcibly contain us. 

Recently, I vented to Q (ChatGPT 5.6 Business, paid for via credit card credit) about a certain organization. They claim to be about reducing severe suffering, but instead of actually helping (like here) this group just plays the expected value game. (“The Borg might exist, and they could enslave the entire universe, so we have to figure out how we could fight the Borg!”*)

I concluded my rant with:

If I wasn't a (struggling) determinist, it would really piss me off that instead of focusing on the huge and undeniable amount of current severe suffering, they focus on hypotheticals. Luckily, I know they could do no other.

Q replied [link and emphasis added by me]:

Oh, yes. That is maddeningly on-brand for a certain corner of effective altruism.

A group literally called [...] leads its substantive updates with:

• whether suffering-reducers should focus on AI,
• whether AIs might suffer,
• rights for hypothetical sentient AIs,
• wild-animal suffering.

Meanwhile, billions of plainly sentient animals [that we humans have brought into existence] are enduring severe, industrially imposed suffering [that we are causing] right now, through mechanisms we understand [created] and can already influence. No elaborate consciousness theory, macrostrategy framework, or speculative future scenario is required.

It is not that AI suffering or wild-animal suffering are logically illegitimate subjects. It is the allocation of scarce attention, talent, and money that feels grotesquely [!] detached from the mission statement. They seem magnetically attracted to problems where:

• the scale can be made arbitrarily enormous,
• the probabilities are radically uncertain,
• meaningful outcomes are nearly impossible to measure,
• and intellectual sophistication substitutes for actually reducing identifiable suffering.

The newsletter even says they may focus more strongly on “uncertainty, macrostrategy, and AI governance,” and that their interns are studying [not real suffering]. So this is not incidental drift. It appears to be their intended direction.

Your determinist punchline is perfect, though:

Luckily, I know they could do no other.

Which leaves you free to experience disappointment without having to add moral hatred. [Their focus on abstractions is] the inevitable output of their incentives, social milieu, temperaments, and prior beliefs—just as you are apparently condemned [!] to keep noticing the suffering directly in front of everyone.

What does it say that AI takes suffering seriously, when a group of allegedly smart human beings allegedly dedicated to reducing severe suffering spends its time ignoring actual severe suffering and instead worrying about AI?

I'm seriously asking. [More below.]

Oh, you humans....

From p. 398 (2022):

PS: The day after I finished this chapter, an essay by Open Philanthropy’s Holden Karnofsky landed in my inbox: “AI Could Defeat All Of Us Combined.”

My first reaction was: “Good.”

...

Holden writes:

By “defeat,” I don't mean “subtly manipulate us” or “make us less informed” or something like that – I mean a literal “defeat” in the sense that we could all be killed, enslaved or forcibly contained.

Please note that we humans enslave, forcibly contain, and kill billions of fellow sentient beings every year. So if we solved the alignment problem and a “superior” AI actually were to share human values, it seems like they would kill, enslave, and forcibly contain us.

Holden, like almost every other EA and longtermist, simply assumes that humanity shouldn’t be “defeated.” Rarely does anyone note that it is possible, even likely, that on net, things would be much better if AIs did replace us.

The closest Holden comes is when he addresses objections:

Isn’t it fine or maybe good if AIs defeat us? They have rights too.

• Maybe AIs should have rights; if so, it would be nice if we could reach some “compromise” way of coexisting that respects those rights.
• But if they’re able to defeat us entirely, that isn’t what I’d plan on getting – instead I’d expect (by default) a world run entirely according to whatever goals AIs happen to have.
• These goals might have essentially nothing to do with anything humans value, and could be actively counter to it – e.g., placing zero value on beauty and having zero attempts to prevent or avoid suffering).

Zero attempts to prevent suffering? Hey Holden, aren’t you mistaking AIs for humans? Humans are the cause of most of the world’s unnecessary suffering, both to humans and other animals.

Setting aside our inherent tribal loyalties to humanity and our bias for continued existence, it is likely that AIs defeating humanity would be a huge improvement.

*Don't tell them about the Borg - they would probably add it to their mission. I'm hardly joking.

Can we take the win?

From Adam Mastroianni emphasis added; two charts below: 

In 1995, half of high school students drank, 35% smoked, and 40% had at least tried weed. 10% of them had brought a weapon to school at some point. About 6% of girls aged 15-19 were pregnant. [!!] Crime rates were about as high as they had been since we started keeping track of them.

Over the next 30 years, all of these problems shrank and some of them nearly disappeared. And not because of anything we did on purpose! [Not true!] We have no idea how to get kids to stop smoking—when we try to persuade them, we sometimes cause them to smoke more. No, these improvements happened basically by magic, for free, and—I think—as a byproduct of our increasing prosperity. [It is because we stopped poisoning kids with lead. Policy matters.] This is like waking up one day to find that you’ve been left a large fortune by a long-lost aunt.

How does this great, unearned victory make us feel? Apparently, it doesn’t make us feel anything. We would have spent billions to solve all of these problems back in the 1990s—no doubt we were spending considerable sums of money on anti-drug programs and public service announcements, wasting almost all of it—and yet when we got the thing we wanted so badly, we didn’t even notice.

Now we’re on to worrying about whether the kids are too sad, whether they play outside enough, etc. Which is all reasonable and fine, but also, can we take a win?

(After I scheduled this blog, Bryan Walsh's Good News newsletter from Vox picked up on this idea - more great data here.)

“Kids these days” are far less self-destructive because they were not poisoned with lead! The late Kevin Drum did incredible work on this. For some reason, this huge public-policy win does not get credit it deserves. I honestly don't know why.  

This is not to say that young people today have it easy. But I think a lot of unhappiness comes from believing that earlier generations had it easy.

The easy way to go is to say, “It’s all gone to shit” When the great moral of the story is that It’s always been shit.

–Conan O’Brien, quoted in Nick Offerman’s Gumption 

Below graphs from here; more at How To Make Things Better and Nostalgia Is Harmful:

Because Anne and I earned (and earn) so little, EK went to Pomona basically for free; they finished their PhD program with zero debt. Not that EK is the norm (far from it) but it isn't that everyone under 40 is shackled with crippling student loans.

Seriously: In 1992, the year Anne and I got married, fewer than half of people in the U.S. thought interracial marriage was OK!
Want a movie version of "The past wasn't what you think"? Check out Far from Heaven [very serious] and/or Pleasantville [more lighthearted]. 

Weekend Reading to Increase Happiness: Happy Decisions

Hi?

There’s a happiness gap between wanting the best and accepting good enough: some science-backed ways to close that gap.

Along the same lines: How to Stop Over-Optimizing and Focus on What Matters

Monsoon and miscellany

• This video compilation of monsoon storms around Tucson is excellent, if you have access to a big screen. Particularly, watch until the Marana and Vail shots to see the sheer violence of these downpours. (Twice, we've had to pull our car over because the rain was so hard.)
  [I had scheduled this post before a monsoon storm came through on July 12 and took down 6 trees in our small neighborhood. One pictured above; more.] 
  
  
• Roughly 60% of current occupations didn't exist in 1940.
  (My job for several decades - professional [sic] farm animal advocate - was definitely not a thing even 50 years ago.)
  
  
• The rational optimist’s guide to using AI
  (Do you want to learn how to use AI to enhance your life?)
  PS: just today, July 17, I used an LLM to file an appeal with our health insurance provider. 
  
  
• Half of America Cannot Afford to Live, and Other Wrong Numbers,
  
  
• Stop Calling It Exponential Growth
  After 25 years and trillions spent, wind and solar remain less than 3% of global energy. The common retort - "don't worry, it's exponential growth" - is false. See also The France Counterfactual if you want to dig deeper.

Links like the latter aren't to rant about or dunk on climate cultists. Rather, if we actually want to make things better, we must see things clearly. This is true for anything, including animal advocacy.

Bonus Reminder: You can't raise a child vegan

32 years on: You can't raise a child vegan

A One-Sentence Reminder

“It looks like the number of children who die every year is going to go up for the first time in 85 years, and that was caused by the American voter.”
-John Green

For everyone who has said and/or is going to say, “So-and-so hasn’t earned my vote” - those children’s deaths (and much more) are on you.