Superintelligence and moral blindness
According to his wife, John von Neumann, talking about his work on the atom bomb, said:
What we are creating now is a monster whose influence is going to change history, provided there is any history left, yet it would be impossible not to see it through, not only for the military reasons, but it would also be unethical from the point of view of the scientists not to do what they know is feasible, no matter what terrible consequences it may have (quoted in Dyson, Turing’s Cathedral, p62)
Richard Feynman reports a conversation with von Neumann from around the same time:
And Von [sic] Neumann gave me an interesting idea; that you don’t have to be responsible for the world you’re in. So I have developed a very powerful sense of irresponsibility as a result of Von Neumann's advice. It’s made me a very happy man ever since. (same book, same page)
I don’t know what you think about this, but my initial reaction to such things is something approaching horror. Both Feynman and von Neumann’s words sound to me like the words of moral monsters: social irresponsibility made Feynman very happy! It would be unethical for a scientist not to do science that is feasible, even if it leads to terrible outcomes, von Neumann thinks! That’s awful!!!
The reason this is concerning is that these guys aren’t nobodies. In addition to being very powerful and influential, they are also very smart. For the purposes of this post, I will concentrate on von Neumann.
Stories of his intelligence are legion. He had an eidetic memory, and could quote whole books from memory. On his deathbed, when his mental faculties were deteriorating, he had someone read Thucydides in Greek to him and he had the wherewithal to correct mistakes or mispronunciation. A scholar of Byzantine history said that ‘Johnny’ knew more about the topic than he did. And so on: anecdotes like this abound, and are easily googleable. He either invented or contributed to a ridiculous number of fields of maths. Looks at his wiki toc:
Pertinently for this post, his mental abilities were such that he has been called an alien or superintelligence, along with others who grew up in Hungary on both sides of 1900. A theoretical physicist proposed:
these people were really visitors from Mars; for them, he said, it was difficult to speak without an accent that would give them away and therefore they chose to present to be Hungarians whose inability to speak any language with accent is well known; except Hungarian, and [these] brilliant men all lived elsewhere” (quoted p106, Rhodes, The Making of the Atomic Bomb)
With that said, here is the thing I am curious about. How are we to reconcile these two facts:
- von Neumann is one of the smartest people ever to live
- von Neumann, per the above quotes, and from my perspective, sounds like he’s seriously wrong about morality.
In particular, I want to explore the possibility of reconciling these two facts with moral anti-realism: with the idea that there are no objective moral truths. This isn’t a view I have particularly strong feelings about either way (antecedently of this post), although I lean towards realism.
I think this question — of how to reconcile smartness and potential moral blindness — is pretty relevant at the moment. A vocal (at least on the internet) minority think that ‘AI alignment’ is a very important problem: that with the coming of superintelligent artificial beings, we should be very concerned that their values don’t mesh with ours. That could lead them, for example, to attempt to minimize the suffering on earth by murdering everyone painlessly, or to any number of other perverse means for realizing human goals (or, indeed, for realizing AI goals).
In a previous post, I argued that some of the problems of superintelligence arise already today (the basic thought is that superintelligence is a relative concept, that we’re super-intelligent relative to even close ancestors, but that our gains in intelligence can be rolled back, for example through cyberwar, given how much our intelligence is intermeshed with the internet, becoming subintelligences relative to our superintelligent enemies.) The aim here is to follow that sort of line by assuming that von Neumann is superintelligence and considering the possibility that the above gives evidence he’s not morally aligned with us.
An argument for anti-realism from superintelligence
We can make a very simple argument for anti-realism that goes as so:
(1) If moral realism is true, then if something is true, it is so because there’s a moral fact which makes it so
(2) As someone noted (Rhodes p108) “(a) Johnny can prove anything (b) anything Johnny proves is correct”
(3) You don’t have to be responsible for the world you’re in (from the fact that von Neumann said it and 2b)
(4) You do have to be responsible for the world you’re in (my intuition)
(5) There’s a fact corresponding to 3 and a fact corresponding to 4
Which is a contradiction (and we’ll assume contradictions are bad). This is heavy going for a very simple point. von Neumann is way smarter than any of us, but seems to be wrong about a basic moral truth. That should undermine our confidence that there are moral facts out there amenable to discovery by smart people, and that seems to challenge a prima facie attractive view of moral truth and understanding. In what follows, I’ll consider various possible responses to the above.
Response 1. I’m wrong.
This is the most obvious one, at least to me. Just as I trust von Neumann more than I do myself about Byzantine history, or attic Greek, or game theory, or quantum mechanics, or the whys and wherefores of the design of the very machine you’re reading this on, I should trust him about moral matters. He is right that you don’t have to be responsible for the world you’re in. You find yourself somewhere and somewhen, and you do what you’re meant to. We could call this the Friedmanian theory of morality: just as for Milton Friedman, the aim of a firm should simply and only be to maximise profits for its shareholders regardless of social impact, so the moral aim of a scientist qua scientist should be simply to advance science, wherever that takes you. The scholarly qualification ‘qua scientist’ is doing work here: this wouldn’t be the view that no one has any social or moral responsibility (Friedman doesn’t think that, he just thought that firms weren’t the locus of that responsibility), but that when one is doing science, one should advance science, and not let morals intrude.
Whether that is coherent morally is, of course, a different question. But it should at least have some plausibility, and be recognizable as something to which we respond. My day job is managing editor of an academic journal; I don’t think academic journals (at least in philosophy) are forces for good, but I try to make my journal as good as it can be (while spending spare time trying to come up with improvements). von Neumann’s view might just be a drastically more momentous version of my dilemma.
The cost of this view is that deeply held moral intuitions are to be rejected. This is a sad face solution, and probably will be hard to take for many. It would also serve as premise for the alignment worrier — if great intelligence is consistent with moral blindness, then super AGI could potentially be greatly worrisome.
Response 2. I’m wrong in a different way.
One gigantic problem here is that I’ve basically picked a couple of quotations that I found arresting while reading the cited book. It doesn’t give the whole story. It doesn’t tell you that the date was during the war; doesn’t tell you that he agitated to get Goedel out of Nazi Europe; and that he hated the Nazis with a loathing that was ‘essentially boundless’ (53 of Dyson, as reported to his wife), that he disliked Europe because of his ‘memory of the total disillusionment in human decency between 1933 and September 1938’ (same book, same location).
That makes it sound like he’s much more intelligible as a moral personality, and so maybe we don’t know enough. This is both worrying and heartening. It’s heartening because it shows what we would view as appropriate moral feeling and perhaps motivation. But it’s also worrying because it means, if we do indeed take the very top quotations as truly representative, that someone in some ways like us morally (disliking Nazis and Nazism) but much smarter also thought it was true that you don’t have to be responsible to the world.
One possibility, I think, is just that there’s too little evidence to go on. Maybe we need to know more (which we can do — there’s a bio of him coming out in a few months). But in advance of that, we might also consider in more detail the time. von Neumann was acutely conscious of the possibility of nuclear war, and was deeply worried that a belligerent country would get an unsurpassable lead in atomic weapons before the US could catch up. A superintelligence chugging through the utilitarian calculus in the face of such a possibility (especially one who invented game theory) might conclude that the moral thing to do is leave one’s moral scruples at the door. Maybe in these sorts of stakes, where the whole world is at risk, our moral intuitions become worthless. Maybe von Neumann was right that a scientist, in the 1940s, in the face of nuclear war, should leave their moral scruples at the door before entering the lab. It would then be perfectly consistent that a random blogger, in 2021, with no nuclear war incredibly close up, would react differently. That seems possible.
Let me briefly expand on the game theory point. von Neumann, along with Oskar Morgenstern, wrote a (/the?) fundamental work in game theory, a discipline concerned with mathematically modelling how people interacting with one another should rationally behave. In some cases, game-theoretic reasoning can lead one to conclusions that seem odd. Particularly salient — and espoused by von Neumann — is the notion of mutually assured destruction. On that idea, the best way to stave off a full out nuclear war that would destroy the whole planet is for all belligerents to have enough nuclear weapons to destroy the rest of the world should they be attacked. So, on this view, countries should stockpile nuclear weapons, to make it clear they could use them, to ensure that they won’t in fact have to use them. Perhaps one could argue that the seeming moral blindness exhibited up top is in fact a result of von Neumann having gamed through the possibilities, something that seems all the more plausible, again, when one considers the circumstances.
This is a happy face solution to the problem. von Neumann isn’t a moral monster. Indeed, maybe’s he someone we can learn from, if we’re ever in as dire situations as he was. One unfortunate consequence of this, however, is that if the above is true, what holds in one situation can’t easily be translated to another: what held for Feynman and Johnny doesn’t hold for us. That explains my bafflement and sense of his wrongness, but it also means we can’t readily learn from people in different moral contexts. And that has the unfortunate side-effect — absent further argument — of compelling us to ditch basically all the great moral teachings handed down to us, as we should expect them to be no less context-bound than von Neumann’s teachings.
Response 3. von Neumann is wrong
By contrast, we might think that von Neumann is wrong, and indeed that he’s wrong in a deep and interesting way, one think reveals something about the nature of moral truth and understanding. There’s a bit of philosophical literature on the notion of moral expertise (I haven’t read it, I’m afraid: this, and references therein, looks like a decent intro). One of the things that literature is concerned with is the way that moral truth and regular truth might differ. A driving intuition in that literature (I think) is that there is something markedly different between these types of truth, something that manifests, for example, in the way we respond to testimony. It’s completely reasonable for you to accept on testimony ‘moral understanding is quite a trendy topic in philosophy journals these days’; there’s (meant to be) something weird about you similarly accepting ‘you shouldn’t eat meat’ merely because I, some random blogger, say so.
One can go on to make a bunch of distinctions, different ways of understanding the nature of moral realism and moral understanding, and ways of making it come out true or false. But without engaging it that, I want to suggest that von Neumann, or an idealized version of him, presents an interesting case, if this were so: someone capable of knowing basically everything, who didn’t know moral truths.
It would make him something that a moral Mary, by which I refer to a famous thought-experiment in philosophy of mind. Mary has been brought up in a monotone black and white room, containing only every neuroscience textbook about colour. She masters them all. She knows everything about colour (in a sense — she’s read all the books). Nevertheless, there’s something she misses: the thing she experiences when she leaves the room and sees the rose for the first time,it seems she learns something, namely what it’s like to see red. On this view, von Neumann would be like Mary while still in the room, oblivious to a part of reality.
I don’t think this is really true: that anti-Nazi stuff would tend to suggest he has a moral sensitivity. But it’s a fun possibility to think about.
Which should we go for?
Before starting writing this, I was puzzled, and thought it was a decent argument against moral realism. Now, I’m not so sure. In particular, I think looking in more detail at the harsh context von Neumann found himself in (as well as my ignorance of his life in general), we can perhaps begin to make sense of his morally blind sounding words. This is comforting: it severs the potential inference from very smart to morally worrisome, an inference alignment people think is a very important one to consider. And it makes of von Neumann not a moral monster.
But its comfortingness has limits. As pointed out, if moral truth is context-bound in this way, then we need to be careful about how we talk to and interpret people when it comes to morality, potentially to the extent of discarding our Tripitakas and Analects.
Anyway, this is my, very tentative, verdict: von Neumann, rightly despising the Nazis, like Odysseus with the Sirens, thought it moral to discard morality, and perhaps rightly. But others — perhaps indeed Feynman — perhaps ought not to have been so receptive to his teachings. And the point generalizes. If a computer scientist today were getting into big data stuff while trying not to face up to the potential ethical issues surrounding it (arising from the GIGO principle that applies to big data sets and the fact that many people, if not garbage, at least aren’t great), then they would be wrong to do so, and the irrelevance of moral considerations to science, if it applies at all, might apply in a very context-bound way.
