HOUSE_OVERSIGHT_016889.jpg

Those are also the processes by which all new knowledge is created: They are how we
innovate, make progress, and create abstract understanding for its own sake. This is
human-level intelligence: thinking. It is also, or should be, the property we seek in
artificial general intelligence (AGI). Here I’ll reserve the term “thinking” for processes
that can create understanding (explanatory knowledge). Popper’s argument implies that
all thinking entities—human or not, biological or artificial—must create such knowledge
in fundamentally the same way. Hence understanding any of those entities requires
traditionally human concepts such as culture, creativity, disobedience, and morality—
which justifies using the uniform term people to refer to all of them.

Misconceptions about human thinking and human origins are causing
corresponding misconceptions about AGI and how it might be created. For example, it is
generally assumed that the evolutionary pressure that produced modern humans was
provided by the benefits of having an ever greater ability to innovate. But if that were so,
there would have been rapid progress as soon as thinkers existed, just as we hope will
happen when we create artificial ones. If thinking had been commonly used for anything
other than imitating, it would also have been used for innovation, even if only by
accident, and innovation would have created opportunities for further innovation, and so
on exponentially. But instead, there were hundreds of thousands of years of near stasis.
Progress happened only on timescales much longer than people’s lifetimes, so in a typical
generation no one benefited from any progress. Therefore, the benefits of the ability to
innovate can have exerted little or no evolutionary pressure during the biological
evolution of the human brain. That evolution was driven by the benefits of preserving
cultural knowledge.

Benefits to the genes, that is. Culture, in that era, was a very mixed blessing to
individual people. Their cultural knowledge was indeed good enough to enable them to
outclass all other large organisms (they rapidly became the top predator, etc.), even
though it was still extremely crude and full of dangerous errors. But culture consists of
transmissible information—memes—and meme evolution, like gene evolution, tends to
favor high-fidelity transmission. And high-fidelity meme transmission necessarily entails
the suppression of attempted progress. So it would be a mistake to imagine an idyllic
society of hunter-gatherers, learning at the feet of their elders to recite the tribal lore by
heart, being content despite their lives of suffering and grueling labor and despite
expecting to die young and in agony of some nightmarish disease or parasite. Because,
even if they could conceive of nothing better than such a life, those torments were the
least of their troubles. For suppressing innovation in human minds (without killing them)
is a trick that can be achieved only by human action, and it is an ugly business.

This has to be seen in perspective. In the civilization of the West today, we are
shocked by the depravity of, for instance, parents who torture and murder their children
for not faithfully enacting cultural norms. And even more by societies and subcultures
where that is commonplace and considered honorable. And by dictatorships and
totalitarian states that persecute and murder entire harmless populations for behaving
differently. We are ashamed of our own recent past, in which it was honorable to beat
children bloody for mere disobedience. And before that, to own human beings as slaves.
And before that, to burn people to death for being infidels, to the applause and
amusement of the public. Steven Pinker’s book The Better Angels of our Nature contains
accounts of horrendous evils that were normal in historical civilizations. Yet even they

86

HOUSE_OVERSIGHT_016889