HOUSE_OVERSIGHT_016874.jpg

mainstream groups, who had more to lose, initially qualified and diluted the message,
taking positions like, “It would make sense in the long term to delegate control over local
matters.” (There were always exceptions: Some public intellectuals proclaimed the
original dissident message verbatim.) Finally, the original message—being, simply,
true—won out over its diluted versions. Estonia regained its independence in 1991, and
the last Soviet troops left three years later.
The people who took the risk and spoke the truth in Estonia and elsewhere in the
Eastern Bloc played a monumental role in the eventual outcome—an outcome that
changed the lives of hundreds of millions of people, myself included. They spoke the
truth, even as their voices trembled.

The Second Message: AI Risk
My exposure to the second revolutionary message was via Yudkowsky’s blog—the blog
that compelled me to reach out and arrange that meeting in California. The message was:
Continued progress in AI can precipitate a change of cosmic proportions—a runaway
process that will likely kill everyone. We need to put in a lot of extra effort to avoid that
outcome.
After my meeting with Yudkowsky, the first thing I did was try to interest my
Skype colleagues and close collaborators in his warning. I failed. The message was too
crazy, too dissident. Its time had not yet come.
Only later did I learn that Yudkowsky wasn’t the original dissident speaking this
particular truth. In April 2000, there was a lengthy opinion piece in Wired titled, “Why
the Future Doesn’t Need Us,” by Bill Joy, co-founder and chief scientist of Sun
Microsystems. He warned:

Accustomed to living with almost routine scientific breakthroughs, we have yet
to come to terms with the fact that the most compelling 21st-century
technologies—robotics, genetic engineering, and nanotechnology—pose a
different threat than the technologies that have come before. Specifically, robots,
engineered organisms, and nanobots share a dangerous amplifying factor: They
can self-replicate. . . . [O]ne bot can become many, and quickly get out of
control.

Apparently, Joy’s broadside caused a lot of furor but little action.
More surprising to me, though, was that the AI-risk message arose almost
simultaneously with the field of computer science. In a 1951 lecture, Alan Turing
announced: “[I]t seems probable that once the machine thinking method had started, it
would not take long to outstrip our feeble powers. . . . At some stage, therefore, we
should have to expect the machines to take control. . . .”21 A decade or so later, his
Bletchley Park colleague I. J. Good wrote, “The first ultraintelligent machine is the last
invention that man need ever make, provided that the machine is docile enough to tell us
how to keep it under control.”22 Indeed, I counted half a dozen places in The Human Use
of Human Beings where Wiener hinted at one or another aspect of the Control Problem.
(“The machine like the djinnee, which can learn and can make decisions on the basis of

21 Posthumously reprinted in Phil. Math. (3) vol. 4, 256-60 (1966).
22 Irving John Good, “Speculations concerning the first ultraintelligent machine,” Advances in Computers,
vol. 6 (Academic Press, 1965), pp. 31-88.

71

HOUSE_OVERSIGHT_016874