HOUSE_OVERSIGHT_016878.jpg

concept of “Pareto-topia”: the idea that AI, if done right, can bring about a future in
which everyone ’s lives are hugely improved, a future where there are no losers. A key
realization here is that what chiefly prevents humanity from achieving its full potential
might be our instinctive sense that we’re in a zero-sum game—a game in which players
are supposed to eke out small wins at the expense of others. Such an instinct is seriously
misguided and destructive in a “game” where everything is at stake and the payoff is
literally astronomical. There are many more star systems in our galaxy alone than there
are people on Earth.

Hope
As of this writing, I’m cautiously optimistic that the AI-risk message can save humanity
from extinction, just as the Soviet-occupation message ended up liberating hundreds of
millions of people. As of 2015, it had reached and converted 40 percent of AI
researchers. It wouldn’t surprise me if a new survey now would show that the majority
of AI researchers believe AI safety to be an important issue.
I’m delighted to see the first technical AI-safety papers coming out of DeepMind,
OpenAI, and Google Brain and the collaborative problem-solving spirit flourishing
between the AI-safety research teams in these otherwise very competitive organizations.
The world’s political and business elite are also slowly waking up: AI safety has
been covered in reports and presentations by the Institute of Electrical and Electronics
Engineers (IEEE), the World Economic Forum, and the Organization for Economic
Cooperation and Development (OECD). Even the recent (July 2017) Chinese AI
manifesto contained dedicated sections on “AI safety supervision” and “Develop[ing]
laws, regulations, and ethical norms” and establishing “an AI security and evaluation
system” to, among other things, “[e]nhance the awareness of risk.” I very much hope that
a new generation of leaders who understand the AI Control Problem and AI as the
ultimate environmental risk can rise above the usual tribal, zero-sum games and steer
humanity past these dangerous waters we are in—thereby opening our way to the stars
that have been waiting for us for billions of years.
Here’s to our next hundred thousand years! And don’t hesitate to speak the truth,
even if your voice trembles.

75

HOUSE_OVERSIGHT_016878