HOUSE_OVERSIGHT_016926.jpg

This geographical division of authority made logical sense when most of the
actors were humans who spent their lives within a single nation state, but now that the
actors of importance include geographically distributed hybrid intelligences such as
multinational corporations, that logic is less obvious. Today we live in a complex
transitional period, when distributed superintelligences still largely rely on the nation
states to settle the arguments arising among them. Often, those arguments are resolved
differently in different jurisdictions. It is becoming more difficult even to assign
individual humans to nation states: International travelers living and working outside
their native country, refugees, and immigrants (documented and not) are still dealt with
as awkward exceptions. Superintelligences built purely of information technology will
prove even more awkward for the territorial system of authority, since there is no reason
why they need to be tied to physical resources in a single country—or even to any
particular physical resources at all. An artificial intelligence might well exist “in the
cloud” rather than at any physical location.

I can imagine at least four scenarios for how machine superintelligences will
relate to hybrid superintelligences.

In one obvious scenario, multiple machine intelligences will ultimately be
controlled by, and allied with, individual nation states. In this state/AI scenario, one can
envision American and Chinese super-AIs wrestling each other for resources on behalf of
their state. In some sense, these AIs would be citizens of their nation state in the way that
many commercial corporations often act as “corporate citizens” today. In this scenario,
the host nation states would presumably give the machine superintelligences the
resources they needed to work for the state’s advantage. Or, to the degree that the
superintelligences can influence their state governments, they will presumably do so to
enhance their own power, for instance by garnering a larger share of the state’s resources.
Nation states’ AIs might not want competing AIs to grow up within their jurisdiction. In
this scenario, the superintelligences become an extension of the state, and vice versa.

The state/AI scenario seems plausible, but it is not our current course. Our most
powerful and rapidly improving artificial intelligences are controlled by for-profit
corporations. This is the corporate/AI scenario, in which the balance of power between
nation states and corporations becomes inverted. Today, the most powerful and
intelligent collections of machines are probably owned by Google, but companies like
Amazon, Baidu, Microsoft, Facebook, Apple, and IBM may not be far behind. These
companies all see a business imperative to build artificial intelligences of their own. It is
easy to imagine a future in which corporations independently build their own machine
intelligences, protected within firewalls preventing the machines from taking advantage
of one another’s knowledge. These machines will be designed to have goals aligned with
those of the corporation. If this alignment is effective, nation states may continue to lag
behind in developing their own artificial-intelligence capability and instead depend on
their “corporate citizens” to do it for them. To the extent that corporations successfully
control the goals, they will become more powerful and autonomous than nation states.

Another scenario, perhaps the one people fear the most, is that artificial
intelligences will not be aligned with either humans or hybrid superintelligences but will
act solely in their own interest. They might even merge into a single machine
superintelligence, since there may be no technical requirement for machine intelligences
to maintain distinct identities. The attitude of a self-interested super-AI toward hybrid

123

HOUSE_OVERSIGHT_016926