HOUSE_OVERSIGHT_016832.jpg

THE PURPOSE PUT INTO THE MACHINE
Stuart Russell

Stuart Russell is a professor of computer science and Smith-Zadeh Professor in
Engineering at UC Berkeley. He is the coauthor (with Peter Norvig) of Artificial
Intelligence: A Modern Approach.

Among the many issues raised in Norbert Wiener’s The Human Use of Human Beings
(1950) that are currently relevant, the most significant to the AI researcher is the
possibility that humanity may cede control over its destiny to machines.
Wiener considered the machines of the near future as far too limited to exert global
control, imagining instead that machines and machine-like control systems would be
wielded by human elites to reduce the great mass of humanity to the status of “cogs and
levers and rods.” Looking further ahead, he pointed to the difficulty of correctly
specifying objectives for highly capable machines, noting

a few of the simpler and more obvious truths of life, such as that when a djinnee is
found in a bottle, it had better be left there; that the fisherman who craves a boon
from heaven too many times on behalf of his wife will end up exactly where he
started; that if you are given three wishes, you must be very careful what you wish
for.

The dangers are clear enough:

Woe to us if we let [the machine] decide our conduct, unless we have previously
examined the laws of its action, and know fully that its conduct will be carried out on
principles acceptable to us! On the other hand, the machine like the djinnee, which
can learn and can make decisions on the basis of its learning, will in no way be
obliged to make such decisions as we should have made, or will be acceptable to us.

Ten years later, after seeing Arthur Samuel’s checker-playing program learn to play
checkers far better than its creator, Wiener published “Some Moral and Technical
Consequences of Automation” in Science. In this paper, the message is even clearer:

If we use, to achieve our purposes, a mechanical agency with whose operation we
cannot efficiently interfere . . . we had better be quite sure that the purpose put into
the machine is the purpose which we really desire. . . .

In my view, this is the source of the existential risk from superintelligent AI cited in
recent years by such observers as Elon Musk, Bill Gates, Stephen Hawking, and Nick
Bostrom.

Putting Purposes Into Machines
The goal of AI research has been to understand the principles underlying intelligent
behavior and to build those principles into machines that can then exhibit such behavior.
In the 1960s and 1970s, the prevailing theoretical notion of intelligence was the capacity
for logical reasoning, including the ability to derive plans of action guaranteed to achieve
a specified goal. More recently, a consensus has emerged around the idea of a rational

29

HOUSE_OVERSIGHT_016832