HOUSE_OVERSIGHT_018428.jpg

look for a chance to “improve” the way we live, to bend us like so many paperclips
into what it seeks? The leap from deciding liver allocations to shutting down liquor
plants might seem pretty short to a rationalizing machine. And if such a machine
could really “think”, Vinge bet it would pretty quickly conclude that the restraints of
its creators were limiting what it had been asked to do. At which point the AI would
turn to thinking about how to escape those bounds. It would be like Deep Blue
programmed to plan its own prison break. And as much as humans might try to
stifle a smart machine, we’d be fighting to contain something more powerful than
we’d ever encountered.

This challenge, which sounds like something out of science fiction, is known by
technologists by a name that does sound like a short story by Isaac Asimov: “The
Confinement Problem”. The computer scientist Butler Lampson named this in 1973
as a sort of task for computer security experts – possibly their last. The assignment:
Not simply to keep malware out of a system, but to keep the mind of a malicious
machine inside. To gate it. Today computer science labs are filled with nervous,
apocalyptic research imagining the impossible troubles of confinement. The debate
divides those who think smart technology can be contained – “Boxers,” they are
called – and those like Vinge who think the AI will always, eventually escape.
“Imagine yourself confined to your house with only limited data access to the
outside, to your master.” he wrote, putting the reader in the place of an AI machine.
“If those masters thought at a rate -- say – one million times slower than you, there
is little doubt that over a period of years (your time) you could come up with
‘helpful advice’ that would incidentally set you free.”

Imagine you are in charge of containing that health-optimizing AI. What if it told you
it had the power to cure all illness and hunger, to ameliorate the misery of the world,
if only it could be permitted to really control access to all the world’s trading and
transport market? Let me out! Would you refuse?268 Would that be ethical?
Eventually, perhaps, the AI would study the physics of its own electrics, discover
laws no human knows, and then slip free from its box on a trail of bits we’d never
imagined, using physical laws we’ll never discover. Impossible? “It seems to me that
historically ‘impossible’ has essentially always meant ‘I can’t figure out how to do it
right now,” the computer scientist Michael Vassar has written about such a situation.
“People proposing AI boxes are a bit like literature majors proposing to
lock McGuyver in a ‘room full of discarded electronics components.’”269 The
computers, built to solve problems, will do exactly that. This is perhaps why some of
the bleakest warnings about AI come from the very New Caste figures now
accelerating their adoption. AI is our “biggest existential threat” they warn, even as
they integrate it more fully into their own products.

268 Let me out: See, for instance, Stuart Armstrong, Anders Sandberg and Nick
Bostrom “Thinking Inside the Box: Controlling and Using an Oracle AI”, Minds &
Machines (2012) 22:299–324
269 People proposing: Michael Vassar (2005) “Re: AI boxing (dogs and helicopters)”
posted to SL4 mailing list

196

HOUSE_OVERSIGHT_018428