HOUSE_OVERSIGHT_013124.jpg

208
12 The Engineering and Development of Ethics

rise to questions about the ethical value of various practical modalities of AGI development,
for instance:

• Should AGI be developed in a top-secret installation by a select group of individuals selected
for a combination of technical and scientific brilliance and moral uprightness, or other
qualities deemed relevant (a "closed approach")? Or should it be developed out in the
open, in the manner of open-source software projects like Linux? (an "open approach").
The open approach allows the collective intelligence of the world to more fully participate
– but also potentially allows the more unsavory elements of the human race to take some
of the publicly-developed AGI concepts and tools private, and develop them into AGIs
with selfish or evil purposes in mind. Is there some meaningful intermediary between these
extremes?
• Should governments regulate AGI, with Friendliness in mind (as advocated carefully by e.g
Bill Hibbard [Hib02])? Or will this just cause AGI development to move to the handful of
countries with more liberal policies? ... or cause it to move underground, where nobody can
see the dangers developing? As a rough analogue, it's worth noting that the US government's
imposition of restrictions on stem cell research, under President George W. Bush, appears
to have directly stimulated the provision of additional funding for stem cell research in other
nations like Korea, Singapore and China.

The former issue is, obviously, highly relevant to CogPrime (which is currently being devel-
oped via the open source CogPrime project); and so the various dimensions of this issues are
worth briefly sketching here.
We have a strong skepticism of self-appointed elite groups that claim (even if they genuinely
believe) that they know what's best for everyone, and a healthy respect for the power of collective
intelligence and the Global Brain, which the open approach is ideal for tapping. On the other
hand, we also understand the risk of terrorist groups or other malevolent agents forking an open
source AGI project and creating something terribly dangerous and destructive. Balancing these
factors against each other rigorously, seems beyond the scope of current human science.
Nobody really understands the social dynamics by which open technological knowledge plays
out in our current world, let alone hypothetical future scenarios. Right now there exists open
knowledge about many very dangerous technologies, and there exist many terrorist groups, yet
these groups fortunately make scant use of these technologies. The reasons why appear to be
essentially sociological – the people involved in these terrorist groups tend not to be the ones
who have mastered the skills of turning public knowledge on cutting-edge technologies into real
engineered systems. But while it's easy to observe this sociological phenomenon, we certainly
have no way to estimate its quantitative extent from first principles. We don't really have a
strong understanding of how safe we are right now, given the technology knowledge available
right now via the Internet, textbooks, and so forth. Even relatively straightforward issues such
as nuclear proliferation remain confusing, even to the experts.
It's also quite clear that keeping powerful AGI locked up by an elite group doesn't really
provide reliable protection against malevolent human agents. History is rife with such situations
going awry, e.g. by the leadership of the group being subverted, or via brute force inflicted by
some outside party, or via a member of the elite group defecting to some outside group in the
interest of personal power or reward or due to group-internal disagreements, etc. There are
many things that can go wrong in such situations, and the confidence of any particular group
that they are immune to such issues, cannot be taken very seriously. Clearly, neither the open
nor closed approach qualifies as a panacea.

HOUSE_OVERSIGHT_013124