HOUSE_OVERSIGHT_013126.jpg

210 12 The Engineering and Development of Ethics

We realize this point may be somewhat contentious – a counter-argument would be that
the human brain is known to support at least moderately ethical behavior, according to human
ethical standards, whereas less brain-like AGI systems are much less well understood. However,
the obvious counter-counterpoints are that:

• Humans are not all that consistently ethical, so that creating AGI systems potentially much
more practically powerful than humans, but with closely humanlike ethical, motivational
and goal systems, could in fact be quite dangerous
• The effect on a human-like ethical/motivational/goal system of increasing the intelligence,
or changing the physical embodiment or cognitive capabilities, of the agent containing the
system, is unknown and difficult to predict given all the complexities involved

The course we tentatively recommend, and are following in our own work, is to develop AGI
systems with explicit, hierarchically-dominated goal systems. That is:

• create one or more "top goals" (we call them Ubergoals in CogPrime )
• have the system derive subgoals from these, using its own intelligence, potentially guided
by educational interaction or explicit programming
• have a significant percentage of the system’s activity governed by the explicit pursuit of
these goals

Note that the "significant percentage" need not be 100%; CogPrime, for example, combines
explicitly goal-directed activity with other "spontaneous" activity. Requiring that all activity
be explicitly goal-directed may be too strict a requirement to place on AGI architectures.
The next step, of course, is for the top-level goals to be chosen in accordance with the
principle of human-Friendliness. The next one of our eight points, about the Global Brain,
addresses one way of doing this. In our near-term work with CogPrime, we are using simplistic
approaches, with a view toward early-stage system testing.

12.4 Ethical Synergy

An explicit goal system provides an explicit way to ensure that ethical principles (as represented
in system goals) play a significant role in guiding an AGI system’s behavior. However, in an
integrative design like CogPrime the goal system is only a small part of the overall story,
and it’s important to also understand how ethics relates to the other aspects of the cognitive
architecture.
One of the more novel ideas presented in this chapter is that different types of ethical intuition
may be associated with different types of memory – and to possess mature ethics, a mind
must display ethical synergy between the ethical processes associated with its memory types.
Specifically, we suggest that:

• Episodic memory corresponds to the process of ethically assessing a situation based on
similar prior situations
• Sensorimotor memory corresponds to "mirror neuron" type ethics, where you feel another
person’s feelings via mirroring their physiological emotional responses and actions
• Declarative memory corresponds to rational ethical judgment

HOUSE_OVERSIGHT_013126