HOUSE_OVERSIGHT_013053.jpg

7.3 Toward a Formal Characterization of Real-World General Intelligence
137

7.3.2 Connecting Legg and Hutter’s Model of Intelligent Agents to
the Real World

A notable aspect of the Legg and Hutter formalism is the separation of the reward mechanism
from the cognitive mechanisms of the agent. While commonplace in the reinforcement learning
literature, this seems psychologically unrealistic in the context of biological intelligences and
many types of machine intelligences. Not all human intelligent activity is specifically reward-
seeking in nature; and even when it is, humans often pursue complexly constructed rewards,
that are defined in terms of their own cognitions rather than separately given. Suppose a certain
human’s goals are true love, or world peace, and the proving of interesting theorems – then these
goals are defined by the human herself, and only she knows if she’s achieved them. An externally-
provided reward signal doesn’t capture the nature of this kind of goal-seeking behavior, which
characterizes much human goal-seeking activity (and will presumably characterize much of the
goal-seeking activity of advanced engineered intelligences also) ... let alone human behavior that
is spontaneous and unrelated to explicit goals, yet may still appear commonsensically intelligent.

One could seek to bypass this complaint about the reward mechanisms via a sort of “neo-
Freudian” argument, via

• associating the reward signal, not with the “external environment” as typically conceived,
but rather with a portion of the intelligent agent’s brain that is separate from the cognitive
component
• viewing complex goals like true love, world peace and proving interesting theorems as in-
direct ways of achieving the agent’s “basic goals”, created within the agent’s memory via
subgoaling mechanisms

but it seems to us that a general formalization of intelligence should not rely on such strong
assumptions about agents’ cognitive architectures. So below, after introducing the pragmatic
and efficient pragmatic general intelligence measures, we will propose an alternate interpreta-
tion wherein the mechanism of external rewards is viewed as a theoretical test framework for
assessing agent intelligence, rather than a hypothesis about intelligent agent architecture.

In this alternate interpretation, formal measures like the universal, pragmatic and efficient
pragmatic general intelligence are viewed as not directly applicable to real-world intelligences,
because they involve the behaviors of agents over a wide variety of goals and environments,
whereas in real life the opportunities to observe agents are more limited. However, they are
viewed as being indirectly applicable to real-world agents, in the sense that an external intelli-
gence can observe an agent’s real-world behavior and then infer its likely intelligence according
to these measures.

In a sense, this interpretation makes our formalized measures of intelligence the opposite of
real-world IQ tests. An IQ test is a quantified, formalized test which is designed to approxi-
mately predict the informal, qualitative achievement of humans in real life. On the other hand,
the formal definitions of intelligence we present here are quantified, formalized tests that are
designed to capture abstract notions of intelligence, but which can be approximately evaluated
on a real-world intelligent system by observing what it does in real life.

HOUSE_OVERSIGHT_013053