foundations of computational agents
An agent chooses actions based on their outcomes. Outcomes are whatever the agent has preferences over. If an agent does not prefer any outcome to any other outcome, it does not matter what the agent does. Initially, we consider outcomes without considering the associated actions. Assume there are only a finite number of outcomes.
We define a preference relation over outcomes. Suppose ${o}_{1}$ and ${o}_{2}$ are outcomes. We say that ${o}_{1}$ is weakly preferred to outcome ${o}_{2}$, written ${o}_{1}\u2ab0{o}_{2}$, if outcome ${o}_{1}$ is at least as desirable as outcome ${o}_{2}$.
Define ${o}_{1}\sim {o}_{2}$ to mean ${o}_{1}\u2ab0{o}_{2}$ and ${o}_{2}\u2ab0{o}_{1}$. That is, ${o}_{1}\sim {o}_{2}$ means outcomes ${o}_{1}$ and ${o}_{2}$ are equally preferred. In this case, we say that the agent is indifferent between ${o}_{1}$ and ${o}_{2}$.
Define ${o}_{1}\succ {o}_{2}$ to mean ${o}_{1}\u2ab0{o}_{2}$ and ${o}_{2}\u22e1{o}_{1}$. That is, the agent weakly prefers outcome ${o}_{1}$ to outcome ${o}_{2}$, but does not weakly prefer ${o}_{2}$ to ${o}_{1}$, and is not indifferent between them. In this case, we say that ${o}_{1}$ is strictly preferred to outcome ${o}_{2}$.
Typically, an agent does not know the outcome of its actions. A lottery is defined to be a finite distribution over outcomes, written as
$$[{p}_{1}:{o}_{1},{p}_{2}:{o}_{2},\mathrm{\dots},{p}_{k}:{o}_{k}],$$ |
where each ${o}_{i}$ is an outcome and ${p}_{i}$ is a non-negative real number such that
$$\sum _{i}{p}_{i}=1.$$ |
The lottery specifies that outcome ${o}_{i}$ occurs with probability ${p}_{i}$. In all that follows, assume that outcomes may include lotteries. This includes lotteries where the outcomes are also lotteries, and so on recursively (called lotteries over lotteries).
(Completeness) An agent has preferences between all pairs of outcomes:
$${o}_{1}\u2ab0{o}_{2}\mathit{\text{or}}{o}_{2}\u2ab0{o}_{1}.$$ |
The rationale for this axiom is that an agent must act; if the actions available to it have outcomes ${o}_{1}$ and ${o}_{2}$ then, by acting, it is explicitly or implicitly preferring one outcome over the other.
(Transitivity) Preferences must be transitive:
$$\mathit{\text{if}}{o}_{1}\u2ab0{o}_{2}\mathit{\text{and}}{o}_{2}\u2ab0{o}_{3}\mathit{\text{then}}{o}_{1}\u2ab0{o}_{3}.$$ |
To see why this is reasonable, suppose it is false, in which case ${o}_{1}\u2ab0{o}_{2}$ and ${o}_{2}\u2ab0{o}_{3}$ and ${o}_{3}\succ {o}_{1}$. Because ${o}_{3}$ is strictly preferred to ${o}_{1}$, the agent should be prepared to pay some amount to get from ${o}_{1}$ to ${o}_{3}$. Suppose the agent has outcome ${o}_{3}$; then ${o}_{2}$ is at least as good so the agent would just as soon have ${o}_{2}$. ${o}_{1}$ is at least as good as ${o}_{2}$ so the agent would just as soon have ${o}_{1}$ as ${o}_{2}$. Once the agent has ${o}_{1}$ it is again prepared to pay to get to ${o}_{3}$. It has gone through a cycle of preferences and paid money to end up where it is. This cycle that involves paying money to go through it is known as a money pump because, by going through the loop enough times, the amount of money that agent must pay can exceed any finite amount. It seems reasonable to claim that being prepared to pay money to cycle through a set of outcomes is irrational; hence, a rational agent should have transitive preferences.
It follows from the transitivity and completeness axioms that transitivity holds for mixes of $\succ $ and $\u2ab0$, so that if one or both of the preferences in the premise of the transitivity axiom is strict, then the conclusion is strict. That is, $\text{if}{o}_{1}\succ {o}_{2}\text{and}{o}_{2}\u2ab0{o}_{3}\text{then}{o}_{1}\succ {o}_{3}$. Also, $\text{if}{o}_{1}\u2ab0{o}_{2}\text{and}{o}_{2}\succ {o}_{3}\text{then}{o}_{1}\succ {o}_{3}$. See Exercise 1.
(Monotonicity) An agent prefers a larger chance of getting a better outcome than a smaller chance of getting the better outcome. That is, if ${o}_{\mathrm{1}}\mathrm{\succ}{o}_{\mathrm{2}}$ and $p\mathrm{>}q$ then
$$[p:{o}_{1},(1-p):{o}_{2}]\succ [q:{o}_{1},(1-q):{o}_{2}].$$ |
Note that, in this axiom, $\succ $ between outcomes represents the agent’s preference, whereas $>$ between $p$ and $q$ represents the familiar comparison between numbers.
The following axiom specifies that lotteries over lotteries only depend the outcomes and probabilities:
(Decomposability) (“no fun in gambling”) An agent is indifferent between lotteries that have the same probabilities over the same outcomes, even if one or both is a lottery over lotteries. For example:
$[p:{o}_{1},$ | $(1-p):[q:{o}_{2},(1-q):{o}_{3}]]$ | ||
$\sim [p:{o}_{1},(1-p)*q:{o}_{2},(1-p)*(1-q):{o}_{3}].$ |
Also ${o}_{\mathrm{1}}\mathrm{\sim}\mathrm{[}\mathrm{1}\mathrm{:}{o}_{\mathrm{1}}\mathrm{,}\mathrm{0}\mathrm{:}{o}_{\mathrm{2}}\mathrm{]}$ for any outcomes ${o}_{\mathrm{1}}$ and ${o}_{\mathrm{2}}$.
This axiom specifies that it is only the outcomes and their probabilities that define a lottery. If an agent had a preference for gambling, that would be part of the outcome space.
These four axioms imply some structure on the preference between outcomes and lotteries. Suppose that ${o}_{1}\succ {o}_{2}$ and ${o}_{2}\succ {o}_{3}$. Consider whether the agent would prefer
${o}_{2}$ or
the lottery $[p:{o}_{1},(1-p):{o}_{3}]$
for different values of $p\in [0,1]$. When $p=1$, the agent prefers the lottery (because, by decomposability, the lottery is equivalent to ${o}_{1}$ and ${o}_{1}\succ {o}_{2}$). When $p=0$, the agent prefers ${o}_{2}$ (because the lottery is equivalent to ${o}_{3}$ and ${o}_{2}\succ {o}_{3}$). At some stage, as $p$ is varied, the agent’s preferences flip between preferring ${o}_{2}$ and preferring the lottery.
Figure 9.1 shows how the preferences must flip as $p$ is varied. On the $X$-axis is $p$ and the $Y$-axis shows which of ${o}_{2}$ or the lottery is preferred. The following proposition formalizes this intuition.
If an agent’s preferences are complete, transitive, and follow the monotonicity axiom, and if ${o}_{\mathrm{1}}\mathrm{\succ}{o}_{\mathrm{2}}$ and ${o}_{\mathrm{2}}\mathrm{\succ}{o}_{\mathrm{3}}$, there exists a number ${p}_{\mathrm{2}}$ such that $\mathrm{0}\mathrm{\le}{p}_{\mathrm{2}}\mathrm{\le}\mathrm{1}$ and
for all $$, the agent prefers ${o}_{2}$ to the lottery (i.e., ${o}_{2}\succ [p:{o}_{1},(1-p):{o}_{3}]$) and
for all $p>{p}_{2}$, the agent prefers the lottery (i.e., $[p:{o}_{1},(1-p):{o}_{3}]\succ {o}_{2}$).
By monotonicity and transitivity, if ${o}_{2}\u2ab0[p:{o}_{1},(1-p):{o}_{3}]$ for any $p$ then, for all $$, ${o}_{2}\succ [{p}^{\prime}:{o}_{1},(1-{p}^{\prime}):{o}_{3}]$. Similarly, if $[p:{o}_{1},(1-p):{o}_{3}]\u2ab0{o}_{2}$ for any $p$ then, for all ${p}^{\prime}>p$, $[{p}^{\prime}:{o}_{1},(1-{p}^{\prime}):{o}_{3}]\succ {o}_{2}$. By completeness, for each value of $p$, either ${o}_{2}\succ [p:{o}_{1},(1-p):{o}_{3}]$, ${o}_{2}\sim [p:{o}_{1},(1-p):{o}_{3}]$ or $[p:{o}_{1},(1-p):{o}_{3}]\succ {o}_{2}$. If there is some $p$ such that ${o}_{2}\sim [p:{o}_{1},(1-p):{o}_{3}]$, then the theorem holds. Otherwise, a preference for either ${o}_{2}$ or the lottery with parameter $p$ implies preferences for either all values greater than $p$ or for all values less than $p$. By repeatedly subdividing the region that we do not know the preferences for, we will approach, in the limit, a value filling the criteria for ${p}_{2}$. ∎
The preceding proposition does not specify what the preference of the agent is at the point ${p}_{2}$. The following axiom specifies that the agent is indifferent at this point.
(Continuity) Suppose ${o}_{\mathrm{1}}\mathrm{\succ}{o}_{\mathrm{2}}$ and ${o}_{\mathrm{2}}\mathrm{\succ}{o}_{\mathrm{3}}$, then there exists a ${p}_{\mathrm{2}}\mathrm{\in}\mathrm{[}\mathrm{0}\mathrm{,}\mathrm{1}\mathrm{]}$ such that
$${o}_{2}\sim [{p}_{2}:{o}_{1},(1-{p}_{2}):{o}_{3}].$$ |
The next axiom specifies that replacing an outcome in a lottery with an outcome that is not worse, cannot make the lottery worse.
(Substitutability) If ${o}_{\mathrm{1}}\mathrm{\u2ab0}{o}_{\mathrm{2}}$ then the agent weakly prefers lotteries that contain ${o}_{\mathrm{1}}$ instead of ${o}_{\mathrm{2}}$, everything else being equal. That is, for any number $p$ and outcome ${o}_{\mathrm{3}}$:
$$[p:{o}_{1},(1-p):{o}_{3}]\u2ab0[p:{o}_{2},(1-p):{o}_{3}].$$ |
A direct corollary of this is that outcomes to which the agent is indifferent can be substituted for one another, without changing the preferences:
If an agent obeys the substitutability axiom and ${o}_{\mathrm{1}}\mathrm{\sim}{o}_{\mathrm{2}}$ then the agent is indifferent between lotteries that only differ by ${o}_{\mathrm{1}}$ and ${o}_{\mathrm{2}}$. That is, for any number $p$ and outcome ${o}_{\mathrm{3}}$ the following indifference relation holds:
$$[p:{o}_{1},(1-p):{o}_{3}]\sim [p:{o}_{2},(1-p):{o}_{3}].$$ |
This follows because ${o}_{1}\sim {o}_{2}$ is equivalent to ${o}_{1}\u2ab0{o}_{2}$ and ${o}_{2}\u2ab0{o}_{1}$, and we can use substitutability for both cases.
An agent is defined to be rational if it obeys the completeness, transitivity, monotonicity, decomposability, continuity, and substitutability axioms.
It is up to you to determine if this technical definition of rationality matches your intuitive notion of rationality. In the rest of this section, we show more consequences of this definition.
Although preferences may seem to be complicated, the following theorem shows that a rational agent’s value for an outcome can be measured by a real number. Those value measurements can be combined with probabilities so that preferences with uncertainty can be compared using expectation. This is surprising for two reasons:
It may seem that preferences are too multifaceted to be modeled by a single number. For example, although one may try to measure preferences in terms of dollars, not everything is for sale or easily converted into dollars and cents.
One might not expect that values could be combined with probabilities. An agent that is indifferent between the money $\$(px+(1-p)y)$ and the lottery $[p:\$x,(1-p)\$y]$ for all monetary values $x$ and $y$ and for all $p\in [0,1]$ is known as an expected monetary value (EMV) agent. Most people are not EMV agents, because they have, for example, a strict preference between $1,000,000 and the lottery $[0.5:\$0,\mathrm{\hspace{0.25em}0.5}:\$2,000,000]$. (Think about whether you would prefer a million dollars or a coin toss where you would get nothing if the coin lands heads or two million if the coin lands tails.) Money cannot be simply combined with probabilities, so it may be surprising that there is a value that can be.
If an agent is rational, then for every outcome ${o}_{i}$ there is a real number $u\mathit{}\mathrm{(}{o}_{i}\mathrm{)}$, called the utility of ${o}_{i}$, such that
${o}_{i}\succ {o}_{j}$ if and only if $u({o}_{i})>u({o}_{j})$ and
utilities are linear with probabilities:
$$u([{p}_{1}:{o}_{1},{p}_{2}:{o}_{2},\mathrm{\dots},{p}_{k}:{o}_{k}])={p}_{1}u({o}_{1})+{p}_{2}u({o}_{2})+\mathrm{\dots}+{p}_{k}u({o}_{k}).$$ |
If the agent has no strict preferences (i.e., the agent is indifferent between all outcomes) then define $u(o)=0$ for all outcomes $o$.
Otherwise, choose the best outcome, ${o}_{best}$, and the worst outcome, ${o}_{worst}$, and define, for any outcome $o$, the utility of $o$ to be the value $p$ such that
$$o\sim [p:{o}_{best},(1-p):{o}_{worst}].$$ |
The first part of the proposition follows from substitutability and monotonicity.
To prove the second part, any lottery can be reduced to a single lottery between ${o}_{best}$ and ${o}_{worst}$ by replacing each ${o}_{i}$ by its equivalent lottery between ${o}_{best}$ and ${o}_{worst}$, and using decomposability to put it in the form $[p:{o}_{best},(1-p):{o}_{worst}]$, with $p$ equal to ${p}_{1}u({o}_{1})+{p}_{2}u({o}_{2})+\mathrm{\cdots}+{p}_{k}u({o}_{k})$. The details are left as an exercise. ∎
In this proof the utilities are all in the range $[0,1]$, but any linear scaling gives the same result. Sometimes $[0,100]$ is a good scale to distinguish it from probabilities, and sometimes negative numbers are useful to use when the outcomes have costs. In general, a program should accept any scale that is intuitive to the user.
A linear relationship does not usually exist between money and utility, even when the outcomes have a monetary value. People often are risk averse when it comes to money. They would rather have $\$n$ in their hand than some randomized setup where they expect to receive $\$n$ but could possibly receive more or less.
Figure 9.2 shows a possible money–utility relationship for three agents. The topmost agent is risk averse, with a concave utility function. The agent with a straight-line plot is risk neutral. The lowest agent with a convex utility function is risk seeking.
The risk averse agent would rather have $300,000 than a 50% chance of getting either nothing or $1,000,000, but would prefer the gamble on the million dollars to $275,000. They would also require more than a 73% chance of winning a million dollars to prefer this gamble to half a million dollars.
For the risk-averse agent, ${u}{\mathit{}}{\mathrm{(}}{\mathrm{\$}}{\mathit{}}{\mathrm{999000}}{\mathrm{)}}{\mathrm{\approx}}{\mathrm{0.9997}}$. Thus, given this utility function, the risk-averse agent would be willing to pay $1000 to eliminate a ${\mathrm{0.03}}{\mathrm{\%}}$ chance of losing all of their money. This is why insurance companies exist. By paying the insurance company, say $600, the risk-averse agent can change the lottery that is worth $999,000 to them into one worth $1,000,000 and the insurance companies expect to pay out, on average, about $300, and so expect to make $300. The insurance company can get its expected value by insuring enough houses. It is good for both parties.
Rationality does not impose any conditions on what the utility function looks like.
Figure 9.3 shows a possible money–utility relationship for Chris who really wants a toy worth ${\mathrm{\$}}{\mathit{}}{\mathrm{30}}$, but would also like one worth ${\mathrm{\$}}{\mathit{}}{\mathrm{20}}$, and would like both even better. Apart from these, money does not matter much to Chris. Chris is prepared to take risks. For example, if Chris had ${\mathrm{\$}}{\mathit{}}{\mathrm{29}}$, Chris would be very happy to bet ${\mathrm{\$}}{\mathit{}}{\mathrm{9}}$ against a single dollar of another agent on a fair bet, such as a coin toss. This is reasonable because that $9 is not much use to Chris, but the extra dollar would enable Chris to buy the ${\mathrm{\$}}{\mathit{}}{\mathrm{30}}$ toy. Chris does not want more than ${\mathrm{\$}}{\mathit{}}{\mathrm{60}}$, because then Chris will worry about it being lost or stolen this will leave Chris open to extortion (e.g., by a sibling).