Overview Emotions View Ecological View Context Media Implications Adaptive Recognition

Ecological Facial Display Recognition

Facial expression is an important element in human communication. The dominant school of thought is that the human face expresses the underlying emotion of the human [1] (the Emotion View). That is, a person is in an emotional state, and this state is expressed in their face. They can attempt to hide the emotion, but there is often some detectable leakage. Furthermore, the emotion view holds that facial expressions are universal.

The emotions view has recently been challenged by a number of authors. They propose an alternative view of facial displays are signals of social intent [2](the Behavioral Ecology View). Behavioural ecologists believe that facial expression is a way of communicating through the medium of a social context. If we hold this view, then it is not possible to characterize facial expressions in terms of universal emotions.

The controversy over these two conflicting views of facial expression has implications for research in human-computer interaction which propose to use facial signals as input. Until now, automatic expression studies have focused on the Emotion View approach, attempting to recognize six basic expressions of emotion (happy, sad, angry, surprised, fearful and disgusted - with sometimes an additional neutral) [3,4]. Recent approaches [5] have focused on a more detailed level of analysis, attempting to recognize the 44 facial action units of the Facial Action Coding System (FACS) [6]. The motivation behind this work is that, once a system can detect these 44 action units, then the emotional state of the human subject can be inferred. The emotional state of the user can then be used to guide the computer side of a human-computer interaction. However, this motivation is not well founded for two reasons.

  1. The mapping between the space of 44 action units and the ephemeral term emotion is unclear. Proponents of the emotions view would argue that it does exist, and is universal. Universality has been called into question, however [7].
  2. It has become clear that detecting emotions of a computer user may not be the optimal way of using facial expression data. Facial expressions are used in normal conversation for many reasons. These include both semantic and syntactic support of what the speaker is saying, as well as reactions in the listener's face to offer support of continuation of the dialogue [8]. Facial displays of this kind are extremely context dependent, for both speaker and listener. That is, the displayer can use a single facial display in many different contexts, and the display will assume a different meaning for the watcher depending on the context he or she perceives. A computer which can pick up on these cues will certainly find them extremely useful in inferring meaning in what the user is communicating. Furthermore, the user will feel like they can communicate naturally with the computer, as they do with other humans. In turn, this will make the experience more enjoyable [9].

Contrary to current beliefs, facial action is not often tied to universal emotions, but are dependent both on context and culture. The paradigm for studying expression recognition in terms of a set of prototypes must be rejected. A system for recognizing facial action with the goal of identifying emotional states will not be sufficient for visual intelligent agents of the future. Instead, such a system must take context into account. It must be adaptible, or unsupervised.

... contra the Emotions View, the Behavioural Ecology View does not require [prototype faces]. Rather, because displays exert their influence in the particular context of their issuance, they may only be interpretable within that context. ([2], p.106).
It is important to note that 44 facial action (AU) units (AUs) are sufficent to characterize any expression, however, and that a recognition system could be built upon this 44-dimensional, (almost completely) boolean, vector. However, it will still be a mistake to then characterize combinations of these AUs as distinct, universal emotions, which are then used to infer appropriate agent behaviour.

Read more

  1. Carroll E. Izard. Emotions and facial expressions: A perspective from differential emotions theory. In James A. Russell and Jose Miguel Fernandez-Dols, editors, The Psychology of Facial Expression, chapter 5, pages 103-129. Cambridge University Press, Cambridge, UK, 1997.
  2. Alan J. Fridlund. The new ethology of human facial expressions. In James A. Russell and Jose Miguel Fernandez-Dols, editors, The Psychology of Facial Expression, chapter 7, Cambridge University Press, Cambridge, UK, 1997.
  3. Michael Black and Yaser Yacoob. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motions. International Journal of Computer Vision , 25 (1):23--48, 1997.
  4. Jesse Hoey and and James J. Little. Representation and recognition of complex human motion. In Proc. IEEE CVPR , Hilton Head, SC, June 2000.
  5. Ying li Tian, Takeo Kanade, and Jeffrey F. Cohn. Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence , 23(2), February 2001.
  6. Paul Ekman and W.Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement . Consulting Psychologists Press, Palo Alto, CA, 1978.
  7. James A. Russell. What does facial expression mean? In James A. Russell and Jose Miguel Fernandez-Dols, editors, The Psychology of Facial Expression, chapter 1, Cambridge University Press, Cambridge, UK, 1997.
  8. Janet B. Bavelas and Nicole Chovil. Faces in Dialogue. In James A. Russell and Jose Miguel Fernandez-Dols, editors, The Psychology of Facial Expression, chapter 15, p.334-346 Cambridge University Press, Cambridge, UK, 1997.
  9. Byron Reeves and Clifford Nass The Media Equation . Cambridge University Press, 1996.