A Framework to Evaluate Evaluative Arguments
Giuseppe
Carenini
The evaluation framework is based on the task efficacy evaluation
method.
In this method, a generation model is evaluated without explicitly
evaluating
its output but by measuring the output's effects on users' behaviors,
beliefs
and attitudes in the context of a task. To apply the task
efficacy
method, I chose a variation of a rather basic and frequent task that
has
been extensively studied in decision analysis: the selection of a
subset
of preferred objects (e.g., houses) out of a set of possible
alternatives
by considering trade-offs among multiple objectives (e.g., house
location,
house quality). This task requires the decision-maker to evaluate and
compare
all the available alternatives and so it is particularly suitable for
evaluating
evaluative arguments.
To support the user in the exploration of the set of alternatives,
I have developed a computer environment for interactive data
exploration
and analysis that provides powerful visualization and direct
manipulation
techniques.
In order to evaluate evaluative arguments within the framework, some
measures of argument effectiveness must be defined and assessed.
Furthermore,
as is the case in any experimental setting, possible confounding
variables
must be controlled. Since evaluative arguments can be used as a form of
persuasive communication, the measures of argument effectiveness and
the
variables to control for are based on principles developed in social
psychology
to study persuasion.
Software: The
framework
is implemented in visage,
an info-centric data exploration environment.
It runs on PCs. If interested in working with
the framework, please contact carenini@cs.ubc.ca
Published papers on the evaluation
framework:
Giuseppe Carenini. A Task-based Framework to Evaluate
Evaluative
Arguments.
International Natural Language Generation Conference, Mitzpe
Ramon,Israel.
2000.
Please, send comments and inquiries to carenini@cs.ubc.ca