A Framework to Evaluate Evaluative Arguments

Giuseppe Carenini

The evaluation framework is based on the task efficacy evaluation method. In this method, a generation model is evaluated without explicitly evaluating its output but by measuring the output's effects on users' behaviors, beliefs and attitudes in the context of a task.  To apply the task efficacy method, I chose a variation of a rather basic and frequent task that has been extensively studied in decision analysis: the selection of a subset of preferred objects (e.g., houses) out of a set of possible alternatives by considering trade-offs among multiple objectives (e.g., house location, house quality). This task requires the decision-maker to evaluate and compare all the available alternatives and so it is particularly suitable for evaluating evaluative arguments.
To support the user in the exploration of the set of alternatives, I have developed a computer environment for interactive data exploration and analysis that provides powerful visualization and direct manipulation techniques.
In order to evaluate evaluative arguments within the framework, some measures of argument effectiveness must be defined and assessed. Furthermore, as is the case in any experimental setting, possible confounding variables  must be controlled. Since evaluative arguments can be used as a form of persuasive communication, the measures of argument effectiveness and the variables to control for are based on principles developed in social psychology to study persuasion.

Software: The framework is implemented in visage, an info-centric data exploration environment.
It runs on PCs. If interested in working with the framework, please contact carenini@cs.ubc.ca

Published papers on the evaluation framework:

Giuseppe Carenini.  A Task-based Framework to Evaluate Evaluative Arguments.
International Natural Language Generation Conference, Mitzpe Ramon,Israel. 2000.

Please, send comments and inquiries to carenini@cs.ubc.ca