An Experiment to Evaluate Aspects of GEA
Giuseppe
Carenini
The experiment I have run within the evaluation
framework empirically tested two specific aspects of GEA.
i) Argumentation theory indicates that supporting and opposing evidence
for the main evaluative claim of an argument should be identified
according
to a model of the reader's values and preferences. In GEA, it is
assumed
that such a model can be effectively represented as an AMVF, a
quantitative
model of preferences originally developed in decision theory. This
assumption
has been tested by comparing the effectiveness of arguments tailored to
the user's AMVF with the effectiveness of arguments tailored to a
default
AMVF, in which all aspects of the evaluated entity are equally
important.
The outcome of this comparison was that tailoring the argument to the
user
model makes a significant difference in argument effectiveness.
ii) Argumentation theory also indicates that evaluative arguments
should
be concise, presenting only pertinent and cogent information. However,
it remains an open question what is the most effective degree of
conciseness.
As a preliminary attempt to determine an optimal level of conciseness
for
evaluative arguments, we have compared the effectiveness of arguments
generated
by our argument generator at two different levels of conciseness. The
outcome
of this comparison was that differences in conciseness significantly
influence
argument effectiveness. However, since only two levels of conciseness
were
compared, more empirical work is needed to determine the optimal level
of conciseness.
The experiment also shows that the framework is usable and robust. More
than 40 subjects accomplished the selection task by interacting with
the
framework and measures of argument effectiveness were successfully
assessed.
Only two subjects did not manage to accomplish the selection task.
Published papers on the experiment
results:
Giuseppe Carenini and Johanna Moore, An Empirical Study of
the Influence
of Argument Conciseness on Argument Effectiveness . The 38th Annual
Meeting of the Association for Computational Linguistics. (ACL 2000)
Hongkong,
China, 2000. [pdf]
Giuseppe Carenini and Johanna Moore, An Empirical Study
of the Influence
of User Tailoring on Evaluative Argument Effectiveness, Proceedings
of the 17th International Joint Conference on Artificial Intelligence
(IJCAI
2001), Seattle, USA, 2001 [pdf]
Please, send comments and inquiries to carenini@cs.ubc.ca