An Experiment to Evaluate Aspects of GEA

Giuseppe Carenini

The experiment I have run within the evaluation framework empirically tested two specific aspects of GEA.
i) Argumentation theory indicates that supporting and opposing evidence for the main evaluative claim of an argument should be identified according to a model of the reader's values and preferences. In GEA, it is assumed that such a model can be effectively represented as an AMVF, a quantitative model of preferences originally developed in decision theory. This assumption has been tested by comparing the effectiveness of arguments tailored to the user's AMVF with the effectiveness of arguments tailored to a default AMVF, in which all aspects of the evaluated entity are equally important. The outcome of this comparison was that tailoring the argument to the user model makes a significant difference in argument effectiveness.
ii) Argumentation theory also indicates that evaluative arguments should be concise, presenting only pertinent and cogent information. However, it remains an open question what is the most effective degree of conciseness. As a preliminary attempt to determine an optimal level of conciseness for evaluative arguments, we have compared the effectiveness of arguments generated by our argument generator at two different levels of conciseness. The outcome of this comparison was that differences in conciseness significantly influence argument effectiveness. However, since only two levels of conciseness were compared, more empirical work is needed to determine the optimal level of conciseness.
The experiment also shows that the framework is usable and robust. More than 40 subjects accomplished the selection task by interacting with the framework and measures of argument effectiveness were successfully assessed. Only two subjects did not manage to accomplish the selection task.

Published papers on the experiment results:

Giuseppe Carenini and Johanna Moore, An Empirical Study of the Influence of Argument Conciseness on Argument Effectiveness . The 38th Annual Meeting of the Association for Computational Linguistics. (ACL 2000) Hongkong, China, 2000. [pdf]

Giuseppe Carenini and Johanna Moore, An Empirical Study of the Influence of User Tailoring on Evaluative Argument Effectiveness, Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI 2001), Seattle, USA, 2001 [pdf]

Please, send comments and inquiries to