Exploring more Realistic Evaluation Measures for Collaborative Filtering

By Giuseppe Carenini (based on joint work with Rita Sharma)

Collaborative filtering is a popular technique for recommending items to people. Several methods for collaborative filtering have been proposed in the literature and the quality of their predictions compared in empirical studies. In this paper, we argue that the measures of quality used in these studies are based on rather unrealistic assumptions. We propose new measures for comparing the effectiveness of collaborative filtering methods which are grounded in decision-theory. Our new measures attempt to compute how useful a recommendation will be for the user. For this, a specification of a user threshold for accepting a recommendation is required. We report preliminary results from a user study in which we elicited both the users' thresholds for accepting a recommendation and their utility functions for ratings in the movie domain. Finally, we discuss the application of our new measures to an existing dataset.


Back to the LCI Forum page