Document Type

Conference Paper

Department/Unit

Department of Computer Science

Title

A user-centric evaluation framework for recommender systems

Language

English

Abstract

This research was motivated by our interest in understanding the criteria for measuring the success of a recommender system from users' point view. Even though existing work has suggested a wide range of criteria, the consistency and validity of the combined criteria have not been tested. In this paper, we describe a unifying evaluation framework, called ResQue (Recommender systems' Quality of user experience), which aimed at measuring the qualities of the recommended items, the system's usability, usefulness, interface and interaction qualities, users' satisfaction with the systems, and the influence of these qualities on users' behavioral intentions, including their intention to purchase the products recommended to them and return to the system. We also show the results of applying psychometric methods to validate the combined criteria using data collected from a large user survey. The outcomes of the validation are able to 1) support the consistency, validity and reliability of the selected criteria; and 2) explain the quality of user experience and the key determinants motivating users to adopt the recommender technology. The final model consists of thirty two questions and fifteen constructs, defining the essential qualities of an effective and satisfying recommender system, as well as providing practitioners and scholars with a cost-effective way to evaluate the success of a recommender system and identify important areas in which to invest development resources. © 2011 ACM.

Keywords

e-commerce recommender, post-study questionnaire, quality of user experience, recommender systems

Publication Date

2011

Source Publication Title

Proceedings of the fifth ACM conference on Recommender systems

Start Page

157

End Page

164

Conference Location

Chicago, United States

Publisher

ACM

DOI

10.1145/2043932.2043962

Link to Publisher's Edition

http://dx.doi.org/10.1145/2043932.2043962

ISBN (print)

9781450306836

This document is currently not available here.

Share

COinS