Recommenders for Information Seeking Tasks Lessons Learned Michael

Recommenders for Information Seeking Tasks: Lessons Learned Michael Yudelson

References o o o Joseph A. Konstan, Sean M. Mc. Nee, Cai-Nicolas Ziegler, Roberto Torres, Nishikant Kapoor, John Riedl: Lessons on Applying Automated Recommender Systems to Information-Seeking Tasks. AAAI 2006 Mc. Nee, S. M. , Kapoor, N. , and Konstan, J. A. 2006. Don't look stupid: avoiding pitfalls when recommending research papers. In Proceedings of the 2006 20 th Anniversary Conference on Computer Supported Cooperative Work (Banff, Alberta, Canada, November 04 08, 2006). CSCW '06. ACM Press, New York, NY, 171 -180. Michael Yudelson, AAAI 2006 Nectar Session Notes 2

Overview o o o Statement of the Problem Theories General Advice Experiment Lessons Learned 3

o “There is an emerging understanding that good recommendation accuracy alone does not give users of recommender systems an effective and satisfying experience. Recommender systems must provide not just accuracy, but also usefulness. ” J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, "Evaluating Collaborative Filtering Recommender Systems", ACM Trans. Inf. Syst. , vol. 22(1), pp. 5 -53, 2004. 4

Statement of the Problem o User is engaged in an information seeking task (or several) n o Movies, Papers, News Goal of the recommender is to meet user specific needs with respect to n n n Correctness Saliency Trust Expectations Usefulness 5

Theories o o Information Retrieval (IR) Machine Learning (ML) Human-Recommender Interaction (HRI) Information Seeking Theories n n Four Stages of Information Need (Taylor) Mechanisms and Motivations Model (Wilson) Theory of Sense Making (Dervin) Information Search Process (Kuhlthau) 6

General Advice o o Support multiple information seeking tasks User-centered design n o Shift focus from system and algorithm to potentially repeated interactions of a user with a system Recommend n n Not what is “relevant”, But what is “relevant for info seeking task X” 7

General Advice (cont’d) o Choice of the recommender algorithm n n Saliency (the emotional reaction a user has to a recommendation) Spread (the diversity of items) Adaptability (how a recommender changes as a user changes) Risk (recommending items based on confidence) 8

What Can Go Wrong o Possible pitfalls (semantic) n n not building user confidence (trust failure) not generating any recommendations (knowledge failure) generating incorrect recommendations (personalization failure), and generating recommendations to meet the wrong need (context failure) 9

Experiment o o Domain - Digital Libraries (ACM) Information Seeking Tasks n n o Find references to fit a document Maintain awareness in a research field Subjects - 138 n 18 students, 117 professors/researchers, 7 non -computer scientists 10

Experiment (cont’d) o Tested recommending algorithms n n User-Based Collaborative Filtering (CF) Naïve Bayesian Classifier (Bayes) Probabilistic Latent Semantic Indexing (PLSI) Textual TF/IDF-based algorithm (TFIDF) 11

Experiment (cont’d) o Walkthrough n n n Seed the document selection (self or others) Tasks (given seeded documents ) o What are other relevant papers in the DL interesting to read o What are the papers that would extend the coverage of the field Compare recommendations of 2 algorithms (each recommends 5 items) o Satisfaction with algorithm A or B on likert scale o Preference of algorithm A or B 12

Experiment (cont’d) o Anticipated Results n n n CF - golden standard PLSI - comparable with CF Bayes - generating more mainstream recommendations, worse personalization TFIDF - more conservative, yet coherent results CF + PLSI vs. Bayes + TFIDF 13

Experiment (cont’d) o Results n Dimensions o o o n n n Authoritative Work, Familiarity, Personalization Good Recommendation, Expected, Good Spread Suitability for Current Task CF + TFIDF significantly better feedback that Bayes + PLSI No significant difference between CF & TFIDF or Bayes & PLSI Contradicts IR & ML literature 14

Experiment (cont’d) o What went wrong n n o Bayes - generated similar recommendations for all users PLSI - random, “illogical” recommendation Both Bayes and PLSI n n n Highly dependant on “connectivity” (co-citation) of papers Suffered from inconsistency Didn’t “fail”, but were “inadequate” 15

Lessons Learned o o Understanding the task is more important than achieving high relevancy of recommendation for that task Understanding whether searcher knows what s/he’s looking for is crucial n o There is no “golden bullet” People think of recommenders as machine learning systems n modeling what you already know, predicting the past and penalizing for predicting the future 16

Lessons Learned (cont’d) o Dependence on offline experiments created a disconnect between algorithms that score well on accuracy metrics and algorithms that prove useful for users n Problem of Ecological Validity 17

Lessons Learned (cont’d) o 1 good recommendation in a list of 5 n o Wins trust of the user Loses trust of user If user needs are unclear n Do a user study to elicit them 18

Thank you! o Questions… 19