Design of a Pilot Experiment for the Evaluation

Design of a Pilot Experiment for the Evaluation of Automated Persuasion systems Marco Guerini (ITC-Irst) Shlomo Hareli (University of Haifa) Shy Tzafrir (University of Haifa)

General Motivations l Persuasive interfaces need to be evaluated: they have to be effective with real users. l Evaluation studies up to now have focused mainly on usability. l Evaluation of effectiveness is not straightforward, specific methodologies have to be used / defined. 2

Approach l Use of experiments/measures coming mainly from Social Psy. l Specific Question: Can display of emotion enhance the persuasive force of a message in HCI? 3

Specific Motivations l 2 specific interests: ¡ Automated persuasive communication: evaluation study of the Promoter Prototype, developed at Irst. ¡ Emotions in persuasive comm. : study of the Fitting Principle (Univ. of Haifa) in persuasive Computer-Human Communication 4

Promoter prototype l Content Ordering Meta-Strategies (COMS). Given basic persuasive strategies, how to compose them to create a complex message. l Example of basic persuasive strategy on positive consequences: “action a has the positive outcome o”. ¡ ¡ ¡ COMS 1 a message made with positive pieces of information. As a pilot experiment we will start from this one. COMS 2 based on framing effect theory (positive consequences of the required action following negative consequences of a conflicting action) etc… 5

The Fitting Principle l The theory we use for emotion displaying is the Fitting principle (University of Haifa): ¡ A message that is accompanied by an emotional expression that fits the valance of the message will be more persuasive than the same message emitted with no emotion or an emotional expression that contradicts the message. 6

Advantages ¡ an evaluation study of the specific persuasion system (Promoter) ¡ a contribution to the design of a general model for evaluation of automated persuasion systems 7

Conditions of the Experiment l The experiment will study one persuasive meta-strategy (i. e. COMS 1) on 3 different conditions (between subjects, each person gets only one condition). 3 conditions are: ¡ Fitting emotion ¡ Unfitting emotion ¡ Neutral, no emotion, just message 8

The message You should come and visit the renewed exhibition on the archaeology of the Land of Israel at the Hecht Museum. It has pieces from the Chalcolithic, through the Hellenistic, Canaanite up to the Byzantine Period. This is a good opportunity to enlarge your knowledge about Israel History, to learn nice anecdotes to tell your friends and it is really easy to reach, since it is inside your own university! 9

Fitting Clip 10

Unfitting Clip 11

Neutral Clip 12

Expected Results l The Fitting message is more persuasive than the other messages. The unfitting message is less persuasive relative to the other messages. 13

Conclusions (1) l Evaluation of Persuasive interfaces: specific methodologies have to be used / defined. l Design of an experiment using measures coming from Social Psy. 14

Conclusions (2) l Specific question: Can display of emotion enhance the persuasive force of a message in HCC? l Advantages: evaluation study of the specific system (Promoter) & contribution for a general model for evaluation of automated persuasion systems 15

Future work (1) l Preamble: We can expect, from studies in HHI and HCI, that each persuasive strategy alone works (COMS, fitting princ. ) l Broader result: not just the replication of studies already done. ¡ Test the interaction of different strategies: there is no results about their mixing. 16

Future Work (2) l Promoter uses four main content-ordering persuasive strategies: the experiment replicated conceptually three times, one for each strategy (after COMS 1, also COMS 2, COMS 3, COMS 4). l Study the interaction between fitting principle and COMS strategies. 17

Ethical Issues l Challenge 1: agents that are ethically aware (i. e. capable of reasoning on the ethicality of their actions and possibly to correct their behaviour) l Challenge 2: a “layered” model of ethical persuasion: ethicality of the act of persuading + ethicality of the induced action l Ethical capabilities to be encoded at the metareasoning level 18