College 5 Designing and Evaluating Groupware Dr Mari

Скачать презентацию College 5 Designing and Evaluating Groupware Dr Mari

b650ec117bcaad2931ce0767ee38dd8a.ppt

Количество слайдов: 60

College 5: Designing and Evaluating Groupware Dr. Mari Carmen Puerta Melguizo

Overview: A bit about… n CSCW n Groupware systems n. Designing Groupware and evaluating

Remember n Distributed work & virtual groups Ø Because of the distributed nature of their interaction they are more or less dependent on groupware systems n n Ø interpersonal communication more difficult : lack of nonverbal signals, unplanned encounters, context awareness… difficulty in collaboration, coordination, developing trust, exchanging knowledge ICT tools have to match with the type of group and its task and to support the group processes

Remember n Group. Ware Ø Ø Technology that people use to work together Collaborative tasks supported n n n E-messages distributed Scheduling Voting Off-line discussion Distributing and storing electronic documents. . . CSCW Ø Ø Study of how people work together as a group and how technology supports/affects it Social issues n n People bring in different perspectives and views to a collaboration environment Goal of Group. Ware is often to establish some common ground and to facilitate understanding and interaction

Designing Groupware n More difficult than traditional software Ø n E. g. Ease-of-use must be better for groupware than for single-user systems because the pace of use of an application is often driven by the pace of a conversation Groupware design involves understanding Ø groups and how people behave in groups n n Ø E. g. million-person groups behave differently from 5 -person groups the performance parameters of the technologies to support different groups vary… networking technology and how affects users n n E. g. delays in synchronizing views The World Wide Web and its associated tools and standards have had a major impact on the possibilities for Group. Ware (Berners-Lee, 1999). These group tools require Ø Browser & plug-ins Ø Also problems with security in internet: development of middleware and Grid Technologies

Designing Groupware n The design and introduction of Group. Ware is more successful if (Andriessen, 2003): Ø The users (stakeholders) concerned are involved in the various stages of the process n n E. g. User-centred design: exploring stakeholders reactions with scenarios and prototypes E. g. participative design: involving stakeholders in the whole design process. Users help develop the information system and their own workplace Ø The wider context is taken into account and tools and organizational context match Ø The organization is also adapted technical innovation cannot operate without organizational innovation n design and utilisation of Group. Ware requires understanding from both technology & organisational change

Designing Groupware n understanding the co-operation Ø n technology related decisions are context dependable & require understanding of the work process methods & their utilisation takes time & resources Ø requires training Ø changes with the context of design & use Ø

The design process n Activities Ø Analysing the context of use n n Ø Current situation The future Task Model 1 Task Model 2 Detail design n Functionality Dialog Representation Ø Evaluation n Usability, easy of use, likeability… Evaluation Ø n Implementation Iterative Process Implementation maintaining consistency

work organization/ practice Contextual analysis Task Model 1 users’ knowledge/ behavior/needs Documents/ artifacts problem analysis/ specification Mental models Knowledge acquisition usability measuring early evaluation validity analysis Task Model 2 specification/ negotiation feedback Client constraints/ opportunities Technology Functionality Scenario Simulation Prototype early evaluation UVM Dialog Representation Implementation maintaining consistency

Analysis of task, context technology, stakeholders Identification of technological and organisational options Development of Future Usage Scenarios Design requirements Identification of Stakeholders’ success factors Choice of Evaluation approach Analysis of potential impacts Iterative tests of prototypes Iterative design and implementation Use, experience, adaptation of new system Concept Evaluation’ Prototype Evaluation Operational Evaluation

1. Analysing the context of use When designing an innovative product, it is important to make sure the new product has some add value • Improving already existing possibilities • Providing new possibilities n designing taking into account the whole context of use n Modelling Ø Ø Current situation The future

The context of use n the users: identifying stakeholders and their requirements Designers must have an understanding of the degree of homogeneity of users, of the possible roles people play in cooperative work and of who key decision-makers are and what influences them Ø The individual user’s needs and requirements n Ø mental models, knowledge, user acceptance, satisfaction, safety… The groups, communities of practice and organizations that will be using the groupware system n Group communication, Identify power structures and roles… n the work Ø The tasks and work practice n Ø E. g. general info on how people work in many different settings (airports, hotels) Equipment: hardware, software and materials n the context of work or situation Ø Ø The physical environment The social environment: Organizational structure, Organizational culture and routines…

Identifying the stakeholders and their requirements n Stakeholders Ø Primary n people who actually use the system: end-users Ø Ø Buying your plane ticket online Secondary People who receive output from the system or provide input to it n Ø Ø Tertiary n People directly affected by the success or failure of the system and are not primary or secondary Ø Ø competitors Facilitating n People involved in the design, development and maintenance of the system Ø n Airline staff Design team Conflicting goals and needs

Studying Work practices and group behaviour n Not only formal routine work, but also informal and local practices n Methodologies Semi-structured interviews Questionnaires Document analysis… Ethnography Ø Ø n Ø Ø Can’t ask workers about their work process (they can’t observe themselves). One can instead observe users (and participate) and then analyse Contextual analysis Distributed Cognition

Contextual analysis n Going to the real situation to explore the impact of the systems in the organizations Ø Organizational structure and routines analysis Ø Organizational culture and history n Ø Socio-political analysis n Ø group and national culture E. g. In individualistic cultures (EU, USA) people prefer direct expression of opinions; In virtual groups they prefer synchronous communication, through telephone, video and chat In collectivist cultures (Asia, Africa) people are sensitive to non-verbal signals and group relations. In virtual groups people prefer asynchronous communication, to be able to express themselves more carefully: e-mail different groups of interests; how the implementation of the system change the power relations in the organization Analysis of work-related meanings, world thoughts/interpretations analysis n different interpretations/meanings in relation with the system by managers, IT and HCI people

Some personal examples n Manufacturing Company Zeta n Dutch Police Force n Assurance Company Y n Banks in different countries (The Netherlands and Romania)

Manufacturing Company Zeta n Setting: Ø n Problem: Ø n collaboration with the User Interface Design team complains from different groups of end-users from different organizations regarding the user interface Findings: Ø cultural issues at level of end-users: national and occupational culture (related especially with the language/jargon) n n Ø Interface Communication between different types of users was also affected strategic issues at the level of organizations n Permission and access to tools in the interface by different users

Dutch Police Force n Setting: Ø collaboration with Usability specialists from Dutch Police Force n Problem: Ø the design of a new integrated crime information system n Findings: Ø national level: n Ø different systems – problems in sharing knowledge and knowledge management local level: n n culture of the regions impact on the use of the system : flexibility vs. hierarchical occupational cultures impact on the use of the systems: officers vs. detectives

Assurance Company Y n Setting: Ø n Problem: Ø n Collaboration with Marketing and ICT department The low percentage use of the software of the company by the assurance agents working in different organizations Findings: Ø Ø Ø The resistance to use the systems was not due to the usability of the systems (which was high, measured with SUMI) but to the fact that systems do not support the work processes and the goals of the agents as they should Using scenarios (personas) for the re-design of the software

The X Bank n Setting: Ø n Problem: Ø n Collaboration with ICT managers of the bank user dis(satisfaction) with IT (bankshops) Findings: Ø system design-related factors n Ø reliability, speed of the systems, navigation structure… organizational-related factors n lack of feedback, help-desk perception, ICT department perception, managers’ attitude towards the technology, the issues of implementation of new systems, training, cultural factors…

2. Detail Design n Functionality specifying want the tool will do to the user; strongly related to the task n Dialog how the user and the system will communicate; commands? menu? n Representation how the system looks like Maintaining consistency!!!

More designing issues n n n n Formal vs. informal communication: awareness Socially vs. Technologically Mediated Communication Customization and grounding Session control Floor control Privacy and anonymity Sharing Information, Identification, and Accountability Avoiding abuse…

Designing for communication n Formal and informal communication is used to coordinate routine tasks Ø brief, informal communication such as spontaneous hallway conversations can help establish trust, promote social relationships and provide background information about the work environment Ø Ø Groupware systems have to incorporate a way of having awareness!!! n Awareness of location and activity of other people

Awareness n Implicit communication helps to Ø Ø Ø Ø Ø establish common ground coordinate activities avoid surprises… indirect gestures What is happening? , Who is there (e. g. IM buddy list) information about people's environment (whether their office door is open or closed) biographical information (what their job position is) letting them know what document you're working on, or how you're feeling at any given time…

Awareness n e. g. flight progress strips in air traffic control are on the focus for discussions and activities and meaningful conversations themselves Ø vocabulary and syntax of annotation and the order of strips are a language that makes sense for experienced observers Ø

Awareness n Awareness information takes many forms Ø Synchronous awareness n n Ø Asynchronous awareness n n n Real-time audio-video systems (e. g. chat systems with buddy lists) Videoconferencing: a wide-angle camera lens can provide a greater degree of environmental awareness Using e-mail, online calendars… to find out when people are available Email: information about the time and date of the message or the signature file of the sender awareness can be at odds with privacy concerns Ø important to give users control over how much information about themselves is made available to others

Designing technologies to support greater awareness n Media spaces - “extend the world of desks, chairs, walls and ceilings” (Harrison et al, 1997) Ø n Examples: Clearboard, Portholes and Cruiser Notification systems Ø Ø Users notify others as opposed to being constantly monitored (cf Portholes) Provide information about shared objects and progress of collaborative tasks n Examples: Tickertape, Babble

Clearboard (Ishii et al, 1993) Clear. Board - transparent board that shows other person’s facial expression on your board as you draw

Portholes (Xerox PARC) Regularly updated digitized images of people in their offices appeared on everyone’s desktop machines throughout day and night

Tickertape (Segall and Arnold, 1997) n n Tickertape is a scrolling one-line window, going from left to right Group name, sender’s name and text message

Babble (IBM, Erickson et al, 1999) Circle with marbles represents people taking part in conversation in a chatroom. Those in the middle are doing the most chatting. Those towards the outside are less active in the conversation.

Socially vs. Technologically Mediated Communication n If the type of communication structure of the group is known, systems can take advantage of the structure to speed up communications and minimize error n Exceptions to the expected structure of communication are extremely common Ø Ø n technologically-mediated communication may actually be an obstruction to getting work done efficiently and may lead people to not use a groupware system or use it incorrectly The designers need to anticipate the range of communication possibilities Facilitation vs. Enforcement Ø Take the common structure of communication to make tasks more straightforward n Ø e. g. by providing a "quick send" button that routes a message to the appropriate person But make sure any kind of message can be sent regardless n Thus, the communication is technologically-facilitated but not technologicallyenforced

Customization and grounding n When groups are working together with the same information, they may individually desire customized views n The challenge of customized views is to support grounding: the establishment of a common ground or shared understanding of what information is known and shared between the different users. Ø E. g. a healthcare setting n n n a physician talks to a lab technician about a patient and have access to the same patient record each may want a view on their computer screen which selects and emphasizes different pieces of information This may cause confusion when a given piece of information is readily available to one person and not the other. Another concern is if one user chooses to display exceptional values in red another chooses to display exceptional values in blue, different users may be confused When working together on the same screen, this inconsistency can result in dangerous miscommunication. In communication situations, it's important Ø to make it very clear what information is private and what is shared n Ø as much as possible, make it clear what information the other user is seeing (e. g. provide a miniature or summary view of the other person's screen) In all cases, be sure to maintain consistency of the data. Users should never see spurious or irreconcilable differences.

Session control n situation where a group of people are in a conversation together at a given time (e. g. chat) n Metaphorically, session control is like a person standing at the door of a room checking IDs and deciding who gets to go in Ø n Session control issues include finding out what rooms are available, determining who can enter and exit the room, and when and how some suggested policies for session control: Ø Ø Ø Decide what limits there are to who can join a session. Are there limits to the number of people or to who is qualified to enter? Allow people to join and leave at any time. Provide a "polite" protocol for doing so Avoid intrusive situations where users are able to invade privacy or impose a session on others Provide a means for preventing interruptions Facilitate people getting together. Provide mechanisms for identifying appropriate conversational partners. Provide a means for setting up side conferences

Floor control n Once people have joined a conversational session, it must be decided what kind of access each person has to shared artifacts, or conversational props Ø E. g. in a shared whiteboard, everyone can draw on it at the same time (simultaneous access), or only one person can access it at a time (by passing a token, or baton), is there a moderator who controls access, and is there a time limit for each person? n Simultaneous access by everyone to everything is often preferred for the most fluid conversation, but it can be vulnerable (especially with a large number of people) n The advantages to providing some kind of mediated access include preventing mistakes, preventing unauthorized access, and avoiding people making conflicting changes. n some intermediate solutions are also possible Ø E. g. there can be multiple whiteboards. Some may be personal and others shared. Personal whiteboards may be visible to other users but noneditable by other users. This allows everyone to work simultaneously without interfering with the work of others.

Privacy and anonymity n Whenever using groupware, some information needs to be shared, and all other information remain private and secure even against aggressive attempts to obtain the information n In many situations, users choose to be anonymous or use a consistent pseudonym. Anonymity can be crucial in encouraging fair participation in discussions and is useful for providing protection from harassment n Negative: Ø n You don’t know who the person communicating with you is. Can result in “flaming” and similar nuisances. Positive: Ø Ø Reduce the effects of power hierarchies (you don’t care as much when you e-mail your boss as when you talk to him) Helps people speak their minds. Electronic chat may be used for meetings to get honest opinions

Sharing Information and Identification Ø there is continuing pressure to share more information. The more information gets shared, the more easily common ground can be achieved Ø Sharing information about yourself enables many systems to provide more useful customization and matching to your interests Ø while anonymity can protect an individual, there also quite legitimate reasons for identifying people for accountability, especially where security and the risk of abusive behaviour are involved

Sharing Information and Identification n Control and Reciprocity Ø To resolve these conflicting needs, it's important to give users as much control as possible over what information gets shared and what remains private Ø One example of privacy policy is the principle of reciprocity n if a user wants information about another user, then they must provide the equivalent information about themselves. Reciprocity isn't always the right policy, but serves as a useful starting point.

Avoiding abuse n n Spamming with email Taking inappropriate advantage of anonymity, sabotaging group work, or violating privacy…

3. Evaluation n Quite challenging Ø Ø Ø n Need more participants Logistically difficult Difficult to generalize from one situation to another As soon as you start making decisions!!! Ø Ø a. Concept evaluation: e. g. scenarios b. Prototype evaluation: e. g. prototypes c. Operational assessment Usability, easy of use, likeability, acceptance of use…

Evaluation methods 1. Inspection methods Ø Ø Heuristic Evaluation Cognitive walk-through 2. Laboratory studies Ø Ø Ø Performance analysis (Human Reliability Analysis) Behaviour analysis (Diagnostic Recorder for Usability Measurement (DRUM) factors related to social settings or to activity are impossible to study in laboratories 3. Field studies and ethnography Ø group processes evolve over time (days, weeks, even months) 4. Effort and satisfaction Ø Ø Measuring the Usability of Multi-Media Systems (MUMMS) Multi. Media Communication Questionnaire (MMCQ) 5. Network performance 6. System usage and interaction registration Ø Ø Automatic registration of the use of the system Coding schemes for communication content

Conducting user studies on system prototypes n More difficult that with single-users systems Ø Organizing and scheduling for groups is more difficult than for individuals Ø Pre-established groups vary in interaction style, and the length of time they've been a group affects their communication patterns Ø Groups are dynamic and change, roles also change Ø Many studies need to be long-term, especially when studying asynchronous groupware Ø Modifying prototypes can be technically difficult because of the added complexity of groupware over single-user software Ø In software for large organizations, testing new prototypes can be difficult or impossible because of the disruption caused by introducing new versions into an organization

Evaluation issues (Andriessen, 2003) • Describe the tool characteristics: • reliability, portability, maintainability, network performance, costs, infrastructural quality, security/privacy and evaluate whether this is adequate (ISO-9126) • Describe the functionalities • Analyse the task and evaluate whether the functionalities fit the task • Analyse the users and evaluate whether the tool fits the users (usability) • Analyse the group (structure, culture, setting) and evaluate whether the tool fits the group • Evaluate whether the tool supports (or at least does not hinder) the group processes: communication, co-operation, co-ordination, learning, social interaction • Evaluate whether the tool contributes to (or at least does not hinder) individual, group, organisational outcomes. • Evaluate to which extend the tools can be adapted to learning and new uses

4. Implementation n Often more complicated feedback and network delays Ø architectures for groupware Ø feedthrough and network traffic Ø purchase, installation and maintenance of multiple applications Ø

Feedback and network delays local machine screen feedback 9 8 network 7 remote machine remote application 6 5 user types n n 1 2 client 3 4 server At least 2 network messages + four context switches With protocols 4 or more network messages

The implementation process n should have the objectives of (Andriessen, 2003): Ø developing a new system in line with the existing organizational structure, culture and processes Ø learning to work with and to manage the new system: training and preparation Ø developing acceptance of the innovation, thereby preventing or reducing resistance to changes

Adoption and Acceptance n Groupware systems cannot be successful unless a critical mass of users chooses to use the system Ø n E. g. a videophone is useless if you're the only one who has one most common reasons for failing to achieve critical Ø lack of interoperability n n Ø In the 90 s, AT&T and MCI both introduced videophones, but their two systems couldn't communicate with each other. This lack of interoperability/compatibility meant that anyone who wanted to buy a videophone had to make sure that everyone they wanted to talk to would buy the same system Compatibility issues lead to general wariness among customers, who want to wait until a clear standard has emerged lack of appropriate individual benefit n n Disparity individual/group benefit: discussed in game theory as the prisoner's dilemma or the commons problem To solve this problem, some groups can apply social pressure to enforce groupware use (as in having the boss insist that it's used), but otherwise it's a problem for the groupware designer who must find a way to make sure the application is perceived as useful for individuals even outside the context of full group adoption

If not… n Failure of implementing an EES (Employee Evaluation System supporting the personnel management function) at a Chemical Company n Differences between Ø Ø n actual culture (real culture of the organization) presumed culture (the culture designers assumed the organization have) The design company went bankrupted because of this

Presumed culture (designers) 1. innovation and action orientation in order to achieve maximal profitability 2. 2. Risk taking encouraged 3. 3. management is centralized and formal horizontal communication prevail. Actual culture (company) 1. CC afford to act conservatively because of the favorable environment 2. EES resisted because was perceived as an innovative action. 2. CC could maintain success without risk taking; CC managers believed in “playing safely” EES was resisted because was perceived as a danger to stable labor relations 3. CC had mainly informal communication and plant managers were semi-independent from the top management EES was resisted because information sharing and EES formalities were not desired within the plants, nor did they foster communication within CC top management

Why groupware fails (Grudin) § Organisational factors that influence the acceptance of groupware n Disparity in work and benefit Ø n Critical mass and Prisoner’s dilemma problems Ø n groupware may not enlist a critical mass of users to be useful or it may not be to any individual’s benefit to use Disruption of social processes Ø n additional work may be required from individuals who do not perceive a direct benefit it may violate social taboos or threaten existing power structures Exception handling Ø it may not support the wide range of exception handling and improvisation associated with group activity

Why groupware fails (Grudin) n Unobtrusive accessibility Ø n Difficulty of evaluation Ø n difficulty of generalisation makes it hard to learn from experience Failure of intuition Ø n features that support group activity are usually used less frequently and need to be unobtrusively integrated with other features intuition in product development is poor for multi-user applications The adoption process Ø groupware requires more careful introduction in the workplace

Some rules for success n Groupware is part of a social system Ø Ø n Design for several levels of interaction Ø n users and possibly other stakeholders should be part of the design process from the beginning Analyse carefully the situation of the users Ø n user friendly interaction, adequate interpersonal communication, group cooperation and organisational functioning Design in a participative way Ø n Design not for a tool as such but for a new socio-technical setting Groupware changes organisational culture. Plan for it! technology should match their skills and abilities, and also their attitudes, otherwise resistance is inevitable Analyse carefully the context Ø The more a new setting deviates from the existing one the more time, energy and other resources should be mobilised to make it a success

Some more rules for success n Introduce the new system carefully Ø Ø Ø Apply proper project management Pick a pilot project rather than major rollout Pick a project with a group that is supportive of technology and innovation Pick a project with visibility, and productivity or learning impact Pick groupware based on a specific problem that needs to be solved Listen to the people involved in pilot project n Train and support end-users extensively n Measure success conditions and success criteria before, during and after the development process Ø to cost-justify groupware implementation

Some more rules for success n Plan for a long process of introduction, incorporation, evaluation and adaptation n Groupware is not a quick fix – no single groupware product can do it all n Keep options open for new ways of working with the groupware, because this may result in creative and innovative processes Ø Don't be afraid to make changes – learn as you go n Make sure software you pick fits with existing systems or doesn’t compromise them n You can’t change people overnight – be prepared for resistance, make it fun n While people take time to change, organisations take longer n Groupware can be very political – need to strive for a win/win outcome

Werkcollege: assignment

Evaluating BSCW n Your group Ø n Print 2 copies of the questionnaire you created to evaluate BSCW During the werkcollege: Pass the questionnaire to the members of another group Ø Ø Each of them have to answer the questionnaire As a group: They have to assess the quality of your questionnaire n n Your group Ø n Does it cover all of the important issues? Are the questions properly done? Which ones are ambiguous? Compare with their on questionnaire Analyse the results of the 2 questionnaire in relation to BSCW In the werkcollege Ø Ø Discussion of findings in relation to BSCW Discussion about quality of questionnaires, difficulties creating it, measures for improvement… n Ø Groups interchange their assessment and ways to improve the tests Participate in the discussion!!!!