Introduction to User Studies Yale Braunstein November 1999
Three Types of Studies 1. Traditional studies of users and usage (often of libraries) 2. Studies that include Internet uses and similar measures 3. Studies of Digital Libraries and their users (“digital communities, ” just starting)
Traditional Studies
Basic Problems • • Is there a theoretical basis to the analysis? Are the results reproducible? What can be changed based on the findings? How do we know changes will help? Possible solution: - Develop theoretical approach early
Typical Steps in Social Science Research • State general nature of problem, question, relationship • Review previous studies • Create and state “stylized facts” • Develop testable hypotheses • Discover or develop data to be used to test hypotheses • Perform appropriate (statistical) tests • Describe results • Draw conclusions
Traditional Bibliometric Model of Scholarly Productivity Socio-economic factors Institutional factors Funding Research output
Revised Bibliometric Model of Scholarly Productivity Socio-economic factors Institutional factors Funding Computer skills/literacy Computer use / Internet use Research output
Revised Bibliometric Model of Scholarly Productivity Socio-economic factors Institutional factors Funding Research output Computer skills/literacy Computer use / Internet use Major issues: - What sample/universe to use? - How to measure inputs? - How to measure outputs? - Functional form? (linear? , additive? )
List of Selected Processes
Our main hypothesis in this research is that scholars' Internet-use data add explanatory power to models of scholarly productivity. The formal null and alternate hypotheses are: H 0: Internet use data does not add explanatory power to the Traditional Publication Model. H 1: Internet use data adds explanatory power to the Traditional Publication Model. We used multiple regressions to estimate the four models listed in Figure 5. Since we are interested in whether a set of one or more Internet-use variables “improves” the explanatory power of the traditional model, the appropriate measure is the F-statistic from a comparison of the restricted (traditional) model with the unrestricted (new) model.
The exact equation, as estimated in Equation 1, is: PUBAV = 17. 1 – 0. 313 AGE + 0. 197 PHD-AGE + 0. 0860 RL – 0. 0794 PIM 1 + 19. 6 (Login-FTP principal component) + 2. 99 (Kermit principal component).
Digital Libraries and Collaborative Knowledge Construction Based on a presentation by Nancy Van House September 15, 1999
Digital Libraries • Support knowledge work • Knowledge work occurs in material and social matrices • DLs mutually constituted with work, practices, tools, artifacts, knowledge communities
Evaluating DLs • Direct effects: how well they meet users’ needs & expectations • Higher-order effects: Understanding how DLs support and potentially change – Work – Its practices and artifacts – People/communities who do the work – Institutions that support it
Possible Theoretical Bases for DL Evaluation • Situated action – work and learning • Science studies – knowledge creation – knowledge communities • Actor-Network Theory – knowledge creation and communities – stabilization of socio-technical systems
Characteristics of Knowledge Work • Situated in place and time • Distributed x people, places, time • Social - collectively decide what is known, how we determine what is known, how we do the work
The UC Berkeley Digital Library Project • Supporting environmental planning • Innovative tools • Large, diverse testbed
This Research • To better understand distributed, collaborative cognitive work, the role of information and information practices and artifacts, and the potential effects of digital information • To investigated the practices of environmental planning, particularly information use and production
This Study • Not about the UC Berkeley DL but about potential of DLs in general, and current issues around sharing of digital data. • Method: Interviews
Knowledge Communities • Two engaged in work related to environmental planning – Water planning – Botanical • Commonalities – Multiple scientific (mostly) disciplines – Various organizational settings – Large datasets of observations and summaries and analyses • collected by many, over time, differing methods – Potentially major economic impact – Often highly political
Findings • Contributions • Use • Cooperation in creating and operating DL
Findings: Barriers to Contribution • Misuse of data – People who ‘don’t understand’ – “Using our data against us’ • ‘Productizing’ – Burden – Mismatched incentives – Making invisible work visible
Findings: Use • User must be willing to trust both content and functionality of DL • Assessing trustability of contents – Assessing sources • Ability and training • Interests – DL’s filtering policies and processes • DL’s functionality – E. g. , search tools, GIS overlays
Findings: Cooperation in Creating and Operating DL • Both a research project and active testbed • Groups to interest: computer scientists and applications area specialists • Cutting edge vs reliable, incremental improvement over past • Differences in decision-making styles: hierarchical, planned vs distributed, emergent • Articulation work
Possible Theoretical Bases for DL Evaluation • Situated action – work and learning • Science studies – knowledge creation – knowledge communities • Actor-Network Theory – knowledge creation and communities – stabilization of sociotechnical systems
Two Broad Approaches to Social Theory • Reductionist: economics, behavioral psychology, structural-functional sociology • Irreductionist: ethnomethodology (Garfinkel), practice/structuration theory (Giddens, Bourdieu, Lave)
Irreductionist Approaches • Social order/institutions not pre-given but continually recreated through the concrete, day-to -day activities Interest in processes by which people construct meaning and re/produce social order • Interest in processes by which this done • Emphasis on practices and artifacts which shape understanding and carry it across time and space
Situated Action, Situated Learning • Knowledge not inert substance transferred from teacher to student • Knowledge and learning on-going construction of meaning through action and interaction • Knowing not activity of single mind but complex social and material phenomenon, “nexus of relations between the mind and work and the work in which it works” (Lave, 1988).
Communities of Practice: the Social Matrix of Knowledge Work • People learn & work within groups who share understanding, practices, technology, artifacts, and language, e. g. , professions, workgroups, and disciplines. • A person typically belongs to multiple communities of practice • Important task of communities of practice: – deciding what is known; – the processes and principles by which knowledge claims are evaluated; – who is entitled to participate in the discussion; – whom to believe.
Information Artifacts • Include texts, images, maps, databases, thesauri, classification systems… • Not simply reflections of knowledge but instrumental in creating it, coordinating work across space and time • Tightly bound up with practices and communities
Science and Technology Studies • Science Studies – practices of scientific work – how the participants come to an agreement about what counts as fact or discovery – what inferences are made from facts – what is regarded as rational or proper conduct – how credibility of claims is assessed • Science and Technology Studies (STS) – Increased emphasis on technology, esp’ly large sociotechnical systems • Science as prototype of rational knowing > applicable to knowledge work generally
Approaches to Research • Sci & tech knowledge neither purely natural nor social • Attention to processes of sci knowledge production • Methodological orientation – “Follow the actor” • Attention to activities • Not deciding ahead of time where to draw boundaries – Funding as part of scientific activity – Inscriptions play key role in sci knowledge production • E. g. , labs as producing papers • Carrying work across space and time
Actor-Network Theory (Bruno Latour, Michel Callon, John Law) • Not a theory but a “method of looking at the actors’ own world building activities” • Doesn’t take the stability of sociotechnical systems as given, but asks how they are created and maintained. • “Organization is an achievement, a process. . . Its components -- hierarchies, organizational arrangements, power relations, flows of information -- are uncertain consequences of the ordering of heterogeneous materials. . ANT is thus a theory of agency, of knowledge, and of machines. ” (Law in systems science (Emphasis added)).
Key Concepts of ANT • • • Action-at-a-distance The actor-network Translation Black-boxing Intermediaries
Key Concepts of ANT • Action-at-a-distance is the problem of coordinating and sustaining collective activities over space and time). • The actor-network “is most simply defined as any collection of human, non-human, and hybrid human/non-human actors who jointly participate in some organized (and identifiable) collective activity in some fashion for some period of time. ” – Perhaps the most radical contribution of ANT is the inclusion of the non-human in its “heterogeneous networks. ”
Key Concepts, cont. • Translation is a key process by which actornetworks are created and stabilized, however temporarily. Actors’ disparate interests get translated into a set of interests that coincide in the network. • Black-boxing is a process of closing questions and debates. • An intermediary is an actor (of any type) that stands between two others and translates between them in such a way that their interaction can be more effectively articulated. – Inscriptions, Latour’s “immutable, combinable mobiles: ”
Approaches to Research of Situated Action, Science Studies, ANT • Contention that sci & tech knowledge neither purely nature nor purely social • Methodological orientation: – “Follow the actor” (ethnography) – Inscriptions
Applied to our DL • Applications area: environmental planning • The DL as a large sociotechnical systems • Method
Implications: Joining Findings & Theory I. DLs and Knowledge Productions • DLs support knowledge work, which is situated, distributed, and social • DL ‘evaluated’ according to its relationship with knowledge work and its social and material matrix
Users’ and Providers’ Concerns – the DL and Knowledge Production • Fear of misuse of info, inability to evaluate credibility = crossing boundaries of communities and assemblages within which/for which created • Practices of (1) work and (2) assessing/demo’ing credibility undermined by – Ready crossing of sociotechnical boundaries – Fluid recombination of inscriptions – Opening of black boxes, e. g. document, methods of work – Closing of others, e. g. analytical and search tools
Implications: Joining Findings & Theory II. The DL as a Sociotechnical System • Successful DL enrolls dynamic network of users, sources (human and nonhuman), builders, and technology • Requires on-going enrollment and multiple translations • Stabilization not equilibrium but on-going achievement
Implications for DL Design • Articulate with existing assemblage of practices, artifacts, participants – Attention to existing and emergent practices of trust and credibility • Design specific to communities and tasks – Tension of specificity and generality • Include high-order user involvement • Design to be emergent and dynamic – on-going enrollment and co-constitution
Implications for DL Evaluation • Not just how well it meets users’ needs but how co-constituted with assemblages of work • Target to specific user communities and tasks • Consider not just service delivery but stabilization of DL as a heterogeneous network
Evaluation Methods • Ethnographic – “Follow the actor” – Study activities in natural settings in which they occur – Focus on what people actually do, not simply their accounts of behavior • Varied – Different methods appropriate for different issues
Conclusion • Understanding knowledge work not simple • Beginning to have an analytical approach from which to better understand KW and DLs • Applies to other areas of electronic collection, delivery, manipulation of information where trust and credibility are important – E. g. , knowledge management
Скачать презентацию Introduction to User Studies Yale Braunstein November 1999
bcf3141ffc31857919b43aada600af49.ppt