Global Developments in Assessment: Lib. QUAL+® and Beyond… Duane Webster Executive Director Association of Research Libraries Selena Lock Research and Development Cranfield University 6 th International JISC/CNI Conference York, U. K. July 6 -7, 2006 ARL Duane Webster, Executive Director www. arl. org
Library Assessment in an Electronic Era What are some of the current developments with library assessments efforts? ARL Stats. QUAL™ E-Metrics Lib. QUAL+® Digi. QUAL™ MINES for Libraries™ Where are the most critical needs and opportunities? What are the lessons learned? Duane Webster, Executive Director www. arl. org
Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication. Members: 123 major research libraries in North America. Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources. Users: Three million students and faculty served. Expenditures: $3. 4 billion annually, $1. 1 billion for acquisitions of which 31 percent is invested in access to electronic resources. Duane Webster, Executive Director www. arl. org
Thinking Strategically About Library Futures • What is the central work of the library and how can we do more, differently, and at less cost? • What important set of services does the library provide that others can’t? What new roles are needed? • What advantages does the research library possess? • What will be the most needed by our community of users in the next decade? How is user behavior changing? • What should our libraries aspire to be ten years from now? What are the implications of technology driven change? • What are the essential factors responsible for the success of the library? Duane Webster, Executive Director www. arl. org
Defining Success in a Digital Environment • Crafting new measures of success. • Moving from measuring inputs to outputs. • Understanding impact of library roles and services. • Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals. • Reallocating and managing capabilities to focus on new definitions of success. Duane Webster, Executive Director www. arl. org
ARL Statistics and Measurement Duane Webster, Executive Director www. arl. org/stats/
Stats. QUAL™ . Duane Webster, Executive Director www. statsqual. org
Stats. QUAL™ Duane Webster, Executive Director www. statsqual. org
Updating the Traditional ARL Statistics • E-Metrics = ARL Supplementary Statistics – On going efforts to update and refine core data. – Exploring feasibility of collecting e-metrics. • ARL Task Force on New Ways of Measuring Collections : – Growing concern with utility of membership index. – Study ARL statistics to determine relevance. – Develop Profile of Emerging Research Libraries. Duane Webster, Executive Director www. statsqual. org
E-Metrics: Recommended Statistics Patron Accessible Electronic Resources (R 1 -3) Use of Networked Resources and Services (U 1 -5) Expenditures for Networked Resources and Related Infrastructure (C 1 -3) Library Digitization Activities (D 1 -3) Duane Webster, Executive Director www. arl. org/stats/newmeas/emetrics/index. html
E-Metrics: Recommended Statistics Patron Accessible Electronic Resources (R 1 -3) • R 1 – Number of electronic full-text journals • R 2 – Number of electronic reference sources • R 3 – Number of electronic books Duane Webster, Executive Director www. arl. org/stats/newmeas/emetrics/index. html
E-Metrics: Recommended Statistics Use of Networked Resources and Services (U 1 -5) • U 1 – Number of electronic reference transactions • U 2 – Number of logins (sessions) to electronic databases • U 3 – Number of queries (searches) in electronic databases • U 4 – Items requested in electronic databases • U 5 – Virtual visits to library’s website and catalog Duane Webster, Executive Director www. arl. org/stats/newmeas/emetrics/index. html
E-Metrics: Recommended Statistics Expenditures for Networked Resources and Related Infrastructure (C 1 -3) • C 1 Cost of electronic full-text journals • C 2 Cost of electronic reference sources • C 3 Cost of electronic books • C 4 Library expenditures for bibliographic utilities, networks and consortia • C 5 External expenditures for bibliographic utilities, networks and consortia Duane Webster, Executive Director www. arl. org/stats/newmeas/emetrics/index. html
Library Digitization Activities (D 1 -3) • D 1 – Size of library digital collection • D 2 – Use of library digital collection • D 3 – Cost of digital collection construction and management data requires staff familiar with the digital (Collecting these environment. ) Duane Webster, Executive Director www. arl. org/stats/newmeas/emetrics/index. html
A Lib. QUAL+® Update • The Lib. QUAL+® premise, dimensions, and methodology • Lib. QUAL+® results • Lib. QUAL+ in action Duane Webster, Executive Director www. libqual. org
A Lib. QUAL+® Update Lib. QUAL+™ Presented by: Selena Lock Cranfield University 6 th International JISC/CNI Conference York, U. K. July 6 -7, 2006 http: //www. libqual. org
A Lib. QUAL+® Update § The Lib. QUAL+® premise, dimensions, and methodology § Lib. QUAL+® results § Lib. QUAL+® in action http: //www. libqual. org
Lib. QUAL+™ The Lib. QUAL+® premise, dimensions, and methodology http: //www. libqual. org
The need for Lib. QUAL+® § Underlying need to demonstrate our worth § The reallocation of resources from traditional services and functions § Rapid shifts in information-seeking behavior • Need to keep abreast of customer demands § Increasing user demands • 37% of UK 16 – 18 year olds expect better libraries in return for their top-up fees http: //www. libqual. org
The Lib. QUAL+® Premise PERCEPTIONS SERVICE “…. only customers judge quality; all other judgments are essentially irrelevant” Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press. http: //www. libqual. org
Multiple Methods of Listening to Customers § § § Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture *A SERVQUAL-type instrument is most suitable for these methods Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D. C. http: //www. libqual. org
Lib. QUAL+® Development § An ARL/Texas A&M University joint developmental effort based on SERVQUAL § Lib. QUAL+® initially supported by a 3 -year grant from the U. S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) § Initial project established a expert team, regrounded SERVQUAL concepts, and designed survey methodology § Survey conducted at over 700 libraries resulting in a data base of over a million user responses http: //www. libqual. org
13 Libraries English Lib. QUAL+® Version 4000 Respondents Emergent 2000 QUAL PURPOSE Describe library environment; build theory of library service quality from user perspective Lib. QUAL+® Project DATA Unstructured interviews at 8 ARL institutions ANALYSIS Content analysis: (cards & Atlas TI) PRODUCT/RESULT Case studies 1 Valid Lib. QUAL+® protocol QUAN Test Lib. QUAL+® instrument Web-delivered survey Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2 QUAL Refine theory of service quality Unstructured interviews at Health Sciences and the Content analysis Smithsonian libraries Cultural perspective 3 QUAL Refine Lib. QUAL+® instrument E-mail to survey administrators Content analysis Refined survey delivery process and theory of service quality 4 QUAN Test Lib. QUAL+® instrument Web-delivered survey Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Refined Lib. QUAL+® instrument 5 QUAL Refine theory Focus groups Content analysis Iterative Vignette Re-tooling 2005 700 Libraries English, Dutch, Swedish, German Lib. QUAL+® Versions 160, 000 anticipated respondents http: //www. libqual. org Local contextual understanding of Lib. QUAL+® survey responses 6
76 Interviews Conducted § § § § York University § University of Arizona State University of Connecticut § § University of Houston § University of Kansas University of Minnesota University of Pennsylvania University of Washington Smithsonian Northwestern Medical http: //www. libqual. org
http: //www. libqual. org Loaded. PT: P 1: 01 xxxxxxx. txt, S: AdminColleenServ. Qual InterviewsTEXT Only 1 xxxxx. txt
http: //www. libqual. org
Dimensions of Library Service Quality Affect of Service Information Control Empathy Scope of Content Responsiveness Assurance Reliability Library as Place Utilitarian Space Symbol Refuge http: //www. libqual. org Model 3 Convenience Ease of Navigation Timeliness Equipment Self-Reliance
Lib. QUAL+® Participants http: //www. libqual. org
Rapid Growth in Other Areas § Languages • American English • British English • French • Dutch • Swedish § In development • Chinese • Greek • Spanish • German § Consortia • Each may create 5 local questions to add to their survey § Types of Institutions • Academic Health Sciences • Academic Law • Academic Military • College or University • Community College • European Business • Hospital • Public • State § Countries • U. S. , U. K. , Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia http: //www. libqual. org
World Lib. QUAL+® Survey 2005 http: //www. libqual. org Participating Libraries
Lib. QUAL+® Languages Over 700 institutions 1, 000 respondents http: //www. libqual. org
Survey Instrument – “ 22 items… http: //www. libqual. org
…and a Box” § Why the Box is so Important: • About 40% of participants provide open-ended comments, and these are linked to demographics and quantitative data • Users elaborate the details of their concerns • Users feel the need to be constructive in their criticisms, and offer specific suggestions for action http: //www. libqual. org
Lib. QUAL+™ Lib. QUAL+® results http: //www. libqual. org
Understanding Lib. QUAL+® Results § For the 22 items Lib. QUAL+ asks users’ to rate their: • Minimum service level • Desired service level • Perceived service performance § This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings § Perception ratings ideally fall within the Zone of Tolerance http: //www. libqual. org
Key to Bar Charts Key to Radar Charts http: //www. libqual. org
Lib. QUAL+® 2005 Summary Colleges or Universities (American English) http: //www. libqual. org
Lib. QUAL+® 2005 Summary Colleges or Universities (British English) http: //www. libqual. org
Longitudinal Analysis for Colleges or Universities (American English) http: //www. libqual. org
Longitudinal Analysis for Colleges or Universities (British English) http: //www. libqual. org
Comparisons by user group College or University (British English) http: //www. libqual. org
General findings § Highly desired • Making electronic resources accessible from my home or office • Print and/or electronic journals I require for my work • A haven for study, learning or research § Lowest • Library staff who instil confidence in users • Giving users individual attention • Space for group learning and group study http: //www. libqual. org
Lib. QUAL+™ Lib. QUAL+® in Action http: //www. libqual. org
UK University Case Study – Using Lib. QUAL+® Results § Strategic Service Developments • Data to support service development • Ability to identify where not meeting expectations • Measure if change has met need § Budget Discussions • Data to support bid for increased funding • Data to support case for change in emphasis (towards e-provision) § Marketing Position • Status of the library within the University • Importance of national & international benchmarking http: //www. libqual. org
Lib. QUAL+® Outcomes Funding received for: § New Web Services Administrator § Increased opening hours • Now providing 222, 578 seat hours per week § Library refurbishment programme reinstated at costs in excess of £ 8 million http: //www. libqual. org
Library Refurbishment From: To: http: //www. libqual. org
Why use Lib. QUAL+®? Feedback from Lib. QUAL+® Users “Why did you choose to use Lib. QUAL+®? ” § Lib. QUAL+® was recommended to us as offering a well designed, thoroughly Library-focused set of survey tools § Opportunity to benchmark § Cost-effectiveness § Automated processing & fast delivery of results § Respectability and comparability (with others and historically) http: //www. libqual. org
The benefits of Lib. QUAL+® has enabled us to find out what a broad range of our users thought of the services we offer; what level of servicedelivery quality we had achieved in their eyes, and to get a clear picture of what they actually wanted the Library to deliver (as opposed to what we thought they wanted). UK HE Institution, 2006 http: //www. libqual. org
In Closing Lib. QUAL+® § Focuses on success from the users’ point of view (outcomes) § Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense § Requires limited local survey expertise and resources § Analysis available at local, national and inter-institutional levels § Offers opportunities for highlighting and improving your status within the institution § Can help in securing funding for the Library http: //www. libqual. org
Lib. QUAL+® Resources § Lib. QUAL+® Website: http: //www. libqual. org § Publications: http: //www. libqual. org/publications § Events and Training: http: //www. libqual. org/events § Lib. QUAL+® Bibliography: http: //www. coe. tamu. edu/~bthompson/servqbib § Lib. QUAL+® Procedures Manual: http: //www. libqual. org/Information/Manual/index. cfm http: //www. libqual. org
Acknowledgements § Fred Heath, Bruce Thompson, Colleen Cook, Martha Kyrillidou and the rest of the Lib. QUAL+® team § Stephen Town, Cranfield University § Helen Durndell and Jacqui Dowd, Glasgow University § All SCONUL Lib. QUAL+® Participants and the SCONUL Working Group on Performance Improvement http: //www. libqual. org
Lib. QUAL+® Contact Information § Martha Kyrillidou • Director, ARL Statistics and Measurement Program • martha@arl. org § Selena Lock • Research and Development, Cranfield University • s. a. lock@cranfield. ac. uk http: //www. libqual. org
Digi. QUAL™ Developing the Digi. QUAL™ Protocol for Digital Library Evaluation • NSF Funding • Building on the Lib. QUAL+® experience • Secures feedback on user’s perceptions of library’s web site • Five questions on services, functionality, and content • Goal is to determine utility, reliability, and trustworthiness Duane Webster, Executive Director www. digiqual. org
Developing Digi. QUAL™ Survey Items Background: Serv. QUAL Lib. QUAL+® Digi. QUAL™ Lib. QUAL+® Dimensions of Service Quality: Digi. QUAL™ 12 themes of service quality: • Affect of Service • Information Control • Library as Place • • • Duane Webster, Executive Director Accessibility Navigability Interoperability Collection building Resource Use Evaluating collections DL as community for users DL as community for developers DL as community for reviewers Copyright Role of Federations DL Sustainability www. digiqual. org
Pilot Testing Survey Items and Implementation Duane Webster, Executive Director www. digiqual. org
Building a Survey • Review and select items • Issue: aligning items to individual DL needs & users – vocabulary and content Duane Webster, Executive Director www. digiqual. org
Building a Survey, cont. Customize Survey Issue: Flexibility vs. Standardization Duane Webster, Executive Director www. digiqual. org
Implementing Survey – Notification Methods Links on site Newsletters Next to resources Issues: no pop-ups, no individual emails Duane Webster, Executive Director www. digiqual. org
Implementing Survey Incentives Issues: must be easily transferable, requires email address - clear IRB Duane Webster, Executive Director www. digiqual. org
Analysis and Reporting Issues (mis)Interpreting results from individual digital libraries in the context of other sites Sites reluctant to share data and results Duane Webster, Executive Director www. digiqual. org
Outstanding Issues and Challenges • Unique DLs: niche market, critical mass, both? • Balance: – custom vs. generic content results – flexible vs. standard implementation scaling • Mixed methods – Preserving user privacy – Collecting truly useful data • Moving target: digital libraries as… it depends. Duane Webster, Executive Director www. digiqual. org
Assessing the Value of Networked Electronic Services: The MINES survey Measuring the Impact of Networked Electronic Services (MINES) MINES for Libraries™ Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
What is MINES? • A research methodology consisting of a web-based survey form and a sampling plan. • Measures who is using electronic resources, where users are located at the time of use, and their purpose of use. • Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003. • Different from other electronic resource usage measures that quantify total usage (e. g. , COUNTER, EQUINOX, EMetrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (Lib. QUAL+®, Digi. QUAL™). Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Questions Addressed • How extensively do sponsored researchers use the new digital information environment? • Are researchers more likely to use networked electronic resources from inside or outside the library? • Are there differences in usage of electronic information based on the user’s location (e. g. , in the library; oncampus, but not in the library; or off-campus)? • What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys? • Are particular network configurations more conducive to studies of digital libraries patron use? Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
MINES Methodological Considerations • A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library. • Random moment/web-based surveys are employed at each site. • Participation is usually mandatory, negating nonrespondent bias, and is based on actual use in real-time. • Libraries with database-to-web gateways or proxy rewriters offer the most comprehensive networking solution for surveying all networked services users during survey periods. Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Web Survey Design Guidelines • Simple text for different browsers – no graphics. – Different browsers render web pages differently. • Few questions per screen or simply few questions. • Easy to navigate. • Short and plain. • No scrolling. • Clear and encouraging error or warning messages. • Every question answered in a similar way - consistent. – Radio buttons, drop downs. • ADA compliant. • Introduction page or paragraph. • Easy to read. – Must see definitions of sponsored research. • Can present questions in response to answers. Dillman, D. A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method. Duane Webster, Executive Director 2 nd Ed. New York: John Wiley & Sons. www. arl. org/stats/newmeas/mines. html
Library User Survey Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Recent Data Collection Activities Data was collected at seven main campus libraries and seven academic health science libraries in the U. S. , between January, 2003 and January, 2005. Main University Libraries Academic Medical Libraries University of Colorado University of Connecticut Health Center University of Connecticut University of North Carolina University of Texas Medical Branch Oregon State University of Texas Southwestern University of Utah University of Virginia Washington University Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Recent Data Collection Activities • More than 45, 000 networked electronic services uses were surveyed. • At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities. • MINES is also being conducted by the Ontario Council of University Libraries (OCUL) in 2004 -2005 in conjunction with ARL. Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Purpose of Use By Location Main Campus Libraries 2003 – 2005 In the Library On-Campus, not in the Library n = 7, 618 n = 6, 641 26% 14% 2% 3% 5% 66% 21% 63% All Usage Off-Campus n = 5, 012 n = 19, 271 33% 24% 2% 6% 59% 2% 11% 63% • 72% of sponsored research usage of electronic resources occurred outside the library; 83% of this took place on campus. Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Location of Users 2003 -2005 Medical Library 76% Main Library 60% 24% In Library Users 40% Remote Users All Libraries 70% 30% Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Demographics by Location of User Main Libraries Inside the Library On Campus, Not in the Library n = 7, 064 n = 6, 391 8% 20% 27% 45% Graduate Students Faculty, Staff, Research Fellows Undergraduate Students All Other Users 24% 39% 40% 33% Total Users Off-Campus n = 4, 953 3% n = 18, 408 27% 38% In the Library 12% 22% Duane Webster, Executive Director Outside the Library 62% www. arl. org/stats/newmeas/mines. html
Electronic Services Sponsored Research Use Compared to Print Journal and Total Library Use Main Libraries E-Resources Print Journals Total Use Duane Webster, Executive Director www. arl. org/stats/newmeas/mines. html
Where are the most critical assessment needs and opportunities? • Complementing Lib. QUAL+® with additional measures. • Developing impact studies on user success, economic value, and community return on investment. • Moving target: what is a digital library? • E-Resources: understanding usage. • Gaining acceptance and use of standard measures for e-resources. • Building a climate of assessment throughout library. Duane Webster, Executive Director
What are the lessons learned? • Understanding changes in users approach to information resources. • Service quality improvement is a key factor. • Understanding the impact of e-resources on library services TRL. • Learning how to compete with Google. • Upfront investment in design and development. • Making the assessment service affordable, practical, & effective. • Assessment needs to be satisfying and fun. Duane Webster, Executive Director
In Closing • As higher education is challenged on accountability and effectiveness issues so will libraries. • A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work. • Basic questions of role, vision, and impact must be answered by library community. Duane Webster, Executive Director
Selected References • Kyrillidou, Martha and Sarah Giersch. “Developing the Digi. QUAL Protocol for Digital Library Evaluation. ” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6 -11, 2005. [Available at http: //www. libqual. org/documents/admin/digiqual-jcdl 05 -v 5. pdf] • Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report. ” Presented at the 6 th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http: //www. libqual. org/documents/admin/FINAL%20 REPORT_Jan 26 mk. pdf] • Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http: //Information. R. net/ir/9 -4/paper 187. html] • Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services. " ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20 -21. [Available at http: //www. arl. org/newsltr/230/usage. html]. • Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) Lib. QUAL+™ Protocol: An E-service for Assessing the Library of the 21 st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2 -3, 2003. [Available at http: //www. libqual. org/documents/admin/NSDL_workshop_web 1. pdf] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections. ” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3 -6, 2004. [Available at http: //www. libqual. org/documents/admin/MERLOT%20 Paper 2_final. pdf] • Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and Lib. QUAL+™ Data Bases. ” 7 th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6 -8, 2005. [Available at http: //www. libqual. org/documents/admin/ISKO. PDF] Duane Webster, Executive Director
Скачать презентацию Global Developments in Assessment Lib QUAL and Beyond
5a66b0eb98b1117b969a6b85c923300c.ppt