Old and New Measures … Why Bother? Georgetown University Library February 24, 2003 • Washington, DC Martha Kyrillidou Senior Program Officer for Statistics and Measurement Consuella Askew Lib. QUAL+™ Program Officer Association of Research Libraries www. arl. org
Association of Research Libraries • Mission: Shaping and influencing factors affecting the future of Research Libraries in the process of scholarly communication • Members: 124 major research libraries in North America • Users: 3 million students and faculty served • Service measures: 73 million circulations 18. 5 million reference transactions 108 thousand group training lessons 1. 6 million participants Association of Research Libraries www. arl. org
ARL Strategic Objectives • 8 including federal relations, scholarly communication, preservation, collections … and • PERFORMANCE MEASURES To describe and measure the performance of research libraries and their contribution to teaching, research, learning and community service Association of Research Libraries www. arl. org
ARL Statistics and Trends • ARL Statistics: Serial and Monograph Costs Interlibrary Loan/Document Delivery Expenditures in ARL Libraries Expenditures for Electronic Resources • ARL Annual Salary Survey Salaries and Human Resources Issues • Preservation Statistics • ARL Law and Medical Library Statistics • Lib. QUAL+ • E-Metrics Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Association of Research Libraries www. arl. org
Georgetown Libraries, 2000 -01 Volumes Held Volumes Added, Gross Current Serials Total Expenditures Total Prof. & Support Staff 1, 612, 448 51, 126 12, 452 11, 552, 918 118 Law 494, 266 22, 970 11, 755 6, 797, 430 66 Med 162, 165 2, 925 1, 524 1, 996, 448 20 2, 268, 879 77, 021 25, 731 20, 346, 796 204 Main Total Association of Research Libraries www. arl. org
Building the Future Main Campus Strategic Plan: 2001 -2006 • Continue library enhancement so that collection development can further support the scholarship and teaching activities of faculty and students. • Integrate technology support for research and teaching of the Library and UIS. • Expand funding for the digital library – Source: http: //provost. georgetown. edu/documents/implement_plan. html Association of Research Libraries www. arl. org
Building the future • What do you do to build the future? • Why is what you are doing relevant to the future? • How do you use the evidence you collect from your everyday transactions to define the future? Association of Research Libraries www. arl. org
What data do you collect? What do you collect? Why do you collect it? Association of Research Libraries www. arl. org How do you use it?
No Gluttony Measures Measure what is important, not just what is measurable because What you measure is what you will pay attention to and work toward Association of Research Libraries www. arl. org
Measures that Matter • Input Outcome Impact • Consistent with organizational mission, goals and objectives • Integration with program review process • Balance customer, stakeholder, and employee interests and needs • Establish accountability • Collection and use of reliable and valid data Association of Research Libraries www. arl. org
Opportunities and Pressures • Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution • Increasing pressure to maximize use of resources through benchmarking resulting in: – Cost savings – Reallocation Association of Research Libraries www. arl. org
Reflections … Educational institutions today face new and significant challenges stemming from disruptions of financial markets, introduction of new technologies, demands for greater efficiency, and unprecedented requirements for investment in faculty, research, and infrastructure Some Early Reflections on TIAA-CREF by Herbert M. Allison (February 2003) Association of Research Libraries www. arl. org
Why New Measures for ARL • Increased customer and stakeholder expectations for services and responsiveness • Greater demands for accountability • Exploding growth in use and applications of technology • Increasing competition for resources • Need for use of reliable and valid data – Benchmarking and best practice – Over time Association of Research Libraries www. arl. org
New Measure How does a library answer the question, Do We Make a Difference? Association of Research Libraries www. arl. org
Evaluation team representing the Middle States Commission on Higher Education A number of areas emerged from an overall observation of the many recommendations in the student section: a concern for diversity issues, desire for increased faculty/student interaction, need for additional space for student use to promote greater campus community, increased funding and better services for graduate students, and improved use of data from outcomes assessment for planning. Report to the Faculty, Administration, Trustees, Students of GEORGETOWN UNIVERSITY Washington, DC 20057 by. An evaluation team representing the Middle States Commission on Higher Education <http: //provost. georgetown. edu/documents/evaluation. html> Association of Research Libraries www. arl. org
The Challenge “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable—i. e. , how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two” Sarah Pritchard Association of Research Libraries www. arl. org
ARL New Measures Initiative • Collaboration among member leaders with strong interest in this area • Retreat setting to define topics • Specific projects developed with different models for exploration • Individual library may take the lead • Intent to make resulting tools and methodologies available to full membership and wider community Association of Research Libraries www. arl. org
Development of Issue Papers (1) Tuscon, January 1999 • • • a) include an outcomes based definition b) identify the need for the topic c) provide some initial thoughts on data d) identify what data is already collected e) consider whether should be a part of ARL f) recommendation for additional research g) note whether topic is long- or short-term; h) recommend possible experts i) prepare a call for pilot projects Association of Research Libraries www. arl. org
Development of Issue Papers (2) Tuscon, January 1999 • • • Ease and Breadth of Access User Satisfaction Library Impact on Teaching and Learning Library Impact on Research Cost Effectiveness of Library Operations and Services • Space and Facilities • Market Penetration • Organizational Capacity Source: <http: //www. arl. org/stats/newmeas/nmbackground. html> Association of Research Libraries www. arl. org
ARL New Measures Projects • Investigation of role libraries can play in higher education outcomes: (a) learning (b) research • Assessing Information Literacy Outcomes (SAILS) • ILL/DD study to confirm changes in operations that lead to best practices • Project to define usage measures for electronic information resources (E-Metrics) • Demonstration project for service effectiveness measures (Lib. QUAL+) • Service Quality perceptions and expectations for the digital library (e-QUAL) Association of Research Libraries www. arl. org
Learning Outcomes Brief History • Engage project consultant with experience in campus assessment activities • Supported with funds from interested members • Small group of leaders engaged in discussion • Preparation of strategy paper for discussion by larger ARL community in May 2000 • Kenneth R. Smith ‘New Roles and Responsibilities for the University Library: Advancing Student Learning Through Outcomes Assessment’ • Summer 2001: Working Group Association of Research Libraries www. arl. org
Learning Outcomes Working Group (1) Summer 2001: Establishment of Learning Outcomes Working Group – Survey accreditation agencies to identify best practice institutions – Compile a list of learning outcomes being used by ARL institutions academic departments and general education requirements (if they exist) – Contact professional associations to identify any that measure learning – Identify offerings that libraries can develop to meet the outcomes identified Association of Research Libraries www. arl. org
Learning Outcomes Working Group(2) – Establish a plan for how to take the development of offerings to the next stage with faculty in a select group of institutions – Explore collaboration with ACRL in offering training on information literacy skills within the ARL community – Define or create generalized tools for assessment – Define learning outcomes for the purpose of this ARL project and determine at what level the project should focus its attention (the student, course, program, department, etc. ) Association of Research Libraries www. arl. org
SAILS Brief History • Experience with SERVQUAL in many libraries over the last 10 years • Texas A&M SERVQUAL assessment • Meeting of interested ARL libraries (ALA Midwinter 2000) • Pilot with 12 ARL libraries (spring 2000) • External funding through FIPSE, U. S. Department of Education (September 2000) • Participation and endorsement by Big 12 Plus Consortium • 43 libraries participating spring 2001 • 164 institutions participating in spring 2002 including Ohio. LINK and AAHSL groups • 316 institutions participating in spring 2003 including NY 3 Rs, 21 British libraries, 1 Dutch, and three institutions experimenting with a French version of the instrument Association of Research Libraries www. arl. org
SAILS Brief History • • • Based on ACRL Standards 92 items developed 70% of ACRL learning outcomes covered Field testing at KSU two semesters; Oregon State Ohio Board of Regents collaborative grant with Bowling Green State University IMLS Grant Association of Research Libraries endorsement Advisory Council 10 libraries currently involved in a pilot Association of Research Libraries www. arl. org
ILL/DD Brief History • ARL/RLG interlibrary loan cost study: a joint effort of the Association of Research Libraries and the Research Libraries Group by Marilyn M. Roche (Washington, DC: Association of Research Libraries, 1993) • 1996 ARL ILL/DD Performance Measures Study – 97 research libraries and 22 Oberlin Group – cost, fill rate, turnaround time, and user satisfaction – Measuring the Performance of Interlibrary Loan Operations in North American Research & College Libraries by Mary Jackson (Washington, DC: Association of Research Libraries, 1998) • 2003 Assessing ILL/DD Services – 75 libraries – Replicate performance measures study – Add mediated and user-initiated (unmediated) component Association of Research Libraries www. arl. org
E-Metrics Brief History • ARL Supplementary Statistics tracking expenditures for electronic resources since 1993 • Facilitated retreat at Scottsdale in February 2000 • Contract with the Information Use and Management Policy Institute at Florida State University – Phase One: Environmental Scan – Phase Two: Proposed Measures and Testing – Phase Three: Training Modules • Measures for Electronic Resources (E-Metrics) by Wonsik ‘Jeff’ Shim, Charles Mc. Clure, and John Bertot (Washington, DC: Association of Research Libraries, 2002) • 2002 -2003 extended pilot with 39 libraries Association of Research Libraries www. arl. org
Project COUNTER • Both libraries and publishers need usage statistics • Counting Online Usage of Networked Electronic Resources (COUNTER) • Gain industry support and increase awareness • Release 1 of Code of Practice – Focus on journals and databases • Identify and define data elements to collect • Usage report content, format and delivery • Guidelines for data processing and auditing Association of Research Libraries www. arl. org
Project COUNTER • Both libraries and publishers need usage statistics • Counting Online Usage of Networked Electronic Resources (COUNTER) • Gain industry support and increase awareness • Release 1 of Code of Practice – Focus on journals and databases • Identify and define data elements to collect • Usage report content, format and delivery • Guidelines for data processing and auditing Association of Research Libraries www. arl. org
Lib. QUAL+ Brief History • • • Experience with SERVQUAL in many libraries past 15 years Texas A&M SERVQUAL assessment Meeting of interested ARL libraries (ALA Midwinter 2000) Pilot with 12 ARL libraries (spring 2000) External funding through FIPSE, U. S. Department of Education (September 2000) • 43 libraries participating spring 2001 (Big 12 Plus) • 164 institutions participating spring 2002 (Ohio. LINK, AAHSL) g • 316 institutions participating spring 2003 (NY 3 Rs, British, Dutch, French language version) Association of Research Libraries www. arl. org
Lib. QUAL+ Brief History • • • Experience with SERVQUAL in many libraries past 15 years Texas A&M SERVQUAL assessment Meeting of interested ARL libraries (ALA Midwinter 2000) Pilot with 12 ARL libraries (spring 2000) External funding through FIPSE, U. S. Department of Education (September 2000) • 43 libraries participating spring 2001 (Big 12 Plus) • 164 institutions participating spring 2002 (Ohio. LINK, AAHSL) g • 316 institutions participating spring 2003 (NY 3 Rs, British, Dutch, French language version) Association of Research Libraries www. arl. org
e-QUAL Brief History • National Science Digital Library Program at NSF • Building on the Lib. QUAL+ experience • NSF funding to modify Lib. QUAL+ for the digital library through NSDL (August 2001) • Participation in the NSDL evaluation workgroup • Understanding NSDL Core Integration Project and distributed collections environment • Qualitative Grounding ongoing • Development of a web-based tool for measuring user perceptions and expectations of the web, digital resource, digital collection, ? Association of Research Libraries www. arl. org
Future of New Measures Initiative • Incorporation of some new data elements in traditional surveys • Development of workshops and consulting services for performance measures • Establishment of data gathering and statistical analysis tools Association of Research Libraries www. arl. org
“Grand Unified Theory” Collaborative action + Commitment to Learning = Progress Association of Research Libraries www. arl. org
What data would support the library’s mission? The Georgetown University Library strives for excellence in acquiring, organizing, interpreting, preserving, and providing access to information resources and services that support the University's curricular and research activities. The Library achieves its mission by: • • Developing and preserving collections that support the curriculum and research interests of the main campus faculty and students • Organizing convenient and seamless access to print, electronic, and other resources to facilitate research and expand scholarship • Teaching patrons to use information resources and technologies effectively and promoting lifelong learning skills • Providing physical and networked environments that encourage study, scholarship, and collaborative learning • Retaining a staff committed to service excellence, continuous learning, and leadership within the library profession • Fostering collaborations throughout the University to create appropriate partnerships and maximize resources • Cultivating effective relationships with administrators, faculty, and students • Cooperating regionally, nationally and internationally with other libraries and consortia to acquire and share collections and resources with the scholarly community. Association of Research Libraries www. arl. org
What data should you collect? Association of Research Libraries www. arl. org
Self-assessment and value judgements Individual choice Association of Research Libraries Social choice www. arl. org
Measurement Tools • Focus Groups • Interviews – Structured – Unstructured • Site Visits • Observation – Obtrusive – Unobtrusive • Case Studies • Surveys • Critical Incident Technologies • User/Transaction Logs • Networked-Based Data Collection • Group Process Surveys • Scenario Development – Group – Individual – Transaction – User Association of Research Libraries www. arl. org
Methods of Assessing Students • Standardized tests – Pre – Post • Assignments – – Papers and essays Oral presentations Demonstrations Exhibitions • Capstone experiences • Surrogates – Grades/GPA – Self-reports – Interviews • Portfolios Association of Research Libraries www. arl. org
Assessment Challenges • • • Resources (i. e. , time and money) Buy-in Access to individuals to evaluate Expertise to conduct evaluation Project management experience Appropriate benchmarks Conceptual clarity Measurement & design requirements Instrument validity and reliability Association of Research Libraries www. arl. org
The Answer is in Building the Future Library effectiveness is becoming less elusive and increasingly recognizable and actionable … define the actions, the measures, and the future before others define them for you. Association of Research Libraries www. arl. org
Martha Kyrillidou, Senior Program Officer for Statistics and Measurement, martha@arl. org Consuella Askew, Lib. QUAL+™ Program Officer, consuella@arl. org Association of Research Libraries 21 Dupont Circle, Ste 800 Washington, DC 20036 202 -296 -2296 (phone) 202 -872 -0884 (fax) Association of Research Libraries www. arl. org
Скачать презентацию Old and New Measures Why Bother Georgetown
3d9b58790d120bb0030eb94a49d32f95.ppt