New measures for new services Indicators for quality

New measures for new services Indicators for quality, cost, and impact of electronic library services Roswitha Poll Münster LIDA 2007

The new electronic library services • catalogue databases • portals • online ordering and delivery • personalized services • online reference • digitized collections • electronic publishing • online self-paced training • Internet access via the library LIDA 2007

Reasons for evaluation of quality and impact accountability special funding needed: evidence of positive effects results-based budgeting justifying investment into change resource management rising expenses for new services increasing demand for new services new allocation of resources necessary promotion of the library‘s role communication of benefits by using library services LIDA 2007

What library standards want to achieve to spread the knowledge about existing tested methods and practices to standardize terminology and definitions methods of data collection methods of analyzing the data with the aim of comparing and aggregating results LIDA 2007

ISO Technical committee 46: Information and documentation Subcommittee 8: Quality, statistics and performance evaluation WG 2: International library statistics ISO 2789 (2006) International library statistics 4 th edition WG 4: Performance indicators for libraries ISO DIS 11620 (2006) Library performance indicators 2 nd edition WG 7: Quality measures for National Libraries ISO TR 281180 Performance indicators for National Libraries in preparation LIDA 2007

Stages of an ISO standard Approved New Work Item Working Draft Committee Draft (CD) Draft International Standard (DIS) voting required Final Draft International Standard (FDIS) Standard LIDA 2007

Problems of standardizing data for electronic services § quick changes in media and information ways § § changes not always predictable State of technical no clear boundaries of documents and development localities varies in libraries data must be collected from many sources (vendors, suppliers, institutional servers) § LIDA 2007

Assessing quantity (ISO 2789) In terms of • collection size • number and type of users • use of services • staff • space • equipment • training LIDA 2007

Electronic library services: counting use electronic collection website electronic document delivery OPAC problem of counting Internet access via the library can be counted manually online reference user training on electronic services LIDA 2007

ISO 2789 Electronic collection databases full text abstract and indexing computer files other electronic serials e. Books electronic patents digital documents networked AV-documents other LIDA 2007

ISO 2789 Forms of use sessions rejected sessions inside the library contents or records download session time searches outside the library, but inside the institution/ population Internet sessions virtual visits from outside the institution/ population LIDA 2007

ISO 2789: Definitions of electronic use session time search specific intellectual query session successful request of a database or the online catalogue rejected session unsuccessful request exceeding the simultaneous user limit period between log-in and implicit or explicit log-off content unit downloaded content unit being successfully requested record downloaded descriptive record being successfully requested Internet session Internet access by a user from inside the library LIDA 2007

ISO 2789: virtual visits a user's request on the website from outside the library premises web browser visit IP visit home page visit LIDA 2007

Counting electronic usage in German academic libraries: Percentage of libraries delivering data 2005 2006 OPAC sessions 45. 3 % 39. 8 % Database sessions 28. 0 % 31. 1 % E-journal sessions 25. 8 % 32. 0 % Digital document sessions 21. 8 % 17. 0 % Downloads from databases 7. 1 % 9. 2 % Downloads from e-journals 9. 3 % 13. 1 % Virtual visits 36. 0 % 34. 5 % LIDA 2007

Assessing quality (ISO 11620) In terms of • user-orientation • reliability • currency • usability/accessability • competence • responsiveness • speed • cost-effectiveness • perceived quality LIDA 2007

ISO 11620: Library performance indicators New structure: Balanced scorecard resources, access, infrastructure use efficiency potentials and development LIDA 2007

ISO 11620 Indicators for "hybrid" services resources, access, infrastructure • percentage of required titles in the collection • speed of interlibrary lending use • library visits per capita (including virtual visits) • percentage of population reached • user satisfaction efficiency potentials, development • cost per library visit • cost per user • correct answer fill rate • attendances at training lessons per staff member LIDA 2007

ISO 11620 Indicators for electronic services resources, access, infrastructure use efficiency potentials, development • percentage of rejected sessions • public access workstations per capita • number of content units downloaded per capita • percentage of information requests submitted electronically • workstation use rate • cost per database session • cost per content unit downloaded • percentage of acquisitions expenditure spent on the electronic collection • percentage of library staff providing and developing electronic services LIDA 2007

Possible performance indicator for website quality Direct access from the homepage Measured by • number of clicks neccessary to find relevant information • adequacy of terms used on the homepage Example: renewals Directly on the homepage: user account, my account Under broader terms: circulation, loan service LIDA 2007

Cost issues Performance indicators for costs ISO 11620 cost per loan cost per database session cost per content unit downloaded costs include • total recurrent expenditure costs include • contractual costs of a database costs include • contractual costs of an electronic resource LIDA 2007

Changing from print to electronic collections Cost issues What changes for the library? In use decrease of traditional services (loans, visits, copying) overall: increase in use increase of electronic use in costs less costs for storage, binding higher systems and staff costs better qualified staff needed lower cost per use LIDA 2007

Changing from print to electronic collections Cost issues What changes for the users? In use access from the working-place or home access at once 24 -hours access additional value searching options, links in costs academics spend half a day time saved ! per week on seeking and no ways, no fetching of materials, information procuring no waiting in queues LIDA 2007

Journals: cost per use Print only subscription costs 257. 737 € Example 1: Medical library Müster usage cost E-only usage per use subscription cost per use costs Users accessed online versions 10 times more than print versions 13. 795 18. 86 € 223. 647 € 64. 491 3. 47 € Only subscription costs are considered! LIDA 2007

Journals: cost per use Example 2: Drexel University Operational costs of print and electronic journals Cost type Space Print Electronic Journals (current) (bound) $ 5, 000 $40, 000 $205, 000 Systems $ 10, 000 $ 2, 500 $ 2, 400 Supplies & Services $ (2, 000) $ 600 $ 8, 000 Staff $125, 000 $46, 000 $ 42, 000 $90, 000 $258, 000 TOTAL OPERATIONAL COSTS $138, 000 Montgomery, Carol Hansen, Donald W. King: Comparing library and user related costs of print and electronic journal collections. D-Lib Magazine October 2002 Drexel University Philadelphia LIDA 2007

Staff costs by function (Drexel) EPrint journals (current) Director/Administration Print (bound) $ 22, 000 0 0 Communications (among staff) $ 7, 000 $ 2, 500 $ 800 Collection development $ 18, 000 $ 6, 000 0 $ 8, 000 $ 4, 000 0 0 $10, 000 $22, 000 Record creation and maintenance $ 16, 000 $10, 000 $ 6, 000 Public relations $ 11, 000 $ 600 Reference $ 37, 000 $ 4, 000 $ 6, 000 $ 1, 000 $ 600 0 $ 4, 000 $ 6, 000 $125, 000 $46, 000 $42, 000 Acquisitions Physical handling: bindery, labeling, reshelving Teaching (users) Other LIDA 2007

Cost per use by journal type (Drexel) recorded use subscription cost per use operational total cost per use print current 15. 000 $ 2. 50 $ 6. 00 $ 8. 50 print bound 9. 000 ---- $ 30. 00 total print 24. 000 $ 2. 50 $ 15. 00 $ 17. 50 total Ejournals 335. 000 $ 1. 40 $ 0. 45 $ 1. 85 LIDA 2007

Journals: cost per use Problems in measuring cost per use For subscription prices package deals for journals u e lower consortia with "additional access" s and lt in publishers insist on "print + electronic" tance su p acce idently re For measuring use igh H ill ev per use dia w cost e E-mreshelving counts may under-estimate use of E-journal use data may differ between publishers For assessing operational costs assessing staff time will be time-consuming long-term preservation costs for E-media still questionable LIDA 2007

Assessing impact/outcome In terms of knowledge information literacy s itude academic or professional es, att c success peten , com socialskills inclusion n ges i chan individual well-being , r aviou beh LIDA 2007

Methods for assessing impact Use statistics as measures of impact electronic media • frequency of use • forms of use (sessions, searches, downloads) Users are familiar • range of resources used with the use of change of use in traditional services change in remote use electronic number of new users resources and • that had not used the library services • that had not used the same service in traditional form percentage of the primary user group using Eservices LIDA 2007

Methods for assessing impact Qualitative methods Surveys (postal, face-to-face, telephone, online) Interviews The "anecdotal Focus groups, discussion groups evidence" supports Self assessment of users the statistics Data and „stories“ must be brought in a systematic order in order to make trends visible LIDA 2007

Methods for assessing impact Quantitative methods Tests of user skills before and after training Observation by trained observers Problem: Influences on individuals Data mining are manifold. The library‘s influence not always clear Analysis of bibliographies in publications of users Comparing library use to academic success LIDA 2007

Assessing the outcome of electronic services Example: Changing the journal collection to electronic form Statistical results § online versions were accessed 10 times more § usage of an online article 5. 4 times cheaper § high decline in print usage § remote use increased § physical library visits decreased change of attitudes Survey to faculty and behaviour • In 1998 a number of academics were sceptical as to E-journals • In 2002 it was exceptional for faculty staff not to use the electronic version LIDA 2007

Projects of assessing outcome e. VALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information services http: //www. evalued. uce. uk/index. htm IMLS (Institute of Museum and Library Services): outcome-based evaluation of projects http: //www. imls. gov/index. htm ARL New Measures Initiative: mehrere Projekte - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services) http: //www. arl. org/stats/newmeas/index. html SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing list http: //www. sconul. ac. uk/activities/performance/impact. html LIDA 2007

Standardization of impact measures? projects still on the way IFLA Section Statistics and Evaluation has instituted a working group Bibliography of projects and literature: http: //www. ulb. uni-muenster. de/outcome LIDA 2007

Experience is the mother of science Traditional Proverb LIDA 2007