Assessing the Value and Impact of Digital Content

Assessing the Value and Impact of Digital Content Brinley Franklin Vice Provost for University Libraries University of Connecticut March 1, 2007

Measuring Print Collection Usage Did we ever know what really happened in those “miles of aisles? ” “It is useless to tell the acquisitions librarian that half the monographs ordered will never be used, unless we can specify which 50% to avoid buying. ” (Galvin and Kent, 1977)

The Brave New World of Digital Content

Digital Content is Truly Omnipresent

ARL New Measures Initiative In January, 1999, the ARL Statistics and Measurement Committee and ARL Research Library Leadership and Management Committee, under Carla Stoffle’s leadership, initiated the ARL New Measures Initiative in response to: 1. Increased demand for libraries to demonstrate outcomes/impacts (instead of inputs and outputs) in areas important to the institution 2. Increasing budgetary and political pressure to maximize efficient use of resources and to identify best practices

ARL New Measures Initiatives • • • Higher Education Outcomes Research Review – Learning Outcomes – Information Literacy (SAILS)* – Research Outcomes Investigation of Cost Drivers Technical Services Cost Study Document Delivery/Interlibrary Loan Study E-Metrics (Electronic Resources Measures)* • • • Stats. QUALTM Lib. QUAL+TM * Digi. QUALTM* MINES for Libraries. TM* Adapted from: Peter T. Shepherd and Denise M. Davis, “Electronic Metrics, Performance Measures, and Statistics for Publishers and Libraries: Building Common Ground and Standards” portal: Libraries and the Academy, Vol. 2, No. 4 (2002)

Vendor Supplied Data and Transaction Based Usage E-metrics (aka ARL Supplementary Statistics), Project. COUNTER, the Penn Library Data Farm, SUSHI, etc.

Reliance on Vendor Statistics Vendor statistics, while more reliable than in the past, are still maturing.

Transaction Based Usage What do those big usage numbers mean? It takes some serious analysis to understand what 1, 354, 478 Science Direct uses really means

Unit Costs of Electronic vs. Print Journals Drexel University Library D-Lib Magazine (2002) Medical Branch Library of the University Library Muenster, Germany Health Information and Libraries Journal (2003) Nonsubscription Periodicals Costs CLIR Study (June 2004)

Drexel University Total Unit Cost per Use Subscription Operational Unit Cost Recorded Use Cost per Use 15, 000 $2. 50 $6. 00 $8. 50 9, 000 NA $30. 00 24, 000 $2. 50 $15. 00 $17. 50 23, 000 $3. 20 $0. 45 $3. 65 134, 000 $2. 25 $0. 45 $2. 70 20, 000 $1. 35 $0. 45 $1. 80 158, 000 $0. 45 $0. 85 335, 000 $1. 40 $0. 45 $1. 85 Print Journals Current Journals Bound Journals Total Print Journals Electronic Journals Individual Subscriptions Publisher’s Packages Aggregator Journals Full-Text Database Journals Total Electronic Journals Adapted from: C. H. Montgomery and D. W. King, “Comparing Library and User Related Costs of Print and Electronic Journal Collections: A First Step Towards a Comprehensive Analysis, ” D-Lib Magazine, October 2002.

Medical Branch Library, University Library Muenster, Germany Unit Costs of Print and Online Journals (in Euros) Print Subscription Cost Print Usage Print Unit Cost Online Subscription Cost Online Usage Online Unit Cost Academic 67, 533 € 3, 350 20. 16 € 54, 241 € 3, 593 15. 10 € Blackwell 35, 742 € 1, 531 23. 35 € 30, 380 € 6, 329 4. 80 € Elsevier 60, 143 € 4. 012 14. 99 € 54, 139 € 8, 248 6. 56 € High Wire 8, 984 € 3, 223 2. 79 € 8, 086 € 25, 975 0. 31 € Springer 85, 353 € 1, 679 50. 82 € 76, 081 € 20, 346 3. 77 € Total 257, 737 € 13, 795 18. 68 € 223, 647 € 64, 491 3. 47 € Adapted from: Oliver Obst, “Patterns and Cost of Printed and Online Journal Usage, ” Health Information and Libraries Journal 2003.

Annual Nonsubscription Costs for Print and Electronic Journals E-Journals Current Print Journals Print Journal Backfiles Cost Per Title ($US) Bryn Mawr $ 10. 95 $ 105. 65 $ 0. 88 Franklin and Marshall $ 10. 67 $ 63. 77 $ 1. 10 Suffolk $ 35. 19 $ 313. 89 $ 0. 32 Williams $ 10. 63 $ 92. 38 $ 0. 63 Schonfeld, King, Okerson, Drexel $ 7. 28 $ 101. 37 $ 0. 39 and Fenton, The George Mason $ 16. 84 $ 74. 06 $ 1. 80 Nonsubscription Side of Western Carolina $ 12. 83 $ 50. 33 $ 0. 98 Periodicals: Changes in Library Operations and Cornell $ 26. 29 $ 49. 82 $ 0. 18 NYU $ 16. 01 $ 41. 77 $ 1. 63 Pitt $ 47. 04 $ 59. 21 $ 1. 15 Yale $ 33. 53 $ 29. 37 $ 1. 15 Costs between Print and Electronic Formats, Council on Library and Information Resources, June 2004.

University of Connecticut Libraries Database Unit Costs FY 2005 Approximate Searches/ Approximate FY 2005 FY 2003 Downloads FY 2005 Cost Unit Cost Searches World. Cat FY 2003 Cost per Search 38, 726 $17, 000 $0. 44 30, 131 $11, 160 $0. 37 137, 722 $30, 000 $0. 22 49, 439 $ 30, 650 $0. 62 15, 075 $13, 000 $0. 86 13, 700 $ 6, 375 $0. 47 Web of Science 118, 151 $165, 000 $1. 40 66, 420 $ 144, 039 $2. 17 Total 309, 674 $225, 000 $0. 73 159, 690 $192, 224 $1. 20 JSTOR First. Search Special thanks to: Deborah Sanford, Stephanie Willen Brown and Nicholas Eshelman (UConn Libraries)

Measuring Digital Content Use The most popular current method of measuring usage of electronic resources by libraries is not through webbased usage surveys, but through vendor supplied data of library patron usage or transaction usage. Web-based usage surveys are increasingly relevant in the collection of usage data to make collection development and service decisions, to document evidence of usage by certain patron populations, and to collect and analyze performance outputs. Brinley Franklin and Terry Plum, “Successful Web Survey Methodologies for Measuring the Impact of Networked Electronic Services (MINES for Libraries. TM)” IFLA Journal 32 (1) March, 2006

An Infrastructure of Assessment Using web-based surveys to probe deeper into digital content use

An Assessment Gateway Setting up an authenticated entrance point to networked resources

An assortment of library assessment tools that collectively describe the role, character, and impact of physical and digital libraries. Consists of Lib. QUAL+™, Digi. QUAL™, and MINES for Libraries™, as well as a growing dataset of survey results that offer insights into contemporary library use and user satisfaction. Stats. QUAL™ operates under the leadership of the Association of Research Libraries (ARL), which has a long-standing role in the development, testing, and application of performance measures, statistics, and management tools.

Lib. QUAL+TM • • Used to identify user perceptions of service quality and gaps between users’ desired, perceived and minimum expectations Consists of 22 core items and “a box” Can be classified into three subscales: Affect of Service; Library as Place, and Information Control Utilized by more than 1000 libraries in the last seven years A new Lib. QUAL+ study looks at responses from more than 225, 000 Lib. QUAL+ participants in the U. S. between 2004 and 2006 The most desired core item was: Making electronic resources accessible from my home or office The second most desired core item was: Print and/or electronic journal collections I require for my work The third most desired core item was: A library web site enabling me to locate information on my own

A grant-funded project that represents a modification of the Lib. QUAL+TM protocol. In its initial implementation, Digi. QUALTM evaluated digital library services provided for the user communities of the National Science, Math, Engineering and Technology Education Digital Library (NSDL) program. Digi. QUALTM was pilot tested on the Digital Library for Earth System Education (DLESE), the Computational Science Education Reference Desk (CSERD), Utopia, The Math Forum@Drexel, and the Multimedia Educational Resource for Learning and Online Testing (MERLOT).

A web-based transactional survey that collects data on users’ demographics and their purpose of use. It is administered in real time over the course of at least a year using a random moments sampling plan. MINES for Libraries. TM has been administered at 40 North American universities in the last four years. More than 100, 000 North American networked services users have been surveyed using a standard protocol.

Library User Survey Patron Status

Library User Survey Affiliation

Library User Survey Location

Library User Survey Purpose

Sample Survey Data File Generated • • • • Other UConn. Instruction/Education/Departmental (Non-Funded) Researchhttp: //newfirstsearch. oclc. org/done=referer; dbname=World. Cat; autho=100122319; FSIP 12: 36: 5012/3/2004 Off Campus. UConn Faculty 12. 101. 104. 84 Family Studies. Instruction/Education/Departmental (Non-Funded) Researchhttp: //www. jstor. org/cgi-bin/jstor/gensearch 12: 37: 4312/3/2004 Of Campus. UConn Undergraduate Student 12. 101. 104. 84 Non-UConn. Other Activitieshttp: //homerweb. lib. uconn. edu/cgibin/Pwebrecon. cgi? DB=local&PAGE=First 13: 08: 4112/3/2004 Off Campus. Non-UConn 12. 18. 36. 40 Non-UConn. Instruction/Education/Departmental (Non-Funded) Researchhttp: //magic. lib. uconn. edu/index_real. html 13: 31: 2912/3/2004 Off Campus. Non-UConn 12. 76. 131. 246 Non-UConn. Other Activitieshttp: //magic. lib. uconn. edu/index_real. html 12: 11: 0612/3/2004 Off Campus. Non-UConn 130. 132. 86. 156 Agriculture & Natural Resources. Instruction/Education/Departmental (Non-Funded) Researchhttp: //magic. lib. uconn. edu/index_real. html 12: 33: 5712/3/2004 Off Campus. Non-UConn 131. 128. 89. 54 Education. Instruction/Education/Departmental (Non-Funded) Researchhttp: //www. euromonitor. com/womdas/12: 57: 4412/3/2004 Off Campus. Non-UConn 134. 241. 135. 70 Non-UConn. Instruction/Education/Departmental (Non-Funded) Researchhttp: //homerweb. lib. uconn. edu/cgi-bin/Pwebrecon. cgi? DB=local&PAGE=First 13: 28: 5212/3/2004 Off Campus. Non-UConn 134. 74. 42. 82 Business Administration. Other Activitieshttp: //homerweb. lib. uconn. edu/cgibin/Pwebrecon. cgi? DB=local&PAGE=First 12: 56: 4612/3/2004 In the Library. UConn Faculty 137. 99. 1. 122 Liberal Arts & Sciences. Other Activitieshttp: //www. siam. org/journals/simax. htm 12: 52: 1712/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 1. 122 Engineering. Instruction/Education/Departmental (Non-Funded) Researchhttp: //homerweb. lib. uconn. edu/cgibin/Pwebrecon. cgi? DB=local&PAGE=First 12: 04: 3112/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 1. 219 Business Administration. Instruction/Education/Departmental (Non-Funded) Researchhttp: //proquest. umi. com/pqdweb? RQT=31812: 16: 3312/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 100. 105 Business Administration. Instruction/Education/Departmental (Non-Funded) Researchhttp: //www. jstor. org/journals/07322399. html 12: 16: 5212/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 100. 105 Business Administration. Instruction/Education/Departmental (Non-Funded) Researchhttp: //homerweb. lib. uconn. edu/cgibin/Pwebrecon. cgi? DB=local&PAGE=First 12: 29: 5312/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 100. 175 Business Administration. Instruction/Education/Departmental (Non-Funded) Researchhttp: //homerweb. lib. uconn. edu/cgibin/Pwebrecon. cgi? DB=local&PAGE=First 12: 48: 4112/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 100. 175 Business Administration. Instruction/Education/Departmental (Non-Funded) Researchhttp: //proquest. umi. com/login? COPT=SU 5 UPTAm. Vk. VSPTIm. REJTPTE 3 Mj. Er. Mysx. Nk. JD&client. Id=4899612: 04: 2312/3/2004 On Campus - Storrs. UConn Graduate Student 137. 99. 100. 175

MINES for Libraries TM Location of Electronic Resources Users U. S. Main Libraries Total Users U. S. Medical Libraries Total Users n = 25, 698 n = 31, 883 OCUL (Canada) Libraries Total Users n = 20, 300

Demographics Jonathan – the University of Connecticut Husky Mascot (circa 1991)

MINES for Libraries™ Demographics by Location of User U. S. Main Libraries Inside the Library On Campus, Not in the Library n = 9, 172 n = 6, 391 Off-Campus n = 4, 953

MINES for Libraries™ Demographics by Location of User U. S. Medical Libraries Inside the Library On Campus, Not in the Library n = 6, 819 n = 19, 582 Off-Campus n = 5, 133

MINES for Libraries TM Demographics by Location of User Ontario Council of University Libraries Inside the Library On Campus, Not in the Library n = 4, 047 n = 7, 090 Off-Campus n = 9, 163

Purpose of Use Are users engaged in coursework, funded (or unfunded) research, public service, patient care, or other activities?

Purpose of Use By Location U. S. Main Campus Libraries 2003 – 2005 On-Campus, Not in the Library In the Library n = 9, 733 n = 9, 460 66% Off-Campus n = 7, 790 Overall Use n = 26, 983 *72% of sponsored research usage of electronic resources occurred outside the library; 83% took place on campus.

Purpose of Use By Location U. S. Medical Libraries 2003 – 2005 In the Library n = 6, 590 Off-Campus n = 4, 852 On-Campus, Not in the Library n = 15, 948 Overall Use n = 27, 390 *83% of sponsored research usage occurred outside the library. 92% of this use took place on-campus.

OCUL Scholars Portal Users Purpose of Use In a sample of 20, 300 electronic resources uses at OCUL libraries, there were four uses outside the library for each use in the library.

Recent Advances in Capturing All Networked Services Use Through Web-based Surveys • An advanced application of EZproxy at UTMB presents the MINES survey to networked services users as they initiate a session, thereby capturing almost all networked services usage both locally and remotely • UCSD is capturing all on-campus and off-campus networked services usage through the proxy server or virtual private network (VPN) at the campus Internet router. This approaches catches all authorized users’ usage during the randomly selected two hours per month MINES surveys.

Look How Far Libraries Have Come (in less than a decade) • For most of our history, evaluation methods for print collections yielded unreliable data for decision-making • International projects have identified relevant data elements and set standards for commercial electronic resource usage; standards have also recently been developed for harvesting usage data • Libraries, consortia (and publishers) routinely calculate unit costs for electronic resources and perform cost/benefit analyses • Lib. QUAL+, and Digi. QUAL and other web-based user surveys have measured user satisfaction with electronic resources and digital library services and reinforced users desires for digital content • MINES and other web-based user surveys provide another level of insight into user behavior in the networked environment • The coming years will see further developments that will even further refine data about users and how they utilize digital content

Questions? This presentation is available at: www. library. uconn. edu/~bfranklin Contact the author at: brinley. franklin@uconn. edu Learn more about Lib. QUAL+TM, Digi. QUALTM, & MINES for Libraries. TM at: www. statsqual. org