Architecting an Extensible Digital Repository Anoop Kumar Ranjani

Architecting an Extensible Digital Repository Anoop Kumar, Ranjani Saigal, Rob Chavez, Nikolai Schwertner Tufts University, Medford, MA

Overview l l Background Information on the evolution of TDL Design Requirements TDL Architecture Applications that interface with TDL – – Tufts DL search VUE

History of Digital Collections at Tufts l About Tufts – – l Interdisciplinary Focus on teaching and learning Digital Collections at Tufts – – Perseus (Classics) Tufts University Science Knowledgebase (TUSK-Medicine) Artifact (Art History) Digital Collections and Archives (DCA) l – Bolles, etc Other (Crime and Punishment)

Projects Materials Tools Perseus DL 50 million words, highly structured TEI Perseus document encoded XML texts of many types. management system and tools 50, 000 images DCA 13 million words, 35, 000 images, geospatial datasets multimedia objects Perseus document management system and tools TUSK 15, 000 documents Includes full-text syllabi, digital slide images, lecture recordings (audio and video) and text notes and exam questions, evaluation forms, and bibliographies linked to full-text articles. Networked course management system interface Artifact 2500 images links to the Art History slide collection database containing 120, 000 entries. On-demand viewing and searching with Internetbased adaptations of traditional learning aids, such as flashcards, for review and study

Why TDL? (Tufts Digital Library) l l The collections were continuously expanding adding content in a variety of formats. The architecture of these libraries was not built to accommodate such expansion. Needed a university wide digital repository that can manage the ever increasing content while continuing to service the discipline specific needs and leveraging existing and new tools and service

Designing TDL l l l Digital Collections and Archives partnered with Academic Technology to create a digital library that can manage the content while supporting teaching and learning. Commitment to comply with standards in the library and the open source community. Ensure Scalability, Flexibility, Reusability, Extensibility and Interoperability

Design Requirements l Ingest: – l Management: – – l Ability to enforce archival standards Use of information packages to facilitate storage and dissemination Ability to incorporate content models Persistence: – – Use of persistent identifiers mapped URNs Requirements System Services Unique and persistent identification of materials Naming Service Use of archival information packages (AIP) Digital Object Provider (DOP) Service -- Fedora Use of submission information Packages (SIP) Drop Box, Ingestion Service Use of Dissemination Information Packages (DIP) DOP Service Authentication and integrity checking DOP Service Dissemination Disseminators, Caching Service, Digital Library Application, Search Service Access Search Service and other applications

Tufts DL Architecture A M A Fedora Client Application Creation Service Application Data U Application Interface Drop Box Fedora Search Interface Naming Service Fedora Ingestion Service Search Indexing Service Search Index U U - Users M - Manager A - Administrators

Component Role Drop Box and Ingestion Service Validation, Tagging, Preprocessing, Ingestion Naming Service Unique persistent identifiers mapped to objects (“tufts: dca: central: MS 102. 33. 1345”) Fedora Repository Management and access framework for digital objects Search and Indexing Service Provides search mechanism Application Creation Service Provides mechanism for external applications to interface with repository Components of TDL

TDL Architecture l l l Drop Box and Ingestion Service Naming Service Fedora Repository Service at Tufts Indexing Service and Search Engine Application Creation Service

Drop Box and Ingestion Service

TDL Architecture l l l Drop Box and Ingestion Service Naming Service Fedora Repository Service at Tufts Indexing Service and Search Engine Application Creation Service

Naming Service l l Assigns, reserves and resolves URN Format tufts: school name: owner: [collection: ]item name tufts: dca: central: MS 102. 33. 1345 l URN Properties – – Provides unique ID to objects deposited into repository Service assures resolution to unique resource.

TDL Architecture l l l Drop Box and Ingestion Service Naming Service Fedora Repository Service at Tufts Indexing Service and Search Engine Application Creation Service

Fedora Repository Service@Tufts l l l Fedora - Key Features Repository at Tufts Content Models at Tufts – l Objects, Behaviors and Disseminator Implementation Challenges

Flexible Extensible Data Object Repository Architecture (Fedora) l l l Support for heterogeneous data types Accommodation of new types as they emerge Aggregation of mixed, possibly distributed, data into complex objects The ability to specify multiple content disseminations of these objects The ability to associate rights management schemes with these disseminations.

Repository Model Medium Bandwidth (20 Mb TIFF) High Bandwidth HTTP Server HTTP Request Storage Device (20 Mb TIFF) sto res Caching Service UR L sf or Processing Service HTTP Request Medium Bandwidth (200 Kb JPEG) Fedora HTT P Re ques t Inter net B andw (200 idth Kb J PEG ) s n tio a Ap c pli User

Content Model (CM) Hierarchy Indexing Disseminators Repository-Level Disseminators • get. Index. Terms • get. Archival. Copy • get. For. Indexing • get. Preview • Etc. • get. Class • Etc. Text CM VUE CM Image CM Binary CM Collection CM • get. TOC • get. Concept. Map • get. Thumbnail • get. Objects • get. Chunks. List • get. Resource • get. Access. High • get. MIME • get. Info • get. Chunk • Etc. • get. Image. Stats • Etc. Specific Implementations (TEI text, EAD text, Encyclopedia, Directory, TIFF image, etc)

Implementation Challenges l l l Processing Large XML Documents Transforming Large Images Modeling Collections Advanced Search Customized Search Caching Disseminations

TDL Architecture l l l Drop Box and Ingestion Service Naming Service Fedora Repository Service at Tufts Indexing Service and Search Engine Application Creation Service

Indexing Service and Search Engine l Indexing – l Implementation – l Lucene Supported Types of Search – – l Specialized Polymorphic Disseminators Basic Keyword Advanced metadata based Accessing the service – – HTTP GET/POST SOAP

TDL Architecture l l l Drop Box and Ingestion Service Naming Service Fedora Repository Service at Tufts Indexing Service and Search Engine Application Creation Service

Application Creation Service l l l An important design requirement for TDL was to allow current digital library applications to easily interface with TDL and provide access to the content in the digital library within their own environments in a seamless fashion. Current applications like Perseus can interface with this service to allow their tools to disseminate the content that resides in TDL The service has been designed not only to support current application but also to accommodate the needs of future yet-tobe-defined applications like course management systems, learning tools, portals etc.

Applications Accessing TDL Content l l Tufts DL Search Visual Understanding Environment (VUE)

Visual Understanding Environment (VUE)

Why TDL? (Tufts Digital Library) l l The collections are continuously expanding adding content in a variety of formats. The current architecture of these libraries is not built to accommodate such expansion. Need a university wide digital repository that can manage the ever increasing content while continuing to service the discipline specific needs and leveraging existing and new tools and service

Future Direction l l l Authentication and authorization service Customization and enhancement to Fedora@Tufts to address a wide variety of needs. Provide automated browsing service for Repository.