Challenges of Digital Media Preservation Karen Cariani Director

Challenges of Digital Media Preservation Karen Cariani, Director Media Library and Archives Dave Mac. Carn, Chief Technologist

Who we are: WGBH Media Library and Archives 2

Challenges § Transitions (Analog to Digital) § What Born Digital Brings — New Workflows — New Capture Formats — Access vs. Preservation — New Systems New Storage 3

Transitions (Analog to Digital)

Transition challenges (Analog to Digital) § Preservation needs are more complicated — — — New and changing content formats Network connections Software Storage media Hardware § Access expectations challenging — Faster access — Anywhere, anytime 5

Common Physical Formats (and some not so common) § § § § 16 mm film (some 35 mm) 2” videotape 1” videotape 3/4” videotape Betacam Betamax MII § § § Digital Betacam D 1, D 2, D 3, D 4, D 5 DVCam DVCPro & DVCPro. HD Mini DV & HDV Keep in mind: The Machine was the Format. 6

Thought Migration § TAPE - Condition - Machinery - Priorities 8 § BORN DIGITAL - Codecs/wrappers - Nested data - Playback - Storage

Thought Migration § Before we had 1 shoot, 10 physical tapes, many shots per tape § Now each time the camera is in record, a new file is created § Now 1 shoot can generate many files across multiple digital storage containers (Optical Disk, Hard Drive, Solid State Memory) 9

Digital Formats

New Workflows

For Access: Data Organizational Issues § Technical Metadata — Characterize files § Descriptive metadata — Need description for video to be useful, findable — How to capture that — How to make sure it is linked to video files 12

Original Footage 13

That was easy, wasn’t it? § Plug-ins to view files § Depending on the file type we may have to re-wrap to a Quick. Time wrapper, or fully transcode the source file § Redundant storage of raw and unwrapped materials § Quality of data 14

Metadata Entry in the Field: § In-camera - encourage tagging files with data. § User-generated clip naming --- (Health 001, Health 002) § Card Labeling § Content Labeling through folder structures 15

Folder Structure 16 § Create folders by card — Assign unique number — Continue numbers — Add description — Place ENTIRE card contents into this folder!!

Video, Where for art thou video? 17

Finding both audio & video 18

Storage and Retrieval How do we: § § § Capture the audio and video generated by myriad cameras Store the project information to allow potential re-edit Store files with rich, meaningful metadata Store born-digital materials Display and retrieve born-digital materials 19

Original Footage © 2011 WGBH 20

Proposed tapeless workflow § Create a mapping document between Filemaker & DAM § Used to generate an xml style sheet § Video is ingested simultaneously with the metadata from Filemaker using the xml style sheet § Technical metadata is ingested simultaneously with the video and production data using the xml generated by the source digital files 21

Access vs. Preservation

Access 23 § File size – need proxies — Speed of access § Formats — Want consistency for playback § Reuse — Retrieval of original files/preservation files § Search/findable — Metadata — Organize files

Preservation Needs § § § Multiple Copies Save original files Validity – check sum Regular storage migration Persistence File format issues — Migration ease — Future playback § Fixity check big files § Big files — Speed of access of preservation files for reuse — Processing speed 24

What makes video different? § Preservation files are large — Uncompressed — Slow to move around § Need proxy files for viewing — Smaller size for quick transport over network § Complicated formats — Not just one file type — Codecs, wrappers, frame speed, etc. 25

New systems New storage

Software / Network 27 § File management — Where are the files? § Needed for access to files — Large preservation files — Smaller access, proxy files § Network speed — Larger files, need faster network to meet speed expectations

Issues with current mgmt. systems/software § Preservation not a priority § Interface issues — Access vs. Preservation § IT relationship — Tech support — Vendor reliance issues — Need library based system for Archivist needs rather than traditional IT company needs § Expense — License cost — Development — Customizations 28

Access repository as preservation § § § Fedora repository only stores proxy files Great interface Great search and indexing Faceted searches No preservation files or migration process 29

Technology Mix: 30

New Tools § § Combine preservation system with access system Better interface Flexible design Easy to evolve 31

Hydra project § § Blacklight Hydra heads Hydra mgmt. layer Fedora repository HSM storage system 32

Hydra Fundamental Assumption #1 & 2 § No single system can provide the full range of repository-based solutions for a given institution’s needs >. . . yet sustainable solutions require a common repository infrastructure. § No single institution can resource the development of a full range of solutions on its own, > …yet each needs the flexibility to tailor solutions to local demands and workflows 33

New Storage Types and Costs § Need hierarchical storage (HSM) — Video files are large — Spinning disks are expensive — Tape can help save cost — Tape copies/migration can be automated 34

Hardware/Storage media: HSM § Access — Online > XX bytes Spinning disk — Offline — Nearline § Preservation (offline) — Robotic tape library system — LT 04 data tapes — 2 copies — One stored off site § Migration needs 3 -5 years — Both tape migration to newer formats — Technology migration 35

New Storage Types and Costs § Proprietary § HSM has licensing issues — Some systems license by gigabyte managed, others by tapes managed § Need Open Source alternative 36

Q&A § Karen: karen_carinani@wgbh. org § Dave: dave_maccarn@wgbh. org 37