Historical lessons inter-disciplinary comparison and their application to

Historical lessons, inter-disciplinary comparison, and their application to the future evolution of the ESO Archive Facility and Archive Services Paul Eglitis and Dieter Suchar ESO, Karl-Schwarzschild Str. 2, 85748 Garching near Munich, Germany Chajnantor (5000 m) Paranal (2600 m) La Silla (2400 m) EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 1

Introduction v Inter-disciplinary thoughts v The ESO Science Archive Facility v Physical experiences: storage and infrastructure v Future growth requirements v Lessons Learnt v Future evolution EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 2

Archive Requirements: Inter-disciplinary Variation and Similarity Space Mission Earth Observation Ground-Based Astronomy Particle Physics Customer Science Service, Science Configuration Stable (Mission) Stable (Long Term) Variable Data Timeliness On Opportunity Periodic Variable In-house++ Always more! Data Volume Interoperability IPDA, SPASE EUROPLANET … EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 3 GEOSS, INSPIRE, GIGAS , HMA FITS, VO, IVOA E-GRID

CHILE DATA HANDLING Replicated Database Infrastructure OBSERVATORIES Traveling Disk Replication On-Line Data Transfer QC STAGING SECONDARY ARCHIVE PRIMARY ARCHIVE ESO SCIENCE ARCHIVE EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 4 USER INTERFACES Local Databases Replicated Database Infrastructure HQ

Physical Archive History GB 2500 2000 1500 cited. be Racks 1000 Hardware Changes CD * DVD * Unit Media (GB) DISK * Inexpensive Upgrades * * * 100 TB Footprint (Racks) 500 0 1996 2 1998 2000 2002 2004 Year EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 5 0. 5 2006 0. 25 2008 2010 2012

“Multi-mission” Concept in Archival VLT CRIRES FLAMES FORS 1 FORS 2 HAWK-I ISAAC NACO SINFONI Very Large Telescope CRyogenic high-resolution Infra. Red Echelle Spectrograph Fibre Large Array Multi Element Spectrograph FOcal Reducer/low dispersion Spectrograph 1 FOcal Reducer/low dispersion Spectrograph 2 High Acuity Wide field K-band Imager Infrared Spectrometer And Array Camera Nasmyth Adaptive Optics System (NAOS) Near-Infrared Imager and Spectrograph (CONICA) Spectrograph for INtegral Field Observations in the Near Infrared NTT ESO 3. 6 m MPG/ESO 2. 2 m Wide Field Imager WFI The Fiber-fed Extended Range Optical Spectrograph FEROS EFOSC 2 ESO Faint Object Spectrograph and Camera High Accuracy Radial velocity Planet Searcher HARPS Son of ISAAC SOFI La Images UVES VIMOS VISIR X-SHOOTER VLTI MIDI VLTI AMBER VISTA MASCOT Ultraviolet and Visual Echelle Spectrograph ALMA APEX The Atacama Large Millimeter/submillimeter Array Atacama Pathfinder Experiment telescope VST E-ELT VLT Survey Telescope nal a Par a Sill Spectra Leading observatories and stateof-the-art instrumentation VIsible Multi. Object Spectrograph VLT Imager and Spectrometer for mid Infrared a multi wavelength (300 -2500 nm) medium resolution spectrograph Very Large Telescope Interferometer MID-infrared Interferometric instrument Astronomical Multi-BEam combine. R Visible and Infrared Survey Telescope for Astronomy Mini All-Sky Cloud Observation Tool r to ana hajn C re utu F European Extremely Large Telescope Calibration and Quality Information Data products managed in the ESO Data Centre and Archive Facilities

ESO Archive System 2009 EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 7

Design and construction of the ESO Data Centre Requirements: • Construction at different physical location • On an area of 7 offices in an existing building • Modular approach: Dismantle, move and rebuild the Data Centre at ESO premises.

Archive Holdings Past and Projected EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 9

Paranal and the ESO Very Large Telescope La Silla EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 10

VISTA-survey-telescope Credit: Steven Beard/UKATC/ESO EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 11

ALMA ~ 1 rack/yr Credit: ALMA EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 12

E-ELT EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 13 17 PB/yr ?

Lessons Learnt v Preconditioning: physical constraints of data flow a. o. domain practices bias archive solutions. v Physical installation: in-house, out-source (all or part)? v Industry support: risks, costs, maintenance continuity. v Engineering changes: return on investment, cost of change, operational costs. v Art-of-balance: especially when a facility serves multiple purposes, e. g. Operations, Preservation, Exploitation. v Interoperability: many parallel efforts are ongoing. Long-term view and inter-disciplinary initiatives like the PVSeries critical. EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 14

ESO Archive Future Evolution “History of effective response to rapid changes has led to system heterogeneity” v Homogenisation of metadata and interfaces v Development of new HMIs v Standards compliance v Converge on next storage technology: Ø Analyse current usage of disk-based system in detail Ø Assessment of growth and access requirements Ø End-User, Operations, Maintenance and Development requirements The ESO Science Archive is a complex part of the complete operational Data Flow System at ESO. EGLITIS and SUCHAR | PV 2009 ESAC | 03. 12. 2009 | Slide 15

Thank you Acknowledgement: observatory images courtesy of the ESO Education and Public Outreach Department. See http: //www. eso. org/gallery