Scoring a novel approach towards automated certification of

Scoring: a novel approach towards automated certification of pipeline products Reinhard Hanuschik, Wolfgang Hummel, Mark Neeser, Burkhard Wolff Data Processing and Quality Control Group European Southern Observatory SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Complexity and mass production • VLT: 11 instruments • VLTI: 2 instruments • 2009/10: 2 survey telescopes • roughly 50 modes • about 150 data types • most of them pipelinesupported SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Complexity and mass production • each data type or instrument component controlled by QC 1 parameters • metrics for instrument performance extracted by data pipelines • stored in database • data frequency varies on timescales between a day and a semester SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Complexity and mass production two challenges … … the complexity challenge! SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Complexity and mass production current VLT instruments: … the volume challenge! VIRCAM: OMEGACAM: avg: 30 GB/night 400 GB/night SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

The solution: scoring: well-established outside astronomy … QC process: • measure quality • compare quality • assess quality: score and certify SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring: the solution • measure quality: done by automatic procedures • pipelines • other QC procedures • implemented for CALIB data • progress on SCIENCE data SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring: the solution • compare quality: trending • first step towards assessment: • same/different behaviour as others? SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring: the solution • assess quality: scoring • compare new result to trending • pick outliers and flag them SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring: the solution • required: thresholds • statistical thresholds: e. g. +/- 3 sigmas + independent from specs - sensitivity variable control charts • specified thresholds: e. g. lower/upper level + stable thresholds - requires careful configuration and thinking saturation noise SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of pipeline products each new pipeline product is scored • most relevant QC 1 parameters: compare to trending, score as • OK: 0 • NOK: 1 (outlier) • count scores, assign total score, e. g. SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of pipeline products score report: SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of pipeline products score reports for multi-detector instruments (CRIRES: N=4) … detailed information on demand! SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of instrument health • all scores go into database • re-arrange them by instrument properties • detector health • calibration lamp performance • system efficiency score-based instrument QC cal file 1 scores FLAT instrument scores cal LAMPs cal file 2 scores ARC SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of instrument health • example: detector properties for GIRAFFE instead of … SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Scoring of instrument health • and finally: www. eso. org/HEALTH SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems

Conclusions • • • scoring: automatic flagging of outliers useful to auto-certify pipeline products quick-look monitoring of instrument health powerful concept to handle large and complex data sets hierarchical approach: from global overview (instrument score) to QC 1 parameter product and detector (6 levels) SPIE Astronomical Instrumentation 2008 Observatory Operations: Strategies, Processes, and Systems