Status and Overview of HMI AIA Joint Science Operations

Status and Overview of HMI–AIA Joint Science Operations Center (JSOC) Science Data Processing (SDP) P. Scherrer • Science Goals • Science Products • Software Infrastructure • Hardware SDO Teams Meeting, March 2008 Page 1

HMI & AIA JSOC Scope • The HMI/AIA Joint SOC consists of two functional parts: – Science Data Processing (SDP) – at Stanford and LMSAL – Instrument Operations Center (IOC) – at LMSAL • JSOC IOC includes: – HMI and AIA Commanding and Health Monitoring – HMI and AIA Engineering support as needed • JSOC SDP includes: – – – • HMI and AIA Science Telemetry Data capture (from DDS) and archive HMI and AIA Level-0 processing and archive HMI processing through to level-2 with archiving of end products AIA processing through level-1 with online archive at Stanford AIA level-2 processing at LMSAL Data export of the above and other HMI and AIA products as needed JSOC does not include tasks such as: – Science analysis beyond level-2 products – HMI and AIA EPO – HMI & AIA Co-I science support SDO Teams Meeting, March 2008 Page 2

JSOC – SDP Stanford Personnel JSOC SDP Team Science Data Processing Teams • Phil Scherrer, Acting SDP Lead • • SDP Software Level-0 Team, Rock Bush, Lead – Jim Aloise, SUMS, PUI – Keh-Cheng Chu – Karen Tian, DRMS, Export – Jennifer Spencer – Art Amezcua, CM, Level-2 Science Products – Carl Cimilluca • Level-1 Team, Jesper Schou, Lead – Jennifer Spencer, Database – Carl Cimilluca, Sys Engineer • – Sebastien Couvidat – Cristina Rabello-Soares – Richard Wachter SDP Hardware – Keh-Cheng Chu, Lead – Brian Roberts, Sys Admin • – Yang Liu – Steve Tomczyk, (HAO group lead) Data Operations – Jeneen Sommers – Hao Thai • HMI Level-2 Science Leads – Rick Bogart – Tom Duvall (GSFC) – J. Todd Hoeksema – Sasha Kosovichev – Jesper Schou SDO Teams Meeting, March 2008 Page 3

JSOC – SDP Locations at Stanford Poplar Cedar South Cypress North P&A 1 st floor P&A Basement SDO Teams Meeting, March 2008 Page 4

HMI & AIA JSOC Architecture White Sands keep e hous GSFC ing MOC DDS Stanford Offline Archiv e Data Export & Web Service AIA Analysis System Local Archive Catalog High-Level Data Import Housekeeping Database Quicklook Viewing Primary Archive 12 -Day Archive SDO Teams Meeting, March 2008 HMI & AIA Operations HMI JSOC Pipeline Processing System Redundant Data Capture System Offsite Archiv e LMSAL World Science Team Forecast Centers EPO Public Page 5

JSOC Interfaces with SDO Ground System S-b Ka- and ban d Instrument Commands RT HK Telemetry (S-band) MOC at GSFC DDS at WSC Science data files (Ka-band) DDS Handshake files Spare Science Data Capture AIA Science System Data Capture HMI Science System Data Capture System RT HK telemetry L-0 HK files FDS products Planning data AIA MON monitoring HMI MON Planning monitoring Planning SDP segment JSOC-SDP Stanford JSOC-SDP Primary responsibilities: Capture, archive and process science data Additional: Instrument H&S monitoring SDO Teams Meeting, March 2008 L-0 HK files FDS products Mission support data Instrument Commands AIA OPS Real-time HMI OPS Inst Real-time monitor and Control Inst monitor and Control RT HK Telemetry AIAQL Quicklook HMI QL Planning Quicklook Analysis Planning Analysis T&C-Segment JSOC-IOC LMSAL JSOC-IOC Primary responsibilities: Monitor instruments health and safety in real-time, 24/7 Control instrument operations and generate commands Support science planning functions Page 6

JSOC Dataflow Rates SDO Teams Meeting, March 2008 Page 7

JSOC Data Volumes from Proposal SDO Teams Meeting, March 2008 Page 8

JSOC Data Volumes from Proposal SDO Teams Meeting, March 2008 Page 9

JSOC-SDP Major Components DDS Support W/S, FDS, L 0 HK, Pipeline User Interface, etc. Data Capture System 2 Dual Core X 86 -64 Processors 8 TB Disk HMI LTO-4 Tape Library SPARE Processors & Export Cache Processors 10 TB Disk Primary Processors 10 TB Disk Secondary Processors 2 TB Disk 10 TB Disk Export 10 -Gig ethernet Pipeline Processor Cluster LMSAL SPARE@MOC Firewall Workstations SUMS Server 512 cores in 64 nodes LTO-4 Tape Library Local Science Link 2 Dual Core X 86 -64 1 TB Disk 1 -Gig Web 2 Dual Core X 86 -64 AIA Processors Offsite 4 Quad Core X 86 -64 LTO-4 Tape Library 2 Dual Core X 86 -64 LTO-4 Tape Library 4 Quad Core X 86 -64 Web Server 2 Quad Core X 86 -64 Processors LTO-4 Tape Library 4 Quad Core X 86 -64 10 TB Disk 2 Dual Core X 86 -64 Processors Database – DRMS & SUMS 2 Quad Core X 86 -64 2 Quad Core Processors X 86 -64 2 Quad Core X 86 -64 Processors 2 Quad Core Processors X 86 -64 2 Quad Core X 86 -64 Processors 2 Quad Core Processors X 86 -64 2 Quad Core X 86 -64 Processors 2 Quad Core Processors X 86 -64 2 Quad Core X 86 -64 Processors 2 Quad Core Processors X 86 -64 Processors 400 TB Disk Tapes 12 LTO-4 Drives 150 TB Disk 2200 Cartridge Per year Library Fast interconnect SDO Teams Meeting, March 2008 Page 10

Hardware Procurement Status • DCS system is in place and acceptance tested – The 1 st OC 3 is in place – The DDS communications rack has been delivered and installed – Test data flows at expected rates and handshaking functions OK • Offsite and Catastrophe Systems in house, will be configured soon • Database, 16 nodes of Pipeline system, tape robotics and drives, disk server and initial 400 TB procurement are installed and nearly online. – Initial tests confirm c. 3 gigabytes/sec disk access • 48 nodes of Processor farm will be delayed until summer 2008 • Building Facility ready and occupied SDO Teams Meeting, March 2008 Page 11

JSOC Processing “Levels” • Tlm is raw telemetry files as received from SDOGS • Level-0 is images extracted from tlm with added meta-data, no change to pixels • Level-1 is cleaned up and calibrated into physical units in standardized form • Level-2 is science data products • Level-3 is higher level products or user produced products and are not JSOC products but may be archived and distributed as desired by owner SDO Teams Meeting, March 2008 Page 12

JSOC DCS Science Telemetry Data Archive • Telemetry data is archived twice • The Data Capture System (DCS) archives tlm files for offsite storage • Archive tapes are shipped to the offsite location and verified for reading • The Data Capture System copies tlm files to the Pipeline Processing System • The Pipeline Processing System generates Level-0 images and archives both tlm and Level-0 data to tape • Only when the DCS has received positive acks on both tlm archive copies does it inform the DDS, which is now free to remove the file from its tracking logic SDO Teams Meeting, March 2008 Page 13

HMI and AIA Level-0 • Level-0 Processing is the same for HMI and AIA • Level 0. 1 – Immediate – Used for Ops quicklook – Reformat images – Extract Image Header meta-data – Add “Image Status Packet” high-rate HK Packet (per image) – Export for JSOC IOC Quicklook • Level 0. 3 – Few minute lag - Used for quicklook science data products – Add other RT HK meta-data – Add FDS S/C info • Level 0. 5 – Day or more lag – Used for final science data products – Update FDS data – Add SDO roll info – Includes final images SDO Teams Meeting, March 2008 Page 14

DDS Level-0 HSB image Immediate or Retransmitted, permanent JSOC IOC Ground Tables DDS JSOC LEVEL-0 Processing Immediate JSOC-IOC quicklook, Temp, 5 d Level-0 HK ISP HK via MOC HK 1553 Other APID Level-0. 1 Level 1. 0 q Flat fielded and bad pixel list included, Temp 1 d Select nearest or average JSOC IOC Command logs if needed MOC FDS predict data dayfiles MOC FDS final data dayfiles FDS series, temp MOC HK 1553 APID dayfiles Level-0 HK, temp MOC SDO HK dayfiles From MOC SDO Teams Meeting, March 2008 Level-0. 3 FDS series, temp Few minutes lag SDO HK lev 0 temp Level 1. 5 q Quicklook Observables, Temp, 5 d Level 1. 5 p Provisional Observables, Links to best avail Level 1. 5 Final Observables, permanent Level-0. 5 Day lag Level 1. 0 Flat fielded and bad pixel list included, Temp 60 d Page 15

Level 1 HMI and AIA Basic Observable Quantities • • HMI and AIA level-1 “levels” are similar but the details differ (a lot). HMI – from filtergrams to physical quantities – 1. 0 Flat field applied to enable limb fit and registration – 1. 5 Final product types • 1. 5 q – Quicklook available in ~10 minutes, saved ~10 days • 1. 5 p – Provisional mix of 1. 5 q and 1. 5 final • 1. 5 – Final best possible product – Products • Continuum Intensity • Doppler Velocity • Line of Sight Field • Vector Field • AIA – Filtergrams are basic product – Quicklook and Final both produced – Planning movies from quicklook – Full details in development SDO Teams Meeting, March 2008 Page 16

JSOC - HMI Pipeline Processing HMI Data Level-0 Doppler Velocity Spherical Harmonic Time series To l=1000 Mode frequencies And splitting Ring diagrams Heliographic Doppler velocity maps Filtergrams Level 2 Data Product Local wave frequency shifts Time-distance Tracked Tiles Cross-covariance Of Dopplergrams function Egression and Ingression maps Wave travel times Wave phase shift maps Internal rotation Ω(r, Θ) (0

Example - Helioseismology Filtergrams Spherical Harmonic Time series To l=1000 Mode frequencies And splitting Ring diagrams Heliographic Doppler velocity maps Doppler Velocity Internal rotation Ω(r, Θ) (0

Example - Global helioseismology Filtergrams Doppler Velocity Heliographic Doppler velocity maps Code: project J. Schou Code: qdotprod J. Schou Status: ready to port SDO Teams Meeting, March 2008 Spherical Harmonic Time series To l=1000 Mode frequencies And splitting Code: med-l peak bagging J. Schou High-l ridge fitting, E. Rhodes Status: needs improvements Internal rotation Ω(r, Θ) (0

Analysis modules: Co-I contributions and collaboration • Contributions from Co-I teams: – Software for intermediate and high level analysis modules – Output data series definition • Keywords, links, data segments, etc. – Documentation (detailed enough to understand the contributed code) – Test data and intended results for verification – Time: Explain algorithms and implementation; Help with verification; Collaborate on improvements if required (e. g. performance or maintainability) • Contributions from JSOC team: – Pipeline execution environment – Software & hardware resources (Development environment, libraries, tools) – Time: Help with defining data series; Help with porting code to JSOC API; If needed, collaborate on algorithmic improvements, tuning for JSOC hardware, parallelization; Verification SDO Teams Meeting, March 2008 Page 20

JSOC data organization • Evolved from FITS-based MDI dataset concept to – Fix known limitations/problems – Accommodate more complex data models required by higher-level processing • Main design features – Lesson learned from MDI: Separate meta-data (keywords) and image data • No need to re-write large image files when only keywords change (lev 1. 8 problem) • No (fewer) out-of-date keyword values in FITS headers • Can bind to most recent values on export – Easy data access through query-like dataset names • All access in terms of sets of data records, which are the “atomic units” of a data series • A dataset name is a query specifying a set of data records (possibly from multiple data series): – Storage and tape management must be transparent to user • Chunking of data records into “storage units” and tape files done internally • Completely separate storage and catalog (i. e. series & record) databases: more modular design • Legacy MDI modules should run on top of new storage service – Store meta-data (keywords) in relational database (Postgre. SQL) • Can use power of relational database to rapidly find data records • Easy and fast to create time series of any keyword value (for trending etc. ) • Consequence: Data records for a given series must be well defined (i. e. have a fixed set of keywords) SDO Teams Meeting, March 2008 Page 21

JSOC DRMS/SUMS Basic Concepts • • • Each “image” is stored as a record in a data “series”. There will be many series: e. g. hmi_ground. lev 0 is ground test data The image metadata is stored in a relational database – our Data Record Management System (DRMS) The image data is stored in SUMS (Storage Unit Management System) which itself has database tables to manage its millions of files. SUMS owns the disk and tape resources. Users interact with DRMS via a programming language, e. g. C, FORTRAN, IDL. The “name” of a dataset is actually a query in a simplified DRMS naming language that also allows general SQL clauses. Users are encouraged to use DRMS for efficient use of system resources Data may be exported from DRMS as FITS or other protocols for remote users. Several Remote DRMS (RDRMS) sites will be established which will “subscribe” to series of their choice. They will maintain RSUMS containing their local series and cached JSOC series. The JSOC will act as an RDRMS to access products made at remote sites. SDO Teams Meeting, March 2008 Page 22

JSOC Pipeline Processing System Components Pipeline Operato r Pipeline processing plan JSOC Science Libraries Processing script, “mapfile” PUI Pipeline User Interface Pipeline Program, “module” List of pipeline modules with needed datasets for input, output Utility Libraries SUMS Disks DRMS Library Record Manage ment Keyword Access Link Manage ment Record Cache Data Access SUMS Storage Unit Management System DRMS Processing History Log Data Record Management System SUMS Tape Farm Database Server SDO Teams Meeting, March 2008 Page 23

JSOC Export • • • ALL HMI and AIA data will be available for export at level-0 through standard products (level-1 for both and level-2 for HMI) It would be unwise to expect to export all of the data. It is simply not a reasonable thing to expect and would be a waste of resources. Our goal is to make all useful data easily accessible. This means “we” must develop browse and search tools to help generate efficient data export requests. Quicklook Products – Quicklook raw images to JSOC IOC – Quicklook Basic Products to Space Weather Users • Prime Science Users – – – • JSOC will support Virtual Solar Observatory (VSO) access JSOC will also have a direct web access There will be remote DRMS/SUMS systems at key Co-I institutions JSOC In Situ Delivery and processing Special Processing at SDP as needed and practical Public Access – Web access for all data; Special products for E/PO and certain solar events SDO Teams Meeting, March 2008 Page 24

Software Development Status • DCS System complete, needs more testing with DDS, 98% complete • SUMS fully operational, needs tuning, 95% complete • DRMS fully operational, some features not fully implemented, 95% complete • PUI development expected to start shortly • Level-0 Image Code: HMI complete, AIA 80% complete • Level-0 HK, FDS and other metadata merge in development, 75% complete • Level-1 (science observables) code work starting now, sample V code done • HMI Level-2 (science data products) work in development – Local Helioseismology: Rings at risk due to funding, Time-Distance 50%, Holography 20% – Global Helioseismology: low-l will be MDI port to DRMS, work beginning, high-l needs funds – Magnetic Field standard products will be MDI ports, work beginning – Vector Field work at risk due to funding and loss of key Co-I • AIA Level-2 – Event catalog “Knowledge Base” development underway • Summary: About where expected for Launch -10 and Phase-E – 12 months SDO Teams Meeting, March 2008 Page 25

Configuration Management & Control • Capture System – Managed by JSOC-SDP CCB after August freeze – Controlled in CVS • SUMS, DRMS, PUI, etc. Infrastructure – Managed by JSOC-SDP CCB after launch – Controlled in CVS • PUI Processing Tables – Managed by HMI and/or AIA Instrument Scientist – Controlled in CVS • Level 0, 1 Pipeline Modules – Managed by HMI and/or AIA Instrument Scientist – Controlled in CVS • Science Analysis Pipeline Modules – Managed by program author – Controlled in CVS SDO Teams Meeting, March 2008 Page 26

JSOC Access Web Pages still in development, but please visit: http: //jsoc. stanford. edu See the wiki: http: //jsoc. stanford. edu/jsocwiki Or look at data: http: //jsoc. stanford. edu/ajax/lookdata. html with Firefox for now. SDO Teams Meeting, March 2008 Page 27