A Future Detachment Computing System Paul Barrett Richard

A Future Detachment Computing System Paul Barrett, Richard Cotton, Steve Abel, Kate Szpek © Crown copyright Met Office

Table of Contents • Motivation for a Detachment Computing System • Requirements of the system • Applications • Hardware • The DATA SYSTEM: discussion © Crown copyright Met Office

Introduction • New computing facility for collection, storage, processing and preliminary analysis of aircraft flight data during OBR detachments. • Requirement from OBR cloud physics group to have better access to data processing and storage during detachments. • Not intended to replace CORE processing although the hardware will be free for all to use. © Crown copyright Met Office

Motivation • • Stems from CONSTRAIN and Pikn. Mix FAAM – TARDIS for CORE and CORE-CLOUD Met Office bought new NAS drive - storage of non-core OBR Cloud Physics Group– personal laptops using Virtual machines and GDL - not ideal Code WG at Met Office – develop consistent flight data analysis tools such as Size Distributions, common parameters, Theta-vl, etc Central code to rebuild MRF_WAVE / MRF_IDL OBR_(insert_your_new_most_fashionable_language) Wanted to: Process Nevzorov data during testing phase, look at WVSS 2 data to check that flights were successful – no facility existed to do these non-core instruments Decided to invest in new computing facilities © Crown copyright Met Office

System Requirements • Simple plug-and-play hardware • Virtualisation of existing Met Office processing machines • User input developing scripts – OBR: Cloud Physics, Aerosol, Radiation, FAAM, NERC • Movable / portable / maintainable • Back-end software (DLR forecast tool, metresearch. com, incoming data handling (BADC? ) © Crown copyright Met Office

Applications • Processing and quick-look analysis of Met Office non-core instrument data • Processing and validation of new instrumentation e. g. Nevzorov, WVSS 2, SID-2 • Campaign specific quick-look at CORE data e. g. TWC, Aerosol, • Collation of all data post-flight – as demonstrated feasible during SAMBBA • Storage and serving of all data • Running the user-side of new DLR forecast tool • Mirror of metresearch. com and other forecast products • Serving of forecast data to users (via laptops / tablets / phones etc) • Back-up of non-core data offsite (bandwidth dependant (BADC? , MO? ) © Crown copyright Met Office

Hardware • • 19” Rack mounted hardware, enterprise grade server, NAS disk, network switch Optional UPS Ruggedised housing for ease of transportation and so system is almost plug and play Open for discussion Movable (>30 kg) computer/server for data storage, backup, processing and weather products. Virtual machines allow all to use what they want / need Facility to send data off site to remote storage for security and for others to analyse © Crown copyright Met Office

• Identify that inversion altitude falls throughout flight • Adiabatic LWC profile, inversion, constant nd gives confidence in success of sortie • Helpful mid-campaign to prioritise which types of sorties to fly • Also plot additional T, Td, liquid water parameters to indentify any instrument bias post flight Requires • Prior knowledge of flight plan • Core Data, Cloud data, Run Table Generator © Crown copyright Met Office T [K] qliq[kg kg-1] θ [K] Td [K] Nd [cm-3] reff [μm] z [m] Purpose z [m] Post Flight Quick Look Analysis – Stratocumulus Case B 576 Above: Profiles of thermodynamic and cloud physics parameters during B 576 (colours). Data from Straight and Level runs in filled grey circles with IQR as bars. [Osbourne and Abel (COALESC Met Office Key Deliv. Report 2012)]

Post Flight Quick Look Analysis –Aerosols PSAP Purpose Below: Time series of instrument data output and derived quantities • Map out the sampled aerosol • Compare data from multiple probes to identify inconsistencies and instrument issues early • Provide quick look to mission scientists • Derive properties such as SSA (single Scattering Albedo) Neph. Green a) Scattering, Nephi, altitude b) Single Scattering Albedo (nephew, PSAP) PCASP # Requires • Prior knowledge of flight plan • Core Data, Cloud data, Figures: Kate Szpek, Run Table Generator B 457 © Crown copyright Met Office Left: geographic distribution of aerosol parameters using CORE and CORE cloud instruments b) Aerosol number, CO concentration (air Mass)

Post Flight Quick Look Analysis – TWC Performance Left: Time series of TWC in 120 s sections from multiple campaigns, high qt and low qt Purpose • UV causes window degradation during campaign - worsening the high frequency response • If fluxes of TWC are crucial then knowledge of this could help decide when to program a soft-down-day or fly • Building up these diagnostics will assist in characterization of instruments over time • Identify problems very quickly • Use the back catalogue of “Quicklooks” to assist in fault finding. Requires • Prior knowledge of flight plan • Core Data, Cloud data, Run Table Generator © Crown copyright Met Office Below: Power spectra of time series data. High frequency response worsens with continued operation. High frequency response is worse in dry conditions

Schematic of Proposed OBR Detachment Computing System © Crown copyright Met Office

Snapshots, Discussion topics • Mission science requires timely ‘quick-look’ at data quality to identify potential issues. Cancel flight to work on instrument. • Missing data. Verify raw data integrity (file size, pre-viewer). Fewer lost flights with missing data. • Met data from UKMO / DLR interface ready for flight decision. • Processing of key instruments (Turbulence probe calibration, Nevzorov). • Build flight-based data archive as we go along. Easier and faster data transfer, logical and consistent data directory. • Flying is expensive and often ‘one-off’ events, while computer hardware is cheap. • Some instrument data is falling through the cracks (SID-2, WVSS) due to operator not being the data owner. • Instead of a zoo of laptops, use server-based computing. Common code and configured correctly before campaign. • All instrument data transferred off site. Scientist at HQ can analyse data Mission scientist can look at data immediately after campaign. Potential © Crown copyright Met Office failure of USB pen-drives, instrument HDD and OBR ‘scratch’ NAS.

Additional Slides © Crown copyright Met Office

Enterprise Grade Hardware • DL 360 Exsi hypervisor • Vsphere Client • VM Workstation – allow user to produce own virtual machines - Hardware independent • Sophos – scan incoming data © Crown copyright Met Office

Enterprise Grade Hardware • • DL 360 Exsi hypervisor Vsphere Client VM Workstation Sophos

Aircraft Instrumentation Networked: Removable hard disk Mission Data System (Optional UPS) Other transfer media Processing Server: Win 7, TARDIS, Sophos VM, other VM NAS storage, RAID 5 Network Switch Met. Data, Met Office metresearch. com, External Internet Connection © Crown copyright Met Office Non-Networked: v. LAN, wifi? External data storage, KVM, disp keyboar Optional X -Kryptor Dumb terminals, laptops, tablets, etc www

Virtual disk DL 360 Exsi hypervisor VSphere client 4 port LAN Network switch V-LAN Internet Met Office VM win 7 VM linu Sophos station plate © Crown ysical copyright VM win 7 Cloud physics Core Virtual network switch