Radiation Belt Storm Probes RBSP Science Data Portal

Radiation Belt Storm Probes RBSP Science Data Portal Michele Weiss SWG Data System Technical Lead michele. weiss@jhuapl. edu 240 -228 -4806 Radiation Belt Storm Probes 1

Outline • Virtual Observatories – History and Introduction – Report on Virtual Observatories • SPASE and SPASEQL – Introduction and Definition – How the Virtual Observatories use SPASE – How SPASEQL fits in • • • Requirements on Instrument Teams Common Data Tools RBSP Science Data Portal Autoplot presentation by Jerome Faden MIDL presentation by Jon Vandegriff RBSP SWG March 4 -5, 2009 2

History of Virtual Observatories (VOs) • In 2007, NASA formalized the Heliophysics Data Environment (HPDE) in the HP Science Data Management Policy. This was endorsed by the Heliophysics Data and Computing Working Group. – http: //hpde. gsfc. nasa. gov/index. html • Newest components of the HPDE are discipline specific Virtual Observatories (Vx. Os) and post-mission Resident Archives – History: • VSO approved by a Senior Review in 2001; Initial studies led to full funding in 2003 (granddaddy of Heliophysics VOs) • 5 Vx. Os funded in 2006/2007. These are currently in their 3 rd year of funding. • Virtual Energetic Particle Observatory (VEPO) - funded 2008, Virtual Wave Observatory (VWO) - funded 2008 and Virtual Model Repository – funded 2009 – In initial phases, but talk to them if you think your data would fit well there. RBSP SWG March 4 -5, 2009 3

Current VOs of Potential Interest to RBSP – RBSP team met w/ each of these VOs • • • Virtual Heliospheric Observatory (VHO) – PI: Adam Szabo – NASA GSFC – http: //vho. nasa. gov/ Virtual Magnetospheric Observatory (VMO) – GSFC – PI: Jan Merka – NASA GSFC – http: //vmo. nasa. gov/ Virtual Magnetospheric Observatory – UCLA – PI: Ray Walker – Institute of Geophysics and Planetary Physics – UCLA – http: //vmo. igpp. ucla. edu/ Virtual Ionosphere Thermosphere Observatory (VITMO) – PI: Daniel Morrison – JHU/APL – http: //vitmo. jhuapl. edu/ Virtual Radiation Belt Observatory (Vi. RBO) – PI: Bob Weigel – George Mason University – http: //virbo. org RBSP SWG March 4 -5, 2009 4

Virtual Heliospheric Observatory (VHO) & Virtual Magnetospheric Observatory (VMO) – GSFC http: //vmo. nasa. gov/ and http: //vho. nasa. gov/ Why use the VHO/VMO? • Latest data available • Descriptions associated with data products • Comprehensive data collection • Data search uses multiple criteria – Date/time – Product name – Measurement type – Parameter values (statistical estimators) – Observatory positions – … more are coming: event lists, regions, . . . RBSP SWG March 4 -5, 2009 5

VHO/VMO – Pre-calculated Statistical Estimators • • Used in parameter value searches as proxies A unique feature of VHO/VMO RBSP SWG March 4 -5, 2009 6

VHO/VMO – Data Finder RBSP SWG March 4 -5, 2009 7

Virtual Magnetospheric Observatory (VMO) – UCLA http: //vmo. igpp. ucla. edu/ The Virtual Magnetospheric Observatory (VMO) is a distributed data environment with a web -accessible graphical user interface and application programming interface that provides a unified data discovery and retrieval. The VMO offers both services and data. Services • Support – – • Search – – – • Structured/Form Word-based (google-like) SPASE-QL Download – – • Domain experts (X-men) Generation, validation, referential tools. Granules Collections (multi-granule) Rendering – Autoplot Data Spacecraft AMPTE_CCE AMPTE_IRM AMPTE_UKS DE-1 DE-2 GOES 10 GOES 8 GOES 9 Geotail IMP 8 Groundbased Augsburg ICESTAR RBSP SWG March 4 -5, 2009 Interball-1 Interball-Tail ISEE-1 ISEE-2 POLAR ST 5 -094 ST 5 -224 THEMIS Wind UCLA WDC 8

VMO Homepage (UCLA & GSFC) – they share the same web entry point Software Search For Data All news Our Vision Helpful Information New Additions Current Events RBSP SWG March 4 -5, 2009 9

Virtual Ionosphere Thermosphere Observatory (VITMO) http: //vitmo. jhuapl. edu/ 1. Interface that is keywords-based using keywords that cover phenomena, fundamental physical properties, and common data measurements. 10 RBSP SWG March 4 -5, 2009 2. Enhanced search services – coincidences between satellite and ground and restricted by geophysical conditions. 3. Single data delivery mechanism for all data, pointers to tools and documentation. 4. Support for Summary Images and Search for Data by Browsing Images 5. Data providers web sites integrated into data browse process. 10

VITMO Integrates Data Providers Web Site into the Browse Process RBSP SWG March 4 -5, 2009 11

Virtual Radiation Belt Observatory (Vi. RBO) http: //virbo. org Vi. RBO Capabilities/Advantages Data • An expanding list of data that we are continually making more accessible (for example, all 2 -second NOAA SEM 2 data, along with corrected data, including NOAA 15, 16, 17, 18, and Met. Op 02 are available in merged CDF files. Previously only available in daily binary files. ) • Early work with Vi. RBO will simplify two-way RBSP/Vi. RBO connections at both the metadata and data level. Metadata • Workflow for contributing or editing existing metadata by entering a web form. See http: //virbo. org/meta/ • Experience with SPASE metadata issues involved with particle data RBSP SWG March 4 -5, 2009 12

Vi. RBO – Vi. RBO Capabilities/Advantages Server • Using OPe. NDAP and a few extensions developed by a former OPe. NDAP employee (D. Lindholm at CU/LASP) that will end up in the OPe. NDAP distribution. All server features are not yet active. Expected completion by June. Demos available by request. • Turns data browsing inside-out: instead of viewing data day-by-day, perform interactive drill down from full time range. (See Autoplot demo by J. Faden). • Allows range requests and data subsetting (e. g. , time>2001 -0101&flux>1 e 6) with expected response times and non-trivial data volumes. • Data requests will return a short IDL or MATLAB script that hides the data transport from the user. (Standard file output will be supported. ) • For performance in plotting and subsetting, server uses caching on data as needed and as allowed. (Traditional long list of files associated with query is still an option. ) Visualization • Autoplot design and implementation effort led by Vi. RBO. RBSP SWG March 4 -5, 2009 13

Vi. RBO – Vi. RBO Capabilities/Advantages - continued Other • Host a SVN and GIT repository and have a wiki page that is used for meetings - increasing visibility on Radiation Belt searches. • Thinking a lot about the issue of how to make contributing data simpler. The current model will not scale to 100 s of active small satellites. Idea would be to provide a simple IDL/MATLAB script that allows the user to “push” data into our cache or permanent storage. Once data is there, all high-level services are available. Working with Das 2/QData and OPe. NDAP developers on this. RBSP SWG March 4 -5, 2009 14

What a Virtual Observatory does for you • • • VOs are continuing to evolve but all have the following common goals Queryable data catalog of distributed space physics data products within a single user interface Allow scientists to access data from many missions from a single “data portal” or even from their own s/w applications directly Simple, uniform access to data from distributed, heterogeneous sources can also enable services such as visualizations, or format translation that enhance the use of these data May provide tools for cross-mission analysis and visualization Enhances ability to obtain and use data efficiently across a broad range of observatories, instruments and data formats Enables integration of complex data systems Could eliminate need for specialized integration software Promotes the distribution of your data RBSP SWG March 4 -5, 2009 15

What a Virtual Observatory does NOT do for you • Store your data – Most VOs only store information or metadata about the data product, rather than the data itself. This will include the following types of metadata: • Descriptive metadata typically includes: – Program, satellite and instrument information, as well as data details like observed phenomena, units of measure, time and spatial coverages, provenance, and processing level • Structural or access metadata typically includes: – Details for data access and retrieval – Resolution of data product • Ancillary metadata typically includes: – Services and tools for a data product – Some VOs may store your data when it doesn’t have a home anywhere else • Strong interaction necessary between you and VO(s) to properly describe your data • The better your metadata, the better a VO and a VO user can use your data RBSP SWG March 4 -5, 2009 16

Making your data available to a VO(s) • It is your responsibility to make your data available to the VO(s) • To make your data available to VO(s): – Your data must be already publicly available. The mechanism, as supported by the VO you are delivering your products to, may include: • • ftp http (i. e. a website) a service a database – Metadata must exist and be current for your data • Metadata requirements are determined by the VO you are delivering your products to • Most VOs use SPASE to define data products metadata • More on metadata and SPASE in subsequent slides • You can make your data available to multiple VOs but each one may require different metadata standards and protocols RBSP SWG March 4 -5, 2009 17

RBSP Project and Virtual Observatories • RBSP Science Data Management Plan received instrument team buy-in to make their products available to a VO • RBSP Project will not recommend a VO or VOs, however it is hoped that at least one VO (a “universal” VO) will contain all of the data products for the entire RBSP mission • RBSP SWG will be the decider of a “universal” RBSP VO RBSP SWG March 4 -5, 2009 18

Future of Vx. Os • The 5 VOs identified here will be part of a Senior Review proposal in May, 2009 • Results of proposal will identify the future level of funding for the VOs as well as help determine their long-term future • NASA Heliophysics Science Data Management Policy states: – “The data from missions is to be made available by them both directly and via Virtual Observatories (VOs) that will provide one-stop access to data from many missions along with tools for cross-mission analysis and visualization. NASA HQ will provide the vision for the DE and ensure, with community input from the HP Data and Computing Working Group (HPDCWG) and Senior Reviews of both missions and Data Centers, that the vision is carried out. NASA Data Centers will provide various services, including archives and cross-disciplinary access to data and services, and coordination of the VOs and other HPDE components. All components of the HPDE will involve competitive selections and reviews to ensure the best quality. ” RBSP SWG March 4 -5, 2009 19

Which VO(s) to use? • Each VO is currently discipline specific – Large overlap across the disciplines • Look at the VOs – URLs are provided here as well as PIs and institutions – Look at datasets they are currently providing and plan to provide – Look at ancillary tools and services that may be of utility to you • Talk to the individual VOs – Many of you are already affiliated w/ one or even several – PIs and affiliations are provided here • Currently at differing levels of maturity – This will change significantly by the time RBSP is flying • Note: AGU meetings usually have at least one VO session per meeting in either SA, SH, SM or IN (Earth Space Science Informatics) RBSP SWG March 4 -5, 2009 20

Spase Physics Search and Extract (SPASE) http: //www. spase-group. org/ • • • Defines metadata that describes Space & Solar Physics datasets SPASE is a consortium consisting of an international team of space and solar physicists and information scientists Provides a unified method to describe data and other resources SPASE data model is comparable to the data models developed by PDS and the International Virtual Observatory Alliance (IVOA) for planetary and astronomical data Major creators of SPASE-based product descriptions are expected to be the VOs, data centers and individual data and model providers – Which could be you if your selected VO does not do it for you • SPASE data model tries to provide enough detail to allow a scientist to understand the content of data products as well as access and contact information – i. e. a set of files for 3 second resolution Geotail magnetic field data for 1992 to 2005 RBSP SWG March 4 -5, 2009 21

SPASE - continued • Most VOs are: – Using SPASE to define data product metadata • VITMO and VSO are current exceptions however their metadata needs are similar to SPASE – Currently defining metadata for data providers. This may change so you may be responsible for your own product metadata in the future. – Typically, only need one metadata file for every different type of data product that you produce and make available through a VO • i. e. , if you are producing 5 different data products 15 times a day for the life of the mission, you would only need 5 metadata files and your metadata file could specify the frequency of the data product and the resolution of the product – Some of the VOs may have finer granularity on their metadata definitions of your data products • It is up to you and your selected VO(s) to adequately define your data products to make them useful to both you and your users – Your data products are only as good as their metadata RBSP SWG March 4 -5, 2009 22

SPASE Metadata • The following SPASE metadata examples are one VO’s implementation of SPASE – the VHO • Each VO may have their own implementation of SPASE so may require different fields • SPASE data model is available at http: //www. spase-group. org/ RBSP SWG March 4 -5, 2009 23

SPASE Metadata Example – Spacecraft/Observatory • SPASE record describing the ACE spacecraft RBSP SWG March 4 -5, 2009 24

SPASE Metadata Example - Instrument • SPASE record describing the ACE EPAM instrument RBSP SWG March 4 -5, 2009 25

SPASE Metadata Example – Numerical Data • SPASE record describing the ACE EPAM 1 -hour Level 2 data product RBSP SWG March 4 -5, 2009 26

SPASE Metadata Example – Physical Parameters • SPASE record describing some of the parameters whose values are part of the ACE EPAM 1 -hour Level 2 data product RBSP SWG March 4 -5, 2009 27

SPASE Query Language (SPASEQL) • SPASEQL is based on the SPASE metadata model • SPASEQL will allow the VOs to communicate with each other in a standardized manner • User would go into their favorite VO to initiate a query. SPASEQL would then provide a frontend to each VO that allows them to speak the same language to each other so your VO could then query another VO to get the data you are requesting • As a data provider, you will never need to implement SPASEQL or learn anything about it. It is simply a VO to VO language and will be transparent to you RBSP SWG March 4 -5, 2009 28

Requirements on RBSP Instrument Teams • You should make your public data available to at least one VO or possibly multiple VOs • RBSP SWG should select at least one “universal” VO that all RBSP data gets sent to • You will need to work with your selected VO(s) to ensure that the metadata accurately and adequately defines and provides access to your data products RBSP SWG March 4 -5, 2009 29

Common Tools that might be of Interest • Look directions & calibrations • Physical & empirical fields and particles models to both understand model the radiation belts • Orientation of spacecraft • Energy flux & pitch angle distribution plots w/ magnetic field data • Alternative sources for time conversions • Examples on the following slides • More on Autoplot and MIDL later • Discussion on tools of interest • Discussion on RBSP Science Data Portal RBSP SWG March 4 -5, 2009 30

Common Tools that might be of Interest Orbit viewer • Orientation of spacecraft • Coincidence locator for other relevant missions • Overlays of relevant models Coincidence locator for other relevant missions Planning tools • Satellite locations • Coordinated observations calendar View from north pole along with orbit RBSP SWG March 4 -5, 2009 31

Common Tools of Interest - continued 2 -D pitch angle plots Energy flux & pitch angle distribution plots w/ magnetic field data Field model at a given L-Shell Angle to Angle plots RBSP SWG March 4 -5, 2009 32

Common Tools of Interest - continued Magnetic field line trace of a simple dipole Magnetic field line trace of a more realistic magnetic field model RBSP SWG March 4 -5, 2009 33

Common Tools of Interest - continued Solar & Geomagnetic Indices Event Lists Planned and as flown Timelines IMF Data RBSP SWG March 4 -5, 2009 34

Common Tools of Interest - continued Publication List Documentation RBSP SWG March 4 -5, 2009 35

RBSP Science Data Portal http: //rbsp. jhuapl. edu • Will not duplicate VO functionality • Comprised of things that could be of utility to the RBSP instrument teams such as: – – – – – Common tools repository Shared software repository Catalog of papers and presentations Documentation IMF data Solar & Geomagnetic indices Latest Space Weather Forecast Relevant links Etc. • Data portal is a team resource governed by RBSP SWG New tab called Data Portal RBSP SWG March 4 -5, 2009 36