A Workshop for Planning the SCAR Scientific Programme

A Workshop for Planning the SCAR Scientific Programme Interhemispheric Conjugacy Effects in Solar-Terrestrial and Aeronomy Research April 22 -23, 2004, Villefranche sur Mer, France ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov VGMO. NET A VIRTUAL GLOBAL MAGNETIC OBSERVATORY Concept and Realization vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page of 15 Vladimir Papitashvili, Anshuman Saxena*, Valeriy Petrov, Robert Clauer, and Natalia Papitashvili** Space Physics Research Laboratory University of Michigan, Ann Arbor, MI, U. S. A. http: //maggy. engin. umich. edu/mist/vgmo. html * Now at TATA Consultancy Services, Bangalore, India ** NSSDC and QSS, Inc. , NASA/GSFC, Greenbelt, MD, U. S. A.

To get scientific data from various, mostly distributed sources, a scientist may have to: ICESTAR VGMO. NET 1. Search through a number of data centers, various institutions, observatories, contact colleagues… Hi, Bob, could you send me data? Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 2 of 15 2. Get data via snail-mail, air-mail, e-mail, Web… Oh, where are data? 4. Process collected data using mostly proprietary codes, run models… and… 3. Then ingest retrieved data into a local database… Ugh, done! 5. Finally, get something meaningful or meaningless… Hurrah! Sounds “exciting”… right? Yes, searching and then converting data from various formats into something useful for local processing and analysis – this can be time consuming, labor intensive, and often frustrating if collected data do not deliver what you have expected!

Current Forms of Sharing Geomagnetic Data: ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page of 15 Centralized distribution schemes – e. g. , World Data Centers (WDC) n Require persistent support for data acquisition, storage, and distribution n Submission of data remains voluntary n Often data are not suitable for submission – e. g. , WDCs only accept absolute geomagnetic measurements; however, geomagnetic data collected outside of standard magnetic observatories are mostly variational in nature Personal Communication – exchange of discs, customized FTP access, etc. n Communication overhead – significant delays between responses n Any change in the structure of remote databases requires renegotiation for access n Need to develop more contacts n Remember - people often change their minds and responsibilities Publishing data through World Wide Web n In order to avoid additional steps of data preparation for submission to WDC n To achieve greater visibility amongst the scientific and user communities n Increased computing power and bandwidth – lots of untapped potential at the edge of networking Such diversity of available geomagnetic databases argues for more sophisticated search engines capable of identifying geomagnetic data repositories and then retrieving data for scientific analyses

VGMO. NET Virtual Global Magnetic Observatory ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov VGMO. NET is a middleware architecture that provides a new way for the worldwide geomagnetic community to share data and functionality in a platform-independent and location-neutral environment Design Goals n Identify prospective geomagnetic data repositories and provide transparent access to these remote databases through a common interface n Perform online processing of the acquired data sets n Ability to construct self-populating databases on individual machines; these selfpopulated sub-centers can be made available to other users through future request chains (i. e. , building a GRID-type access and computing) vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov FFMN IDL Matlab Simulink FORMAT CONVERSION (A 2 F) DATA ACQUISITION (HTTP/FTP/OPe. NDAP/…. ) LOCATION DISCOVERY (Geo. Ma. C) April 23, 2004 Page 4 of 15 A four-tier architecture of VGMO. NET ü Lowest layer - Location Discovery Module, Geo. Magnetic Crawler (Geo. Ma. C), which continuously forages Web for prospective geomagnetic data repositories ü Data Acquisition downloads requested data from remote sites using specific site information from the shared data structures ü A 2 F converts downloaded data into a Flat. File Format (or even to a proprietary format) ü Integrated Visualization Layer, e. g. , Flat File Manager (FFMN) or other tools, can be used to perform analysis of data

VGMO. NET - The Local Database ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 5 of 15 o Geomagnetic data are published in widely different, often proprietary formats o We convert all downloaded data sets into a Flat-File database o Databases built via VGMO. NET conform to the Flat-File DBMS architecture Flat DBMS revisited [A. Smith, C. R. Clauer, 1984] o Each dataset consists of two files: a header file, which is an ASCII description of the dataset and a binary data file that is the data itself o Leverages advantages of ASCII presentation (readable and editable data description), as well as binary presentation (compact data storage and fast random access) o A sample header file: Name of header and data files: Date files created: Record length of data file, in bytes: Number of columns: Number of rows: Flag for missing data: # 1 2 3 4 name Time VOCE VOSH VOSZ units seconds n. T source type T Antarctic magnetometer R VOS 01 13 -May-2002 20 4 3137310 -0. 10 E+33 loc 1 9 13 17 NOTES: Start time = 01 -JAN-01 00: 02: 00. 000 End time = 31 -DEC-01 23: 58: 00. 000 Antarctic magnetometer high resolution data END Note that the local database might hold a mixture of various data (flat files): the interplanetary magnetic field/solar wind data, ionospheric data, etc.

VGMO. NET - The Local Database (cont’d) ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page of 15 Directory structure and naming convention File Name consists of three parts – a station IAGA 3 -letter code, followed by a timestamp in YYYYMMDD format and some special tags that are attached for housekeeping purposes: Special Tags: absolute measurements: a variation measurements: v public access: p restricted access: r rate of data sampling (in sec): 60/30/1/ For example, a publicly accessible dataset consisting of 60 -sec samples of absolute geomagnetic measurements from Antarctic magnetic observatory VOSTOK for December 2002 will be stored in the flat files named: 20006MAGVOS 2000600_60 pa. hed VOS 2000600_60 pa. dat

VGMO. NET - Architecture Unleashed ICESTAR A 2 F - Any to Flat File Conversion Module Active Section VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Remote Site Info Format Info ftp. dmi. dk 19802002 /pub/wdcc 1/obsdata/1 minval/ YYYY/ ftp. ngdc. noaa. gov 19702002 /STP/GEOMAGNETIC_DATA/ONE_ MINUTE_VALUES/YYYY/ …………………… Valeriy Petrov ……… INTERNET vpetrov@engin. umich. edu Robert Clauer Conversion Pointer Prospective Section. rclauer@umich. edu Remote Site Info Format Info Conversion Pointer Natalia Papitashvili ftp. iki. rssi. ru - - - ftp. abs. xyz. edu - - - natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 7 of 15 Geo Magnetic Crawler (Geo. Ma. C) LOOKUP TABLE FFMN Flat File Manager

Two Implementations of VGMO. NET ICESTAR VGMO. NET Vladimir Papitashvili Web-Based Portal – available at http: //maggy. engin. umich. edu/mist/vgmo. html Ø A secure, scalable, platform independent, and user-friendly software framework for remote access to VGMO. NET Flat File Manager Ø The Flat File Manager Client is written to the Java 2 platform that requires a Java Web Start (JNLP - Java Network Launching Protocol) papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 8 of 15 Self-Populating Standalone Version - available for download from above Web site Ø An alternate standalone version to create, manage, and populate local geomagnetic databases from INTERNET; aims on building a geomagnetic GRID access

VGMO. NET Highlights http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Remote (Client) Machine Requirements • Java Runtime Environment (JRE), version 1. 2. 2 or later • Java Web Start (available for Windows 98/ME/NT/2000/XP, Linux, and Solaris OE) • The library and “Java thin client” for the FFMN Client Server Requirements • Any standard Web server configured for JNLP (Java Network Launching Protocol) • Flat File Manager DLLs and Flat File Manager Server software Platform Independence • Client Side Security and Notification of Application’s Origin • The FFMN service provider signs the downloadable code to ensure that no other party can impersonate the application on the Web; thus, the VGMO framework provides flexibility without compromising security. • The user is shown a dialog displaying the application's origin (based on the signer's certificate) before the application is launched; thereby, the user can make an informed decision whether to grant additional privileges to the downloaded code • If the user trusts the FFMN service provider, he/she can choose to grant additional system privileges, such as a write access to a local disk Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 9 of 15 FFMN Server can be deployed on a wide-variety of platforms (Linux, Solaris OE, Windows 98/ME/NT/2000/XP) and launched remotely from any platform

Launching VGMO. NET http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 10 of 15 q The very first initiation of FFMN is through a Web browser. As initiated, the remote machine is checked for necessary software (e. g. , JRE, the Java Runtime Environment). If that software is not found, the user can choose an option of “One Click” installation or download various components manually and launch FFMN later. However, the “One Click” option requires least manual intervention and prepares the platform to launch FFMN remotely. q As the FFMN client software is downloaded from the Web server, it is launched locally via JNLP (Java Network Launch Protocol). Note that the downloaded application runs at the remote machine in a protective environment (“sandbox”). q The “Java thin client” opens a control connection (out-of band signaling) with the FFMN Server and instructs the server to run Flat File Manager locally but the output is posted at the remote machine. Later this control connection initiates FTP sessions on demand allows users to download requested files. q Subsequent initiations of FFMN from the remote machine can be independent of a Web browser; the application can be launched through desktop shortcuts, making launching the Web-deployed application similar to launching a native application. The user is prompted for creating a desktop shortcut in Step (1).

VGMO. NET at Work http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 11 of 15 • FFMN Main Menu allows the user to select up to three data sets (File), then do certain operations with selected data sets (Action) by setting Options • The File item allows the user to open the server database files or to create a temporary data set for the selected geomagnetic stations (selected either by names or geographic location) • If the selected data are found in the server’s database, then the FFMN Server retrieves requested data for the plotting (and possible uploading) to the remote, FFMN client machine • In addition, if the “Search worldwide” box is checked, the FFMN Server will look for the selected data on a number of remote FTP sites (listed in the FFMN Lookup File); these data are then downloaded, converted to flat files, and added to the FFMN server database • When new FTP sites with geomagnetic data are found, they can be easily linked through additions to the FFMN Lookup File

VGMO. NET Actions http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 12 of 15 FFMN Client shows all data available from the Server (that is, included in the temporary data set for the selected time interval) and then the user can Select All or mark (+) the data subset for Action • Plot –a stack of magnetograms plotted in accordance with the changeable Options menu • Download – save the selected dataset to a file (in various formats, including IAGA-2002) and then download this file to the remote machine via FTP. If there were a few opened files on the server, then all selected data will be combined into a single file • Merge rows – merge two similar data sets with interlaced time intervals • Filter – apply the low, high, or bandpass filter to the selected data • Calculate – enter a formula to compute new physical quantity from the given columns in the opened data files (e. g. , compute the total geomagnetic field intensity from three orthogonal components) • Shift time – to change timing in one of the data sets for the plotting purposes (e. g. , showing delays between data)

VGMO. NET Search and Plot Example http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page of 15

VGMO. NET: World Wide Web Search http: //maggy. engin. umich. edu/mist/vgmo. html ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 14 of 15 • By default all the sites presented in the list are contacted for world wide search • The user can drop some sites from the list by making appropriate selections • Each site remains in one of the following states Not connected : Site has not yet been contacted Connecting : Synchronization with the site is in progress Completed : Synchronization with the site has been completed • The list of matching stations found are listed against each site

Summary ICESTAR VGMO. NET Vladimir Papitashvili papita@umich. e du Anshuman Saxena anshuman. saxe na@tcs. com Valeriy Petrov vpetrov@engin. umich. edu Robert Clauer rclauer@umich. edu Natalia Papitashvili natasha@mail 6 30. gsfc. nasa. gov April 23, 2004 Page 15 of 15 Ö Existing World Data Centers continue to serve the worldwide scientific community in providing free access to global geophysical databases Ö Recently many digital geomagnetic datasets have been placed on the Web, often in near-real time, but some of these data are not even submitted to any data center Ö In this study, we formulated a concept and developed a prototype of a Virtual Global Magnetic Observatory (VGMO) that uses a pre-set list of FTP-based geomagnetic data holders to retrieve requested data Ö Saving retrieved data locally over multiple requests, a VGMO user begins to build his/her own data sub-center, which does not need searching the Web if a new data interval is within a span of earlier downloaded data Ö At the same time, these self-sustained sub-centers become available to other VGMO users - this network of “GEOMAGstered” users establishes VGMO. NET where Web data-crawling becomes transparent to the users Ö However, more studies are needed to help identifying newly “Webbed” digital geomagnetic data – and Semantic Web is the most promising approach