Developing a Net CDF-4 Interface to HDF 5

Developing a Net. CDF-4 Interface to HDF 5 Data Russ Rew (PI), UCAR Unidata Mike Folk (Co-PI), NCSA/UIUC Ed Hartnett, UCAR Unidata Quincey Kozial, NCSA/UIUC John Caron, UCAR Unidata Robert E. Mc. Grath, NCSA/UIUC NASA award AIST-02 -0071

Unidata: A Community Endeavor • Community of educators and researchers at 120 universities, 30 other institutions, international in scope • Managed by the University Corporation for Atmospheric Research • Mission: providing data, tools, support, and community leadership for enhanced earth-system education and research • Atmospheric science community, expanding to oceanography, hydrology, other geosciences • Unidata Program Center: 25 staff, 15 2

Overview • • • What is net. CDF? What is HDF 5? Why develop a net. CDF interface to HDF 5? What is the current project status? What still needs to be done? Do we have the necessary resources? What are the prospects for success? 3

Net. CDF-3 and HDF 5 Ad hoc standards are useful standards • Standard Data Models for scientific data and data abstractions • Standard Interfaces between data providers and data users • Standard Libraries for data access from various languages • Standard Formats for portable binary data • Users need not know about the format 4

Data Models net. CDF-3 HDF 5 Variables Dimensions Attributes Datasets Dataspaces Attributes Coordinates Element types Datatypes Groups Links References Property Lists 5

Libraries net. CDF-3 HDF 5 one interface level serial I/O C, C++ Fortran-77, -90 Java (pure) high- and low-level interfaces serial. parallel (MPI) I/O C, C++ Fortran-90 Java (native) Perl Python Ruby IDL Matlab 6

Formats net. CDF-3 HDF 5 XDR direct access efficiently extendible 32 -bit file offsets XDR and native direct access efficiently extendible 64 -bit file offsets chunked access compound structures nested structures compression efficient schema changes 7

Other Characterisitics Net. CDF-3 HDF 5 Availability free Development and maintenance UCAR Unidata NCSA HDF Group Primary funding NSF NASA, DOE ASCI Advantages popular, simple, lots of tools, multiple implementations powerful, highperformance, storage efficiency, extensibility 8

Goals of Net. CDF/HDF Combination • Create net. CDF-4, combining desirable • characteristics of net. CDF-3 and HDF 5, while taking advantage of their separate strengths • Widespread use and simplicity of net. CDF-3 • Generality and performance of HDF 5 Make net. CDF more suitable for highperformance computing • Provide simple high-level interface for HDF 5 • Demonstrate benefits of combination in advanced Earth science modeling efforts 9

Net. CDF-4 Features Enabled by HDF 5 • • Large file support Parallel I/O Multiple dynamic dimensions Packed data, compression New data types Dynamic schema modifications Other possibilities: groups, user-defined types, better coordinate support, … 10

Approach • Implement net. CDF-3 over HDF 5, to • • • demonstrate backward compatibility with • Programming interface • Format Design net. CDF-4 interface Implement net. CDF-4 over HDF 5 to add enhancements made possible with HDF 5 Foster continued collaboration between Unidata and NCSA in design, development, testing, and support 11

Net. CDF-4 Architecture net. CDF-3 Interface net. CDF-4 Library HDF 5 Library • Access to net. CDF-3, net. CDF-4, and HDF 5 data created through net. CDF-4 interface 12

User View of Net. CDF-4 • Net. CDF-4 library accesses either the net. CDF-3 or HDF 5 library to read or write data 13

Current Technical Status Implement net. CDF-3 over HDF 5, to demonstrate backward compatibility with API and format done Determine needed HDF 5 enhancements done Prepare net. CDF-3 for incorporation with net. CDF-4 nearly done Design net. CDF-4 interface to add enhancements made possible with HDF 5 in progress Implement needed HDF 5 enhancements in progress Implement net. CDF-4 over enhanced HDF 5 not started yet 14

Net. CDF-3 Interface Using HDF 5 • • • 13, 000 lines of C code Passes all net. CDF-3 tests Demonstrates HDF 5 practical for net. CDF-4 Identifies HDF 5 enhancements needed Shows read/write times and file sizes satisfactory • Validates approach to backward compatibility • API compatibility: only recompilation and relinking needed for existing net. CDF-4 programs • Format compatibility: accesses all current net. CDF files as well as new HDF 5 files transparently 15

Net. CDF-3 Enhancements for Net. CDF -4 • To provide • • • stable foundation for incorporating net. CDF-4 • smooth transition for current users Automated multi-platform testing Documentation converted to maintainable form, new language-independent Users Guide Added large file support with backward compatibility Added default format interfaces Better Windows and. Net support 16

HDF 5 Additions for Supporting Net. CDF 4 • HDF 5 enhancements • • numeric type conversions zero-dimensional datasets overflow handling improvements flexible parallel I/O • HDF 5 design specifications • dimension scales for coordinate systems • shared object proposal 17

Project Schedule Currently on schedule for a July 2005 release • July 2004: version 3. 6. 0 - revised documentation, 64 -bit file offsets, default format functions • October 2004: version 3. 7. 0 - use of autotools • January 2005: version 3. 7. 1: net. CDF-4 prototype included, support for multiple unlimited dimensions • March 2005: version 4. 0. 0_beta - test relelase • July 2005: version 4. 0. 0 - first net. CDF-4 production release 18

Net. CDF-4 Design Issues • Issue: support for coordinate systems in net. CDF and HDF 5 data models? under consideration • Issue: addition of HDF 5 Groups abstraction to net. CDF data model? yes, tentatively • subset of HDF 5 Group features • constrained by backward compatibility with net. CDF-3 • no Group aliases but try to support Variable aliases and Dimension scoping? • Issue: can we just adopt Northwestern/Argonne pnet. CDF interface for adding parallel I/O? 19

What remains to be done? • Next for net. CDF-4: interface additions for multiple unlimited dimensions, group interfaces, dynamic schema modification, new data types, packed data, parallel I/O, compression • HDF 5 enhancements • zero-length attributes • shared dimensions • creation order access for objects • Testing in models (CCSM, WRF, ESMF, . . . ) 20

Papers, Posters, Presentations 1. R. Rew, M. Folk, E. Hartnett, and R. Mc. Grath: Plans for an Enhanced Net. CDF-4 Interface to HDF 5 Data. HDF/HDF-EOS Workshop VII, Silver Springs, September 2003. Poster and presentation. 2. M. Folk, R. Rew, K. Yang, R. Mc. Grath: Net. CDF-4: Combining net. CDF and HDF 5 Data. AGU Fall Meeting, San Francisco, December 2003. Poster. 3. R. Rew and E. Hartnett: Merging Net. CDF and HDF 5. 20 th International Conference on Interactive Information Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, Seattle, January 2004. Paper and poster. 4. E. Hartnett: Merging the Net. CDF and HDF 5 Libraries to Achieve Gains in Performance and Interoperability. 2004 Earth Science Technology Conference, Palo Alto, June 2004. Paper and presentation. 21

Excellent Prospects for Success • More software engineering than research • Net. CDF-4 web site just announced: • www. unidata. ucar. edu/packages/netcdf-4/ • Unidata and NCSA developers collaborating via email, teleconferences • On schedule for July 2005 release: • www. unidata. ucar. edu/packages/netcdf/release_schedul e. html • Great interest in status of project! Ultimate goal to make earth science researchers more productive. . . 22

Questions? ? 23