ICAT Overview Tom Griffin ISIS Facility ICAT Developer

ICAT Overview Tom Griffin, ISIS Facility ICAT Developer Workshop The Cosener’s House, Abingdon August 2009 tom. griffin@stfc. ac. uk

The Problem(s) ICAT • • • • Large Data Volumes High Throughput Proliferation of data formats Multiple Data Analysis Step Increasing complexity of data Data Access requirements (Sharing and Restriction) Versioning of data formats and associated software Distributed Computation (accessed offline from research chain) Common names and units for temperature, pressure etc. Changing / differing metadata requirements International users / federation of data from facilities Relating to Proposals and Publications Ontologies Provenance (Creation, Ownership, History) Governments want return on investment

What is ICAT ? ICAT is a database (with a well defined API) that provides a uniform interface to experimental data and a mechanism to link all aspects of research from proposal through to publication. Access data anywhere via the web Annotate your data Search for data in a meaningful way e. g. taxonomy, Sample, temperature, pressure etc Share data with colleagues Access data via your own programs (C++, Fortran, Java etc. ) via the ICAT API Identify potential collaborations Utilise integrated e-Science High. Performance Computing and Visualisation resources Link to data from your publications Etc. Example ISIS Proposal What is ICAT? ICAT GEM – High intensity, high resolution neutron diffractometer Proposals Experiment Once awarded beamtime at ISIS, an entry will be created in ICAT that describes your proposed experiment. Data collected from your experiment will be indexed by ICAT (with additional experimental conditions) and made available to your experimental team H 2 -(zeolite) vibrational frequencies vs polarising potential of cations Analysed Data You will have the capability to upload any desired analysed data and associate it with your experiments. B-lactoglobulin protein interfacial structure Publication Using ICAT you will also be able to associate publications to your experiment and even reference data from your publications.

Overview ICAT User Database System Single Sign On Data Storage/ Delivery System Proposal System Publication System ICAT API e-Science Services RDBMS Software Repository Web Services API Command Line Tools Fortran C++ Glassfish / JBOSS Java

Federation ICAT User Database System Single Sign On SNS ISIS User Database System Single Sign On Data Storage/ Delivery System User Database System Proposal System Data Storage/ Delivery System Single Sign On Proposal System ANSTO Data Storage/ Delivery System Publication Proposal System ICAT API Publicatio n System ICAT API e-Science Services RDBMS Software Repositor y e-Science Services RDBMS Web Services API Software Repositor y Repository Web Services API Data Portal Top. Cat e-Science Services

Data Model ICAT Name Parent Id Topic Level User Id Role e. g Admin, Deleter, Updater, Reader, Creater, Downloader etc. Element Type Element Id Full Reference Topic URL Repository Authorisatio n Name Sample Id Description Name Description Version Location Format Source Datafile Id Format Version Destination Datafile Id Create Time Relation Modify Time S/W Application Size S/W Version Checksum Publication Investigation Dataset Keyword Reference / Proposal Id Investigator Previous Reference Facility Instrument Title Abstract Sample Etc. Datafile Dataset Parameter Related Datafile Parameter Name User Id Role Name Sample Chemical Formula Parameter Safety Information Name Units Parameter String Value Numeric Value Range Top Range Bottom Name Error Units String Value Numeric Value Range Top Range Bottom Error Name Units String Value Numeric Value Range Top Range Bottom Name/Units/Value etc Error Searchable Is Sample Parameter Is Dataset Parameter Is Datafile Parameter Verified

Investigation Facility: ISIS Instrument: MERLIN Title: Si. Mn. Si 2 100 mev 8 s 300 k in CCR 45 x 45 mm inv_type: experiment Bcat_inv_string: Mark Dr A - Universit. Dr A, , Dataset Name: Default, Type: experiment_raw Dataset_Status: complete Description: MER 03766 Mark Dr A Si. Mn. Si 2 100 mev 8 s 300 k Datafile Name: MER 03790. raw Desc: Yb 0. 9 Y 0. 1 In. Cu 4 15 me. V 4 S 40 K 3 Kbar Cu. Be cell 10 x 22 mm Format: isis neutron raw Keyword: Name: RAL Name: g_large Name: OSIRIS Name: YCo 3 D 1. 3 Sample: Name: Vanadium L 2=158 (Gm=91) Investigation Investigator Dataset Sample Parameter Datafile Dataset Parameter Datafile parameter: Name: total_proton _charge Units: u. Amp. Hours Value: 0. 233844 Datafile Parameter Sample Parameters: Name: sample_state Units: N/A String value: powder Name: sample_sitution Units: N/A String value: CCR

ICAT API ICAT • Service Oriented Architecture – Services exposed as Web Services – User required to authenticate in order to obtain Session Token – Token is used in all subsequent API calls to for authorisation • The API is modular in order to fit the needs of the facilities – Plugin own user database – Plugin data delivery system • Chracteristics – – Platform independent [Java] Application Server independent [EJB 3] Database Independent (Almost!) [JPL] Language independent [Web Services] • Internals – Core functionality implemented as POJOs using JPA – For deployment EJB 3 Session Beans bind the core API, user db and data delivery aspects together – Services are unit tested using JUNIT – Services are logged at every interaction point using LOG 4 J

ICAT API Continued ICAT

ICAT Client ICAT

Data Portal ICAT

Security ICAT • Role based permissions – – – – [Super] Admin Create Delete Update Download Read • Data Policy – – – • SSL 3 year embargo on data (+1 if requested) Commercial data is never made public Instrument Scientists can access all data from their beamline Calibration data is public Any data that involves IPR (e. g. analysed) is private for perpetuity unless explicitly shared by user

Installation / Development ICAT Installation » » » Any O/S Oracle 10 G/11 G Java 6 Update 6 Apache Ant v 1. 7+ Glassfish v 2 UR 2 Installed & Configured Cog Kit » Unzip download bundle » Update properties files e. g. database details » Run Ant commands Development Technologies Used – – – – Java Net. Beans 6. 1 Glassfish UR 2 Ant JUnit JMeter Log 4 J EJB 3 JPA JAX-WS JAXB Oracle (10 G / 11 G) Subversion

User Database ICAT

Data Delivery ICAT 1 User performs search via application e. g. Data 2 3 Permitted results are returned to application 4 Results are displayed to the user 5 6 7 User performs request to download datafile, multiple datafiles or dataset ICAT creates http GET link and passes to back to user (routed through application) session. Id email (optional) file. Id(s) or dataset. Id action (i. e. download, zip, compressed) User clicks http link 8 Data. ISIS call ICAT API to check permissions session. Id & datafile. Id(s) or dataset. Id 9 1 0 7 Portal Search is executed in ICAT Return Exception on failure or Download. Object on success - user. Id - array [filename, cycle, run number] User gets their data! 1 0 5 Data. ISIS 8 1 4 Data Portal 2 3 ICAT API 6 9

Data Delivery Continued ICAT

XML Ingest ICAT User Database System Single Sign On Validation Data Storage/ Delivery System Proposal System Publication System XSD ICAT API Investigation. Id RDBMS XMLIngest(xml) Web Services API Client e-Science Services Software Repository

ISIS Integration ICAT Trigger • NXIngest • Raw. Ingest

Developers ICAT

Future Developments ICAT • Design and develop new interface • Release Top. Cat to ISIS users • Move XML Ingest into asynchonous Message Driven Bean • Rule-based policy implementation • Expand improve the supplied interface • Proposal System integration • Publication System integration • Database independent • Derived and simulated data upload • Consequence… • Look at issue/tickets & forum!

Damian Flannery Summary ICAT • At ISIS – – Volume of data ~4 TB ~3 M datafiles (22 instruments, 330/hour) 6. 7 GB metadata, 33 M rows 550+ unit & stress tests • Attempt to solve problems as outlined earlier in this talk • Software characteristics – Scalability – Maintainability – Reliability – Availability – Extensibility – Performance – Manageability – Security • We want to drive this forward • We would like to do it in collaboration with other facilities

Damian Flannery Acknowledgements ICAT • ISIS – Damian Flannery, Robert Mc. Greevy, Kenneth Shankland, Stuart Ansell – Freddie Akeroyd, Chris Moreton-Smith, Matt Clarke, Kevin Knowles, Steven King, Adrian Hillier, Alex Hannon, Rob Dalgleish • e-Science – Glen Drinkwater, Shoaib Sufi, Kerstin Kleese Van Dam, Laurent Lerusse, Rik Tyer, Phil Couch – Gordon Brown, Kier Hawker, Carmine Coiffe – Roger Downing

Damian Flannery Questions ICAT http: //code. google. com/p/icatproject