CIMA in Australia Ian Atkinson Ian Atkinson jcu edu

CIMA in Australia Ian Atkinson Ian. Atkinson@jcu. edu. au HPRC Manager, ITR School of Maths, Physics and IT James Cook University ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

DART: Dataset Acquisition, Acquisition & e-Research Technologies • A DEST-SII funded project (http: //www. dart. edu. au & http: //plone. jcu. edu. au/dart) to develop tools to handle the data & information management needs of diverse research environments. • DART is developing tools to handle typical research data & information management needs, such as: 1. collecting data 2. managing data 3. analyzing that data to produce useful information 4. managing that information 5. collaboration & annotation on the information 6. publishing information 7. searching on the information • DART has built three demonstrators: • Climate Research • X-ray crystallography • Indigenous Digital History ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

DART - workflows ICTP, 23 -28 April 2007

Instrument Taxonomy Rick Mc. Mullen ICTP, 23 -28 April 2007

Remote Instrumentation Goal: Develop a Grid enabled and Web services based portal system for collaborative remote access to scientific instruments and their data. Why? Maximise resource utilization and return on investment/s Outreach, diffusion, training and education Grid? Access to compute cycles for processing and modeling and flexible distributed storage management that may further facilitate remote access Multi-Site Collaboration Goal for production level service: Significant collaboration between USyd, JCU, IU (and many others: UAdel, Monash, UQ, DART…) Deployment and user focus issues (this is hard) ICTP, 23 -28 April 2007

Common Instrument Middleware Architecture (CIMA) Offers a general and re-usable model for instrument access and management using Web/Grid services; adaptable to different instrument settings Grid enablement of instruments & sensors Flexible and extensible with modular use of plug-ins, Common programmable interface – reusable code base Publish-subscribe or registration model Basis for a standardised implementation/deployment system ICTP, 23 -28 April 2007

Common Instrument Middleware Architecture (CIMA) ICTP, 23 -28 April 2007

Common Instrument Middleware Architecture (CIMA) • Integrate instruments and sensors (e. g. real-time data sources) into a grid environment with Web Services interfaces • Abstract instrument capabilities and functions to reduce data acquisition and analysis applications dependence on specialized knowledge about particular instruments • Move production of metadata as close to instruments as possible and facilitate the automatic production of metadata • Develop a standard, reusable methodology for grid enabling instruments Rick Mc. Mullen, Knowledge Acquisition and Projection Laboratory, Indiana University ICTP, 23 -28 April 2007

CIMA Elements • Schema for instrument functionality in WSDL • Data model for representing instrument metrics and calibration • A small, high performance Web Services stack – Service implementation for accessing the instruments functionality • Ability to dynamically insert new protocols into running instances • OGSA compliant functions to register with a directory service, authenticate users, provide access control to instrument controls and data, and co- schedule the instrument into a Grid computing and storage context; ICTP, 23 -28 April 2007

CIMA X-ray Portal www. instrumentmiddleware. org. au ICTP, 23 -28 April 2007

Indiana CIMA X-Ray system ● ● ● 4 Components i. Instrument Representative, ii. My Manager, iii. Data Manager, iv. Portal Interface Instrument Rep, My Manager, Data Manager Service implemented in c++, g. Soap, Perl Storage – ext 3 file-system, My. SQL database Portal – jrs 168 portlets, gridsphere Issues – My Manager & Portal implementations domain specific i. e. Crystallography vs oceanographic monitoring ICTP, 23 -28 April 2007

DART Implementation CIMA SOAP Instruments Live data feeds Interface Instrument Representative CIMA SOAP Data Manager Workflow & comp pipelines. Automatic metadata generation. Video feeds Grid. Sphere Portal PGL: SRB access & schemas Data Processing Applications, Annontation and visualisation CIMA core code from IU DART/JCU/UQ/Usyd/MU Implementation SRB HSM Storage MCAT SRB data repository (data and metadata) ICTP, 23 -28 April 2007

Data Management - workflow NFS and My. SQL replaced with Storage Resource Broker (SRB) Kepler workflow used for My. Manager Ability to customise data storage via a workflow system Highly extensible, not restricted to SRB as repository ICTP, 23 -28 April 2007

Data Management - datagrid Storage Resource Broker (SRB) Data Grid middleware from SDSC (San Diego Supercomputing Centre) • Uniform interface to heterogeneous resources over a network. • Logical View of data • User defined metadata structures • Works in conjunction with MCAT. ICTP, 23 -28 April 2007 http: //www. sdsc. edu/srb

DART - User Portal Use of Personal Grid Library (PGL) for SRB data manipulation / metadata display Metadata schema definitions applied to experimental data - chitter chatter Stored experimental data is able to be easily retrieved, secured annotated Visual diagnostics for sensors (e. g. Graphs, Visual Indicators) Live CCD image analysis (experimental status) ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007 A. Beitz, et al. Monash

ICTP, 23 -28 April 2007 J. Hunter, et al. UQ

Data Federations with SRB Initial Instrument Representatives deployed at JCU, USyd, Monash & UQ Goal Each site having their own Data Manager and SRB storage facility which is federated across sites Shibbolith based AAA and Virtual Organisations Federate/Replicate data into a a national SRB store…

Future Developments – Data federations (SRB) – Security and Rights Management • Shibboleth – Analysis tools (Portals/portlets) – Workflows • Goal is an end-to-end workbench • Extensions to larger and smaller instruments (e. g. sensors) ICTP, 23 -28 April 2007

Acknowledgements: JCU S. Ness M. Morgan F. Eilert Sydney Uni D. du Boulay C. Chee Indiana Uni K. Chiu (SUNNY) Adelaide Uni A. Wendelborn P. Coddington D. Zhang Department of Education Science and Training: Dataset Acquisition, Accessibility, and Annotation e-Research Technologies (DART) project Australian Research Council: e-Research Seed Funding Programme and the Research Networks Programme (MMSN: Molecular and Materials Structure Network). ICTP, 23 -28 April 2007

ICTP, 23 -28 April 2007