ICAT developper Workshop 25 26 August 2009 Integration of

ICAT developper Workshop (25/26 August 2009) Integration of the ICAT API at Institut Laue-Langevin (ILL) (Grenoble, France) Tuesday, August 25, 2009 Jean-François Perrin

Agenda • Overview (ILL - Project) • Data(file) import (ILL data import into ICAT) • User Interface (web-based user Frontend) • Conclusions (Problems, Questions)

IT Service ILL • Besides two other IT services at ILL: – SCI (Instrument Control) and CS (Computing for science) • Responsibilities (groups): – – – Infrastructure (network and phone) DB Usersupport Developpement Administration • ICAT Team (Developpement group): – Holger Gebhard – Najor Cruz-Cruz (since July)

ILL-ICAT-Project • Current state at ILL (IDA application): – text and script based system, – user must know how to retrieve date (not a search) • 3 steps to integrate ICAT at ILL – replace existing (IDA) functionalities – public, more comfortable access for a larger group of users – Extend meta-data base and search functionalities • Practical point of view Installation and exploration data-import GUI (Web App)

Data Import 1 • General: – Since 1973 all ILL data are available online – about 40 active instruments (60 overall) – Filesystem organised by cycle – instrument stored on NFS (sometimes CIFS) – Total number of files: 29. 145. 595 (148 cycles) – Average size: 229 kb max ~ 40 mb (compressed) • ILL formats: – Historical ILL Formats (ascii) – Nexus (2 Instruments) – edf (proprietary binary format) • Impacts for ICAT integration at ILL relevant parameters are specific to (groups) of instrument(s). Need for expert advice

Data Import 2 Example: Import first cycle of 2006 for instrument d 22 No relation to proposals IDA replacement ! Analyse file level Analyse directory level …/061/d 22/ 015673. Z … 015713. Z 021354. Z …. 020967. Z … 021354. Z Read parameters and metadata One specific parser class per (group) of instrument(s) Integrate found • Data files • Data sets • Investigations into Database Map files to dataset

Data Import 3 Used (filled) tables: • DATASET • • DATAFILE_PARAMETER • INSTRUMENT (# 110, 61 activ) • INVESTIGATION • FACILITY_CYCLE (#142) • ICAT_AUTHORIZATION: • INVESTIGATION_TYPE – (user_id = any, role=downloader) • PARAMETER – (ex. wavelength, pressure, voltage) • DATAFILE_FORMAT • ILL Nexus, ILL ASCII • DATASET_STATUS (complete) • DATASET_TYPE(experiment_raw) • USER_ROLES (some testusers)

ILL User Interface Available in a stand-alone application and/or integrated into intra-/internet As described in documentation: Compile WSDL to generate remote proxy and classes for business delegates Call remote proxy methods to interact with ICAT API

Feedback • Tests and developments done so far worked fine for us • ICAT Installation: straightforward – Database Scripts, ICAT API – Test with a dummy prototype • Certificate handling and user authentication during installation • No performance critical tests

ILL’s open issues ICAT integration: • Mapping between data files and proposal data no proper notion of "ICAT investigation" at ILL • Specimen data (and others) are stored with proposal • Control of data format and parameter variety General • ILL Data Policy – data ownership – delay until public availability • Unification of data file formats – Competence and responsibility – Complexity

Some open issues • Search by parameter in ICAT • Future prospect: Opening ICAT’s Authentication mechanism to ILL’s SSO/LDAP • Future prospect: Search between ICAT facilities repositories

Thanks Comments? Questions?

Technical envorinment • Hardware and Database – Virtual Server (Suse SLES 10. 2) for ICAT – Oracle 10 g (Cluster RAC) as DB • Development Environment: – Java 1. 6, – Glassfish 1. 4, – Ant 1. 7. 1 • ICAT version: 3. 3. 3

Question details searchable parameters Parameter table: ‘Wavelength’ , ‘Å’, ‘Amstrong’ , 02: 20, ‘NAJOR’, Y, N Y, Y, N, Y, Y, ‘Desc Wavelenght’, Y, 999, GEBHARD, 31/07/09 02: 20, 31/07/09 Also: Voltage and Pressure Join to tables: data_file_parameter and data_set_parameter Known searches: • Search. By. Advanced • Search. By. Keywords • Search. By. Run. Number • Search. By. User. Id • Search. By. User. Sur. Name • Search. Datasets. By. Sample Advanced Search Fields private java. lang. String datafile. Name; private java. util. Calendar date. Range. End; private java. util. Calendar date. Range. Start; private java. lang. String experiment. Number; private java. lang. Long grant. Id; private java. lang. String[] instruments; private java. lang. String investigation. Abstract; private uk. icat 3. client. Investigation. Include investigation. Include; private java. lang. String investigation. Name; private java. lang. String investigation. Type; private java. lang. String[] investigators; private java. lang. String[] keywords; private java. lang. Double run. End; private java. lang. Double run. Start; private java. lang. String sample. Name; private java. lang. String visit. Id;

ILL-ICAT-Project 2 From the practical point of view • ICAT installation and exploration • Data Import (File metadata to ICAT, ~ 10 of 40 instruments) • Webbased User Interface (Simple version, planned for end of the year) • Extension of metadata and userinterface Automated, continious data import (all intruments, all cycles)

Data File example RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR 068762 0 4 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Inst User L. C. Date Time D 19 tf sam 14 -May-09 10: 16: 11 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA 80 1 Title Scantype backgr om-mono IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 31 4 nvers ntype kctrl manip nbang nkmes npdone jcode ipara ianal imode itgv iregul ivolt naxe npstart ilast 1 isa flgkif ih ik nbsqs nb_det nbdata icdesc 1 icdesc 2 icdesc 3 icdesc 4 icdesc 5 icdesc 6 4 2 4 8 1 21 21 1 1 0 0 0 0 1 163840 8 0 0 0 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 50 10 H (Hmin) K (Kmin) L (Lmin) phi chi omega 2 theta (gamma) psi ub(1, 1) ub(1, 2) ub(1, 3) ub(2, 1) ub(2, 2) ub(2, 3) ub(3, 1) ub(3, 2) ub(3, 3) wavelength dmonochromator danalyser energy Hmax Kmax Lmax Delta. H -0. 10000000 E+01 0. 50000000 E+00 -0. 49880001 E+02 0. 15721400 E+03 0. 23701000 E+02 0. 64004000 E+02 0. 0000 E+00 0. 12238280 E+00 0. 53997743 E-01 -0. 64825200 E-02 -0. 53893240 E-01 0. 12252440 E+00 0. 37250000 E-02 0. 69433160 E-02 -0. 74291900 E-03 0. 14317410 E+00 0. 24220001 E+01 0. 10000000 E+01 0. 00000000 E+00 0. 0000 E+00