Astro-WISE Grid Fokke Dijkstra Donald Smits

Astro-WISE & Grid Fokke Dijkstra – Donald Smits Centre for Information Technology Andrey Belikov – Omega. CEN, Kapteyn institute University of Groningen

Target • • • Non-commercial consortium to provide infrastructure in Northern Netherlands for scientific and business applications Many different applications (Life sciences, Cultural heritage, Astronomy, . . ) Support storage and computing infrastructure Industrial partners – a. o. IBM, Oracle Collaboration with Dutch NGI Bi. G Grid http: //www. rug. nl/target 20/03/2018 2

Target infrastructure • 10 PB of storage: disk & tape – IBM GPFS file system managing this • Compute facilities – University cluster (~3000 cores) – New Grid cluster (~900 cores) • Users – LOFAR – KIDS/Viking – Other applications 20/03/2018 3

Why EGI Grid? • Standards based – Computing – Storage – Authorization/authentication • Resources available on national and international scale • Sharing resources • Challenges: – Abstraction layers make software more complex – Difficult to use for end user 20/03/2018 4

Bi. G Grid • • • Dutch NGI, part of EGI Grid infrastructure based on g. Lite ~ 9000 cores 4 main sites 5 PB online storage > 8 PB nearline storage 20/03/2018 5

Local Grid node: status • New Grid cluster at University of Groningen • 58 node cluster – 16 cores – 64 GB memory – 4 TB local disk • 10 Gb. E network • Connected to Target storage infrastructure • Used by a. o. Omega. CEN (KIDS/VIKING), LOFAR 20/03/2018 6

Astro-WISE www. astro-wise. org • • Originally: scientific information system to process widefield astronomical images Presently: system approach to development of information system for science (WISE technology) 20/03/2018 7

Astro-WISE - Grid Integration Workgroup Omega. CEN: Belikov, Vriend, Boxhoorn; CIT: Dijkstra, Dröge Astro-WISE EGI, one way full integration Access and store data on EGI, launch Astro-WISE applications on EGI Authentication & Authorization (Grid certificate + Astro-WISE A&A) How-to available on www. astro-wise. org Link from Metadatabase to SRM Incorporation of third-party products into Astro-WISE (j. Lite, My. Proxy, SRM) Use of Astro-WISE data models and data processing on the Grid Multiple copies of a file (dataserver, EGI Grid) 20/03/2018 8

Astro-WISE – Grid integration: Features Port the system to the Grid, not an application Easy-to-use approach Distributed Processing Unit 20/03/2018 9

Target goals • • • Integrate 2 grid infrastructures: Astro-WISE Grid and external Grid (currently using g. Lite as middleware) Provide support in the use of Grid for existing and upcoming Target projects Use experience gained in the development of new information systems Hide complexity of Grid for users Collaboration with Big Grid important 20/03/2018 10

LOFAR: User of integration Ingest procedure Code base Files Station data Nearline storage & processing Blue Gene SAS Archive Grid - Jülich Germany Big Grid – SARA Amsterdam Target - CIT – Groningen Metadata Storage + compute Oracle DB Tier 0 Tier 1 Astro-WISE view Euro VO Lofar Communities (Grid interfaces) Tier 2 20/03/2018 11

Current status • KIDS/Viking pipelines running on Grid • LOFAR – Software ported to the Grid – Integration with Astro-WISE ongoing • Other applications being ported – Handwritten text recognition – Other astronomy projects in near future 20/03/2018 12

Future Goals Improve interfaces to Target storage Improving Astro-WISE DPU capabilities Less complexity in Authorization/authentication Better support for parallel jobs on Grid necessary • • Global solve for 240 subbands in LOFAR Multithreaded runs MPI runs of imager • • • Improve software deployment Data location awareness • • I/O latencies Target can provide a local file system to local resources Data access from/to Grid (including DBMSes) IVOA interfaces Cloud computing: feasibility study • • 20/03/2018 Virtualization 13

Proposal to community • Knowledge dissemination • Exchange with non-astronomical communities • Public access to data stored on Grid (registry, interfaces, accessibility) • Regular workshops/meetings for Astro community • Information about experience with middleware and third-party software 20/03/2018 14

Acknowledgements • Enabling Grids for E-scienc. E III – EU FP 7: INFSO-RI-222667 • Target: – – – Provinces of Groningen and of Drenthe European Fund for Regional Development Dutch Ministry of Economic Affairs, Pieken in de Delta. Samenwerkingsverband Noord Nederland. Target operates under the auspices of Sensor Universe • Big Grid – Netherlands Organisation for Scientific Research (NWO) 20/03/2018 15

Relevant publications • K. Begeman, A. N. Belikov, D. Boxhoorn, F. Dijkstra, E. A. Valentijn, W. J. Vriend, Z. Zhao, Merging Grid technologies, J. Grid computing, 8, 2, 199221 (2010) • K. G. Begeman, A. N. Belikov, D. Boxhoorn, F. Dijkstra, H. Holties, Z. Meyer. Zhao, G. A. Renting, E. A. Valentijn, W. J. Vriend, LOFAR Information System, 27, 3, 319 -328 (2011) • A. N. Belikov, F. Dijkstra, J. A. Gankema, J. B. A. N. van Hoof, R. Koopman, Information Systems Playground - The Target Infrastructure, Scaling Astro. WISE into the Petabyte range, Experimental Astronomy, accepted for publication (2011) 20/03/2018 16