Development of GRID environment for interactive applications Jesús

Development of GRID environment for interactive applications Jesús Marco de Lucas (marco@ifca. unican. es) Instituto de Física de Cantabria, IFCA Consejo Superior de Investigaciones Científicas, CSIC, Santander, SPAIN DATAGRID DISSEMINATION DAY 14 -V-2003 BARCELONA 1

The EU Cross. Grid Project European Project ( ~5 M€, 3 year project started March 2002 ) proposed to CPA 9, 6 th IST call, V FP Polish (Cracow & Poznan) / Spanish (CSIC & CESGA) / German (FZK) initiative with the support of CERN (thanks to Fab!) CYFRONET (Cracow) is the coordinator of the project (Michal Turala, project leader) Objectives: Extension of GRID in Europe, assuring interoperability with Data. Grid Interactive Applications (“human in the loop”): • Environmental fields (meteorology/air pollution, flooding crisis management) • High Energy Physics (interactive analysis over distributed datasets) • Medicine (vascular surgery preparation) Need: • Develop corresponding middleware and tools • Deploy on a pan-european testbed Partners: Poland (CYFRONET, PSNC, ICM, INP, INS), Spain (CSIC: IFCA, IFIC, Red. IRIS, UAB, USC), Germany (FZK, USTUTT, TUM), Slovakia (II SAS), Ireland (TCD), Portugal (LIP), Austria (U. Linz), The Nederlands(Uv. A), Greece (DEMO, Au. TH), Cyprus (UCY) Industry: Datamat (I), Algosystems (Gr) Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 2

Surgical Planning Problem: vascular diseases Solution: placement of a bypass by a surgeon Planning for intervention is based on 3 D images obtained from MRI or CT scans. The attainable improvement in blood flow should determine which possibility is the best for a particular patient. A 3 D arterial model is built on the basis of the images, and presented to the surgeon in an inmersive intuitive environment Jesús Marco de Lucas A CT scanner Observation Stenosis (narrowing of an artery) Viewing the arterial structure in an immersive 3 D environment Data. Grid Dissemination Day (Barcelona 14 -V-2003) 3

Surgical Planning Goal: Simulate vascular reconstruction Method: Interactive Virtual Reality Environment to Arterial structures from scans with proposed bypasses • View scanned data • Define proposed interventions • View simulation results Advanced fluid code to simulate flows Simulated flows Need Grid in interactive mode (the surgeon should not wait long…) Access distributed computational resources for flow simulation and visualization, so get a high performance environment at low cost • Distribute simulations for different bypass configurations Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 4

Flood management Problem: flooding crisis in Slovakia Solution: monitoring, forecasting, simulation, real-time actions Precipitation forecasts based on meteorological simulations of different resolution from the meso-scale to the stormscale. For flash floods, high-resolution (1 km) regional atmospheric models have to be used ng ori along with remote sensing data (satellite, it on radar) m From the quantitative precipitation forecast, hydrological models are used to determine the discharge from the affected area. Then hydraulic models simulate water flow ing through various river structures to predict the ast ec impact of the flood for Crisis management teams should consult various experts, before making any decisions. The experts should be able to run simulations with different parameters and analyze the impact (“what-if” analysis). ion lat u sim Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 5

Flood management Goal: Flooding risk prediction Method: Cascade of simulations • Meteorological • Hydraulic Virtual Organization Need Grid in interactive mode (simulation results for “what-if” ) seamlessly connect together experts, data and computing resources needed for quick decisions highly automated early warning system, based on hydro-meteorological (snowmelt) rainfall-runoff simulations Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 6

Flood management Web portal for access Job submission Visualization See DEMO outside Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 7

HEP interactive analysis The next Large Hadron Collider (LHC) at CERN, will accelerate protons to an energy enough to produce a particle hundreds times heavier: the Higgs Boson, the last piece in the Standard Model, key for understanding the origin of the mass. Problem: All collisions will be recorded by sophisticated detectors, and the information stored in distributed databases with a volume of millions of gigabytes. But only few of those complex collisions will produce a Higgs Boson… Solution: On-line filtering techniques + sophisticated mathematical algorithms for physics analysis, like neural networks leve l 1 - 40 M s ec Hz l 2 - 75 p. KHial h (40 TB/sec ) z (75 ardw emb leve l 3 5 KHezdded pr GB/secare ) ( leve 5 G oce - 1 0 s data (10 PC 0 Hz B/sec) ssors 0 offl recor. MB/sec) ine ding ana & lysis Physicists across the world are collaborating in this search… Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 8

HEP interactive analysis Goal: Physics analysis on large distributed databases Method: Distributed computing for • Access to databases • Complex algorithms, like Neural Networks Use Web Portal as GUI Need Grid in interactive mode (physicists try different hypos) Reduce the waiting time to test a new algorithm or a new hypothesis from hours down to minutes by processing in distributed mode (DEMO TODAY) Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 9

Meteo and Air Pollution Problem: Improve local predictions and refine air-pollution modeling close to a thermical power plant. Solution: data-mining on databases of outputs from atmospheric circulation models, to improve downscaling Typical database (ERA-15, ECMWF) Daily forecasts on a reticule covering the globe from 1979 -1993 Atmospheric circulation pattern: v=(T(1 ooomb), T(850 mb), . . . , Z, H. . . ) The dimension can reach 104 Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 10

Meteo and Air Pollution SIMILAR PATTERNS close in the grid and in the CPs space!! Goal: Data-mining on databases and improvement on airpollution prediction Method: Distributed computing for Need Grid in interactive mode (so the • Data-Mining algorithm SOM power plant reacts on time) • Air-Pollution STEM II Try different air-pollution estimations according to meteo predictions 2/1/1979 Atmospheric circulation pattern: 1/1/1979 v=(T(1 ooomb), T(850 mb), . . . , Z, H. . . ) The dimension can reach 104 Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 11

Application development Good interaction with final user community (clear use cases) Vascular Surgery: Leiden Hospital Flooding crisis management: authorities in Slovakia HEP interactive physics analysis: LHC physicists Meteo and Air Pollution: power plant managers Middleware and Tools (significative effort): Basic middleware: Globus 2 + Data. Grid Distributed computing using MPI: MPICH-G 2 • Support for correct use of MPI: profiling interface (MARMOT) • Benchmarking on a grid context and performance prediction Optimization of data access Monitoring: • the application itself, the network use, and the hardware Scheduling: • Support for allocation with priority of resources needed for MPI Portals and Roaming Access • Web Portal + VNC (Migrating Desktop) Testbed: Support development, test and deployment of applications, tools, and middleware Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 12

Architecture Migrating Desktop Application Tool Benchmark OCM-G Grid Visualization Kernel Portal and Roaming Access Simulation Output (Parallel) Application Running User Interaction Services Infrastructure Monitoring Scheduling Agent Data Access Data. Grid Data Management Data. Grid Job Management Globus Toolkit Multiple Sites Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 13

The Cross. Grid Testbed 16 sites (small & large) in 9 countries, connected through Géant + NRe. Ns + Grid Services: EDG middleware (based on Globus) RB, VO, RC… Géant TCD Dublin Uv. A Amsterdam PSNC Poznan ICM & IPJ Warsaw FZK Karlsruhe USC Santiago CYFRONET Cracow CSIC-UC IFCA Santander LIP Lisbon CSIC Red. Iris Madrid II SAS Bratislava Auth Thessaloniki UAB Barcelona CSIC IFIC Valencia DEMO Athens Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) UCY Nikosia 14

Using the Testbed Parallel Jobs (HEP Prototype using MPICH-G 2) Running Across Sites Globus II JSS LB Globus Site 1 Globus ork tw ne … Site i Globus Jesús Marco de Lucas Grid Services (LIP) Globus Data. Grid Dissemination Day (Barcelona 14 -V-2003) 15

Testbed Status http: //mapcenter. lip. pt Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 16

User Support Software repository http: //gridportal. fzk. de Customized GNU Savannah (based on Source. Forge ) CVS browsable repository Main current usage: • ca. 1000 web-hits per day 7000 files, 356 MB, 850. 000 code-lines, 15. 000 doclines + 174 doc/pdf-files Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 17

Integration work… Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 18

IST Demonstration Cross. Grid has participated in the World Grid demonstration involving European and US sites from Cross. Grid, Data. Grid, Gri. Phy. N and PPDG, that took place in November 2002. It was the largest grid testbed in the world. Applications from the CERN/LHC experiments CMS and Atlas Cross. Grid participated with 3 sites: LIP - Lisbon FZK - Karlsruhe IFIC - Valencia Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 19

Extending the GRID in Europe Close collaboration and complementarity with Data. Grid Interactive and parallel applications Extending the GRID into new countries and communities Keeping interoperability, in particular for the testbed Outreach and dissemination (visit our booth outside !!!): High impact at the national research level: • See Poland, Germany, Spain, Greece examples ACROSSGRID conference in Santiago de Compostela, great success! Dissemination effort to new communities (i. e. South. East Europe, Latin America) New application areas start to be interested Reforcing effort via Grid. Start (concertation meeting in June, 18 -19) Starting to establish company and final user contacts: • Companies interested in middleware and tools • Institutions and companies interested as final users Involved in proposals for new 6 th FP: Health. Grid Flood. Grid RT Grids… Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 20

Extending the GRID in Europe …and pushing for a common grid infrastructure for e-Science in Europe: EGEE Keep in contact with us: http: //www. eu-crossgrid. org Thanks in advance for your interest! Jesús Marco de Lucas Data. Grid Dissemination Day (Barcelona 14 -V-2003) 21