e Xegrid Workflow Support and Resource Management

e. Xegrid – Workflow Support and Resource Management PGM-RG Workshop, GGF 8, Seattle, June 26 th 2003

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Profile of the Fraunhofer-Gesellschaft Founded in: 1949 Staff of 12 000 (2002) Research volume of 1 billion Euro (2002) 57 research establishments, operating as independent » Profit-Centers « JNF/841, 2002 Page 2

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Fraunhofer Gesellschaft 57 Institutes mainly engineering Production technology IT Microelectronics Material Sciences Energy and Building Life Sciences 12000 Scientists JNF/841, 2002 Page 4

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid International locations Fraunhofer USA Boston (Massachusetts), Pittsburgh (Pennsylvania), Plymouth (Michigan), Providence (Rhode Island), Newark (Delaware), College Park (Maryland), Peoria (Illinois) Fraunhofer Asia Beijing (China), Singapore, Jakarta (Indonesia), Tokyo (Japan) JNF/841, 2002 Page 5

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid e. Xegrid An open source approach for the industrial use of grids JNF/841, 2002 Page 12

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Main assumptions Ø Grid Job can be a very complex workflow (there is a difference between grid jobs and common jobs) Ø No user interaction for job scheduling and data transfer Ø Mapping between software and hardware must be automated Ø Need for a Grid Application Description. Language GADL Ø Job submission as easy as possible JNF/841, 2002 Page 13

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid e. Xegrid in brief Web Portal Problem Build control and Jobs Resource Information Authentication/ Security Selected Resources Resource and Accounting Information Job Builder Grid Job Workflow Send and control Jobs Job Handler Scheduler JNF/841, 2002 Load information Page 14

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Overview Grid Access + current network JNF/841, 2002 Page 15

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Application Description Language (GADL) GADL Set of XML-based description languages needed to define and to execute grid applications The GADL consists of: GResource. DL Description of resources GJob. DL Description of grid jobs Set of resources + workflow GInterface. DL Interface definition of software components GData. DL Description of data JNF/841, 2002 Page 16

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid GADL: Grid Resource Description Language (GResource. DL) GResource. DL Description language for categorization and description of resources Used to select suitable resources to solve a given problem (task mapping) Definition of dependencies between resources Extension of resource descriptions with inheritance allows formulation of recursive descriptions Everything is a resource! Software components Hardware resources Measuring devices JNF/841, 2002 Data Page 17

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid GResource. DL example that depends on other resources depends, conflicts, . . . provides, suggests . . . JNF/841, 2002 Page 18

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid GResource. DL example that provides other resources JNF/841, 2002 . . . Page 19

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid The Web Portal – on the outside Security Proxy User Interne t – Fh. RG certificate – input data – requests – ID check – mapping to Globus account – access rights on web portal Web Portal – HTML info – servlets for dynamic requests, e. g. task mapping – web-/grid-services – grid tools – resource repository – user data repository – Globus client node JNF/841, 2002 Page 20

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Building and executing a Grid Job Builder – grid service request for resource information (GRDL) – build grid applications – start Job Handler with job parameters (GJDL) Job Handler JNF/841, 2002 – map software to resources – generate atomic jobs – ask broker for compute resources – start and control jobs via Globus Page 21

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets Places Files, buffers, control places (active, done, . . . ) Transitions Software components, control transitions Arcs from places to transitions (Place is input place of transition) Arcs from transitions to places (Place is output place of transition) Tokens Data Rules A transition is activated if all input places are filled with tokens and all output places have not reached their maximum capacity of tokens Refinement A transition can be replaced by a Petri Net JNF/841, 2002 Page 22

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Control and Data Flow as a Petri Net Control place Control transition Data place (linked to a file) start exit status stdin Software transition (linked to a software component) stdout stderr input 1 input 2 output 1 JNF/841, 2002 Page 23

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid GADL: Grid Job Definition Language (GJob. DL) Problem Description of complex workflows of grid jobs DAG Directed Acyclic Graph (see e. g. Condor) no bidirectional coupling (interaction) no loops A B C D Petri Nets PARENT A CHILD B C PARENT B C CHILD D Graphical flow control of discrete systems A B JNF/841, 2002 Page 24

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Description of Petri Nets in XML PNML Petri Net Markup Language (modified from Jüngel, Kindler, Weber; HU- Berlin) JNF/841, 2002 true start program output Page 25

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid GJob. DL A JNF/841, 2002 Page 26

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid MAGMAsoft Simulation Environment for foundries a commercial application A simple example of Fh. RG usage JNF/841, 2002 Page 27

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid MAGMAsoft : Use your laptop and some remote site to solve the simulation prolem magma start input magma. Visu exit status stdout stderr output log start end exit status stdout stderr magma. Data magma. Log JNF/841, 2002 Page 28

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid. FTP with Petri Nets using refinement local file remote file program local file remote file Grid. FTP local file program JNF/841, 2002 Page 29

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid MAGMAsoft with refinement ( file transfers are inserted automatically) transfer. Executable(magma. Visu) magma. Visu. sh. tar. gz magma. sh. tar. gz unpack() magma. sh magma. Visu. sh transfer. Data(magma) transfer. Data(magma. Visu) magma. Visu magma start input exit status stdout stderr output log transfer. Data(magma. Visu) start end exit status stdout stderr magma. Data magma. Log JNF/841, 2002 Page 30

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Tools that help Visualisation of the Grid Job Description Jobhandler: Runs the complete Job sequence JNF/841, 2002 Page 31

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Simple Petri Nets B begin A end D C failed B begin end A done C JNF/841, 2002 Page 32

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri nets: while( condition = true ) do condition = true A end condition = false JNF/841, 2002 Page 33

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Fh. RG Loop begin end count 7 count < 7 transfer. Executable(add. Hostname) add. Hostname unpack() Host. List start old. Host. List exit status stdin stdout stderr input transfer. Data(add. Hostname) new. Host. List transfer. Data(add. Hostname) JNF/841, 2002 Page 34

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Workflow and Dataflow management with Petri Nets failed begin cat. sh done start d 25. dat stdin failed exit status stdout stderr end cat. sh input 1 input 2 d 26. dat output 1 done start exit status stdin d 25 -26. dat stdout stderr input 1 d 27. dat input 2 d 25 -27. dat output 1 JNF/841, 2002 Page 35

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Eramas – System Architecture User with information request Web Interface ERAMAS Server X 1 FIRST IBB DGC Repository Main Server Input / Output Meta database Server X 2 other potential service systems … Server Xn Internet Server A 1, A 2. . . An Atmospheric transport models (incl. local databases) Server I 1, I 2. . . In Information systems (databases, online measurements etc. ) Fraunhofer FIRST Server AZ 1, AZ 2. . . AZn Models for the aeration region (incl. local databases) DGC Server G 1, G 2. . . Gn Groundwater models (incl. local databases) Server E 1, E 2. . . En Exposition models (incl. local databases) Server B 1, B 2. . . Bn Assessment models (incl. local databases) IBB JNF/841, 2002 Page 36

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Pollutant Transport in the Atmosphere begin diapar. in dwminput User interface dwm. out Diagnostic wind model dwm. out lacon. dat oro. in latim. dat lapos. dat momat. dat mit 95. out momet. dat Preprocessing morel. dat movar. dat Lagrangian model Monte-Carlo, evaporation conv. out var. out JNF/841, 2002 Page 37

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Parts of our e. Xecution grid Ø Grid Application Description Language incl. GJob. DL based on Petri nets and able to handle very complex workflows Ø Jobbuilder : user interface for creating grid job workflows Ø Jobhandler : managing the execution Ø resource broker Ø web portal for secure access, task mapping , . . . Ø grid repository : Resource descriptions, User rights, . . Ø based on Globus but independent of Globus Ø easily portable to OGSI Ø role based security JNF/841, 2002 Page 38

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Thank you for your attention! Questions? JNF/841, 2002 Page 39

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Contact Information http: //www. fhrg. fhg. de Juergen Falkner Dr. Franz-Josef Pfreundt (Fh. RG Project Manager) Fraunhofer IAO Fraunhofer ITWM Nobelstr. 12 Europaallee 10 D-70569 Stuttgart, Germany D-67657 Kaiserslautern, Germany Tel : +49 -(0)711 -970 -2414 Tel : +49 -(0)631 -303 -1800 Fax: +49 -(0)711 -970 -2401 Fax: +49 -(0)631 -303 -1811 E-Mail: Juergen. Falkner@iao. fraunhofer. de E-Mail: Pfreundt@itwm. fraunhofer. de JNF/841, 2002 Page 40

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets (from van der Aalst und Kumar, 2000) Task Sequence Choice JNF/841, 2002 Page 41

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets Parallel execution without synchronization JNF/841, 2002 Page 42

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets Parallel execution with synchronization JNF/841, 2002 Page 43

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets Wait all with time out JNF/841, 2002 Page 44

Fraunhofer-Institute for Industrial Engineering / Fraunhofer Resource Grid Petri Nets Wait any with time out JNF/841, 2002 Page 45