EU FP 6 Project Instruments and Sensors on

EU FP 6 Project Instruments and Sensors on the Grid The Grid. CC Project Gaetano Maron gaetano. maron@lnl. infn. it Istituto Nazionale di Fisica Nucleare – Laboratori di Legnaro (PD), Italy

Outline n n n GRIDCC project: goals & objectives Architecture overview Strategic targets Pilot applications Focus on the new provided capabilities ¨ Instrument Element ¨ RMM-JMS as fast publishing system ¨ Virtual Control Room Conclusions www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 2

GRIDCC project Grid Enabled Remote Instrumentation with Distributed Control and Computation n It is a 3 -years EU FP 6 project started in September 2004 n Web site: www. gridcc. org Participant name Country Istituto Nazionale di Fisica Nucleare Italy Institute Of Accelerating Systems and Applications Greece Brunel University UK Consorzio Interuniversitario per Telecomunicazioni Italy Sincrotrone Trieste S. C. P. A Italy IBM (Haifa Research Lab) Israel Imperial College of Science, Technology & Medicine UK Istituto di Metodologie per l’Analisi ambientale – Consiglio Nazionale delle Ricerche Italy Universita degli Studi di Udine Italy Greek Research and Technology Network S. A. Greece www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 3

GRIDCC: Goals & Objectives 1. Remote control and monitoring of complex and distributed instrumentation 2. Tight integration between instrument grid and classical computational grid 3. Human interaction with Grids via Virtual Control Room (collaborative environment) 4. Enactment of complex workflows www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 4

GRIDCC: Architecture (I) Instruments Grid IE IE DATA Instrument Element Instrument Manager Ser vice Information & Monitoring System Ori Problem Solver VCR ent e d. A VCR rch i VCR Collaborative Environment www. gridcc. org (EU FP 6 Contract 511382) tec Workflow VCR The GRIDCC services are Computational. Web Services compliant, Grid according to the WS-I convention. n Any type of client WS-I CE compliant, can access the GRIDCC services without any specific software library. n X 509 based security is used CE n When performances are required , a kerberos based mechanism is used to access SE IEs. n ture ES –S OA Execution Services G. Maron, KVI Seminar, Groningen, March 2007 5

GRIDCC Architecture Security Services Virtual Control Room Information and Monitoring Services (IMS) Global Problem Solver Collaborative Services (CS) Instrument elements Storage Instrument Element (SE) (IE) Element Storage Element (IE) (SE) Storage Element (SE) Compute element Computing element (CE) Element (CE) Web Service Interface WMS Work Management System Wf. MS Work Flow Mng System Execution Wf. MS Service WMS AS Broker Information System (BDII) AS Agreement Service www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 6

Grid. CC Permanent. Test Bed VCR, IE Wf. MS VOMS IE, CE/CREAM, SE, LFC AS, Krb. Server WMProxy www. gridcc. org (EU FP 6 Contract 511382) VCR, SE, IE, BDII G. Maron, KVI Seminar, Groningen, March 2007 7

Strategic plan: 3 main target areas Remote process control Accelerator control (Tele-) Biomedicine Robotics Automotive Electronic microscopes (Large-scale) scientific experiments High energy particle physics (Radio-) Telescopes GRIDCC Middleware Widely Sparse Instrumentation Power Grids Monitoring of the territory Monitoring of the sea Geo-hazard prediction Distributed laboratories Transportation monitoring Sensor network www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 8

GRIDCC pilot applications Power Grid Particle Accelerator High Energy Physics CMS Meteorology Device Farm Geohazard Monitoring www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 9

Pilot applications: CMS - I n n The Grid. CC middleware has been deployed to control the run of the CMS (Compact Muon Solenoid), one of the four high energy experiments in LHC (Large Hadron Collider) at CERN laboratory. CMS Magnet Test and Cosmic Challenge (MTCC), a milestone in the CMS construction, positively carried out. CMS Detector CMS Control User Top IE Interface Structure CSC IE Tracker HCAL IE IE DAQ IE RPC IE DAQ IMS FB RB Trigger IE DT IE ECAL IE DAQ RS FF xdaq www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 10

Pilot applications: Power grid GRIDCC deployed to monitor: • A 50 k. W generator • A 1 kw Photo-Voltaic array Virtual Control Room Instrument Manager Power Grid V. O. Power Grid. GRIDCC deployed to monitor a 50 k. W generator and a 1 Solar Panel k. W Photo-Voltaic array Gas Instrument Element . . . www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 11

Pilot applications: Remote Operation of an Accelerator Elettra Synchrotron www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 12

GRIDCC: other applications n Meteorology (Ensemble Limited Area Forecasting) ¨ n Device Farm for the Support of Cooperative Distributed Measurements in Telecommunications and Networking Laboratories ¨ n Weather forecasting system to detect hazardous weather The Device Farm consists of a pool of Measurement Instruments for Telecommunication Experiments Geo-hazards: Remote Operation of Geophysical Monitoring Network The monitoring net will be characterized by different levels of activity: stand-by, pre-alert, plus a control modality ¨ An event worth to be monitored is for example the evolution of a “landslip” ¨ www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 13

Focus on the new capabilities provided by GRIDCC n Instrument Element ¨ Complete integration of Instruments into SOA. Instruments become “services” accessible via standard Web Service ¨ Complete integration into the computational and storage GRID with the embedded Sorage Element (SE) ¨ High virtualization and abstraction of diverse physical devices ¨ A very scalable and platform-independent framework (Java based) n Fast Data Publishing System ¨ High-performance reliable multicast via RMM-JMS n Virtual Control Room ¨ Allows effective human interactions with the grid ¨ Provides a homogeneous collaborative environment www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 14

Instrument Element: Versatile I/O multichannel The IE permits ¨ SOA accessible operations to control and monitor the instruments (via VIGS), such as: • • ¨ execute a command get / set parameters different data outputs: • • • data mover to/from a grid Storage Element (via SRM) high bandwidth channel for data publishing (via IMS). low bandwidth channel for logs, states etc. (via IMS). Data rs Status Parameters Grid Interaction VIGS Commands VIGS IMS SRM cribe IMS VCR ES Subs L St ogs ate , E s, rro Mo rs, nit ors IE SRM n Instruments Storage Element (SE) Virtual Instrument Grid Service Information and Monitor Service Storage Resource Management www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 15

Instrument Element: Instruments as Services n n The Instrument Element (IE) is one of the novel elements of the GRIDCC architecture. It offers a standard web service interface to integrate scientific and general purpose instruments and sensors within the grid. An IE can control a set of instruments with the possibility to decide on which subset to operate. The IE is the key decoupling element between real instruments and the grid Service Oriented Access Web Services Instrument Element Any Protocol or physical connection Sensor Network www. gridcc. org (EU FP 6 Contract 511382) Instrumentation Instrument G. Maron, KVI Seminar, Groningen, March 2007 16

Instrument Element: Instrument Virtualization Instrument Parameters Attributes Control Mode XML Based Language IMS IE SE The IE grid instruments representation is basically based on: ¨ providing the XML description of the instruments managed by the IE ¨ providing the Finite State Machine that defines the dynamic behaviour of the control performed on the set of instruments managed by the IE ¨ simple grid accessible operations to control and monitor the instruments like execute a command, get or set a parameter or a set of parameters of the instrument (VIGS). ¨ providing different data acquisition output paths ranging from the accessing of a Grid Storage Element (SRM/SE) to a very efficient communication channel when an high bandwidth output stream is required (via IMS). VIGS n Instruments www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 17

Instrument Element Architecture Problem Solver Instrument Element SRM/SE Grid. FTP SE IMS Resource Service The term Instrument Element describes a set of services that provide the needed interface and implementation that enables the remote control and monitoring of physical instruments. Data Flow Instrument Manager Data Publisher IMS Access Control Manager VIGS IMS State Flow Error Flow Monitor Flow Control Flow Data Collector IMS Proxy Control Manager Instrument Protocols Control Manager Event Processor FSM Engine Input Manager Resource Proxy Real Instruments www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 18

Instrument Element Framework Characteristics n n n Customizable plug-in modules are used to interface the framework with the communication protocol (both hardware and software) used by the instruments. the dynamic behaviour of the control that has to be performed is programmable via the built in Finite State Machine (FSM). relatively fast control feedback loops can be provided using the customizable built in Event Processor (EP). Error repairing actions due to malfunctioning of the instruments can even be handled by EP or by the Local Problem Solver (LPS). www. gridcc. org (EU FP 6 Contract 511382) Access Control Manager n IE The IE framework is modular, fully customizable and can be adapted to different environments and different type of instruments to be controlled. VIGS n Resource Service IMS Data Mover Local Problem Solver Data Collector Instrument Instrument Manager IMS Proxy Control Manager Event Processor FSM Engine Input Manager Resource Proxy G. Maron, KVI Seminar, Groningen, March 2007 19

Instrument Manager Problem Solver IMS Proxy SRM/SE Grid. FTP Data Publisher Instrument Element Instrument Manager Data Collector www. gridcc. org (EU FP 6 Contract 511382) IMS SE Resource Service IMS Access Control Manager VIGS IMS Control Manager G. Maron, KVI Seminar, Groningen, March 2007 20

21 Instrument Manager Customizable Control Manager Event Processor FSM Engine Input Manager Resource Proxy IMS Proxy Plug-in modules to interface to the instruments Data Collector Data Flow State Flow Instruments Monitor Flow Control Flow Error Flow IM is composed by 3 main components: - Control Manager: - Input Manager. It handles all the input events of the IM. These includes commands from GUIs or other IMs, errors/state/log/monitor messages. - Event Processor. It handles all the incoming message and decide where to send them. It has processing capability - FSM. A finite state machine is implemented - Resource Proxy. It handles all the outgoing connections with the resources. - Data Collector. It get data from the controlled instruments and make them available to the data mover. A local storage of the data is even foreseen. - IMS Proxy. It receives error/state/log/monitor information from the controlled resources and forward them to IMS www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 21

DB persistency S -JM MM Access Control Manager enable/disable data stream VIGS Data Collector Resource Service Local Problem Solver IMS Instrument Element Status • data collection • data publishing on RMM-JMS • data on file in progress • data on DB to be started • sync commands (enable/disable done trough the “set param” of the IE façade (VIGS) Data Mover IE File System Instrument Manager Data Collector www. gridcc. org (EU FP 6 Contract 511382) R IMS Proxy , or nit rs be. ) ri sc etc b Su ysis, ta l Da ana ta da o (M Control Manager Event Processor FSM Engine data stream Instrument Input Resource Instrument Manager Proxy G. Maron, KVI Seminar, Groningen, March 2007 22 Instrument

Resource Service Instrument Manager Data Collector www. gridcc. org (EU FP 6 Contract 511382) Data Publisher Instrument Element IMS Proxy SRM/SE Grid. FTP SE Problem Solver IMS Access Control Manager VIGS IMS Control Manager G. Maron, KVI Seminar, Groningen, March 2007 23

Partition/Configuration retrieve methods Partition and Lock setting methods Discovery Manager Subscribe Manager Configuration setting methods Discovery methods n n n Partition&Lock Configuration Manager Available Resources Partition Definitions Manager Configuration Definitions RS Data Bases Resource Service Architecture The Resource Service (RS) handles all the resources of an IE and manages their partition (if any). A resource can be any hardware or software component involved in the IE (instruments, Instrument Managers, IMS components) RS stores the configuration data of the resources and download them to resource target when necessary Resources can be discovered, allocated and queried. It is the responsibility of the RS to check resource availability and contention with other active partitions when a resource is allocated for use. A periodic scan of the registered resources keeps the configuration database up to date. www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 24

Resources Discovery n Technology used http: //www. jxta. org/ www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 25

Information and Monitor Service Data Publisher Instrument Element Instrument Manager IMS Proxy SRM/SE Grid. FTP SE Problem Solver Data Collector www. gridcc. org (EU FP 6 Contract 511382) IMS Resource Service IMS Access Control Manager VIGS IMS Control Manager G. Maron, KVI Seminar, Groningen, March 2007 26

27 n n n IMS Proxy SUBSCRIBERS IMS Proxy Publish / Subscrib PERSISTENCY SUBSCRIBERS Information and Monitor System (IMS) IMS Proxy The Information and Monitor Service (IMS) collects messages and monitor data coming from the IMS Proxy of the IMs. The messages are catalogued according to their type, severity level and timestamp. Data can be provided in numeric formats, histograms, tables and other forms. The IMS collects and organizes the incoming information either in a database or a pub/sub system or both. It publishes the data to subscribers. These subscribers can register for specific messages categorized by a number of selection criteria, such as timestamp, information source and severity level. www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 27

IE Data Subscribers IE Data Producer IMS Fast Data Publishing: RMM-JMS www. gridcc. org (EU FP 6 Contract 511382) Same data are sent to several subscribers. Multicast protocols can have a benefic impact on performances • JMS provides a standard set of APIs for the communication • Many commercial and academic JMS implementations both in C/C++ and Java (Narada. Brokering, Sun, IBM) • GRIDCC (IBM Haifa lab) has implemented a Reliable Multicast protocol (RMM) JMS compliant • RMM-JMS works within a LAN but an efficient bridge technology has been developed to allow inter-LAN multicast communication G. Maron, KVI Seminar, Groningen, March 2007 28

Problem Solver IMS Proxy SE Instrument Manager SRM/SE Grid. FTP Data Publisher Instrument Problem Element Solver Data Collector www. gridcc. org (EU FP 6 Contract 511382) IMS Resource Service IMS Access Control Manager VIGS IMS Control Manager G. Maron, KVI Seminar, Groningen, March 2007 29

Problem Solver Step 3 On-line information can be analyzed in order to detect possible malfunctions On Line Analisys Data Mining Tools Step 1 The control manager can perform an autonomous recovery action where the cost for the determination it is not so heavy. Pub/Sub Instrument Manager IMS Proxy DB Algorithms evaluations : Rule Induction, Tree, Functions, Lazy, Clusters and Associative Control Manager State Flow Error Flow Monitor Flow Instrument Manager IMS Proxy Control Manager Step 2 Persistent information can be analyzed in order to extract knowledge www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 30

Resource Service Local Problem Solver IMS Instrument Element Instrument Manager SE Backend SRM Access Control Manager VIGS Embedded SRM compliant Grid Storage Element (SE) GRID accessibile data IE File System Status: installed and running • SRM 2. 2 • STORM implemetation (INFN) • Bestman implementation (Berkeley)) • XFS IE file system www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 31

Instrument Element: Scalable on embedded systems Custom Board IE Xilink Virtex IV PPC 405 Standalone Axis FPGA Linux Custom Logic Java. VM Grid. CC IM Web Service Instrument Manager Custom Electronics Instrument Manager on a chip … … towards a grid on a chip 1 Gbps Ethernet Montavista www. montavista. com USER INTERFACE Jam. VM http: //jamvm. sourceforge. net www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 32

IE technologies n Web Service compliant (WS-I) n Tomcat + Axis (and Java) and Axis standalone are the main technologies of the IE n All the services are deployed on a single or multiple instances of Tomcat, according to the needs of the application n Message oriented middleware (Pub/Sub) is based on the Java Messaging System (JMS). The following implementations are used in the project Sun ¨ Narada Brokering ¨ RMM - JMS (Grid. CC IBM) ¨ n My. SQL and Oracle are used as Data Base for the RS n SRM interface version 2. 2 used www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 33

Virtual Control Room n n The VCR is a collaborative web portal Enable the user to run, aggregate and display plug-ins that act as “mini applications” www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 34

Conclusion - I n The Grid. CC project is integrating instruments into the “classic” computational/storage Grids. n Novel concepts introduced by Grid. CC are: ¨ The Instrument Element, allowing a virtualisation of the instruments to be controlled and their insertion in a Grid ¨ The Virtual Control Room, providing an highly interactive environment with IEs/CEs/SEs. VCR even provides cooperative tools to allow the cooperation (logbook, chat, videoconf. , etc. ) between remote users ¨ hard (reservation of IE) and soft (statistical prediction) guarantees of the IE’s methods execution times ¨ Fast Data Publication via a Message Oriented middleware (RMM-JMS) to distribute data and information from an IE to the world wide Grid. n The IE is highly customizable and can be adapted in different environments. www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 35

Conclusion - II n The IE can be shrunk down into a chip allowing grid enabled embedded control of the instrumentation n Several heterogeneous pilot applications are deploying and running the IE middleware n We support and encourage the adoption of our middleware in other projects/experiments. www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 36

Thank you for your attention n Any Questions? www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 37

Spare www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 38

Tecnologie usate http: //www. jxta. org/ http: //gladgw: 2002/Instrument. Element. Keeper/ da dove vedere la lista degli strumenti http: //gladgw. lnl. infn. it: 2002/Seeds. Finder/ da dove scaricare l’IE da web-start

Grid. CC Scalability Demo Imperial Brunel VCR Monitor GUI Elettra INFN IASA IBM GRNet CNIT Network Core Machine Instrument Element Uni. UD IMAA www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 40 Network Information Provider

www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 41

Pilot applications: CMS - II 2 107 electronics channels 40 MHz O(104 ) distributed Objects to be ¨ controlled CMS Detector ¨ configured ¨ monitored n n n www. gridcc. org (EU FP 6 Contract 511382) Highly interactive system (human reaction time - fractions of second) n 100 Hz On-line diagnostics and problem solving capability World Wide distributed monitor and control G. Maron, KVI Seminar, Groningen, March 2007 42

Instrument Element – Interconnections II Fast Data Publishing Fast Data Display Existing Grid Elements Work. Flow Mng. System IE Control and Status VCR Control Panel Data Consumer IE Computing Element (CE) Storage Element (SE) IE Log Display www. gridcc. org (EU FP 6 Contract 511382) Information and Monitor System G. Maron, KVI Seminar, Groningen, March 2007 Log Persistency 43

References n n n n - WS-I, http: //www. ws-i. org - Java Message Service (JMS), http: //java. sun. com/products/jms - RMM-JSM, http: //www. haifa. ibm. com/projects/software/rmsdk/gridcc. html - g. Lite, http: //glite. web. cern. ch/glite/ - SRM, http: //sdm. lbl. gov/srm-wg - Sto. RM, http: //grid-it. cnaf. infn. it/storm - Montavista, http: //www. montavista. com - Jam. VM, http: //jamvm. sourceforge. net www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 44

Acknowledgments The INFN team working in Legnaro, in collaboration with CERN at Geneva: n n n E. Frizziero M. Gulmini F. Lelli G. Maron A. Petrucci S. Traldi www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 45

Instrument Element: Versatile I/O multichannel Data Virtual Control Room IE 2) 3) 4) IE Grid Interaction IMS WMS AS 1) s SE VIGS Status Wf. MS cribe r DP Commands Execution Service Subs A Virtual Instrument Grid Service (VIGS) interface has been defined. The VIGS provides a Web Service acccess to the instrumentation The Data Publishing (DP) channel disseminates the acquired information to the data subscribers A SRM/SE interface is provided to allow file movement with other Grid Element, like SE, CE and other IEs. Information messages (logs, errors, etc. ) are published from the IE (via IMS) www. gridcc. org (EU FP 6 Contract 511382) , rors rs Er gs, Monito Lo s, te Sta Storage Element (SE) Compute element Computing element (CE) Element (CE) VIGS Virtual Instrument Grid Service IMS Information and Monitor Service SE Storage Element DP Data Publishing G. Maron, KVI Seminar, Groningen, March 2007 46

Fast Data Publishing: Some results Message Rate: Case many-to-one • 32 Dual Xeon 2. 4 GHz 1. 5 GB RAM machines, 1 GB Ethernet switch • At most 1 publisher, subscriber, or broker(Sun MQ 3. 6) per machine • No message lost • RMM throughput: 75 -90 Mbytes/sec. (for 5 and more publishers) www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 47

Architecture: New “instruments grid” Component Name Description Instrument Element (IE) This is a unique concept to GRIDCC. It consists of a coherent collection of services which provide all the functionalities to configure, partition and control the physical instruments Information & Monitor Service (IMS) It gathers from GRIDCC resources information and monitor data to be disseminated through a publish / subscribe systems or to be stored in persistent repositories. Problem Solver (PS) It offers automated problem solving in a Grid environment at two levels. A local PS, within a given Instrument Element, allows to solve local problems related to functionalities of a given instrument. A global PS, allows to solve system-wide problems. Virtual Control Room (VCR) It provides a common set of collaboration tools and allows users to build complex workflows, which are then submitted to the Execution Services, and to directly monitor and control remote instruments in real-time. Execution Services (ES) They control the execution of the workflows defined by the user in the VCR, maintaining the status of the tasks that make up the workflow. They also support the advance reservation of resources. Security Services (SS) GRIDCC uses a split security system. When interacting with components of other Grids the GSI security will be used and the users identified by their X. 509 proxy certificate. When interacting with the IE the user will be identified by a Kerberos ticket. www. gridcc. org (EU FP 6 Contract 511382) G. Maron, KVI Seminar, Groningen, March 2007 48