FAIR Controls H G Essel J Adamczewski B

FAIR Controls H. G. Essel, J. Adamczewski, B. Kolb, M. Stockmeier GSI, Oct 2005 Hans G. Essel DAQ Control 1

CMS XDAQ • • • XDAQ is a framework targeted at data processing clusters – Can be used for general purpose applications – Has its origins in the I 2 O (Intelligent IO) specification The programming environment is designed as an executive – A program that runs on every host – User applications are C++ programmed plug-ins – Plug-ins are dynamically downloaded into the executives – The executive provides functionality for • Memory management • Systems programming queues, tasks, semaphores, timers • Communication Asynchronous peer-to-peer communication model Incoming events (data, signals, …) are demultiplexed to callback functions of application components Services for configuration, control and monitoring Direct hardware access and manipulation services Persistency services GSI, Oct 2005 Hans G. Essel DAQ Control 5

XDAQ event driven communication • • • Dynamically loaded application modules (from URL, from file) Inbound/Outbound queue (pass frame pointers, zero-copy) Homogeneous frame format Readout component Generates a DMA completion event Executive framework Demultiplexes incoming events to listener application component Computer foo( ) Application component Implements callback function Peer transport Receives messages from network GSI, Oct 2005 Hans G. Essel DAQ Control 8

XDAQ: I 2 O peer operation for clusters • • • Application component Processing node Controller node device IOP host • • Messaging Layer Homogeneous communication – frame. Send for local, remote, host – single addressing scheme (Tid) Application framework Messaging Layer ‚ ‡ Peer Transport Agent ƒ † I 2 O Message Frames GSI, Oct 2005 Peer Transport Agent Executive Application ˆ „ Peer Transport … Hans G. Essel DAQ Control ‰ Application 9

XDAQWin client Configuration tree XML based configuration of a XDAQ cluster GSI, Oct 2005 Daqlet window Daqlets are Java applets that can be used to customize the configuration, control and monitoring of all components in the configuration tree Hans G. Essel DAQ Control 11

BTe. V: RTES deliverables A hierarchical fault management system and toolkit: – Model Integrated Computing • GME (Generic Modeling Environment) system modeling tools – – ARMORs (Adaptive, Reconfigurable, and Mobile Objects for Reliability) • – and application specific “graphic languages” for modeling system configuration, messaging, fault behaviors, user interface, etc. Robust framework for detection and reaction to faults in processes VLAs (Very Lightweight Agents for limited resource environments) • GSI, Oct 2005 To monitor/mitigate at every level – DSP, Supervisory nodes, Linux farm, etc. Hans G. Essel DAQ Control 18

Modeling Synthesis RTES structure Global Fault Manager Soft GSI, Oct 2005 Region Operations Mgr Analysis Performance Diagnosability Reliability Region Fault Mgr Logical Data Net Global Operations Manager Logical Control Network Experiment Control Interface Fault Algorithms Behavior Synthesis Logical Data Net Design and Analysi s Runtime Feedback Resource Reconfigure L 2, 3/CISC/RISC Real Time Hans G. Essel DAQ Control L 1/DSP Hard 20

GME: data type modeling • Modeling of Data Types and Structures • Configure marshallingdemarshalling interfaces for communication GSI, Oct 2005 Hans G. Essel DAQ Control 23

RTES: GME fault mitigation modeling language (2) FMML Model – Behavior Aspect Translator ARMOR Microkernel Switch(cur_state) case NOMINAL: I f (time<100) { next_state = FAULT; } Break; case FAULT if () { next_state = NOMINAL; } break; class armorcallback 0: public Callback { public: ack 0(Controls. Cection *cc, void *p) : Callback. Fault. Inject. Tererbose>(cc, p) { } void invoke(Fault. Injecerbose* msg) { printf("Callback. Recievede dtml_rcver_Local. Armor_ct *Lo; mc_message_ct *pmc = new m_ct; mc_bundle_ct *bundlepmc->ple(); pmc->assign_name(); bundle=pmc->push_bundle(); mc); } }; • • • Fault Tolerant Custom Element Communication Custom Element Model translator generates fault-tolerant strategies and communication flow strategy from FMML models Strategies are plugged into ARMOR infrastructure as ARMOR elements ARMOR infrastructure uses these custom elements to provide customized fault-tolerant protection to the application GSI, Oct 2005 Hans G. Essel DAQ Control 27

ARMOR • Adaptive Reconfigurable Mobile Objects of Reliability: – – • Completely transparent and external support Enhancement of standard libraries Instrumentation with ARMOR API Internal architecture structured around event-driven modules called elements. Elements provide functionality of the runtime environment, error-detection capabilities, and recovery policies. Deployed ARMOR processes contain only elements necessary for required error detection and recovery services. ARMOR processes resilient to errors by leveraging multiple detection and recovery mechanisms: – – – • ARMOR Runtime environment is itself checking. ARMOR processes designed to be reconfigurable: – – – • System management, error detection, and error recovery services distributed across ARMOR processes. 3 -tiered ARMOR support of user application – – – • Provide error detection and recovery services to user applications Hierarchy of ARMOR processes form runtime environment: – – • Multithreaded processes composed of replaceable building blocks Internal self-checking mechanisms to prevent failures from occurring and to limit error propagation. State protected through checkpointing. Detection and recovery of errors. ARMOR runtime environment fault-tolerant and scalable: – 1 -node, 2 -node, and N-node configurations. GSI, Oct 2005 Hans G. Essel DAQ Control 30

ARMOR system: basic configuration Execution ARMOR Oversees application process (e. g. the various Trigger Supervisor/Monitors) Exec ARMOR App Process Daemons Detect ARMOR crash and hang failures network Daemon Heartbeat ARMOR Detects and recovers FTM failures Fault Tolerant Manager (FTM) Fault Tolerant Manager Highest ranking manager in the system ARMOR processes Provide a hierarchy of error detection and recovery. ARMORS are protected through checkpointing and internal self-checking. GSI, Oct 2005 Hans G. Essel DAQ Control 31

EPICS overview EPICS is a set of software components and tools to develop control systems. The basic components are: OPI (clients) – Operator Interface. This is a UNIX or Windows based workstation which can run various EPICS tools (MEDM, ALH, Oracle. Archiver). IOC (server) – Input Output Controller. This can be VME/VXI based chassis containing a Motorola 68 xxx processor, various I/O modules, and VME modules that provide access to other I/O buses such as GPIB, CANbus. PV (Process variables) – Named objects located in data base records (data and functions) of IOCs LAN (communication) – Local area network. This is the communication network which allows the IOCs and OPIs to communicate. EPICS provides a software component, Channel Access, which provides network transparent communication between a Channel Access client and an arbitrary number of Channel Access servers. GSI, Oct 2005 Hans G. Essel DAQ Control 32

Hierarchy in a flat system IOC • • • IOCs – One IOC per standard CPU (Linux, Lynx, Vx. Works) clients – on Linux, (Windows) Agents – Segment IOCs beeing also clients tasks IOC Client GSI, Oct 2005 tasks IOC Name space architecture! IOC tasks IOC Hans G. Essel DAQ Control 35

Local communication (node) Node commands Task intertask Task IOC command thread working thread message thread status segment memory Task • • • messages GSI, Oct 2005 Hans G. Essel DAQ Control Commands handled by threads Execution maybe in working thread Message thread maybe not needed 36

Screen shot FOPI GSI, Oct 2005 Hans G. Essel DAQ Control 38

SMI++ and State Management Language SML – Classes and Objects • Allow the decomposition of a complex system into smaller manageable entities – Finite State Machines • Allow the modeling of the behavior of each entity and of the interaction between entities in terms of STATES and ACTIONS – Rule-based reasoning • Allow Automation and Error Recovery – SMI++ Objects can be: • Abstract (e. g. a Run or the DCS) • Concrete (e. g. a power supply or a temp. sensor) – Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts) – Logically related objects can be grouped inside "SMI domains" representing a given sub-system – Finite State Logic • Objects are described as FSMs their main attribute is a STATE – Parallelism • Actions can be sent in parallel to several objects. Tests on the state of objects can block if the objects are still “transiting” – Asynchronous Rules • Actions can be triggered by logical conditions on the state of other objects GSI, Oct 2005 Hans G. Essel DAQ Control 40

SMI++ Run-time Environment Obj SMI Domain Obj – Device Level: Proxies • C, C++, PVSS ctrl scripts • drive the hardware: – deduce. State – handle. Commands – Abstract Levels: Domains • Internal objects • Implement the logical model • Dedicated language – User Interfaces • For User Interaction – Communication DIM Proxy Hardware Devices GSI, Oct 2005 Hans G. Essel DAQ Control 41

Kind of conclusion • • RTES: Very big and powerful. Not simply available! – Big collaboration – Fully modelled and simulated using GME – ARMORs for maximum fault tolerance and control XDAQ: Much smaller. Installed at GSI. – Dynamic configurations (XML) – Fault tolerance? EPICS: From accelerator controls community. Installed at GSI – Maybe best known – No fault tolerance – Not very dynamic SMI++ DIM: State machines and light weight communication – Used in JCOB (CERN) GSI, Oct 2005 Hans G. Essel DAQ Control 47