Complexity Management Solutions for High Energy Physics Control

Complexity Management Solutions for High Energy Physics Control Systems: The CMS experiment Ildefons Magrans de Abril CMS Trigger Software Technical Coordinator, CERN, Geneva Zurich (IBM Research Laboratory) 23 th January 2008 Ildefons Magrans, CMS Trigger Software Technical Coordinator

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 2

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 3

CERN, European Organization for Nuclear Research CERN provides research facilities to particle physicists worldwide Large Hadron Collider Ildefons Magrans, CMS Trigger Software Technical Coordinator 4

Large Hadron Collider (LHC) Largest superconducting installation: • 27 Km ring • 3 billion euros CMS and ATLAS detect collision information (event): • 40 million events/second Ildefons Magrans, CMS Trigger Software Technical Coordinator 5

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 6

Compact Muon Solenoid (CMS) Numeric complexity: 21. 6 m long 15 m diameter 12500 tones 4 Tesla solenoid (100. 000 time earth mag. Field) Human complexity: 39 countries 182 Institutes (CERN is 1) 1200 m 3/hour of water for cooling (~gva jet d’eau 1800) 10 MWatts required for operation (~10. 000 houses) 3330 people ~800 students! Time complexity: Design stated 20 years ago! 7 -8 years for construction 15 years of expected operational life time Already developing upgrades Ildefons Magrans, CMS Trigger Software Technical Coordinator 7

The CMS “sensor” Electromagnetic Calorimeter: Particle physicist Measure energy of particles interacting electromagnetically Hadronic Calorimeter: Measure energy of particles interacting via the strong nuclear force (heavy neutral particles) ? Silicon Tracker: Find charged particle tracks and momentum Ildefons Magrans, CMS Trigger Software Technical Coordinator Muon detector: Find muon tracks 8

The CMS Trigger and Data Acquisition System Solution based on two filter levels: Level-1 Trigger (HW) High Level Trigger (SW) 40 million events/second ~55 million Channels ~1 Mbyte per event We can just store 100 events per second Ildefons Magrans, CMS Trigger Software Technical Coordinator Control system coordinates experiment operation 9

About this talk L 1 Decision Loop and detector front-ends. HARDWARE Ildefons Magrans, CMS Trigger Software Technical Coordinator CMS Control System. SOFTWARE 10

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 11

Context complexity Numeric dimension: Time dimension: Thousands of hardware modules and the same order of electronic links • L 1 Trigger development starts the year 2000 • L 1 Trigger design for the SLHC has already started! • CERN Linux platform upgrades every 2 years → Periodic Software & Hardware upgrades Human & political dimension: • Large number of independent research institutes with similar requirements using different technologies (e. g. FPGA vs ASIC, VME vs PCI vs tiny …) • Most people are particle physicist with few % of time dedicated to SW development. ~20% students Ildefons Magrans, CMS Trigger Software Technical Coordinator 12

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 13

XSEQ: A Software environment model XML Device Description Devices Control Sequence (XSEQ) Device Data Interpreter Processes platform independent control sequences 1. XML as uniform data representation format for both data and code • Long term technologic inversion (XML is here to stay) • Maximize usage of standard tools • Simplify software configuration management 2. Interpreted approach for the code • Execute code independently of the platform Ildefons Magrans, CMS Trigger Software Technical Coordinator 14

XSEQ example 1: Hello world XSEQ language (XML-based sequencer): ·Syntax specified in xsd documents + Extensions: file system, SOAP, DOM, HW access (PCI & VME). ·Exception handling with error recovery mechanism ·Other: object oriented and design by contract extensions. XSEQ syntax core definition Every tag is a function Exception handling Ildefons Magrans, CMS Trigger Software Technical Coordinator 15

XSEQ example 2: hardware access Decoupling syntax and semantic enhances sharing code between sub-systems with similar requirements Extends interpreter in order to execute a new syntactic ecd 6 Common tools for fd 05 processing code 0 and data. Simplifies core Scoped variable development Not accessible in upper hierarchical my_device levels CTRL my_data Ildefons Magrans, CMS Trigger Software Technical Coordinator Device specifications … 16

Online software integration Xseq program (URL) Interpreter plug-in Peer transport (SOAP) XDAQ application XDAQ executive (one per host computer) XDAQ framework: CMS in house developed C++ Middleware SOAP message specifies the URL of the XSEQ program or embeds a the program itself The running XSEQ program can access the original SOAP message and retrieve parameters Return message generated by the XSEQ program Ildefons Magrans, CMS Trigger Software Technical Coordinator 17

XSEQ example 3: distributed system Client: Remote server: CERN. Geneve HEPHY. Vienna. Global Trigger server SOAP pt SOAP Interpreter Xdaq plug-in executive Standalone Interpreter … … … … SOAP message sent by Client Bus protocol SOAP exension SOAP messae returned to the client … Ildefons Magrans, CMS Trigger Software Technical Coordinator msg … board fname chip … my_msg 18

XSEQ conclusions “Good”: • Suitable technologic investment (XML is here to stay) • Reduces in house development (Large asset of standard tools) • Enhances code sharing among sub-systems (extension mechanism) • Enhances platform evolution (interpreted approach) • Simplifies software configuration management (uniform usage of XML for code/data) “Bad”: • XML is verbose (programming with XSEQ is not fun), but: – An XML editor could help – XSEQ could serve as the underlying syntax to store virtual instrumentation developed with graphical tools like Labview • Just a prototype. It is not being used for production Ildefons Magrans, CMS Trigger Software Technical Coordinator 19

Outline 1 CERN and the LHC 2 The CMS experiment 3 Enhancing complexity management with web services 3. 1 Software environment model: XSEQ 3. 2 Concrete architecture: The CMS Trigger Supervisor Ildefons Magrans, CMS Trigger Software Technical Coordinator 20

CMS Trigger Supervisor Context CMS Control System. SOFTWARE L 1 Decision Loop. HARDWARE ~55 Million Channels, ~1 Mbyte per event Ildefons Magrans, CMS Trigger Software Technical Coordinator 21

HW context: L 1 -Trigger Decision Loop Configuration: • 64 crates • O(103) boards • Firmware ~ 15 MB/board • O(102) regs/board Testing: • O(103) links Integration coordination: L 1 decision loop operation ~ “business” • 27 research institutes Time: • Research: 1992 -2000 • Development: 2000 -present • Fully replaced by 2010 Ildefons Magrans, CMS Trigger Software Technical Coordinator 22

SW context: Experiment control system Run Control and Monitoring System (RCMS): • Overall experiment control and monitoring • RCMS framework implemented with java L 1 -Trigger Control and Hardware Monitoring System: Provides a machine and a human interfaces to operate, test and monitor the Level-1 decision loop hardware components. (8) Experiment control system ~ “business” IT infrastructure Detector Control System (DCS): • Detector safety, gas and fluid control, cooling system, rack and crate control, high and low voltage control, and detector calibration. • DCS is implemented with PVSSII Ildefons Magrans, CMS Trigger Software Technical Coordinator Cross-platform Data Ac. Quisition middleware (XDAQ): • C++ component based distributed programming framework • Used to implement the distributed event builder 23

Project phases and terminology New “business capabilities”: e. g. configuration Services Architecture Prove of concept System Framework Services and core developments Prototype Concept SW Context “Business” software infrastructure Ildefons Magrans, CMS Trigger Software Technical Coordinator • Business needs • Project team HW Context “Busines”: To filter the “best” events 24

Business needs and project team Trigger Supervisor GUI TS team (2 + 1 or 2 students) : • Services + core developments • Architecture • Business capabilities • Sub-system developers coordination & support • Communication pattern comp. Experiment control system 0. . n 1 1 0. . n 1 developer subsystem: • Uses services to develop the subsystem architecture • Customizes subsystem architecture as required by TS team Trigger Supervisor 1 Business need: coordinate operation of CMS subsystems (eg. Configuration and test) 1 1 DT TF 1 CSC TF G. Muon Trigger G. Cal. Trigger 1 1 1 CSC hits 1 DT hits Ildefons Magrans, CMS Trigger Software Technical Coordinator 1 RPC hits 1 GT/TCS 1 1 R. Cal. Trigger 1 HCAL energy ECAL energy HF energy 25

Baseline service infrastructure CMS official software frameworks to develop distributed systems: DCS, RCMS, XDAQ: Subsystems Online Soft. Ware Infrastructure needs to be integrated Subsystem OSWI integration effort (C++, Linux) DCS (PVSSII, Windows) RCMS (Java) XDAQ (C++, Linux) Infrastructure should be oriented to develop SCADA systems Supervisory and Control Infrastructure development effort ++ Ok + XDAQ-based baseline solution + additional development to reach SCADA framework Ildefons Magrans, CMS Trigger Software Technical Coordinator 26

Core development: The Cell Control panel plug-ins + HTTP/CGI: Automatically generated e. g. DTTF panel Cell plug-ins (FSM, commands, control FSM operation panels) e. g. Cell hide HW and SW platform evolution Other plug-ins: e. g. GT panel • Command: RPC method. SOAP API extensions Synchronous and Asynchronous SOAP API • Monitoring items FSM Plug-ins Ildefons Magrans, CMS Trigger Software Technical Coordinator Xhannel infrastructure: Designed to simplify access to web services (SOAP and HTTP/CGI) from operation transition methods • Tstore (DB) • Monitor collector • Cells 27

Service providers: building blocks Cell: Facilitates subsystem integration. DB interface. Exposes SOAP. Log Collector: and Tstore: Monitor sensor: Cell operation (additional development, next slide). • 1 per system. interface to poll monitoring • Collects log statements from cells and 1 per crate. information. forward them to consumers. Mon. Collector: Polls all cell sensors. 1 per cell. 1 per system. Mstore: interface M. on these components Architecture based uniquely collector with Tstore. 1 per system. XS: Reads logging data base. Job control: Remote startup of 1 per cell. XDAQ applications. 1 per host. RCMS components Ildefons Magrans, CMS Trigger Software Technical Coordinator 28

Architecture Building blocks Control system + • User’s guide • Workshops Monitoring system = • Support + Subsystem Usage model proposal Ildefons Magrans, CMS Trigger Software Technical Coordinator Logging system Start-up system 29

Control architecture Stable infrastructure in top of what new “business” capabilities can be defined Hierarchical control system 1 crate ~ 1 cell Centralized access to DBs Multicrate subsystems ~ 2 level of subsystem cells (1 subsystem central cell) Ildefons Magrans, CMS Trigger Software Technical Coordinator 30

Monitoring architecture Infrastructure that facilitates the hardware monitoring Centralized access to DBs Centralized system: 1 Collector, 1 Mstore 1 cell ~ 1 sensor Ildefons Magrans, CMS Trigger Software Technical Coordinator 31

Logging and start-up architecture Auxiliar infrastructure 1 cell ~ 1 XS Centralized system: 1 host ~ 1 JC 1 Collector Ildefons Magrans, CMS Trigger Software Technical Coordinator 32

New business capabilities: “How to” e 12() S 1 e 23() S 2 e 34() S 4 S 3 e () New “business” capabilities can be coordinated by particle physicist managers without SW expertise 43 Particle physicist manager Subsystem SW developer e 12() S 1 Entry cell Operation states e 23() S 2 Operation transitions Ildefons Magrans, CMS Trigger Software Technical Coordinator e 12() S 3 S 1 Operation transition methods e 23() S 2 S 3 Service test 33

Trigger Supervisor conclusions • Design: Services, architecture and “business” capabilities 1 Services: – Reduced number of building blocks already developed in-house (but the Cell) – Main building block: Cell • Isolates Hardware/Software evolution from architecture implementation • Adapts sub-system integration tasks to the human context academic background (Non SW experts) 2 Architecture: – Uniquely based on 7 building blocks • Simplifies sub-system integration coordination – Stable infrastructure • Isolates services evolution from the implementation of business capabilities 3 New “business” capabilities: – Coordination methodology associated with the architecture • Facilitates the implementation of new “business capabilities” taking into account the academic background of managers (Non SW experts) Ildefons Magrans, CMS Trigger Software Technical Coordinator 34

Summary Enhancing control systems design & development with web-services technologies: 1. XML-based programming language: • Maximizes usage of existing XML standards and tools, good tech. investment, max. code sharing and platf. evolution 2. Control system design example: • Services: Hides HW/SW evolution • Architecture: Hides Services evolution, stable infrastructure • Business capabilities: Developed in top of the architecture Ildefons Magrans, CMS Trigger Software Technical Coordinator 35

Thank you very much! … For more information: Ildefons. magrans@cern. ch http: //triggersupervisor. cern. ch Ildefons Magrans, CMS Trigger Software Technical Coordinator 36