EPICS Experience at Fermilab Geoff Savage August 2005

EPICS Experience at Fermilab Geoff Savage August 2005 Controls and Monitoring Group 31 Aug 2005 MICE DAQ and Controls Workshop 1

Outline Fermilab and DØ u What is EPICS? u EPICS at DØ - Run. IIa u EPICS at DØ – Run. IIb u EPICS and the CFT u EPICS at Fermilab u EPICS lessons u 31 Aug 2005 MICE DAQ and Controls Workshop 2

Fermilab Chicago p p 1. 96 Te. V Booster p CDF DØ Tevatron p source 31 Aug 2005 p Main Injector & Recycler MICE DAQ and Controls Workshop 3

DØ Detector Liquid Argon/Uranium Calorimeter Forward Muon Layer C Superconducting Solenoid Scintillating Fibers Central Muon Layer A Silicon Tracker Central Muon Layer B Central Muon Layer C Muon Toroid 31 Aug 2005 MICE DAQ and Controls Workshop 4

What is EPICS? u Experimental Physics and Industrial Control System • Collaboration • Control System Architecture • Software Toolkit Integrated set of software building blocks for implementing a distributed control system u www. aps. anl. gov/epics u 31 Aug 2005 MICE DAQ and Controls Workshop 5

Why EPICS? Availability of device interfaces that match or are similar to our hardware u Ease with which the system can be extended to include our experiment-specific devices u Existence of a large and enthusiastic user community that understands our problems and are willing to offer advice and guidance. u 31 Aug 2005 MICE DAQ and Controls Workshop 6

EPICS Architecture Operator Interface (OPI) OPI – a workstation running EPICS tools. Operating Systems: Linux and Windows XP. LAN – communication path for Channel Access (CA), the EPICS communication protocol. Local Area Network (LAN) IOC – platform supporting EPICS run-time database. Example: VME based PPC processor running vx. Works, Linux workstation. 31 Aug 2005 MICE DAQ and Controls Workshop Input Output Controller (IOC) 7

IOC Architecture 31 Aug 2005 MICE DAQ and Controls Workshop 8

IOC Database Collection of records u Each record represents a system parameter (process variable, PV) u • Unique name • Set of attributes • Attributes and value can be modified u Records must process to do something • An input record can read a value every 10 seconds • A CA write to an output record causes the record to • process Either input or output, not both 31 Aug 2005 MICE DAQ and Controls Workshop 9

EPICS Records u u Input • • • Analog In (AI) Binary In (BI) String In (SI) • • Calculation (CALC) Subroutine (SUB) • • Analog Out (AO) Binary Out (BO) Algorithm/control Output Custom – only needed when existing record types or a collection of existing record types are inadequate 31 Aug 2005 MICE DAQ and Controls Workshop 10

Channel Access EPICS Tool (CA Client) LAN IOC (CA Server) 31 Aug 2005 • One client can connect to many servers. • A channel connects to the value or an attribute of a PV. • Each channel has a unique name = record name + attribute. • CA Services: • Search – find a channel by name • Get – retrieve a channel value • Put – modify a channel value • Add monitor – notification of state change MICE DAQ and Controls Workshop 11

EPICS at DØ - Run. IIa u u u u Support for new devices and additional field busses Centralized database with supporting tools for defining the control system configuration Report EPICS alarms to our global event reporting system Interface to the Python programming language used to develop host tools for downloading and monitoring Detector configuration management - COMICS Archiving Gateways 31 Aug 2005 MICE DAQ and Controls Workshop 12

MIL-STD-1553 B Bus Restricted detector access while running u Provides a robust and highly reliable connection to electronics in the remote collision hall u Developed a queuing driver, device support, and a generic record u 12 IOCs with ~70 1553 busses from the counting house to the detector and ~10 busses within the counting house u 31 Aug 2005 MICE DAQ and Controls Workshop 13

DØ Specific Records SVX sequencer - configures and controls the ADCs in the tracking detectors u High voltage u • Implemented using a state machine model • Linear ramping with retries • Trip condition recovery • Limits control u Rack monitor - Generic I/O module u MIL-1553 – debugging/testing on a 1553 bus • 64 A/D inputs, 4 D/A output, 4 16 -bit I/O words 31 Aug 2005 MICE DAQ and Controls Workshop 14

DØ Specific Device Support Access hardware registers across the MIL-1553 bus and VME backplane using standard EPICS records – long input, long output, … u Support for DZERO specific records u • Bira 4877 power supplies • SVX sequencers • Rack monitor u Complex MIL-1553 bus transactions • Dual port memory on the AFE boards • Write the address to access followed by a read or write of the data 31 Aug 2005 MICE DAQ and Controls Workshop 15

Describe a Device with Records Multiple record definitions with substitution Template parameters This defines a device. File Instances of records read by IOC to create record database. ASCII file EPICS tool Database Load EPICS Database Generator Define instances of a template. Assign values File to the substitution parameters in the template file. 31 Aug 2005 MICE DAQ and Controls Workshop 16

Centralized Database System Template File Instance Creation DB Creation Python Scripts Oracle Hardware Database Record Extract EPICS Database 31 Aug 2005 Generator File EPICS Database Web Access Template Extract IOC Id Generator File MICE DAQ and Controls Workshop 17

PV Naming Convention u <det> [subdet]_ <devtype>_ <loc>/ <attr> [: io][. field] • • u <det> - Required detector system § CTL, SMT, CFT, CPS, CAL, FPS, ICD, MUO [subdet] - Optional subdetector § North, South, East, West, Forward, Test <devtype> - Required Device Type § ADC, BLS, HV, PLS, CETEC, . . . <loc> - Required Location indicator § For example a crate number: M 201, PC 03, F 313 <attr> - Required device attribute § PEDS, MODE, SFM, CTEC, HEAD, VER, . . . [: io] - Optional Input/Output designation Direction is from the viewpoint of the front end crate [. field] - Optional Epics field name, default is. VAL Example: CTL_LV_M 201/P 5 V 31 Aug 2005 MICE DAQ and Controls Workshop 18

Global Event Reporting u Collect and distribute significant events from detector and DAQ • Run control • Alarms - data corruption u Added a connection to EPICS u Significant Event System (SES) • Allows for notification of all EPICS alarms • Otherwise a user must register an event 31 Aug 2005 MICE DAQ and Controls Workshop 19

Front Ends SE Host Apps L 1 L 2 L 3 Collector /Router SE SE Data Logger SE COOR SE SE Log File Disk SE SE Logger SE SES Server FM Alarm Watcher RC FIL Message Key: SE = Significant Event FIL = Filter configuration RC = Run Control FM = Filtered SE Message = Filter 31 Aug 2005 FM Guidance Files Alarm Display MICE DAQ and Controls Workshop 20

Alarm Display 31 Aug 2005 MICE DAQ and Controls Workshop 21

Python Interface “Python is an interpreted, interactive, objectoriented programming language. ” u A Python interface to Channel Access (C library) has been developed and extended to be almost object oriented u Tkinter and Python Mega Widgets used to develop graphical user interfaces (GUI) u Developed a GUI framework to speed development time and provide a consistent look and feel u 31 Aug 2005 MICE DAQ and Controls Workshop 22

IOC Resource Monitor Display Left click each tab to move between displays. Each box is a PV Look for status information. Processor name. 31 Aug 2005 MICE DAQ and Controls Workshop 23

31 Aug 2005 MICE DAQ and Controls Workshop 24

Detector Configuration Management The COMICS server manages the configuration of the detector u Receives load requests from: u • Run control • Expert GUIs • Program APIs u Load map is a directed graph (tree) • Tree node (intermediate) § Establishes a layered hierarchy § Establishes an execution order • Action node (leaf) § Issues EPICS CA requests 31 Aug 2005 MICE DAQ and Controls Workshop 25

Configuration Management Tree Root Node DØ CAL SMT Intermediate Tree Node S 0 S 1 S 2 DEV 0 DEV 1 T 0 Action Node Uses EPICS Channel Access 31 Aug 2005 MUO Links determine the order of execution DEV 2 MICE DAQ and Controls Workshop CFT Configuration Data DEV 3 26

Comics Expert GUI 31 Aug 2005 MICE DAQ and Controls Workshop 27

Archiving Data u u u Maintain a history of PV values Channel Archiver – EPICS tool • • • Fast sampling rates Immediate analysis Infrequent access to old historical data Large files Stored to tape once a week • • • Long-term studies Slow sampling rates Easy access to data at any moment Minimal maintenance Data stored directly in an Oracle database EPICS/Oracle Archiver - DØ 31 Aug 2005 MICE DAQ and Controls Workshop 28

Gateways u Accelerator u Cryogenics and Gas systems • Gateway link to ACNET system • Bidirectional • Data access only (no control) • Gateway link to DMACS system • Portable Channel Access Server – EPICS tool • Read-only • Data access only (no control) 31 Aug 2005 MICE DAQ and Controls Workshop 29

ACNET Gateway CLIENTS xmlrpc Server xmlrpc ITC CA To. From. ITC To. EPICS RECORDS CA To. ACNET C A C H E From. ACNET xmlrpc From. EPICS D 0 31 Aug 2005 R E M O T E E X P O R T BD MICE DAQ and Controls Workshop 30

EPICS at DØ - Run. IIb u u u Can. Bus driver for EPICS • Weiner power supplies • • Gigabit Ethernet communication Raw Ethernet protocol • • Asynchronous Driver Support “asyn. Driver is a general purpose facility for interfacing device specific code to low level communication drivers. ” “A primary target for asyn. Driver is EPICS IOC device support but, other than using lib. Com, it is independent of EPICS. ” Ethernet driver for EPICS Both use support from asyn. Driver – EPICS tool • u Framework reduces development time. 31 Aug 2005 MICE DAQ and Controls Workshop 31

Support for CFT Device support for the AFE board using the MIL-1553 driver u AFE and AFE power supply templates u Record and device support for the SVX sequencers u Expert GUI for downloading u • Does not use COMICS u Calibration software • Runs on processor in VRB crate • Communicates with AFE 31 Aug 2005 MICE DAQ and Controls Workshop 32

Analog Front End Fiber MIL-1553 Communication Cryostat Wave guide Analog Front End VLPC Stereo Board Cassette 31 Aug 2005 MICE DAQ and Controls Workshop 33

SVX Sequencer S e q u e n c e r AFE MIL-1553 Communication P r o Calibration c e Software s s o r 31 Aug 2005 V R B C V R B VME S e q u e n c e r V B D MICE DAQ and Controls Workshop S e q u e n c e r Level 3 34

EPICS at Fermilab u SMTF u ILC • Superconducting module test facility • Cavity test in October • DOOCS for LLRF – speaks CA • EPICS for everything else • No ACNET – beams division controls system • Control system to be decided • Management structure now forming 31 Aug 2005 MICE DAQ and Controls Workshop 35

EPICS Lessons u Three Layers Applications • u Tools • u • Development § EPICS tools § OPI development - CA library § Build base § Build application - combine existing pieces § Develop device templates § Records, device, and driver support EPICS is not simple to use • • Need expertise in each layer Support is an email away IOC operating system selection 31 Aug 2005 MICE DAQ and Controls Workshop 36

The End 31 Aug 2005 MICE DAQ and Controls Workshop 37