The European X-Ray Laser Project Summary of XFEL

The European X-Ray Laser Project Summary of XFEL DAQ and Control for photon beam systems workshop 10 -11 th March 2008 Agenda and presentations are available on: https: //indico. desy. de/conference. Display. py? conf. Id=762 The workshop was a Pre-XFEL project partially funded by the European Commission under the 7 th Framework programme. Christopher Youngman, DESY Photon Beamline Systems meeting - March 18, 2008 XFEL X-Ray Free-Electron Laser

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 47 registered participants ALESSANDRO, Polini alessandro. polini@bo. infn. it ANGELSEN, Christian. C. Angelsen@rl. ac. uk BILLICH, Heiner. Billich@psi. ch BOUROV, Sergueiserguei. bourov@desy. de CHELKOV (JINR), Georgychelkov@nu. jinr. ru CLAUSEN, Matthias. Clausen@desy. de CLAUSTRE, Laurentclaustre@esrf. fr COPPOLA, nicola, coppola@mail. desy. de COUGHLAN, John, J. COUGHLAN@RL. AC. UK DECKING, Winfried, winfried. decking@desy. de DIMPER, Rudolf, rudolf. dimper@esrf. fr DUVAL, Philip. Duval@desy. de ESENOV, Sergey, sergey. esenov@desy. de FURUKAWA, Yukito, furukawa@spring 8. or. jp GRAAFSMA, Heinz, heinz. graafsma@desy. de GUELZOW, Volker, volker. guelzow@desy. de GöTTLICHER, Peter, peter. goettlicher@desy. de HALSALL, Rob, R. Halsall@rl. ac. uk HATSUI, Takaki, hatsui@spring 8. or. jp HENRICH, Beat, beat. henrich@psi. ch HOMS PURON, Alejandro, homs@esrf. fr HOTT, Thomas. Hott@desy. de JALOCHA, Pawel. Jalocha@psi. ch KLEESE VAN DAM, Kerstink. kleese@dl. ac. uk KLORA, Jörg klora@cells. es KRACHT, Thorsten thorsten. kracht@desy. de KUGEL, Andreas, andreas. kugel@ziti. uni-heidelberg. de MANT, Geoffrey, g. r. mant@dl. ac. uk MURIN, Pavel, murin@vk. upjs. sk NICHOLLS, Tim, t. c. nicholls@stfc. ac. uk PERAZZO, Amedeo, perazzo@slac. stanford. edu PLECH, Anton, anton. plech@uni-konstanz. de POTDEVIN, Guillaume, guillaume. potdevin@desy. de REHLICH, Kay, kay. rehlich@desy. de RYBNIKOV, Vladimir. Rybnikov@desy. de SCHWARZ, Andreas, andreas. schwarz@desy. de SCHöPS, Andreas, andreas. schoeps@desy. de STEFFEN, Lothar, lothar. steffen@desy. de STEPHENSON, Paul, p. c. stephenson@dl. ac. uk TIEDTKE, Kai, kai. tiedtke@desy. de TRUNK, Ulrich, ulrich. trunk@desy. de VAN BAKEL, Niels, nielsvb@slac. stanford. edu VAN BEUZEKOM, Martin, martinb@nikhef. nl VISSCHERS, Janjanv@nikhef. nl WINTER, Graeme, g. winter@dl. ac. uk YOUNGMAN, Christopher. Youngman@desy. de ZIMMER, Manfred, manfred. zimmer@desy. de ALBA, DESY, Daresbury, ESRF, ITEP, JINR, Nikhef, PSI, RAL, SLAC, Spring 8, Bologna, Heidelberg, Konstanz Christopher Youngman, DESY January 29, 2008 2

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims Review ALL areas of WP 76 = 8 sections & 20 talks Machine parameters and timing Photon beam line instruments and detectors Control systems Archiving and data processing DAQ and control at other Labs Infrastructure requirements Perspectives for data rejection and size reductions 2 D pixel detectors Aims meet other groups, exchange ideas, etc. produce a list of work, milestones required = any fires clarify, if possible, work with other WPs identify regions of in-kind contribution is sufficient manpower and other resources available? … Christopher Youngman, DESY January 29, 2008 3

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Nomenclature used here: pulse = packet of photons or electrons sometimes called a bunch (e-beam context) train = consecutive group of pulses or bunches sometimes called a bunch train (e-beam) sometimes called a pulse train (γ-beam) sometimes called a macro-bunch (? ) train number unique incremented number for each train generated by XFEL ~1010 after 20 years of 30 Hz operation sometimes called the event number Christopher Youngman, DESY January 29, 2008 4

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser How to summarize 20 talks or 529 slides ? Talks reviewed: sometimes performed individually (Decking, Rehlich, …) sometimes as part of a related group (2 D-pixel detectors, …) comments/highlights added to slides are in green Talks not reviewed – due to lack of time and their offline nature: Large scale data retrieval and processing Analysis tools Talks not reviewed – due to lack of time Fast 2 D data acquisition at ESRF beam lines Other simplifications: My talks not reviewed – instead I have built them into the slides Christopher Youngman, DESY January 29, 2008 5

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL machine parameters relevant to beam lines – W. Decking talk walked through the electron and photon machine elements e beam dumps = limits pulses per beam line to 1500 (new) described the time structure 10 Hz standard, but up to 30 Hz with reduced energy pulse to pulse time = 200 ns to few micro-secs train to train variation of pulses delivered possible described photon beam distribution options fast and slow kicker operation slow kicker distributes between beam lines fast kicker selectively removes bunches energy variation pulse to pulse and train to train chirps (continuous and stepped) and sweeps time schedule Christopher Youngman, DESY January 29, 2008 6

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Electron Beam Dumps – W. Decking bunch compressor diagnostic dumps 0. 5 and 2. 5 Ge. V small fraction of max. beam power injector dumps 130 Me. V Pave= 12 k. W max. beam power main beam dumps up to 25 Ge. V Pave= 300 k. W 1/2 max beam power limit beam magnified slow sweep to distribute heat Christopher Youngman, DESY January 29, 2008 7

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Fast and Slow Kicker Distribution – W. Decking 200 ns fast slow • low accuracy (>1 %) • 5 MHz burst operation slow switch 300 ms • high accuracy (< 0. 01 %) • 10 Hz operation Christopher Youngman, DESY January 29, 2008 8

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Beam Distribution Options – W. Decking DC Magnet One beamline only Safe, Easy Commissioning option Christopher Youngman, DESY January 29, 2008 Slow switch pulse to High Q Resonator pulse Fixed bunch Duty cycle pattern, full duty reduced by # of cycle beamlines Safe, Easy Fast kicker for individual bunches Baseline Extended baseline Flexible Accuracy 9

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Energy Variation – Within Bunch Train – W. Decking E 0+1. 5% DE/Emax = 3 % E 0 -1. 5% 280 ms 1500 bunches SASE 1, SASE 3 280 ms 1500 bunches SASE 2, UN 1, UN 2 E 0+1. 5% 20 ms DE/E = + 10 -4 / ms - 10 -3 / ms E 0 20 ms E 0 -1. 5% 300 ms 1500 bunches SASE 1, SASE 3 Christopher Youngman, DESY January 29, 2008 300 ms 1500 bunches SASE 2, UN 1, UN 2 10

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Schedule cont. – W. Decking Gmbh = start delayed 1 year w. r. t. TDR – possible to recover ½ year if work faster Christopher Youngman, DESY January 29, 2008 11

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Summary – W. Decking • XFEL can provide flexible bunch and energy patterns for its various experiments • This requires the operation of LLRF, accurate fast and slow kickers, energy and position feedbacks with unprecedented performance parameters • The full flexibility will only be reached through an extensive research and commissioning program, where parts can only be done on the real machine • A definitive time schedule for the availability of these options is thus impossible to give Comments and wishes from users are very welcome to tailor the possible options to their needs Experiments would prefer the machine stating what is maximally possible, as these parameters determine pipeline lengths, variable clocks, etc. Christopher Youngman, DESY January 29, 2008 12

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Timing & data synchronisation for machine and experiments – K. Rehlich talk vision femto-sec to pico-sec stable clocks, event triggers, XFEL wide train# and pulse# unique tag of expt. and machine data XFEL timing = development of FLASH timing experience few pico-sec jitter for experiment clocks and events (≤ ½ ns required) ~10 femto-sec jitter for more demanding users (RF feedback, Laser) Experiment interfaced to system via TCA board development of Tönsch IP module at FLASH distributes received timing events (e. g. train start) and datagram Ø delayed by programmable value Ø datagram = train number, bunch pattern, energies input from users required Ø datagram definitions (defn. of data required, LAN requirement? ) prototype expected end 2008 Christopher Youngman, DESY January 29, 2008 13

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser K. Rehlich DAQ Christopher Youngman, DESY January 29, 2008 14

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser K. Rehlich Christopher Youngman, DESY January 29, 2008 15

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser K. Rehlich Christopher Youngman, DESY January 29, 2008 16

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser K. Rehlich Christopher Youngman, DESY January 29, 2008 17

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Photon beam line and diagnostic systems – K. Tiedtke talk Beam line systems: design and prototyping for XFEL ongoing Systems: shutters, mirrors, … Ø Use beam line systems at FLASH as baseline Ø Control system = HASYLAB (DORIS and PETRA) Ø Train sensitive Diagnostic systems: design and prototyping for XFEL ongoing Systems: pulse intensity, wave front, … Ø Must redesign detectors and DAQ for higher rates, lower cross-sections, etc. Ø Control system = ? Ø Pulse sensitive Still missing basic LISTS of what & where, responsibilities for readout, requirements on other groups !! Not easy, but must be done. Christopher Youngman, DESY January 29, 2008 18

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Photon beam diagnostics (online) User require pulse-resolved and non-destructive photon diagnostics to monitor important beam parameter like: • intensity (XGMD, bolometer, use fluorescence, etc. ) • beam position • lateral and temporal radiation pulse profile (e. g. interferometer or cross correlator) • spectral distribution (photoelectron spectrometer, etc. ) • Wave front (wave front sensor, interferometer) List items ! • coherence So far most of these XFEL detectors do not exist! We only have design concepts or prototypes, in particular for the intensity and beam position monitoring, but their DAQ and control system requirements are similar to XGMD… Christopher Youngman, DESY January 29, 2008 19

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser To determine the pulse energy on a shot-to-shot basis we need… Pressure and temperature => gas density (RS 232) Kind of gas and wavelength => cross section (SPS & DOOCS energy server) Calibrated ion current (electrometer) (GPIB) Integrated electron peaks (fast integrator -> fast ADC) (VME crates of the control system group) Bunch pattern (rep. rate, number of bunches in the train) (DOOCS server) Information about bunch loses (e. g. LOLA) (DOOCS server) Evaluated data must be correlated with “machine time stamp” and saved to the DAQ system In addition, one need status information about gas attenuator, apertures, and beam shutter (SPS, encoder values, DOOCS server) List items ! Christopher Youngman, DESY January 29, 2008 20

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Motion Control (speed 10 Hz or slower) SASE 1 beamline, TDR 2006: (other beamlines similar) total: up to 61 stepper motors, 16 channels piezo control (RS 232): • 6 mirrors with each 5 stepper motor axes+ 2 piezos = 30 stepper motors + 12 channels piezo control • Diagnostics mono+ Diamond Mono: each 8 stepper motor + 2 piezos = 16 step motors + 4 piezos • BM absorber (1 stepper motor), High power slits (6), 1: 1 focusing (2), 6 screens (6) = 15 stepper motors List items ! further: about 20 channels for temperature readout, 30 control units (RS 232) for ion pumps and vacuum gauges, about 4× 1 D-detectors readouts (PIN diodes, Si-diodes) for flux calibration 6 survey cameras (look at fluorescent screens) Christopher Youngman, DESY January 29, 2008 21

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Motion control software Existing solutions from 3 rd generation synchrotrons are sufficient, e. g. SPEC, Spectra, medm-EPICS, Tango, Tine, …. Should be compatible with (more demanding) slow control solutions for experiments. Basic requirements for beamline control software: • should talk to standard (e. g. OMS) control boards • easy configuration of motor control parameters (speed, limits, acceleration, dial position, user position, …) • easy configuration of ‘scan’ parameters: which motor? which counter? how long? delay between steps? configure data format with additional channels (e. g. temperatures). • visualization of scan, go to maximum etc. • reliable (no reset button in tunnel needs to be pushed) Assume that we use the HASYLAB control system under DOOCS for beamline. Christopher Youngman, DESY January 29, 2008 22

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Experiment systems 0 D, 1 D and 2 D detectors – H. Graafsma talk 3 beam line XFEL startup scenario = need experiments, … detectors in various stages of prototyping/design 0 D little work done 1 D initial work started Ø collaboration with FLA started and possibly NIKHEF Ø DAQ and control look like a slice of 2 D Ø 1 D development to be headed Ramos (quick start is important) 2 D proposals in, DAQ and control relatively advanced Ø see 2 D-pixel detector later talks 1 D and 2 D detectors should share solutions generic backend DAQ and control generic clock&control hardware generic (slow) control systems Requirements list for operating modes given. Christopher Youngman, DESY January 29, 2008 23

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Undulator and Beamline Layout – H. Graafsma XFEL startup scenario Distribution of the 5 (3) beamlines, with 2 experimental stations each Christopher Youngman, DESY January 29, 2008 24

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser What do we know? – H. Graafsma Scattering experiments: Beamline Elastic Scattering 1000 x 1000 = 1 Mpixels (day 1) 4, 16, 25 Mpixels Christopher Youngman, DESY January 29, 2008 25

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser This means: - H. Graafsma • 2 bytes/pixel; 1 Mpixels/image; 500 images/train; 10 times per second 10 Gbytes/sec sustained. (More numbers in Detector specific talks). Principle: store images during BT, and readout between BT • But for different experiments we need different detectors (low energies, high energies, small pixels large pixels, …) Different frontend but generic backend. – 3 developments for large 2 D detectors: • HPAD (see Goettlicher): coherent imaging and XPCS hard X-ray • LPD (see Coughlan): Femto-second non-crystalline diffraction • DEPFET (see Kugel): coherent imaging and XPCS soft X-ray – We will also use 0 D and 1 D detectors (diagnostics and exp. ) Christopher Youngman, DESY January 29, 2008 26

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Other boundary conditions: • We need to modify the experiment: add, remove (auxiliary) detectors within a day flexibility • We want to be able to use different (new) detectors flexibility and standardization. • We want to be able to use larger detectors in the future (4, 9, 25, … Mpixels) Modular approach • We need to CONTROL the experiment (see SR-talks). Means “move and count” (part of) the data needs to be visible “immediately” • We need to store other data (machine and experiment) with the images • We want to store only “useful” images (fast veto) • We will have single module prototypes by 2010 (LCLS; Petra, …) Christopher Youngman, DESY January 29, 2008 27

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Layout of 1 D detector array (1 D-DA) H. Schlarb PD array ADC N x 10 bit DATA processing ASIC control Controls/ Arch. Crate u. TCA FPGA … N channels 128, … 1048? Gain & Shaping Rocket IO 5 MHz Two readout streams: 1. Fast feedback lvds 2. IP readout to storage needed Christopher Youngman, DESY January 29, 2008 >50 MHz Machine Clock & Trigger Feed back 28

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Conclusion: • Many parameters and milestones are yet undefined. • They will only get defined once the station scientists are in place (this year? ) • The large 2 D-detectores are the most demanding, and they are getting well defined • 1 D detectors will (hopefully) look like a slice through 2 D. • We are just starting…. . Christopher Youngman, DESY January 29, 2008 29

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Control systems at DESY– P. Duval talk introduction: what control systems have to deal with non legacy control systems used at DESY: in house: DOOCS, TINE, SPECTRA (hasylab) external: EPICS (cryogenics+utilities), TANGO (hasylab) commercial: PVSS (H 1), D 3 (cryogenics) Grand unification of control systems effort underway TINE-DOOCS, EPICS-DOOCS, etc. but further diversification (TANGO) = more interfaces. no agreement likely for using one Reviewed DAQ archiving and viewing requirements data storage (DAQ requirements) database (thin and thick) user requirements (offline and online data access) data browsing tools For HEP person worth reading to understand light source control software Christopher Youngman, DESY January 29, 2008 30

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Control Systems (one way or another) have to deal with … - P. Duval Distributed end points and processes Data Acquisition (front end hardware) Real-time needs (where necessary) Process control (automation, feedback) Central Services (Archive, Alarm, Name Resolution, …) Security (who’s allowed to do what from where? ) States (Finite State Machines, sequencing, automation…) Time synchronization (time stamps, cycle ids, etc. ) Databases (configuration, machine data, post-mortem data, …) Statistics (control system itself, operation, …) Logging (central, local, application, …) Data transport (data flow, control system protocol, scalability) DAQ : focus on Data Acquisition and Central Services ! Agrees with our requirements if experiment algorithm software is added Christopher Youngman, DESY January 29, 2008 31

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ Users Want … P. Duval Control guys want to debug/improve operations – Machine parameters Experimental analysis guys want to examine the stored data and look for things (off-line? ) – Experimental data (and machine parameters) Easy to add things to the system – Plug-and-play Need to ‘browse’ the DAQ in quasi-realtime – You may have terabytes of data somewhere, but if it takes tera-seconds to retrieve it, you’ll go crazy. – Browsing the DAQ = Browsing the Archive System Christopher Youngman, DESY January 29, 2008 32

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Browsing the DAQ (DOOCS) – P. Duval DOOCS calls to the “LONGARCHIVER” Christopher Youngman, DESY January 29, 2008 33

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Browsing the DAQ (CSS) – P. Duval “DAL/DAA” calls to the underlying control system archiver. Christopher Youngman, DESY January 29, 2008 34

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Browsing the DAQ (TINE) – P. Duval 3 Hera Lumi Runs shortly before the End of Operations Protons : LUMI : ? ? Pick and Choose other Zeus Parameters : Christopher Youngman, DESY January 29, 2008 35

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser TINE Video (Multicasting + Scheduling) – P. Duval . 5 Mbyte Video Frames @ 10 Hz multicast (100 Mb ethernet). (also runs fine @ 20 Hz) Uses the CM_NETWORK switch. Server calls the Scheduler when a new frame is grabbed => as real-time as it gets !!! Christopher Youngman, DESY January 29, 2008 36

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Conclusions – P. Duval We are able to mix control system protocols fairly easily these days – Use the best tool for the job – But … • you might be introducing headaches for somebody • So don’t go overboard! – A DAQ system won’t care where the data came from Central DAQ – Data stored in the same (expected) way from all sources Parallel DAQ-Lite – Quick data browsing – In some cases is all you need anyway Agreed ! Christopher Youngman, DESY January 29, 2008 37

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Using FLASH as a prototype for XFEL – V. Rybnikov talk design http: //doocs. desy. de outwardly complete user interface layer: GUIs (panels look old = XView? ) middle layer: FSM, Name service, EVB, Archive, Webservices hardware interface layer: DOOCS servers (many one offs) implementation outwardly complete implementation: C++, Java, MATLAB, LABview, Oracle Applications: Doocs Data Display, MATLAB, ROOT, e. Log DAQ functionality: Run control, error handling Used by FLASH and XFEL machines and some FLASH experiments Archive file formats: ROOT insufficient performance measured Move to something else – RAW format being tested what about LCLS experience (Nexus) ? Workshop result: DAQ+control starting with DOOCS is sensible Christopher Youngman, DESY January 29, 2008 38

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Rybnikov FLASH goals coincide with XFEL Christopher Youngman, DESY January 29, 2008 39

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Rybnikov Continuous upgrade of DOOCS = suits us as we may need new features Christopher Youngman, DESY January 29, 2008 40

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Rybnikov Expertise present Christopher Youngman, DESY January 29, 2008 41

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Rybnikov Christopher Youngman, DESY January 29, 2008 42

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DESY site data archiving – V. Guelzow+M. Gasthuber talk – given by V. Guelzow Five components of analysis chain network = 10 later 40/100 GE (Gbit/s Ethernet) - OK computer resources Ø Use GRID solutions = available, tuned by LHC, worldwide access and computing model. Ø workgroups computing (on- and off-site) Ø Data management and access using Grid tools Ø expect increased CPU performance via multi-core tech. storage = use disk cache and storage silos - OK Ø XFEL requirements look OK, 2013 ~3 x 5 GB/s, 2016 more. Ø costs money software – needs effort Ø OS (Linux) and compiler optimize for multi-core CPUs support = money and manpower (operation and software) Urgently need Computing Model/TDR document Christopher Youngman, DESY January 29, 2008 43

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber IT ongoing test program for CPUs and connectivity Christopher Youngman, DESY January 29, 2008 44

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber Christopher Youngman, DESY January 29, 2008 45

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber This is data management and offline computing requirements = new WP Christopher Youngman, DESY January 29, 2008 46

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber This is a fire ! Christopher Youngman, DESY January 29, 2008 47

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at ESRF – L. Cluastre talk ESRF beamlines BLISS (Beamline Inst. Software Support group) low-level driver to analysis and visualization software large group 18 people Control system development 1990 - TACO for acc. Control 1994 - SPEC as main control program 2004 - BLISS + TANGO (TACO compatible collab. DESY+) Future challenges 2008+ - addressing many XFEL type problems visualization on+offline analysis tools automated sample, exposure and result handling new beamline and experiment detectors Many parallels to DESY development ! Could not include slides easily – sorry. Christopher Youngman, DESY January 29, 2008 48

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at Diamond – T. Nicholls talk Diamond some numbers: Phase 1 – 7 beamlines 2007 Phase 2 – 15 beamlines 2012 1000 experiment proposals per year (8 hr to many day operations) Data management per year: 103 TB, 106 files, … Control and DAQ EPICS used for acc. and beamline control GDA – Generic Data Acquisition sits above EPICS and detector interfaces (non EPICS) GDA is the Diamond equivalent of DOOCS Java, XML, CORBA service broker, … similar three layer structure to DOOCS developed by STFC (RAL, Daresbury, …) DOOCS could profit from GDA cross developments (visualization, …) contacts made Christopher Youngman, DESY January 29, 2008 49

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at LCLS – A. Perazzo talk Data system architecture: readout chain: det. >frontend>acquistion (L 1) >processing (L 2) >archive control is L 0 (L=Level) 120 Hz timing system EVG+EVR system (Event Gen. and Recv. ) L 1 can veto and L 2 reject pulses L 1 common interface to all frontend systems Control system architecture EPICS based because used by machine (only reason) Networking Highly partioned: separate frontend thru L 2 slice, user networks, beamline systems, etc. Christopher Youngman, DESY January 29, 2008 50

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at LCLS – A. Perazzo talk – part 2 RCE = Reconfigurabe Cluster Element L 1 acquisition node Ø custom module Ø ATCA based, System On Chip technology – Xilinx Vitrex 4 Ø Small footprint Pretty Good Protocol (PGP) used for p-2 -p connections, many features (reliable, deterministic low latency, …) Ø PGP used to interface RCE to frontend CIM = Cluster Interconnect Module Switch module Ø custom built Ø 24 port fulcrum switch ASIC Ø interconnects RCE and L 2 networked nodes LCLS solutions very close to our baselines – but presently more thought through. Christopher Youngman, DESY January 29, 2008 51

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Data System Architecture – A. Perazzo Detector specific Photon Control Data Systems (PCDS) Beam Line Data L 1: Acquisition Detector + ASIC FEE Detector Timing L 0: Control L 2: Processing L 3: Data Cache – Experiment specific – May be bump-bonded to ASIC or integrated with ASIC Front-End Electronics (FEE) – Provide local configuration registers and state machines – Provide ADC if ASIC has analog outputs – FEE uses FPGA to transmit to DAQ system Christopher Youngman, DESY January 29, 2008 52

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Register Command Data Interface – A. Perazzo Reg. Addr[23: 0] Reg. Data. Out[31: 0] Reg. Req Reg. Op Req. Ack Reg. Fail Reg. Data. In[31: 0] Cmd. Ctx. Out[23: 0] Cmd. Opcode[6 : 0] Cmd. En Frame. Tx. Enable Frame. Tx. Sof Frame. Tx. Data. Width Frame. Tx. Eofe Frame. Tx. Data[15: 0] MG T Register Block Data Block MG T L 1 Node Interface defined between FEE and L 1 PGP Command Fiber Transceiver PCDS blocks Transceiver Detector specific blocks Bloc k – Common interface among different experiments – Provide data, command register interfaces – Custom point-to-point protocol (Pretty Good Protocol, PGP) implemented as FPGA IP core – FEE FPGA assumed to be Xilinx Virtex-4 FEE FPGA FX family with Multi Gigabit Transceivers (MGT) Statement is define a common interface for 2 D, 1 D and other detectors Christopher Youngman, DESY January 29, 2008 53

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Back End Zone Network Diagram – A. Perrazo DMZ SLAC Domain Service Traffic User Accelerator Domain Control & DAQ nodes NFS, DNS, NTP, AAA Service CDS We will have similar network partitioning Christopher Youngman, DESY January 29, 2008 Science bulk data Data cache machines DSS Front End Zone 54

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Conclusions and Path Forward – A. Perrazo LCLS common custom DAQ hardware devices fully prototyped – RCE, CIM and front-end development board Interface with FEE defined – Uses common (among different experiments) communication protocol • Implemented as IP cores interfaced to detector specific user logic in FEE FPGA – Need to develop detector specific user logic for ASIC operations Begun definition of the interface with off-line system Rather late – hope for us – Currently investigating HDF 5 as common data format for data cache Software development in progress on: DAQ partition management, user interface, calibration framework and data-flow Continue development on EPICS drivers for the control system of first LCLS experiment (AMOS) Christopher Youngman, DESY January 29, 2008 55

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Infrastructure requirements – T. Hott talk 5 important slides: extract milestones. ensure that our requirements present. Christopher Youngman, DESY January 29, 2008 56

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser T. Hott Christopher Youngman, DESY January 29, 2008 57

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser T. Hott Christopher Youngman, DESY January 29, 2008 58

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser T. Hott Christopher Youngman, DESY January 29, 2008 59

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser T. Hott Christopher Youngman, DESY January 29, 2008 60

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser T. Hott Christopher Youngman, DESY January 29, 2008 61

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Perspectives for data rejection and size reductions – G. Potdevin talk – first results from simulated pictures on large 2 D pixel detectors Data rejection Veto frames if no Fluorescence light seen in (e. g. ) PM system Ø assumed to work 100%, what about noise – needs simulation Online reduction in backend – orientation varies, physics varies Data reduction Zero suppression Ø assume no pedestal Data rejection+reduction needed to reduce archive rate Ø approach: simulate photons, add flat noise, add fluorescence, add detector imperfections. Ø use jpeg compresson algorithm to estimate reduction Ø once noise added little gain in data size Feature extraction Ø store difference between consecutive frames Need – more work, initial results not encouraging large set of images for different experiments improve background simulation cross check results with expt. Christopher Youngman, DESY January 29, 2008 62

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser G. Potdevin Data Rejection: Online analysis Intensities distribution and values vary greatly With the experiment type (XPCS, Single Particle Imaging, Pump. Probe diffraction, plasmas …) The size of the sample, its orientation… Ex. Three possible good measurements Nothing doable ? Christopher Youngman, DESY January 29, 2008 (case of several samples into the beam? Techniques Specific Criterias) 63

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser G. Potdevin Data Reduction: What do the images look like Unique Molecule Ferritin are se ho 24 T k. Da 1*1*1 crystal Unit 5*5*5 crystal Unit s!! ge ma si les ise No Images courtesy of Pr. Franz Pfeiffer, SLS & Lausanne University Christopher Youngman, DESY January 29, 2008 64

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser G. Potdevin Data Reduction: What do the images look like noise Contribution to the noise Sample Fluores cence Dirt e luor F g erin tt Sca or ct te De ise no e enc sc Can we suppress the noise from the image? Only the fixed pattern due to the detector. All recorded incoming photons must be kept. Christopher Youngman, DESY January 29, 2008 65

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Data Reduction: zero suppression (2) Lame attempt to simulate compression: – Conversion in jpeg with best quality (~lossless) Single Molecule, noiseless image 99. 8% blank pixels Reduced to 6% of original size Large crystal, noiseless image 92. 4% blank pixels Reduced to 80% of original size Large crystal noised image Christopher Youngman, DESY January 29, 2008 42. 8% blank pixels No gain 66

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Conclusion - G. Potdevin Data rejection with veto What will be the proportion of rejection is unknown Data reduction possible with Zero suppression What the data will look like know How strong the background will be know we don’t But, preliminary simulations tend to show that not so much can be gained in this direction Christopher Youngman, DESY January 29, 2008 67

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL 2 D pixel detector status Review as a single block HPAD LPD LSDD – now DEPFET Similarities: initially 1 Mpixel detector, later 2, 4, 8, 16, 25, … similar geometrical tile design sensor > ASIC > frontend > backend readout modular design, e. g. 32 modules = full detector similar ASIC 5 MHz ADC digitize data into pipeline similar readout pipeline processing/readout during inter-train gap similar control requirements Christopher Youngman, DESY January 29, 2008 68

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL 2 D pixel detector status Differences pixel sizes (LPD 500 x 500, HPAD&LSDD 200 x 200μm) gain handling: HPAD switch dynamically LPD 3 gains pick best LSDD DEPFET specific layout: HPAD Sensor&ASIC&frontend on detector head LPD Sensor&ASIC one detector head, frontend O(10)cm away LSDD Sensor&ASIC one detector head, frontend O(10)m away … Christopher Youngman, DESY January 29, 2008 69

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Modular frontend Auxiliary ASIC and passive components Flex hybrid Optional heat spreader Bump and wire bonds r/o ASICs sensitive DEPFET array Three building blocks sensor+ASIC tile module full detector LSDD and HPAD sensor and ASIC bump bonded LPD sensor and ASIC separated Modular design = allows increase in final full detector size !! Christopher Youngman, DESY January 29, 2008 70

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 2 D pixel detector sensor/ASIC/frontend parameters Parameter HPAD LPD LSDD technology Si, … Si - DEPFET pixel size 200 x 200 μm 500 x 500 μm 200 x 200 μm topology tile roof tile single γ sensitive yes yes soft x-ray sensitive no no yes Max. digitizing rate 5 MHz gain control switched 1 fold 3 fold DEPFET dynamic range (γ count) 0 to 5 x 104 0 to 105 0 to 104 ADC 14 bit (12 eff. ) 14 bit (12 eff) 10 bit (eff? ) pixel data size 2 bytes frame pipeline depth ≤ 400 512 500 Module count 32 32 16 readout IO channels 128 x 1 GE 96 x 1 GE 64 x 1 Gbit/s LVDS startup pixel count 1 k x 1 k startup frame size 2 Mbytes Heat load sensor/total ~1. 2 / 2. 4 k. W ? ? Detectors DAQ 2 D frontend parameters very similar = suit generic backend Christopher Youngman, DESY January 29, 2008 71

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Front end time sliced DAQ operation 0. 6 ms bunch train 99. 4 ms bunch gap frontend capture data to pipeline frontend format data for transfer f+backend build frames backend do something – analyze? backend build bunch trains Time slicing the data transfer and processing simplifies the conceptual design Using the bunch train gap also fixes the design – 30 Hz operation lower frame count Christopher Youngman, DESY January 29, 2008 72

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Basic approach slow info frontend timing backend DAQ and control implementation frontend = LSDD, HPAD, LPD specific: sensor, ASIC, pipeline buffers, … backend = generic readout, control and data archiving and management Baseline solution agreed to by the participating groups Christopher Youngman, DESY January 29, 2008 73

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 2 D pixel backend – C. Youngman At least three backend solutions have existed: HPAD 2007 PC farm backend LPD Eo. I ATCA backend LSDD e. PCI proposal backend XFEL advisory committee recommended having a generic backend design used by all detectors Christopher Youngman, DESY January 29, 2008 74

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser HPAD DAQ connectivity and time allowance wiretime/ available x 8 (ms) Two PC layers L 1 and L 2 collect complete frames in L 1 analyze frame properties 66/100 accept/reject frames ? 2 FL switch 1 x 12 collect bunch train frames in L 2 UDP 66/100 analyze bunches FL PC x 108 accept/reject bunches ? TCP 66/100 compress frames ? 2 SL switch 1 write to mass storage x 12 Data transfer 9 x 66/900 1 Gbit/s links 1 2 SL PC x 9 UDP rocket IO time sliced transfer in bunch gap 7128/8100 Transfer to archive requires big archiver 1 2 x 9 Will work for 1 Mpixel – scalability might be issue servers with 10 Gbit/s links Module 1 2 2 MB x 400 frames / (8 RIs x 12 channels)/1 Gbit/s = 66 ms Christopher Youngman, DESY January 29, 2008 75

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser LPD Straw Man Event Builder – J. Coughlan out in PC backend farm 1 board = 1 Mpixel 1 MPix Line Card Receiver 10 Gb. E Event builder and Processor. Assuming Data Processing with Data Reduction Need access to complete frames? Data Processing engines FPGA vs FPNA vs Micro. Processor? FPt ? ? Implementation AMC Mezzanines on Commercial ATCA Carrier Board Or AMCs in Micro. TCA Large scale data switching problem -> ATCA Serial mesh fabric sharing between 1 MPix cards Christopher Youngman, DESY January 29, 2008 76

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser LSDD DAQ box – A. Kugel Similar to ATLAS „ROS“ PC Intermediate layer with existing hardware – DAQ software framework – Global calibration – Control • FPGA – – – Serial link interface (8 links/card) 1 GB local memory to assemble images Online calibration, offset/gain compensation (Simple) Data processing 10 Gb. E output to BE-DAQ using UDP Christopher Youngman, DESY January 29, 2008 07. 03. 2008 XFEL DAQ Workshop 77 77

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser How to progress on backend Need agreement on a generic solution for all detectors Agree on protocol from frontend - assume UDP - need test measurements (from HPAD developers? ) Scalability issues for >1 Mpixel detectors Intermediate layer between frontend and backend PC farm Sub group formed to look at other intermediate layer solutions: M. French, J. Couglan, T. Nicholls, R. Halsall, P. Goettlicher, M. Zimmer, A. Kugel, J. Visschers, M. v. Beuzekom, C. Youngman Produce an on paper design of a ATCA intermediate to investigate feasibility. Assume Ø 10 GE inputs Ø 1 board per 1 Mpixel input Ø Build frames and ordered trains Ø Send result to backend PC farm Christopher Youngman, DESY January 29, 2008 78

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Other 2 D pixel issues Clock and Control interface Need definitive definition of machine parameters Starting point W. Decking’s talk at workshop Can a generic design be provided for Train config (LAN? ) Bunch clk (5 MHz) Train start Bunch veto Bunch clk (5 MHz) Train start Clock&Control Busy Frontend 1 or 10 GE Christopher Youngman, DESY January 29, 2008 Intermediate layer XFEL timing system PC 10 Gbit/s PC PC backend farm 79

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims - revisted Results and fires meet other groups, exchange ideas, etc done useful input from all: LCLS particularly interesting w. r. t. DAQ and control implementation DOOCS – GDA contact being established NIKHEF interested in in-kind work Slovak group interested in backend/data management/offline how to handle multiple requests (Uppsala) ? produce a list of work, milestones required = any fires (yes) need a computing TDR, see V. Guelzow’s talk still to do – extract milestones and full work list need definitive documents: machine parameters, experiment clock&control requirements timing system interface impelementation. Christopher Youngman, DESY January 29, 2008 80

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims – revisted cont. Results and fires continued clarify, if possible, work with other WPs will need a data management and offline WP soon no significant progress defining beamline and diagnostic interaction identify regions of in-kind contribution sub group for intermediate layer founded – potentially in-kind NIKHEF have submitted a participation outline: detector development 1 D frontend, 2 D cooling, … DAQ and control (trigger, FPGA, DSP) expertise is sufficient manpower and other resources available? No – needs finalizing, but could be to do control work need for HPAD (probably LSDD) need 1 person to select and integrate control systems 1 technical person for control circuitry work (possibly clock&control) Christopher Youngman, DESY January 29, 2008 81

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Acknowledgements Thanks to: Imke Gembalis the secretary The international organizing committee Heinz Graafsma and Andreas Schwarz for their suggestions The speakers for their presentations The participants for their attention Thomas Hott for leading the visit to FLASH and Hall 3 Christopher Youngman, DESY January 29, 2008 82