The European X-Ray Laser Project Summary of XFEL

The European X-Ray Laser Project Summary of XFEL DAQ and Control for photon beam systems workshop 10 -11 th March 2008 Agenda and presentations are available on: https: //indico. desy. de/conference. Display. py? conf. Id=762 The workshop was a Pre-XFEL project partially funded by the European Commission under the 7 th Framework programme. Christopher Youngman, DESY XFEL meeting - March 26, 2008 XFEL X-Ray Free-Electron Laser

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 47 registered participants ALESSANDRO, Polini alessandro. polini@bo. infn. it ANGELSEN, Christian. C. Angelsen@rl. ac. uk BILLICH, Heiner. Billich@psi. ch BOUROV, Sergueiserguei. bourov@desy. de CHELKOV (JINR), Georgychelkov@nu. jinr. ru CLAUSEN, Matthias. Clausen@desy. de CLAUSTRE, Laurentclaustre@esrf. fr COPPOLA, nicola, coppola@mail. desy. de COUGHLAN, John, J. COUGHLAN@RL. AC. UK DECKING, Winfried, winfried. decking@desy. de DIMPER, Rudolf, rudolf. dimper@esrf. fr DUVAL, Philip. Duval@desy. de ESENOV, Sergey, sergey. esenov@desy. de FURUKAWA, Yukito, furukawa@spring 8. or. jp GRAAFSMA, Heinz, heinz. graafsma@desy. de GUELZOW, Volker, volker. guelzow@desy. de GöTTLICHER, Peter, peter. goettlicher@desy. de HALSALL, Rob, R. Halsall@rl. ac. uk HATSUI, Takaki, hatsui@spring 8. or. jp HENRICH, Beat, beat. henrich@psi. ch HOMS PURON, Alejandro, homs@esrf. fr HOTT, Thomas. Hott@desy. de JALOCHA, Pawel. Jalocha@psi. ch KLEESE VAN DAM, Kerstink. kleese@dl. ac. uk KLORA, Jörg klora@cells. es KRACHT, Thorsten thorsten. kracht@desy. de KUGEL, Andreas, andreas. kugel@ziti. uni-heidelberg. de MANT, Geoffrey, g. r. mant@dl. ac. uk MURIN, Pavel, murin@vk. upjs. sk NICHOLLS, Tim, t. c. nicholls@stfc. ac. uk PERAZZO, Amedeo, perazzo@slac. stanford. edu PLECH, Anton, anton. plech@uni-konstanz. de POTDEVIN, Guillaume, guillaume. potdevin@desy. de REHLICH, Kay, kay. rehlich@desy. de RYBNIKOV, Vladimir. Rybnikov@desy. de SCHWARZ, Andreas, andreas. schwarz@desy. de SCHöPS, Andreas, andreas. schoeps@desy. de STEFFEN, Lothar, lothar. steffen@desy. de STEPHENSON, Paul, p. c. stephenson@dl. ac. uk TIEDTKE, Kai, kai. tiedtke@desy. de TRUNK, Ulrich, ulrich. trunk@desy. de VAN BAKEL, Niels, nielsvb@slac. stanford. edu VAN BEUZEKOM, Martin, martinb@nikhef. nl VISSCHERS, Janjanv@nikhef. nl WINTER, Graeme, g. winter@dl. ac. uk YOUNGMAN, Christopher. Youngman@desy. de ZIMMER, Manfred, manfred. zimmer@desy. de ALBA, DESY(20), Daresbury(4), ESRF(3), ITEP, JINR, NIKHEF(2), PSI(3), RAL(4), SLAC(2), Spring 8(2), Bologna, Heidelberg, Konstanz, Slovakia, … Christopher Youngman, DESY XFEL meeting - March 26, 2008 2

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims Review ALL areas of WP 76 = 8 sections & 20 talks & 529 slides Machine parameters and timing Photon beam line instruments and detectors Control systems Archiving and data processing DAQ and control at other Labs Infrastructure requirements Perspectives for data rejection and size reductions 2 D pixel detectors Aims meet other groups, exchange ideas, etc. produce a list of work, milestones required = any fires clarify, if possible, work with other WPs identify regions of in-kind contribution is sufficient manpower and other resources available? … Christopher Youngman, DESY XFEL meeting - March 26, 2008 3

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser How to summarize 20 talks or 529 slides ? Talks reviewed: sometimes performed individually (Decking, Rehlich, …) sometimes as part of a related group (2 D-pixel detectors, …) comments/highlights added to slides are in green Talks not reviewed – lack of time Large scale data retrieval and processing Analysis tools Fast 2 D data acquisition at ESRF beam lines Other simplifications: My talks not reviewed – instead I have built my comments into the slides Christopher Youngman, DESY XFEL meeting - March 26, 2008 4

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL machine parameters relevant to beam lines – W. Decking talk = understand the machine parameters seen by instruments. explain allowed bunch structure - walk through e+γ machine find “new” facts every time reviewed: e beam dumps limit pulses per beam line to 1500 (new) at full E train to train variation of pulses delivered possible energy variation pulse to pulse and train to train Parameter Nominal Max variation range beam energy (E) 17. 5 Ge. V ? ≤E≤? train repetition rate 10 Hz 30 Hz at ½ E pulses train (N) 3000 0 ≤ N ≤ 3000 pulse separation 200 ns 200 ≤ N ≤ few micro-secs ΔE pulse in train (sweep/chirps/steps) 0 +/- 1. 5% over delivered train DAQ frontend+backend readout configuration driven by timing parameters Facts needed for frontend systems – need definition document ! Will need a tool to decide which beamline gets what Christopher Youngman, DESY XFEL meeting - March 26, 2008 5

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Timing & data synchronisation for machine and experiments – K. Rehlich talk = how do the instruments get triggered / synchronized Kay’s vision femto-sec to pico-sec stable clocks, event triggers, XFEL wide train# and pulse# unique tag of expt. and machine data XFEL timing = FLASH development = more than 1 system few pico-sec jitter for experiment clocks and events (≤ ½ ns required) ~10 femto-sec jitter for demanding users (RF feedback, Laser) Experiment interfaced to system via TCA board development of Tönsch IP module at FLASH distributes received timing events (e. g. train start) and datagram Ø delayed by programmable value Ø datagram = train number, bunch pattern, energies, etc. input from users required Ø datagram definitions (defn. of data required, LAN requirement? ) prototype expected end 2008 Need a final definitions document (connectors, datagrams, etc. ) Need review of usage at 2 D, 1 D, 0 D instruments (sharing, etc. ) Christopher Youngman, DESY XFEL meeting - March 26, 2008 6

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Photon beam line and diagnostic systems – K. Tiedtke talk = what’s there and what does DAQ and control look like ? Beam line systems: design and prototyping for XFEL ongoing Systems: shutters, mirrors, … Ø Use beam line systems at FLASH as baseline Ø Control system = HASYLAB (DORIS and PETRA) Ø Train sensitive timing (~0. 1 s) Diagnostic systems: design and prototyping for XFEL ongoing Systems: pulse intensity, wave front, … Ø Must redesign detectors and DAQ for higher rates, lower cross-sections, etc. Ø Control system = use FLASH solution Ø Pulse sensitive need connection to timing system Still missing basic LISTS of what & where, responsibilities for readout, requirements on other groups !! Not easy, but must be done. Christopher Youngman, DESY XFEL meeting - March 26, 2008 7

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Experiment systems 0 D, 1 D and 2 D detectors – H. Graafsma Startup scenario from June 2007 foresees 3 beamlines (red) T. Tschentscher et al. Distribution of the 5 (3) beamlines, with 2 experimental stations each Christopher Youngman, DESY XFEL meeting - March 26, 2008 8

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Experiment systems 0 D, 1 D and 2 D detectors – H. Graafsma talk = what experiment detectors are expected. detectors in various stages of prototyping/design 0 D little work done 1 D initial work started (128 pixel strips) Øcollaboration with FLA started and possibly NIKHEF ØDAQ and control look like a slice of 2 D ØSensor/ASIC/~512(? ) deep pipeline/frontend ØData transfer during train gap (~1 MB/s) Ømore pixels by adding slices together. Three 2 D proposals in, DAQ and control being developed Øsee 2 D-pixel detector talks later ØDAQ and control ØSensor/ASIC/≤ 512 deep pipeline/frontend ØData transfer during train gap (~5 GB/s) Ø 1 Mpixel detectors now, but 2, 4, 8, 16… later. Christopher Youngman, DESY XFEL meeting - March 26, 2008 9

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Layout of 1 D detector array (1 D-DA) H. Schlarb PD array ADC N x 10 bit DATA processing ASIC control Controls/ Arch. Crate u. TCA FPGA … N channels 128, … 1024? Gain & Shaping Rocket IO 5 MHz Two readout streams: 1. Fast feedback lvds 2. IP(? ) readout to storage needed Christopher Youngman, DESY XFEL meeting - March 26, 2008 >50 MHz Machine Clock & Trigger Feed back 10

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Experiment systems 0 D, 1 D and 2 D detectors – H. Graafsma talk = what experiment detectors are expected ? 1 D and 2 D detectors should share solutions generic backend DAQ and control generic clock&control hardware generic (slow) control systems Christopher Youngman, DESY XFEL meeting - March 26, 2008 11

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Other boundary conditions: • We need to modify the experiment: add, remove (auxiliary) detectors within a day flexibility • We want to be able to use different (new) detectors flexibility and standardization. • We want to be able to use larger detectors in the future (4, 9, 25, … Mpixels) Modular approach • We need to CONTROL the experiment (see SR-talks). Means “move and count” (part of) the data needs to be visible “immediately” • We need to store other data (machine and experiment) with the images • We want to store only “useful” images (fast veto) • We will have single module prototypes by 2010 (LCLS; Petra, …) Heinz’s wish list ! Christopher Youngman, DESY XFEL meeting - March 26, 2008 12

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Control systems at DESY– P. Duval talk = what control systems are in use at DESY non legacy control systems used at DESY: in house: DOOCS, TINE, SPECTRA (hasylab) external: EPICS (cryogenics+utilities), TANGO (hasylab) commercial: PVSS (H 1), D 3 (cryogenics) Grand unification of control systems effort underway TINE-DOOCS, EPICS-DOOCS, etc. but further diversification (TANGO) = more interfaces. no agreement likely for using one, so use the most convenient. For the uninitiated – explains why there is not one light source control package. Christopher Youngman, DESY XFEL meeting - March 26, 2008 13

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Control Systems (one way or another) have to deal with … - P. Duval Distributed end points and processes Data Acquisition (front end hardware) Real-time needs (where necessary) Process control (automation, feedback) Central Services (Archive, Alarm, Name Resolution, …) Security (who’s allowed to do what from where? ) States (Finite State Machines, sequencing, automation…) Time synchronization (time stamps, cycle ids, etc. ) Databases (configuration, machine data, post-mortem data, …) Statistics (control system itself, operation, …) Logging (central, local, application, …) Data transport (data flow, control system protocol, scalability) DAQ : focus on Data Acquisition and Central Services ! Agrees with our requirements if experiment algorithm software is added Christopher Youngman, DESY XFEL meeting - March 26, 2008 14

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Using FLASH as a prototype for XFEL – V. Rybnikov talk = explains why we start with DOOCS for the control software design http: //doocs. desy. de outwardly complete user interface layer: GUIs (XView panels look old – replace with Java) middle layer: FSM, Name service, EVB, Archive, Webservices hardware interface layer: DOOCS servers (one per instrument type) implementation outwardly complete implementation: C++, Java, MATLAB, LABview, Oracle Applications: Doocs Data Display, MATLAB, ROOT, e. Log DAQ functionality: Run control, error handling Used by FLASH and XFEL machines and some FLASH experiments Archive file formats: ROOT insufficient performance measured Move to something else – RAW format being tested what about LCLS experience (Nexus) ? Workshop result: DAQ+control starting with DOOCS is sensible Christopher Youngman, DESY XFEL meeting - March 26, 2008 15

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Rybnikov Ongoing development = we profit from their work and can potentially get our requirements inserted. Christopher Youngman, DESY XFEL meeting - March 26, 2008 16

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DESY site data archiving – V. Guelzow (+M. Gasthuber) talk – storage, data management and the GRID Five components of analysis chain network = 10 later 40/100 GE (Gbit/s Ethernet) - OK computer resources Ø Use GRID solutions = available, tuned by LHC, worldwide access and computing model. Ø workgroups computing (on- and off-site) Ø Data management and access using Grid tools Ø expect increased CPU performance via multi-core tech. storage = use disk cache and storage silos - OK Ø XFEL requirements look OK, 2013 ~3 x 5 GB/s, 2016 more. Ø costs money software – needs effort Ø OS (Linux) and compiler optimize for multi-core CPUs support = money and manpower (operation and software) Urgently need Computing Model/TDR document Christopher Youngman, DESY XFEL meeting - March 26, 2008 17

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber Christopher Youngman, DESY XFEL meeting - March 26, 2008 18

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber This is data management and offline computing requirements = new WP Christopher Youngman, DESY XFEL meeting - March 26, 2008 19

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser V. Guelzow+M. Gasthuber This is a fire ! Christopher Youngman, DESY XFEL meeting - March 26, 2008 20

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at ESRF – L. Cluastre talk = useful parallel developments? ESRF beamlines BLISS (Beamline Inst. Software Support group) low-level driver to analysis and visualization software large group 18 people Control system development 1990 - TACO for acc. Control 1994 - SPEC as main control program 2004 - TANGO (TACO compatible collab. with DESY+) Future challenges 2008+ - addressing many XFEL type problems visualization on+offline analysis tools automated sample, exposure and result handling new beamline and experiment detectors Future developments might be of interest to XFEL. Christopher Youngman, DESY XFEL meeting - March 26, 2008 21

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at Diamond – T. Nicholls talk = useful parallel developments? Diamond some numbers: Phase 1 – 7 beamlines 2007 Phase 2 – 15 beamlines 2012 1000 experiment proposals per year (8 hr to many day operations) Data management per year: 103 TB, 106 files, … Control and DAQ EPICS used for acc. and beamline control GDA – Generic Data Acquisition sits above EPICS and detector interfaces (non EPICS) GDA is the Diamond equivalent of DOOCS Java, XML, CORBA service broker, … similar three layer structure to DOOCS developed by STFC (RAL, Daresbury, …) DOOCS could profit from GDA cross developments (visualization, …) contacts made Christopher Youngman, DESY XFEL meeting - March 26, 2008 22

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at LCLS – A. Perazzo talk = asked to concentrate on hardware developments Data system architecture: readout chain: det. >frontend>acquistion (L 1) >processing (L 2) >archive control is L 0 (L=Level) 120 Hz timing system EVG+EVR system (Event Gen. and Recv. ) L 1 can veto and L 2 reject pulses L 1 common interface to all frontend systems Control system architecture EPICS based because used by machine (only reason) Networking Highly partioned: separate frontend thru L 2 slice, user networks, beamline systems, etc. Christopher Youngman, DESY XFEL meeting - March 26, 2008 23

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control at LCLS – A. Perazzo talk – part 2 RCE = Reconfigurabe Cluster Element L 1 acquisition node Ø custom module Ø ATCA based, System On Chip technology – Xilinx Vitrex 4 Ø Small footprint Pretty Good Protocol (PGP) used for p-2 -p connections, many features (reliable, deterministic low latency, …) Ø PGP used to interface RCE to frontend CIM = Cluster Interconnect Module Switch module Ø custom built Ø 24 port fulcrum switch ASIC Ø interconnects RCE and L 2 networked nodes LCLS solutions very close to our baselines – but presently more thought through. Christopher Youngman, DESY XFEL meeting - March 26, 2008 24

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Data System Architecture – A. Perazzo Detector specific Photon Control Data Systems (PCDS) Beam Line Data L 1: Acquisition Detector + ASIC FEE Detector Timing L 0: Control L 2: Processing L 3: Data Cache – Experiment specific – May be bump-bonded to ASIC or integrated with ASIC Front-End Electronics (FEE) – Provide local configuration registers and state machines – Provide ADC if ASIC has analog outputs – FEE uses FPGA to transmit to DAQ system Christopher Youngman, DESY XFEL meeting - March 26, 2008 25

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Register Command Data Interface – A. Perazzo Reg. Addr[23: 0] Reg. Data. Out[31: 0] Reg. Req Reg. Op Req. Ack Reg. Fail Reg. Data. In[31: 0] Cmd. Ctx. Out[23: 0] Cmd. Opcode[6 : 0] Cmd. En Frame. Tx. Enable Frame. Tx. Sof Frame. Tx. Data. Width Frame. Tx. Eofe Frame. Tx. Data[15: 0] MG T Register Block Data Block MG T L 1 Node Interface defined between FEE and L 1 PGP Command Fiber Transceiver PCDS blocks Transceiver Detector specific blocks Bloc k – Common interface among different experiments – Provide data, command register interfaces – Custom point-to-point protocol (Pretty Good Protocol, PGP) implemented as FPGA IP core – FEE FPGA assumed to be Xilinx Virtex-4 FEE FPGA FX family with Multi Gigabit Transceivers (MGT) Statement is define a common interface for 2 D, 1 D and other detectors Christopher Youngman, DESY XFEL meeting - March 26, 2008 26

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Back End Zone Network Diagram – A. Perrazo DMZ SLAC Domain Service Traffic User Accelerator Domain Control & DAQ nodes NFS, DNS, NTP, AAA Service CDS XFEL will have similar network partitioning Christopher Youngman, DESY XFEL meeting - March 26, 2008 Science bulk data Data cache machines DSS Front End Zone 27

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser DAQ and control infrastructure requirements – T. Hott talk = when and what infrastructure is required? Planning status for Experimental hall and tunnels. little planned so far ! requirements for: space, power, air/water cooling, network connections Milestones: Areas accessible for installation: Ø Ø XTD ~ Apr. 2011 XTD 6 -10 ~ Feb. 2012 (photon beamlines) Expt. Hall underground ~ May 2012 (hutches) Expt. Hall surface ~ Dec 2012 Current first dates for: Ø 1 st beam injector ~ end. 2012 Ø 1 st beam Linac ~ end. 2013 Ø 1 st SASE(1) at 0. 2 nm ~ end. 2014 Needed now: (Thursday: 1 st meeting with IT hall+tunnel IT infrastructure) Space requirements catalogue for Expt. Hall – soon Power (UPS, etc. ) air/water cooling, networks – June 2008 Cables and Fibre routing – end 2008 define budget requirement – soon Christopher Youngman, DESY XFEL meeting - March 26, 2008 28

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Perspectives for data rejection and size reductions – G. Potdevin talk – first results from simulated pictures on large 2 D pixel detectors Data rejection and reduction needed to reduce archive rate !! Frame rejection ideas Veto frames if no Fluorescence light seen in (e. g. ) PM system Ø assumed to work 100%, what about noise – needs simulation Online reduction in backend – orientation varies, physics varies – need sophisticated analysis software Data reduction ideas Zero suppression Ø assume no pedestal Ø approach: simulate photons, add flat noise, add fluorescence, add detector imperfections. Ø use jpeg compression algorithm to estimate reduction Ø once noise added little gain in data size Feature extraction Ø store difference between consecutive frames Christopher Youngman, DESY XFEL meeting - March 26, 2008 29

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Data Reduction: zero suppression (2) Lame attempt to simulate compression: – Conversion in jpeg with best quality (~lossless) Single Molecule, noiseless image 99. 8% blank pixels Reduced to 6% of original size Large crystal, noiseless image 92. 4% blank pixels Reduced to 80% of original size Large crystal noised image Christopher Youngman, DESY XFEL meeting - March 26, 2008 42. 8% blank pixels No gain 30

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Conclusion - G. Potdevin Data rejection with veto ® What will be the proportion of rejection is unknown Data reduction possible with Zero suppression ® What the data will look like we don’t know ® How strong the background will be we don’t know But, preliminary simulations tend to show that not so much can be gained in this direction Early days – more work needed, initial results not encouraging large set of images for different experiments improve background simulation cross check results with expt. Christopher Youngman, DESY XFEL meeting - March 26, 2008 31

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL 2 D pixel detector status Review as a single block HPAD LPD LSDD – now DEPFET Similarities: initially 1 Mpixel detector, later 2, 4, 8, 16, 25, … similar geometrical tile design sensor > ASIC > frontend > backend readout modular design, e. g. 32 modules = full detector similar ASIC 50 MHz ADC digitize data into pipeline similar readout pipeline processing/readout during inter-train gap similar control requirements Christopher Youngman, DESY XFEL meeting - March 26, 2008 32

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser XFEL 2 D pixel detector status Differences pixel sizes (LPD 500 x 500, HPAD&LSDD 200 x 200μm) gain handling: HPAD switch dynamically LPD 3 gains pick best LSDD DEPFET specific layout: HPAD Sensor&ASIC&frontend on detector head LPD Sensor&ASIC one detector head, frontend O(10)cm away LSDD Sensor&ASIC one detector head, frontend O(10)m away … Christopher Youngman, DESY XFEL meeting - March 26, 2008 33

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Modular frontend Auxiliary ASIC and passive components Flex hybrid Optional heat spreader Bump and wire bonds r/o ASICs sensitive DEPFET array Three building blocks sensor+ASIC tile module full detector LSDD and HPAD sensor and ASIC bump bonded LPD sensor and ASIC separated Modular design = allows increase in final full detector size !! Christopher Youngman, DESY XFEL meeting - March 26, 2008 34

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 2 D pixel detector sensor/ASIC/frontend parameters Parameter HPAD LPD LSDD technology Si, … Si - DEPFET pixel size 200 x 200 μm 500 x 500 μm 200 x 200 μm topology tile roof tile single γ sensitive yes yes soft x-ray sensitive no no yes Max. digitizing rate 5 MHz gain control switched 1 fold 3 fold DEPFET dynamic range (γ count) 0 to 5 x 104 0 to 105 0 to 104 ADC 14 bit (12 eff. ) 14 bit (12 eff) 10 bit (eff? ) pixel data size 2 bytes frame pipeline depth ≤ 400 512 500 Module count 32 32 16 readout IO channels 128 x 1 GE 96 x 1 GE 64 x 1 Gbit/s LVDS startup pixel count 1 k x 1 k startup frame size 2 Mbytes Heat load sensor/total ~1. 2 / 2. 4 k. W ~1. 0– 2. 0 k. W ? Detectors DAQ 2 D frontend parameters very similar = suit generic backend Christopher Youngman, DESY XFEL meeting - March 26, 2008 35

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Basic approach slow info frontend timing backend DAQ and control implementation frontend = LSDD, HPAD, LPD specific: sensor, ASIC, pipeline buffers, … backend = generic readout, control and data archiving and management Baseline solution agreed to by the participating groups Christopher Youngman, DESY XFEL meeting - March 26, 2008 36

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser 2 D pixel backend – C. Youngman At least three backend solutions have existed: HPAD 2007 PC farm backend LPD Eo. I ATCA backend LSDD e. PCI proposal backend XFEL advisory committee recommended having a generic backend design used by all detectors Christopher Youngman, DESY XFEL meeting - March 26, 2008 37

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser HPAD – PC farm backend 1 Module wiretime/ available x 8 (ms) 2 66/100 FL switch 1 2 x 12 66/100 UDP FL PC x 108 TCP SL switch Two PC layers L 1 and L 2 collect complete frames in L 1 collect bunch train frames in L 2 1 Gbit/s links UDP rocket IO 1 2 66/100 Use 10 GE as standard x 12 9 x 66/900 SL PC 1 2 x 9 archiver 1 2 x 9 Christopher Youngman, DESY XFEL meeting - March 26, 2008 Possible to build trains in 1 layer 10 GE links need more memory at frontend 7128/8100 obviously fewer links 38

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Front End Readout 2 nd Stage – J. Coughlan Close to Detector FEM Flexible Layer Concentrator Buffering DMA controller Calibration corrections Peds. . XFEL Formatting (headers id) Serial Optical output GEthernet Traffic Shaping + Reject bunches MSCs FPGA Interface to DAQ Interface to CTR Fast Timing and Ctrls System Receive clocks, synch, event nr, bunch fill information, vetos? Christopher Youngman, DESY Configuration User Controls via DOOCs XFEL meeting - March 26, 2008 39

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser LPD backend Straw Man Event Builder – J. Coughlan out in PC backend farm 1 board = 1 Mpixel 1 MPix Line Card Receiver 10 Gb. E Event builder and Processor. Assuming Data Processing with Data Reduction Need access to complete frames? Data Processing engines FPGA vs FPNA vs Micro. Processor? FPt ? ? Implementation AMC Mezzanines on Commercial ATCA Carrier Board Or AMCs in Micro. TCA Large scale data switching problem -> ATCA Serial mesh fabric sharing between 1 MPix cards Christopher Youngman, DESY XFEL meeting - March 26, 2008 40

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser LSDD backend DAQ box – A. Kugel Similar to ATLAS „ROS“ PC Intermediate layer with existing hardware – DAQ software framework – Global calibration – Control • FPGA – – – Serial link interface (8 links/card) 1 GB local memory to assemble images Online calibration, offset/gain compensation (Simple) Data processing 10 Gb. E output to BE-DAQ using UDP Christopher Youngman, DESY XFEL meeting - March 26, 2008 07. 03. 2008 XFEL DAQ Workshop 41 41

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser How to progress on backend Need agreement on a generic solution for all detectors Agree on protocol from frontend - assume UDP - need test measurements rates/errors (from HPAD developers and WP 76) Scalability issues for >1 Mpixel detectors Intermediate layer between frontend and backend PC farm Sub group formed to look at other intermediate layer solutions: M. French, J. Couglan, T. Nicholls, R. Halsall, P. Goettlicher, M. Zimmer, A. Kugel, J. Visschers, M. v. Beuzekom, C. Youngman Produce an on paper design of a ATCA intermediate to investigate feasibility. Assume Ø 10 GE inputs Ø 1 board per 1 Mpixel input Ø Build frames and ordered trains Ø Send result to backend PC farm LPD had looked at other solutions: i. WARP, … Christopher Youngman, DESY XFEL meeting - March 26, 2008 42

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Other 2 D pixel issues Clock and Control interface Need definitive definition of machine parameters Starting point W. Decking’s talk at workshop Aim at generic design for all 1 D and 2 D detectors Train config (LAN? ) Bunch clk (5 MHz) Train start Bunch veto Bunch clk (5 MHz) Train start Clock&Control Busy Frontend 1 or 10 GE Christopher Youngman, DESY XFEL meeting - March 26, 2008 Intermediate layer XFEL timing system PC 10 Gbit/s PC PC backend farm 43

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims - revisted Results and fires meet other groups, exchange ideas, etc done useful input from all: LCLS particularly interesting w. r. t. DAQ and control implementation DOOCS – GDA contact being established NIKHEF interested in in-kind work Slovak group interested in backend/data management/offline how to handle multiple requests (Uppsala) ? produce a list of work, milestones required = any fires (yes) need a computing TDR, see V. Guelzow’s talk hall and tunnel infra structure requirements need definitive documents: milestones and full work list machine parameters, experiment clock&control requirements timing system interface impelementation. Christopher Youngman, DESY XFEL meeting - March 26, 2008 44

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Aims – revisted cont. Results and fires continued clarify, if possible, work with other WPs will need a data management and offline WP soon no significant progress defining beamline and diagnostic interaction identify regions of in-kind contribution sub group for intermediate layer founded – potentially in-kind NIKHEF have submitted a participation outline: detector development 1 D frontend, 2 D cooling, … DAQ and control (trigger, FPGA, DSP) expertise is sufficient manpower and other resources available? No – needs finalizing, but could be to do control work need for HPAD (probably LSDD) need 1 person to select and integrate control systems 1 technical person for control circuitry work (possibly clock&control) Christopher Youngman, DESY XFEL meeting - March 26, 2008 45

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Acknowledgements Thanks to: Imke Gembalis the secretary The international organizing committee Heinz Graafsma and Andreas Schwarz for their suggestions The speakers for their presentations The participants for their attention Thomas Hott for leading the visit to FLASH and Hall 3 Christopher Youngman, DESY XFEL meeting - March 26, 2008 46

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Spare slides Christopher Youngman, DESY XFEL meeting - March 26, 2008 47

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser Nomenclature used here: pulse = packet of photons or electrons sometimes called a bunch (e-beam context) train = consecutive group of pulses or bunches sometimes called a bunch train (e-beam) sometimes called a pulse train (γ-beam) sometimes called a macro-bunch (? ) train number unique incremented number for each train generated by XFEL ~1010 after 20 years of 30 Hz operation sometimes called the event number Christopher Youngman, DESY XFEL meeting - March 26, 2008 48

The European X-Ray Laser Project XFEL X-Ray Free-Electron Laser HPAD = time sliced DAQ operation 0. 6 ms bunch train 99. 4 ms bunch gap frontend capture data to pipeline frontend format data for transfer f+backend build frames backend do something – analyze? backend build bunch trains Time slicing the data transfer and processing simplifies the conceptual design Using the bunch train gap also fixes the design – 30 Hz operation lower frame count Christopher Youngman, DESY XFEL meeting - March 26, 2008 49