HCAL Trigger Readout HTR Status and Clocking Issues

HCAL Trigger Readout HTR Status and Clocking Issues D. Baden, T. Grassi http: //www. physics. umd. edu/hep/esr_dec_2002. pdf CMS ESR December 2002 1

FE/DAQ Electronics S-Link: 64 bits @ 25 MHz Trigger Primitives CAL REGIONAL TRIGGER READ-OUT Crate S B S Rack CPU C L K D C C H T R QIE QIE QIE 12 HTRs per Readout Crate, 2 DCC Shield Wall GOL CCA QIE CCA HPD Fibers at 1. 6 Gb/s 3 QIE-channels per fiber GOL CCA (On detector) 32 bits @ 40 MHz H T R 16 bits @ 80 MHz TTC FRONT-END RBX Readout Box H T R FE MODULE CMS ESR December 2002 2

HTR Principal Functions 1. Receive front-end data for physics running • Synchronize optical links • Data validation and linearization • Form TPG’s and transmit to Level 1 at 40 MHz • Pipeline data, wait for Level 1 accept • Upon receiving L 1 A: • Zero suppress, format, transmit to the concentrator (no filtering) • Handle DAQ synchronization issues (if any) 2. Calibration processing and buffering of: • Radioactive source calibration data • Laser/LED calibration data 3. Support a VME data spy monitoring CMS ESR December 2002 3

Readout VME Crate “BIT 3” board – Slow monitoring over VME – Commercial VME/PCI Interface to CPU Front End Electronics TTC fiber Fiber 1. 6 Gb/s Fan. Out board – Takes TTC stream in – Clone and Fanout timing signals HTR (HCAL Trigger and Readout) board D – C C– Spy output over VME FE-Fiber input – TPG output (SLBs) to CRT – DAQ/TP Data output to DCC (Data Concentrator Card) board – Input from HTRs – Spy output – Output to DAQ F a H H B n D T T I O C R R T u C 3 t . . . H H T T R R VME CRATE DAQ 20 m Copper 1. 2 Gb/s Calorimeter Regional Trigger CMS ESR December 2002 4

“Old” HTR Design (Summer 2002) • Board organized around 2 identical sets of circuitry: • • • Optical inputs • 1. 6 GHz, 8 B/10 B frames, 3 ch/link • Dual LC detectors and drivers TI TLK 2501 Deserializers • Crystal Ref. Clk • TTC 80 MHz backup Xilinx Virtex FPGA XCV 1000 E • 24 channels each TPG signals • Sent to SLB over backplane, LVDS • SLBs mounted 6 to a transition board Level 1 accept output to DCC • LVDS output OLD DESIGN VME • Altera FPGA and firmware CMS ESR December 2002 5

HTR Functional Experience • What was tested: • VME fully tested and working • Some changes necessary to conform to CMS VME standards • Optical links and synchronization • No indication of any problems. Big success here – was a real worry • LVDS to DCC • Tested, working (Will change cable/connector to Cat 6/RJ 45) • Fanout of timing signals on two Cat 5 cables • Plan to change to a single Cat 6 or Cat 7 cable (very low cross-talk) • Firmware – full tests of: • Pipeline and L 1 A triggering capability • In-line histogramming for source calibration • TTCrx • Not working at all (4 bad on 4 tested). • What was not tested: Anything to do with TPG CMS ESR December 2002 6

HTR Board Experience • Produced ~12 boards • Several bare boards were delivered warped • Many opens under FPGA after assembly (~9 boards) • Some fixed after reflow (a few) • Some worse after reflow (shorts) • X-rayed a few boards, sometimes inconclusive • Some opens on VME side • Non BGA FPGA, indicates bad vias • Few other various open circuits • Finally got ~8 boards to “work” • Questionable reliability CMS ESR December 2002 7

Modifications • Change board from using white-tin to gold traces • This process was sold to us by the board maker. Our mistake. • Used only for very high volume, cost competitive products, very difficult and expensive to control. • Gold is flatter and not very much more expensive (~$50/board), better for FPGAs • Change assembly house • Insufficient Quality Control on current assembler – they are fired. • We visited 2 high-end assemblers • Modern Machines • Step up and step down oven temp control. • In-line X-ray for BGA QC • Manufacturability Review • Add stiffeners to HTRS • Flexability of 9 U VME boards was underestimated • Worry: fine-line BGA (FBGA) can pop connections CMS ESR December 2002 8

Modifications (cont) • Change from FBGA to BGA • FBGA 1. 0 mm pitch, change to BGA 1. 27 mm pitch • No additional expense, plenty of available real estate, no need to push • We are just being very conservative here • JTAG capabilities added • Will help with debugging • By making these changes… • We have profited from the summer • We have reduced our production risk considerably CMS ESR December 2002 9

HTR Design Changes • SLB transition board issues: • Worries about so many LVDS signals over backplane for old design • Routing is too complicated • Many signals going to same backplane location • Requires multi-layer routing with many vias • TPG cables very thick • Mechanical issues are very worrisome • SLB changes needed (e. g. height reduced after ECAL redesign…) • Solution: move SLB’s to HTR motherboard • Benefits: • Mechanically attach SLB’s to HTR front panel for mechanical stability • Eases routing requirements, reduces board and assembly risks, cheaper too • Change from Xilinx Virtex. E to Virtex 2 • More resources, block ram, hardware multipliers • Big cost reduction (save $180 k) • More modern chip for long-term maintenance • Clock synchronization • Decouple “ 80 MHz” crystal from FPGA system clock • Will allow us to use crystal to maintain synchronization of serdes • This gives us 2 solutions for our “ 40 ps” jitter requirement issue CMS ESR December 2002 10

New HTR Conceptual Design 8 -way TI TI to DCC LVDS FPGA Xilinx XC 2 V LVDS SLB to Level 1 Cal Trigger TI SLB TI TI SLB FPGA Xilinx XC 2 V SLB RJ 45 CMS ESR December 2002 LC LC VME FPGA P 2 TI SLB from Fanout TI P 1 LC LC TTC TI TI LC LC No P 3! Fibers LC LC TI 11

Clocking Schematic • Start with Fanout card • TTCrx Maryland mezzanine card or CERN TTCrm daughterboard • QPLL • Fanout on Cat 6/7 quad twisted pair TTC, BC 0, 40 MHz, 80 MHz • In HTR: • Send TTC signal to TTCrx mezzanine board, access to all TTC signals • Send 80 MHz clean clock (cleaned by QPLL) to mux • Select 80 MHz clean clock OR crystal to TI deserializers Cat 6/7 quad cable (allows PECL) TTC Fanout Board TTCrx QPLL TTC BC 0 80 MHz 40 MHz 80 MHz LVPECL Crystal 80 MHz BC 0 1 to 8 Fanout 80 MHz 40 MHz TTC Single width VME CMS ESR December 2002 1 to 8 Fanout TTC mezz TI (16) F P G A SLB SLB SLB 40 MHz TTC broadcast bus 12

HCAL TRIDas Clock Scheme Fanout Card TTCrx Cat 6/7 RJ 45 QPLL TTC RJ 45 TTCMezz CC 80 TTC BC 0 CC 40 CC 80 4 twisted pair… (‘CC’ means Clean Clock) TTC broadcast, L 1 A, BCR, EVR, CLK 40 Xilinx CC 40 BC 0 CMS ESR December 2002 SLB HTR Board 13

Fanout – HTR scheme Notes: SLBs require fanout of CLK 40, BC 0. FE-link possibly requires CLK 80. PECL fanout was tested in TB 2002. One Cat 6 E cable (low x-talk) replaces the 2 Cat 5 cables used in TB 2002. TTC and BC 0 remain LVDS as in Weiming’s board. HTR needs Broadcast bus, BCnt. Res and L 1 A: from TTCrx if we get it to work, otherwise we have to fan them out. TTC fiber Fanout buffer TTC O/E TTC LVDS Low-jitter Fanout x 15 TTC Brdcst<7: 2>, Brcst. Str TTCrx ~Fifteen RJ 45 connectors LVDS BC 0 FPGA (or daughter card) CLK 40 LVDS QPLL PECL fanout CLK 80 LVDS AN 1568/D Fig 11 Onsemi. com . . . PECL fanout Cat 6 E or Cat 7 cable 2 Test Points for CLK 40 and BC 0 NB 100 LVEP 221 is LVDS compatible Fanout Board Brdcst<7: 2>, Brcst. Str, CMOS BCnt. Res, L 1 A ……. . LVDS or diff PECL 15 Cables & Connectors tbd 15 connectors on bottom layer ? 9 U Front-panel space = 325 mm ; => space per connector ~ 21. 5 mm CMS ESR December 2002 RX_BC 0 LVDS . . . LVDS Fanout x 8 Diff. to 6 SLBs Single-end to 2 xilinx Q 1 Q 2 CLK 40 Q 3 3. 3 V-PECL Q 4 Q 5 Q 6 Q 7 Q 8 MC 100 LVE 310 3. 3 V PECL CLK 80 3. 3 V-PECL ……. . Fanout x 15 e. g. DS 90 LV 110 RJ 45 . . TTC daughter card IN IN_b DS 90 LV 001 80. 0789 MHz 3. 3 V crystal Diff. PECL MC 100 LVEL 37 CK CK CK/2 Brdcst<7: 2>, Brcst. Str, L 1 A, BCnt. Res to xilinx and SLBs To 6 SLBs 8 clks to TLKs Diff. to 2 Xilinx + termin. PCK 953 LVPECLto-LVTTL Fanout (top layer) 8 clks to TLKs + TPs HTR Tullio Grassi 14

TTCrx Mezzanine card • Very simple card: • 2 PMC connectors • TTCrx chip • TTC signal driver on motherboard • Will be sent out for prototype next week • Used by HTR, DCC, and Fanout cards CMS ESR December 2002 15

TTC Distribution – Fanout Card • Currently HCAL has 6 TTC partitions: • Each partition requires TTCvi and TTCex • Each HCAL VME crate will have a single TTCrx receiving data directly from TTCex in a single VME card (Fanout Card) • Fanout TTC signal to HTR mezzanine card with TTCrx chip • Use quad twisted pair CAT 6/7 cable allows PECL fanout • TTC raw, BC 0, 40 MHz clean, 80 MHz clean fanout • Cost savings and simplification • TTC monitoring by Fanout card over VME • Count resets, etc… CMS ESR December 2002 16

Random Latency Issue • Texas Instruments TLK 2501 Serdes • Run with 80 MHz frame clock – 20 bits/frame, 1. 6 GHz bit clock • 625 ps bit time • Latency from data sheet: • ~20 ns variation (overall latency between 47 and 67 ns) • Fiber to fiber alignment could cross a 40 MHz bucket boundary. • How to fix? • SLB “knows” this latency – we will read it out after each reset • HCAL LED fast rise time • Can pulse during abort gap and align channels • Requires LED pulsing alignment • FE will send BC 0 signal on all fibers • Will measure this alleged latency with new HTR boards CMS ESR December 2002 17

TPG Alignment • TPG alignment performed in SLB • Necessary: All HTRs will send common BC 0 to SLB’s within each of 16 VME crates • Calibration procedure to be performed for crate-crate alignment • Initial alignment with LEDs, laser, etc. • Final alignment with LHC first beam data • Use “ 1 -bucket” running to check everything • This will ensure successful alignment CMS ESR December 2002 18

DAQ Alignment • DAQ data must also be aligned • Must know L 1 A bucket for zero suppression • Only if we will do ZSP on 1 or 2 HCAL channels centered on L 1 A bucket • If ZSP done with sum over 5 channels, then this alignment is not critical • Solution: discussed in previous slide • Read from SLB • FE sending known ID after with fixed offset relative to BC 0 during abort gap • Comparison of the two for error checking • DAQ check on BC 0 in DCC for alignment • Will send BC 0, BCN, and EVN with the data to DAQ CMS ESR December 2002 19

MISC Errors • What happens if DCC finds mismatch in EVN? • DCC will then issue resynch request to a. TTS system • Details not yet defined but is fully programmable • Fiber Link/synchronization errors (GOL/TI) • Work out protocols to inform DCC • Reset requests to a. TTS as well • FE Clock/GOL PLL link errors • If GOL loses synch, then transmitter will send out IDLE characters • IDLE characters are illegal in a pipelined system! • HTR will trap on IDLE as a signal that FE/GOL is having trouble CMS ESR December 2002 20

Schedule CMS ESR December 2002 21