Prototype Process B Carlson 2008 -Dec 2 -3

Prototype Process B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process EVLA Correlator CDR Dec. 2 -3, 2008

Outline • Design • Build stages • Testing • ASIC • Summary B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 2

Design • Formally review (Co. DR, PDR) architecture and design. • Formally develop “RFS” (Reqs + Func. Spec. ) doc. – Not always done due to time constraints; sometimes board functionality is largely defined by FPGA RFS. • Board schematic capture and layout done in-house by engineers. – Tried local external contractor, but too complex, time consuming. – Board and FPGA/ASCI engineer needs to be intimately involved. – Engineers have best idea of high-frequency, tx line, SI reqs. • Detailed peer review of schematics before start layout. • Detailed peer review of schematics and layout before build. – For the Station and Baseline Boards, each review was a week-long process. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 3

Build Stages • 4 stages of build and test. – Increasing quantities with each stage to manage cost-risk. • Not all boards require all stages…some require just one build (e. g. RPMIB). • Some boards abandoned as architecture changed/converged. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 4

Build Stages • “Stage 1” – Alpha proto, a. k. a. “FTB”. – Large boards: 1 functional board, 2 mechanical boards. Too cost-risky to build more than 1. – Small boards: 5 functional boards (most cost is NREs…). • “Stage 2” – Beta proto. Incorporates design changes from Stage 1 testing. – Large boards: 2 functional boards. – Small boards, build 5 if needed. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 5

Build Stages • “Stage 3” – Pre-production units. – Incorporate fixes in design from Beta protos. – No hard and fast rule for quantities: • Minimum quantities to reduce cost-risk. • Quantities to meet deployment, telescope testing goals. • Enough quantity to discover lurking bugs + defects not apparent in few quantities. – For the large boards, settled on quantity 14 each. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 6

Build Stages • “Stage 4” – Full production. Major cost expenditure, ~$6. 5 M USD ($2 M for Station Boards, $4. 5 M for Baseline Boards). – Only after successful pre-production testing, Critical On-The-Sky testing, and Critical Design Review. – 133 Station Boards, 146 Baseline Boards, ~80 Cross-bar Boards. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 7

Testing • Functionality. Does the board function at speed as designed? • Performance and margin. – – – Signal integrity. Power distribution. Timing. Thermal. Physical/mechanical. • Primary engineer develops checklist of tests to perform on the board. – In reality, many more ad-hoc tests are performed, but the checklist is used for final check off. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 8

Testing – Functionality • Each FPGA (and ASIC) has an RFS document which describes its requirements, functionality and I/O in detail. – S/W developed according to RFS’ “register set”. – Incremental build and simulation (RTL + gate level) test chip according to RFS(s). – Test chip and board according to RFS(s). – Built-in comm error testing in each chip. – Sig. Pro functionality requires comparing data products with simplified S/W based model of function, with the same vectors. – Due to long integration times, sometimes only statistical comparison performed. • Real-time and GUI software developed for testing and final use. • In some cases, in-house throw-away software developed to expedite board testing (Station Board). B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 9

Testing – Functionality • When mature enough, on-telescope testing used to verify that the hardware performs as designed. – “Critical On-The-Sky” (COTS) testing. • COTS testing started ~July 5, 2008. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 10

Testing – Signal Integrity • Preliminary “what if” analysis with Mentor’s Signal Vision during design exploration/development. • Post-layout exhaustive SI analysis using Mentor’s ICX. – Pretty good agreement with real board, but cross-talk analysis only 1 st order approximation. Can exhaustively test every net. • Probe every available access point (vias) on at least 2, more if time, instances of every chip (several hundred pins per chip!) – Check for jitter, voltage margin, acceptable over/undershoot, cross-talk. • Jitter spectral analysis invaluable to locate jitter sources. – Mostly judgement call; sometimes (especially jitter) there is an exact specification…need margin…but vias are stubs along the transmission line, and receiver input capacitance and on-chip termination provide incomplete picture of what receiver sees. • Secondary information on longer term tests, over temperature, speed and distance margin, more quantity etc. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 11

Testing – Power Distribution • Check each DC voltage at each device to ensure within spec (~+/-50 m. V max). – Voltage gradients across board due to IR drops. Set power supply sense line at the approx center of the power plane. • Check voltage drop across power supply pins to PCB. – High currents can have large IR drops and I 2 R heating. – Use mini-planes on every layer around high I pins with multiple vias, to deal with high current density. – Check power supply pin temperature. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 12

Testing – Power Distribution • Check noise on each power supply on at least 2 devices of each type. – Decoupling and Vanalog (VPLL VGXB) bead isolation critical! – Run separate AGND and DGND planes. • On the Baseline Board, connect with ferrite bead at one point (but did provide array of 0402 pads to connect together just in case). – Several decades of decoupling caps required (680 u. F, 100 n. F)…but all high freq caps are 100 n. F. • 0402 100 n. F caps have excellent performance. • Using smaller caps is “textbook”, but only space for one cap per power pin, and some pins get starved of current if smaller caps used. • For the ASIC, additionally use “PCB stacked capacitor” for best high freq decoupling, but only ~100 p. F capacitance. – Found that in some cases, VPLL oscillating…due to PLL itself, and larger caps required. – Spectral analysis helpful to determine source of noise. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 13

Testing – Timing • Group delay of signals from chip-to-chip the most important. – Set/guaranteed by careful layout. • Clock phase relative to data largely unimportant, as long as stable. – Use DPA and DDR sampling throughout. • Bought tools for board level timing analysis, but never used! B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 14

Testing – Thermal • Early reliability study indicated that Tj < 50 o. C is an important requirement for reliability. • Full thermal mock-up and test of rack. • Monolithic heatsinks to minimize board thermal gradients. • Thermal imaging of front/back of boards during tests. • Operation at Tamb=50 C, down to 0 (burn-in conditions also). B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 15

Testing – Phys/Mech • No vibration testing performed. – Boards will not live in a harsh vibration environment. – Use thermal shock (0 -80 o. C cold/hot box) to discover marginal solder joints. • Still, use good practices to ensure no problems: – Glue down “floppy” electrolytic caps. – Use lockwashers (Bellville) or locktight on every fastener. – Tie-wraps to secure cables. • Brick power supplies use press-fit sockets due to difficulties in removing soldered-in connections. – Could be a problem in a high vibration/mechanical shock environment. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 16

ASIC • Subject to separate formal reviews. Just summarize development process here. • Incremental RTL design and simulation (mid-2002). • ~2 years of detailed RTL testing by independent tester (Smegal). – Bit-exact comparisons against simple S/W correlator model. • Long painful process of vendor selection. Selected i. Sine Inc. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 17

ASIC • RTL sign-off CDR (Jan 2005) • RTL vs gate-level simulation comparison…bit exact results. • Signed off for proto fabrication ~Dec 2005 (report A 25082 N 0004) • We developed standalone > at speed tester. – 30 test cases…at speed. – Production screen coverage for that missed by scan-test due to ripple accumulators. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 18

ASIC • Prototypes delivered ~June 2006. – Delays in testing due to delays in Baseline Board first proto build, and problems with socketing on motherboard. – Could test with standalone tester, but wanted board and S/W to test exhaustively. – In the end, the proto worked first time…no re-spin required. • Production sign-off CDR June 2007. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 19

ASIC • Contracted Mu. Analysis Ottawa to run JEDEC qual tests, and to develop and execute production screen. – Use our standalone tester for pass/fail qualification. – Passed all JEDEC tests. Cost ~100 chips. • Warning that chip package construction is susceptible to fine conductive particle accumulation. For long-term reliability, want clean room. – Production screen (168 hrs at 125 C bake, for bond wire failure) • Out of 10, 800 devices, there were 83 failures. 44 were likely due to incomplete scan-test coverage, 7 “broke in the tester”, and the rest ~32 likely due to infant mortality. • It is possible that the screen saved us ~32 BGA rework cycles. • All devices are now at Brecon. Ridge, waiting for full production. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 20

Summary • Prototyping and test process aimed at gradually increasing costs as risk decreases. – Gradually increase the number of boards built with increasing confidence. • Try to exhaustively test, but still susceptible to human error, selection bias etc. – – Ad hoc communication within team. Peer reviews to go over designs and test results. Formal reviews. But primary responsibility is still with primary engineers as they know the design, functionality, and performance requirements most intimately. • Learning process with large boards, high speed, new technology. B. Carlson, 2008 -Dec 2 -3 EVLA Correlator CDR - Prototype Process 21