White Rabbit Sub-Nanosecond timing over Ethernet Oh dear

White Rabbit Sub-Nanosecond timing over Ethernet “Oh dear! I shall be too late!” 1) 1) H. Z. Peek Nikhef Amsterdam Electronics. Technology Alice in Wonderland, Lewis Carroll (1865) H. Z. Peek. . . on behalf of the White Rabbit collaboration KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 1

Outline What is White Rabbit Network Building blocks Measurement results Applications Conclusion H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 2

What is White Rabbit Networking and timing Based on well-known technologies / standards Open Hardware and Open Software International collaboration Main features: Transparent, high-accuracy time distribution, Low-latency, deterministic data delivery, Designed for high reliability, Plug & play. H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 3

Timing over Ethernet History 1985 NTP (RFC 1305) 2002 PTP (IEEE 1588) 2012 White Rabbit (? ) sub milli-second sub micro-second sub nano-second H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 4

White Rabbit timing distribution Synchronization with sub-ns accuracy over fiber A combination of: Synchronous Ethernet (Sync. E) syntonization Digital Dual-Mixer Time Difference (DDMTD) phase detection (j) Precision Timing Protocol (PTP) synchronization H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 5

Timing distribution Reference Clock Tx Master Rx j Ordinary serial data communication High Precision Low jitter channels! Tx channels j Averaging • H. Z. Peek Nikhef Amsterdam Electronics. Technology SFP Slave Rx SFP Tx Slave SFP Rx High Precision Low jitter Rx SFP Tx all clock edges “mean phase” Syntonization • “The adjustment of two electronic circuits or devices in terms of frequency” KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 6

PTP (IEEE 1588) t 1 Tx SFP t 4 Ref Clk Master Cnt Slave t 1 t 2 Rx t 4 SFP Rx Tx j t 3 j =0 t 1 t 4 Cnt t 2 t 3 toffset t 1 Time Stamp Data (Packets) Clock Follow_Up Message (t 1) t 4 H. Z. Peek Nikhef Amsterdam Electronics. Technology (= Cnt + Master. Bit. Slide + j) 1. t 2 –t 1=offset + MSdelay 2. t 4 –t 3=-offset + SMdelay 3. MSdelay=SMdelay Sync message Delay_Req Message Delay_Response Message (t 4) Master Clock Time Slave Clock Time t 2 (= Cnt + Slave. Bit. Slide) t 1 t 2 t 3 t 1 t 2 t 4 t 3 t 1 t 2 KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 7

White Rabbit network H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 8

White Rabbit Switch V 2 Central element of White Rabbit network Fully custom design, designed from scratch 10 SFP ports (1000 Base-LX) Capable of driving long distance Single Mode fiber H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 9

White Rabbit Node Simple PCIe FMC Carrier (SPEC) Currently available H. Z. Peek Nikhef Amsterdam Electronics. Technology http: //www. ohwr. org/projects/spec/wiki KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 10

White Rabbit Switch V 3 2 uplink ports, 16 downlink ports Hardware just assembled. Hardware & Software currently being tested. Production expected Q 1 -2012 H. Z. Peek Nikhef Amsterdam Electronics. Technology http: //www. ohwr. org/attachments/741/White_Rabbit_Technical_Spec_05. pdf KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 11

Measurement test setup H. Z. Peek Nikhef Amsterdam Electronics. Technology Hot-air gun demo: http: //www. youtube. com/watch? v=ZSRQEExbdq 8 KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 12

Measurement results H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 13

Applications Digital to Time Converter (DTC) = Alarm Clock Time to Digital Converter (TDC) = Time stamping H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 14

Applications-II H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 15

Conclusions White Rabbit enables measurement and control applications which are using distributed system technologies. Such applications may be spread over large distances. Data transmission delay changes are continuously measured and compensated. System timing of White Rabbit nodes are synchronized with high precision. H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 16

Conclusions-II White Rabbit = Networking and timing Is ideally suited for Prototype hard- soft-ware becomes available this year. Open hard- soft-ware sources available which enables tendering. Standardization effort (IEEE and ITU) is foreseen. H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 17

White Rabbit a multi-laboratory, multi-company effort H. Z. Peek Nikhef Amsterdam Electronics. Technology Thank you Coming Soon! KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 18

H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 19

How do we know the time offset between Master and Slave? Toffset = Total delay(Master>Slave>Master) / 2 Can we measure propagation delay using existing serial communication channels? Serial Communication Coding Properties: Clock & Data coded into one stream 2 DC-Balance 3 Special code-groups / Word Alignment 1 H. Z. Peek Nikhef Amsterdam Electronics. Technology KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 20

Measure propagation delay using FPGA Ser. Des technology 1 3 Regenerate system clock at the receiver Using a barrel shifter for word alignment delay known with bit clock resolution Transmitter Lattice x 20 System Clock Receiver Xilinx LFSCM 25 Virtex-5 Ser. Des SFP H. Z. Peek Nikhef Amsterdam Electronics. Technology System Clock Lattice SC PCI Expressx 1 Evaluation Board Start Xilinx ML 507 Board SFP 100 Km Fiber Ser. Des LEDs Stop KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 21

Time offset measurement test setup 10 Km fiber Stop Loopback the recovered clock with a Voltage Controled Xtal Oscillator Clock Loopback (DPLL) DAC Start Master H. Z. Peek Nikhef Amsterdam Electronics. Technology Slave VCXO KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 22

Resynchronization + Barrel shifting action 0111010110000010101101110101 3 1 0 Rx. Rec. Clk Bit. Slide(4: 0) 0 H. Z. Peek Nikhef Amsterdam Electronics. Technology Unit Interval (UI) 0011 0000 0001 = 1 3 0 Start/Stop delay Algorithm: Propagation Delay = “Start-Stop” Delay + “LED Value” * UI Details : VLVn. T 09, October 15, 2009 in Athens • Presentation: http: //www. nestor. noa. gr/vlvnt 09/pres/Jansweijer_Meauring. Propagation. Delay. ppt • Paper: http: //dx. doi. org/10. 1016/j. nima. 2010. 04. 126 KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 23

Measure time offset Bidirectional + Loopback the recovered clock 1. 25 Gbps (IEEE 802. 3 1000 BASE-X = Gigabit Ethernet) Time offset is determined by: Stop Tx. Usr. Clk Start Reference Clk Count H. Z. Peek Nikhef Amsterdam Electronics. Technology 1. 2. 3. Rx. Usr. Clk j Tx Master Slave SFP Rx. Usr. Clk SFP Rx Fine time Coarse time Rx Tx toffset # of system clocks # bit clocks (i. e. barrel shifts) Phase between Master node Tx and Rx clock Tx. Usr. Clk 1. 25 Gpbs 20 x #bit clocks 800 ps 10. . 100 ps For details please see Technical Report “ETR 2010 -01”: http: //www. nikhef. nl/pub/services/biblio/technicalreports/ETR 2010 -01. pdf KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 24

Time offset and fibre dispersion Master time Slave time toffset l 1 tpdl 1 dm. Tx ds. Rx l 2 dm. Rx H. Z. Peek Nikhef Amsterdam Electronics. Technology tdisp tpdl 1 ds. Tx (tdisp = 15150 ps over 10. 7 km => D(l) = l 416 ps/km) KM 3 Ne. T General Assembly Meeting Catania, 20 -23, February 2012 25