Multi-Gigabit Trials on GEANT Collaboration with Dante. Richard Hughes-Jones The University of Manchester www. hep. man. ac. uk/~rich/ then “Talks” ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 1
Outline u. What is GÉANT 2 u What is it interesting? n 10 Gigabit Ethernet n UDP memory-2 -memory flows u. Options using GÉANT Development Network n 10 Gbit SDH Network u. Options Using the GÉANT Light. Path Service n Po. P Location for Network tests u. PCs and Current 10 Gbit Tests n PC Servers n Some Test results ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 2
GÉANT 2 Topology ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 3
GÉANT 2: The Convergence Solution EXPRe. S PC 10 GE NREN Access 1678 MCC Existing IP Router L 2 Matrix TDM Matrix Dark Fibe r GÉANT 2 POP A GÉANT 2 POP B 1678 MCC 1626 LM L 2 TDM 1626 LM Managed Lambda’s EXPRe. S PC 10 GE ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester Existing IP Router 4
From Po. S to Ethernet Connect. Communicate. Collaborate • More Economical Architecture • Highest Overall Network Availability • Flexibility (VLAN management) • Highest Network Performance (Latency) IP Links VLANs Router 1678 MCC Transport Node VC-4 -nv Channels 1/10 Gigabit Ethernet L 2 Matrix TDM Matrix ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 5
What do we want to do? u Set up 4 Gigabit Lightpath Between GÉANT Po. Ps n Collaboration with Dante n PCs in their Po. Ps with 10 Gigabit NICs u VLBI Tests: n UDP Performance l Throughput, jitter, packet loss, 1 -way delay, stability n Continuous (days) Data Flows – VLBI_UDP and n multi-Gigabit TCP performance with current kernels n Experience for FPGA Ethernet packet systems u Dante Interests: n multi-Gigabit TCP performance n The effect of (Alcatel) buffer size on bursty TCP when using BW limited Lightpaths u Need A Collaboration Agreement ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 6
Options Using the GÉANT Development Network u 10 Gigabit SDH backbone u Alkatel 1678 MCC u Node location: n London n Amsterdam n Paris n Prague n Frankfurt u Can do traffic routing so make long rtt paths u Available Dec/Jan 07 u Less Pressure for long term tests ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 7
Options Using the GÉANT Light. Paths u Set up 4 Gigabit Lightpath Between GÉANT Po. Ps n Collaboration with Dante n PCs in Dante Po. Ps u 10 Gigabit SDH backbone u Alkatel 1678 MCC u Node location: n Budapest n Geneva n Frankfurt n Milan n Paris n Poznan n Prague n Vienna u Can do traffic routing so make long rtt paths u Ideal: London Copenhagen ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 8
4 Gigabit GÉANT Light. Path u Example of a 4 Gigabit Lightpath Between GÉANT Po. Ps n PCs in Dante Po. Ps n 26 * VC-4 s 4180 Mbit/s ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 9
PCs and Current Tests ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 10
Test PCs Have Arrived u Boston/Supermicro X 7 DBE u Two Dual Core Intel Xeon Woodcrest 5130 n 2 GHz n Independent 1. 33 GHz FSBuses u 530 MHz FD Memory (serial) u Chipsets: Intel 5000 P MCH – PCIe & Memory ESB 2 – PCI-X GE etc. u PCI n 3 8 lane PCIe buses n 3* 133 MHz PCI-X u 2 Gigabit Ethernet u SATA ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 11
Bandwidth Challenge wins Hat Trick u The maximum aggregate bandwidth was >151 Gbits/s n 130 DVD movies in a minute n serve 10, 000 MPEG 2 HDTV movies in real-time u 22 10 Gigabit Ethernet waves Caltech & SLAC/FERMI booths n In 2 hours transferred 95. 37 TByte n 24 hours moved ~ 475 TBytes SC 2004 101 Gbit/s u Showed real-time particle event analysis u SLAC Fermi UK Booth: n 1 10 Gbit Ethernet to UK NLR&UKLight: l transatlantic HEP disk to disk l VLBI streaming n 2 10 Gbit Links to SALC: l rootd low-latency file access application for clusters l Fibre Channel Stor. Cloud n 4 10 Gbit links to Fermi l Dcache data transfers ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester In to booth Out of booth 12
SC|05 Seattle-SLAC 10 Gigabit Ethernet u 2 Lightpaths: n Routed over ESnet n Layer 2 over Ultra Science Net u 6 Sun V 20 Z systems per λ u dcache remote disk data access n 100 processes per node n Node sends or receives n One data stream 20 -30 Mbit/s u Used Neteion NICs & Chelsio TOE u Data also sent to Stor. Cloud using fibre channel links u Traffic on the 10 GE link for 2 nodes: 3 -4 Gbit per nodes 8. 5 -9 Gbit on Trunk ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 13
Lab Tests 10 Gigabit Ethernet u 10 Gigabit Test Lab being set up in Manchester n Cisco 7600 n Cross Campus λ <1 ms n Server quality PCs n Neterion NICs n Chelsio being purchased u B 2 B performance so far n Super. Micro X 6 DHE-G 2 n Kernel (2. 6. 13) & Driver dependent! n One iperf TCP data stream 4 Gbit/s n Two bi-directional iperf TCP data streams 3. 8 & 2. 2 Gbit/s n UDP Disappointing u Propose to install Fedora Core 5 Kernel 2. 6. 17 on the new Intel dual-core PCs ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 14
10 Gigabit Ethernet: TCP Data transfer on PCI-X u Sun V 20 z 1. 8 GHz to 2. 6 GHz Dual Opterons u Connect via 6509 u XFrame II NIC u PCI-X mmrbc 4096 bytes 66 MHz Data Transfer u Two 9000 byte packets b 2 b u Ave Rate 2. 87 Gbit/s CSR Access u Burst of packets length 646. 8 us u Gap between bursts 343 us u 2 Interrupts / burst ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 15
10 Gigabit Ethernet: UDP Data transfer on PCI-X u Sun V 20 z 1. 8 GHz to 2. 6 GHz Dual Opterons u Connect via 6509 u XFrame II NIC u PCI-X mmrbc 2048 bytes 66 MHz u One 8000 byte packets n 2. 8 us for CSRs n 24. 2 us data transfer effective rate 2. 6 Gbit/s Data Transfer CSR Access 2. 8 us u 2000 byte packet, wait 0 us n ~200 ms pauses u 8000 byte packet, wait 0 us n ~15 ms between data blocks ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 16
Any Questions? ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 17
More Information Some URLs 1 u u u u u UKLight web site: http: //www. uklight. ac. uk MB-NG project web site: http: //www. mb-ng. net/ Data. TAG project web site: http: //www. datatag. org/ UDPmon / TCPmon kit + writeup: http: //www. hep. man. ac. uk/~rich/net Motherboard and NIC Tests: http: //www. hep. man. ac. uk/~rich/net/nic/Gig. Eth_tests_Boston. ppt & http: //datatag. web. cern. ch/datatag/pfldnet 2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http: // www. hep. man. ac. uk/~rich/ TCP tuning information may be found at: http: //www. ncne. nlanr. net/documentation/faq/performance. html & http: //www. psc. edu/networking/perf_tune. html TCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 PFLDnet http: //www. ens-lyon. fr/LIP/RESO/pfldnet 2005/ Dante PERT http: //www. geant 2. net/server/show/nav. 00 d 00 h 002 ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 18
More Information Some URLs 2 u Lectures, tutorials etc. on TCP/IP: n n n www. nv. cc. va. us/home/joney/tcp_ip. htm www. cs. pdx. edu/~jrb/tcpip. lectures. html www. raleigh. ibm. com/cgi-bin/bookmgr/BOOKS/EZ 306200/CCONTENTS www. cisco. com/univercd/cc/td/doc/product/iaabu/centri 4/user/scf 4 ap 1. htm www. cis. ohio-state. edu/htbin/rfc 1180. html www. jbmelectronics. com/tcp. htm u Encylopaedia n http: //www. freesoft. org/CIE/index. htm u TCP/IP Resources n www. private. org. il/tcpip_rl. html u Understanding IP addresses n http: //www. 3 com. com/solutions/en_US/ncs/501302. html u Configuring TCP (RFC 1122) n ftp: //nic. merit. edu/internet/documents/rfc 1122. txt u Assigned protocols, ports etc (RFC 1010) n http: //www. es. net/pub/rfcs/rfc 1010. txt & /etc/protocols ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 19
Backup Slides ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 20
Bandwidth on Demand Our Long-Term Vision Bandwidth Request Policy Middleware Network Resource Mgr Bandwidth Request Applications e. g. GRID Research Activity UNI-C Command 1678 MCC Ethernet 1678 MCC GMPLS Ethernet 1678 MCC ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester Applications e. g. GRID 21
Super. Computing ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 22
10 Gigabit Ethernet: UDP Throughput u u u 1500 byte MTU gives ~ 2 Gbit/s Used 16144 byte MTU max user length 16080 Data. TAG Supermicro PCs Dual 2. 2 GHz Xenon CPU FSB 400 MHz PCI-X mmrbc 512 bytes wire rate throughput of 2. 9 Gbit/s u u CERN Open. Lab HP Itanium PCs Dual 1. 0 GHz 64 bit Itanium CPU FSB 400 MHz PCI-X mmrbc 4096 bytes wire rate of 5. 7 Gbit/s u u SLAC Dell PCs giving a Dual 3. 0 GHz Xenon CPU FSB 533 MHz PCI-X mmrbc 4096 bytes wire rate of 5. 4 Gbit/s ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 23
10 Gigabit Ethernet: Tuning PCI-X u 16080 byte packets every 200 µs u Intel PRO/10 Gb. E LR Adapter u PCI-X bus occupancy vs mmrbc n n mmrbc 512 bytes Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5. 7 Gbit/s mmrbc 1024 bytes CSR Access mmrbc 2048 bytes PCI-X Sequence Data Transfer Interrupt & CSR Update mmrbc 4096 bytes 5. 7 Gbit/s ESLEA-FABRIC Technical Meeting , 1 Sep 2006, R. Hughes-Jones Manchester 24
Скачать презентацию Multi-Gigabit Trials on GEANT Collaboration with Dante Richard
a951dbda2b55fd82331a2fc6a991cc12.ppt