High Performance Routers Slides originally by Nick Mc. Keown Professor of Electrical Engineering and Computer Science, Stanford University nickm@stanford. edu www. stanford. edu/~nickm Stanford High Performance Networking group: http: //klamath. stanford. edu Nick Mc. Keown IEE, October 2001 1
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future Nick Mc. Keown IEE, October 2001 2
What is Routing Forwarding? R 3 R 1 A R 4 B D C E D R 2 Destination Next Hop D R 3 E R 5 R 3 F Nick Mc. Keown D R 5 IEE, October 2001 F 3
What is Routing? R 3 R 1 A 4 16 D 1 D Ver HLen T. Service 20 bytes B Fragment ID TTL C Flags Fragment Offset Header Checksum Source Address R 2 Destination Address Hop Destination Next D R 3 Options (if any) F Nick Mc. Keown D D D 32 Total Packet Length Protocol E R 4 Data R 5 E F R 3 R 5 IEE, October 2001 4
What is Routing? R 3 A R 1 R 4 B C Nick Mc. Keown D E R 2 R 5 IEE, October 2001 F 5
Points of Presence (POPs) POP 2 A POP 1 POP 4 B C Nick Mc. Keown POP 3 E POP 5 POP 6 POP 7 IEE, October 2001 D POP 8 F 6
Where High Performance Routers are Used (2. 5 Gb/s) R 1 R 2 R 5 R 4 R 3 R 8 R 9 R 10 Nick Mc. Keown R 7 R 11 R 14 R 13 (2. 5 Gb/s) R 6 R 15 (2. 5 Gb/s) R 12 R 16 (2. 5 Gb/s) IEE, October 2001 7
What a Router Looks Like Cisco GSR 12416 Juniper M 160 19” Capacity: 160 Gb/s Power: 4. 2 k. W 6 ft 3 ft 2 ft Nick Mc. Keown Capacity: 80 Gb/s Power: 2. 6 k. W 2. 5 ft IEE, October 2001 8
Generic Router Architecture Header Processing Data Hdr Lookup Update IP Address Header IP Address 1 M prefixes Off-chip DRAM Nick Mc. Keown Queue Packet Data Hdr Next Hop Address Table Buffer Memory IEE, October 2001 1 M packets Off-chip DRAM 9
Generic Router Architecture Data Hdr Header Processing Lookup IP Address Buffer Manager Update Header Buffer Memory Address Table Data Hdr Header Processing Lookup IP Address Buffer Manager Update Header Data Hdr Memory Header Processing Lookup IP Address Buffer Manager Update Header Buffer Memory Address Table Nick Mc. Keown Data Hdr Buffer Address Table Data Hdr IEE, October 2001 10
Why do we Need Faster Routers? 1. To prevent routers becoming the bottleneck in the Internet. 2. To increase POP capacity, and to reduce cost, size and power. Nick Mc. Keown IEE, October 2001 11
Why we Need Faster Routers 1: To prevent routers from being the bottleneck Packet processing Power Link Speed 10000 2 x / 7 months 100 10 1 1985 1990 1995 Fiber Capacity (Gbit/s) 2 x / 18 months 1000 2000 0, 1 TDM DWDM Source: SPEC 95 Int & David Miller, Stanford. Nick Mc. Keown IEE, October 2001 12
Why we Need Faster Routers 2: To reduce cost, power & complexity of POPs POP with smaller routers POP with large routers v v Nick Mc. Keown Ports: Price >$100 k, Power > 400 W. It is common for 50 -60% of ports to be for interconnection. IEE, October 2001 13
Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law: v The bottleneck is memory speed. v Memory speed is not keeping up with Moore’s Law. Nick Mc. Keown IEE, October 2001 14
Why are Fast Routers Difficult to Make? Speed of Commercial DRAM 1. It’s hard to keep up with Moore’s Law: v The bottleneck is memory speed. 1. 1 x is months v Memory speed / 18 not keeping up with Moore’s Law 2 x / 18 months Nick Mc. Keown IEE, October 2001 15
Why are Fast Routers Difficult to Make? 1. It’s hard to keep up with Moore’s Law: v The bottleneck is memory speed. v Memory speed is not keeping up with Moore’s Law. 2. Moore’s Law is too slow: v Routers need to improve faster than Moore’s Law. Nick Mc. Keown IEE, October 2001 16
Router Performance Exceeds Moore’s Law Growth in capacity of commercial routers: v v v Capacity 1992 ~ 2 Gb/s Capacity 1995 ~ 10 Gb/s Capacity 1998 ~ 40 Gb/s Capacity 2001 ~ 160 Gb/s Capacity 2003 ~ 640 Gb/s Average growth rate: 2. 2 x / 18 months. Nick Mc. Keown IEE, October 2001 17
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future Nick Mc. Keown IEE, October 2001 18
First Generation Routers Shared Backplane Li CP In ne U te rf ac M e em or y CPU Route Table Buffer Memory Line Interface MAC MAC Typically <0. 5 Gb/s aggregate capacity Nick Mc. Keown IEE, October 2001 19
Second Generation Routers CPU Route Table Buffer Memory Line Card Buffer Memory Fwding Cache MAC MAC Typically <5 Gb/s aggregate capacity Nick Mc. Keown IEE, October 2001 20
Third Generation Routers Switched Backplane Line Card Li In ne CP Ute rf ac M e em or y CPU Card Line Card Local Buffer Memory Routing Table Local Buffer Memory Fwding Table MAC Typically <50 Gb/s aggregate capacity Nick Mc. Keown IEE, October 2001 21
Fourth Generation Routers/Switches Optics inside a router for the first time Optical links 100 s of metres Switch Core Nick Mc. Keown Linecards 0. 3 - 10 Tb/s routers in development IEE, October 2001 22
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future Nick Mc. Keown IEE, October 2001 23
Generic Router Architecture Header Processing Lookup IP Address Buffer Manager Update Header Address Table Buffer Memory Header Processing Lookup IP Address Buffer Manager Update Header Buffer Memory Address Table Nick Mc. Keown IEE, October 2001 24
IP Address Lookup Why it’s thought to be hard: 1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 120, 000 entries today, and growing. 3. The lookup must be fast: about 30 ns for a 10 Gb/s line. Nick Mc. Keown IEE, October 2001 25
IP Lookups find Longest Prefixes 128. 9. 176. 0/24 128. 9. 16. 0/21 128. 9. 172. 0/21 65. 0. 0. 0/8 0 128. 9. 0. 0/16 128. 9. 16. 14 142. 12. 0. 0/19 232 -1 Routing lookup: Find the longest matching prefix (aka the most specific route) among all prefixes that match the destination address. Nick Mc. Keown IEE, October 2001 26
IP Address Lookup Why it’s thought to be hard: 1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 120, 000 entries today, and growing. 3. The lookup must be fast: about 30 ns for a 10 Gb/s line. Nick Mc. Keown IEE, October 2001 27
Address Tables are Large Source: Geoff Huston, Oct 2001 Nick Mc. Keown IEE, October 2001 28
IP Address Lookup Why it’s thought to be hard: 1. It’s not an exact match: it’s a longest prefix match. 2. The table is large: about 120, 000 entries today, and growing. 3. The lookup must be fast: about 30 ns for a 10 Gb/s line. Nick Mc. Keown IEE, October 2001 29
Lookups Must be Fast Year 40 B packets (Mpkt/s) 1997 1999 2001 2003 Nick Mc. Keown Line 622 Mb/s 2. 5 Gb/s 10 Gb/s 40 Gb/s 1. 94 7. 81 31. 25 125 IEE, October 2001 30
IP Address Lookup Binary tries 0 1 f d e g h i abc Example a) b) c) d) e) f) g) h) i) j) Prefixes: 000010 00011 001 011 100 1010 1100 11110000 j Nick Mc. Keown IEE, October 2001 31
Prefix Length Distribution Source: Geoff Huston, Oct 2001 99. 5% prefixes are 24 -bits or shorter Nick Mc. Keown IEE, October 2001 32
24 -8 Direct Lookup Trie 0000…… 0000 24 bits 1111…… 1111 0 224 -1 8 bits 0 28 -1 v When pipelined, allows one lookup per memory access. v Although inefficient use of memory, total memory cost < $20. Nick Mc. Keown IEE, October 2001 33
IP Address Lookup Summary v Lookup limited by memory bandwidth. v Lookup uses high-degree trie. v State of the art: 10 Gb/s line rate. v Scales to: 40 Gb/s line rate. v By 2008, entire IPv 4 address space will fit on one $20 DRAM! Nick Mc. Keown IEE, October 2001 34
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future Nick Mc. Keown IEE, October 2001 35
Generic Router Architecture Header Processing Lookup IP Address Buffer Queue Manager Packet Update Header Buffer Memory Address Table Header Processing Lookup IP Address Queue Buffer Packet Manager Update Header Buffer Memory Address Table Nick Mc. Keown IEE, October 2001 36
Fast Packet Buffers Example: 40 Gb/s packet buffer Size = RTT*BW = 10 Gb; 40 byte packets Write Rate, R 1 packet every 8 ns Buffer Manager Read Rate, R 1 packet every 8 ns Buffer Memory Use SRAM? Use DRAM? + fast enough random access time, but - too low density to store 10 Gb of data. + high density means we can store data, but - too slow (50 ns random access time). Nick Mc. Keown IEE, October 2001 37
Packet Caches Small ingress SRAM cache of FIFO heads Small ingress SRAM cache of FIFO tails Arriving Packets 60 59 58 57 56 55 1 97 96 2 Buffer Manager Q SRAM 91 90 89 88 87 4 3 2 1 5 4 3 2 1 6 5 4 3 2 1 2 Departing Packets Q b>>1 packets at a time DRAM Buffer Memory 54 53 52 51 50 10 9 8 7 6 5 8 7 6 11 10 9 8 7 DRAM Buffer Memory 95 94 93 92 91 90 89 88 87 86 1 Nick Mc. Keown 15 14 13 12 11 10 9 86 85 84 83 82 IEE, October 2001 1 2 Q 38
Packet Buffers Summary Packet buffers limited by memory bandwidth. v Packet buffer caches use hybrid SRAM+DRAM. v State of the art: 10 Gb/s line rate. v Scales to: 40 Gb/s line rate. v Nick Mc. Keown IEE, October 2001 39
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future Nick Mc. Keown IEE, October 2001 40
Generic Router Architecture Data Hdr Header Processing Lookup IP Address Update Header 1 1 Buffer Memory Address Table Data Hdr Header Processing Lookup IP Address Queue Packet Update Header 2 2 NQueue line rate times Packet Buffer Memory Address Table N times line rate Data Hdr Header Processing Lookup IP Address Update Header N N Buffer Memory Address Table Nick Mc. Keown Queue Packet IEE, October 2001 41
Generic Router Architecture Data Hdr Header Processing Lookup IP Address Update Header Address Table Data Hdr Update Header Address Table Nick Mc. Keown Queue Packet 2 2 Data Hdr Buffer Memory Header Processing Lookup IP Address 1 Buffer Address Table Data Hdr 1 Data Hdr Memory Header Processing Lookup IP Address Queue Packet Update Header Queue Packet Scheduler N N Buffer Data Hdr Memory IEE, October 2001 42
Output Buffered Switches The best that any queueing system can achieve. Nick Mc. Keown IEE, October 2001 43
Input Buffered Switches Head of Line Blocking The best that any queueing system can achieve. Nick Mc. Keown IEE, October 2001 44
Head of Line Blocking Nick Mc. Keown IEE, October 2001 45
Virtual Output Queues Nick Mc. Keown IEE, October 2001 46
A Router with Virtual Output Queues The best that any queueing system can achieve. Nick Mc. Keown IEE, October 2001 47
Maximum Weight Matching A 1(n) A 11(n) S*(n) L 11(n) 1 1 D 1(n) A 1 N(n) AN(n) DN(n) AN 1(n) N ANN(n) N LNN(n) L 11(n) Maximum Weight Match LN 1(n) “Request” Graph Nick Mc. Keown IEE, October 2001 Bipartite Match 48
The Evolution of Switching Different weight functions, incomplete information, pipelining. Theory: Input Queueing (IQ) IQ + VOQ, Maximum weight matching 58% [Karol, 1987] 100% [M et al. , 1995] 100% [Various] Randomized algorithms 100% [Tassiulas, 1998] IQ + VOQ, Maximal size matching, Speedup of two. 100% [Dai & Prabhakar, 2000] Practice: Input Queueing (IQ) Nick Mc. Keown IQ + VOQ, Sub-maximal size matching e. g. PIM, i. SLIP. IEE, October 2001 Various heuristics, distributed algorithms, and amounts of speedup 49
Switching Summary v Routers use virtual output queues, and a centralized scheduler. State of the art: 2. 5 Tb/s. v Scales to: ~10 Tb/s. v v 100% throughput will be standard soon. Nick Mc. Keown IEE, October 2001 50
Current Internet Router Technology Summary v There are three potential bottlenecks: Address lookup, v Packet buffering, and v Switching. v v Techniques exist today for: 10+Tb/s Internet routers, with v 40 Gb/s linecards. v v But what comes next…? Nick Mc. Keown IEE, October 2001 51
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future v v Nick Mc. Keown More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching? IEE, October 2001 52
External Parallelism: Multiple Parallel Routers What we’d like: IP Router capacity 100 s of Tb/s R R Nx. N R R The building blocks we’d like to use: R R R Nick Mc. Keown R IEE, October 2001 53
Multiple parallel routers Load Balancing R R R R/k 1 2 … R/k … R R R k Nick Mc. Keown IEE, October 2001 54
Intelligent Packet Load-balancing Parallel Packet Switching Router R/k rate, R 1 R/k 1 1 rate, R 2 rate, R N N rate, R k Nick Mc. Keown Bufferless IEE, October 2001 55
Parallel Packet Switching Advantages v Single-stage of buffering v No excess link capacity v kh a power per subsystem i v kh a memory bandwidth i v kh a lookup rate i Nick Mc. Keown IEE, October 2001 56
Parallel Packet Switch Theorem If S > 2 k/(k+2) @ 2 then a parallel packet switch can precisely emulate a single big router. Nick Mc. Keown IEE, October 2001 57
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future v v Nick Mc. Keown More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching? IEE, October 2001 58
![Eliminating schedulers Two-Stage Switch [Chang et al. , 2001] External Inputs Internal Inputs External](https://present5.com/presentation/e950308036f2bb65b4805b5b4d91ff66/image-59.jpg "Eliminating schedulers Two-Stage Switch [Chang et al. , 2001] External Inputs Internal Inputs External")
Eliminating schedulers Two-Stage Switch [Chang et al. , 2001] External Inputs Internal Inputs External Outputs 1 1 N N 1 N Load Balancing First Round-Robin Nick Mc. Keown IEE, October 2001 Second Round-Robin 59
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future v v Nick Mc. Keown More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching? IEE, October 2001 60
Do optics belong in routers? They are already there. v Connecting linecards to switches. Optical processing doesn’t belong on the linecard. v You can’t buffer light. v Minimal processing capability. Optical switching can reduce power. Nick Mc. Keown IEE, October 2001 61
Optics in routers Optical links Switch Core Nick Mc. Keown Linecards IEE, October 2001 62
Complex linecards Typical IP Router Linecard Lookup Tables Optics Physical Layer Framing & Maintenance Buffer & State Memory Packet Processing Buffer Mgmt & Scheduling Buffer & State Memory Switch Fabric Arbitration 10 Gb/s linecard: Number of gates: 30 M v Amount of memory: 2 Gbits v Cost: >$20 k v Power: 300 W v Nick Mc. Keown IEE, October 2001 63
Replacing the switch fabric with optics Typical IP Router Linecard Lookup Tables Packet Processing Optics Physical Layer Typical IP Router Linecard Buffer & State Memory Switch Fabric Framing & Maintenance Buffer Mgmt & Scheduling Buffer & State Memory 1. 2. 3. 4. MEMs. Packet Processing Optics Physical Layer Buffer Mgmt & Scheduling Typical IP Router Linecard optical Buffer & State Memory Req/Grant Arbitration Lookup Tables Buffer Mgmt & Scheduling Switch Fabric Framing & Maintenance Nick Mc. Keown Physical Layer Fast tunable lasers + passive optical couplers. Diffraction waveguides. Electroholographic materials. Buffer Mgmt & Scheduling Buffer & State Memory Optics Framing & Maintenance Buffer & State Memory Buffer Mgmt & Scheduling Packet Processing Arbitration Candidate technologies Typical IP Router Linecard Lookup Tables Buffer Mgmt & Scheduling electrical Buffer Mgmt & Scheduling Lookup Tables Packet Processing Optics Framing & Maintenance Req/Grant IEE, October 2001 Physical Layer Buffer Mgmt & Scheduling Buffer & State Memory 64
The Stanford Phicticious Optical Router Li 160 Gb/s r eca n 1 d# n ng tio ina cessi m ter t pro ing ne e r • Li pack uffe b • IP cket • Pa 160320 Gb/s Optical 2 -stage Switch Line c 160320 Gb/s ard #62 5 • Li n • IP e term • Pa packe inatio n t cke t bu proce s ffe ring sing 40 Gb/s 100 Tb/s IP Router, 625 linecards, each operating at 160 Gb/s. Nick Mc. Keown IEE, October 2001 65
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future v v Nick Mc. Keown More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching? IEE, October 2001 66
Evolution to circuit switching • Optics enables simple, low-power, very high capacity circuit switches. • The Internet was packet switched for two reasons: v Expensive links: statistical multiplexing. v Resilience: soft-state routing. • Neither reason holds today. Nick Mc. Keown IEE, October 2001 67
Fast Links, Slow Routers Processing Power 2 x / 2 years Link Speed (Fiber) 2 x / 7 months Source: SPEC 95 Int; Prof. Miller, Stanford Univ. Nick Mc. Keown IEE, October 2001 68
Fewer Instructions per packet since 1996 Nick Mc. Keown IEE, October 2001 69
Outline Background v v v What is a router? Why do we need faster routers? Why are they hard to build? Architectures and techniques v v The evolution of router architecture. IP address lookup. Packet buffering. Switching. The Future v v Nick Mc. Keown More parallelism. Eliminating schedulers. Introducing optics into routers. Natural evolution to circuit switching? IEE, October 2001 70
References General 1. 2. 3. J. S. Turner “Design of a Broadcast packet switching network”, IEEE Trans Comm, June 1988, pp. 734 -743. C. Partridge et al. “A Fifty Gigabit per second IP Router”, IEEE Trans Networking, 1998. N. Mc. Keown, M. Izzard, A. Mekkittikul, W. Ellersick, M. Horowitz, “The Tiny Tera: A Packet Switch Core”, IEEE Micro Magazine, Jan-Feb 1997. Fast Packet Buffers 1. Nick Mc. Keown Sundar Iyer, Ramana Rao, Nick Mc. Keown “Design of a fast packet buffer”, IEEE HPSR 2001, Dallas. IEE, October 2001 71
IP Lookups 1. 2. 3. 4. 5. 6. Nick Mc. Keown References A. Brodnik, S. Carlsson, M. Degermark, S. Pink. “Small Forwarding Tables for Fast Routing Lookups”, Sigcomm 1997, pp 3 -14. B. Lampson, V. Srinivasan, G. Varghese. “ IP lookups using multiway and multicolumn search”, Infocom 1998, pp 124856, vol. 3. M. Waldvogel, G. Varghese, J. Turner, B. Plattner. “Scalable high speed IP routing lookups”, Sigcomm 1997, pp 25 -36. P. Gupta, S. Lin, N. Mc. Keown. “Routing lookups in hardware at memory access speeds”, Infocom 1998, pp 1241 -1248, vol. 3. S. Nilsson, G. Karlsson. “Fast address lookup for Internet routers”, IFIP Intl Conf on Broadband Communications, Stuttgart, Germany, April 1 -3, 1998. V. Srinivasan, G. Varghese. “Fast IP lookups using controlled prefix expansion”, Sigmetrics, June 1998. IEE, October 2001 72
References Switching • • • N. Mc. Keown, A. Mekkittikul, V. Anantharam, and J. Walrand. Achieving 100% Throughput in an Input-Queued Switch. IEEE Transactions on Communications, 47(8), Aug 1999. A. Mekkittikul and N. W. Mc. Keown, "A practical algorithm to achieve 100% throughput in input-queued switches, " in Proceedings of IEEE INFOCOM '98, March 1998. L. Tassiulas, “Linear complexity algorithms for maximum throughput in radio networks and input queued switchs, ” in Proc. IEEE INFOCOM ‘ 98, San Francisco CA, April 1998. D. Shah, P. Giaccone and B. Prabhakar, “An efficient randomized algorithm for input-queued switch scheduling, ” in Proc. Hot Interconnects 2001. J. Dai and B. Prabhakar, "The throughput of data switches with and without speedup, " in Proceedings of IEEE INFOCOM '00, Tel Aviv, Israel, March 2000, pp. 556 -- 564. C. -S. Chang, D. -S. Lee, Y. -S. Jou, “Load balanced Birkhoff-von Neumann switches, ” Proceedings of IEEE HPSR ‘ 01, May 2001, Dallas, Texas. Nick Mc. Keown IEE, October 2001 73
References Future • • • Nick Mc. Keown C. -S. Chang, D. -S. Lee, Y. -S. Jou, “Load balanced Birkhoffvon Neumann switches, ” Proceedings of IEEE HPSR ‘ 01, May 2001, Dallas, Texas. Pablo Molinero-Fernndez, Nick Mc. Keown "TCP Switching: Exposing circuits to IP" Hot Interconnects IX, Stanford University, August 2001 S. Iyer, N. Mc. Keown, "Making parallel packet switches practical, " in Proc. IEEE INFOCOM `01, April 2001, Alaska. IEE, October 2001 74
Скачать презентацию High Performance Routers Slides originally by Nick Mc
e950308036f2bb65b4805b5b4d91ff66.ppt