Routers with Small Buffers NANOG October 25 2005

Routers with Small Buffers NANOG, October 25, 2005 Yashar Ganjali High Performance Networking Group Stanford University yganjali@stanford. edu http: //yuba. stanford. edu/~yganjali/ Joint work with: Guido Appenzeller, Mihaela Enachescu, Ashish Goel, Tim Roughgarden, Nick Mc. Keown Special thanks to: Level 3 Communications

The Story # packets at 10 Gb/s 1, 000 Intuition & Proofs Sawtooth Peak-to-trough Evidence Simulated Single TCP Flow 10, 000 Smoothing of many sawtooths Simulations, Experiments 20 Non-bursty arrivals Simulated Many TCP Flows (1) Assume: Large number of desynchronized flows; 100% utilization (2) Assume: Large number of flows; <100% utilization October 2005 Routers with Small Buffers 2

Backbone Router Buffers Source Router C Destination 2 T n Universally applied rule-of-thumb: q A router needs a buffer size: n n n 2 T is the two-way propagation delay (or just 250 ms) C is capacity of bottleneck link Context q q q Mandated in backbone and edge routers. Appears in RFPs and IETF architectural guidelines. Usually referenced to Villamizar and Song: “High Performance TCP in ANSNET”, CCR, 1994. Already known by inventors of TCP [Van Jacobson, 1988] Has major consequences for router design October 2005 Routers with Small Buffers 3

Single TCP Flow Only W packets may be outstanding Rule for adjusting W q q October 2005 If an ACK is received: If a packet is lost: Routers with Small Buffers W ← W+1/W W ← W/2 4

Single TCP Flow Only W packets may be outstanding Rule for adjusting W q q If an ACK is received: If a packet is lost: W ← W+1/W W ← W/2 Source Dest Window size t October 2005 Routers with Small Buffers 5

Time Evolution of a Single TCP Flow Time evolution of a single TCP flow through a router. Buffer is 2 T*C October 2005 Time evolution of a single TCP flow through a router. Buffer is < 2 T*C Routers with Small Buffers 6

Synchronized Flows t n n n Aggregate window has same dynamics Therefore buffer occupancy has same dynamics Rule-of-thumb still holds. October 2005 Routers with Small Buffers 7

Many TCP Flows B 0 Buffer Size October 2005 Routers with Small Buffers Probability Distribution 8

Required Buffer Size Simulation October 2005 Routers with Small Buffers 9

Real Network Experiments n n Stanford University dorm traffic Network Lab (Cisco routers) at University of Wisconsin Internet 2 Operational Internet backbone October 2005 Routers with Small Buffers 10

Internet Backbone Experiment n n Buffer sizes 190 ms, 10 ms, 5 ms, 2. 5 and 1 ms Load balancing High link utilization Long duration (about two weeks) October 2005 n n Drops, utilization data collected every 30 seconds Test flows Routers with Small Buffers 11

Packet Drops vs. Link Load Buffer size = 190 ms, 10 ms, 5 ms MAX October 2005 Routers with Small Buffers 12

Packet Drops vs. Link Load Buffer size = 1 ms October 2005 Routers with Small Buffers 13

Relative Link Utilization of the link with 1 ms buffer / Utilization of the link with 190 ms buffer October 2005 Routers with Small Buffers 14

Relative Utilization (Cont’d) October 2005 Routers with Small Buffers 15

Theory vs. Practice Theory (benign conditions) Poisson M/D/1 D B Loss independent of link rate, RTT, number of flows, etc. Typical OC 192 router linecard buffers over 1, 000 packets 5 orders of magnitude difference! October 2005 Can we make traffic look “Poisson-enough” when it arrives to the routers…? Routers with Small Buffers 16

Paced Injections Assume: Buffer size > Distance between consecutive packets of a single flow S > q q n q q Limited injection rate Flows are not synchronized; and Start times picked randomly and independently We can prove that the packet drop probability is very low. Similar results from Cambridge/UCL, UMass and Stanford See October 2005 papers in: ACM Computer with Small Buffers Routers Communications Review, July 2005 17

O(log W) Buffers Assumptions: q Internet core is over-provisioned n q Example: Load < 80% There is spacing between packets of the same flow: n n Natural: Slow access links Artificial: Paced TCP Result: Traffic is very smooth, and loss rate is very low, independent of RTT, and number of flows. With a buffer size of just 10 -20 packets we can gain high throughput. October 2005 Routers with Small Buffers 18

Leaky Bucket – Paced vs. Reno Bucket drains with a constant rate. Load is 90% for both cases. October 2005 Routers with Small Buffers 19

TCP Reno sends packets in a burst High drop rate October 2005 Routers with Small Buffers 20

Paced TCP Spacing packets Much lower drop rate October 2005 Routers with Small Buffers 21

O(log W) Buffers Regular TCP With Pacing October 2005 Routers with Small Buffers 22

O(log W) Buffers October 2005 Routers with Small Buffers 23

Ideal Experiment Packet Trace Monitor n n Packet Trace Monitor Highly loaded link, with real/realistic traffic Precisely controlled router buffers Packet traces with precise timestamps Work in progress: Sprint, Verizion, Telcordia, Lucent, … October 2005 Routers with Small Buffers 24

Conclusion and Future Work n Theory: q Reducing buffer sizes by a factor of sqrt(N) does not affect the network performance. q Reducing the buffer sizes to O(log. W) does not affect the network performance if: n n The network is over provisioned; and We use Paced TCP; or Have slow access links Experimental Validation: q Thousands of ns 2 simulations q Stanford dorm, University of Wisconsin Testbed, Internet 2, Level 3 Communications, … q Ongoing work and need your help October 2005 Routers with Small Buffers 25

Thanks! More Info? yganjali@stanford. edu http: //www. stanford. edu/~yganjali http: //yuba. stanford. edu/~yganjali/research/bsizing/ October 2005 Routers with Small Buffers 26

O(log W) Buffers With a large ratio between core and access link bandwidth Bottleneck Bandwidth = 1 Gb/s October 2005 Routers with Small Buffers 27