Re-engineering TCP Vegas Neal Cardwell Boris Bak 6 15 99

Re-engineering TCP Vegas Neal Cardwell Boris Bak 6/15/99 1

Building a Better TCP l In deciding when to send, TCP errs on the side of simplicity u u l Some more aggressive approaches: u u l Doesn’t find the right rate Sends in bursts TCP Vegas: finding the right rate Pacing: smoothing the bursts Old ideas, but not deployed u 6/15/99 Don’t fully understand benefits, costs 2

Overview l l Congestion control review The current approach & its problems An alternative: TCP Vegas Implementing Vegas in the real world: The devil is in the details u Problems and patches u 6/15/99 3

Congestion Control: The Problem l l How fast should TCP send? Assumptions: IP’s best-effort service model u Primary feedback: u t t ACKs from receiver Packet loss No information from routers u Perhaps info from past or current flows along the same path u 6/15/99 4

The Sub-Problems l l l Finding the right rate at startup Staying at the right rate Reacting to changes: Path or traffic changes u More or less bandwidth available u 6/15/99 5

TCP Reno l l l cwnd: bounds un-ACKed packets ssthresh: guess as to safe cwnd By default, probe for more bandwidth slow start: cwnd *= 1. 5 each RTT u congestion avoidance: cwnd+=. 5 each RTT u l Packet loss signals congestion u u 6/15/99 Fast retransmit, fast recovery: cwnd/=2 Retransmission timeout: cwnd=1 6

Problems with TCP Reno l During slow start u l Underutilizes and then swamps path No “right rate”: cwnd traces a sawtooth Underutilizes path u Increases queuing delay u Causes loss, reducing throughput u Inherently biased against long RTTs u 6/15/99 7

Alternatives: Startup l l Original [Jacobson 88]: always cwnd*=1. 5 Several studied in [Allman, Paxson 99] Tracking slow start flights [BP 95] u Closely-spaced ACKs [Hoe 96] u Tracking closely-spaced ACKs [AD 98] u Receiver-side estimation [AP 99] u l But no studies w/ typical paths 6/15/99 8

Alternatives: Steady-State l l l l Original [Jacobson 88]: always cwnd++ CARD [Jain 89]: RTT Tri-S [Wang, Crowcroft 91]: rate DUAL [Wang, Crowcroft 92]: RTT TCP Vegas [Brakmo, Peterson 94]: rate Buffer-Fill Avoidance [Awadallah, Rai 98]: RTT New, from UCB (Mo, Walrand 99): RTT, rate 6/15/99 9

TCP Vegas l In congestion avoidance cwnd = (actual rate)x(base. RTT) + 2 pkts u Each RTT, tweak cwnd by 1 pkt if needed u l During slow start To reduce overshoot, increase cwnd only every other RTT u Exit slow start when u t 6/15/99 cwnd > (actual rate)x(base. RTT) + 1 pkt 10

Vegas in Theory and Practice l l l In isolation, 3 -70% higher BW than Reno Loses to Reno b/c it queues few packets In steady state, 2 pkts queued per flow u l l No loss Stable Basically fair w. r. t. other Vegas flows No RTT bias, unlike Reno u Slight bias toward newer connections u 6/15/99 11

Our Experience l l l Plan: implement Vegas for Linux 2 months fixing Linux TCP bugs Found problems with Vegas u l Some fixed, some are works in progress. . . This congestion control business is tricky stuff. . . 6/15/99 12

Time Stamp Granularity l l l Arizona Vegas used 1 ms Linux has 10 ms-granularity time in kernel Nice for smoothing out noise? Nope! u l l RTT to UCB goes from 2 ticks to 3 - whoa! Need resolution much smaller than typical RTT to avoid granularity artifacts We now use microsecond resolution 6/15/99 13

Slow Start Problems l Vegas slow start is too slow. . . Increase by 1. 5 x every other RTT u Most flows are short, so. . . ouch! u l But still eventually overshoots… Suppose cwnd of W = bottleneck BW u It will send 1. 5*W for two RTTs before slowing down u l Our implementation increases every RTT u 6/15/99 Need a better heuristic for exiting slow start 14

Delayed ACKs l When receiver’s delayed ACK timer fires: u u u l Causes ~100 ms delay that looks just like queuing Artificially small actual rate Can compensate by min filtering When receiver ACKs every other packet: u u 6/15/99 Sender sends burst of two packets Only get RTT sample for 2 nd packet of bursts But this packet was queued behind the 1 st… Can compensate by allowing >1 packets queued 15

Idleness Bug l Arizona Vegas assumes flow never idle u l l thus always has recent actual rate info On restart from idle, must wait until we get fresh info before making a decision Our implementation incorporates this fix 6/15/99 16

Loss Hurts Vegas More l l l TCP uses duplicate ACKs to detect losses So big windows help loss recovery But Vegas tries to keep a small window For my 512 Kbps line, Vegas cwnd=4 u If Vegas loses 2 packets to my house, RTO u l Could compensate w/ Net. Reno [Lin, Kung 98] Send new packet out for every duplicate ACK u No RTO unless all packets in flight are lost u 6/15/99 17

Route Changes l l Long noted: If new route is longer, this looks like queuing delay Vegas slows when it should speed up! Simulated proposals: choose base. RTT as min of RTTs over last N ACKs or T secs Our experience: tricky; depends on bandwidth, delay 6/15/99 18

Persistent Queuing l l l Long noted: Vegas can’t distinguish propagation delay from persistent queuing New flows get bigger base. RTT, cwnd Two approaches: OK; prioritizes short flows u Not OK; causes unacceptable queuing delay u l l Proposal: treat as route change We have yet to try this 6/15/99 19

Conclusions l Hard to fine-tune and be robust to: Random queuing delay u Delayed ACKs u Route changes u Link layer loss, retransmission, compression u l l Reno may be dumb, but it’s robust I still have high hopes for Vegas. . . 6/15/99 20

Future Directions l Lots of experiments & tweaking Different approaches to slow start u Different heuristics for detecting route change u Low bandwidth, and high bandwidth-delay u l l l Integrate Vegas with FACK Competing with Reno; more aggressive? What if we could get help from routers? . . . 6/15/99 21

Bibliography: Experience l l l L. Brakmo, S. O'Malley, and L. Peterson. “TCP Vegas: New techniques for congestion detection and avoidance. ” SIGCOMM 94. ftp: //ftp. cs. arizona. edu/xkernel/Papers/vegas. ps L. S. Brakmo and L. L. Peterson, "TCP Vegas: End to End Congestion Avoidance on a Global Internet. ” IEEE JSAC, Vol. 13, No. 8, October 1995. ftp: //ftp. cs. arizona. edu/xkernel/Papers/jsac. ps. Z J. S. Ahn, Peter B. Danzig, Z. Liu and L. Yan, "Evaluation of TCP Vegas: Emulation and Experiment. ” SIGCOMM 95. http: //catarina. usc. edu/yaxu/Vegas/Reference/vegas 95. ps 6/15/99 22

Bibliography: Modeling l l l T. Bonald. “Comparison of TCP Reno and TCP Vegas via Fluid Approximation. ” http: //www. inria. fr/mistral/personnel/Thomas. Bonald/po stscript/vegas. ps. gz J. Mo, R. La, V. Anantharam, J. Walrand. “Analysis and Comparison of TCP Reno and Vegas. ” INFOCOM 99. http: //walrandpc. eecs. berkeley. edu/Papers/vegas. pdf G. Hasegawa, M. Murata, H. Miyahara. “Fairness and Stability of Congestion Control Mechanism of TCP. ” INFOCOM 99. 6/15/99 23