Proposed future direction for CHEETAH Malathi Veeraraghavan University

Proposed future direction for CHEETAH Malathi Veeraraghavan University of Virginia August 23, 2006 l Outline l Strategy discussion: l What's our goal for the CHEETAH network: § l Bandwidth sharing mode: § l e. Science network or a scalable GP network? Book-Ahead (BA) or Immediate-Request (IR)? Tactical aspects: l Network evolution l Networking software modules l Application software modules l Interconnection to HOPI/DRAGON 1

Observation l "Many e-science experiments are unique applications that involve collaboration among a handful of facilities. As a result, networks supporting these experiments are optimized to provide maximum throughput to a few facilities, as opposed to moderate throughput to millions of users, which is the raison d'etre for commercial networks. " 2

e. Science networks l e. Science network requirements l l l Number of users small Hard to achieve high utilization; also not impt. Overprovision network to keep call blocking rate low We can then focus on creating software to allow scientists to automatically create highspeed application-specific topologies: AST, UCLP, OSCARS, USN scheduler, BRUW Bandwidth-sharing algorithms of less concern 3

General-purpose commercial networks l Has to be scalable: large number of users l l Metcalfe's statement: Value of a network increases exponentially with the number of users High utilization is an important goal Low call blocking probability or low waiting time for resources Focus on efficient bandwidth-sharing algorithms 4

Circuit/VC service on GP commercial networks l Just for ISPs/enterprise admins: l l needs similar to e. Science l router-to-router circuits l limited number of users l high-bandwidth, long-held circuits l low price not a high priority l need BA mode of bandwidth sharing For end users l large number of users can only offer moderate BW and limited call holding times IR mode of sharing becomes feasible 5

BW sharing modes in circuit/VC networks m is the link capacity expressed in channels e. g. , if 1 Gbps circuits are assigned on a 10 Gbps link, m = 10 Large m Moderate throughput Small m immediate-request with call blocking + retries ("call queueing") Short calls Bank teller (video, gaming) immediate-request with delayed-start times (file transfers) l l High throughput Long calls Doctor's office book-ahead Mean waiting time is proportional to mean call holding time Can afford to have a queueing based solution if calls are short 6

m=10 Pq=41% Offered load: call arrival rate/call departure rate Prob. of arriving job finding all m circuits busy Impact of increasing m at different values of link utilization Ud Link capacity expressed in channels High-rate per-call circuits Low-rate per-call circuits 7

Mean waiting time for delayed calls Number of ports aggregating traffic on to the link Impact of mean call holding time : per host call-generation rate Ud: 90% 8

Main findings of analysis l Two key parameters: l If m is small (per-circuit BW is high) l and mean call holding time is large § l and mean call holding is small (file transfers) § l l then need BA to avoid long waiting times then use "call queueing" If m is large, switch hardware costs increase l N, number of aggregation ports, high l level of demultiplexing high Moderate m: best choice 9

Book-Ahead (BA) or Immediate-Request (IR)? Bandwidthsharing mechanisms Book-Ahead (BA) Immediate-Request (IR) e. Science networks very large file transfers need high-BW and long holding time + remote viz. need to reserve other resources such as displays None? general-purpose networks circuit service to only ISPs/enterprise admins - router-to-router circuits circuit service for end users - host-to-host + router-torouter (end-to-end) - partial-path router-to-router circuits on congested links (called in by end user) 10

Support for the BA mechanism of bandwidth sharing l l l Since RSVP-TE does not have parameters for BA calls (call duration, start time), this mode is not implemented in switch controllers Need an external scheduler to manage bandwidth into the future Easiest to make it centralized - one per domain Cannot utilize the BW management software implemented in switch controllers as part of GMPLS control-plane software The BA mode is necessary for high-BW, long-held calls 11

Support for the IR mechanism of bandwidth sharing l l Switches have built-in (G)MPLS controlplane software (RSVP-TE/OSPF-TE) Bandwidth management is part of RSVP-TE switch controller software l l l Hence it is distributed bandwidth management Need to limit call holding time - reminders for renewals and automatic release Moderate-to-high per-call bandwidth 12

To implement BA, IR, or both? l Implement only BA l l l Develop and "standardize" protocols for scheduler-toscheduler signaling for interdomain circuits (one centralized scheduler per domain) Implement scheduler and test with other networks Create software tools to enable scientists and ISP/enterprise admins to visualize network topologies and request appropriate circuits/VCs High-BW, long-held: Therefore AAA is a must Path being pursued by DRAGON, USN, OSCARS, UCLP 13

Opportunity missed if the whole optical testbed community only experiments with BA l What opportunity? l l Enable the creation of large-scale circuit/VC networks with moderate-rate circuits that can support a brand new class of applications l economic value for the telcom industry A "reservations-oriented" mode of networking to complement today's connectionless Internet l ala airlines that complement roadways Could prove useful to FIND, GENI, net-neutrality Alternative pricing models for bandwidth 14

What "brand new class of applications? " l l l Video, video Gaming Remote software access + Sync. storage Async storage Multimedia (large) files in web sites 15

Video applications l l l Improve quality of conferencing, telephony, surveillance, entertainment and distancelearning by a significant degree Expend bandwidth for a higher-quality, lower latency, multi-camera, auto-movement, automixing experience Make the "flat world" flatter Energy savings/environmental benefits Moderate bandwidth - IR with call blocking/retries 16

Gaming applications l l l Current gamers buy personal graphics cards Players talk of "lag" caused by differences in graphics processing speeds Moderate-speed circuits can enable a new class of games in which rapidly-changing scenes are possible l compare movies in which multiple story lines keep scenes changing vs. gaming scenes Players connect to graphics servers Data transferred is not GL commands, but rather rendered bits (doable? ) Moderate bandwidth - IR with call blocking/retries 17

Remote software access/sync storage l Remote software access l l l Synchronous storage access l l Reduce computer administration cost Personal computers vs. machine rooms I loaded 22 new applications on my new laptop l Instead: connect and run! Virtual Computing Laboratory: Mladen Vouk, NCSU Disaster recovery Moderate bandwidth - IR with call blocking/retries 18

Asynchronous storage l Asynchronous storage depots will lower costs for l l l backups disaster recovery Need for increased storage grows with multimedia files High bandwidth, short calls IR with delayed start 19

Larger files in web sites l Multimedia files in web sites l Imagine the use of video/audio files in all sorts of web sites instead of ASCII l l My own course PPT files: I use audio sparingly because of bandwidth Think assembly instructions for electric fans, furniture § l Think hotel web pages § l l l Kinesthetic learning - show me a video Show me exactly where the beach is relative to my room; do I have a balcony - saying it in text format is one thing; seeing it in a video format quite another! Content distribution network & web caching High bandwidth, short calls IR with delayed start 20

Are all these "high"-BW apps just a matter of increasing BW of links in the current Internet? l l l No The socialistic mode of bandwidth sharing on the Internet discourages individual investment in network bandwidth Age-old question: l should we pay for bandwidth with tax dollars "free" for the whole community? l l "Tragedy of the commons" (Tanenbaum) should we create a network where individuals can pay for bandwidth on congested links more directly? - think higher-toll HOV lanes 21

What does all this mean? l Let's build a scalable circuit/VC network in which bandwidth is shared in IR mode l l Scalability will create "Metcalfe's value" Provides an opportunity to finally recoup our investment in (G)MPLS technologies l standards creation effort l implementation: Cisco, Juniper, Sycamore, Movaz Assign at least a few of the optical testbeds that we are investing in now to study whether this IR mode of bandwidth sharing can help with our understanding of net-neutrality, economic growth, FIND questions IR more natural in data world unlike in airlines (BA) 22

Argument: IR is just a "now" in BA l BA and IR cannot coexist without some form of bandwidth partitioning l l l BA allows for high-BW, long-duration calls IR calls will suffer a high call blocking rate if supported through BA scheduler (the "add-now -as-an-option-in-scheduler" solution) Should you admit an IR call if it arrives a few seconds before start time of a BA call and hope it completes before the BA call start time, or reject the call and waste bandwidth? 23

CHEETAH and TSI l l The CHEETAH network solves only part of the TSI problem Other problems l l Cray computer I/O problem Local-area connectivity within NCSU If the CHEETAH project was a production solution to support TSI, we should spend money to solve these two problems for TSI But as an experimental short-lived networking project, where should we focus? 24

Outline l Strategy discussion: l What's our goal for the CHEETAH network: l l Bandwidth sharing: l l e. Science network or a scalable GP network? Book-Ahead (BA) or Immediate-Request (IR)? Tactical aspects: l CHEETAH network evolution l Networking software modules l Application software modules l Interconnection to HOPI/DRAGON 25

Network evolution to support IR l Current CHEETAH network only supports 10 circuits per OC 192 link l l remember IR mode does not work well when m, the link capacity in channels, is small (i. e. , 10) Recoup OC 1 -crossconnect capability of the SN 16 Ks from its current 1 Gbps use l Has three advantages l l l supports higher m; better for IR GMPLS standards based signaling Call setup delay: 166 ms for two-hop instead of 1. 5 sec! 26

Network evolution options l Four options: l l VLAN-enabled NICs + VLSR for SN 16 K 15454 with VLSR IP router with VLSR Ethernet switch with VLSR 27

Example: web caching application VT UVa C'ville, VA mvsut 6 UTK duke VLANs CHEETAH zelda 4 ORNL wukong UN C Raleigh, NC Atlanta, GA Gatech zelda 3 NCSU UGa Xiuduan Fang, xf 4 c@virginia. edu Bob Gisiger, rwg 5 f@virginia. edu 28

VLAN-enabled NICs + SN 16 K VLSR l l SN 16 K has data-plane support to map a sub-Gb/s VLAN on an Ethernet port to a corresponding number of OC 1 s on a SONET port But, it does not have control-plane support for this type of circuit l l Even GMPLS support for the Gb. E port mapping to a 21 -OC 1 VCAT signal is an experimental release just for CHEETAH usage Because GMPLS support for such hybrid circuits is nonstandard Can implement our own (non-standard) solution as a VLSR But, goal is to use off-the-shelf switches with GMPLS support to demonstrate IR mode 29

15454/VLSR at each Po. P l l Make the 15454 serve as the intermediary between Ethernet NICs in hosts and SONET based SN 16 Ks at CHEETAH Po. Ps A 15454 VLSR could be useful for other projects, UCLP, Ultralight l l Two problems: l l l Cisco has no plans to implement a GMPLS control-plane engine for the 15454 Non-standard solution for hybrid circuits VLAN ID continuity requirement Cannot support partial-path circuits 30

IP router/VLSR at each Po. P l l Use channelized OCxx SONET interfaces to connect IP router to SN 16 K Connect web caches to router Have routers initiate pure SONET circuit setup Use PBR or just ordinary routing table update to map flows to different OCxx circuits; support multiple circuits from one web cache 31

CHEETAH wide-area network I OP UVa /H tex r Vo Via PI ernet/HO Via Nys Raleigh Po. P (MCNC) SN 16000 via NCREN OC 192 Control Gb. E/ card 10 Gb. E card CUNY NCSU End hosts OC-192 (via NLR/SLR/NCREN) ORNL Po. P SN 16000 End hosts Atlanta Po. P (SOX/SLR) Gb. E/ Control OC 192 10 Gb. E card SN 16000 OC 192 Control Gb. E/ 10 Gb. E card End hosts OC-192 ORNL Ga. Tech 32

CHEETAH evolution to support sub-Gb/s circuits UVa Gb. E Raleigh Po. P SN 16000 OC 192 Control OC 192 card CUNY Gb. E NCSU End hosts OC-192 (via NLR/SLR/NCREN) ORNL Po. P SN 16000 Gb. E End hosts Atlanta Po. P OC 192 Control OC 192 card OC-192 Gb. E ORNL SN 16000 OC 192 Control OC 192 card Gb. E End hosts Gb. E Ga. Tech 33

IP router/VLSR at each Po. P l Can support end-to-end circuits l l l web caching CDN servers video apps at 10 -15 Mbps - map to one OC 1 storage depots Has the potential to support PPCs (partial-path circuits) Place router with VLSR in enterprises at edge of Gb. E cheetah access link 34

Ethernet switch/VLSR at each Po. P l Does not help with the problems noted in today's Gb/s circuit use of the SN 16 K l l long call setup delays: 1. 5 sec non-standard solution high per-circuit BW Using an Ethernet switch/VLSR at an enterprise (e. g. CUNY) requires all VLANs sharing 1 Gbps CHEETAH access link to be switched to the same exit SN 16 K. Even worse, m=1 if whole 1 Gb/s link used for a circuit 35

Software modules required l Networking software: l l l CVLSR for IP router CTCP code to support multiple simultaneous flows Application software: l l l Add CHEETAH API to web caching squid software Write software for video apps CDN and storage software 36

Upcoming year goals specified in special report Work item Milestones and deliverable Responsible individuals Item I Stabilize the CHEETAH network; increase user base; complete doc. MV and Xuan/Tao Item II Extend CHEETAH network by adding routers/VLAN switches/MSPPs MV and Tao Item III Interconnect to testbeds, such as HOPI, USN, DRAGON MV and Tao 37

Upcoming year goals specified in special report Work item Milestones and deliverable Responsible individuals Item IV Develop software, both apps and networking s/w, such as CVLSRs Router CVLSR: Mark CTCP: Helali Item V Support ORNL and NCSI in TSI apps Mark and Tao/MV Item VI Enhance theoretical understanding of sharing modes IR blocking/retries: XF BA: Xiangfei IR delayed start: Mark Apps: Xiuduan 38

Equipment required l IP routers with channelized SONET cards with GA GMPLS UNI implementation l l l need one for ORNL Po. P if we can partner with SOX in ATL, NCREN in Raleigh, MAX in Mc. Lean, purchase channelized OC 192 cards IP routers with Gb. E blades for V. Tech and UVA If NC 454 transponders are unavailable, purchase transponders for DC-Raleigh NLR link - since HOPI doesn't have this link Colocation costs at NLR Mc. Lean and Raleigh 39

Interconnect CHEETAH and HOPI l l l Through IP routers With our IP router/VLSR combo, setup router-to-route SONET OC 1 circuits via cheetah and router-to-router VLAN virtual circuits through HOPI. At routers, do PBR mapping for flows or just update routing tables This means packets go back to IP layer between two networks 40

CHEETAH-HOPI interconnection Web cache VLAN Web cache Mc. Lean, VA MPLS Web cache NC 10 Gb. E Web cache CHEETAH SONET HOPI network: courtesy of Rick Summerhill Web cache TN Web cache 41 GA

HOPI and web caching l l Seems like a good match Rick's black cloud experiment - same as web caching Exercises "hybrid" goal of HOPI Small per-circuit BW possible with VLANs 42

Connection to DRAGON l l l Spoke with Jerry Sobieski, Aug. 12, 2006 He said DRAGON Po. Ps have Ethernet VLAN switches Therefore, can use similar IP router demarcation points to interconnect CHEETAH/HOPI to DRAGON 43

Conclusions l We'd like to l l enable and demonstrate general-purpose apps using circuit/VC service with scalability as a key goal support IR mode of bandwidth sharing with limited per-call bandwidth and limited call holding times l l call blocking with retries delayed start 44