Globus Advance Reservation and Network Environment Testbed GARNET

Globus Advance Reservation and Network Environment Testbed GARNET Volker Sander, Alain Roy sander@mcs. anl. gov Argonne National Laboratory 10/07/1999

Outline l The Basic concept of GARA l GARNET and Experiences l How to join the Effort l Demonstration 2

Goals l Provide end-to-end Quality of Service to applications. This requires: – Discovery and selection of resources – Allocation of resources – Advance reservation of resources Supercomputer Workstation 3

Difficulties/Solutions l Lack of support for advanced reservations – NGI Bandwidth Broker effort is based on immediate reservations l Heterogeneous resources – More than just Networking Qo. S – We provide uniform interfaces l Need to work with complex sets of resources – Multi-domain, Multi-Resource-type – We use co-reservation agents 4

Solution: GARA Globus Architecture for Reservation and Allocation l Three important contributions: – Support for Advance Reservations (AG) – Uniform treatment of underlying resources > Network, disk, CPU etc… > Currently supported: Differentiated Services, DPSS, and DSRT – Layered architecture enables generic coreservation agents > User is able to select API best suitable for his need 5

GARA Basics—Reservations l There is a generic “reservation”, which has several properties: – Start Time (“now” or future) and Duration – Resource type/Underlying resource identifier – Resource-specific (bandwidth, % CPU…) l All reservations are treated uniformly: – Create/Modify (Given properties) => Returns Reservation Handle – Destroy – Monitor (Callbacks or Polling) 6

A picture of what happens (creating a reservation) User Program or Agent Create Reservation Handle Gatekeeper (Authenticate and Authorize) Gatekeeper Service Local Resource Manager API (LRAM) Resource Manager Resource (Router. . . ) 7

A picture of what happens (claiming a reservation) User Program bind(handle) Resource Manager Setup reservation LRAM API Resource Enforce Reservation Note: GARA supports third party binding 8

Example: Network reservation l In advance, make a reservation: è RSL: & (reservation-type=network) (start-time=now) (duration=3600) (endpoint-a=128. 135. 11. 4) (endpoint-b=128. 135. 11. 1) (bandwidth=1000) ç Handle: Opaque string H 1 l At run-time: – Bind(H 1) 9

The Resource Manager l l l To track advanced reservations, we use a simple “slot table” We assume exclusive access to resource through the resource manager We aren’t tied to this particular resource manager 100% % of available resource allocated 0% Time 10

Co-Reservation Agents l When multiple resources are needed, an agent: – Discovers applicable resources via the MDS – Reserves resources – Informs user when resources are ready. – Meta-RM support for integrating other RMs l The uniform interface enables these agents to be created easily. – Given this ease, it is convenient to express different strategies such as: > Depth-first vs. best-first > Coping with failures 11

The Big Picture MDS Co-Reservation Agent Gatekeeper GRIO LRAM Scheduler LRAM Gatekeeper Diffserv LRAM DSRT LRAM Workstation Supercomputer Router Workstation 12

End-to-End Network Reservations l Algorithm for a single network reservation – Discover which gatekeepers/resource managers need to be contacted (MDS) – Make reservation at each one. End-to-End Co-Reservation Agent Resource Manager User process MDS Resource Manager Workstation 13

The GARA API l globus_gara_reservation_create() è Gatekeeper contact è RSL reservation specification ç Reservation Handle or Error l globus_gara_reservation_modify() è Old Reservation Handle è RSL reservation specification ç New Reservation Handle or Error 14

The GARA API (continued) l globus_gara_reservation_cancel() è Reservation Handle ç Error l globus_gara_reservation_status() è Reservation Handle ç Status/Error l globus_gara_reservation_callback_register() è Reservation Handle è Callback Function/User Parameter ç Error 15

The End-to-End Agent API l For making network reservations – Deals with co-reservation along path. l Very similar to GARA API: – Ex: globus_gara_end 2 end_create() l l l Handles MDS lookups. Handles making multiple reservations, but provides user with single reservation handle. Probably the API you want to use. 16

GARNET goals l Provide dedicated testbed for evaluating current and future mechanisms used for implementing GARA – Qo. S-Type Network: similar to IETF’s diffserv model – Qo. S-Type CPU: DSRT-Scheduler – Qo. S-Type Disk: Distributed Parallel Storage System (DPSS) l Demonstrate validity of concept – End-to-End Reservation – Application level Qo. S 17

Quality of Service for IP Traffic l l The ability to define or predict the performance of traffic on a network Needed in case of congestion – Leave the current best-effort model – End-User is able to specify his requirements Guaranteed bandwidth major concept l Building blocks – Link Layer Qo. S (ATM Qo. S) Create VC with specific parameters (ABR, CBR, UBR, …) – IP-Qo. S Flow Based Qo. S: Integrated Services (RSVP) Aggregate Based Qo. S: Differentiated Services (Premium service) 18

The Basic Concept Aggregate based scheduling Flow based Set IP precedence for non exceeding flows with bound GARA Reservation 19

The Extension Server Client GARA API Diffserv Resource Manager 20

The Multidomain Extension Server Client GARA API Diffserv Resource Manager 21

Building Blocks for Diffserv l Traffic marking based on a specific policy l Admission control l Congestion handling – Congestion avoidance – Congestion management l Traffic shaping Service Level Agreement (SLA) 22

Actual Mechanisms (Cisco) l Committed Access Rate (CAR) – Per flow based at End-Systems > Mark packets if they are not exceeding a rate limit > Drop packets beyond the limit or transmit them via best effort – Per Precedence between domains > Corresponds to a static SLS > RM will configure the limits on demand in later releases – Allows short term bursts l Weighted Random Early Detection (WRED) l Weighted Fair Queuing l Generic Traffic Shaping or PVC attributes 23

GARnet- Globus Advance Reservation and Network Env Testbed Linux SUN Ultra 60 WFQ CAR WFQ Cisco 7507 SUN Ultra 2 WFQ Cisco 7507 CAR WFQ Cisco 7507 SUN Ultra 2 Fast. Ethernet link OC-3 link 24

25

Evaluation Tools l Several Modified versions of ttcp (premium) – GARA-enabled (wait for reservation) – Support for a desired application rate > long term control > short term control – Consecutive bandwidth reporting – Bulk transfer ttcp in work l UDP traffic generator (competetive traffic) – Andy Adamson, University of Michigan – Added support for automated testing l Modified version of DPSS (premium) – GARA enabled – Nice example on how to use GARA 26

Overhead for packet headers (40 Bytes) 27

BUFFER 28

The TCP “Problem” Receiver Transmitter TCP’s sliding window introduces burstiness of Traffic Socket rcv buffer Socket send buffer MSS-sized (media specific) TCP segments 40 Byte Header overhead Ack Ack Maximum outstanding of 1 actual window Last. Byte. Sent - Last. Byte. Acked <= Rcv. Window. 29

ze TCP Congestion Window Si The TCP “Problem: Reno-TCP 45 Slow Start Exponential Growth 40 35 30 25 Average 20 window size 15 Congestion Avoidance Phase Linear Growth Half Window Size in case of duplicate Acks 10 5 0 0 time 30

The TCP “Problem” l TCP’s flow control – Traffic becomes bursty if the actual window size is large – Bandwidth*Latency Product as ideal window size l TCP’s congestion control – “Slow Start Phase” > shrink Window to 1 > increase size with every Ack – Congestion Avoidance > Half Window size > Increase size by one for every full window roundtrip 31

32

33

34

WRED impacts on RTT 35

WRED impacts on the Window size 36

… the Throughput 37

… some updates 38

… increasing tcp_cwnd_max 39

… new throughput 40

41

… new throughput 42

Using WFQ 43

Using WFQ 44

Using WFQ 45

Current Status l An early working prototype (alpha!) of GARA exists – Differentiated Services-like Qo. S – Real-Time CPU Scheduling (DSRT) – DPSS Disk Access l l l An early prototype of the end-to-end reservation agent exists. Successful demonstration on GARNET and over ESnet’s NGI testbed Work with bleeding-edge adopters has already started. 46

The GARA Source Tree l Subdirectories represent different layers – end 2 end – gara – lram – resource_manager – slot_manager – (dsrt) l l We are providing examples for every layer docs directory contains lots of useful information 47

Doing Tests l Select your favorite application l Determine port numbers l l edit (set endpoint systems) and run script: gara/utilities/demo_make_reservation Create congestion Try demo_modify_reservation for getting different premium bandwidth Try demo_cancel_reservation 48

Install GARA l Get GARA (currently through E-Mail from sander/roy @mcs. anl. gov) l Get SSLeay 0. 9. 0 and install it l Get Globus 1. 1 configure and compile it deploy is not necessary for doing tests Run grid-cert-request to apply for a Grid X 509 Certificate l Read gara/docs/* l Edit gara/build l Run gara/build l Customize GARA – gara/resource_manager/programs/diffserv_manager. conf – gara/resource_manager/programs/setup_flow. cfg – (Samples are in the documentation directory) l Experiment! 49

Router Requirements l Cisco 75 xx VIP 2 -50 l Get new IOS XE beta release Experimental Version 12. 0(19990728: 032821) [rolsen-conn_4 xe_fix 243] Available at ftp: //ftpeng. cisco. com/rolsen/rsp-pv-mz. 091099 l l l ip cef distributed Define an empty policy-map set-precedence on the edge router Define a consecutive set of access-lists to be used by GARA (100 -199) Define class-map for precedence 3 or 5 on every router Define a policy-map containing the class assigned with the supported premium BW l Assign this as service policy on each output interface l Define CAR per precedence on the ingress site 50

Future Work l Add new features – Authorization – Dynamic SLS – Enhance CLI – Increase support for adaptive applications l Deployment with Globus – Packaging – Easy install l Supporting more resource types – Job Schedulers – Alternate Network Qo. S (RSVP) – Disk (GRIO) – Others (Memory, etc…) 51

Conclusions l GARNET is ready. . . l GARA could prove its capabilities – Try it… – Once installed, it is easy to use! – Suggestions are welcome!!!! l Take a look at http: //www. mcs. anl. gov/qos 52