Lambda Station On-demand flow based routing for data

Lambda Station: On-demand flow based routing for data intensive GRID applications over multitopology networks Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford, Andrey Bobyshev, Maxim Grigoriev California Institute of Technology. Harvey Newman (co-PI) ESCC/Internet 2 Joint Techs Workshop Fermilab, July 15 -19, 2007

Outline of the talk ● goals and major directions of the project ● software architecture, API, middleware ● results of using Lambda Station service ● problems and challenges, plans

Basic terms Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications ● ● PBR – policy based routing PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing ● Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP). ●

The goal of the project. . . deals with the last-mile problem in local area networks The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks. – selective forwarding on a per flow basis – alternate network paths for high impact data movement – access control in site edge routers for those selected flows – on-demand from applications (authentication & authorization) – current implementation based on policy-based routing & including the support of DSCP marking

Flows and DSCP tagging Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis. Typical steps of alternative path reservation: generate request for service (application, application's proxy, admin) ● LS negotiates service and parameters with remote site ● configure local and wide area network (not yet available) ● marking traffic (if specified). ● Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started: ● endpoint IP addresses ● DSCP

Lambda Station Building Blocks Storage & application space Management Remote. Lambda. Station SOAP/JClarens LSInterface LS-Management & Reporting Interface LS request interface LS-to-LS service my. SQL: requests, Authorization SOAP over HTTPS Service-based Architecture: LS Controller – manages LS persistence, controls other services ●LS persistence is stores current state of the system ●LS request Iinterface webservice for placing all kinds of “ticket” related requests ●LS-to-LS – webservice for LS definitions propagation and LS discovery ● NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN ● local definitions LS Controller LS persistence online updates NETWORK CONFIGURATOR Vendor specific modules CISCO Data Exchange Control & Management Force 10 WAN LS-2 -LS service

LS JAVA API Components

Java implementation of LS Software Service Oriented Architecture ● utilized JClarens and Axis framework as a web-services toolkit ● messages are defined and strongly validated by XML schema ●LS service is multi-threaded, one thread for LSController, one thread for LS 2 LS service and threads pool for open. Svc. Ticket requests ● LS 2 LS and client-2 -LS authentication is based on g. Lite library and supports standard Grid proxies and KCA-issued certificates ●Authorization is based on rules set ●General framework persistence is accomplished by My. SQL DB backend ● secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1. 1) ●

Java implementation of LS Software (continued) Automated LS and PBR client configuration management ●Automated deployment ( one can install on any Linux box) ●LS Controller, LS 2 LS , LS AAA, LS client interface are ready for deployment. Supported java and perl clients. ●Some interest from ANL to support C client for Globus toolkit ●Network Config calls implemented in interface and may relay requests to perl service ( SOA at work ) ●Currently deployed and work ( exchanging PBR and LS configurations) at FNAL ( 2 stations) and Cal. Tech ( 1 station) ●

DSCP Tagging Complexity of using DSCP tagging: ● preservation of DSCP is not guaranteed in WAN DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for highperformance transferring ) ● LS software does support two different modes of DSCP tagging : ● fixed DSCP values to identify site's traffic. ● DSCP value is assigned dynamically on per ticket base.

Netconfig Module dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs. ● Tasks to configure PBR in Cisco devices: ● interface on which PBR is applied needs to be configured with “ip policy route-map” statement ● route map needs to be configured as ordered list of match/action statements ● match criteria need to be associated with ACLs

open. Svc. Ticket request Accepts many input parameters, most of them could also be determined automatically: ● Remote Lambda. Station site's identifiers ● Source/Destination PBRClient identifier or/and list of IP netblocks ●IP protocols, protocol's ports or port range ● local. Path, remote. Path identifiers ( subject to available resources) ● Bandwidth out, Bandwidth in ● board. Time, start. Time, end. Time, travel. Time Returns ID of locally assigned ticket Operational modes (subject to authentication, authorization): new ticket - creating a new ticket ● join - provides ID and required parameters(DSCP) of already opened ticket ● extend ticket – allows to extend already created ticket. ●

LSiperf End-to-End Test 1. Data transfer started: – 10 GE host; 5 tcp streams – Network path is via ESnet – OC 12 bottleneck… – Path MTU is 1500 B – Lambda. Station service ticket is opened 4 2. Lambda. Station changes network path to USN 3. Host path MTUD check detects a larger path MTU 2 4. Lambda. Station service ticket expires: – Network path changed back to ESnet 1 3

Lambda Station Test. Bed

SRM/DCache 1. 7 LS-awareness Advanced Networks Cal. Tech Wide Area Network SRM Cal. Tech Lambda. Station Star. Light FNAL Lambda. Station CMS core router Site Network CMS SRM USCMS Tier 1 normal traffic flow High Impact traffic Production USCMS SRM server sends requests to Lambda Station to stir a high-impact traffic into Advanced Network infrastructure How to demonstrate increasing transfer rate ? ? In tests we could generate a data stream and monitor its rates when switching traffic between alternative paths. Production traffic is not deterministic.

Project accomplishments Software version 1. 0 (a fully functional prototype supporting whole cycle of LS functionality) ● positive results of testing between Fermilab and Caltech ●lsiperf, ls. Traceroute – wrappers around well known applications to add Lambda Station awareness ( based on prototype version 1. 0) ● SRM/d. Cache integration – testing, added in production 1. 7. 0 release ● run LS-aware SRM/d. Cache on production cluster at Fermilab ●Interoperable Java implementation of the major components of Lambda. Station ( perl, java clients available) ●

Problems and challenges Traffic Asymmetry is bad for high performance applications ● Network (Lambda Station) awareness is too complex ● Definition of PBR Clients is a complex issue, auto definition is not yet available ● Plans ● release fully functional Java LS SRM/d. Cache with production quality LS support ● add real-time monitoring of resources ( perf. SONAR ) ● add WAN control plane module ● integration with OSCARS ( unified Network Path Reservation Model ? ) ● References: http: //www. lambdastation. org/

END

Miscellaneous Slides Simulation of multiple Lss ● screen shoots of ticket's queue ● SC 05 Demo ●

Netconfig Module dynamically modifies the configurations of local network devices. ● a vendor dependent components. ● Cisco routers with IOS version supporting sequencing type of named ACLs. ● Tasks to configure PBR in Cisco devices: ● interface on which PBR is applied needs to be configured with “ip policy route-map” statement ● route map needs to be configured as ordered list of match/action statements ● match criteria need to be associated with ACLs

open. Svc. Ticket request Accepts many input parameters, most of them could also be determined automatically: ● Dst site's identifiers ● Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR) ● IP protocols, protocol's ports ● local. Path, remote. Path identifiers ● BWout, BWin ● board. Time, start. Time, end. Time Returns ID of locally assigned ticket Operational modes (subject authentication, authorization and quoting): new ticket - creating a new ticket ● join - provides ID and required parameters(DSCP) of already opened ticket ● extend ticket – allows to extend already created ticket. ●

Directions of the project. building a wide-are testbed infrastructure ● designing, developing Lambda Station software, Lambda Station (network) aware applications ● researching effects of flow based switching on applications ●

Application's behavior in flow based switching environment Tuning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows DSCP tagging with IPtables ● switching between two paths with different MTUs ●

Effect of DSCP tagging with IPTables

Motivation of the project unprecedented demands for data movement in physics experiments such as CDF, D 0, Bar and coming LHC experiments ● massive, globally distributed datasets growing to the 100 petabytes by 2010 ● collaborative data analysis by global communities of thousands of scientists ● available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs. ● Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints ●

LS multitopology network model NG-ADM NG-B RT 1 Multiple Network Toplogies Admission Group of network devices Blue Green RT 3 RT 2 Red RED-B-IN NG-C RT 1 RT 3 RT 2 BLUE is Production Path Client. A rules for Red& GREEN topologies: GREEN-OUT RT 1 H 2 PBR-clients or regular clients at the remote sites RED-Client. A-IN GREEN-Client. B-IN RED-Client. B-IN GREEN-Client. B-IN Client A, RED & GREEN rules for NGC PBR-client H 3 PBR-client B H 1 RT 2 H 2 RED-OUT RT 3 H 3 PBR-client A NG-A Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs

Lambda. Station SC 05 Demo Fermilab SC 05/Seattle Commodity Internet/SCinet lambdastation@FNAL lambdastation@SC 05/High. Speed Links nws-lab. fnal. gov A 122. 302. sc 05. org lsiperf charley. fnal. gov srmcp A 126. 302. sc 05. org

PMTU D Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.