Internet 2 Supporting Next Generation Networking NASA IT

Internet 2: Supporting Next Generation Networking NASA IT Summit 2010 Christian Todorov August 16, 2010

Session Agenda • Internet 2 Overview • Network Architecture • Network Services • • IP Circuit Service Use of Dynamic Circuits Waves and Private Networking • Supporting Advanced Research • Discussion

INTERNET 2 OVERVIEW

The Internet 2 Network Overview • Layer 1: Managed wavelengths from Level(3) Communications • Level(3) owns and manages Infinera optical gear: responsible for software upgrades, equipment maintenance, remote hands, sparing, NOC services • Internet 2 NOC has total provisioning control • Layer 2: Internet 2 owned and managed equipment • Using OSCARS and DRAGON GMPLS control plane • Layer 3: Internet 2 owned and managed Juniper MX 960 s • Expanded Observatory • Platform for layer 1/3 network performance data collection, collocation, experimentation • perf. SONAR integration for intra- & inter-network performance analysis • International connectivity • Layer 1 network extended to international exchange points in Seattle, Chicago and New York City • Peering points in Seattle, PAIX, Equinix Chicago 4

Internet 2 Connected Community • • • 5 217 Leading US research institutions ~50, 000 K 20, libraries, museums, zoos and aquariums 38 states covered by SEGP program Federal network peerings and International peerings reaching 80 countries Private networking provided to NOAA and Department of Energy (ESnet); Piloting with USFS Named participant in the FCC’s RHCPP; connecting rural healthcare networks Actively working with the VA JPL and Goddard are Affiliate Members Internet 2 peers with NASA networks: NREN and NISN as well as others such as DREN and USGS

Internet 2 Universities 217 University Members as of June 2010

Sample Listing of Government Affiliates

Internet 2 Affiliate Members • ACUTA: The Assn. for Information Communications Professionals in Higher Ed. • Altarum • Ames Laboratory, US Department of Energy • Argonne National Laboratory • Association of Universities for Research in Astronomy • Berklee College of Music • Brookhaven National Laboratory • Cine. Grid, Inc. • Coalition for Ne. Cleveland Institute of Music • tworked Information (CNI) • Department of Veterans Affairs (VA) • Desert Research Institute • EDUCAUSE • ESnet • Federal Highway Administration, Turner. Fairbank Highway Research Center • Fermi National Accelerator Laboratory • HIMSS (Healthcare Information and Management Systems Society) • Howard Hughes Medical Institute • IEEAF (Internet Educational Equal Access Foundation) • Indian Health Services • Indiana Higher Education Telecommunication System (IHETS) • ISOC (Internet Society) • Jet Propulsion Lab • JSTOR • Laboratory for Telecommunications Sciences (LTS) • Lawrence Berkeley National Laboratory • Library of Congress • Los Alamos National Laboratory • Manhattan School of Music • NASA Goddard Space Flight Center • National Archives and Records Administration • National Emergency Number Association (NENA)

Internet 2 Affiliate Members • • • • National Geographic National Institutes of Health National Science Foundation New World Symphony NIST (National Institute of Standards and Technology) NITLE (National Institute for Technology and Liberal Education) NOAA (National Oceanic & Atmospheric Administration, Washington, D. C. ) Oak Ridge National Laboratory OCLC Online Computer Library Center Pacific Northwest National Laboratory Princeton Plasma Physics Laboratory Ruth Lilly Health Education Center Stanford Linear Accelerator Center SURA (Southeastern Universities Research Association) The Children's Hospital of Philadelphia • • • The Philadelphia Orchestra Association The World Bank Thomas Jefferson National Accelerator TOPIX (Torino E Piemonte Exchange Point) UCAR/NCAR (University Corporation for Atmospheric Research) United Nations System of Organizations (UN) United States Antarctic Program (USAP) United States Department of Agriculture, Forest Service (USFS) United States Department of Commerce Boulder Labs United States Department of State United States Holocaust Memorial Museum University of North Carolina General Administration

The Internet 2 Network Design • Built on dedicated fiber from Level(3) Communications – 13, 000+ mile footprint • Currently based on 10 G waves; moving to 100 G • Built on Infinera innovative optical technology • Simple and convenient add/drop technology • Simple and convenient wave setup • Demonstrated high reliability in initial period of operation on the Level(3) network • Economics of Infinera system are disruptive in the market place • Architecture has maximum flexibility • Every direct connector can access every wave on the system if needed • Adding add/drop points doesn’t require network redesign 10

Internet 2 Network International Connectivity Pac. Wave: APAN/Trans. Pac 2, AARnet, KOREN/KREONet 2, CA*Net 4, GEMNET, REANNZ, T 640 NGIX-Ames: AARnet France Telecom (v 6), HARNET, CERN Star. Light: CERN, JGN 2, ASNet, CERN, KOREN/KREONet 2, CA*Net 4, CERN, JGN 2, SINET, GLORIAD, TANet 2/TWAREN MANLAN: TANet 2/TWAREN, TENET, MCIT/ENERGI, LHCnet, Qatar. FN, CA*Net 4, SURFNet, CERN, GEANT(2), SINET, Nether. Light/IEEAF T 640 Pac. Wave: UNINET, SINET, Qatar. FN, APAN/Trans. Pac 2 TANet 2/TWAREN, Sing. AREN T 640 NGIX-East/A-Wave: CLARA, GEANT, RNP 2 T 640 AMPATH/SFGP: ANSP, REACCIUN-2, red. CLARA, CUDI via CALREN/Pac. Wave T 640 CUDI via UTEP / UT Slide 12

Last updated: April 2005 Internet 2 Network International Connectivity Europe-Middle East Asia-Pacific Austria (ACOnet) Australia (AARNET) M Luxembourg Belgium (BELNET) China (CERNET, CSTNET, (RESTENA) Croatia (CARNet) NSFCNET) alta (Univ. Malta) Czech Rep. (CESNET) Netherlands (SURFnet) Hong Kong (HARNET) Cyprus (CYNET) Japan (SINET, WIDE, JGN 2) Norway (UNINETT) Denmark Korea (KOREN, KREONET 2) Poland (POL 34) (Forskningsnettet) Singapore (Sing. AREN) Portugal (RCTS 2) Estonia (EENet) Philippines (PREGINET) Qatar (Qatar FN) Finland (Funet) Romania (Ro. Edu. Net) Taiwan (TANet 2, ASNet) France (Renater) Thailand (UNINET, Thai. SARN) Russia (RBnet) Germany (G-WIN) Slovakia (SANET) Greece (GRNET) Slovenia (ARNES) Hungary Spain (Red. IRIS) Algeria (CERIST) (HUNGARNET) Sweden (SUNET) Egypt (EUN/ENSTIN) Iceland (RHnet) Switzerland (SWITCH) Morocco (CNRST) Ireland (HEAnet) United Kingdom Tunisia (RFR) Israel (IUCC) (JANET) South Africa (TENET) Italy (GARR) Turkey (ULAKBYM) Latvia (LATNET) *CERN Lithuania (LITNET) Africa Slide 13 Americas Argentina (RETINA) Brazil (RNP 2/ANSP) Canada (CA*net) Chile (REUNA) Mexico (Red-CUDI) United States (Abilene) Peru (RAAP) Venezuela (REACCIUN-2) Central Asia Armenia (ARENA) Georgia (GRENA) Kazakhstan (KAZRENA) Tajikistan (TARENA) Uzbekistan (UZSCI)

Internet 2 Network Federal Peering Pac. Wave: DREN, NREN, ESnet T 640 New York: ESnet Star. Light: Tera. Grid, NREN, DREN, NISN, USGS, ESNet NGIX-Ames: NREN, DREN, NISN, USGS T 640 DC: DRAGON NGIX: DREN, USGS, NISN, NREN, Wright Patt AFB, ESNet T 640 Slide 14

Network Architecture

Objectives • The vision for the Internet 2 Network is a seamless integrated network facility that allows for applications and users to transparently utilize the services and network layers that most appropriately serve their needs, when they need it, in a cost effective manner. • This network facility will allow users to focus on their work and not on the network.

Scalability and Operability • The Internet 2 Network is based on a unique arrangement with Level 3 that represents a hybrid approach to carrier provided resources. • Internet 2 has full control over the provisioning on the network but does not bear the responsibility of supporting and maintaining the physical infrastructure: fiber, amps, transport equipment, etc. • Level 3’s support of the physical network frees Internet 2 of having higher levels of specialized engineering resources dedicated to network support. • The Internet 2 NOC has a full view into the underlying transport equipment and works jointly with a dedicated NOC group within Level 3. • The Internet 2 network is constructed on a dedicated fiber pair and with dedicated transport equipment • The Infinera, and Juniper equipment used in the network are 40 G capable and each has 100 G on their roadmaps 17

Collocation Facilities • 30 facilities across the US • 27 x Level 3 • 1 x Westin Building, Seattle • 1 x 32 Ao. A, New York • 1 x 710 N. Lakeshore Drive, Chicago • Dedicated suites or colo areas – secured • Supports Internet 2, ESnet, NOAA and some Internet 2 connector equipment

Optical Network • Infinera optical transmission equipment • Managed by Level 3 Communications • Internet 2 has direct provisioning control • Internet 2 has direct TL-1 monitoring • Infinera nodes • Redundant Add / Drop (28) • Regeneration (28) • Optical Amplification (193)

Internet 2 Network IP Network Services • Carrier-class IP service • Natively supports IPv 4 and IPv 6, as well as multicast and jumbo frames • Carried over 10 Gbps waves on the extremely reliable Infinera backbone • 40 Gbps and 100 Gbps are future enhancements • Backhaul to the nearest router is included in the IP connection • Redundancy options • One option is a geographically diverse backup connection. This connection is viewed as a backup and the aggregate traffic flow over the primary and redundant circuit must not exceed that of the primary circuit • Internet 2 currently supports 2 x 5 Gig and 2 x 10 Gig connections as well. These can be in geographically diverse locations. • Protection against fiber cuts using the automatic restoration capabilities of the Juniper MX’s MPLS Fast Reroute as an optional service • Additional IP circuits are also available Slide 20

Internet 2 Network Wave. Co Static Circuit Services • Are currently set up manually by the Internet 2 NOC. • Can have any bandwidth up to a full 10 Gbps wave or multiple 10 G waves • Can be provisioned directly over the Infineras (waves) or through the Junipers (sub-rate circuits) • This provisioning can occur over the same physical interface that is used for the Dynamic Circuit Network • Circuits can be offered both as protected and unprotected • Can be ordered for a period of weeks up to years • Have a fee for service that is based on distance, bandwidth, duration and protection scheme. Slide 21

Internet 2 Network Dynamic Circuit Network • The dynamic circuit network (ION) automatically provisions circuits among participants deploying the dynamic networking control plane and protocol • Each participant may connect to one or more circuits allowing them to customize the infrastructure to meet their application needs • The ION service is currently based on connections into the Juniper MX 960 s. • As the ION service grows and other R&E networks across the world “peer” with the ION Network, through exchange points, world wide reachability will grow. Slide 22

ION Service • ION has moved from a SONET-based network on the Ciena’s to an MPLS-based service operating on the current IP network. • This transition took place during the summer • ION connectivity and provisioning options for Internet 2 Network Participants will increase. • A Connector need only maintain a single connection that can provide both IP and ION services on the same port. The Connector is empowered to appropriate resources in a way that works with their regional model. • The shared MPLS model can increase the efficiency of the backbone. • Bandwidth reserved for a circuit instantiation is available for use by other network users when the circuit owner is not utilizing the circuit for transfer • There’s opportunity to provide circuits that can burst above their requested commit rate, if there is sufficient headroom available. • • • ION is a production service managed by the IU NOC. ION circuits are provisioned using a simple and secure web based interface. ION is available to any institution regardless of their membership status. The goal is to make the service available to the entire connector community. Connectors or sites that use ION should contact us to discuss the best way to utilize this new model. • Contact Dale Finkelson (dmf@internet 2. edu) for further information.

I 2 DCN Software Suite • OSCARS (IDC) • Originally an ESnet Project, now joint development by Internet 2 and ESnet • Open source project maintained by Internet 2 and ESNet • DRAGON (DC) • NSF-funded • Open source project maintained by USC ISI EAST and MAX

Inter. Domain Controller (IDC) Protocol (IDCP) • Developed via collaboration with multiple organizations • Internet 2, ESnet, GEANT 2, Nortel, University of Amsterdam, others • The following organizations have implemented/deployed systems which are compatible with this IDCP • • • Internet 2 Dynamic Circuit Network (ION) ESNet Science Data Network (SDN) GÉANT 2 Auto. Bahn System Ciena (Nortel) (via a wrapper on top of their commercial DRAC System) Surfnet (via use of above Nortel solution) LHCNet (use of I 2 ION Software Suite) Nysernet (use of I 2 ION Software Suite) University of Amsterdam (use of I 2 ION Software Suite) DRAGON Network • The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: • Lambda. Station (Fermi. Lab) • Tera. Paths (Brookhaven) • Phoebus

ION Control Plane

Multi-Service/Domain/Layer/ Vendor Provisioning • • • Multi-Domain Provisioning Interdomain ENNI (Web Service and OIF/GMPLS) Multi-domain, multi-stage path computation process AAA Scheduling GEANT TDM GUI XML Internet 2 Network Regional Network API Dynamic Ethernet TDM ESNet Domain Controller Ctrl Element Ethernet SONET Switch Router Dynamic Ethernet Data Plane Control Plane Adjacency LSP Slide from Tom Lehman, ISI-East IP Network (MPLS, L 2 VPN)

Application Interfaces to ION • Phoebus is network middleware to facilitate high throughput for the non-wizard • Based on a “Session” Protocol (Layer 5) that talks to Phoebus Gateways • These Gateways do Transport protocol translation and tuning • Transparently allocates circuits over the Internet 2 ION • Tera. Paths is a system from BNL • Creates end-to-end virtual paths and guaranteed bandwidth for specific flows with Diff. Serv marking • Web interface for manual reservations and a Web Services interface for automatic reservation • Allocates resources on the ESNet SDN or Internet 2 ION

• Phoebus Gateways are placed at network access points and can be thought of as network “on-ramps • At a high level, the goal is to get network data to these gateways which can then intelligently move the data as efficiently as possible to the other edge of the network • Phoebus will automatically utilize ION resources as needed and as available • At the other edge, it again appears as legacy traffic does today, thus neither client or server need to be modified

Waves and Private Networking • Internet 2 makes dedicated waves available over its network to support researchers and organizations • Internet 2 has provided nation-wide dedicated network infrastructure to support the needs of advanced research networks like ESnet (Do. E) and N-Wave (NOAA) allowing for private research networking that has access to the broader Internet 2 community

Monitoring and Measurement

Internet 2 Network Observatory • Dependence on the network is increasing • Distributed applications & moving larger data sets • Network is growing much more complex • Need to better understand the network • Transparent network operation • User must know what performance levels to expect • Network operators must be able to demonstrate that the network meets or exceeds those expectations. • Application developers must have access to tools that differentiate between network problems and application problems. Slide 32

Observatory Services • Several tools and services are available for network troubleshooting: • Active testing to the middle to divide and conquer • OWAMP (latency), BWCTL (throughput), NDT (last-mile/end station) • perf. SONAR (cross-domain tool results, link utilization, and test point locations) • BGP feeds to Route. Views • Link utilization at 10 second intervals • “visible backbone” gives router configuration snapshots • “Router” proxy for interactive query • http: //www. internet 2. edu/observatory/ (data views) • http: //noc. net. internet 2. edu/ 33

perf. SONAR • A way to export measurement data from your network, and consume it from many networks (including your own) • “Interoperable network measurement middleware” • A set of schemas / “APIs” and protocols 35

perf. SONAR Integrates • Network measurement tools and archives (e. g. stored measurement results) • Data manipulation • Information Services • Discovery • Topology • Authentication and authorization 36

Many collaborations are inherently multi-domain, so for an end-to-end monitoring tool to work everyone must participate in the monitoring infrastructure user performance GUI m 1 m 4 measurement archive m 3 FNAL (AS 3152) [US] Analysis tool measurement archive m 1 m 3 m 4 measurement archive m 1 m 4 m 3 GEANT (AS 20965) [Europe] m 1 m 4 m 3 ESnet (AS 293) [US] 37 – 3/18/2018, © 2009 Internet 2 DESY (AS 1754) [Germany] DFN (AS 680) [Germany] 37

Meeting Upcoming Application Demands 38

Demands on the Network • Entering the age of large scientific facilities • Large Hadron Collider at CERN • Very Long Baseline Arrays (radio astronomy) • Large Synoptic Survey Telescope (2010 -13) – 30 TB/night • An increasingly diverse set of demanding applications are utilizing network resources • Telemedicine: BIRN project, proteomics, tele-surgery, remote ICU, radiology: high-resolution 3 D color f. MRI brain scan = 4. 5 PB • Telepresence: master classes, virtual classrooms, tele-psychiatry • High performance video delivery: Uncompressed HD, Cinegrid • Disaster Recovery and distributed storage 40

Tier 1 to Tier 2 Traffic

The New Requirements • High performance applications are dependent on high performance networks • Networks must be fast, reliable, scalable, have flexible architectures, be cost effective, capable of delivering multiple services across multiple network layers, easy to operate and maintain, and have a view towards the future • Enable the user – the network as a service 42

Questions? 43

44