Nether Light day-to-day experience APAN lambda networking BOF

Nether. Light day-to-day experience APAN lambda networking BOF Erik Radius Manager Network Services SURFnet bv Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

SURFnet • Provides the Dutch National Research Network • Not for profit company, 50 employees • 150 connected organizations, 500, 000 users • Turnover (2002): 30 million Euro • Infrastructure services: – innovation paid for by government – cost effective exploitation for higher education and research Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

SURFnet 5 • Partners BT and Cisco • 15 Po. Ps connected by thirty 10 Gbit/s lambdas • Dual stack IPv 4 and IPv 6 since 2001 • More than 100 institutes connected at Gbit/s level • 500, 000 users Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Nether. Light • Optical Internet exchange point in Amsterdam • Built and operated by SURFnet • Experiments with light path provisioning in a multi domain environment • www. netherlight. net Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Nether. Light Network 2001 • 2. 5 Gbit/s lambda between Star. Light, Chicago, USA and Nether. Light, Amsterdam, NL • Lambda terminated on Cisco ONS 15454 muxes, – WAN side: SONET framed: OC 48 c – LAN side: Gb. E interfaces to computer clusters Nether. Light Star. Light Gb. E 2. 5 G lambda Gb. E Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Nether. Light Network at i. Grid 2002 New York 10 Gbit/s IEEAF Amsterdam Nether. Light 10 Gbit/s Level 3 Chicago Star. Light 2. 5 Gbit/s SURFnet DWDM SURFnet 2. 5 Gbit/s CERN Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27 Geneva CERN Dwingeloo Astron/Jive

Nether. Light equipment in A’dam Lambdas to Chicago, Geneva, Prague etc Ethernet switch (Cisco 6509) DWDM over dark fiber to JIVE DWDM line terminal (Cisco ONS 15252) L 1 (SONET) multiplexer (Cisco ONS 15454) optical cross connect (Calient PXC) 10 GBase-LR 1000 Base. T 1000 Base. SX 1000 Base. LX (SM fiber, (copper) (MM fiber, (SM fiber, l = 1310 nm) l = 850 nm) l = 1310 nm) Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27 Computer clusters Uv. A/NIKHEF

International lightpath network 1 Q 2004 10 Gbit/s New York MANLAN Stockholm Northern. Light IEEAF 10 Gbit/s 2. 5 Gbit/s 10 Gbit/s 2 x 10 Gbit/s Tokyo WIDE 2. 5 Gbit/s Chicago IEEAF 10 Gbit/s 2 x 10 Gbit/s 2. 5 Gbit/s SURFnet 10 Gbit/s Amsterdam DWDM SURFnet NSF 10 Gbit/s SURFnet 10 Gbit/s Tokyo APAN London UKLight lambda service path IP service path Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27 Dwingeloo ASTRON/JIVE Geneva CERN 2. 5 Gbit/s Prague Czech. Light

Day-to-day lightpath use • Part of lambda capacity statically configured for IP production traffic – Router-to-router SONET circuits between Amsterdam and Chicago for SURFnet and others (e. g. NSF link) – Link & interface state monitored by SURFnet NOC • Part of lambda capacity for lightpath research community – Changed on weekly/monthly basis upon request – Has a ‘test’ status; is hardly monitored by NOC Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

NSF funded 10 G lambda US - EU Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Application for a lightpath • Researchers can apply for a lightpath by an e-mail to netmaster-optical@surfnet. nl • Information in the application should include: – Test purpose – End points A and Z [CERN, Star. Light, Stockholm] – Interface type [1 Gb. E: SX or LX, 10 Gb. E WAN PHY] – Number of lightpaths needed – Lightpath capacity in multiples of SONET STS-6 (311 Mb/s): [STS-6 / STS-12 / STS-24 / STS-48 / STS-192] * • Full Gb. E capacity: STS-24 * – VLAN ID (if applicable) – Start date, end date – Contact details (e-mail, phone) * Bandwidth granularity depends on transport equipment, e. g. Cisco ONS 15454 Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Day-to-day operations (1) • SURFnet NOC manages production network plus Nether. Light – Part of Nether. Light has production status – Nether. Light test circuits get lower priority than SURFnet 5 IP network • Some learning curve to get NOC engineers familiar with transport (SONET) troubleshooting – ONS 15454 network operations: craft terminals only • DWDM link to Dwingeloo (ASTRON): very stable, no pro-active management of optical system yet – DWDM (ONS 15252) craft terminal is not very user-friendly Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Day-to-day operations (2) • International lambda network monitoring – SNMP monitoring by NOC for SONET alarms and port errors – BER level monitoring on ONS 15454 SONET ports • BER parameter is part of SLA with carriers – Gb. E ports: errors mostly attributable to LAN troubles – No IP traffic monitoring possible on SONET ports • ONS 15454 cannot look inside SONET circuits, only provide overall quality information (BER, errored-seconds etc) • No trouble with clocking observed yet – SONET network should have one master clock – ONS 15454 internal Stratum III clock is sufficient, no external clock needed Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Day-to-day operations (3) • Layer 0: optical cross connect – Calient Photonic Cross Connect, automated patch panel – Research status only, managed by University of Amsterdam – Topology discovery difficult • Layer 1: SONET layer – Very flexible Gb. E path setups on +- biweekly basis – 10 G end-to-end also possible, for couple of months at a time – Multi-domain setup works very well by building half-circuits – ONS 15454 networks from Canet and SURFnet pre-wired at Star. Light Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

What did we learn / need to work on • Point to point lambdas are a powerful service – High-performance research groups are pulling at us after getting a taste of 10 GE all-the-way… • Straightforward to build a hybrid network architecture on top of optical: – Some lambdas to interconnect your IP prod. routers – Some lambdas for new high speed few-to-few usage • Current optical-electrical-optical equipment can allocate sublambdas to individual applications – Ethernet-over-SONET based, this will get more advanced soon (we need to learn what LCAS can do here) • Management is still cumbersome, learning curve for all of us – How to make this dynamic? Or self-provisionable…? Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27

Light Path Provisioning in SURFnet 6 Operations Management Broker OGSA Applications Web service Layer 3 flow data collection and analysis Light Path Provisioning via Control Plane SURFnet 6 Layer 2 / Layer 1 network Light Path Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27 SURFnet 6 Routers

Erik. Radius@SURFnet. nl www. netherlight. net www. surfnet. nl Techs in Paradise 2004, Honolulu / Lambda Networking BOF / Jan 27