SWITCHlambda — Experiences with national dark fibers

SWITCHlambda - Experiences with national dark fibers @ SWITCH Willi Huber / Urs Eppenberger huber@switch. ch / eppenberger@switch. ch 2003 © SWITCH

The SWITCHlambda Project goals: • Provide fore front network services to the Swiss Universities at low recurring cost • Network scalable to multiples of 10 Gbps with low additional cost • Long living infrastructure, no need to change provider every couple of years • Independence of carrier market 2003 © SWITCH 2

SWITCHlambda Project Basic decisions: • Replacement of the STM-1 links by Gigabit. Ethernet connections based on dark fiber • Long term lease of dark fiber, indefeasible right of use (IRU) for at least 10 years • Use of DWDM on main trunk lines • Redundant wide area links • Simple network design (no SDH, no ATM): Use of Ethernet-Interface all over the place 2003 © SWITCH 3

SWITCHlambda Network Design • Hierarchy of technologies – DWDM on main trunk lines (16 l possible today, expandable if later needed) – CWDM or single GE to smaller sites off the backbone • Transparent optical path between Zurich and Geneva – No regeneration between the two main sites of SWITCH • Optical path on main trunks designed for 10 Gbps / l – Chromatic dispersion compensated – Polarization mode dispersion under control • Bi-directional use of fibers – lower fiber costs 2003 © SWITCH 4

SWITCHlambda Network end of 2003 350 km Basel Zurich Bern Lausanne Geneva dark fiber based links (approx. 1200 km) leased GE link 2003 © SWITCH 5

Bi-directional Use of Fiber • Only one fiber used for the DWDM system • Second fiber free to connect smaller sites inbetween Universities with cheap GE technology DWDM node Router n * 1 GE / 10 GE implemented with DWDM P 2 P Gigabit. Ethernetconnections Router 2003 © SWITCH DWDM node One pair of fiber 6

Typical SWITCHlambda node Bi-directional transmission mode: i. e. a single fiber is used for signals in both directions, leaving the second fiber free for future use. 2003 © SWITCH 7

Sorrento Gigamux 2003 © SWITCH 8

Sorrento CWDM Jump. Start-400 CWDM System – 8 wavelengths (4 receive, 4 transmit) – Bi-directional operation – Optional backup path SWITCH uses currently 1 pair of Jumpstarts for two p 2 p GE connections over on single fiber. 2003 © SWITCH 9

Bi-directional Gigabit. Ethernet Cheap solution for p 2 p GE connection over one single fiber with colored GBICs and passive optical devices Switch / Router Splitter Signal reflection (e. g. bad connector) Coloured GBIC OADM Switch / Router Splitter Coloured GBIC OADM 2003 © SWITCH 10

Testing 10 Gbps/channel capability Goals • verify equipment capability under real life conditions • verify optical design (dispersion compensation) • verify fiber quality for 10 Gbps application (PMD and other adverse characteristics) Realization • DWDM interface cards on loan from Sorrento (10 GE WAN-PHY) • IP equipment from Force 10 (2002) and Cisco (2003) • traffic generation with Linux based workstations and the free software “mgen” 2003 © SWITCH 11

10 Gbps tests 10 Gbps/l, 360 km, bi-directional transmission 1. Test with signal regeneration optical - electrical optical in the middle (at Bern) 2. Test with transparent optical signal Zurich - Geneva 3. Same as 2. , but with additional attenuation of 3 d. B of optical signal at the sender Each test: 8 Gbps of traffic during 35 hours, 0 bit errors Þ BER < 8 * 10 -15 Þ SWITCHlambda ready for 10 Gbps ! 2003 © SWITCH 12

Pros and Cons: Technical View • Stable, long living infrastructure • High future potential • This new stuff is fun 2003 © SWITCH • none 13

Is this reliable? • Stable infrastructure with occasionally long interruptions – Interruptions mainly due to planned maintenance – True fiber cuts are rare, but last for hours if not days (e. g. fiber on high voltage power lines, fiber along railway) – Experience of a big Swiss carrier: Fiber cuts happen mostly in the local loop (city) • Our experiences so far – Several interruptions due to planned maintenance (lasting up to 1 day) – Broken optical amplifier (interruption of 4 hours) » Redundant links are mandatory for a good sleep! 2003 © SWITCH 14

No connection, what now? • 24/7 monitoring by Ascom Netwatch – ping, SNMP – alarm SWITCH staff – basic analysis • SWITCH NOC for analysis and action decision • University IT staff – power problems – physical equipment check • Onsite support by Ascom Netwatch – localize cable problems with OTDR (optical time domain reflectometer) – Cisco hardware • Onsite support by Deltanet for Sorrento DWDM 2003 © SWITCH 15

Pros and Cons: Operational View • we are the boss 2003 © SWITCH • Big effort for negotiations with a lot of contract partners • no single provider to blame in case of connection loss • need a car to fix problems instead of telnet 16

Network Cost, general Remarks • Cost for wide area links – Well known in advance • Local loop cost – Vary a lot from place to place – Good (say cheap) solutions need a lot of planning and patience – Building own infrastructure is economically better than renting, but not possible in all cities – In some cities there is a de facto monopole of the city’s industrial works – The railways are an excellent partner for wide area links: low local loop cost because the schools are mostly located near railway stations 2003 © SWITCH 17

Network Cost, general Remarks (2) • Cost for optical equipment – DWDM is expensive – Long 10 Gbps/l DWDM trunks are very expensive (dispersion compensation, amplification) – CWDM stuff is getting cheap • Routers / Switches – Ethernet interfaces are cheaper by factors than SONET ones 2003 © SWITCH 18

Cost Structure Depreciation period used for the calculations: Fiber optic infrastructure (wide area & local loop): 10 years Fiber optic equipment DWDM & CWDM: 5 years 12 % 6% 31 % wide area fiber local loop fiber DWDM equipment CWDM equipment maintenance & support 7% 44 % Total yearly cost: 2. 05 MEUR Cost include: Investment years 2001 -2004 10 Gbps between the major sites maintenance & support 2003 © SWITCH 19

Comparison with predecessor SWITCHng SWITCHlambda Years covered: 1999 -2002 2001 -2004 Technology: ATM/STM-1 Sites connected: 13 universities dark fiber, GE & 10 GE 21 universities Yearly cost: 2. 05 MEUR 2003 © SWITCH 2. 95 MEUR 20

Isn’t it cheapter to buy it? We have not done a serious market review lately or even negotiated for capacity to get the real prices But we have a figure we assume is the best current offer on the market for high capacity links: 100 k. EUR – limited to 300 Mbit/s – local loop must exist For 21 sites: 2. 1 MEUR Þ Similar cost as SWITCHlambda, but less flexibility and performance. 2003 © SWITCH 21

Pros and Cons: Economic View • Low recurring cost • Low marginal cost for additional bandwidth 2003 © SWITCH • High investment • Uncertainity about development of bandwidth prices 22