Implementing and Maintaining an ISP Backbone Kevin Butler

Tier 1 ISP Backbones • Comprise some of the world’s largest IP networks • Tier 1 companies include Sprint, AT&T, PSINet • UUNET has the world’s largest IP data network, presence on four continents and future expansion into Latin and South America

Service Level Agreements • SLAs are an important and prestigious tool in attracting and maintaining customers • Comprised of uptime guarantees and bounds on latency through various geographic regions • most ISPs currently have latency < 50 ms across the US

Supporting the Customer • Quality and expertise of first-line customer support varies wildly between companies • depending on size, geographic location and company focus, some front-line support teams outsourced to third parties • some in-house high level support teams have skills equivalent or superior to NOCs

Network Operations Centres • Generally the teams concerned with backbone maintenance and support • trend towards consolidation into “Super. NOCs” (eg. one for Americas, one for Europe) • specialisation within NOC for product support (eg. dial, VPN, backbone NOCs)

NOC Tools • NOCOL - Network Operations Centre On Line (freeware UNIX) • Mediahouse monitoring (mainly web) • Micromuse Netcool (now owned by Lucent) - used by MCI World. Com, PSINet, BT

Dial Access • Dial is a major selling point, especially with customers who travel a lot or are their own ISPs • connections made through an Ascend MAX TNT, which can support up to 720 concurrent callers • back-end is a DS-3 into a backbone router, routers advertised by an IGP (eg. RIP)

Dial-Related Technologies • COBRA (Central Office Based Remote Access) allow building of virtual POPs by backhauling PRIs • RADIUS (Remote Authentication Dial In User Service)

Integrated Services Digital Network • ISDN customers authenticate by RADIUS similar to dial users • underlying architecture similar but dial equipment often administrated differently • ISDN maintained within same AS as backbone whereas dial often in its own AS

DS-1 and high-speed access • Customer connections usually multiplexed, come into DSU as a channelised DS-3 • gateway routers on ISP side usually Cisco 7500 series, increasingly using Cisco 12000 • customers connect using Cisco 1604, 2621, some 3600 series, very large customers use 7500 series routers

Gateway Routers • obtain routes from customers usually statically, but sometimes by BGP • usually run link-state IGP within AS (eg. OSPF, IS-IS) • Cisco 7513 backplanes 1. 8 Gbps while 12008 does 40 Gbps

Where does traffic go from here? • Most ISPs have two levels of networks above the access router • Metropolitan networks aggregateway traffic, generally city-wide (if multiple POPs in city) • transit networks aggregate metro network’s traffic, responsible for inter-city transport

ATM Switches • Terminate long-haul OC-12, OC-48 circuits and metro rings • Choice of vendor contingent on ISP, commonly Newbridge, Fore Systems (ASX 1000 and ASX-4000)

Example of an ATM interface TR 1. EG 1: interface ATM 2/0 description To HA 13. BLAH 1 3 C 1 atm vc-per-vp 512 atm pvc 16 0 16 ilmi ! interface ATM 2/0. 195 point-to-point description To XR 1. BLAH 1 ATM 6/0 ip address 146. 188. 200. 98 255. 252 ip router isis Net-Backbone atm pvc 195 0 195 aal 5 snap clns router isis Net-Backbone

Implementation of BGP • BGP run between autonomous systems and peers, as well as multi-homed customers • monolithic AS broken up into BGP confederations for ease of work • routes controlled using access lists and route maps

BGP • Communities are destinations that share common attributes (eg. through access-list filters) BGP table version is 23718690, local router ID is 205. 150. 242. 2 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete Network *>i 24. 64. 0. 0/19 *>i 24. 64. 0. 0/14 *>i 24. 64. 32. 0/19 *>i 24. 64. 96. 0/19 *>i 24. 64. 192. 0/19 *>i 24. 64. 224. 0/19 *>i 24. 65. 0. 0/19 *>i 24. 65. 96. 0/19 *>i 24. 65. 128. 0/19 Next Hop 198. 133. 49. 7 198. 133. 49. 7 Metric Loc. Prf Weight Path 100 0 6327 6172 100 0 6327 i 100 0 6327 6172 100 0 6327 6172 i i i i i

Advantages of BGP for User • Allows for load-sharing and redundancy • routes can be biased through AS path prepending • requirement is high-quality router with close to 100% uptime to avoid connection flaps and subsequent route dampening

Common Customer Issues • Static routes on backbone - often difficult to spot, can cause very strange routing results • pull-up routes for netblocks smaller than /24, required to avoid BGP dampening • BGP recalculations - if done on a transit router, entire backbone segments can experience outages

Customer Requirements of the Backbone • Redundancy - networks are redundant but card failures can take down whole routers • physical connection to POP from customer is SPF • low latency - massive increases in demand on backbone makes this difficult • over $2 million a day spent on global backbone upgrades

DSL: low cost, high speed • DSL might phase out ISDN connections • difficult to troubleshoot from network standpoint • connections pass through telco’s frame or ATM cloud between DSLAM and VR • Red. Back SMS (Subscriber Management System) 1000 commonly used as VR

Red. Back SMS 1000 • Supports up to 4000 sessions • OC-3 out to metro network • traffic-shaping accomplished with profiles atm profile samplecust counters shaping vbr-nrt pcr 1000 cdvt 100 scr 100 bt 10

Increasing Capacity • Backbone capacity increasing at a huge rate • Traffic engineering combined with high backplane becoming increasingly important • many ISPs turning to Juniper routers • UUNET rolled out production OC-192 c with Juniper M 160 running MPLS

Juniper Routers • JUNOS supports MPLS and RSVP isis { interface all; } ospf { area 0. 0 { interface so-0/0/0 { metric 15; retransmit-interval 10; hello-interval 5; } } } [edit]

Distributed DOS attacks • Can be very detrimental to backbone (even causing switch crashes) • Combated by rate-limiting ICMP on routers • Most effective defense is community-wide egress filtering; requires co-operation throughout the Internet

Canadian Network Challenges • Geographically, population resides in virtually a straight line across the south • major focus is on southbound capacity to the US • CRTC regulations on telcos create different arrangements • heterogeneous network to the US, integration a big issue

Questions? • Anything I can clarify or expand on. . . • Thank you!