The Evolution of the U S Internet Peering

The Evolution of the U. S. Internet Peering Ecosystem William B. Norton Co-Founder and Chief Technical Liaison Equinix, Inc.

Internet Researcher • • • 90% with Internet Companies Engineers EQIX: Massive Carrier Neutral Colocation Lots of Blinking lights… Observe: documentation on HW&Protocols Lack of Operations documents Research How does Peering work? White paper process. .

Community Operations Research • • • Ground Truth w/dozens of experts Write White Paper v 0. 1 Walk community through WP for comments Revise White Paper into new version Present White Paper at conferences Solicit comments over lunches and dinners White papers so far…

Internet Operations White Papers 1) 2) 3) 4) “Interconnection Strategies for ISPs” “Internet Service Providers and Peering” “A Business Case for Peering” “The Art of Peering: The Peering Playbook” 5) “The Peering Simulation Game” 6) “Do ATM-based Internet Exchanges Make Sense Anymore? ” Freely available. Send e-mail to wbn@equinix. com New white Paper: The Evolution of the U. S. Peering Ecosystem, Agenda…

Agenda • Definitions to level set – Peering, Transit, ISPs – Synch Point here. • Introduction to notion: “Peering Ecosystem” – Internet=Many Internet Regions each with – Tier 1 ISPs, Tier 2 ISPs, Content Providers • Evolution of the U. S. Peering Ecosystem – 3 major evolutions

Def: Internet Service Provider • Def: Internet Service Provider is an entity that sells access to the entire Internet. This service is often called “Internet Transit” • Def: Transit is a business relationship whereby one entity sells access to the entire Internet.

Definition of Transit Def: Transit is the business relationship whereby one ISP announces (sells) reachability to the *entire* Internet to a customer. 2) Reachability Announcement ISP A 1) ISP A buys Transit Service Upstream Transit Providers Provider 3) Traffic Flows • $$$ ? Typically usage-based • Pricing ’ 04: $18 -$200/Mbps • Volume based on 95 th Percentile measure • Transit is Simple, Convenient: • Upstream handles the delivery of packets to the Internet by some means Usage: “I’m purchasing transit from Level 3. ” I N T E R N E T W O R K S

Cost of Transit Traffic Exchange Source: 2004 Survey Avg. Range: $20 -$400/Mbps 95 th Percentile 2001 Pricing Sampling Range: $100 -$1200/Mbps 1/15/04 field spot quotes: GX: $65/Mbps, HE: $25/Mbps My Financial Models used $388/Mbps Cogent $20/Mbps UU: $45/Mbps, etc. varied commits 0 -1000 Mbps Why Peer?

Scaling at the Edge The Zone: +1 M gaming user/mo! Streaming Media Broadcast. com: 100, 000 concurrent unicast streams 15 million streaming hrs/mo, 1300 Live events/day Streaming: Broadcast U IG M Telephony B Video P Multimedia $$$ T Interactive Gaming S Content Heavy ISP SPOT Events V T R E $$$ A Access Heavy ISP M S 56 k 384 k 1. 5 m Access 1 T 1 consumer streaming 24/7 cost $388/mo in transit Tier 1 Equipment: $13 B/’ 00 -> $42 B/’ 04 AOL+ DSL: 1, 000/day (Source: Infonetics) Roadrunner Cable Modems: 1 M subscribers Negotiate Peering with Upstream ISPs Or focus on peering with like-minded ISPs…

Why Peer? Motivations for Peering • Financial: Reduce load on expensive Transit service • Traffic src/dest • Measure vs Intuit • Usage-based Billing • Engineering: Lower latency ISP A Seek transport Interconnection $ 1 st Stage of Peering: Top 10 destination ISP list x Transit $$$ Transit ISP B Transit $$$ Top 10, Def: Peering…

Sample Top 10 Destination List Def: Peering

Definition of Peering Def: Peering is the business relationship whereby ISPs reciprocally announce reachability to each others’ transit customers. Peering West. Net USNet Routing Tables East. Net Transit • Peering is *not* a transitive relationship • Peering *does not* provide access to the entire Internet Usage: “I buy transit from UUNet and Peer with East. Net, West. Net. ” Cost Peering vs. Transit

The Cost of Peering 1) Monthly Circuit Fee into IX 2) Monthly IX Port Fees

When does Peering Make Sense Financially?

Generalization: ISP Peering Breakeven Analysis Graphs $/Mbps Exchanged Breakeven Point (ISPs Indifferent between Peering and Transit traffic exchange) Peering Prefer Peering Risk Cost of Transit Cost of Traffic Exchange in Peering Relationship Number of Mbps exchanged

Synch Point • You should have – a working lexicon at this point – A sense of the motivations for Peering & Transit • Any questions so far?

The Internet Peering Ecosystem Pre-Crash : circa 2000

Peering Ecosystem • System of interconnected (Peering and Transit) players • Each group with homogenous peering motivations and observed behaviors – Tier 1 ISPs – Tier 2 ISPs – Content Providers

Def: Tier 1 ISP • Definition: A Tier 1 ISP is an ISP that has access to the entire Internet Region solely through peering relationships. (i. e. doesn’t buy transit from anyone to see regional routes) • Motivation: Doesn’t need to peer with anyone else, so • Peering only with other Tier 1 ISPs

Def: Tier 2 ISP • Definition: A Tier 2 ISPs is any ISP that has to purchase transit to reach destinations in an Internet Region • Motivation: Peer to decrease costs, improve performance • Peering: with Tier 1’s is preferable but hard to get, peering with Tier 2’s as appropriate • Must purchase Transit

Def: Content Providers • Def: A Content Provider is an entity that produces content for delivery over the Internet but does not sell transit. • Motivation: Best Customer Experience (engineered performance) • Peering: Not typically done, prefer SLAs. • Transit from ISPs

1998 -2000 Tier 1 ISPs $ Transit Tier 2 ISPs Thickness represents In-Group Peering Content

The Evolution of the U. S. Peering Ecosystem Act II: 2000 -2003

Evolution of the U. S. Peering Ecosystem Four key events leading to drastic disruption in the Peering Ecosystem: 1) The 1999/2000 Telecom Collapse 2) The growth of the Used Equipment Market 3) The Upstream Provider for the Cable Companies (@Home) went bankrupt 4) Peer-to-Peer file sharing systems (like Kazaa) grow in popularity, traffic grows exponentially between Access Providers (1. 5 MB MP 3 700 MB AVI files) 5) Transit Prices Drop, Transport Prices Drop Leads to 3 Major Evolutions

Evolution #1 – Cable Companies are Peering • • 40% of traffic Kazaa 3 -5 Gbps of transit traffic-> Gbps of peering potential ! Significant because 1) Volume 2) Open Peering 3) Kazaa Effect

U. S. Cable Companies Eyeballs

AOL

1998 -2000 Tier 1 ISPs $ Transit Tier 2 ISPs Thickness represents In-Group Peering Content

2000 -2003 Evolution #1 – Cable Companies Peering Tier 1 ISPs Cable. Cos Tier 2 ISPs Content Peering

Evolution #2 – Network Savvy Large Scale Content Companies get into Peering • To reduce transit costs • To improve the end user experience • They need to move out of bankrupt colo anyway Significant because 1) Volume of traffic is huge 2) Content Providers have Open Peering 3) Leading players pave the way

1998 -2000 Tier 1 ISPs $ Transit Tier 2 ISPs Thickness represents In-Group Peering Content

2000 -2003 Evolution #2 – Large Scale Network Savvy Content Peers Tier 1 ISPs Tier 2 ISPs LSNSC Content

Other Broadband Players • Significant, but… • Not Open Peers so disincentive to peer • So less volume and less disruptive

Evolution #3 – Cable Companies Freely Peer with Content Companies • Content literally on the Cable Company Network • Lowest possible latency from content to eyeballs • Internet Gaming, Broadband Streaming, etc.

1998 -2000 Tier 1 ISPs $ Transit Tier 2 ISPs Thickness represents In-Group Peering Content

2000 -2003 Evolution #1 – Cable Companies Peering Tier 1 ISPs Cable. Cos Tier 2 ISPs Content

2000 -2003 Evolution #2 – Large Scale Network Savvy Content Peers Tier 1 ISPs Tier 2 ISPs LSNSC Content

2000 -2003 Evolution #3 – Cable Peers with Large Scale Network Savvy Content Tier 1 ISPs Cable. Cos Tier 2 ISPs LSNSC Content

International Dynamics – Separate White Paper JP U. S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet Region Japan Tier 1 ISPs are Tier 2 ISPs in the U. S. Internet Region

Q&A