Build vs Buy Internet Datacenter Operations Research INTAP

Build vs. Buy: Internet Datacenter Operations Research INTAP March 6, 2001 Tokyo

Agenda • • My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs. Buy Internet Datacenter” White Paper – 40 minutes Future: Where our Industry is heading – 5 minutes

NSFNET to Post-NSFNET Era Internet NSFNET (1987 -1994) Regional Tech Meetings NWNet Merit Oar. Net Nyser. Net NEARNet BARRNet NSFNET –Performance & Operations Regional –Reporting & Coordination SDSC –Routing & Network Planning Regional Sesqui NCAR Oar. Net NWNet Privatization (1994 -Now) North American Network Operators Group Regional Merit ANS Nyser. Net UUNet Sprint MCI (NANOG) Meetings AT&T Regional MY JOB: 5 Years as Chair of NANOG 12 Years in Internet…No Internet Preso… Regional

Internet Statistics • 87. 5% of all Internet statistics are made up. Source: I made it up.

Equinix, Inc. • Scale – size of Internet Business Exchanges, how many, where they are, future plans • “Virtual Fly Through” Tour of the IBX - 6 minutes

Industry Leading Security IBX™ Centers • 5 security checkpoints to customer equipment • State of the art video imaging and archiving • On site security personnel 24 x 365 IP/Carrier Knowledgeable Staff Customer Friendly Spaces – Conducting Business & Equipment Preparation • Designed from customer experience • Ergonomic work spaces Fully Redundant Backup Power • Built to 3 x power capacity • 24 x 365 availability • 3 levels of power backup • Customer troubleshooting • Forward looking design for growing industry • Smart Hands

IBX™ Growth Sites currently open or Under Development

Equinix Internet Business Exchange Model • Infrastructure Content Providers • Players • Neutrality… Neutral Internet Business Exchange Model Internet Service Providers Carriers

Carrier Internet Exchange Model • Infrastructure Carrier Operated Internet Exchange • Players Internet Service Providers Non Carrier-Neutral No Content Support Economic Distortions Carrier Operator

Job Title: Equinix Co-Founder & Chief Technical Liaison • So, my job is to take friends out for lunches and dinners… • Internet Operations Research & • Write White Papers • Summary: What did I learn?

No Internet Operations Documentation To make things more difficult, subtle language differences!

The Internet Operators: All about relationships

My Job: White Papers New Job Title: Equinix Co-Founder & Chief Technical Liaison Document Internet Operations Research – “Interconnection Strategies for ISPs” – “Internet Service Providers and Peering” – “Peering Case Studies” – “Peering Simulation Game” – “Build vs. Buy Internet Datacenter” today

White Paper: Interconnection Strategies for ISPs Direct circuits replaced with: • OC-12 into Exchange • Fiber cross connects Cost Drivers: • Big OC-12 Direct Circuit • Cheap Fiber Cross Connects • Aggregation Efficiency over OC-12 (2: 1, 3: 1) • Rack Fees Exchange Model Cost Formula: Exchange. Model. Cost=Big. Direct. Circuit()+(n-1)*x/2+Rack. Fees() OC-12 $23, 000/mo XC=$200/mo Rack=$1500/mo So does the Exchange-based model make sense?

White Paper: Internet Service Providers Transit & Peering ISP A $$ Transit ISP B Transit Peering $$ Peering West. Net Routing Table USNet East. Net

White Paper: Case Study: Peering in the Bay Area

White Paper: The Peering Simulation Game

White Paper: Build Vs. Buy Internet Data. Center • Modeling: • When does it make sense to build your own? • To Outsource? • Issues to consider • Economies of Scale? • Multiple Business Models

Service Level Agreements in Principle • Guarantee a higher level quality • Define Measurement Criteria for adhering to higher quality service Level • Meeting Performance Level critical to success of business • Failure to meet Service Levels penalty significantly worse than cost of service – e. Bay: $64, 000/minute of downtime

SLAs in Practice • No SLA Pricing – ISPPrice. NO SLA=X • SLA Pricing – Probability. Of. Missing. SLA=p – ISPPrice. SLA 1=X*(1+p) • Network does not typically change to meet new criteria • CIO Security Audit Story • Equinix Switch SLA: 99. 999% Uptime

SLA-Datacenters • • • State-of-the-art Datacenters Infrastructure Failure unlikely SLAs rare, but… Typically Limited to free month service (Insufficient to potential loss customer faces) • But what is an Internet Datacenter? • Include hosting? Include Internet Service? Include Caching, Content Distribution? • Models help Define and Frame the Problem…

SLA Datacenter Parameters • If there were Internet Datacenter SLAs • Parameters: – Installation Lead Time – Service Availability – 24/7 Customer Support – Provisioning/Responsiveness • Independent of Transport and Internet Service Issues.

Creating the Build vs. Buy Internet Datacenter Whitepaper • Create a community white paper on issues associated with the building & scaling of Internet Datacenters • Identify the issues that folks should consider when making the build vs. buy decision Today: • Demonstrate the document & models to date and get feedback NOTE WELL: • EARLY STAGE OF BVB WHITE PAPER (v 0. 4)! • This Model may be of help to you in your SLA Discussions

Toured 32 (now) data centers

Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour

Some with less secure fiber entry… Source: Don’t Ask.

Some Very Secure… http: //www. thebunker. net

Some Cat 5 Hurricane proof…

Some not quite cat proof. Loading Dock

Great religious debate over ties Velcro vs. Plastic cable ties vs. String

Time to white paper what I’ve seen & Facilitate BOF Discussion • Datacenter BOF I NANOG Oct ’ 98 Atlanta • Build vs. Buy White Paper • Internet Datacenter Model – Definition of Internet Datacenter • Focused discussion of Each Component of the Model

Internet Datacenter Model Dependence Relationship Increased Value

Customer Business Model Value Dependence

Internet Datacenter Model Example IXP Value Dependence

Size Ultimate Goal • Build Financial models that allow an ISP to say: • “Given our size requirements, it makes sense to build our own. ” Or • “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale. ” Unit cost + + = Buy Build

Flavor of the Real Estate Model • • • Place to put stuff Proximity to Fiber Space Size Utility Power Facilities Yard Scaling of Datacenter Real Estate? Increased Value Relationship

Scaling of the Real Estate Model $/sq. ft. Premium Real Estate Market (Tokyo, NYC, …) Real Estate Market (everywhere else) # Sq. Ft.

Real Estate Size is Important • Choose Too Small: Run out of space, lose economies of scale • Choose Too Big: Cost per Rack very high Vs. • Right of First Refusal (some Internet Datacenters allow this) • Expandability of Space

Commitment to Data. Center • 1 yr contract Vs. • 10 year lease Time to Market Issue – 6 – 9 months to build

Hidden Charges • Meet Me Rooms • Required products: – Must buy port on switch or certain volume of BW • Cross Connect Fees

Size & Finding Real Estate • 100, 000 sq ft in downtown DC? • 10, 000 sq ft in downtown DC • Vs. • Fiber Miles to Internet Datacenter

Flavor of the Infrastructure Model • Heating Venting Air Conditioning (HVAC) • Security Infrastructure • Power Infrastructure • Fire Suppression

Heating, Venting, Air Conditioning (HVAC) Infrastructure • Controlled Humidity ~45% • Too much Humidity: System Failure • Too Little Humidity: Static Electricity • Air Temperature: ~72 (IBM: 65 -85) • • Solutions: DX Systems Air Cooled Water-Based • How do these Scale?

HVAC Cost HVAC System Cost: Economies of Scale Direct Expansion HVAC Scaling $1, 000/ton Air Cooled $478/ton Chilled Water System $250/ton DX Systems: 25 -300 tons of Cooling ~ 10 K sq ft or ~150 -200 racks Terramark Outsource of HVAC to FPL… # Racks

Crossover point where Chilled Water lowest per-rack solution

Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tou

Security Infrastructure • 24 x 7 Manned • Access: Biometrics, card keys • Video Surveillance Sliding Glass

Power Infrastructure • Sun Netra 1 U=90 Watts – 40 U Rack=3600 Watts! • Power Source – Power Grid (dual feeds) – Co-Generation – UPS/CPS/Generators • Energy Storage – UPS Batteries – Flywheel (no UPS) – Fuel Cells (service) • Monitoring Issues • Redundancy Issues • TBD: Scaling Issues

Fuel Cells for Generators

Fire Suppression Infrastructure • Two Dominant Systems: • VESPA air sampling • Dry pipe, pre action water system vs. FM-200 • Smoke & Heat Required to activate region

Infrastructure Operations • Operations of HVAC, Security, Power systems • N+1 Redundancy

Internet Datacenter Model

Transport Service Model • Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc. • Business Model? – Owned by Carrier? Will others come in? – Not owned by Carrier? Will Carriers come in? • What type of services available? • How long to Provision? • Big Problem: Keeping up with BW Demand

Internet Datacenter Model

Internet Service Model • ISP(s), Aggregator(s) like Inter. NAP? • Business Model – Owned by ISP? Will others come in? – Not owned by ISP? Will others come in? • Capacity Issues • Latency Issues • Interconnection Strategies Allowed?

Internet Datacenter Model

Content Provider Model • Hoster Focus: – Installing New Servers, Provisioning Host Services, – Managing Server Farm, SLAs, Tech Support, 24/7 NOC, Back Office Solutions • Hoster owns Datacenter? – – Pro: Can buy Transit Services Con: Expensive w/Bandwidth Hungry Apps Con: Shared bandwidth issues (Yahoo! attack) Con: Local Loop Expensive • Business Model Content Distributor(s), Hoster(s)

Internet Datacenter Model

Future of the IXP? DSL Access Providers Wireless Access Svc Dial-in Modem Operators USENET Svcs Virtual E-Mail Svcs ISP Web. Host Svcs Search Engines Cache/Content Distribution Sparing Service Providers Internet Service Provider Carrier Carriers Market. Place: Stratification of component operations, fast creation & provisioning of packaged operations services Implication: IXP Content Sources, bandwidth demands

Future SLA Challenges 1. Bundling of Internet Operations Services… 2. Into Cohesive SLA-able services 3. Notion: Trickle-down SLA Bundle Participant Operations Agreements Financial Agreements Example: Bundle Internet Data. Center, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?

Bv. B Document Availability • All references are in the Bv. B white paper – Hyperlinks work in electronic version of document – Early Stage of document: Comments Welcome! • Send e-mail to wbn@equinix. com Subject: Build vs. Buy White Paper

Thanks to • Sean Donelan (Equinix) • Friends at Exodus, Frontier • Jason Colangelo (APC) Global Center, Level 3, etc. • Daniel Golding (Net. Rail) • Stephen Stuart/Phil Steffora (MFN) • Kenney Huang (Asia. Infra) • All Colo/ISPs tour • Sandip Gupta (Ensim) facilitators • Rob Glenn (Sun) • Jay Adelson (Equinix) • Everyone here! Please e • Josh Horn (NOTA) mail feedback. • Justin Newton (Net. Zero)