- Количество слайдов: 62
Build vs. Buy: Internet Datacenter Operations Research INTAP March 6, 2001 Tokyo
Agenda • • My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs. Buy Internet Datacenter” White Paper – 40 minutes Future: Where our Industry is heading – 5 minutes
NSFNET to Post-NSFNET Era Internet NSFNET (1987 -1994) Regional Tech Meetings NWNet Merit Oar. Net Nyser. Net NEARNet BARRNet NSFNET –Performance & Operations Regional –Reporting & Coordination SDSC –Routing & Network Planning Regional Sesqui NCAR Oar. Net NWNet Privatization (1994 -Now) North American Network Operators Group Regional Merit ANS Nyser. Net UUNet Sprint MCI (NANOG) Meetings AT&T Regional MY JOB: 5 Years as Chair of NANOG 12 Years in Internet…No Internet Preso… Regional
Internet Statistics • 87. 5% of all Internet statistics are made up. Source: I made it up.
Equinix, Inc. • Scale – size of Internet Business Exchanges, how many, where they are, future plans • “Virtual Fly Through” Tour of the IBX - 6 minutes
Industry Leading Security IBX™ Centers • 5 security checkpoints to customer equipment • State of the art video imaging and archiving • On site security personnel 24 x 365 IP/Carrier Knowledgeable Staff Customer Friendly Spaces – Conducting Business & Equipment Preparation • Designed from customer experience • Ergonomic work spaces Fully Redundant Backup Power • Built to 3 x power capacity • 24 x 365 availability • 3 levels of power backup • Customer troubleshooting • Forward looking design for growing industry • Smart Hands
IBX™ Growth Sites currently open or Under Development
Equinix Internet Business Exchange Model • Infrastructure Content Providers • Players • Neutrality… Neutral Internet Business Exchange Model Internet Service Providers Carriers
Carrier Internet Exchange Model • Infrastructure Carrier Operated Internet Exchange • Players Internet Service Providers Non Carrier-Neutral No Content Support Economic Distortions Carrier Operator
Job Title: Equinix Co-Founder & Chief Technical Liaison • So, my job is to take friends out for lunches and dinners… • Internet Operations Research & • Write White Papers • Summary: What did I learn?
No Internet Operations Documentation To make things more difficult, subtle language differences!
The Internet Operators: All about relationships
My Job: White Papers New Job Title: Equinix Co-Founder & Chief Technical Liaison Document Internet Operations Research – “Interconnection Strategies for ISPs” – “Internet Service Providers and Peering” – “Peering Case Studies” – “Peering Simulation Game” – “Build vs. Buy Internet Datacenter” today
White Paper: Interconnection Strategies for ISPs Direct circuits replaced with: • OC-12 into Exchange • Fiber cross connects Cost Drivers: • Big OC-12 Direct Circuit • Cheap Fiber Cross Connects • Aggregation Efficiency over OC-12 (2: 1, 3: 1) • Rack Fees Exchange Model Cost Formula: Exchange. Model. Cost=Big. Direct. Circuit()+(n-1)*x/2+Rack. Fees() OC-12 $23, 000/mo XC=$200/mo Rack=$1500/mo So does the Exchange-based model make sense?
White Paper: Internet Service Providers Transit & Peering ISP A $$ Transit ISP B Transit Peering $$ Peering West. Net Routing Table USNet East. Net
White Paper: Case Study: Peering in the Bay Area
White Paper: The Peering Simulation Game
White Paper: Build Vs. Buy Internet Data. Center • Modeling: • When does it make sense to build your own? • To Outsource? • Issues to consider • Economies of Scale? • Multiple Business Models
Service Level Agreements in Principle • Guarantee a higher level quality • Define Measurement Criteria for adhering to higher quality service Level • Meeting Performance Level critical to success of business • Failure to meet Service Levels penalty significantly worse than cost of service – e. Bay: $64, 000/minute of downtime
SLAs in Practice • No SLA Pricing – ISPPrice. NO SLA=X • SLA Pricing – Probability. Of. Missing. SLA=p – ISPPrice. SLA 1=X*(1+p) • Network does not typically change to meet new criteria • CIO Security Audit Story • Equinix Switch SLA: 99. 999% Uptime
SLA-Datacenters • • • State-of-the-art Datacenters Infrastructure Failure unlikely SLAs rare, but… Typically Limited to free month service (Insufficient to potential loss customer faces) • But what is an Internet Datacenter? • Include hosting? Include Internet Service? Include Caching, Content Distribution? • Models help Define and Frame the Problem…
SLA Datacenter Parameters • If there were Internet Datacenter SLAs • Parameters: – Installation Lead Time – Service Availability – 24/7 Customer Support – Provisioning/Responsiveness • Independent of Transport and Internet Service Issues.
Creating the Build vs. Buy Internet Datacenter Whitepaper • Create a community white paper on issues associated with the building & scaling of Internet Datacenters • Identify the issues that folks should consider when making the build vs. buy decision Today: • Demonstrate the document & models to date and get feedback NOTE WELL: • EARLY STAGE OF BVB WHITE PAPER (v 0. 4)! • This Model may be of help to you in your SLA Discussions
Toured 32 (now) data centers
Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour
Some with less secure fiber entry… Source: Don’t Ask.
Some Very Secure… http: //www. thebunker. net
Some Cat 5 Hurricane proof…
Some not quite cat proof. Loading Dock
Great religious debate over ties Velcro vs. Plastic cable ties vs. String
Time to white paper what I’ve seen & Facilitate BOF Discussion • Datacenter BOF I NANOG Oct ’ 98 Atlanta • Build vs. Buy White Paper • Internet Datacenter Model – Definition of Internet Datacenter • Focused discussion of Each Component of the Model
Internet Datacenter Model Dependence Relationship Increased Value
Customer Business Model Value Dependence
Internet Datacenter Model Example IXP Value Dependence
Size Ultimate Goal • Build Financial models that allow an ISP to say: • “Given our size requirements, it makes sense to build our own. ” Or • “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale. ” Unit cost + + = Buy Build
Flavor of the Real Estate Model • • • Place to put stuff Proximity to Fiber Space Size Utility Power Facilities Yard Scaling of Datacenter Real Estate? Increased Value Relationship
Scaling of the Real Estate Model $/sq. ft. Premium Real Estate Market (Tokyo, NYC, …) Real Estate Market (everywhere else) # Sq. Ft.
Real Estate Size is Important • Choose Too Small: Run out of space, lose economies of scale • Choose Too Big: Cost per Rack very high Vs. • Right of First Refusal (some Internet Datacenters allow this) • Expandability of Space
Commitment to Data. Center • 1 yr contract Vs. • 10 year lease Time to Market Issue – 6 – 9 months to build
Hidden Charges • Meet Me Rooms • Required products: – Must buy port on switch or certain volume of BW • Cross Connect Fees
Size & Finding Real Estate • 100, 000 sq ft in downtown DC? • 10, 000 sq ft in downtown DC • Vs. • Fiber Miles to Internet Datacenter
Flavor of the Infrastructure Model • Heating Venting Air Conditioning (HVAC) • Security Infrastructure • Power Infrastructure • Fire Suppression
Heating, Venting, Air Conditioning (HVAC) Infrastructure • Controlled Humidity ~45% • Too much Humidity: System Failure • Too Little Humidity: Static Electricity • Air Temperature: ~72 (IBM: 65 -85) • • Solutions: DX Systems Air Cooled Water-Based • How do these Scale?
HVAC Cost HVAC System Cost: Economies of Scale Direct Expansion HVAC Scaling $1, 000/ton Air Cooled $478/ton Chilled Water System $250/ton DX Systems: 25 -300 tons of Cooling ~ 10 K sq ft or ~150 -200 racks Terramark Outsource of HVAC to FPL… # Racks
Crossover point where Chilled Water lowest per-rack solution
Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tou
Security Infrastructure • 24 x 7 Manned • Access: Biometrics, card keys • Video Surveillance Sliding Glass
Power Infrastructure • Sun Netra 1 U=90 Watts – 40 U Rack=3600 Watts! • Power Source – Power Grid (dual feeds) – Co-Generation – UPS/CPS/Generators • Energy Storage – UPS Batteries – Flywheel (no UPS) – Fuel Cells (service) • Monitoring Issues • Redundancy Issues • TBD: Scaling Issues
Fuel Cells for Generators
Fire Suppression Infrastructure • Two Dominant Systems: • VESPA air sampling • Dry pipe, pre action water system vs. FM-200 • Smoke & Heat Required to activate region
Infrastructure Operations • Operations of HVAC, Security, Power systems • N+1 Redundancy
Internet Datacenter Model
Transport Service Model • Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc. • Business Model? – Owned by Carrier? Will others come in? – Not owned by Carrier? Will Carriers come in? • What type of services available? • How long to Provision? • Big Problem: Keeping up with BW Demand
Internet Datacenter Model
Internet Service Model • ISP(s), Aggregator(s) like Inter. NAP? • Business Model – Owned by ISP? Will others come in? – Not owned by ISP? Will others come in? • Capacity Issues • Latency Issues • Interconnection Strategies Allowed?
Internet Datacenter Model
Content Provider Model • Hoster Focus: – Installing New Servers, Provisioning Host Services, – Managing Server Farm, SLAs, Tech Support, 24/7 NOC, Back Office Solutions • Hoster owns Datacenter? – – Pro: Can buy Transit Services Con: Expensive w/Bandwidth Hungry Apps Con: Shared bandwidth issues (Yahoo! attack) Con: Local Loop Expensive • Business Model Content Distributor(s), Hoster(s)
Internet Datacenter Model
Future of the IXP? DSL Access Providers Wireless Access Svc Dial-in Modem Operators USENET Svcs Virtual E-Mail Svcs ISP Web. Host Svcs Search Engines Cache/Content Distribution Sparing Service Providers Internet Service Provider Carrier Carriers Market. Place: Stratification of component operations, fast creation & provisioning of packaged operations services Implication: IXP Content Sources, bandwidth demands
Future SLA Challenges 1. Bundling of Internet Operations Services… 2. Into Cohesive SLA-able services 3. Notion: Trickle-down SLA Bundle Participant Operations Agreements Financial Agreements Example: Bundle Internet Data. Center, Transport, Internet, MSP. What happens when Transport service fails? Who covers the cost of the SLA?
Bv. B Document Availability • All references are in the Bv. B white paper – Hyperlinks work in electronic version of document – Early Stage of document: Comments Welcome! • Send e-mail to [email protected] com Subject: Build vs. Buy White Paper
Thanks to • Sean Donelan (Equinix) • Friends at Exodus, Frontier • Jason Colangelo (APC) Global Center, Level 3, etc. • Daniel Golding (Net. Rail) • Stephen Stuart/Phil Steffora (MFN) • Kenney Huang (Asia. Infra) • All Colo/ISPs tour • Sandip Gupta (Ensim) facilitators • Rob Glenn (Sun) • Jay Adelson (Equinix) • Everyone here! Please e • Josh Horn (NOTA) mail feedback. • Justin Newton (Net. Zero)