Optimising ASP/ISP Interconnections Panos Gevros University of Cambridge Computer Laboratory panos. gevros@cl. cam. ac. uk TAPAS project meeting - Dortmund, Germany 10 February 2003
outline Application & Internet Service Providers issues • similarities and differences in their operations, optimisation goals The network environment • topology, domains, routing, interconnection types, performance implications Interconnection choice : a common ISP & ASP problem: • Selecting which ISP(s) to interconnect with (involves facility location and ISP assessment aspects) Network design : an ISP-specific problem: • delay analysis and capacity provisioning
background & basic facts • • • For ASP/ISP network performance is crucial for delivering high quality services (offer outsourcing, move bits around). In network engineering there are two basic philosophies for introducing high performance (low delay, high bandwidth) Internet services, namely : • Qo. S / service differentiation i. e. giving preferential treatment to selected traffic flows using router mechanisms and/or new service models (Intserv, Diffserv). • bandwidth provisioning i. e. provide enough bandwidth so that the most stringent delay requirements are being met without differentiation between users. It is unlikely that a new Qo. S service model will replace the existing best-effort one For some time now the major players in the area base their operations on intelligent decisions about provisioning and interconnection of their networks However, the landscape in the area is far from clear…
Comparing ASP and ISP operations • An ASP should • connect to ISPs so that application service performance, fulfils the terms specified in the SLAs it has with its customers (in the absence of SLAs the perceived performance should be considered satisfactory). • establish presence in network places which are “close” to target customers (same metropolitan area, exchange etc. ) • The difference between an ISP and an ASP is that the ISP operation is generally more broad, • ISP customers are not only end-users, but also other network providers, • the ISP may act as a customer w. r. t some ISPs and as a provider w. r. t to others • • • An ISP and an ASP are similar in that their operations depends on the relationships they establish with other ISPs. An ISP in many cases operates as an ASP by bundling with the network connectivity offering, infrastructure services like DNS, Mail Relay, firewall, webcaching etc. The ISP operation goals will become clear after we discuss the interconnection environment.
The interconnection environment • Macroscopic view of the network Abstract view as 3 -tier hierarchy of providers Backbone (Tier 1) (e. g UUNET, C&W, Sprint) Regional (Tier 2) (national backbones, part leased from Tier 1’s) Access (Tier 3) (e. g. AOL) • Brief description of BGP routing • Types of relationships between ISPs transit (“customer – provider”) peering • The role of Internet Exchanges (IX-es)
Basic definitions • Internet : as an interconnection of Autonomous Systems (ASes): • Autonomous System (AS): a collection of networks under a single administrative authority distinguished by AS number (e. g an ISP is associated with one AS number). • Backbone: a network of geographically dispersed POPs (Points of Presence) interconnected with high capacity links, operated by a single administrative authority. • Point of Presence (POP) : a physical location where a collection of • • routers belonging to a single provider are used for the interconnection of its customers to the wide-area backbone. Route : an association between a destination network (network prefix) in the Internet and a next hop router towards that destination. (~112, 000 IP subnets in the global Internet routing table). BGP : Border Gateway Protocol, routing protocol operating at the borders of the providers’ networks (between neighbor AS-es) its operation determines the routes (paths taken by the traffic flows).
A (very) brief description of BGP Operation (1/) • BGP (Border Gateway Protocol) is a path vector protocol. De-facto standard inter-domain routing protocol • Operation: in-bound exchange of Route Advertisements between border routers • Route advertisement (rtadv) consists of • Destination network prefix (e. g. 192. 168. 0. 0/16), • next hop router IP address, • AS path info (list of AS#) • If a border router of AS-x sends a route advertisement for network N to a border router of AS-y this implies that AS-x accepts to forward packets with destination address an address in N on behalf of AS-y.
A (very) brief description of BGP Operation (2/) • AS-path info has dual functionality, • Loop detection • Route selection (shorter paths are preferred) • Inter-domain traffic engineering with BGP filters • Input filters select which rtadv will be accepted (example policy: accept a route advertisements if and only if the AS#s in the AS path info belong to “trusted” AS-es) • Output filters selects which routes are being advertised to its neighbors • Decision process: applies (configurable) route selection criteria • AS-path length, med, e. BGP>i. BGP, for tie-break keep oldest route) • Selected route is installed in forwarding table (for each packet does longest match prefix of its destination address and indicates outgoing interface).
Transit and Peering : definitions (1/) • Route advertisements are advertisements of value being offered and they are at the heart of the contracts between ISPs. Depending on the routes exchanged between two neighbor ISPs there are two types of (bilateral) relationship: • Transit is a “customer-provider” relationship between two ISPs in which one ISP (the provider) sells to the other ISP (the customer) connectivity to all its destinations that are available in its routing table. • Peering is the relationship in which the two ISPs mutually provide connectivity to their customers (only) without any money exchange.
Transit and Peering: example (2/) Suppose ISP - A is buying transit from ISP-D (denoted as A ->D) then ISP-D has the obligation of carrying A’s traffic to/from any Internet destination/source. Let also ISP-A has a peering relationship with ISP-B (A <p> B) and ISP-B in turn peers with ISP-C (B <p> C). Peering is a non-transitive relationship (i. e the routes of C’s customers are visible to B but they do not get propagated further to A, so traffic originating from a customer of A destined to a customer of C will not be routed via B instead it will be routed via ISPD from which ISP-A buys transit. In the same manner ISP-A’s routes do not get propagated to ISP-C. ISP C ISP 1 ISP B ISP A Customers
Transit and Peering: economic facts (1/) • Value and money flow in opposite directions • All ISPs (except the Tier-1 s) have to buy transit • Transit fees constitute the main operational costs. • Selling transit is necessary and desirable – (increases the number of your customers and they are the ones who pay) • However, ISP cannot survive by re-selling transit alone, – (the ISP becomes the unnecessary middleman which the customers bypass and connect directly to its upstream) • Can be a non-starter, depending on selling transit for value growth, while value is a pre-requisite for attracting customers and selling transit …
Transit and Peering: economic facts (2/) • Reason why peering becomes important: • it is an economic optimisation over buying transit, reduces average per-bit costs. • creates value growth (increases the amount of bandwidth the ISP has to sell to its customers) • Parties in a peering relationship perceive that the cost of peering (fixed costs plus carrying peers traffic) is offset • Tier-1 s tend to peer only between themselves, • do not want to commoditise IP – and prefer to compete on the basis of “better service” • do not have incentives to improve the performance of smaller ISPs enabling them to compete against them in the market.
Peering (+)s • Reduces transit costs • buying transit costs 10 ths or 100 ths thousands $$/month • Peering is low cost, provides direct traffic exchange, reduces the load on expensive transit links. • Reduces latency (end-to-end delay) • Direct interconnection to a competitor ISP nearby provides lower delay compared to traversing several transit providers. • Increases demand for bandwidth • High packet loss rates and high delays reduce bandwidth consumption (adaptive TCP!) and therefore revenues. This can be a problem (reducing revenues) with usage based charging.
Peering (-)s • Peering is not free – Router ports, circuits (to the exchange), man hours, cost money, lengthy negotiations • Assymetric traffic – When Traffic ratio across peering boundary (outbound/inbound) exceeds certain limits then the peering may no longer be seen as beneficial to one of the parties involved (in practice ratios up to 4: 1 can be found and still be considered acceptable). • Lack of SLAs – Peering relationships unlike “customer” relationships are not followed by SLA “guarantees” for reliability, rapid repair of problems, or penalty when a connection performs poorly. – Side-effect: determining the performance of an end-to-end path becomes unpredictable when the path traverses several peering boundaries (public internet exchanges) which tend to be congested.
Internet Exchanges (IXes) (1/2) • Exchange is essentially the “marketplace” where ISPs peer or buy/sell transit (first IXes were only for peering). • IX-es offer cost savings by aggregating traffic from many neighbours on one link (the link into the exchange). (downside increased risk due to a single point of failure) • Today IXes tend to differentiate towards transit-only or peering only.
Internet Exchanges (2/2) • An IX it should satisfy certain criteria for ISP participation: access issues, operational, cost, reputation, viability, neutral operation, opportunities (for selling transit or for peering) – Neutrally operated • not aligned with specific carriers or ISPs which may impose restrictions on the participants as to where they should be buying access circuits from etc. – The cost of using the exchange • Rack fees, port fees, cross connect fees, installation fees – Reputation / Credibility / financial viability • Will the exchange continue to exist in the future • ISPs tend to be “risk averse” ; prefer established, well-populated exchanges (and are willing to pay an “insurance” premium, instead of landing in an untested IX) • ISP min-conf : presence two transit and one peering IX-es
Service Provider Interconnection decisions • ASP should access the network so that its customers perceive a service that is fast and reliable. • ISPs (Tier-2, 3) face a somewhat similar problem. • Commonality : the importance of location it determines the quality of So the questions faced by both service providers (ISPs/ASPs ) are I. to which locations, POPs, should we buy links into (“where should we land capacity “), II. which other ISPs should we interconnect with, III. under which terms we should be interconnecting with a certain ISP. No–single correct answer, – depend on provider; customer base, growth plans, budget limitations, (regulatory framework in which it operates) etc. General principles apply (usually)
The ASP interconnection • Depending on the services offered by the ASP there are different strategies, 1. 2. 3. • Before choosing an ISP the ASP should evaluate the candidate, criteria involve • • Co-locate close to the customers Select the same access ISPs as the target customer group Act as an advisor, assess performance/connectivity offered by different ISPs to the ASP site -- advise customers as to which ISPs provide “better” access the service. Network capacity number of peering arrangements information about peering policies (link capacities, upgrade procedures, presence of SLAs) Improve connectivity, reduce risk • • multihoming (use >1 upstream ISPs) use redundancy (have multiple links to the same ISP, possibly using BGP policies for load balancing).
ASP interconnection ASP ISP ISP customer ISP ISP ISP
solution 1: strategic co-location (possibly) VPN ASP ISP ISP customer
solution 2 : common access ISPs ASP ISP ISP customer
solution 3 : issue ISP recommendations ASP ISP ISP customers
ISP interconnection (1) • • • Interconnection decisions based on intuition and higher-level business considerations (strategic partners, alliances), often corporate ties often override technical justification. On the technical side the real AS path is selected by BGP using shortest path (myopic - ignores amount of traffic and cost – there is space for optimisation there). Decisions at this point important for the viability of the business Pattern : ISP min-conf : presence two transit and one peering IX-es transit usually straightforward – (cash issue, small number of players) • peering more complicated (reduce traffic on expensive transit links – – identify targets (IX-es, ISPs) assesment, (monitoring (i/o) traffic flows, Netflow data from routers) negotiation, (peering policy of the identified “target” determine the outcome) Key Question : “How much traffic are we sending to or through the AS’s that participate in the exchange? ”
ISP interconnection (2) • Criterion: Exchanged Traffic Volume between AS-es, distinguish between – Transit traffic passes through an AS – Terminating traffic sinks in an AS • • • create AS rankings based on “transit/terminating traffic scores”. Assign weights to the AS-es based on their distance from the destination subnets. Find those AS that appears to be central in most popular paths and try to peer with those AS!
ISP interconnection (3) • More tangible evaluation criteria – The ownership of the links ; determines the quality of the offered service. When the ISP owns the links it has full control over potential problems. – The physical and link layer technologies deployed – compatibility of the technologies and their intrinsic value (i. e. state-of-the art versus outdated transmission technologies) can determine the reliability of the service, how easy/fast it will be to schedule future capacity upgrades (e. g light-up more fiber is easy) etc. – The diversity of the physical routes – physical path separation inside the core provider’s network is important so that customers’ mission critical traffic is not vulnerable to outages or single points of failure. Sometimes the logical topology maps can be deceptive as they may show route diversity but the underlying physical links are not diverse. Also in the metropolitan area where the links terminate there should be POPs in different physical locations (buildings).
Decision-making framework (1) • The minimal ISP reference model – buy Transit from two other ISPs – Peer with as many ISPs as possible (meaningful) – sell transit to customers • Create an optimal “portfolio” of peering and transit interconnections.
Decision-making framework (2) Inputs: • transit traffic vector • terminating traffic vector • cost of peering with an AS (infinite if not available) • cost of buying transit from an AS • cost of peering with an exchange Algorithm for determining the threshold for participating in an exchange Peering provides redundancy reduces connectivity risk • Prob[ dest-i unreachable ] • weight dest-i based on traffic transitting through or terminating to it
Decision-making framework (3) Criteria for the selection of an upstream provider: • • • geographic proximity (access circuit cost) Network reliability (uptime) Pricing (unit bandwidth cost) Customer support Value added services (etc. . ) Optimisation Algorithm (most likely heuristic)
Network design Two problems: – capacity assignment for delay minimisation – assumes delay analysis
Delay Analysis (1/) • Model: N-nodes, M-links, capacities Ci , i=1, . . , M measured in packets/sec (assumption all packets have the same length 1/ ) • Poisson process with mean rate i, j (packets/sec) external traffic entering (and leaving – i. e. no termination inside the network) node -j network node -k
Delay Analysis (2/) • Link in the network is modelled as an M/M/1 “server with a queue” the arrival process of the packets at link-i is I • Let Zj, k is the expected delay for a packet entering the providers network at node j and exiting at node k, • Let T be the average (expected) delay a packet will experience using the network Decompose T in sum of link delays Ti be the average delay a packet incurs at link-i (waiting time in the queue plus transmission time).
Delay Analysis (3/) Where link delays Ti are given by standard M/M/1 results (arrival rate i and service times 1/ Ci )
Capacity Assignment (1/) • • The provider knows • the traffic (rate bit/s) on each of the links { i }, • the topology and the routing, The goal is to minimise the mean delay T incurred by a packet crossing the network where the minimisation happens with respect to the link capacities { Ci } subject to total cost constraints. The cost incurred by the network provider to install a link of capacity Ci is given by a function d(Ci) (linear, logarithmic, follow a power law etc. depending on the physical length of the link, the technology etc). Obviously Ci > i / and the optimisation problem becomes
Capacity Assignment (2/) • The problem is solved for linear cost function d( Ci ) = di Ci and the yields Where Dx = D - i di/ is the excess cost ( i / is the minimum required amount) and n is the average path length defined as n = /
Summary • • • ISPs and ASPs face similar interconnection problems, new economic models for the ISP industry focus on user control on the edges and less concern about extracting rent from content hosting. There has been no systematic way to approach these interconnection problems There is need for new tools and models to assist decision making in the peering/transit arena, facility location These will help the ISP to manage both their interconnection costs and the performance of their network then they will be able to “standardise” offered services and possibly provide meaningful SLAs to the customers. http: //www. cl. cam. ac. uk/~pg 281/tapas/
Скачать презентацию Optimising ASP ISP Interconnections Panos Gevros University of Cambridge
cfdcabe8ecdc559fa4a30eb86dc6fad9.ppt