A victim-centric peer-assisted framework for monitoring and troubleshooting

A victim-centric peer-assisted framework for monitoring and troubleshooting routing problems

How to Monitor? Four schemes How to monitor? Pros Cons Monitor devices such as routers No overhead Device status does not directly translate into user perceived performance Monitor BGP updates No overhead; Know what happens in other network Do not see some data-plane anomaly Monitor flowlevel traffic No overhead; Real traffic; Witness direct impact of failures Do not witness failures directly Active probing Extra overhead; May Witness direct impact of failures not mimic the real traffic

What Constraint to Monitor? l Network meets ISP’s goals l l Resource utilization Routing goes as specified by policy … Network meets users’ goals l l l Reachability l Most fundamental end-to-end property l Easy to define and formulate Delay, loss l Less easier to define and formulate Application level: Bulk transfer, VOIP l Depends on reachability, delay, loss, etc

Our Monitor Scheme l Monitor reachability using active probing l l Focus on reachability Use ping – no need for remote cooperation Trade off between probing efficiency and probing coverage (challenges) Disclaimers l l Do not monitor delay or loss Do not consider ISP’s goals

Troubleshooting -Next Step to Monitoring l Goal of troubleshooting l l l Localize the root cause How local? Depends on the nature of the cause Purpose of troubleshooting l Local root cause l l Pin-point the problem and fix it Remote root cause l l Contact the responsible networks to solve the problem By-pass the faulty network

Localize the Root Cause Topology dimension AS 1 a->b b->c c->d AS 2 m->n m->l n->l AS 3 x->y y->z z->x Firewalls (those who prevent forwarding) Forwarding paths (those who do forwarding) Control plane Physical and link layers Protocol dimension Localize the cause Link level at AS level Localize the cause at protocol AS and protocol level Both level

Troubleshooting: Three building blocks l Tool l l Data: generated by tool l l traceroute, ping, netflow, looking glass, etc e 2 e reachability, BGP updates, traffic profile, etc Brain: the intelligent part, usually network operator l l Digest the data, make inference, leverage dependency, draw from past experience The key of troubleshooting. Hard problem

What Can We Do to Improve? l Improve the tool l l Promote the cooperation among networks Traceroute -> resilient remote traceroute BGP feed -> resilient remote BGP feed Improve the automation of brain l Unify previous work

Automatic Brain l It’s a challenging problem l l l Fault may occur at multiple levels Involve machine learning Example work: Enterprise network services, sigcomm’ 07, by Paramvir Bahl et al.

Dependency Graph Approach l l Decompose a large system into components Infer the dependencies among components l l l A set of observations on some components l l A depends on B: If B fails, A fails Lead to a hierarchy of dependencies: dependency graph (like Makefile) For example, F, H, X works but G fails Infer the status of other components using dependencies, finally locate the root cause component

Dependency Graph Example 1 Multi-tier dependency graph. Diagnoses multi-level fault but needs automated construction. [ From Paramvir Bahl et al, sigcomm’ 07 ]

Dependency Graph Example 2 Flat dependency graph. Diagnoses simple fault. [From Ramana Kompella et al, infocom’ 07 ]

Trade-off in Decomposition l l The granularity of decomposition determines the how specific the troubleshooting is Fine-grained decomposition l l l Advantage: more specific Disadvantage: graph is more complex, constructing and solve it is challenging Coarse-grained decomposition l l Advantage: graph is simple, constructing and solving it is less challenging Disadvantage: less specific

Dependency Graph Regarding Internet Routing A B p can ping q A depends on B p can send packets to q q can send packets to p Forwarding path p->q is OK Physical path p->q is OK Link u_i->u_{i+1} is up …… Control plane info is correctly propagated AS N_i has correct route N_{i+1} AS N_i imports routes of prefix p Path p->q before failure: IP hops: u_0, u_1, …, u_n, AS hops: N_0, N_1, …. , N_m

Dependency Graph Regarding Internet Routing (cont. ) l Account for three common root causes l l Link/router failure Router misconfiguration leading to missing route (i. e. does not import route) Router misconfiguration or attack leading to prefix hijacking Topology-wise locate the root cause, and also tell among the three root causes l Reasonably specific

Recent Work on Network Troubleshooting l Infocom’ 07, Detection and Localization of Network Black Holes, by Ramana R. Kompella et al l l Co. Next’ 07, Net. Diagnoser: Troubleshooting network unreachabilities using end-to-end probes and routing data, by Amogh Dhamdhere et al l l Automate the “brain”. Consider both physical failure and control plane fault. For inter-domain. Flat dependency graph. Sigcomm’ 07, Automating Cross-layer Diagnosis of Enterprise Wireless Networks, by Cheng et al l l Automate the “brain”. Consider only physical failure. Mainly for intra-domain. Flat dependency graph. Improving the “tool”. Measure and infer various delays in a wireless environment Sigcomm’ 07, Towards Highly Reliable Enterprise Network Services Via Inference of Multi-level Dependencies, by Paramvir Bahl et al l Automate the “brain”. Mainly for enterprise network and services. Deal with multi-level faults. Automatically generate multi-tier dependency graph.

Net. Diagnoser: Overview l l Troubleshooting unreachability Fault assumption: l l l Link failure, router misconfiguration causing partial link failure (in particular BGP export filter misconfiguration) Deal with filtered traceroute More comprehensive than previous work Infrastructure: sensors, all pair-wise traceroute Mechanisms: l l l Binary tomography Per-neighor-basis logical link modeling control plane Combining BGP withdraw message

Net. Diagnoser: Logical Links

Netdiagnoser: Dependency Assumption P can send packets to q Forwarding path p->q is OK Physical path p->q is OK Link u_i->u_{i+1} is up A B Control plane info is correctly propagated AS N_{i+1} exports prefix q to AS N_i A depends on B P->q: IP hops: u_0, u_1, …, u_n, AS hops: N_0, N_1, …. , N_m