Inter-Peer NOC Communication Mike Hughes mike linx net

Inter-Peer NOC Communication Mike Hughes mike@linx. net

Scene Setting: Straw Poll n Who here in this room does peering?

Scene Setting: Straw Poll n Who here in this room does peering? n Have you ever had issues resolving problems with your peerings? – Difficulties contacting peers, finding the right contact, communication problems?

Scene Setting: Straw Poll n Who here in this room does peering? n Have you ever had issues resolving problems with your peerings? n Do you maintain a local db of contacts? – Why? Issues with freshness of data?

Scene Setting: Straw Poll n Who here in this room does peering? n Have you ever had issues resolving problems with your peerings? n Do you maintain a local db of contacts? n When a peer needs to talk to you, where does their call/email arrive? – Main NOC contact? Dedicated peering contact? “Customer Care”?

Scene Setting: Straw Poll n Who here in this room does peering? n Have you ever had issues resolving problems with your peerings? n Do you maintain a local db of contacts? n When a peer needs to talk to you, where does their call/email arrive? n Some names have been changed to protect the innocent… and guilty…

Why do you go peering? n Long term money savings n Less Transit n Lower latency, better performance n Traffic Control n Diversity, Reliability n Presence …and so on…

Where’s the problem? n Poor inter-peer communication seems to be common – Friendly IX operator called in to “mediate” n Communication hitting the wrong place – Customer NOCs – IX Operator – IP address maintainer (e. g. whois contact)

Identifying the right contact n Sources of information: – – Whois queries to databases IXP-maintained NOC and Peering contact db Internal databases Third-party voluntary databases • http: //puck. nether. net/netops list • peeringdb. com n All above are vulnerable to information “rot”

How to drive RIPEdb/RA, etc n Some really subtle differences in the implementations – RIPE expects “AS” before an AS number! Which contacts are useful n Which objects to look up n – Like the Peer ASN, not the Peer IP address! n Why can’t ASN be logged in adjacency changes on routers? – This seems to drive IP-based lookups

Drive the Data Sources Properly! n Example: using WHOIS queries n “Oh, I have an outage on WAIX, I’ll look up the IP address” $ whois -h whois. arin. net 198. 32. 212. 11|less … Org. Name: Exchange Point Blocks … RTech. Handle: WM 110 -ARIN RTech. Name: Manning, Bill RTech. Phone: +1 -310 -322 -8102 RTech. Email: bmanning@karoshi. com

Bad Data Enters the System n “Okay, I’ll phone Bill Manning” – But all Bill did was give WAIX some v 4 space – Bill doesn’t run WAIX, and isn’t an operational contact for WAIX So, Bill either ignores your voicemail, or tells you to call someone else n Whatever – it’s added delay, increased frustration – it’s how not to do it n

Driving Whois Properly n Always lookup the PEER ASN – Not the IP address! – It’s a BGP problem, we use ASNs in BGP $ whois -h whois. ra. net AS 3856|less aut-num: AS 3856 as-name: UNSPECIFIED descr: Packet Clearing House www. pch. net admin-c: Bill Woodcock tech-c: Bill Woodcock remarks: peering@pch. net, +1 866 BGP PEER

Driving Whois Properly n Always lookup the PEER ASN – Not the IP address! – It’s a BGP problem, we use ASNs in BGP $ whois -h whois. ra. net AS 3856|less aut-num: AS 3856 as-name: UNSPECIFIED descr: Packet Clearing House www. pch. net admin-c: Bill Woodcock tech-c: Bill Woodcock remarks: peering@pch. net, +1 866 BGP PEER

So you’ve found the contact n How do they respond to you? – Confusing recursive call trees? – Recalcitrant ticketing systems? – First-line NOC – “Is it switched on? ” – “You’re not a customer, go away” n Once negotiated, peering is an engineering relationship – So backbone ops, not “customer care”

Expectations of Peer Contacts n Choose your points of contact carefully n Big problems with – What’s peering/BGP/WAIX? – Are you a customer? – What’s your circuit ID? – Go away, you aren’t a customer n All serious no-no’s – be peers! nice to your

PCH INOC-DBA Phones n PCH operate a “dial by ASN” NOC hotline system – They run the SIP registry/proxy – “Bring your own” SIP compliant phone n The idea is that it should get through to someone clueful – No call-trees, no music-on-hold n http: //www. pch. net/inoc-dba/

Suggested Role Contacts n Peering@ – For setting up new peerings, changing existing ones, no 24 x 7 expectation – Shouldn’t go to exclusively to sales@ ; -) n NOC@ – Reaches your 24 x 7 NOC, which is either BGP friendly and has enable, or knows when, how and where to escalate n Support@ – Is generally your “customer-care”/call center

Getting the message across n Okay, so you’ve made contact – Now, make your point n Provide the peer with useful information – Start with the subject line – Be informative, who, when, what – Messages like “Help” and “Peering down” aren’t helpful

How not to do it… -----Original Message----From: Joe Schmoe Sent: Wednesday, January 25, 2006 5: 41 PM Subject: Maintenance Notification Dear Peers, … n Where? How does it affect me? – All detail buried in wordy message body n When? No TZ stamp! n Help me handle my huge NOC inbox!

Example: Useful Subject Headers AS 7132’s preferred subject line format: - - - - - Example subject line: Equinix-Ashburn - RCN/6079 - SBC/7132 - new session turn-up - 29 - Mar-06 - 9: 45 am EST Thanks to Ren Provo

Look clueful Subject: Traffic Drop Dear Peer, We suddenly noticed a 300 Mb drop in traffic on our connection to the PIE-IX. Can you investigate, and help us find where the traffic has gone? Regards, … n What does this say about your peer? – Don’t you think they look silly? n Run tools to help you answer these questions yourself – Netflow, MAC accounting, etc.

How to escalate Check your equipment first n Ask your peer - “What’s up? ” n – Often you can resolve a problem bi-laterally n Go to the IX only if you need to – Not all IX operators can provide a 24 x 7 contact n When to escalate a customer fault – Don’t stonewall customer reports – Don’t point them to the IX operator – Co-ordinate directly with your peers

How the IXP Op can help n Provide an up-to-date list of IX participants and their NOC/Peering contact information – Usually password protected n Help break comms deadlock – Help fix “dead ends” n Otherwise, they can only help with “physical” problems – “link down”, packet loss, broken cables, packet corruption to all destinations connected to the IXP

In Summary n Keep your own information up to date – Whois db objects, third party dbs n Make sure your peering and NOC contacts are appropriate – No-one likes call-trees and holding n Find the right contacts at your peers n Be nice to your peers!

Thanks n mike@linx. net