Inter-domain Multicast in European Research Networking TEN-155 Operational

Inter-domain Multicast in European Research Networking: TEN-155 Operational Experience and Deployment on GÉANT NANOG 24 Miami 12 th February 2002 Agnes Pouélé, DANTE Ltd. Network Engineer Jan Novak, Cisco Systems Inc. Network Consulting Engineer Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 1

Agenda • DANTE • TEN-155 Operational Experience – Evolution of the TEN-155 multicast topology from 1998 to 2000 – Operational Experience • GÉANT deployment – Overview of GÉANT Network – GÉANT’s Multicast design – GÉANT Multicast and Unicast Coverage – GÉANT Multicast Service and Monitoring • Conclusion Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 2

DANTE, TEN-155, GÉANT • DANTE is a not-for-profit company set up in 1993 by European National Research Network organizations. • TEN-155 • was an ATM based network built initially on OC 3 links and then upgraded in 2000 • GÉANT • successor of TEN-155 • launched in December 2001 • 10 Gbps Pan-European Network Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 3

Part I TEN-155 Operational Experience Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 4

Starting point MBONE TUNNELED TOPOLOGY Typical mess of DVMRP tunnels on Sun WSs with usual tunnel routing problems. Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 5

1999: PIM-SM and DVMRP set-up PIM-SM domain OSLO IETF transmitted over both native STM-1 ATM based network and DVMRP tunnelled infrastructure Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 6

Multicast deployment in TEN-155 • Based on this first set-up and successful operation, we concluded to enable multicast on all production routers in TEN-155. • Multicast code stable, CPU usage ok, M-BGP ok • PIM-SM “only” for production service • PIM-SM to DVMRP border works, but not possible to operate routinely (NOC) • From 1999 to 2000 each country was migrated from the DVMRP cloud to a interconnection with TEN-155 using PIM-SM/MBGP/MSDP Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 7

2000: TEN-155 final topology Belgium Netherlands Nordics Israel Czech Rep NL Germany SE external Spain AT & T UK AS 8933 OSPF + internal MBGP FR tunnel dedicated mcast unicast/mcast line IT DE AT CH ABILENE US UUNET Hungary BGP Italy Poland external France Greece external United Kingdom Portugal Luxembourg Switzerland Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) Slovenia 8

TEN-155 Operational Experience Performance Impact • Parameters to be considered for the exploitation of Multicast – CPU - parameters • about 1600 forwarding (mroute) entries • max 8 outgoing interfaces (average 2 -3) • max 20 Mbit/s of data forwarded by one router resulting in 5 -10% of CPU increase (mainly PIM) Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 9

TEN-155 Operational Experience Performance Impact • Parameters to be considered for the exploitation of Multicast – Memory • about 3000 SA messages in the cache • about 1600 forwarding entries (mroute) • about 10 000 routes in the MBGP table • These parameters didn’t represent a significant memory usage Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 10

Usage of MSDP – new component – new storms Group A MSDP peers default peer • Default peer in redundant topology Group C Group B • “Redundant” mesh-groups Impact of a MSDP storm on the CPU load Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 11

Usage of MSDP – early implementation problems • Cisco IOS 12. 0. 6 S and lower – ghost SA entries in the MSDP cache • SA messages recreated by the incoming (S, G) joins • FIXED – Origination of SA messages only when source registered to the originating RP – “ip msdp redistribute” command – without arguments • re-originated all known SAs • caused huge increase of the SA counts worldwide • FIXED Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 12

TEN-155 MSDP monitoring • Monitoring set-up DE. TEN-155 router WS - DANTE MSDP peering • Software – adapted C++ MSDP • implementation of Steve Rubin • MSDP usage monitoring • number of RPs, average 90 (40 EU) • number of groups • number of pairs source, group (S, G) • number of SA messages per minute Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 13

TEN-155 MSDP monitoring number of RPs Number of RPs announced to TEN-155 before and after the loss of US connectivity Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 14

TEN-155 MBGP monitoring • Monitoring set-up WS - DE Po. P MBGP peering DE. TEN-155 router • Software - Merit’s MRTD • modified SAFI definition for MBGP • MBGP monitoring • stability/updates • number routes, max about 9000 (760 EU) • number of ASNs, average 240 (80 EU) Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 15

TEN-155 MBGP monitoring stability/updates • In blue, • the number of routes originated from one AS • In green, • the number of updates originated by the same AS Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 16

TEN-155 Multicast Monitoring tools • http: //www. dante. net/pubs/dip/40/40. html • http: //www. dante. net/pubs/dip/41/41. html • http: //www. dante. net/mbone/ • http: //sigma. dante. org. uk/stats/mrtg/msdp/data/ • http: //www. dante. net/mbone/mbgp The graph values from mbgp and msdp monitoring are historical values. Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 17

TEN-155 Operational Experience Conclusion • Move to native and SM multicast in national networks (NRENs) • Unicast and multicast non congruent • MSDP peer doesn’t need to be RP. • Concept of two BGP tables and “multi-protocol” RPF check often still misunderstood. • Inter-domain Multicast debugging => Almost impossible to fix problems in just one week Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 18

PART II NEXT GENERATION GÉANT DEPLOYMENT Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 19

www. dante. net/nep/GÉANT-MULTICAST/map. html Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 20

GÉANT Services • GÉANT Standard IP Service – IP traffic from NREN to NREN and Research peerings. • Multicast Service (rolling out now) • Replacement of TEN-155 Managed Bandwidth Service – GÉANT Premium IP Service – Layer-2 VPNs (forthcoming) • Upcoming • Security and Dos attack detection, IPV 6 Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 21

GÉANT’s Customers and Other Peerings • GÉANT Unicast Customers – 27 countries in Europe • GÉANT Multicast Customers – 24 countries in Europe • GÉANT Unicast and Multicast research and commercial peerings – Abilene, Canarie and ESnet via GTREN – Infonet, UUNET Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 22

GÉANT Multicast Customers www. dante. net/nep/GÉANT-MULTICAST/map. html Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 23

GÉANT Multicast Implementation • Current design built on – The experience gained from TEN-155 – Guidelines from multicast experts – Juniper Laboratory tests – GÉANT Multicast services – Multicast transit domain NRENs to NRENs – Multicast transit domain NRENs to other PEERS – Beacon monitored backbone • GÉANT Multicast routing policy at: – http: //www. dante. net/nep/GEANTMULTICAST/routing-policy. html Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 24

TEST-BED LAB 26/27 th Nov 2001 Host. X Group 233. 1. 10. 1 Source 10. 1. 10. 2 • Junos 5. 0 R 3. 3 • mesh group Network 10. 1/16 -AS 101 Sulfur Iridium Mangan ese Group 233. 0. 1. 1 Source 10. 0. 1. 2 M-BGP peering I-MBGP full mesh I-MSDP full mesh Network 10. 0/16 -AS 100 Group 233. 0. 3. 1 Source 10. 3. 30. 2 Gallium MSDP peering Helium Host. Z Host. Y Xenon Network 10. 2/16 –AS 102 Group 233. 2. 20. 1 Source 10. 2. 20. 2 Tin Network 10. 3/16 –AS 103 Group 233. 3. 30. 1 Source 10. 3. 30. 2 Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 25

PIM SMv 2 GÉANT domain • GÉANT single PIM-SMv 2 – Version 2 , enabled on all interfaces. • Three Rendez-vous Points with private anycast address. – backup for internal sources and receivers. – Private Anycast address (filtered out) – closest RP based on the OSPF cost • All other interconnected administrative domain have to be PIM-SM v 2 enabled with their dedicated RP. Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 26

PHYSICAL TOPOLOGY: RP IUCC Janet Nordunet NY 4 -1 9 uk Eenet Litnet Latnet se pl 40 630 Heanet 40 Surfnet 7 Posnan Rendez-vous Point ie nl Multicast access 35 640 5 Belnet 40 be Restena 40 640 Renater 159 10 640 NY 4 -2 Infonet lu 10 de fr Rediris 7 FCCN Po. P DFN Cesnet 10 cz 40 es 40 20 7 sk 35 ch 7 20 hu it 10 170 at 160 si 160 gr Grnet Hungarnet Ro. Edu. Net 40 INFN Switch&Cern Sanet Carnet Aconet Cynet Unicom-b www. dante. net/nep/GÉANT-MULTICAST/map. html Arnes STM 64/OC 192 STM 16/OC 48 STM 4/OC 12 STM 1/OC 3 Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 27

GÉANT Design: MBGP and MSDP • MBGP • Separate multicast routing table (inet. 2) • Currently congruent BGP and MBGP topology in Europe • i. MSDP • MSDP is fully meshed between 19 Po. Ps • Use of mesh group • i-MSDP Peering with loopback addresses (Not the anycast address !!) • e. MSDP • NREN <---> GÉANT Access Router Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 28

MSDP peering Logical view External MSDP peering NRN ----- GÉANT router RP Janet GRnet Iucc Cynet Unicom-b Renater gr ch es Belnet be it INFN Arnes uk si at GTREN Eenet Latnet Abilene de 2 i. MSDP Mesh Group i. MBGP Full Mesh Carnet hu Aconet Ro. Edunet Hungarnet Nordunet Litnet se fr Switch&Cern Rediris Ny 4 -1 sk Sanet cz pl Infonet de 1 DFN NY 4 -2 nl Surfnet pt FCCN lu ie Restena Heanet Posnan Cesnet www. dante. net/nep/GEANT-MULTICAST/map. html Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 29

MSDP SA Filtering • A list of filtered SA is defined at: – http: //www. dante. net/nep/GEANTMULTICAST/deployment-msdp. html • We filter the recommended list. • We authorise 239. 194. 0. 0/16 from the IPV 4 Organisation Local Scope through GÉANT. Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 30

GÉANT Multicast and Unicast Coverage CAnet Esnet Abilene RESEARCH Peerings DFN GTREN www. dante. net/nep/gtren. html DE UK GÉANT AS 20965 NREN 2. . . STM-16 STM-4 JANET NREN 4. . . Third party provider European Distributed Access Commodity Internet Access Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 31

GÉANT Multicast Service • Access to the service – Via the primary access to GÉANT – Via a GRE tunnel (currently nobody) – Support of PIM-SM v 2 only • Operational procedures (rolling out now) – Goal: same level of service as Unicast. • Troubleshooting – Extension of the trouble ticket systems to multicast incidents Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 32

GÉANT multicast monitoring • Beacon – Tool initially developed by Kai Chen from NLANR • dast. nlanr. net/projects/Beacon/ • Relies on a number of Agents spread over the network which simultaneously send and receive multicast packets carrying a packet sequence number and a timestamp. • Communicates with a central server which displays matrices of Agents via web pages. Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 33

GÉANT multicast monitoring • Recommended by TF-NGN group – ( www. dante. net/tf-ngn) • To use from day 1 – monitoring of multicast inside and outside of GÉANT. • Server code enhanced with historical functionality · http: //noc. man. poznan. pl/noc/index/strony (Menu item “Applications”) · romradz@man. poznan. pl · Multicast Beacon Agent written in C · http: //www. cesnet. cz/tf-ngn/multicast/ Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 34

GÉANT multicast monitoring • Beacon’s matrices – One for the internal sources of GÉANT • Each GÉANT POP has a beacon agent installed – One for the external sources of GÉANT • http: //beaconserver. geant. net: 19999/ – We have assigned two multicast groups from GLOP range [RFC 2770] for each matrix • Parameters monitored • Loss • Delay • Jitter Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 35

Beacon internal/external matrix www. dante. net/nep/GÉANT-MULTICAST/deployment-beacon. html Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 36

Other Monitoring Tools • Per group monitoring from TEN-155 – Based on the IETF IP-MROUTE MIB – Shows traffic per multicast group per interface – Under installation on ws 1. se. geant. net • MSDP and MBGP monitoring tools – Not yet available Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 37

CONCLUSION • Conclusion – From TEN-155 to GÉANT Unicast and multicast moves to a congruent topologyacross Europe and towards research peerings. • Deployment status • Links – http: //www. dante. net/nep/GEANT-MULTICAST/ – http: //beaconserver. geant. net: 19999/ Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 38

THANKS We would like to thank and acknowledge the help of the people who worked and are working with us on these projects, mainly from all EU and US research networks Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 39

Questions ? Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 40

SA Filter list Agnès Pouélé (agnes. pouele@dante. org. uk), Jan Novak (janovak@cisco. com) 41