UH SWARM Dense perf SONAR Deployment With Small

UH SWARM: Dense perf. SONAR Deployment With Small, Inexpensive Devices Alan Whinery U. Hawaii ITS September 29, 2015 Slide 1

The Whole Small perf. Node Thing At a $2000 to $5000 price point, a typical perf. SONAR node gets deployed at REN leaves and branches At $50 price point, you can buy 40 to 100 times as many for the same amount Some focus on p. S node equivalence, Intel compatibility Deployment tends to be relatively sparse (~$200 price point) Some focus on value in smaller ($50) nodes Slide 2

The Whole Small perf. Node Thing Perf. SONAR developer fork for small devices: https: //github. com/perfsonar/project/wiki/perf. SONAR-Endpoint. Node-Project $150 - $250 price range, focus on Intel architecture RNP Brasil – Mon. IPE http: //monipe. rnp. br/ Same type of nodes – ARM architecture - as our $50 price range UH SWARM Our thing – Beaglebones, Raspberries, etc ARM/$50. Slide 3

The Swarm Wrote paragraph into our CC-NIE campus networking proposal about making use of the recent availability of ~$50 computers to “sense” the network, using elements of perf. SONAR. Funded a project to deploy 100 nodes on one campus over 2 years, exploiting a ~$50 price point to deploy many nodes on campus as a dense mesh. Slide 4

Goals/Challenges Finding nodes to buy in the face of market exhaustion Getting node deployment work-flow down to nil Getting recoveries of off-line nodes to a minimum Tracking assets and reliability, generating metrics Evaluating capabilities of the whole set-up Developing a test program for many nodes Slicing/Dicing data to see what it has to tell us Developing visualizations and distillations to put tools in hands of network maintainers, merging into p. S Toolkit Slide 5

Devices We Have Raspberry Pi – famous, $50, med-perf, file system on SD card, 100 Mb Ethernet, USB 2. 0 Beagle. Bone Black – $50, more perf, FS on internal flash, and/or SD card, 100 Mb, USB 2. 0 Honorable mention: Cu. Box i 4 – $147, more perf, FS on SD, Gig. E, Wi. Fi, USB 2. 0 Mira. Box $149 – most perf, FS on SD, dual Gig. E, Wi. Fi, USB 3. 0 Slide 6

Reliability Raspberry Pi (July 2014) 22 SD card hard failures UH ITS owns 47 – 1 has failed 10 file-system failures Beagle. Bone Black Rev A/C. (December 2013/April 2015) UH ITS owns 60, 1 has corrupted firmware Of nodes in production, one had to be power-cycled, once Cu. Box – one deployed 6 months of service zero problems. (using SD from OEM). Mirabox – promising, dual Gig. E ($150), wimpy kernel Slide 7

SD Cards DANE ELEC 8 GB Class 4 10 cards, 2 failures in light duty San. Disk Ultra 8 GB Class 10 Kingston 8 GB Class 10 cards, 0 failures, 3 FS corrupted in 42 k hours 10 cards, 0 failures, 7 FS corrupted, in 42 k hours Kingston 4 GB Class 4 20 hard failures in less than 20 k hours (100% across 6 weeks, < 1000 Hr MTBF) San. Disk Ultra – 8 GB Class 10 Most recent batch of replacements Slide 8

Year 1 Tried 10 Beagle. Bones, liked them And a few Raspberries Pi The market vacuum around the release of BBB Rev. C made BBB impossible to obtain Bought 43 Raspberries Although we are going with Beagle. Bone Black for the completion, we could make Raspberries work if necessary. Bought 2 Dell rack servers as test facilitators, data archives. Slide 9

2 nd Year Completion 50 Beagle. Bone Black Rev. C (4 GB internal flash) Internal + SD card enables separating system/data partitions BBB Internal flash is more reliable than SD Better 100 Mb Ethernet performance 5 Raspberry Pi 2 Model B As number deployed approaches 100, we will be placing nodes in new/special roles. Correlating topology from netdot, MRTG graphs for context Slide 10

Management Puppet/The Foreman http: //theforeman. org/ Easy to push changes, updates out to the swarm. https: //puppetlabs. com/ Easy to push errors out to the swarm and require 50 SSH sessions. Work-flow Try to minimize per node actions and attended setup RPi – ua-netinstall with tweaks for Puppetization BBB – custom SD that auto-images the internal flash Make individual nodes as interchangeable as possible (If you have a choice use one type of device) Slide 11

Characteristics Of Dense Sensor Deployment Within An Enterprise A “sensor” is less complicated than a perf. SONAR toolkit node Central perf. SONAR buoy/MA orchestrates Having many observations makes the loss of a single one less important. You can correlate topo and test results to “triangulate” on the source of a problem. It takes planning avoid affecting user traffic Strategy is to “be” user traffic p. S Toolkit as-built isn't really made for 100 nodes Slide 12

Test Programs: powstream (owamp) powstream puts 10 packets/second on the wire, 24 hours a day (there's been discussion about increasing the rate) To some extent, apparently stochastic/probabilistic loss resembles stochastic/probabilistic loss at much higher rates – meaning – the probabilistic loss that powstream encounters is probably the minimum of what a throughput test will encounter. Slide 13

Side. Bar: Regular Global perf. SONAR powstream Log-scaled loss to color gradient An early idea about how to unify many graphs in front of your cerebral cortex. Black = 0% loss Green->Yellow->Red gradient: low → medium → higher loss Log scaled to avoid hiding low loss In our campus network, everything was always black (no appreciable loss) (Gray – no data) Time: left to right Each 10 pixel row is one path. Slide 14

Test Programs: powstream (owamp) powstream from p. S Toolkit node to/from each sensor node Really, really boring at first glance. All loss appears to be about zero. Always one or two losing a packet per day (1 in 864000) Standard deviation in latency groups somewhat interesting, may reflect queuing, flares in latency std dev may precede loss events Longitudinal analysis reveals damaging loss rates that would otherwise be invisible Higher packet rates might expose low loss probabilities in shorter time Slide 15

30 nodes, in/out

Slide: Phil Dykstra SC 2006 Used with permission Mathis, Semke, Mahdavi, "The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm”, ACM SIGCOMM, Vol 27, Number 3, July 1997

Slide: Used with permission

Speed Limits You Can't See For 45 milliseconds RTT, typical minimum to get onto continental US from Hawaii Loss Rate 10 pps Powstream Packets Lost Per day TCP AIMD Coastal Limit @1460 MSS (Mbits/sec) TCP AIMD Coastal Limit @8960 MSS (Mbits/sec) 45 m. S RTT 1. 82 E-005 15. 75 42. 56 261. 18 2. 25 E-006 1. 94 121. 11 743. 23 1. 87 E-006 1. 62 132. 76 814. 72 9. 38 E-007 0. 81 187. 58 1151. 16 6. 05 E-007 0. 52 233. 55 1433. 28 5. 93 E-007 0. 51 236. 03 1448. 52 3. 35 E-007 0. 29 314. 03 1927. 21 2. 51 E-007 0. 22 362. 49 2224. 57 1. 74 E-007 0. 15 435. 64 2673. 49 Slide 19

Test Progams: 50 Node Full Mesh TCP Throughput <= 100 Mbps RPi, BBB throughput tests resemble real -life user flows Unlike a high performance iperf tester which “punches the network in the face” I run a 50 x 50 full mesh iperf matrix (2450 tests) in about 7 hours, (5 second tests). Full-mesh traceroute is collected concurrently By scoring every hop encountered on the average peformance for paths it appears in, “per-hop confidence” can be derived. Using multi-rate UDP vs. TCP is worth investigating. Slide 20

Destinations The Matrix Sources Cut-out view of iperf 3 tests to/from a chosen node. . . This row/column represents all tests to/from that chosen node. Leaves one wondering what the correlation is between the pink squares showing retransmissions Slide 22

Correlating Full Mesh Throughput And Traceroute Results For Fault Isolation Slide 23

Graph of per-hop “confidence” with colored links where retransmissions were observed (names/addresses obfuscated) This graph shows hops involved in in-bound Throughput testing between a chosen node and all partners. Each oval represents an IP interface as reported in Traceroute output. Graph rendered from test data with Graph. Viz. (Graph. Viz. org)

Data Archiving perf. SONAR MA Exposing some ways in which MA handling of long-term, diverse data could be optimized Correlating such things as early/late “bathtub curve” failures per equipment life cycle (see Wikipedia ^^^ ) Trending probabilistic loss by months/years Etc Slide 25

Ongoing perf. SONAR toolkit integration Not so much new development as making some pieces fit together Correlation of other sources to zero in on a fault Net. Dot Flows/MRTG Ancillary programs Log collection (honeypot-ish info) Name resolution tests v 6/v 4 precedence Slide 26