Avid System Monitor Ed Harper November 2010 1

Avid System Monitoring overview • Avid System Monitor delivers Enterprise wide monitoring solution for Avid systems and infrastructure switches Overview • Single GUI visibility to whole infrastructure • Standards based polling and event notification; SNMP, IP, HTTP • Tightly integrated with Avid Health Monitor • Integrate with enterprise management Devices Managed • ISIS 5000 & 7000; System Director • Interplay; Media Indexer, Look Up Server, Interplay Engine, Capture, ASF services, Capture • SNMP Network Switches (Cisco, Foundry, Force 10) Capabilities • Real Time Statistics, thresholds • Events, Alarms, Notifications (email) • Historical statistics • Surveillance Dashboard • Flexible reporting tools Benefits • Proactive real time status and statistics - Identify anomalies, prevent outages - System wide diagnostic tools, faster restoration • Trend analysis 2

What it is • • • A tool to increase the system availability by identifying issues in real time A tool to help identify potential problems in a system as they are occurring A single tool for monitoring all necessary components of the “system”, including Avid gear, network infrastructure, 3 rd party devices A tool that collects performance data over time so that it can be graphed (and trends identified) A tool that will continually evolve to identify known problems within a system (after the knowledge of those problems have been learned during Code Blues, etc) A window into specific state of the Avid & selected infrastructure system components at a given point in time. It also provides enough flexibility for customers to refine and fine tune the tool’s outputs once the basic functions are mastered. 3

Overview • Avid System Monitor delivers enterprise solution monitoring for Avid systems and infrastructure – – Pro-active system health and status monitoring Statistics gathering, graphing and thresholds Event logging, intelligent alarm processing and notification Dashboard views showing outages and availability • – Simple drill down to isolate issues Standards based • SNMP, HTTP & IP port status – Avid Monitoring Gateway service installed on Framework (ASF) enabled devices to provide visibility to Avid System Monitor via HTTP 4

Monitoring components Monitored Node Agents Monitoring Server Recommended platform SR 2500 GUI, SNMP & HTTP collection SQL Database Java (JDK) Environment • Interplay Engine • Stream Server • Capture • Media Indexer • Interplay Lookup Service (LUS) • ISIS 7000 System Director • ISIS 5000 3 rd Party • Cisco switches • Foundry Switches • Force 10 Real Time Audit Agentless • Avid Service Framework • Provides time sync • Air. Speed, Air. Speed Multi Stream • Capture Manager • DNS, DHCP services • Time Sync 5

Monitoring Environment • Monitored Avid Services & Devices – Detailed monitoring including status, statistics etc. • • • Real-time inventory – Device up/down status without detailed monitoring • • Avid Service Framework (ASF) – Media Indexer (MI) – ASF Lookup Service Interplay Engine Stream Server Interplay Capture ISIS 5000 & 7000: System Director Workflow Engine, i. News FTS, Workstation Service , Time Sync service, Multicast repeater, Low. Res Encoder 3 rd Party Elements – Windows services; DNS, DHCP etc – Network Switches • Cisco, Foundry, Force 10 6

Dashboard • • Single screen view with Intelligent grouping of devices & domains High level status – – • Alarms Notifications Node Status Resource Graphs Click on any device group to automatically filter information for selected devices 7

Events & Alarms • Extensive Event Logging – – – • Severity, source etc Acknowledgement Search Fine grain event details Correlating up/restore Alarms – Flexible rules to allow event aggregation in alarm view to count multiple occurrences of same event • • • – Severity Last time of event Count occurrences Link to event details Option to auto-clean events Operator Instructions specific to alarm & device type 8

Notifications • Flexible notification to email – • Individuals or groups Automatic Escalation – Escalation to higher level group if notification is not acknowledged within certain time • • Example; Minor event sent to Ops team, if unacknowledged for 20 minutes raised priority to Major and issues notification to Management team Notification logging, with timestamps including response time 9

Statistics & Charts • • Historical statistics gathering, trending, charts Thresholds set to trigger events and notifications on ‘interesting’ conditions – Specifically tuned to Avid components, based on real world experience 10

Threshold Event Notification • Flexible Threshold engine – – – Configurable on any counter in the system Extensive pre-programmed thresholds provided in Avid monitoring package Simple process to add customer specific threshold Media Indexer Media Files Admin configurable trigger levels 11

Threshold Configuration • Custom configuration of Threshold Event – Any counter value collected by Open. NMS – – – – Type; High, Low, Relative Change, Absolute Change Datasource; Entity to collect counter data (graph properties) Datasource Type; Node or interface Datasource Label; String displayed in event Value; Threshold value Re-arm; Reset/ Cleared value Trigger: Number of times the threshold must be broken to create an event 12

Node View • Single screen dashboard per node – – Current Status – Notifications, Recent Events, Recent Outages Availability; system and individual services 13

Outages & Availability • Current Outages – Node or Service down • Calculated 30 -Day Availability – Color Coded • Grouped by Device / Service Type – Click to drill down 14

Surveillance View: Flexible Grouping • Current Outages by; – – Device Type Workgroup or location • Grouping by – Service – Category – Simple customization 15

Node Discovery • Configure Open. NMS to discover devices and services on specific IP address or range – – Automated capability query of generic IP, SNMP and Avid specific services & device capabilities Add device names to nodes for readability if desired • • IP address and DNS names displayed by default Automated capabilities scan every 24 hours 16

Network Switch Monitoring • SNMP monitoring and statistics gathering for Cisco, Foundry & Force 10 infrastructure Zone 2 switches SNMP • Link Up • Link Down Network • Spanning Tree Topology Change • Bandwidth Utilization Thermal • Max temp exceeded System • Memory utilization • Processor utilization Cisco • Configuration change Foundry • Startup config change • Running config change • Telnet login / logout 17

Maps • Open. NMS provides mapping tool with device status – – Multiple maps to allow views for LAN, editors etc Link discovery find node connectivity • Not all links shown correctly; ISIS switches not manageable so devices appear connected to adjacent switch 18

Proving it’s Value (a real field example) • Phased Roll-out – • Monitoring SNMP switches (only) Customer Reported Air. Speed “Slow Down” – – Avid CS / Systems Engineers queried Open. NMS remotely Pulled switch bandwidth utilization • • – Switches operating correctly Within a few minutes troubleshooting team moved on to investigate specific devices Without Open. NMS proving switch operation required access labor intensive process of monitoring scripts and driving traffic loads • Time consuming ~ 1 day to prove switches Faster resolution Greater customer satisfaction 19

Example • • Memory Utilization on Interplay Media Indexer Charts show steady consumption of server RAM memory during load test Performance impacted as memory maxed out Thresholds provide notification when x% exceeded 20

Server & System Requirements Category Requirement Avid System Monitor Server Recommended; Intel SR 2500 Server Operating System Windows 2003 Processor 2 GHz or better Memory 2 GB Java JDK Provided with Avid System Monitor Postgre. SQL Database Provided with Avid System Monitor Client Browser Adobe SVG viewer Required for Internet Explorer client browser to view map pages (Firefox etc have SVG viewer built in) 21

Pricing, Availability etc • Delivery – Value-add offered to customers with Avid Uptime support • Software download • Phased roll-out at selected customer Production networks – Typically switch monitoring • Pricing – Avid System Monitor available to Avid Uptime support contract customers – PSG installation • PSG engagement required 22

Summary • Real-Time monitoring of devices, services, networks & infrastructure – Avid Customer Success – Customer IT / Admin • • Statistics, thresholds, events and notifications Broad Enterprise system support – Increasing breadth and depth • • Pro-active warnings and notification of potential problems Improved time to resolution 23

Avid Monitoring Solution ISIS client, Editor Open. NMS GUI ICMP (Ping) Avid TCP Port monitoring DNS ICMP HTTP/TCP SNMP Data collection Trap receiver Avid TCP Port monitoring DNS, time sync ICMP SNMP LAN Switches SNMP Interplay Engine, Stream Server, Archive Air. Speed Service / IP monitoring ICMP SNMP ASF Monitoring Gateway ICMP only ASF Health Monitor SNMP ISIS ISB, ISIS switch System Director ISIS Engine Lookup Server Media Indexer Air. Speed. MS Full Monitoring; events, statistics 25

Failure Modes Monitored • Avid System Monitor is tuned to identify specific failure modes – As found in field experience / Code Blue • Media Indexer • • • MI in the HAG with a weight of "0": Indicates an "election issue" which can cause major system slowdown. Number of quarantined files growing: Indicates a faulty ingest device creating bad files. Different file count between each of the HAG MI's: Indicates issue with ISIS notifications. Some files will appear offline to some clients. Different time on each of the machines in the WG: Can be the cause of lost ISIS notifications (see above). MI Heap usage running dangerously high: Indicates your WG file count or client count is causing too much stress on that MI. Eventually, the MI will thrash. Number of files added/updated on last full resync, when it's greater than 0. This value is displayed in the Health Monitor, under each storage pane of the MI. • Interplay Engine • • • Time to perform login - should be below 15 seconds: indicates engine slowness Number of journal files - should be below 50: indicates journal integration stuck/dead Number of deletes - should be below 100 for 5 minute polling intervals during normal production time: indicates deletion during production time Number of loaded objects/number of total objects - should be above 30%: indicates engine cache warm-up causing slowness Backup running flag - should be off during production time • Avid Service Framework Lookup Service (LUS) • • • For LUS, here are things we could check today via SNMP Gateway. However, these monitor points don't really contribute to most of the problems we see related to ASF. They are the only data points that are available today. Monitor Handle Count (either via gateway or MSFT agent) - should be below some threshold (<5000) Monitor Thread Count (either via gateway or MSFT agent) - should be below some threshold (<500) Monitor Events In Queue (via gateway) - should be less than 50 Check that a process is bound to port 4160 on the box (don't know how to do that with Open. NMS) - confirms that the LUS process is running Monitor Memory Usage (either via gateway or MSFT agent) - should be below some threshold (<200 MB) • ISIS monitors a number of critical areas and sends an event to the Windows event log when values reach a defined value or threshold. You can configure ISIS to send an email when an error or warning event occurs. You can also configure the System Director to generate an SNMP trap when the event occurs. The top areas include the following: Temperature and presence of components such as switches, storage elements, and power supplies. Workspace usage thresholds. For example, an Admin can enable warning and error thresholds. If you can set the workspace threshold to 90%, ISIS will generate an error event when a workspace reaches 90% full Disk health issues such as disk failed or disk performance degraded based on continuous monitoring. Server failover notifications. For example, on a failover system you are notified when the system fails over to the other node. Metadata problems. For example: if there is a problem opening a metadata file or if the metadata in a file seems out of date • • • 26

Monitored Device Matrix Device / Service Version(s), Notes Inv Unity ISIS v 2. x √ Interplay Engine V 2. x √ Media Indexer V 2. x √ Interplay Engine V 2. x √ Interplay Lookup Service (LUS) V 2. x √ Air. Speed Multi Stream √ Capture Manager Mon √ Interplay Capture V 2. x √ DNS & DHCP services √ Avid Time Sync Service √ 3 rd Party Network Switches Cisco / Foundry / Force 10 √ Windows Services DNS, DHCP, Time, Anti-virus, auto-update etc √ Inv Real time Inventory; service or server Up/Down Mon Full monitoring; detailed alarms, statistics etc 27