Lessons Learned from the April 16 2007 Tragedy

Lessons Learned from the April 16, 2007 Tragedy at Virginia Tech EDUCAUSE Live! November 5, 2007 Erv Blythe, VP for Information Technologies 1

Presentation Topics • April 16, 2007 • The impact – load on the communications infrastructure • Web Communications • The notification issue • Radio communications interoperability • Identity management and privacy • Data preservation and computer forensics • A new perspective on IT 2

April 16, 2007 • 49 people shot • 32 dead plus shooter • 9 minutes 3

Crisis Response • About 1500 first responders • 27 ambulances from 14 agencies and five hospitals 4

Communications Figure 1: Communications Infrastructure Stress Factors April 16, 2007 System Normal April 16 Effect University Web Site Access 455 gigabytes per MONTH (largest ever) 432 gigabytes in a DAY 3000 % increase Virginia Tech Police Dispatch Center 400 - 500 calls per day 2, 027 calls 45 0% increase Cellular Provider Capacity and Coverage Designed for non emergency peak load, limited in building coverage Added 3 COLTs, 2 in -building antenna systems, 200 phones By April 17, temporary coverage/capacity added Internet gateway capacity 500 Mbps Added 1 Gbps over 10 GE research link 300% increase University Switchboard 3, 200 calls handled per week 9, 878 calls handled 4/16 -4/21 300% increase Telephone calls into Blacksburg Central Office Reported by local provider Virginia Tech Telephone System Inbound Calls 25, 000 calls inbound daily on average 75, 000+ calls inbound on April 16 300% increase Centralized Computing Systems Data Storage Prior to 4/16, roughly 300 Terabytes/day Since 4/16, over 600 Terabytes/day 100% increase Data Preservation (12 week period) 3, 000 tapes 11, 700 tapes 390% increase 5 - Multifold Increase

A humongous traffic spike … 6

Response by Private Carriers • After the initial response period on April 16, cellular providers including AT&T, Sprint-Nextel, Verizon Wireless, and US Cellular all dispatched technicians to increase tower capacity. • By April 17, Sprint-Nextel, Verizon Wireless, and US Cellular each had “Cell on Light Truck” systems operating on campus and each had provided emergency-use phones and accessories. 7

Infrastructure Facilities During the crisis, Virginia Tech IT faculty and staff were called upon to: • Install telephone and data communications for – At least 9 geographically dispersed command centers – As well as media workrooms and counseling centers • Perform network traffic balancing under unusual load • Begin data collection and preservation activities • Assist in obtaining location information or class roster information 8

Lessons Learned: Communications Infrastructure • • 9 Public safety access point forwarding Information centers Command control centers Language barriers

Lessons Learned: Web Communications • Due to unprecedented load, the webpage had to be stripped down to bare essentials from the normal design to a “Lite” design 10

Normal home page before 4/16 11

“Lite” home page on 4/16 12

Lessons Learned: Radio Communications • Responders from several jurisdictions, all with different radios • Radio transmission was less than ideal in some locations • Deficiencies in interoperability and coverage of police, fire, and rescue radio communications are decades-old problems in the United States. 13

Notification System A significant challenge during an emergency is providing mass notification of a threat and instructions for response. All of these methods were used on April 16: • Broadcast e-mail to @vt. edu addresses (via LISTSERV) • Broadcast voicemail to campus phones (offices and residence halls) • Recorded message on the Weather. Line/Hotline • VT. edu (www. vt. edu) and the Virginia Tech News website • University switchboard • Public media (TV, radio, news websites) • Siren system 14

VT Alerts • A short list of vendors for this service was identified prior to 4/16/07 • Post-4/16 the vendor review process was expedited • 3 n was selected to provide hosted services for sending emergency messages – – – 15 Through cellular phones PDAs and other wireless devices SMS/ text messaging Email Or to voice services

VT Alerts Continued • Virginia Tech students, faculty, and staff can “opt in” to subscribe to an automated event notification system that will notify them of an emergency situation. Each subscriber can select up to three different methods to be contacted. These contact methods include: • Text message (SMS) to mobile devices • Instant message (AOL, MSN, ICQ, and Yahoo) • E-mail (including non-Virginia Tech addresses) • Phone call to office • Phone call to residence • Phone call to mobile number • Phone call to another destination 16

Questions 17

Lessons Learned: Privacy • Maintaining vigilant attention to the restrictions of FERPA and HIPAA added significant complexity to the urgent data needs of April 16 th and the activities of the days that followed. • Immediately following the tragedy, it was recognized that information concerning victims should be restricted from inclusion in the public directory. The University Registrar’s office was asked to mark the student records as “confidential” in Banner to ensure the information would not be publicly accessible from the Enterprise Directory. • For selected individuals identified by university, state, or federal authorities, copies of the e-mail content, Filebox content, e. Portfolio content, and voicemail recordings were preserved on compact disk (CD) or digital video disk (DVD. ) 18

Lessons Learned: Identity Management • Across many of the response activities, one overarching difficulty from a data perspective was the consideration of the diverse constituencies. • An integrated, enterprisecentric identity management system would have reduced some of the complexity of this process. 19

Data Preservation • Support for investigators • Support for families and victims • e-Discovery requirements 20

Questions 21

A New Perspective on IT Legacy Systems TELEPHONE Near Term Recommendations Strategic Recommendation Integrated IP -Based Information Architecture Anywhere, Any Device, Any Service WUVT Alert Phone Features DATA CATV EDUCATIONAL SYSTEMS CENTRAL IT SERVICES Improve Cellular INTEGRATED APPLICATIONS VT Alerts Notification CATV Alert System Crestron Control ADVANCED IP NETWORK Update Personal Info Massive Data Storage and Access High Performance Computing Cyber. Security Inherent Resilience, COOP Blackboard/Scholar Location Awareness 911 DISPATCH Pervasive Computing DIVERSE OPTICAL CORE Integrated Services Integrated Emergency Applications Sensor Networks PSAP Forwarding Police Dispatch DC ATLANTA SERVICES & GLOBAL ACCESS EMERGENCY RADIO Role of IT in Emergency Management 22 Local Interop/Data RECOVERY MITIGATION RESPONSE PREPAREDNESS RESPONSE

23

Questions 24

Other Resources Governor’s Review Panel final report http: //www. vtreviewpanel. org/report/index. html Information and Communications Infrastructure Group report http: //www. vtnews. vt. edu/documents/2007 -0822_communications_infrastructure. pdf 25