Lessons Learned from Hurricane Katrina Azim Ashraf Manager

Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response

Personal Naiveté • • • Personal Preparations Some sense of excitement Estimation of what may occur Weather Channel – always on A bit of ‘Snow Day’ mentality

Hurricane Katrina Initial Projected Path Thursday August 25 Sunday August 28 Saturday August 27 3/19/2018 Tuesday August 23 LOUISIANA STATE UNIVERSITY 3

Monday, August 29 - Landfall • Katrina’s Immediate Effects – Makes landfall 6: 10 a. m. – Lower LA Parishes swamped by storm surge; no real word out – Parts of New Orleans flooded, at least one levee over-topped, but city seems to have survived – SE Louisiana devastated by winds/rain – Mississippi seems hardest hit • Monday 5 pm Meeting at LSUPD Station – LSU is OK – LSU Survived … just a little damage on campus – Data Center Lost power but fail-over to back-up worked perfectly – Everything Looks “Good to Go” for Tuesday clean-up, Wednesday start-up, and Thursday-as-usual – Mood lightened – Power restored to campus ~6: 15 pm

Tuesday 8/30 – Bad gets worse • First confirmed reports of a levee failure in New Orleans occur at 1: 30 AM CDT • By mid-day >80% of New Orleans is under water • Evacuees en route • LSU contacted about expanding routine special evacuee facilities into a broader purpose – Medical Triage (Pete Maravich Assembly Center) – Special Needs Facility (Field House) – First IT needs – Phones, phones and more phones

Called to assist • IT personnel needed to respond • It was not going to be anything like a ‘snow day’

First Impressions

First Impressions

First Impressions

LSU – A city within a city Large H. Ed. institutions uniquely positioned to respond • Infrastructure, knowledge, manpower, affiliations – PMAC/Field House – Became the largest acute care hospital to date in in U. S. history • Over 40, 000 (? ) patients processed during Hurricanes Katrina and Rita – Established a Hurricane command center • Coordinated information for students, and evacuees, as well as directing resources to where they were needed – – – – Faculty, staff, and student volunteers Housing for responders Crowd control Food and laundry services Long distance charges Managed volunteers Received and distributed donations

LSU – A city within a city (cont’d) – Tracked patients, volunteers, responders, supplies, etc. . – Provided Web page re-direction (and other IT services) for UNO – Leveraged communications hardware and services to facilitate data or phone support for: • Command centers • Responders • Govt. Agencies • Affected Universities • Evacuees • Etc. – LSU expended over $1 M (not reimbursed) • Over $100 K out of CIO’s budget – LSU Became perhaps the most critical facility in support of disaster relief/response in the State of Louisiana

Lessons Learned at LSU • Buildings can be rebuilt; hardware can be replaced. Data is the basis of continuity. • Knowing what you’ll need to do and having it organized is more important than knowing exactly ‘how’ you’ll do it • IT enables everything in the 21 st Century • IT Personnel = First Responders • Disaster Recovery and Business Continuity Planning is not a luxury • Be prepared to be flexible; adapt, improvise, overcome

Lessons Learned at LSU (cont’d) • Have a good stock of networking equipment, and mobile and desktop computing in the storeroom • Have strong relationships with key vendors • And most importantly…

People are your most key asset • Know who does what and have them ‘on reserve’ • Expect them to be burdened with other priorities • Be prepared to be amazed…

Key changes in LSU’s Plan • Formal LSU EOC • Formal Memoranda of Agreements (MOAs) – State agencies – Private sector • diesel fuel from local refinery • water from local bottler, etc…. – Secondary suppliers backing up primaries • Chancellor requested written plans from all units on campus • Full-time generator for PMAC • Logistics now pre-planned

Traditional Disaster Recovery - You’re down, everything else is fine • Do you have a workable DR plan? • Do you know where on campus you’ll go? • Did you take necessary back-ups and do you have them ready to re-produce production files? • What vendors will you need to tap – and for what? • How will you quickly re-establish network connectivity? Phone service? Web presence? Email? Mission critical information systems?

Broader Disaster Recovery - You (and everyone around) you are down • Are your off-sites conveniently (and perhaps tragically) close? • Do you have arrangements to get key services restored at a distance – Web, E-mail, Financial/HR, Student Information, CMS • Hot-sites may be too expensive – but can you find suitable raised floor/HVAC/power to ‘re-build’ • Can you support your administration “in exile? ” – Internet access, computers, cell phones, e-mail, IM • Is your ‘life-boat’ plan portable over larger distances? • Can you grab your key people? Can you care for them?

One Possible Tool In The Arsenal: Data Center Lifeboat • Situation: What if we had very short notice (4 -8 hours) notice of the need to abandon our data center/campus and set-up elsewhere (>50 miles away) • Goal #1: Re-establish some critical subset of services • Goal #2: Support the re-establishment of some subset of university administration

Lifeboat • Key things to recover: – Payroll/Financial Data – Web presence • Splash/priority information screens • As much content as possible – E-mail service for faculty/staff/students – Portal interface – Student Information Systems – HR, Procurement Systems – CMS – What else? • Budgets ($25 K, $50 K, $100 K) • Key things to address – Off-site storage of critical back-ups – Ability to ‘grab and go’ key data and hardware – List of key hardware needed later from vendors – Disaster Supplies Crate • What would we put into an 8 x 12 truck for rapid evac? – Equipment for a mobile or relocated university command post • Laptops, radios, phones, etc. – Identify Key IT personnel • Who does what w/back-up • “Scoop ‘em up” – Where might you go?

Survivor Disaster Recovery You’re the last ones standing • Dealing with unimaginable demands – Start imagining it • Do you have a stock of equipment to set up a large support operation in short-order? – Networking gear, computers, cables, supplies, telephone service • Value of a flexible and capable staff • Consider how you’ll do all this on top of your normal jobs, as campus life resumes and student enrollment increases • How ready is your campus administration to take on the role of disaster response center? – Facilities, public safety/police, communications, academic affairs – Is the CEO (Chancellor, or President) prepared?

Final Thoughts • Imagine the questions first so that you can find the answers • Next time, you may not be watching it on CNN – you may be living it • Do the right thing • Now is the time to think, plan, and take action – later it will be too late

Final Thoughts • • Data is the basis of continuity Have a flexible plan People are your most key asset Do the right thing because in the end its really all about…

Service

Credits • The staff of LSU ITS who helped make the relief effort a success. • Brian Voss (CIO) – ‘In the Wake of Katrina’ • Brian Nichols (CISO) – ‘At Katrina’s Edge’ • Frank O’Quinn (DR) – ‘Weathering the Storm’ • Sheri Thompson, Jim Zietz, and others- photographs • John Borne – excerpts from Master’s Thesis • Margo Jolet, LSU Office of Public Affairs - ‘LSU in the Eye of The Storm’

Lessons Learned from Hurricane Katrina Azim Ashraf Manager – Network Security & Incident Response