I 2 Virtual Conference Network Recovery after the

I 2 Virtual Conference Network Recovery after the Attack On the World Trade Center. Tim Lance, NYSERNet Doug Carlson, New York University Alan Crosswell, Columbia University Armand Gazes, The Rockefeller University

I ntroduction Restoring Service After the Attack. Tim Lance, NYSERNet. Photo by Doug Carlson of Manhattan Skyline prior to the attack on the World Trade Center.

Relationship between 7 World Trade Center and 140 West Street prior to September 11 th.

Image of Manhattan from IKONOS Satellite taken Summer 2000. 60 Hudson Street Images courtesy www. spaceimages. com. 140 West Street World Trade Center

Image of Manhattan from IKONOS Satellite taken shortly after the attack on the World Trade Center. 60 Hudson Street Images courtesy www. spaceimages. com. 140 West Street World Trade Center

World Trade Center complex showing relationship between 7 World Trade Center and 140 West Street.

Photographs Copyright 2001 Verizon Communications. All rights reserved. Exterior view of 140 West Street showing debris and damage from collapse of 7 World Trade Center.

Photographs Copyright 2001 Verizon Communications. All rights reserved. An interior view of 140 West Street showing damage from collapse of 7 World Trade Center.

N etwork Status - NYSERNet o o o o As a result of the attack… CO at 140 West Street severely damaged. NYSERNet OC-48 SONET ring operational but no longer a ring. NYSERNet R&E Network operational. NYSERNet POP at 60 Hudson Street operational. Abilene connections operational. v. BNS+ connection down. R&E service for SUNY Stony Brook and AMNH is down.

N etwork Status - Commodity o o o o As a result of the attack… CO at 140 West Street severely damaged. Applied. Theory equipment largely undamaged. Applied. Theory Network operational. Applied. Theory POP at 60 Hudson Street operational. Customers connecting to 60 Hudson operational. Customers connecting at 140 West are down. Customers connecting via Garden City, Deer Park, and White Plains are down.

R estoring Service Columbia, IP Telephony & The New York Academy of Medicine. Alan Crosswell, Columbia University Photo of 140 West Street and WTC 7 courtesy of the Westchester Emergency Communications Association, www. weca. org

R estoring Service o o Early Morning, September 11 th Dealing with start of semester file server performance issues after a failed Summer file server upgrade project. Trying to get a handle on filtering Code Red with our Catalyst 6509’s. Turning off many user network ports for compromised hosts day in and day out.

R estoring Service o o Post Attack, September 11 th Trying to find out where all missing staff are. Discover phone trunks are overloaded. Send email to friends asking them to call our families and let them know we are OK.

R estoring Service o Post Attack, September 11 th o Administration establishes response team. They request that we: o o o Set up large lecture halls with CNN; Students are crowding around TV monitors in lounge areas. Establish net 2 phone-like functionality so students can phone home. Send a mass e-mail to the Columbia community from the President.

R estoring Service o o Getting CNN Out There, September 11 th Start planning to drag an IPTV encoder over to a cable box in a dorm. Staff drag some coax from a lounge TV monitor to a large lecture hall. Read on wg-multicast list that Northwestern has started multicasting CNN, but find our multicast connectivity is broken.

R estoring Service o o o Getting Phones Up, September 11 th One staff member tests net 2 phone…. Send email to Videnet & others + CS SIP group requesting use of PSTN gateways to route around local congestion. ~11 am. End up configuring H. 323 Polycoms via UNC Chapel Hill, and Cisco IP phones via Penn State, and SIP phones via 4 sites (Yale, Dynamicsoft, Nortel, Clarent).

R estoring Service o o o Getting Phones Up, September 11 th Net 2 phone works but it’s not easy & requires a credit card…. Polycoms work but also not so easy…. CS puts 4 SIP phones in CS conference room. 2: 20 pm We put 2 Cisco IP phones via Penn State in student center lobby. Just dial 8 and get a PSU dial tone and call home! 4: 00 pm.

R estoring Service o o Getting Phones Up, September 11 th By the time we had the IP phones in place trunk congestion had declined. We had many other Videnet sites offer their H. 323 – H. 320 PSTN gateways. Thanks to all of you!

R estoring Service o o o September 11 th Onward One of our dialup modem pools is out (and still out). A major expensive carrier. Our free T 1’s from a small carrier are up on our main pool. Weekly Email volume increased by 40% over prior week. Daily volume on Thursday 9/13 was 800, 000 messages: double last year’s.

R estoring Service o o September 11 th Onward Filtering SIRCAM, nimda, Code Red, WTC viruses. Turning off lots of ports of compromised hosts and attempting to deal with helping people reformat and reinstall their systems. Established outbound P 2 P traffic shaping.

R estoring Service o o o September 18 th After a week, NYAM’s T 1 service via 140 West St is still down and low priority for restoration. Set up an 802. 11 b link across Central Park and turn it up Wednesday 9/19.

R estoring Service Restoring Service to Mt. Sinai & School of Medicine Doug Carlson, New York University Photo of 140 West Street. Copyright 2001 Verizon Communications. All rights reserved.

R estoring Service o o Post Attack Administration establishes NYU Command Center. Campus phone system becomes overloaded for a period of time (no dial-tone). Internet connections provide links to family, friends, co-workers: o o Email and instant messaging. CISCO phones to (primarily) co-workers off campus.

R estoring Service o o o Post Attack Information Technology Services (ITS) takes point in keeping NYU community informed via Web, mass-mailings via email, voice mail announcements and Help Desk services. Mt. Sinai Hospital traffic routed through NYU. Phone banks set up to allow students to call home for free. Temporary shelter set up for students and staff who were evacuated.

R estoring Service o o Post Attack Evacuated students and staff are relocated to hotels around the city: o o Students given some money for essentials. Arrangements are made to get new sets of books for students. Loans of laptops and desktop computers offered. Preparations made to increase dial-in capacity via ISP. (Eventually not needed due to students returning to their residence halls. )

R estoring Service o o We were Lucky! If 60 Hudson Street had gone down, we would have been out of business.

R estoring Service Restoring Service to Rockefeller, Weill, & HSS. Armand Gazes, The Rockefeller University Photo of new cable being placed on West Street. Copyright 2001 Verizon Communications. All rights reserved.

key Commodity Path Weill Medical Center of CU Hospital for Special Surgery The Rockefeller University R&E Path Rockefeller Router Applied. Theory Router Rockefeller NYSERNet Switch Applied. Theory Router Applied. Theory NYSERNet Internet 2 th Network Connectivity Prior to September 11.

key Commodity Path Weill Medical Center of CU Hospital for Special Surgery The Rockefeller University R&E Path Rockefeller Router Applied. Theory Router Rockefeller NYSERNet Switch Applied. Theory Router Applied. Theory NYSERNet Internet 2 th Network Connectivity Subsequent to September 11.

key Commodity Path Weill Medical Center of CU Hospital for Special Surgery R&E Path The Rockefeller University Rockefeller Router Applied. Theory Router Rockefeller NYSERNet Switch Applied. Theory Router Applied. Theory NYSERNet Router NYSERNet Buffalo Applied. Theory Router Internet Commodity Service Restored. Internet 2

R estoring Service o o o Restoring commodity service… To Rockefeller University, Weill Medical Center, and the Hospital for Special Surgery. Service restored through Rockefeller’s R&E connection and emergency installation of a jumper between NYSERNet and Applied. Theory in Buffalo. Key - cooperation among campus personnel, commercial service providers and NYSERNet. Service restored by Wednesday afternoon, the 12 th.

R estoring Service Restoring Commodity Service to Long Island. Tim Lance, NYSERNet. Photo of Manhattan Skyline on September 23, 2001 by Doug Carlson.

Applied. Theory Long Island NYSERNet Applied. T heory 60 Hudson St. 140 West St. Verizon NYSERNet Metropolitan OC-48 SONET Ring Broad St. Verizon th NYSERNet Metro Ring & Applied. Theory Prior to September 11.

Applied. Theory Long Island NYSERNet Applied. T heory NYSERNet Applied. T hory 60 Hudson St. 140 West St. Verizon NYSERNet Metropolitan OC-48 SONET Ring Broad St. Verizon th NYSERNet Metro Ring & Applied. Theory After September 11.

Applied. Theory Long Island NYSERNet Applied. T heory NYSERNet Applied. T hory 60 Hudson St. 140 West St. Verizon NYSERNet Metropolitan OC-48 SONET Ring Broad St. Verizon Service Restored Utilizing NYSERNet SONET Ring.

C onclusions

N etwork Status - Commodity o o o Restoring commodity service… Polytechnic University - commodity service restored on 10/1. Long Island – commodity traffic still transiting NYSERNet SONET Ring. White Plains – connected to Garden City and then onto NYSERNet Ring. Rockefeller Solution - evolving into standby service for all customers having both NYSERNet R&E service and Applied. Theory commodity service. o West Street Service – a few commodity T 1’s still down.

N etwork Status – NYSERNet o o Restoring R&E service… American Museum of Natural History SUNY Stony Brook – service restored on 9/25. v. BNS+ – service restored almost immediately. – service restored on 9/20.

O ther Stories o New York University extended service to Mt. Sinai and School of Medicine. o Hofstra University offered NYSERNet bandwidth on its OC 3 if it could be utilized to serve any impacted institutions. o City University of New York hosted the New York City Board of Education Web site to ensure that parents could remain informed on the status of their children’s schools. o American Museum of Natural History staff offered to serve as remote hands for any institution needing onsite service. o Columbia University provided New York Academy of Medicine with access via wireless service. o NYSERNet offered to allow Applied. Theory to utilize its Manhattan SONET ring to restore commodity service, if feasible.

L essons Learned o o How much can be accomplished in a short time when the urgency is clear. The value in having multiple independent paths to the Internet. The value of the relationships. The survivability of the technology.

M aking it happen Many individuals and organizations contributed their time and creativity to this effort. Here a few of them: Sheryl Soskel - CUNY Anil Khullar - CUNY. Michael Benedetto – AMNH Jim Whitlock - Buffalo Mike Philman - Lawrence Berkley Lab Eric Damboise - Univ. Maine Gabriel Moulton - OSU Greg Economides - TAMU. edu Steve Blaire - Univ. Penn Phil Coolick - Penn State Steve Good - RIT Tim Poe - UNC (Chapel Hill) Bob Bloom, Applied. Theory Steve Knapp, Applied. Theory Bill Owens, NYSERNet Anne Dunford, NYSERNet Sharon Akkoul, NYSERNet Jeff Harrington, NYSERNet Videnet - UNC Cisco Accord Nortel Networks Clarent. com Dynamicsoft. com Radvision Broadwing Communications Verizon Communications And Computing & Communications staff at many institutions.

T hank you