Data Center Consolidations Webinar 09 16 11 Bethann

Data Center Consolidations Webinar 09. 16. 11 Bethann Canada, Virginia Peter Tamayo, Washington Baron Rodriguez, PTAC Data Center Consolidation Considerations & Best Practices

Agenda • Overview of data center consolidation considerations. • Virginia’s data consolidation lessons learned – Bethann Canada • Washington’s data center consolidation effort – Peter Tamayo • Questions: Please send your questions to AEM Host through the chat box feature. 2

Political Reality • Dwindling state resources • Political drivers • Savings? • Increased security? • Increased reliability? • Centralized control of state assets • Increased capacity needs as a result of SLDS initiatives 3

Issues to consider… • Physical & Logical Access • Data & System Inventories • Consolidated Data Center Staffing • Business Continuity/Disaster Recovery Plans 4

Security Policies • Data Access & Use Policy – This policy specifies data access controls and defines appropriate data use. Such policies often break down specific roles and job responsibilities related to educational data. • Non-Disclosure Policy – Educational agencies should require individuals who have access to personally identifiable education records to sign affidavits of non-disclosure. • Data Retention/Destruction Policy – The data center provider generally is responsible for backup and archival of electronic systems. Therefore, a data retention and destruction policy should be crafted based on a careful review of local, state and federal policies. • Data Breach Policy – This policy outlines procedures to follow in the event of a data breach. It should include references to any applicable state, local, and federal laws and provide appropriate escalation procedures to follow in the event of a data breach. 5

Other considerations… • Access/Audit Log Review • Legal Ramifications 6

Virginia’s “Transformation” “It’s like building your house with your own hands and then watching someone take it apart piece-by-piece with a crowbar. ” Bethann Canada on VDOE’s infrastructure consolidation

2003 -2009 1. 2. 3. 4. 5. 6. 7. Personnel Assets Network Re-IP Messaging Desktop Server Legislation requires Executive Branch agencies to turn over IT infrastructure to a central IT Agency. Services are then outsourced to Northrup Grumman. Governor directs agencies to “play nice”. The Emperor is still naked…

Service Center Offerings 24 x 7 x 365 Security Desk All movement in/out through main lobby Automated access control Controlled visitor access Personnel badges, camera surveillance, camera analytics Support from local fire and police Backup data center in Southwest VA

Security Offerings Firewalls Intrusion Detection/Prevention Web Proxy for Internet usage and Anti-Virus Email Proxy for Spam and Malware Security Information and Event Management system Vulnerability and compliance scanning

Network Offerings Single and Dual-factor VPN Site-to-Site VPN Encryption Router to router Remote Access Site-to-Site VPN E-mail Full Disk Web Proxy

Moving to a consolidated data center: Runbook TOC Change Log Vendor and Agency Contacts (work, home, cell) Issues Register Resource Plan Application and Database Overview Application Dependencies VDOE’s server move was Backup Overview outsourced to EMC. Server Details Planning, documentation, and Storage Details execution were extremely Network Details thorough and the move was Server Runbook carried out flawlessly. Hour-by-Hour Rollback

Moving to a consolidated data center – Hour-by-Hour Plan Task Posted start Estimated time duration Application 10/16/09 Shutdown 18: 00 Database 10/16/09 Shutdown 18: 00 Server 10/16/09 Backup 18: 00 Server 10/17/09 Shutdown 5: 00 Server 10/17/09 derack 6: 05 load into 10/17/09 truck 6: 05 Transport 10/17/09 equipment to 8: 20 CESC Posted Estimated completion Actual start Start Time time 10/16/2009 18: 15 18: 00 10/16/09 18: 00 10/16/2009 18: 10 10/17/2009 4: 00 10/17/2009 9: 25 10/17/2009 9: 05 10/16/2009 18: 15 10/17/2009 4: 45 10/17/2009 5: 18 10/17/2009 3: 45 10/16/09 18: 00 10/16/09 18: 30 10/17/09 4: 45 10/17/09 5: 18 10/17/09 6: 15 1: 00 10/17/2009 9: 20 7: 30 0: 15 0: 10 10: 00 4: 25 3: 00 Percent complete time In process Time Remaining Estimate completion Actual completion 10/16/09 18: 15 100% 2: 57 0: 00 complete 100% 2: 27 0: 00 complete 100% 16: 12 0: 00 complete 100% 15: 39 0: 00 complete 100% 14: 42 0: 00 complete 10/17/09 7: 30 100% 13: 27 0: 00 complete 10/17/09 8: 00 10/16/09 18: 15 10/17/09 4: 45 10/17/09 5: 18 10/17/09 6: 45 10/17/09 7: 30 Rack equipment 10/17/09 10: 20 3: 30 10/17/2009 13: 50 9: 00 10/17/09 8: 50 100% 12: 07 0: 00 complete 10/17/09 12: 02 Start Servers 10/17/09 13: 50 3: 00 10/17/2009 16: 50 12: 02 10/17/09 12: 02 100% 8: 55 0: 00 complete 10/17/09 19: 24 Start Databases 10/17/09 16: 40 0: 30 10/17/2009 17: 10 19: 24 10/17/09 14: 45 100% 6: 12 0: 00 complete 10/17/09 15: 29 Start 10/17/09 Applications 17: 10 1: 30 10/17/2009 18: 40 15: 29 10/17/09 15: 50 100% 5: 07 0: 00 complete 10/17/09 18: 54 Test 10/17/09 Applications 18: 40 6: 00 10/18/2009 10/17/2009 0: 40 18: 54 10/17/09 17: 33 100% 3: 24 0: 00 complete 10/17/09 20: 24

Moving to a consolidated data center: Hour -by-Hour Rollback Plan Failed lift and ship (truck crashes, servers destroyed) Database failure Storage failure 3 days Larry Pathrow Request new Storage device 4 hours Install and prepare new array. (initialize the replacement, Larry Pathrow carve up the SAN Storage and reallocate the NAS Storage) 1 hour Verify/update the zoning on the switches if there are new Larry Pathrow WWNs on the storage processors. Back. Up Task(s) Karen Lusk Restore Data from Backup Tapes (4 TB data) Application Task(s) 5 mins Application Startup Application Owner 15 mins. Application Test Application Owner

Fast-forward two years… No MOU or SLAs No evidence that backups are taking place 7 -month turnaround on work requests for new servers 2 -month turnaround on work requests for additional storage No knowledge of who has access to data Annual audit points DR cost-prohibitive

Costs 399 PC’s and Laptops “help desk” 357 Mailboxes 30 Network Printers 26 Servers Storage $37, 422 $5, 354 $5, 244 $1, 569 $17, 776 $28, 340 Total monthly bill for June, 2011 $90, 969

Benefits Relief – no servers in building COOP advantage No midnight calls when the AC fails Shift in focus to business needs Information Management Data Stewardship Accountability Systems Longitudinal Data System

The Changing Infrastructure for K -12 systems in Washington PTAC Webinar, September 16, 2011 18

Washington’s K-12 Education System Fast Facts ◦ ◦ ◦ ◦ ◦ Number of Schools / Districts / ESDs: 2, 253 / 295 / 9 Number of K-12 students: 1, 023, 000 Number of K-12 school staff: 90, 000 Number of SEA employees: 400 staff located in a building built in 1895 State Funding in K-12: More than $6 B annually LEAs have Local control Elected Superintendent Dedicated K-20 network and State Government Network Multiple billion dollar budget gaps over the past several years September 2011 19

The Business Case to Transform the State’s IT Infrastructure Why? ◦ ◦ ◦ IT asset value in excess of $1 B State and local governments have to live within their means No growth in near future Opportunities for improving service to constituents Need to be more “Green” How? ◦ ◦ ◦ Leverage economies of scale to drive down costs Fully utilize assets Standardize infrastructure Consolidate data centers Use the state’s buying power to purchase commoditized products Cloud computing September 2011 20

The Business Case to Transform the State’s IT Infrastructure New WA State Data Center (SDC) and Office Building ◦ $255 million dollar construction cost ◦ Consolidates at least 32 data centers ◦ RFI in Summer 2010 for vendor information on a “turnkey” approach for data center operations ◦ New Data Center will have four data halls within the 50, 000 sq. ft. SDC ◦ Office Building completed in June 2011 ◦ SDC is partially operational serving building staff ◦ Plans to migrate agencies to the new data center have lost momentum due to the reorganization of the state’s IT department September 2011 21

Current strategies to mitigate impact to the WA SEA Engage with agency leadership, legal and financial staff and legislators early and often with the K-12 requirements and business cases Discuss with WA LEAs and other SEAs on lessons learned o Participate with the other WA state agencies and the central IT department in the planning process o o NCES Forum, MIS Lift and Shift or “End of Life” replacement migration Server virtualization project Negotiate preliminary agency migration sequence Review and provide feedback to preliminary documents Utilize sound and rigorous project management practices September 2011 22

OSPI today after virtualization September 2011 23

Planned strategies to mitigate impact to the WA SEA Key items on the project plan ◦ Review policy, security and legal requirements and perform gap analysis WA State Policies and Standards: Personnel Security, Physical Security, Data Security, Network Security, Access Security, Records Retention ◦ Stakeholder Management Coordinate with the K-12 application development staff Coordinate with OSPI program staff and user community Peer communication in a shared environment ◦ Patch hardware with the latest software updates ◦ Clean up folders and files ◦ Freeze environment September 2011 24

Strategies to mitigate impact to the WA SEA Key items on the project plan (continued) ◦ Update IT Portfolio and other as-is and to-be documentation ◦ Conduct proof-of-concept projects, pilots or other phased-in approaches ◦ Develop test plans ◦ Develop cut-over and support plan with updated SLAs ◦ Update and exercise Business Continuity Plans ◦ Implement a safety net Full backups Develop roll-back plans Get another set of eyes on the plans and documentation September 2011 25

Contact Information Peter Tamayo, Chief Information Officer ◦ peter. tamayo@k 12. wa. us ◦ (360) 725 -6134 September 2011 26

Data Center Consolidations – Final Thoughts • Efforts are partnerships! • Work with your state attorney general to ensure compliance with federal, state, and local laws. • Contact the Family Policy Compliance Office should you need assistance. 27

PTAC Help Desk & Website Send PTAC your questions on privacy, confidentiality, and data security related to longitudinal data systems (LDSs) Contact the Help Desk o Privacy. TA@ed. gov o Toll Free Phone: 855 -249 -3072 o Toll Free FAX: 855 -249 -3073 Get copies of PTAC resources, join our email list @ http: //nces. ed. gov/programs/ptac 28

Data Center Consolidation Webinar 09. 16. 11 Thank You for Participating Data Center Consolidation Considerations & Best Practices