Catastrophic Hardware Failure Recovery with Exchange Server 2003

Catastrophic Hardware Failure Recovery with Exchange Server 2003 Eileen Brown (eileenb@microsoft. com) IT Evangelist Microsoft UK http: //blogs. msdn. com/eileen_brown

Topics What’s new in Exchange 2003 and Windows 2003 Disaster Recovery Questionnaire Active Directory Overview and Disaster Recovery Exchange 2003 Overview and Disaster Recovery Database Disaster Recovery SP 1 update

What’s New In Exchange 2003 Database snapshot through Volume Shadow Copy Services Recovery Storage Group RPC/HTTP support for Outlook 2003 IPSec support between front-ends and back-end clusters IIS 6 runs in Dedicated Mode Clustering

What’s New In Exchange 2003 Linked Value Replication Support Enhanced Management Tools Inet. Org. Person-enabled mailboxes Native Mode Dependencies Exchange Store automatically ignores and removes zombie ACE entries Query-based Distribution Groups (QDGs)

Windows Server 2003 AD Less traffic between replicas Create branch office replica from media Active Directory improvements Backup through Volume Shadow Copy Service Linked Value Replication When in forest and domain functional level 2 Group Membership replication improvements Inter site replication topology generator Domain rename (without Exchange installed)

Disaster Recovery Questionnaire Is your data stored off-site? Time to reconstruct your business data? What is the cost per hour of server downtime? What is critical data? Have your configurations been mapped? Exch. Dump. exe and Clustool. exe What training is required? Who else knows how to recover your data?

Disaster Recovery Active Directory Recovery

Active Directory Database Ntds. dit – the database Edbxxxxx. log – transaction logs Edb. chk – checkpoint file Res 1. log and Res 2. log – reserved log files Logs are of fixed size (10 mb for AD) Three categories of directory data are replicated between domain controllers: Domain data (accounts…) Configuration data (list of domains…) Schema data (definition of all objects…)

Active Directory Backup System State Components: System Start-up Files (boot files) System registry Class registration database of COM+ SYSVOL What Is A Good Backup? System State, system disk contents, and the SYSVOL folder Consider tombstone age set in Active Directory Default is 60 days If data older than the tombstone lifetime - restore disallowed Backup data from a DC can only be used to restore that DC

Types Of Disaster Determine the type of disaster Database corruption Damaged disks DC hardware failure Software failure – server cannot boot Data corruption Accidentally deleted object from directory Methods to restore Windows 2003 DC: Re-installation Backup

Restore Through Re-Installation New DC receives the same name as failed DC: Remove the ntds. DSA object of the failed DC using ntdsutil Use ntdsutil “metadata cleanup” command connect to the remote DC remove orphaned DC

Removing An Orphan ntds. DSA Object

Restore From Backup Non-Authoritative Restore Default method for the restoration of Active Directory DC is then updated using normal replication techniques Authoritative Restore ntdsutil

Authoritative Restore Follow non-authoritative restore before initiation object attributes version number Incremented entire directory subtree individual object Used when human error is involved Accidentally deleted a number of objects which cannot be recreated easily

Authoritative Restore Considerations Impact on Group Membership significant issue possible loss of group membership information Depends upon which objects replicate first The User object OR The Group object Impact on Trust relationship Password Reset http: //support. microsoft. com/default. aspx? scid=KB; [LN]; 840001

Recovering A Global Catalog Server Restore from backup or: Add additional GC Create branch office replica from media - dcpromo /adv Restore GC onto different hardware - issues Different HALs Incompatible Boot. ini file Different network or video cards

AD Domain Rename (Rendom. exe) Forest mapping tool – Rendom. exe /list Forest constraints - you cannot: Change which domain is the forest root domain. Drop domains from or add domains to the forest. Reuse a domain name. Domain rename operation is not supported if Exchange 2000 is deployed Detected by domain rename tool. Now supported with Exchange 2003/SP 1 and XDRFixup. exe tool

AD Forest Recovery

AD Forest Recovery Call Microsoft PSS first !!! Restoring forest DC’s Steps involved: Recover the forest root domain first Recover the remaining domains in the forest ONE DC from backup DC’s through reinstallation

AD Forest Recovery - High Level Steps Identify single DC for restore Shut down ALL DC’s Recover first DC in root domain 1. Primary SYSVOL restore, disable GC flag 2. Configure DNS 3. Raise value of RID pool by 100, 000 cn=RID Manager$, cn=System, dc= 4. Seize all (FSMO) roles (ntdsutil) 5. Clean metadata of ALL DC’s in the root (ntdsutil)

AD Forest Recovery - High Level Steps Recover FIRST DC in the root domain (cont. ) 6. Delete server and computer objects of all other DC 7. Reset the computer account of the DC twice (netdom) 8. Reset the krbtgt password twice (ADUC) 9. Reset the trust password twice (netdom) Restore FIRST DC in each other remaining domains Primary SYSVOL restore for domain Same steps as previously (domain wide) Enable GC flag DO FRESH BACKUP Install other DC’s using dcpromo

AD Forest Recovery - High Level Steps White paper http: //download. microsoft. com/download/win 2000 srv/Utility/1. 001/NT 5/E N-US/forestrecovery. exe AD Fast recovery (VSS) – white paper available

Exchange 2003 Disaster Recovery

Disaster Recovery Issues Resource intensive Excessive downtime for clients Maintenance of recovery forest/server Goals Reduce resource requirements Perceived client availability Product supported/Integrated functionality

Where Is Exchange Information Stored? Registry settings and metabase System state backup AD Directory Objects store “Recipient” information Users, Groups, and Contacts. Replicated to GCs Most Exchange information placed on existing objects are replicated between Global Catalogs AD Configuration Exchange System Objects Public Folder Directory entries Active Directory Connector (ADC) settings

Levels Of Disaster Recovery Restoring mailboxes Recovery Storage Group / Separate server / 3 rd party backup utility Restoring one or more Exchange databases Backup software Restoring multiple databases - single storage group Backup software Complete disaster - full server recoveries

Disaster Recovery What You Need Similar replacement hardware Windows 2003/Exchange 2003 install CDs, hotfixes System state backups IIS Metabase, AD, Registry, Certificate Server Exchange database backups Event ID 8019 (VSS/System State simultaneous backup fails) System/application drive backups

Disaster Recovery Steps To Recover Reconfigure hardware and drives Reinstall operating system Restore drive backups Restore System state Install Exchange Setup /Disaster. Recovery Restore Exchange databases

Move Exchange To New Hardware (Exchange 2003 = GC) If server is a domain controller: Deletion of computer account / NTDS Settings Object DCPROMO /FORCEREMOVAL – “NEW” Keeping the same server name Take existing Exchange 2003 computer offline Reset existing Exchange 2003 computer account Bring the new computer online using same name Log on using Exchange 2003 Full Administrator account Exchange 2003 Setup /disasterrecovery Mount stores - check client connectivity and mail flow.

Using Exchange 2003 Stand-By Recovery Server What you need System State backup C: Windows folder backup Exchange 2003 database backups Steps to recover Start stand-by server Restore %System. Root% folder and System State Run Exchange 2003 setup in disaster recovery mode Restore databases Recovery Using Images Drive signature issue prevents logon after recovery Fix using Q 249321 and Q 223188

Alternate Server Restores (Before) Active Directory Forest Restore Production Server Copy . pst Restore Server Copy

Recovery Storage Group RSG per Server/ Information Store Restore mailbox DBs from same SG Restore SG/DBs from same AG User mailboxes remain disconnected Only MAPI protocol supported Restores default into RSG Active/Passive one restore storage group per EVS ONE recovery storage group per cluster supported

Benefits Do not need to maintain restore forest/server WARNING: If E 2 K db is restored to EX 2003 No changes to backup processes RSG the db will be upgraded to EX 2003 Utilise existing resources version and won't be mountable back to E 2 K Minimises downtime without data loss box. Access to disconnected mailboxes (Ex. Merge) WARNING: If db already exist in RSG all additional db’s must be from the same Allows single item/mailbox storage group as the first dbrestore!!!

Recovery Of Other Exchange 2003 Services Connectors Lotus Notes Novell Group. Wise Exchange Calendar Connector Custom OWA Clusters Volume Mount Points Majority Node Set (MNS) Clusters Resource Kit clusdiag tool

Exchange 2003 Connectors Lotus Notes connector Back up the Lotus Notes client software Configuration file Notes. ini Back up conndata and its subdirectories Novell Group. Wise connector Ensure that Group. Wise Gateway Network Service can be restored correctly Back up conndata and its subdirectories Install Exchange 2003 in disaster recovery mode using setup /disasterrecovery Clean directories from temporary work files *. seq and *. ck before you start connector

Custom OWA Themes Outlook Web Access in Exchange 2003 supports the concept of 'Themes'. To create your own: OWA themes, Custom Form for logon – logon. asp STEP 1 On the front-end and back-end, create a directory in …exchwebthemes, e. g. called “XBOXtheme" STEP 2 Copy or create new versions of 9 images STEP 3 Edit the CSS file to come up with your own colors and styles. STEP 4 back-end, add reg key …

Custom Xbox OWA Theme

Exchange 2003 Clustering What to back up Cluster Administrative software Quorum System State Exchange 2003 Server Cluster Disaster Recovery types Recover shared disk resource (Clusdb – Chkxxx. tmp Q 224999) Restore Quorum Resource Replace a damaged node Restore an entire Exchange 2003 cluster Majority Node Set (MNS) Cluster, ASR for cluster Windows 2000 to Windows Server 2003 rolling upgrades supported Support for Mount Points

ASR For Clusters Automated System Recovery – ASR can completely restore a cluster in a variety of scenarios, including damaged or missing system files complete OS reinstallation due to hardware failure a damaged Cluster database, and changed disk signatures (including shared)

Removing orphaned Exchange Server Active Directory Sites and Services snap-in Services: Microsoft Exchange: organisation_name: Administrative Groups: Servers Delete same named server object If cluster is gone you cannot delete Exchange Virtual Server resources from AD Bind to DC using LDP: ConfigurationServicesMicrosoft ExchangeOrganizationAdministrative GroupServers Right click: Delete orphan EVS entries No option of Disaster Recovery Setup for EVS

Database Disaster Recovery

Logical Versus Physical Corruption Three layers of corruption that can occur Page level ESE level Store level To remove corruption Restore an uncorrupted backup of the database Repair the database Expunge the corrupted pages from the database Salvage data and generate a new database

File Header Information Signatures for crossmatching logs and databases Database state Log files needed Backup information (last time done, type done) Shadow Headers Database header changes are not logged Crash Recovery Data locations In a crash, all cache buffer data is lost Transaction log data can reconstruct the cache Damaged database files? Damaged log files?

Errors 1018 and 1019 Error 1018: JET_err. Read. Verify. Failure Bad checksum / Wrong page number Hardware / Firmware File system corruption How serious are 1018 Errors? During normal operation (somewhat serious) During startup (likely fatal) During backup (may be minor) Error 1019: JET_err. Page. Not. Initialized What causes Error 1019? Special case of error 1018 (page is replaced with zeroes) Bad page links

Errors 1022 and 1216 Error 1022: JET_err. Disk. IO Disk I/O failure File damage or truncation File locked by another process Anti-virus software Error 1216 (Q 296843) files in the database's running set are missing or have been replaced When storage group starts system analyses header information If logs are missing: Restore the database from backup Repair the database by using ESEUTIL /P followed by ESEUTIL /D and ISINTEG -fix Q 296843 – more details

Best Practices

Best Practices - Hardware Protect databases with RAID 5, RAID 0+1 or RAID 10 Protect logs with mirroring (separate volume) Keep disks < 50% full Deploy fastest possible backup solutions Never use circular logging for mailbox storage groups

Replica Based Recovery Writer Microsoft Exchange Freeze write operations to ensure data consistency Volume Shadow Copy Service Requestor Third Party Vendor Integration Module for Exchange VSS HW Provider SG-1 Data Source SG-1 Data Clone SG-1 Log Source SG-1 Log Clone Exchange Storage Group – Data and Log Source and Clone Targets

Microsoft IT Design Goals 8000 200 MB mailboxes (4000 per EVS) 1. 2 IOPS per mailbox peak + 20% buffer 16 ms read, 3 ms write response at peak load Scalable clustered Exchange configuration Deploy transportable VSS functionality Windows Server 2003 RTM Exchange Server 2003 RTM/SP 1 72 GB 10 K disks for production data and log 146 GB 10 K disks for clones

Best Practices - Conclusion Develop backup and restore strategies rendom. exe /list, Exch. Dump. exe, Ex. Deploy to map environment Give dedicated administrator backup and restore responsibilities Keep three copies of the backup media. Back up an entire volume Perform a trial restoration periodically Test recovery procedures. Windows startup disks function correctly. Uninterruptible power supply (UPS) on the computers running Windows Server Test your disaster recovery plan. Test restores from daily, weekly, and monthly backup media.

SP 1 Enhancements

Service Pack 1 Self-heal removes 40% of 1018 errors No need for Ex. Merge – RSG Wizard Right click on Storage Group to merge data Change of database format Database is not upgraded at once – only pages that are changed getting upgraded Support for i. SCSI and NAS devices Once used SP 1 must continue with SP 1

SP 1 – Self-Healing Error 1018 - the most familiar (and dreaded) Exchange database error New Error Correcting Code (ECC) algorithm Exchange automatically corrects damage to the database file if caused by a "bit flip" 40% of damaged Exchange pages are caused by bit flips

Upgrading DB to SP 1 tips If databases with 1018 errors are upgraded to SP 1: ECC information needed to correct an error does not exist on previously written database pages To upgrade all pages in a database at once Take the database offline and defragment Eseutil /D [db_filename] Will not be able to replay logs Make new full backup and mark old sets invalid Transaction log replays

Restoring SP 1 DB You can restore a pre-SP 1 backup set to an SP 1 server Not the other way round Pre SP 1 Exchange can't understand the ECC data on the page interpreted as damaged

SP 1 Dialtone SP 1 enhances Recovery Storage Group interface with automated merge of data between: Dialtone database Last backup

DIALTONE MERGE SP 1

Conclusion Review your disaster recovery plan when upgrading / deploying Exchange 2000/2003 Backup all data needed for full recovery Verify disaster recovery and restore plans through drills Read Exchange 2003 mailbox and disaster recovery whitepapers regularly Audit your Best Practices Request Microsoft PSS Operations Assessment

© 2004 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.