51e9bd2a2c0c8f19d5472400b8a2cff1.ppt
- Количество слайдов: 82
Backup Methods For a Hot Site Dieter W. Storr Los Angeles Times 23 August 2005 Dieter W. Storr -www. storrconsulting. com
B/R Methods n n n n Existing Backup Method Experiences Mirroring or Replicating Fast Copy of Data Proposals and Costs Future Technology Lessons learned 23 August 2005 Dieter W. Storr -www. storrconsulting. com 2
Existing Backup Method • From disk (databases) • Copy to • 3490 / 3590 -1 / VTS • Then, copy to • 3590 -1 (cartridge) 23 August 2005 Dieter W. Storr -www. storrconsulting. com 3
ADABAS 6. 2. 2 Back-up at LA Times 23 August 2005 Dieter W. Storr -www. storrconsulting. com 4
B/R Methods “Companies that relied on tape or on third-party provider found in many cases they had difficulty meeting their recovery time objectives. ” Source: http: //www. drj. com/articles/spr 02/1502 -07. html 23 August 2005 Dieter W. Storr -www. storrconsulting. com 5
B/R Methods “Flaws in tape-based data backup may be leaving enterprises without key information and could lead to legal exposure under emerging laws such as Sarbanes-Oxley, say data backup and recovery experts. “ Source: 15 Apr 2004 | Search. Security. com 23 August 2005 Dieter W. Storr -www. storrconsulting. com 6
B/R Methods n n n In a survey of 500 IT departments completed … found that as many as 20% of routine, nightly backups fail to capture all data. 40% of IT managers had been unable to recover data from a tape when they needed it More than 23% sought to use data stored on tape backups more than 20 times in a year Source: 15 Apr 2004 | Search. Security. com 23 August 2005 Dieter W. Storr -www. storrconsulting. com 7
B/R Methods Are tapes really so bad? LA Times experiences? 23 August 2005 Dieter W. Storr -www. storrconsulting. com 8
Tape Problems 1 November 2002: n Six tape drive errors n Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 9
Tape Problems 24 March 2003: n n Only two channel paths per tape controller were provided Slow restore time 23 August 2005 Dieter W. Storr -www. storrconsulting. com 10
Tape Problems 5 October 2003: n n 3590 tape drives were not defined to DFSMS (SMS) ADABAS restore and application test cancelled 23 August 2005 Dieter W. Storr -www. storrconsulting. com 11
Tape Problems 6 December 2003: n n VTS problems with GDG datasets End-user functions couldn’t be tested 23 August 2005 Dieter W. Storr -www. storrconsulting. com 12
Tape Problems 5 August 2004: n n Restore jobs had to wait for an input tape that was being used by another restore job Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 13
Tape Problems 30 October 2004: n n Packages didn’t arrive in time, due to a thunderstorm that affected Fed. Ex delivery Major delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 14
Tape Problems 30 October 2004: n n Automated tape library experienced unit address problems during the restore process Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 15
Tape Problems 30 October 2004: n n VTS logical tapes were not shipped to Wood Dale (HSM level 2, SAR level 2) Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 16
Tape Problems 30 October 2004: n n Confusion about when to load DRP 1 and DRP 2 tapes, before or after IPL Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 17
Tape Problems 30 October 2004: n n ICIS libraries were not backed up to tape Application tests were not possible 23 August 2005 Dieter W. Storr -www. storrconsulting. com 18
Tape Problems 8 December 2004: n n n Load problems Tapes were loaded before IPL and not after IPL Major delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 19
Tape Problems 8 December 2004: n n Experienced problems when trying to restore MIG 1 data, e. G. DRADABC 0 job Major delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 20
Tape Problems 8 December 2004: n n n Recall sent by Fed. Ex tapes to Sun. Gard One damaged package arrived without tapes Restored DATA one generation back (-1) System was generation (0) 23 August 2005 Dieter W. Storr -www. storrconsulting. com 21
Tape Problems 21 March 2005: n n Level 2 tapes for VTS not being sent off-site (but have been on the list) Application team couldn’t test all data 23 August 2005 Dieter W. Storr -www. storrconsulting. com 22
Tape Problems 5 August 2005: n 3590 -1 cartridges ejected, not found n n DSS 8370 W - TMS SHOWS TAPE N 00318 OUT OF AREA “DRP 1”, SLOT 00031 Delay 23 August 2005 Dieter W. Storr -www. storrconsulting. com 23
Time Warner employee data missing May 2, 2005: 5: 51 PM EDT NEW YORK (CNN) - Time Warner Inc. said Monday that data on 600, 000 current and former employees stored on computer backup tapes was lost by an outside storage company and that the Secret Service is now investigating. Dieter W. Storr -- 23 August 2005 www. storrconsulting. com 24
Lost Backup Tape Held Ameritrade Client Data Wednesday, April 20, 2005 - LA Times … package was damaged during shipping between vendors …. . fourth tape is still missing…… The tapes may have included customers’ Social Security numbers …. . 23 August 2005 Dieter W. Storr -www. storrconsulting. com 25
Info On 3. 9 M Citigroup Customers Lost Monday, June 6, 2005 – CNN. COM Citigroup, the nation's biggest financial services company, said that UPS lost the tapes while shipping them to a credit bureau in Texas. 23 August 2005 Dieter W. Storr -www. storrconsulting. com 26
Costs for Tape Backups n n n Sun. Gard recovery services Offsite tape storage Tape handling Shipping per test Special extra pick-ups Yearly $150, 000 23 August 2005 Dieter W. Storr -www. storrconsulting. com 27
Costs n Not capable to restore one day n n n $$ ? ? ? Last December: 2 weeks to rebuild manually (? ) customer tables Does it make sense to restore more than 2 days back ? ? 23 August 2005 Dieter W. Storr -www. storrconsulting. com 28
Costs Example: 20 employees x $140 per day x 10 days = $28, 000 And they couldn’t work on other projects $140 is based on $51, 100 yearly income 23 August 2005 Dieter W. Storr -www. storrconsulting. com 29
Quantitative Risk Analysis Single Loss Expectancy n n n SLE = Single Loss Expectancy EF = Exposure Factor, for example 50% or. 50 AV = Asset Value, for example $1, 000 SLE = EF * AV SLE =. 5 x $1, 000 = $500, 000 23 August 2005 Dieter W. Storr -www. storrconsulting. com 30
B/R Methods Reducing tapes 23 August 2005 Dieter W. Storr -www. storrconsulting. com 31
B/R Methods Reducing tapes n n Stacking datasets to 3590 -1 cartridges Using Delta Save Facility from ADABAS 23 August 2005 Dieter W. Storr -www. storrconsulting. com 32
B/R Methods Reducing tapes n n Using Forward Index Compression (FIC) from ADABAS Using larger block size for 3590 tapes = 256 K, supported by ADABAS 23 August 2005 Dieter W. Storr -www. storrconsulting. com 33
Delta Save Facility (DSF) 23 August 2005 Dieter W. Storr -www. storrconsulting. com 34
Delta Save Facility 23 August 2005 Dieter W. Storr -www. storrconsulting. com 35
B/R Methods Forward Index Compression Rochester Gas & Electric Space savings: n Normal Index: 37% - 55% n Upper Index: 21% - 69% Within an index block the part of the index value that is identical to the forward part of the previous index value is suppressed. 23 August 2005 Dieter W. Storr -www. storrconsulting. com 36
B/R Methods IBM Magstar 3494 / Virtual Tape Server (VTS) Sun. Gard LA Times 23 August 2005 Dieter W. Storr -www. storrconsulting. com 37
B/R Methods VTS problems LA Times: n Completion code A 78 RC 18 n We switched from VTS to 3590 -1 cartridges 23 August 2005 Dieter W. Storr -www. storrconsulting. com 38
B/R Methods VTS problems Virginia Information Technologies Agency: n Ran 2003/2004 into the same problem system completion code A 78 RC 18 n We … converted … to 3490/3590 physical tapes n Problem solved 23 August 2005 Dieter W. Storr -www. storrconsulting. com 39
B/R Methods Disk to Disk n Mirroring n n n Hardware Software Replicating n Software 23 August 2005 Dieter W. Storr -www. storrconsulting. com 40
B/R Methods – Enterprise Server NT / 2000 / XP UNIX 23 August 2005 Hot Site Dieter W. Storr -www. storrconsulting. com 41
B/R Methods – Open System Hot Site 23 August 2005 Dieter W. Storr -www. storrconsulting. com 42
B/R Methods Marty Stewart Disaster Recovery Manager An. Med Health: “…we’d rather have a server that’s running slower than having no server at all. ” 23 August 2005 Dieter W. Storr -www. storrconsulting. com 43
Disk Mirroring ASSO Benefits n n DATA Asynchronous disk mirroring can provide better physical protection by supporting extended physical distances. No loss of committed transactions in synchronous storage (mirroring/RAID) on a CPU failure 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA 44
Disk Mirroring ASSO DATA Limitations n n n No protection from data corruption Secondary site is not guaranteed to be transitionally consistent, in the case of asynchronous mirroring. Client application must be re-started after failure and need to be aware of failure 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA 45
Disk Mirroring ASSO DATA Limitations n n Synchronous mirroring and RAID devices can add overhead to application performance. Redundant/specialized high availability hardware/software can be expensive and restricted to use for backup purposes only. 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA 46
Disk Mirroring ASSO DATA Limitations n n Secondary copy of data is not available for use – low hardware utilization. Need to replicate everything on disk, no selectivity of data replication ASSO DATA 23 August 2005 Dieter W. Storr -www. storrconsulting. com 47
Example For Disk Mirroring Back Up / Hot Site S/390 UNIX EMC 5700 SRDF remote mirrored synchronized OC-3 link 12 -15 miles SRDF remote mirrored synchronized EMC 5700 S/390 23 August 2005 Main Platform Dieter W. Storr -www. storrconsulting. com UNIX 48
B/R Methods n n n Can we buy used Enterprise Servers? Yes…. . and inexpensive OP system is free for D/R Search for “selling used mainframes, ” for example: http: //www. used-line. com/fdc 3236 -find-dealer. htm http: //www. azure. co. uk/ etc. 23 August 2005 Dieter W. Storr -www. storrconsulting. com 49
Dedicated line broadband speeds and prices T-1 - 1. 544 megabits per second (24 DS 0 lines) Ave. cost $400. -$650. /mo. n T-3 - 43. 232 megabits per second (28 T 1 s) Ave. cost $6, 000. -$16, 000. /mo. n OC-3 - 155 megabits per second (100 T 1 s) Ave. cost $20, 000. -$45, 000. /mo. n OC-12 - 622 megabits per second (4 OC 3 s) no price n OC-48 - 2. 5 gigabits per seconds (4 OC 12 s) no price n OC-192 - 9. 6 gigabits per second (4 OC 48 s) no price Source: http: //www. infobahn. com/research-information. htm prices updated: 12 May 2005 n 23 August 2005 Dieter W. Storr -www. storrconsulting. com 50
Peer-to-Peer Remote Copy Extended Distance (PPRC-XD) PPRC = 60 miles - PPRC-XD = continent Flash. Copy ESS Shark Also see Time. Finder from EMC 23 August 2005 Dieter W. Storr -www. storrconsulting. com - IBM ESS DASD - HDS also support PPRC 51
External Back-up Systems Fast Copy of Data n Snapshot n n n No data movement A virtual copy by copying pointers Copy Process n n 23 August 2005 Physical copy async. from the log. copy No impact on applic. on the original data Dieter W. Storr -www. storrconsulting. com 52
External Back-up Systems Fast Copy of Data n Specific Hardware Required n n Software works only with the hardware Work on Volume Level n Some snapshot only tools work also on dataset level 23 August 2005 Dieter W. Storr -www. storrconsulting. com 53
Snapshot & Physical Copy IBM n n Hardware: Enterprise Storage Server Software: Flash. Copy http: //www. share. org/proceedings/sh 98/data/S 3087. PDF EMC 2 n Hardware: Symmetrix Remote Data Facility n Software: EMC Time. Finder http: //www. emc. com/interactive_center/media/timefinder/tf_no. RC. html 23 August 2005 Dieter W. Storr -www. storrconsulting. com 54
Flash Copy 23 August 2005 Dieter W. Storr -www. storrconsulting. com 55
How It Works Pre-defined time window Suspend Read / update Read only: update requests are queued Resume Read only Read / update snap Source Data Snapshot Physical Backup Source: SAG ADADBS TRANSACTIONS SUSPEND, TTSYN=60, TRESUME=120 23 August 2005 Dieter W. Storr -www. storrconsulting. com 56
Replication Benefits n Warm standby systems can be configured over a Wide Area Network, providing protection from site failures. n Ability to more quickly swap to the standby system in the event of failure, as backup database is already on-line. n Data corruption is typically not replicated as transactions are logically reproduced rather than I/O blocks mirrored. 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA WORK DATA ASSO 57
Replication Benefits n Automatic switch over for clients using a switching mechanism, no client restart needed. n Originating applications are minimally impacted as replication takes place asynchronously after commit of the originating transaction. n The warm standby database is available for read-only operations, allowing better utilization of backup systems. 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA WORK DATA ASSO 58
Replication ASSO DATA WORK Benefits n Ability to resynchronize and easily switch back to primary system when it becomes available without loss of data. WORK DATA ASSO 23 August 2005 Dieter W. Storr -www. storrconsulting. com 59
Replication Limitations n Warm standby system will be out-ofdate by transactions committed at the active database that have not been applied to the standby. n Protection is limited to components supporting Warm Standby (e. g. DBMS data sources may be protected but file systems may not be supported). 23 August 2005 Dieter W. Storr -www. storrconsulting. com ASSO DATA WORK DATA ASSO 60
Entire Transaction Propagator n The Entire Transaction Propagator allows for asynchronous data replication. n Replicated data can be updated and synchronized with master data at user specified intervals. 23 August 2005 Dieter W. Storr -www. storrconsulting. com 61
ADABAS Data Replication n n n n Logical dissemination of ADABAS Data to homogeneous or heterogeneous targets Near real time propagation Event driven at the Transaction level Implemented at the Database/file level for Store, Delete and Update commands Define Replication rules through subscriptions Minimal Impact on normal nucleus activity Strategic for Enterprise Data Sharing Replace Entire Transaction Propagator 23 August 2005 Dieter W. Storr -www. storrconsulting. com 62
ADABAS Data Replication z/OS Image A Target Field Origin Target File Target DBMS Field File Target DBMS z/OS Table Image B 23 August 2005 Image C Unix Server D Dieter W. Storr -www. storrconsulting. com 63
Possible Hot Site Solutions Enterprise Server Los Angeles Shark EMC Converter ESCON OC 3 Shark OC 3 EMC FICON Fiber Optic OC 3 EMC Own Enterprise Server Hot Site 23 August 2005 Dieter W. Storr -www. storrconsulting. com 64
Costs for Tape Backups n n n Sun. Gard recovery services Offsite tape storage Tape handling Shipping per test Special extra pick-ups Yearly $150, 000 23 August 2005 Dieter W. Storr -www. storrconsulting. com 65
Costs for Real Disaster n n n n Sun. Gard Declaration Fee D/R Site Daily Usage Fee Office Space Daily Usage Fee Work Group Declaration Fee Work Group Daily Usage Fee LAN Bridge Declaration Fee LAN Bridge Daily Usage Fee 30 Days $475, 000 23 August 2005 Dieter W. Storr -www. storrconsulting. com 66
Costs for Own Hot Site n n n n Used IBM Z 800 -0 X 2 Mainframe Used IBM 2105 -F 20 Shark Storage Used IBM 3494 Library, VTS and Tape Drives 3 rd Party Next Day HW Maintenance Printer and Terminal Controller Re-location Costs 3490 Tape Drive and Controller Re-location Costs Other Costs 1 st Year $520, 000 After 5 Years Total $735, 000 23 August 2005 Dieter W. Storr -www. storrconsulting. com 67
Costs for Own Hot Site 5 Years Sun. Gard = $750, 000 30 Days Real Disaster = $475, 000 5 Years Own Facility = 23 August 2005 Dieter W. Storr -www. storrconsulting. com $735, 000 68
Restore Times (Min) 23 August 2005 Dieter W. Storr -www. storrconsulting. com 69
Benefits of Own Hot Site n n n Financial savings > $150, 000 annually providing an almost 5% ROI Reduced recovery time Reduced impact due to road and airport closures Elimination of reliance on external vendors Mainframe and open system can use the same facility 23 August 2005 Dieter W. Storr -www. storrconsulting. com 70
Grid Computing virtual machine virtual memory virtual storage virtual I/O 23 August 2005 Dieter W. Storr -www. storrconsulting. com 71
23 August 2005 Dieter W. Storr -www. storrconsulting. com 72
Interface Grid Computing User Passkey Hardware Software Information Service 23 August 2005 Resource Broker Computer Element(s) Locations Replica Catalog Dieter W. Storr -www. storrconsulting. com Storage Element(s) 73
Grid Computing 23 August 2005 Dieter W. Storr -www. storrconsulting. com 74
Grid Computing n BBC builds distributed grid for content sharing (Gridcast). 62. 73. 167. 57/publicdocs/ppt/prompeg 307. ppt file: ///C: /My%20 Documents/Dieter/my%20 presentations/prompeg 307. ppt 23 August 2005 Dieter W. Storr -www. storrconsulting. com 75
Grid Computing For Backup? n Intra or Extra Grid? n Pull or Push? n Grid Software http: //www. gridcomputing. com/ http: //www. gridforum. org/ggf_grid_understand. htm http: //gridcafe. web. cern. ch/gridcafe/animations. html 23 August 2005 Dieter W. Storr -www. storrconsulting. com 76
Backup Methods Mostly used by other companies Source: DRJ Magazine n n VTS Disk to disk n Is more and more common for enterprise storage servers and AIX server technology, for example. Source: @server Magazine 23 August 2005 Dieter W. Storr -www. storrconsulting. com 77
B/R Methods Problems for other companies n n High third-party hot site costs, approx. $10, 000 - $70, 000 per month Restore time 24 -30 hours 23 August 2005 Dieter W. Storr -www. storrconsulting. com 78
How Far is ‘Far Enough? ’ (http: //www. drj. com/articles/spr 03/1602 -02. html) n n Alternate Facility Offsite Storage Facility Answer = 105 miles …so the survey 23 August 2005 Dieter W. Storr -www. storrconsulting. com 79
Lessons Learned (http: //www. drj. com/articles/spr 02/1502 -07. html) n Distance is key Streets, bridges, tunnels, airports are closed n Tape recovery is not effective n All applications are critical n Inconsistent back-up is no back-up at all 23 August 2005 Dieter W. Storr -www. storrconsulting. com 80
Lessons Learned (http: //www. drj. com/articles/spr 02/1502 -07. html) n n n People-dependent processes do not suffice Two sites are not enough People are hard to replace but information is irreplaceable 23 August 2005 Dieter W. Storr -www. storrconsulting. com 81
…. . we should have an excellent HOT SITE! 23 August 2005 Dieter W. Storr -www. storrconsulting. com 82
51e9bd2a2c0c8f19d5472400b8a2cff1.ppt