CIT 470 Advanced Network and System Administration Backups

CIT 470: Advanced Network and System Administration Backups CIT 470: Advanced Network and System Administration 1

Topics 1. 2. 3. 4. 5. 6. 7. Backup Decisions Types of Backups Backup Hardware Backup Software Snapshots and CDP Cron Backup Security CIT 470: Advanced Network and System Administration 2

Backup Decisions Why? Why are you backing up data? What would happen if you lost data and didn’t back up? What types of data do you have? What? What to back up—entire system, or specific filesystems? What OS to backup? What other things to backup—MBR, LVM? When? When is the best time to backup? How often? Where? Where will backup occur? Where to store backup volumes? Who? Who is going to provide backup system? Who will do backups? How? How are you going to do backups? Tape, mirrors, off-site, etc. CIT 470: Advanced Network and System Administration 3

Why Backups? 1. 2. 3. 4. 5. Accidental deletions. Hardware failures. Data corruption. Security incidents. Plan for the worst. 1. System catches fire. 2. Fire spreads to replicated systems. 3. Sprinklers destroy backup system in data ctr. CIT 470: Advanced Network and System Administration 4

What to Backup? Backups for your backups – Be able to restore your backup server (software + backup volume db) in case it’s down. Which peripherals? – How many drives per server? – What is the capacity of each drive? How were they partitioned? – Drive partitions must be same as before disaster for restore from backup to work. – fdisk –l – lvmcfgbackup CIT 470: Advanced Network and System Administration 5

Filesystem / Data Types Operating System Standard OS image on server. Software + config files specific to server. Data files specific to server. CIT 470: Advanced Network and System Administration 6

Backing Up Selected FS Saves media space and network traffic – But OS is small compared to data today. Harder to administer – Must remember or document which fs to backup for each server. Easier to split up between volumes – Can easily distribute backup of a server across different backup volumes on a per fs basis. Worst case – Forget to backup an important filesystem. CIT 470: Advanced Network and System Administration 7

Backup Entire System Complete automation – Create script to parse /etc/fstab and LVM, then backup every disk filesystem. – Do this once, then it works on all servers. Worst case – Increase network traffic by a few percent. CIT 470: Advanced Network and System Administration 8

Backup Types Image level – Backup raw disk partition or entire disk. – Back up every byte on drive, used or not. – Use compression to eliminate GBs of zeros. – Cannot restore individual files. Filesystem level – Backup files within filesystem. – Backup tool must understand filesystem. – Can restore files, no backup of unused blks. CIT 470: Advanced Network and System Administration 9

Backup Types Full backup Complete copy of all files from a particular time. Backup: slow, requires high capacity. Restore: fast, simple. Incremental backup Storage of changed files since last backup. Backup: fast, may store multiple per tape. Restore: slow, complex (requires multiple tapes) CIT 470: Advanced Network and System Administration 10

Capacity Planning: Space Partition: 40 GB Full backup every week. Daily incremental backups. 50% full now, grows 2 GB per day Tape capacity needed Day 1: 40 GB Day 2: 2 GB … Day 7: 12 GB Day 8: 40 GB CIT 470: Advanced Network and System Administration 11

Capacity Planning: Time Fileserver: 4 TB Full backup must finish overnight (8 hours) Tape drive: 40 MB/s = 144 GB/hr = 1. 15 TB/night Need 4 tape drives running simultaneously. Additional concerns: Network performance btw file & backup servers. Does any capacity need to be reserved for restores? Actual performance vs. manufacturer specs. CIT 470: Advanced Network and System Administration 12

Capacity Planning: Media How much media do you need? Determined by policy and schedule. How long are full backups kept? How often are incrementals recycled? How often are tapes moved off-site? CIT 470: Advanced Network and System Administration 13

Choosing a Backup Drive 1. 2. 3. 4. 5. 6. 7. Reliability Flexibility Transfer speed Time-to-data Capacity Compatibility Cost CIT 470: Advanced Network and System Administration 14

Reliability MTBF – Remember that drives fail faster than MTBF claims early and late in lifespan. – Talk to people who have used system. Duty cycle – Expected usage per day. – 40% duty cycle = 10 hours per day – MTBF based on listed duty cycle. Hard drives are more reliable than tape. – Closed system protects from contaminants. CIT 470: Advanced Network and System Administration 15

Flexibility means – Able to respond to different data rates. – Can be used in different ways. Tapes aren’t very flexible – Typically require a standard data rate. – Slower/faster rates result in I/O errors. – Can only read/write sequentially. Disks are very flexibile – – Random access medium. Can change data rates rapidly. Combine with RAID or LVM for capacity or perf. Virtual tape software allows disks to appear as tape. CIT 470: Advanced Network and System Administration 16

Transfer Speed Compare native sustained transfer rates – Transfer rates often assume compression – Burst or synchronous rates are temporary, bestcase scenarios. Disks are much faster than tape. CIT 470: Advanced Network and System Administration 17

Time to Data – How long to load a volume + – Seek to appropriate place on volume + – Start reading data. Load time can include – Time to manually find and load tape. – Time for tape robot to locate and load tape. Most restores are for a few files. – User deletions. – Time to Data matters more than Transfer Rate CIT 470: Advanced Network and System Administration 18

Capacity Want one backup to fit on single volume. – Easy to manage than backups across multiple volumes. – Tape capacity grows slower than disk capacity. Cost – Lots of small backups to a single volume. – Reduces number of volumes to purchase + store High capacity is faster – Fewer volume switches when backing up. CIT 470: Advanced Network and System Administration 19

Compatibility Want to be able to restore everything. – If new system incompatible with old, must transfer old backups to new format. Use a single format for easy of management. CIT 470: Advanced Network and System Administration 20

Backup Media Flash Memory Very expensive, small media, personal use only. Super floppies ZIP 750 MB, small capacity, high $/GB media. Optical CD-R cheap drives, slow + small (650 MB). DVD-R cheap drives, slow + small (4. 7 GB). Ultra Density Optical (UDO) expensive but larger (60 GB). Hard disk Large capacity (1 TB), bulky, low $/GB media. Tapes Large capacity (800 GB), low $/GB media; expensive drives. CIT 470: Advanced Network and System Administration 21

D 2 D 2 T Backup data first to disk – Take advantage of fast disk speeds. – Complete backups within nightly window. Copy backup disks to tape – Copy backup data from disks to tape. – Disks aren’t in production, so this can be slow. Reuse backup disks each night. CIT 470: Advanced Network and System Administration 22

Current Tape Formats LTO (Linear Tape Open) Uses Ultrium format tapes. LTO 4: 800 GB capacity, 120 MB/s, 49 s seek Super DLT (Digital Linear Tape) Backward compatible with DLT formats. DLT-S 4: 800 GB capacity, 60 MB/s, 79 s seek Super. AIT (Advanced Intelligent Tape) SAIT-1: 500 GB capacity, 30 MB/s, 45 s seek CIT 470: Advanced Network and System Administration 23

Common Tape Features Form Factor 5. 25” FH SCSI drives Media are ~1/2” wide tape stored in cartridges. Hardware compression Usually cited as 2: 1, some cite higher. Depends heavily on nature of data stored. Future Roadmaps Plans to double capacity in next few years. CIT 470: Advanced Network and System Administration 24

Hardware vs Software Compression – Compress data via software before writing. – Can use high compression tools like 7 zip, bzip 2. – Lowers amount of data xfer across network. – Higher CPU usage. Hardware Compression – Compress using specialized hardware on tape. – Does not require additional CPU usage. – Increases throughput of drive. CIT 470: Advanced Network and System Administration 25

Tape Autochangers Stackers Works sequentially through a stack of tapes. Autoloader / Jukebox Provides random access to set of tapes. Library / Silo Multiple drives w/ random access to set of tapes. May incorporate bar code reader, ethernet, etc. CIT 470: Advanced Network and System Administration 26

Backup Software OS Provided (backup individual systems) cpio, dump, tar, ntbackup Open source (backup servers) AMANDA Bacula Commercial (backup servers) Tivoli Storage Manager (IBM) Veritas Storage Manager CIT 470: Advanced Network and System Administration 27

Provided Backup Software UNIX/Linux – – tar cpio dump dd Windows – ntbackup – Windows Restore Mac. OS – ditto CIT 470: Advanced Network and System Administration 28

Windows Restore Points Auto backup of registry + critical files – Must have System Restore enabled. – Done automatically + can create manual too. – Useful when software install has corrupted. Recovering Windows with a Restore Point – Boot into safe mode. – Select “Restore my computer to an earlier time” – Choose date from list of restore point dates. – If that doesn’t work, reboot, try an older one. CIT 470: Advanced Network and System Administration 29

Backup Software Feature tar cpio dump List files on vol Slow—search entire backup (tar –t) Slow—search entire backup (cpio –it) Fast—index at front (restore –t) Incremental backup Easy (-newer) Must use find Easy (set level) List files as backed up tar cvf cpio –v Only after backup Compatibility Multi-platform with ASCII header Readable between some platforms. Backup specific files Yes Whole fs only Stop reading tape after restore file is found No No Yes Likelihood file exists in TOC but not in backup Low Medium (TOC is made before backup) Recovery from I/O errors No No Yes CIT 470: Advanced Network and System Administration 30

Open Source Backup Tools Amanda – Single master backup server. – Simple, fast, uses native backup tools. Bacula – Client/server backup system. – More advanced features but slower than Amanda. Backup. PC – Web-based so works with any client OS. – Backs up PCs and laptops to disk on server. CIT 470: Advanced Network and System Administration 31

Snapshots Virtual read-only copy of filesystem – Snapshot has same contents that filesystem has when snapshot was made. – Snapshot uses pointers to data and copy-on-write to avoid making a copy of entire fs. – Snapshots require ~1% of fs size, depending on updates. CIT 470: Advanced Network and System Administration 32

Snapshot Applications Quick restore times – Snapshots take seconds to create, restore from. – Won’t help you if disk or other hardware fails. – Snapshot lifetime could be short as 1 m or as long as a few days for this purpose. Staging for backups – Snapshot filesystem before starting backup. – Files on snapshot do not change during backup. – Snapshot lifetime is how long it takes to backup. CIT 470: Advanced Network and System Administration 33

LVM Snapshots LVM can create snapshots of logical volumes. lvcreate -L 500 M -s -n backup /dev/db/db 1 – Creates snapshot (-s) volume named backup – Volume is snapshot of /dev/db/db 1 LV – Can make 500 M of changes. What happens if >500 M of changes? – LV can’t receive copies of old data if changes are made to original logical volume. – Ensure 500 M is more than enough space for changes made during lifetime of snapshot. CIT 470: Advanced Network and System Administration 34

Continuous Data Protection Copy every file change to backup server. – Stores changes in a log like RCS or a database. – Can restore to any point of time. Near-CDP – Snapshots + replication. – Does not provide a log, so can only restore to saved snapshots, not to any change like CDP. CIT 470: Advanced Network and System Administration 35

Backing up Virtual Machines Back up VMs as physical machines – Connect VM to your standard backup system. – Have to configure backups for each VM. Back up VM files – Can back up all VMs on host at once. – VM files are constantly changing, so either • Suspend VM • Snapshot filesystem with VM files CIT 470: Advanced Network and System Administration 36

Automation The key to efficiency and reliability. Use cron instead of manually backing up. Single tapes require manual media change. Tape libraries automate this process. Other automated tasks Monitoring (up/down, disk space, security) Logs (rotation, monitoring) File distribution CIT 470: Advanced Network and System Administration 37

Cron Performs tasks at scheduled times. Crontab files specify schedule of tasks root: /etc/crontab users: /var/spool/crontabs/* Cron may log activities and errors. Timing limitations: Runs tasks (if any) every minute. Does not perform scheduled tasks if system down. May or may not perform tasks on DST transition. CIT 470: Advanced Network and System Administration 38

Crontab Format minute hour day month weekday user command Examples 30 4 * * 0 root yum –y update 3 * * root (cd /var/www; make) 20 1 * * * root /usr/local/rot-logs CIT 470: Advanced Network and System Administration 39

Managing Automated Tasks Divide by time: Hourly, daily, weekly, monthly tasks Crontab uses run-parts meta-script: 17 * 25 6 47 6 52 6 *** **7 1** root run-parts --report /etc/cron. hourly run-parts --report /etc/cron. daily run-parts --report /etc/cron. weekly run-parts --report /etc/cron. monthly Add crons by placing script in time directory. Add random delay if all hosts share same crontab. CIT 470: Advanced Network and System Administration 40

Backup Security Tape security Tapes contain all of your important data. Data isn’t secure unless tapes are secure. Solutions: tape vault, encrypted tapes. Backup server security Has read access to all important data. If backup server isn’t secure, data isn’t secure. Solutions: integrity checking, least privilege Restore process Who can request files to be restored? Where will restored file be placed? What will its ACL be? CIT 470: Advanced Network and System Administration 41

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. AIT, AIT Forum, http: //www. aittape. com/index. html, 2006. Lynne Avery, “A Brief History of Tape, ” Exabyte white paper, http: //www. kontron. com/techlib/whitepapers/A_brief_history_of_tape. pdf, 2000. Aeleen Frisch, Essential System Administration, 3 rd edition, O’Reilly, 2002. LTO, http: //www. ltotechnology. com/newsite/index. html, 2006. Peter Mc. Gowan (ed), Quantum DLTape Handbook, http: //downloads. quantum. com/sdlt 320/handbook. pdf, 2001. Evi Nemeth et al, UNIX System Administration Handbook, 3 rd edition, Prentice Hall, 2001. Shelley Powers et. al. , UNIX Power Tools, 3 rd edition, O’Reilly, 2002. W. Curtis Preston, Backup & Recovery, O’Reilly, 2007. Quantum, “Tape Storage Automation, ” http: //www. dlt. com/storage/whitepapers/quantum/dlt/Tape%20 Storage%20 Automation. pdf “The Tao of Backup, ” http: //www. taobackup. com/ Wikipedia Contributors, “Magnetic Tape, ” http: //en. wikipedia. org/wiki/Magnetic_tape, 2005. Elizabeth Zwicky, “Torture Testing Backup and Archive Programs, ” ftp: //ftp. berlios. de/pub/star/testscripts/zwicky/testdump. doc. html, 1991. CIT 470: Advanced Network and System Administration 42