IBM e Server Linux on z Series Module

IBM e. Server™ Linux on z. Series Module 13: Cloning Linux Images on z/VM 2004 IBM Corporation

IBM e. Server™ Objectives § Objectives: 4 Describe why cloning is useful 4 List two reasons cloning is used 4 Describe how the file system should be split for cloning 4 Explain the main consideration in setting up a shared file system for administration and maintenance 4 State one file that needs to be altered 4 List two things that a clone should be able to alter 4 List two areas in which consistency is important 4 Describe the utility and application of policies needed by users 4 Describe the process of updating a clone 4 Understand how a Linux-centric approach to cloning can be achieved 2 © 2004 IBM Corporation

IBM e. Server™ Why Is Cloning Necessary? § System administrators need to be able to quickly bring up dozens (or even hundreds) of new Linux guest machines, or clones. § The process of repeatedly installing similar Linux images from scratch can quickly become tedious. § System administrators need a simple, efficient, and repeatable process for creating these images. § Cloning gives you a unique opportunity to save disk space by using shared directories and installations. 3 © 2004 IBM Corporation

IBM e. Server™ Cloning Steps 1. 2. 3. 4. 5. 6. Create a common build that can be used as a template to quickly create a new image. Copy and duplicate the guest’s disks. Copy and configure one Linux image; when satisfied with it make it into the master copy or “golden image. ” Copy the golden image to produce the next Linux image. Once that image is created, you can modify the new image, or clone, for the specific needs of the user. Finally, in cloning you need to copy the virtual machine definitions and the disks. This can be easily automated with z/VM and tools like DIRMAINT, so you will not be responsible for the z/VM setup unless you are the system administrator. 4 © 2004 IBM Corporation

IBM e. Server™ Linux Setup § The first step in setting up the Linux cloning environment is to decide where the system will be split into a read-write and a read-only part § The easiest way to do this is to just move the /opt and /usr directory tree onto a read-only file system § Here is an example showing the space usage from the “/” directory: root: / # du -h -s * 6. 2 M bin 3. 3 M 24 k dev 22 M 20 k home 9. 5 M 12 k media 12 k 514 M proc 876 k 32 k tmp 697 M boot etc lib mnt root usr 0 0 16 k 105 M 9. 7 M 21 M cdrom floppy lost+found opt sbin var 5 © 2004 IBM Corporation

IBM e. Server™ Linux Setup Hints 4 Ignore directories that don’t have install files because these will probably not be programs (user files/data), and therefore not part of your standard build 4 Only 9% of the data are in directories other than /opt and /usr 4 The root user shares 91% of the system as read only 4 Place system data on one minidisk, then for each image simply define a read-only link to it (each clone will end up using around 1500 cylinders) 6 © 2004 IBM Corporation

IBM e. Server™ Linux Setup § Effects on maintenance and administration The larger the portion of the system that is kept read-only, the less damage users can do by mistake 4 The % of operating system kept read only can even be limited down to the clone-specific level if you are making one clone at a time. 4 – For example, you might decide that one clone user should be allowed access to change networking configurations, while most others should not. Only files that the user will need to write to must be copied to a private read-write file system (/lib, /bin, /usr, /sbin, /var and parts of /etc) 4 This makes it much easier to put update rpms into these systems, as this replaces or adds files not only in /usr and /opt, but also in /lib, /bin, or /etc 4 7 © 2004 IBM Corporation

IBM e. Server™ System setup drwxr-xr-x lrwxrwxrwx drwxr-xr-x dr-xr-xr-x lrwxrwxrwx drwxr-xr-x Lrwxrwxrwx drwxr-xr-x lrwxrwxrwx 2 3 6 25 1 6 4 2 11 43 1 11 4 1 17 1 root root root root root root root root 4096 May 7 17: 12 4096 May 13 13: 25 12288 May 13 11: 56 4096 May 13 13: 27 8 May 10 09: 48 4096 May 8 13: 59 4096 May 7 17: 07 4096 May 10 09: 48 4096 May 7 17: 11 0 May 13 13: 29 8 May 10 09: 49 4096 May 13 16: 25 4096 May 10 13: 01 7 May 10 09: 49 4096 May 13 16: 26 8 May 13 16: 25 8 bin boot dev etc home -> rw/home/ lib media mnt opt proc root -> rw/root/ rw sbin tmp -> rw/tmp/ usr var -> /rw/var/ © 2004 IBM Corporation

IBM e. Server™ RW Directory Tree § Your rw directory would contain 4 drwxr-xr-x 2 root 4096 May 13 13: 29 lib 4 drwxr-xr-x 2 root 4096 May 13 13: 29 bin 4 drwxr-xr-x 2 root 4096 May 13 13: 29 usr 4 drwxr-xr-x 2 root 4096 May 13 13: 29 sbin 4 drwxr-xr-x 2 root 4096 May 13 13: 29 dev 4 drwxr-xr-x 4 root 4096 May 13 13: 27 etc 4 drwxr-xr-x 3 root 4096 May 7 17: 07 home 4 drwxr-xr-x 14 root 4096 May 13 16: 25 local 4 drwx------ 4 root 4096 May 13 09: 21 root 4 drwxrwxrwt 8 root 4096 May 13 13: 35 tmp 4 drwxr-xr-x 17 root 4096 May 7 17: 12 var 9 © 2004 IBM Corporation

IBM e. Server™ Benefits of a Three-Copy Setup § Maintain three copies of the Linux image First copy is for the master instance, as the working operating system 4 Second copy is used for testing changes in the configuration and maintenance scripts 4 The third copy is the master copy that is linked to the clones 4 § When a configuration change or maintenance job is thoroughly tested, the test disks are copied to the master copy disks § The clones pick the new disks up the next time they are booted § This also makes it possible to have several copies of the disk sets, and therefore, if a maintenance update breaks the system, a system administrator can easily switch back to a previous service level by linking to an older set of disks (the backup copy) 10 © 2004 IBM Corporation

IBM e. Server™ Alter Setup Files § Normal Linux installations are not set up to be divided into read-only and read -write portions § Create two copies of the rc. config file One in the /etc directory 4 The other, the renamed version, for the clone in the rw/etc directory 4 § Make the following changes: Only the rc. config-share file uses the /etc/init. d/boot startup script 4 So change: . /etc/rc. config to. /etc/rc. config-shared 4 Disable the Su. SEconfig scripts because they will not work in this kind of environment: 4 – ENABLE_SUSECONFIG="no“ 4 For the TCP/IP dynamic configuration, one line has to be added at the end: –. /rw/etc/rc. config-tcpip 4 Parameters for the syslog have to be changed in the script so that the syslog uses a device in /rw/dev: SYSLOGD_PARAMS="-p /rw/dev/log" 11 © 2004 IBM Corporation

IBM e. Server™ Changes to /etc/init. d/boot script § Comment out the command to give rw acess to the root file system: # mount -n -o remount, rw / § Place a symbolic link from /etc/mtab to /proc/mounts: lrwxrwxrwx 1 root 12 May 13 10: 48 mtab -> /proc/mounts (using the “link” command) § Comment out the line that deletes leftover files from the last time the system was up: # rm -f /etc/mtab* /etc/nologin /fastboot § Comment out the line to initiate the zic program to verify the time zone setting: # /usr/sbin/zic -l $TIMEZONE § The rc. config file is already missing in the Su. SE 8. 0 professional version 12 © 2004 IBM Corporation

IBM e. Server™ Warning: Consistency § Multiple Linux guests become impossible to manage if they are all different 4 Distributional consistency 4 Functional consistency among guests § Create naming conventions and keep similar files in the same places 13 © 2004 IBM Corporation

IBM e. Server™ Warning: Policies § Create formal service level agreements 4 Keep consistent times of operation/maintenance 4 Guaranteed levels of performance – Software levels – General agreement, not one for each server 4 Maintain a general service agreement that specifies user/administrator responsibilities 14 © 2004 IBM Corporation

IBM e. Server™ Written policies § For cloning a new server § For fixing

IBM e. Server™ Customers Responsible if They… § § Need to own the R/W

IBM e. Server™ System Administrator’s Jobs § Have all agreements, forms and policies readily accessible (on the web if possible) § Maintain policy on how to obtain a server § Maintain all how-to documents for users 4 Setting up KDE or GNOME 4 Creating users and groups (IDs and passwords) 4 Keeping Linux secure (setting up firewalls) 4 Running Samba 4 Running Apache, Tomcat, etc 17 © 2004 IBM Corporation

IBM e. Server™ Maintaining Clones § With private read-write file system and a read-only file system maintenance is different § Must deal with rpm (Red. Hat package Manager) tool when it tries to install into the read § § § -only part of the directory tree See http: //www. rpm. org/ if you are unfamiliar with rpm Use the --relocate option to change the path to the local directory 4 Example: rpm -i --relocate /usr=/usr/local --relocate /etc=/usr/local/etc mysql. rpm --badreloc Translates to /usr will be translated to /usr/local and /etc to /usr/local/etc If package can not be relocated, --badreloc option is used for enforcement RPM pre and post install scripts can be printed using: rpm -qp mysql. rpm --scripts 4 Automatic execution of the scripts will fail 4 Pre- and post-install actions have to be done manually 4 Execution of scripts disabled with the command option --noscripts. 18 © 2004 IBM Corporation

IBM e. Server™ Golden Image § After one clone is completely configured life becomes easy!!! § Use it as the golden image and only the unique identifiers have to be changed § Just copy virtual machine definitions and disks 19 © 2004 IBM Corporation

IBM e. Server™ RPM databases § Need two, one for maintenance updates of the

IBM e. Server™ Updating Clones § For updates or installs of packages use --dbpath option to use the shared § § § § rpm database All updates of the system disk should always be done on the private copy z/VM directory entries can be changed to point to the new master disk Different versions and maintenance levels can be kept for system disk During reboot, guest will be configured to use the new z/VM guest definition Clone will be brought automatically to the new maintenance level Update private copy of the rpm database 4 Use option to only update the database (without changing database) 4 Done automatically at startup Must add jobs to run updates for example 4 run-maintenance 4 run-post-scripts 21 © 2004 IBM Corporation

IBM e. Server™ Monitoring Maintenance § Must check that jobs did not fail (usually performed by the system § § § administrator) Jobs send email to maintenance email address Here is a small sample shell script: 4 #! /bin/sh 4 mutt -s test maintenance@central <<EOF 4 Maintenance 2002052201 successfully applied! 4 EOF Send output of maintenance jobs to central syslog facility 4 /etc/syslog. conf must be changed: local 0. * @central Routes all syslog entries to syslog server central Message lines sent from maintenance jobs using logger command: logger -p local 0. info -t clone 5 “Hey how u doin? " 22 © 2004 IBM Corporation

IBM e. Server™ Flexibility of the Clone § Clone user must be able to install additional application and packages 4 Needs an Add-on for individual startup script 4 Simply add script /etc/init. d/run-rw to Su. Se startup § User needs to create new users and passwords 4 Files with this info moved to /etc/rw directory 4 To prevent corruption of the file, lock files and backup files are created prior to changing /etc 23 © 2004 IBM Corporation

IBM e. Server™ z/VM setup § The following few slides contain some configurations that need to be done through z/VM § To minimize the effort of defining the z/VM guest definitions for the Linux clones, always map disks to the same addresses for the guests § In the setup used here, the network is implemented using one Linux system as the router This system exclusively owns one GB Ethernet card and is connected to the Linux farm clones via IUCV 4 Generation of network definitions for a new clone can be easily automated with some scripting 4 24 © 2004 IBM Corporation

IBM e. Server™ Change the TCP/IP setup § TCP/IP setup should not be accessible by each clone user § Keep some scripts in the read-only file system that automatically configures § § TCP/IP at startup Find the z/VM guest name Using the name, enter the file on the shared read-only file system where, in this example, for each clone there is an entry containing the guest name, the IP address, and the peer IP address Network connection will be an IUCV connection to another Linux router system These values are then used to override the initial information in /rw/etc/rc. config to /rw/etc/rc. config-tcpip 25 © 2004 IBM Corporation

IBM e. Server™ Perl Script § Code used to detect VM name and settings: 4 4 4 4 #!/usr/bin/perl use strict; my($HCP) = '/sbin/hcp'; my($HOSTNAMES) = '/usr/var/config/hosts'; my($CONFIG_LOCAL) = '/rw/etc/rc. config-tcpip'; my($DEBUG) = 0; my($guest) = ''; my($ip) = ''; my($peer) = ''; my($GUESTNAME) = split(/ /, `$HCP query userid`); print("HOST: $GUESTNAMEn") if $DEBUG; open(HOSTNAMES, "< $HOSTNAMES") || \ die("Could not open $HOSTNAMES: $!n"); while(<HOSTNAMES>) …continued on next page 26 © 2004 IBM Corporation

IBM e. Server™ Perl Script continued …continued from previous page 4 {chomp; 4 ($guest, $ip, $peer) = split; 4 last if ($guest eq $GUESTNAME); } 4 close(HOSTNAMES); 4 print("Guestname: $guestn") if $DEBUG; 4 print("IP: $ipn") if $DEBUG; 4 print("Peer: $peern") if $DEBUG; 4 open(CONFIG_LOCAL, "> $CONFIG_LOCAL") || \ 4 die("Could not open $CONFIG_LOCAL: $!n"); 4 print(CONFIG_LOCAL \ 4 "#Automatically generated configuration file. Do not edit!n"); 4 print(CONFIG_LOCAL qq{NETCONFIG="_0"n} ); 4 print(CONFIG_LOCAL qq{IPADDR_0="$ip"n} ); 4 print(CONFIG_LOCAL qq{IFCONFIG_0="$ip pointopoint $peer"n} ); 4 close(CONFIG_LOCAL); 27 © 2004 IBM Corporation

IBM e. Server™ TCPIP Config § § § With IUCV, use z/VM’s TCP/IP as virtual router Linux guests must be subnet of z/VM IP address Need one IP address for each virtual server Need only one additional address as “peer” address for all servers No broadcast IP with this method IUCV /sec, IUCV approx 400 mb/sec 28 © 2004 IBM Corporation

IBM e. Server™ Multiple Stacks § Multiple stacks for testing, production, alternate methods; better performance on multiprocessor machine § Don’t have to take down all Linux servers when testing methods on one server § Optimum is 10 -12 Linux instances per TCPIP stack 29 © 2004 IBM Corporation

IBM e. Server™ Disk Setup § Disk Mount: 4 Disks common to all clones should be listed in /etc/fstab and automatically mounted by the Su. SE startup scripts 4 /etc/fstab stays on the read-only root file system 4 Mount private disks and disks that are not common to all clones 4 Common disk address ranges are used for private disks 4 Read disk information and mount the disks later in the boot process § Note: all disks need not be configured with same virtual addresses in z/VM guest configuration § In Linux kernel, parmfile and configuration disks can be dynamically added to a script using: echo "add device range=devno-range" >> /proc/dasd/devices 30 © 2004 IBM Corporation

IBM e. Server™ “Rules of thumb” for including and defining paging and SPOOL space and defining the DIRMAINT DASD pool: § For approximately each 40 to 45 Linux images (per z/VM), add one 33903 for paging space 4 Ensure all the paging volumes are added to the SYSTEM CONFIG file, which is found on the MAINT CF 1 disk § You need 500 cylinders of 3390 space for each Linux image with a shared read-only (R/O) /usr and /usr/src and DASD for swap 4 300 cylinders for a read-write (R/W) 4 200 cylinders for swap 4 One 3390 -3 pack can contain 6 cloned images. 31 © 2004 IBM Corporation

IBM e. Server™ “Rules of thumb” continued R/O /usr and /usr/src and using VDISK for swap • This is 300 cylinders for a R/W / • One 3390 -3 pack can contain 10 to 11 cloned images. You need 1000 cylinders of 3390 space for each Linux image with a shared R/O /usr and a R/W /usr/src. The 800 cylinders break down to: • 300 cylinders for a R/W / • 500 cylinders for a R/W /usr/src • 200 cylinders for swap. One 3390 -3 pack can clone 3 images The console logs of the LXOPR, LXMSTR, LXADMIN, and LXCNTL virtual machines are spooled for debugging purposes. • You should add 300 cylinders for SPOOL space to hold these console logs 32 © 2004 IBM Corporation

IBM e. Server™ Linux-centric Approach to Cloning § This technique only applies to on demand cloning (one clone at a time), not continuous cloning (mass cloning; for example, 100 identical images at a time) § This is a simple and flexible methodology that is least dependent on z/VM automation § For more z/VM-centric methodologies see: Linux on IBM e. Server z. Series and S/390: Cloning Linux Images in z/VM Redbook 33 © 2004 IBM Corporation

IBM e. Server™ This Approach Includes: § Two z/VM guests that manage the entire cloning process 4 Master. Linux 1: a Linux guest that owns the master root and R/O /usr filesystems used to create the clones, and scripts to customize the networking info for each clone and initiate the DDR script 4 Cloner: a CMS guest which actually performs the DDR copy of the model root file system owned by Master. Linux 1 to the clone’s minidisk § A handful of flat files that hold variable data during cloning § Availability of the DIRMAINT z/VM tool 34 © 2004 IBM Corporation

IBM e. Server™ First: Set up the Environment § Install Linux on a z/VM guest that will act as the Master. Linux 1 system (you already have an image configured) 4 Define three mountpoints: /, modelroot, /usr § Use the z/VM DDR utility to create a “golden copy” that you want to clone (as explained in previous slides) 4 You can actually have as many golden copies as you DASD permits – This doesn’t have to be done via DDR § Mount the golden copy to a directory called modelroot 4 Modelroot contains everything but /usr 4 /usr will be shared by all clones in read-only mode 4 You can create different types of modelroot directories for different types of clones (model. WAS, model. DB 2, model. Samba, etc) 35 © 2004 IBM Corporation

IBM e. Server™ First: Set up the Environment continued § Install the clone-eth base script in /root/bin on Master. Linux 1 § Prepare the modelroot directory to be used by the cloning script 4 Define networking flat files in a directory called /modelroot/etc and /modelroot/etc/sysconfig/network 4 These are basically the standard networking config files like hosts, routes, etc, but with substitution variables that will be used by clone-eth § Make sure the cpint module is loaded, and the hcp command defined 4 These are open source and come with SUSE 4 They let you issue z/VM CP commands from Linux Guest § Run zipl to change the modelroot Linux parmfile to indicate the /usr filesystem will be used read-only § Define and start the clone CMS guest, defining it as a secondary console for all Linux machines 36 © 2004 IBM Corporation

IBM e. Server™ Second: Begin Cloning § Telnet or ssh into Master. Linux 1 § From the /root/bin directory, execute the clone-eth script § High level view of the script: 4 Clone-eth() – PROMPT user for input on networking configuration for clone – READ the input – UPDATE /modelroot/etc/model-xxx files with input, and SAVE into /modelroot/etc/xxx. new – MOVE /modelroot/etc/xxs. new files into /modelroot/etc/xxx files – Use HCP command to start DUP exec on CLONE CMS Machine – In a loop, ping new clone until it responds – Tell user that the replication is now complete 37 © 2004 IBM Corporation

IBM e. Server™ Pros of this technique: § Pros: Very simple 4 Flexible: You can use the same approach to clone different distributions, different type of Linux guests (WAS guests, DB 2 guests, etc) 4 Can be used to share any portion of the filesystem you want with the clones (/, /usr/src, etc) 4 – Also, since z/VM definitions for each clone are made by hand, they can be unique to each user (some could use VDISK for swap, others DASD, etc) – Not tied to use of DIRMAINT, so can choose to use that or to update z/VM user directory by hand – Network info can be determined on the fly as each new clone is created • Doesn’t have to be predetermined or pre-loaded 38 © 2004 IBM Corporation

IBM e. Server™ Cons of this technique: § Not completely automated. All z/VM definitions for each clone must be made by hand § No web front end for initiating the cloning § Requires cpint module to be loaded on Master Linux image (but not on clones) § Can only clone guest at a time 39 © 2004 IBM Corporation

IBM e. Server™ Conclusion § Cloning is a process whereby you copy and duplicate a guest’s disks to produce a new Linux image. § Cloning is used when: 4 many images need to be brought up, and starting from scratch becomes tedious and time consuming 4 you are in a real-time environment and demand requires a new server to be brought up as soon as possible § The file system should be divided into a shared and local area. Everything that can possibly be shared should be placed there to save disk space. 40 © 2004 IBM Corporation

IBM e. Server™ Conclusion 4 The main consideration in setting up a shared file system from an administration/maintenance standpoint is that the larger the portion of the system that is kept read-only, the less damage users can do to themselves by mistake. Also, the closer to a standard build the clones remain, the easier it is for the administrator to maintain the environments. 4 Several modifications must be made when setting up a shared system, such as to the startup files. 4 Modifications must also be made to the TCP/IP settings so that the main setup is not accessible to every clone and the new parameters, such as IP address and ID, can be changed. 4 The clone user must be able to install additional applications and packages and new user IDs and passwords 41 © 2004 IBM Corporation

IBM e. Server™ Conclusion 4 Consistency is crucial in a large shared environment. It is necessary to have one distribution used on all guests as well as having similar functions and file systems. 4 Policies are needed to create formal service level agreements that tell guests what performance they can expect and what they are allowed to do with their resources. It needs to be clearly stated who is responsible for what in a given scenario. 42 © 2004 IBM Corporation

IBM e. Server™ Conclusion § The process of updating a clone begins with maintaining two rpm databases; one for maintenance updates of the read-only shared system and one for the private use of the clone § Updates of the system disk are done on the private copy; then z/VM directory entries can be changed to point to the new master disk. During reboot, guests are configured to use the new z/VM guest definition. Clones automatically brought to the new maintenance level § Update the private copy of the rpm database without changing the database; then update at startup by adding a few jobs 43 © 2004 IBM Corporation