VIO Server architecture design and build Alain Lentini

e-business hosting services Some explainations of the presentation q VIOs architecture design : Ø Standard LVM mirroring for VIO rootvg (separate SCSI adapters in each VIO) Ø Both VIOs active for virtual SCSI (used for AIX partitions rootvg mirroring) Ø Active/passive model for Ethernet access Ø Mutual takeover (heartbeat) Ø Failure of one VIO transparent to partitions (a few IP packets lost) Ø Allows hot maintenance of VIOs Ø Each VIO on a different LAN switch Ø Multipathing for SAN access inside each VIO (sddpcm) + pathing failover in the AIX partition between the 2 VIOs (Mpio) Ø Various SAN ACCESS ( SDDPCM for Ess DS 8000 and SVC – RDAC ) 2 Alain Lentini / Co. C Unix Lge

e-business hosting services Virtualization of Rootvg Disk access 3 Alain Lentini / Co. C Unix Lge

e-business hosting services Rootvg disks access configuration VIOS 1 AIX client partition vtserv 1 hd 0 LVM partitionning hdisk 8 rootvg LV xxx VIOS 2 vtserv 1 hd 3 Pg. Sp LVM partitionning + mirroring vtserv 1 hd 2 vtserv 1 hd 1 LVM partitionning hdisk 8 hdisk 9 hdisk 0 hdisk 2 Vhost 7 Slot 34 SCSI adapter Vhost 6 Slot 32 Vscsi 0 Slot 32 Vscsi 2 Slot 34 hdisk 9 hdisk 1 hdisk 3 Vscsi 1 Slot 33 Vscsi 3 Slot 35 Vhost 7 Slot 35 Vhost 6 Slot 33 SCSI adapter POWER hypervisor serv 1 hd 0 lv serv 1 hd 2 lv hdisk 8 hdisk 9 naming convention - example of commands on server VIO 1 Even slots on VIOs 1 and odd slots on VIOs 2 mklv -lv serv 1 hd 0 lv vioserver 1 VG 30 G hdisk 8 mklv -lv serv 1 hd 2 lv vioserver 1 VG 20 G hdisk 9 mkvdev -vdev serv 1 hd 0 lv -vadapter vhost 6 -dev vtserv 1 hd 0 mkvdev -vdev serv 1 hd 2 lv -vadapter vhost 7 -dev vtserv 1 hd 2 4 Alain Lentini / Co. C Unix Lge serv 1 hd 1 lv serv 1 hd 3 lv hdisk 8 hdisk 9

e-business hosting services Rootvg disks configuration on vioservers On vioserver 1 Create the virtual server device in Web. Sm mkvg -f -vg vioserver 1 VG hdisk 1 hdisk 5 extendvg vioserver 1 VG hdisk 2 hdisk 6 Example for partition mopesaxsap 15 p mklv -lv saxsap 15 phd 0 LV vioserver 1 VG 30 G hdisk 1 mklv -lv saxsap 15 phd 2 LV vioserver 1 VG 20 G hdisk 5 mkvdev -vdev saxsap 15 phd 0 LV -vadapter vhost 0 -dev vtsaxsap 15 phd 0 mkvdev -vdev saxsap 15 phd 2 LV -vadapter vhost 1 -dev vtsaxsap 15 phd 2 on vioserver 2 Create the virtual server device in Web. Sm mkvg -f -vg vioserver 2 VG hdisk 1 hdisk 5 extendvg vioserver 2 VG hdisk 2 hdisk 6 Example for partition mopesaxsap 15 p mklv -lv saxsap 15 phd 1 LV vioserver 2 VG 30 G hdisk 2 mklv -lv saxsap 15 phd 3 LV vioserver 2 VG 20 G hdisk 6 mkvdev -vdev saxsap 15 phd 1 LV -vadapter vhost 0 -dev vtsaxsap 15 phd 1 mkvdev -vdev saxsap 15 phd 3 LV -vadapter vhost 1 -dev vtsaxsap 15 phd 3 5 Alain Lentini / Co. C Unix Lge

e-business hosting services Rootvg disks configuration on client partition 6 Alain Lentini / Co. C Unix Lge

e-business hosting services result after the creation of 4 virtual scsi client adapter On client partition Lspath Enabled hdisk 0 vscsi 0 Enabled hdisk 1 vscsi 1 Enabled hdisk 2 vscsi 2 Enabled hdisk 3 vscsi 3 hdisk 0 U 9117. 570. 65 A 1 DA 0 -V 5 -C 36 -T 1 -L 8100000 Virtual SCSI Disk Drive hdisk 1 U 9117. 570. 65 A 1 DA 0 -V 5 -C 37 -T 1 -L 8100000 Virtual SCSI Disk Drive hdisk 2 U 9117. 570. 65 A 1 DA 0 -V 5 -C 38 -T 1 -L 8100000 Virtual SCSI Disk Drive hdisk 3 U 9117. 570. 65 A 1 DA 0 -V 5 -C 39 -T 1 -L 8100000 Virtual SCSI Disk Drive 7 Alain Lentini / Co. C Unix Lge

e-business hosting services Shared Ethernet Adapter configuration 8 Alain Lentini / Co. C Unix Lge

e-business hosting services Shared Ethernet Adapter principle 1/2 Note: Note that a numerically lower priority receives a higher overall priority. 9 Alain Lentini / Co. C Unix Lge

e-business hosting services Create the Virtual LAN adapter on the Hmc In our example, the VIO is secondary server for the vlan 99 : trunk priority = 2 sur vio 1 We share the Networkload between the 2 vios : 1 vlan on each VIO managed with the priority. 10 Alain Lentini / Co. C Unix Lge

e-business hosting services create the virtual adapter for control channel This will generate a device on the VIO server ( cfgdev or cfgmgr ) To find the device, use the slot number in the command lsdev –slots : U 9117. 570. 65 A 1 DA 0 -V 2 -C 5 Virtual I/O Slot ent 8 U 9117. 570. 65 A 1 DA 0 -V 2 -C 3 Virtual I/O Slot ent 9 11 Alain Lentini / Co. C Unix Lge

e-business hosting services SEA command creation q mkvdev -sea ent 0 -vadapter ent 10 -defaultid 101 –attr ha_mode=auto ctl_chan=ent 13 ( vlan 9 for exemple ) sea = ent 14 q Check the priority in VIOs errpt : E 136 EAFA 0519102508 I H ent 12 BECOME PRIMARY q ent 0 = network physical interface ( vlan 101 ) q ent 10 = virtual adapter generated by hmc menu config q ent 13 : control, channel q lsdev –slots q U 9117. 570. 65 A 1 DA 0 -V 2 -C 2 Virtual I/O Slot ent 10 q U 9117. 570. 65 A 1 DA 0 -V 2 -C 5 Virtual I/O Slot ent 13 12 Alain Lentini / Co. C Unix Lge

e-business hosting services Shared Ethernet Adapter principle 2/2 AIX Client LPAR 1 AIX Client LPAR 2 en 0 (if) ent 0 (virt) Hypervisor PVID=100 • 2 client partitions are represented only PVID=100 VLAN 1 PVID=100 VIOS 1 ent 1 (virt) ent 2 (SEA) PVID=10 Control Channel – VLAN 10 PVID=10 ent 3 (virt) en 2 (if) ent 0 (phy) en 2 (if) Primary Secondary Untagged ent 1 VIOS 2 (virt) ent 2 (SEA) ent 0 (phy) Untagged Ethernet Switch 13 This diagram is simplified to show only the logical implementation of the solution: Untagged VLAN Alain Lentini / Co. C Unix Lge Ethernet Switch • 1 customer VLAN and the Heartbeat VLAN are represented when the real system contains 4 VLANs. • The real config would contain 3 physical port per VIOs: one for each VLAN. • The VLAN highavailability is transparent to the client partition. It's provided by the SEA failover feature of the VIOs.

e-business hosting services SEA detailed configuration example 14 Alain Lentini / Co. C Unix Lge

e-business hosting services San access configuration 15 Alain Lentini / Co. C Unix Lge

e-business hosting services Data access on SAN through sddpcm on VIOs principles 1/2 Attention: When using multiple Virtual I/O Servers, and exporting the same LUN to the client partitions, only mapping of hdisks is supported (logical volumes are not supported). Attaching IBM Total. Storage Enterprise Storage Servers to the Virtual I/O Servers, only MPIO using the SDDPCM is supported. 16 Alain Lentini / Co. C Unix Lge

e-business hosting services Data access on SAN through RDAC on VIOs principles 2/2 On both Virtual I/O Servers, you have to set the hdisk attribute reserve_policy to no. This attribute prevents the Virtual I/O Server setting a reservation flag on the disk at the time of mapping. The MPIO component on the client partition will take the responsibility of managing the disk. 17 Alain Lentini / Co. C Unix Lge

e-business hosting services Data access on SAN through sddpcm (no mirroring) VIOS a AIX client partition LVM partitionning vtserv 1 hd 8 SDDPCM hdisk 8 hdisk 9 Default PCM + MPIO Hdisk 70 hdisk 71 hdisk 70 hdisk 71 hdisk 8 FC adapter port 2 vtserv 1 hd 9 datavg LV xxx vtserv 1 hd 9 port 1 VIOS b Vhost 24 Slot 44 Vscsi 4 Slot 44 hdisk 9 hdisk 8 hdisk 70 hdisk 71 hdisk 9 Vscsi 5 Slot 45 FC adapter Vhost 24 Slot 45 port 1 port 2 POWER hypervisor Fabric 1 hdisk 70 hdisk 71 Bay 1 18 Alain Lentini / Co. C Unix Lge Fabric 2 naming convention - example of commands on server VIO 1 mkvdev -vdev hdisk 70 -vadapter vhost 24 -dev vtserv 1 hd 8 mkvdev -vdev hdisk 71 -vadapter vhost 24 -dev vtserv 1 hd 9

e-business hosting services Data access on SAN through sddpcm (mirroring) VIOS 1 AIX client partition datatvg vtserv 1 hd 9 VIOS 2 LV xxx vtserv 1 hd 8 LVM partitionning + mirroring SDDPCM hdisk 70 hdisk 71 hdisk 70 hdisk 71 hdisk 8 hdisk 9 SDDPCM hdisk 70 hdisk 71 hdisk 8 hdisk 9 FC adapter port 1 Vhost 24 Slot 44 port 2 Vscsi 4 Slot 44 Vscsi 5 Slot 45 Vhost 24 Slot 45 port 1 port 2 POWER hypervisor Fabric 1 Fabric 2 Basic MPIO command example Lspath|grep hdisk 8 Enabled hdisk 8 vscsi 4 Enabled hdisk 8 vscsi 5 hdisk 70 Bay 1 19 hdisk 71 Bay 2 Alain Lentini / Co. C Unix Lge

e-business hosting services Data access on SAN through RDAC (no mirroring) VIOS a AIX client partition LVM partitionning vtserv 1 hd 8 RDAC hdisk 8 hdisk 9 Default PCM + MPIO hdisk 70 hdisk 71 hdisk 8 FC adapter port 1 vtserv 1 hd 9 datavg LV xxx vtserv 1 hd 9 VIOS b Vhost 24 Slot 44 port 2 Vscsi 4 Slot 44 hdisk 9 hdisk 8 hdisk 70 hdisk 9 Vscsi 5 Slot 45 hdisk 71 FC adapter Vhost 24 Slot 45 port 1 port 2 POWER hypervisor Fabric 1 Fabric 2 hdisk 70 hdisk 71 FASTT 20 Alain Lentini / Co. C Unix Lge naming convention - example of commands on server VIO 1 mkvdev -vdev hdisk 70 -vadapter vhost 24 -dev vtserv 1 hd 8 mkvdev -vdev hdisk 71 -vadapter vhost 24 -dev vtserv 1 hd 9

e-business hosting services Data access on SAN through RDAC (mirroring) VIOS 1 AIX client partition vtserv 1 hd 9 datatvg vtserv 1 hd 9 VIOS 2 LV xxx vtserv 1 hd 8 LVM partitionning + mirroring RDAC hdisk 70 hdisk 71 hdisk 8 hdisk 9 FC adapter port 1 Vhost 24 Slot 44 port 2 Vscsi 4 Slot 44 Vscsi 5 Slot 45 Vhost 24 Slot 45 port 1 POWER hypervisor Fabric 2 Fabric 1 hdisk 70 FASTT B 2 21 Alain Lentini / Co. C Unix Lge Basic MPIO command example Lspath|grep hdisk 8 Enabled hdisk 8 vscsi 4 Enabled hdisk 8 vscsi 5 hdisk 71 FASTT B 4 port 2

e-business hosting services VIO SAN Connexions example VIO 2 VIO 1 fcs 4 fcs 3 fcs 2 fcs 1 fcs 0 fcs 1 3 3 swds 4300 b 2_01 fabric 1 5 5 swds 4300 b 2_02 fabric 2 swb 2 TMMFC-01 fabric 1 swb 2 TMMFC-02 fabric 2 SOD fabric 1 SOD fabric 2 22 Alain Lentini / Co. C Unix Lge fcs 2 3 6 fcs 0 fcs 3 fcs 4

e-business hosting services San disks devices creation q q On vioservers create the vhost servers adapter on the server and on the client partition mkvdev -vdev hdisk 70 -vadapter vhost 24 -dev vtserv 1 hd 8 mkvdev -vdev hdisk 71 -vadapter vhost 24 -dev vtserv 1 hd 9 On the client partition Change priority chpath -l hdisk 7 -p vscsi 4 -a priority=1 chpath -l hdisk 7 -p vscsi 5 -a priority=2 Priority Check lspath -AHE -l hdisk 7 -p vscsi 4 priority 1 Priority True lspath -AHE -l hdisk 7 -p vscsi 5 priority 2 Priority True q Devices check lscfg hdisk 40 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 500000 Virtual SCSI Disk Drive hdisk 41 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 600000 Virtual SCSI Disk Drive hdisk 42 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 700000 Virtual SCSI Disk Drive q q q Change the hcheck_interval parameter on each disk device chdev -l hdisk 7 -a hcheck_interval=20 –P hcheck_interval CHECK lsattr -El hdisk 7 | grep hcheck_interval 20 q q 23 Alain Lentini / Co. C Unix Lge

e-business hosting services Vio configuration check with padmin user 24 Alain Lentini / Co. C Unix Lge

e-business hosting services Vio configuration check On vio server : lsdev –slots U 9117. 570. 65 A 1 DA 0 -V 2 -C 24 Virtual I/O Slot vhost 2 U 9117. 570. 65 A 1 DA 0 -V 2 -C 26 Virtual I/O Slot vhost 3 U 9117. 570. 65 A 1 DA 0 -V 2 -C 28 Virtual I/O Slot vhost 4 U 9117. 570. 65 A 1 DA 0 -V 2 -C 30 Virtual I/O Slot vhost 5 U 9117. 570. 65 A 1 DA 0 -V 2 -C 32 Virtual I/O Slot vhost 6 U 9117. 570. 65 A 1 DA 0 -V 2 -C 34 Virtual I/O Slot vhost 7 On vio client partition lspath|grep hdisk 28 Enabled hdisk 28 vscsi 4 Enabled hdisk 28 vscsi 5 lscfg hdisk 40 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 500000 Virtual SCSI Disk Drive hdisk 41 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 600000 Virtual SCSI Disk Drive hdisk 42 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 700000 Virtual SCSI Disk Drive hdisk 43 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 800000 Virtual SCSI Disk Drive hdisk 44 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -La 900000 Virtual SCSI Disk Drive hdisk 45 U 9117. 570. 65 A 1 DA 0 -V 5 -C 40 -T 1 -Laa 00000 Virtual SCSI Disk Drive 25 Alain Lentini / Co. C Unix Lge

e-business hosting services Vio configuration check with lsmap Lsmap –vadapter vhost 10 | more SVSA Physloc Client Partition ID ------------------------------vhost 10 U 9117. 570. 65 A 1 DA 0 -V 2 -C 40 0 x 00000005 VTD vt. HA 11 thd 32 Status Available LUN 0 x 9 d 0000000 Backing device hdisk 116 Physloc U 7879. 001. DQDZLWK-P 1 -C 6 -T 2 -W 5005076300 C 4 A 620 -L 511 C 000000 VTD vt. HA 11 thd 33 Status Available LUN 0 x 9 e 0000000 Backing device hdisk 101 Physloc U 7879. 001. DQDZLWK-P 1 -C 6 -T 2 -W 5005076300 C 4 A 620 -L 5043000000 VTD vt. HA 11 thd 34 Status Available LUN 0 x 9 f 0000000 Backing device hdisk 102 Physloc U 7879. 001. DQDZLWK-P 1 -C 6 -T 2 -W 5005076300 C 4 A 620 -L 5044000000 26 Alain Lentini / Co. C Unix Lge

e-business hosting services LAN FREE SERVER 27 Alain Lentini / Co. C Unix Lge

e-business hosting services LAN FREE SERVER ARCHITECTURE design 28 Alain Lentini / Co. C Unix Lge