Open VMS Solutions Center Lab Project

Open. VMS Solutions Center Lab Project - Spring 2004 : Oracle 9 i RAC DT/HA in a distributed Open. VMS Environment Phase I – Failover

RAC DT/HA – Goals – Phase I Ø First: Ø Ø Demonstrate that Oracle 9 i. RAC continues to run during simulated network failure using LAN Failover and fail. SAFE IP configurations. Second: Ø Measure the latency effect of failover when RAC instances are connected over long distance (100 km).

RAC DT/HA – What is Failover? 1. Oracle RAC failover: The ability to resume work on an alternate instance upon instance failure 2. Oracle TAF (Transparent Application Failover): Runtime failover which enables client applications to automatically reconnect to the database if the connection fails 3. LAN Failover: Hardware failover from failed network interface card (NIC) to another NIC configured as part of LAN failover set 4. fail. SAFE IP: Address failover to alternate interfaces

RAC DT/HA – Hardware Config 1. 2 4 -cpu GS 160, with Shared Cluster System disk, a Shared Oracle install disk on Enterprise Storage Array connected via Fibre SAN A Switch 2. DE 602 -AA (EIA) NIC’s, using Twisted Pair on 100 m-bit LAN Extreme Summit 4 Switch 3. 5 -DEGPA-SA, 1 -DEGXA-SA (EWA-D) NIC’s, 1 Gbit fiber on 1 Gbit LAN Digital Networks DNSwitch 800 4. 100 km cable - Gbit SCS Extreme Summit 7 i Switch

RAC DT/HA – Server Config 1. Open. VMS 7. 3 -2, TCPIP 5. 4 2. Oracle Server 9. 2. 0. 4, with Oracle patch for bug fix 3026720: Excessive CPU and BUFIO for LMD 0 and SMON processes when >2 cpu 3. Running 2 RAC instances, in 2 node cluster 4. Requires the INIT<SID>. ORA parameter CLUSTER_INTERCONNECTS to specify alternate network interface for RAC communication

RAC DT/HA – Client Config 1. 9. 2 SQLNet Client, on PC running Windows 2000 2. Benchmark/Load Generating software: 1. Swingbench 2. 1 f- An ‘unofficial’, Java based, client load generating tool from Oracle, which allows a ‘load’ to be generated and the transactions/response times to be charted 2. Configured to connect 100 clients, load balanced between the 2 instances, and run 50, 000 ‘typical’ Order Entry transactions

RAC DT/HA – Test Plan 1. Restore from disk backup before each test run to ensure same starting point 2. Ensure RAC instances communicating over specified network interface 3. Run 3 iterations of same benchmark load while collecting data 1. Run Benchmark load, no failures 2. Run Benchmark load, fail instance 3. Run Benchmark load, fail network connection between instances

RAC DT/HA – Data collection 1. T 4 running on both nodes, 10 sec sampling interval 2. Saved Swingbench data results after each run 3. Executed and ‘saved’ output of VMS commands during network failures to see status of network devices and Oracle processes $ MC LANCP SHOW DEVICE/CHARATERISTICS LLA 0 $ TCPIP SHOW INTERFACES/FULL $ PIPE SHO SYS|SEA TT: ORA_CPU

Tabular Timeline Tracking Tool – T 4 Ø Ø Created by Open. VMS Sustaining Engineers to help diagnose OS functionality. Uses Open. VMS Monitor data, stored in Comma Separated Value file format (. csv file), which can then be used by a variety of applications (spreadsheets, Tl. Viz, etc) Download from web. Shipped with Open. VMS 7. 3 -2, in SYS$ETC directory v http: //h 71000. www 7. hp. com/openvms/products/t 4/index. html Ø Ø Ø Users are able to queue data collection and configure data collection frequency Helpful in establishing baseline performance footprint which can then be used in before and after comparisons of system changes T 4 ‘hooks’ for Oracle and Rdb Server being created

RAC DT/HA – EIA Network

RAC DT/HA – T 4 data - EIA

RAC DT/HA - LAN Failover Network

RAC DT/HA – LAN Failover DCL $ MCR LANCP SHOW DEVICE/CHAR LLA 0 Before NIC ‘fails’ Device Characteristics LLA 0: Value -----256 Yes. . 1000 "EWB 0" "EWA 0". . 0 Characteristic -------Max receive buffers Full duplex enable. . Line speed (mbps) Failover device (active). . Failover priority After NIC ‘fails’ Device Characteristics LLA 0: Value -----256 Yes. . 1000 "EWB 0" "EWA 0". . 0 Characteristic -------Max receive buffers Full duplex enable. . Line speed (mbps) Failover device (active) Failover device. . Failover priority

RAC DT/HA-T 4 LAN Failover EWA/B EWA 0 cable pulled EWB 0 cable pulled

RAC DT/HA-T 4 LAN Failover LLA 0

RAC DT/HA-T 4 Overlay of EWA/LLA 0

RAC DT/HA – fail. SAFE IP Network

RAC DT/HA – fail. SAFE IP DCL $ TCPIP SHOW INTERFACE/FULL Route Tree for Protocol Family 2: default 161. 114. 69. 1 10. 4. 4/24 10. 4. 4. 2 10. 4. 4/24 10. 4. 4. 3 10. 4. 4. 2 10. 4. 4. 3 UGS U U UHL 0 274 0 0 7999 408185 445714 0 14 IE 0 WE 3 WE 4 WE 3: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> fail. SAFE IP Addresses: inet 10. 4. 4. 3 netmask ffffff 00 broadcast 161. 114. 69. 63 (on QBB 3 WE 4) *inet 10. 4. 4. 2 netmask ffffff 00 broadcast 10. 4. 4. 255 ipmtu 1500 WE 4: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> fail. SAFE IP Addresses: inet 10. 4. 4. 2 netmask ffffff 00 broadcast 161. 114. 69. 63 (on QBB 3 WE 3) *inet 10. 4. 4. 3 netmask ffffff 00 broadcast 10. 4. 4. 255 ipmtu 1500

RAC DT/HA – fail. SAFE IP DCL Failed 1 $ TCPIP SHOW INTERFACE/FULL Route Tree for Protocol Family 2: default 161. 114. 69. 1 10. 4. 4/24 10. 4. 4. 2 10. 4. 4/24 10. 4. 4. 3 10. 4. 4. 2 10. 4. 4. 3 UGS U U UHL 0 274 0 0 7999 408185 445714 0 14 IE 0 WE 3 WE 4 WE 3: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> *fail. SAFE IP - interface is in a failed state fail. SAFE IP Addresses: inet 10. 4. 4. 3 netmask ffffff 00 broadcast 161. 114. 69. 63 (on QBB 3 WE 4) *inet 10. 4. 4. 2 netmask ffffff 00 broadcast 10. 4. 4. 255 (on QBB 3 WE 4) WE 4: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> *inet 10. 4. 4. 3 netmask ffffff 00 broadcast 10. 4. 4. 255 ipmtu 1500 inet 10. 4. 4. 2 netmask ffffff 00 broadcast 161. 114. 69. 63 ipmtu 1500

RAC DT/HA – fail. SAFE IP DCL Failed 2 $ TCPIP SHOW INTERFACE/FULL Route Tree for Protocol Family 2: default 161. 114. 69. 1 10. 4. 4/24 10. 4. 4. 2 10. 4. 4/24 10. 4. 4. 3 10. 4. 4. 2 10. 4. 4. 3 UGS U U UHL 0 274 0 0 7999 408185 445714 0 14 IE 0 WE 3 WE 4 WE 3: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> *inet 10. 4. 4. 2 netmask ffffff 00 broadcast 10. 4. 4. 255 ipmtu 1500 inet 10. 4. 4. 3 netmask ffffff 00 broadcast 161. 114. 69. 63 ipmtu 1500 WE 4: flags=c 43<UP, BROADCAST, RUNNING, MULTICAST, SIMPLEX> *fail. SAFE IP - interface is in a failed state. fail. SAFE IP Addresses: inet 10. 4. 4. 2 netmask ffffff 00 broadcast 161. 114. 69. 63(on QBB 3 WE 3) *inet 10. 4. 4. 3 netmask ffffff 00 broadcast 10. 4. 4. 255 (on QBB 3 WE 3)

RAC DT/HA – T 4 data fail. SAFE IP EWD 0 cable pulled EWE 0 cable pulled

RAC DT/HA – 100 km cable Network

RAC DT/HA – T 4 EWA 0 w/100 km cable

RAC DT/HA – T 4 EIA compared w/ EWA

RAC DT/HA – Load Generation Data 50 k Transactions, no RAC or Network Failure Network Interface Total duration TPS Baseline (EIA 161. 114. 69. x) 30: 08 27. 8 Lan Failover (EWA 10. 3. 3. x) 30: 02 27. 9 Fail. Safe IP (EWD 10. 4. 4. x) 30: 02 27. 9 100 km Baseline (EWA 10. 3. 3. x) 29: 52 28. 0

RAC DT/HA – Load Generation Data 50 k Transactions, Network failover Network Interface Total duration TPS Baseline (EIA 161. 114. 69. x) N/A Lan Failover (EWA 10. 3. 3. x) 30: 02 27. 9 Fail. Safe IP (EWD 10. 4. 4. x) 30: 13 27. 7 N/A 100 km Baseline (EWA 10. 3. 3. x)

RAC DT/HA – Load Generation Data 50 k Transactions, 1 RAC instance failed Network Interface Total Duration TPS 50 client Failover Baseline (EIA 161. 114. 69. x) 33: 25 25. 0 00: 37 Lan Failover (EWA 10. 3. 3. x) 29: 54 28. 0 00: 39 Fail. Safe IP (EWD 10. 4. 4. x) 30: 02 27. 7 00: 39 100 km Baseline (EWA 10. 3. 3. x) 29: 39 28. 0 00: 43

RAC DT/HA – Conclusions 1. RAC seemed to have no problems when running with network configured to use LAN Failover or fail. SAFE IP (on the same node). 2. There seems to be a definite distributing effect on network traffic when Oracle init. ora parameter CLUSTER_INTERCONNECTS is used

RAC DT/HA – Phase II and III 1. Phase II: Configure Oracle 9 i. RAC 2 -node cluster using Raid-1 Shadow Sets for database and logfiles, and test recently released Host Based Mini-Merge (HBMM) functionality in a variety of configurations. 2. Refer to: http: //h 71000. www 7. hp. com/news/hbmm. htm 3. Phase III: Distribute nodes in cluster over 100 km+ distance and test failover and HBMM functionality

RAC DT/HA - References Open. VMS Technical Journal: 1. Matt Muggeridge’s July 2003 - V 2 Article: Configuring TCP/IP for High Availability http: //h 71000. www 7. hp. com/openvms/journal/v 2/ articles/tcpip. pdf 2. Steve Lieman’s January 2004 - V 3 Article: Time. Line-Driven Collaboration with T 4 & Friends: A Time-saving Approach to Open. VMS Performance http: //h 71000. www 7. hp. com/openvms/journal/v 3/ t 4. pdf

RAC DT/HA – References (con’t) 1. TCPIP docs: 2. http: //h 71000. www 7. hp. com/doc/tcpip 54. html 3. Open. VMS docs: http: //h 71000. www 7. hp. com/doc/os 732_index. ht ml 4. HP TCP/IP Services for Open. VMS Management: Chapter 5 Configuring and Managing Fail. SAFE IP 5. http: //h 71000. www 7. hp. com/doc/732 final/docum entation/pdf/aa-lu 50 m-te. pdf

RAC DT/HA – References (con’t) 1. HP Open. VMS System Management Utilities Reference Manual: Chapter 13, LAN Control Program (LANCP) Utility 2. http: //h 71000. www 7. hp. com/doc/732 FINAL/DOC UMENTATION/PDF/aa-pv 5 ph-tk. PDF 1. HP Open. VMS System Manager’s Manual, Volume 2 -Tuning, Monitoring, and Complex Systems: Chapter 10, Managing the Local Area Network (LAN)Software 2. http: //h 71000. www 7. hp. com/doc/732 FINAL/aa- pv 5 nh-tk/aa-pv 5 nh-tk. pdf

RAC DT/HA – References (con’t) Oracle References: v Swingbench – an ‘unofficial’ load generating benchmarking tool, developed in Java, which allows a load to be generated and the transactions/response times to be charted v http: //www. dominicgiles. com/swingbench. php v OTN otn. oracle. com Real 24/7: Use Oracle 9 i RAC and TAF to guarantee availability. http: //otn. oracle. com/oramag/oracle/02 may/o 32 clusters. html

RAC DT/HA – References (con’t) Oracle Metalink articles: metalink. oracle. com. Note: 183340. 1 - Frequently Asked Questions About the. v CLUSTER_INTERCONNECTS Parameter in 9 i. v v Note 220970. 1 - “Which network is Oracle using for RAC traffic? " v Note: 162725. 1 - OPS/RAC VMS: Using alternate TCP Interconnects on 8 i OPS and 9 i RAC on Open. VMS. v Note: 226880. 1 – Configuration of Load Balancing and Transparent Application Failover.

Open. VMS Solutions Lab 1. Available to customers to test new hardware, software, applications 2. Alpha and Integrity systems available for use 3. To get the most benefit from the Lab, customer is expected to be prepared with exact list of hardware and software requirements, test plan and goals

Open VMS Solutions Center Lab Project — Spring