Fermi. Grid-HA (Delivering Highly Available Grid Services using Virtualisation) Keith Chadwick Fermilab chadwick@fnal. gov 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA
Outline Who is Fermi. Grid? What is Fermi. Grid? Current Architecture & Performance Why Fermi. Grid-HA? Reasons, Goals, etc. What is Fermi. Grid-HA? Fermi. Grid-HA Implementation Design, Technology, Challenges, Deployment & Performance Conclusions Future Work 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 1
Fermi. Grid - Personnel Eileen Berman, Fermilab, Batavia, IL 60510 berman@fnal. gov Philippe Canal, Fermilab, Batavia, IL 60510 pcanal@fnal. gov Keith Chadwick, Fermilab, Batavia, IL 60510 chadwick@fnal. gov * David Dykstra, Fermilab, Batavia, IL 60510 dwd@fnal. gov Ted Hesselroth, Fermilab, Batavia, IL, 60510 tdh@fnal. gov Gabriele Garzoglio, Fermilab, Batavia, IL 60510 garzogli@fnal. gov Chris Green, Fermilab, Batavia, IL 60510 greenc@fnal. gov Tanya Levshina, Fermilab, Batavia, IL 60510 tlevshin@fnal. gov Don Petravick, Fermilab, Batavia, IL 60510 petravick@fnal. gov Ruth Pordes, Fermilab, Batavia, IL 60510 ruth@fnal. gov Valery Sergeev, Fermilab, Batavia, IL 60510 sergeev@fnal. gov * Igor Sfiligoi, Fermilab, Batavia, IL 60510 sfiligoi@fnal. gov Neha Sharma Batavia, IL 60510 neha@fnal. gov * Steven Timm, Fermilab, Batavia, IL 60510 timm@fnal. gov * D. R. Yocum, Fermilab, Batavia, IL 60510 yocum@fnal. gov * 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 2
Fermi. Grid - Current Architecture Periodic VOMRS Server nit y-i th -u VO ser rox reg ist ers wi p 1 Ste Synchronization Ste p 2 r use se -u es ssu s-p vom ri ves ei rec g s si ned de cre VOMS Server Periodic Synchronization als nti S UM ts G ues le req Ro ay w ate O& V –G on p 5 ed Ste bas ing pp Ma vom Site Wide Step 3 – user submits their grid job via globus-job-run, globus-job-submit, or condor-g p 4 ay c hor izat ion hec ks a gain Ser v st ice 6 ar rw fo is er b st jo clu rid get - G tar to SAZ Server d de ior er Int 07 -Nov-2007 atew Aut Blue. Arc r rio te –G ep (dcache SRM) Ex Site St clusters send Class. Ads via CEMon to the site wide gateway FERMIGRID SE Gratia Ste Gateway GUMS Server CMS WC 1 CMS WC 2 CMS WC 3 CDF OSG 1 CDF OSG 2 D 0 CAB 1 Keith Chadwick - Fermi. Grid-HA D 0 CAB 2 GP Farm GP MPI 3
Fermi. Grid - Current Performance VOMS: Current record ~1700 voms-proxyinits/day. Not a driver for Fermi. Grid-HA. GUMS: Current record > 1 M mapping requests/day Maximum system load <3 at a CPU utilization of 130% (max 200%) SAZ: Current record > 129 K authorization decisions/day. Maximum system load <5. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 4
Why Fermi. Grid-HA? Fermi. Grid core services (GUMS and/or SAZ) control access to: Over 2000 systems with more than 9000 batch slots (today). Petabytes of storage (via g. Plazma which calls GUMS). An outage of either GUMS or SAZ can cause 5, 000 to 50, 000 “jobs” to fail for each hour of downtime. Manual recovery or intervention for these services can have long recovery times (best case 30 minutes, worst case multiple hours). Automated service recovery scripts can minimize the downtime (and impact to the Grid operations), but still can have several tens of minutes response time for failures: How often the scripts run, Scripts can only deal with failures that have known “signatures”, Startup time for the service, A script cannot fix dead hardware. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 5
Fermi. Grid-HA - Requirements: Critical services hosted on multiple systems (n ≥ 2). Small number of “dropped” transactions when failover required (ideally 0). Support the use of service aliases: – VOMS: – GUMS: – SAZ: fermigrid 2. fnal. gov -> fermigrid 3. fnal. gov -> fermigrid 4. fnal. gov -> voms. fnal. gov gums. fnal. gov saz. fnal. gov Implement “HA” services with services that did not include “HA” in their design. – Without modification of the underlying service. Desirables: Active-Active service configuration. Active-Standby if Active-Active is too difficult to implement. A design which can be extended to provide redundant services. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 6
Fermi. Grid-HA - Technology Xen: SL 5. 0 + Xen 3. 1. 0 (from xensource community version) – 64 bit Xen Domain 0 host, 32 and 64 bit Xen VMs Paravirtualisation. Linux Virtual Server (LVS 1. 38): Shipped with Piranha V 0. 8. 4 from Redhat. Grid Middleware: Virtual Data Toolkit (VDT 1. 8. 1) VOMS V 1. 7. 20, GUMS V 1. 2. 10, SAZ V 1. 9. 2 My. SQL: Multi-master database replication. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 7
Fermi. Grid-HA - Challenges #1 Active-Standby: Easier to implement, Can result in “lost” transactions to the backend databases, Lost transactions would then result in potential inconsistencies following a failover or unexpected configuration changes due to the “lost” transactions. – GUMS Pool Account Mappings. – SAZ Whitelist and Blacklist changes. Active-Active: Significantly harder to implement (correctly!). Allows a greater “transparency”. Reduces the risk of a “lost” transaction, since any transactions which results in a change to the underlying My. SQL databases are “immediately” replicated to the other service instance. Very low likelihood of inconsistencies. – Any service failure is highly correlated in time with the process which performs the change. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 8
Fermi. Grid-HA - Challenges #2 DNS: Initial Fermi. Grid-HA design called for DNS names each of which would resolve to two (or more) IP numbers. If a service instance failed, the surviving service instance could restore operations by “migrating” the IP number for the failed instance to the Ethernet interface of the surviving instance. Unfortunately, the tool used to build the DNS configuration for the Fermilab network did not support DNS names resolving to >1 IP numbers. – Back to the drawing board. Linux Virtual Server (LVS): Route all IP connections through a system configured as a Linux virtual server. – Direct routing – Request goes to LVS director, LVS director redirects the packets to the real server, real server replies directly to the client. Increases complexity, parts and system count: – More chances for things to fail. LVS director must be implemented as a HA service. – LVS director implemented as an Active-Standby HA service. – Run LVS director as a special process on the Xen Domain 0 system. LVS director performs “service pings” every six (6) seconds to verify service availability. – Custom script that uses curl for each service. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 9
Fermi. Grid-HA - Challenges #3 My. SQL databases underlie all of the Fermi. Grid-HA Services (VOMS, GUMS, SAZ): Fortunately all of these Grid services employ relatively simple database schema, Utilize multi-master My. SQL replication, – Requires My. SQL 5. 0 (or greater). – Databases perform circular replication. Currently have two (2) My. SQL databases, – My. SQL 5. 0 circular replication has been shown to scale up to ten (10). – Failed databases “cut” the circle and the database circle must be “retied”. Transactions to either My. SQL database are replicated to the other database within 1. 1 milliseconds (measured), Tables which include auto incrementing column fields are handled with the following My. SQL 5. 0 configuration entries : – auto_increment_offset (1, 2, 3, … n) – auto_increment (10, 10, … ) 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 10
Fermi. Grid-HA - Component Design VOMS Active LVS Active GUMS Active Heartbeat LVS Standby GUMS Active SAZ My. SQL Active LVS Replication My. SQL Active Standby Active SAZ Active 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 11
Fermi. Grid-HA - Host Configuration The fermigrid 5&6 Xen hosts are Dell 2950 systems. Each of the Dell 2950 s are configured with: Two 3. 0 GHz core 2 duo processors (total 4 cores). 16 Gbytes of RAM. Raid-1 system disks (2 x 147 Gbytes, 10 K RPM, SAS). Raid-1 non-system disks (2 x 147 Gbytes, 10 K RPM, SAS). Dual 1 Gig-E interfaces: – 1 connected to public network, – 1 connected to private network. System Software Configuration: LVS Director is run on the Xen Domain 0 s. Each Domain 0 system is configured with 4 Xen VMs. Each Xen VM, dedicated to running a specific service: – VOMS, GUMS, SAZ, My. SQL 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 12
Fermi. Grid-HA - Actual Component Deployment Xen Domain 0 VOMS Active GUMS LVS (Active) Xen Domain 0 Xen VM 1 VOMS fg 5 x 1 Active Xen VM 2 GUMS LVS (Standby) Xen VM 1 fg 6 x 1 Xen VM 2 Active fg 5 x 2 Active fg 6 x 2 SAZ Xen VM 3 Active fg 5 x 3 Active fg 6 x 3 My. SQL Active 07 -Nov-2007 Xen VM 4 My. SQL fg 5 x 4 fermigrid 5 Active Keith Chadwick - Fermi. Grid-HA Xen VM 4 fg 6 x 4 fermigrid 6 13
Fermi. Grid-HA - Performance #1 Stress tests of the Fermi. Grid-HA GUMS deployment: The initial stress test demonstrated that this configuration can support >4. 3 M mappings/day. – The load on the GUMS VMs during this stress test was ~1. 2 and the CPU idle time was 60%. – The load on the backend My. SQL database VM during this stress test was under 1 and the CPU idle time was 92%. A second stress test demonstrated that this configuration can support ~9. 7 M mappings/day. – The load on the GUMS VMs during this stress test was ~9. 5 and the CPU idle time was 15%. – The load on the backend My. SQL database VM during this stress test was under 1 and the CPU idle time was 92%. GUMS uses hibernate which is why the backend My. SQL database VM load did not increase between the two measurements. Based on these measurements, we’ll need to start the planning for a third GUMS server in Fermi. Grid-HA when we hit the ~7. 5 M mappings/day mark. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 14
Fermi. Grid-HA - Performance #2 Stress tests of the Fermi. Grid-HA SAZ deployment: The SAZ stress test demonstrated that this configuration can support ~1. 1 M authorizations/day. – The load on the SAZ VMs during this stress test was ~12 and the CPU idle time was 0%. – The load on the backend My. SQL database VM during this stress test was under 1 and the CPU idle time was 98%. The SAZ server does not (currently) use hibernate. – This change is “in the works”. The SAZ server (currently) performs a significant amount of parsing of the users proxy to identify the DN, VO, Role & CA. – This will change as we integrate SAZ into the Globus Auth. Z framework. – The distributed SAZ clients will perform the parsing of the users proxy. We also take a careful look at the SAZ server to see if there are optimizations that can be performed to improve the performance. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 15
Fermi. Grid-HA - Performance #3 Stress tests of the combined Fermi. Grid-HA GUMS and SAZ deployment: Using a GUMS: SAZ call ratio of ~7: 1 The combined GUMS-SAZ stress test which was performed yesterday (06 -Nov-2007) demonstrated that this configuration can support ~6. 5 GUMS mappings/day and ~900 K authorizations/day. – The load on the SAZ VMs during this stress test was ~12 and the CPU idle time was 0%. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 16
Fermi. Grid-HA - Production Deployment Our plan is to complete the Fermi. Grid-HA stress testing and deploy Fermi. Grid-HA in production during the week of 03 -Dec 2007. In order to allow an adiabatic transition for the OSG and our user community, we will run the regular Fermi. Grid services and Fermi. Grid-HA services simultaneously for a three month period. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 17
Fermi. Grid-HA - Future Plans Redundant side wide gatekeeper: We have a preliminary “Gatekeeper-HA” design. . . We will also be installing a test gatekeeper to receive Xen VMs as Grid jobs and execute them: This a test of a possible future dynamic “VOBox” or “Edge Service” capability within Fermi. Grid. Prerequisites: We will be “recycling” the hardware that is currently supporting the non. HA Grid services, So these deployments will need to wait until the transition to the Fermi. Grid-HA services has completed. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 18
Fermi. Grid-HA - Ancillary Services Fermi. Grid also runs/hosts several ancillary services which are not critical for the Fermilab Grid operations: Squid, My. Proxy, Syslog-Ng, Ganglia, Metrics and Service Monitoring, OSG Security Test & Evaluation (ST&E) tool. As the Fermi. Grid-HA evolution continues, we will evaluate if it makes sense to “HA” these services. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 19
Fermi. Grid-HA - Conclusions Virtualisation benefits: + + + Significant performance increase, Significant reliability increase, Automatic service failover, Cost savings, Can be scaled as the load and the reliability needs increase. Virtualisation drawbacks: - Significantly more complex design, Many “moving parts”, Many more opportunities for things to fail, Many more items that need to be monitored. 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 20
Fin Any Questions? 07 -Nov-2007 Keith Chadwick - Fermi. Grid-HA 21
Скачать презентацию Fermi Grid-HA Delivering Highly Available Grid Services using
0c5c53a1fec057e5846ca461e0de9efd.ppt