Using New Features in Condor 7 2 Condor

Using New Features in Condor 7. 2 Condor Project Computer Sciences Department University of Wisconsin-Madison

Outline › › › › Startd Hooks Job Router Hooks Power Management Dynamic Slot Partitioning Concurrency Limits Variable Substitution Preemption Attributes www. cs. wisc. edu/Condor 2

Startd Job Hooks › Users wanted to take advantage of Condor’s resource management daemon (condor_startd) to run jobs, but they had their own scheduling system. h. Specialized scheduling needs h. Jobs live in their own database or other storage rather than a Condor job queue www. cs. wisc. edu/Condor 3

Our solution › Make a system of generic “hooks” that you can plug into: h. A hook is a point during the life-cycle of a job where the Condor daemons will invoke an external program h. Hook Condor to your existing job management system without modifying the Condor code www. cs. wisc. edu/Condor 4

How does Condor communicate with hooks? › Passing around ASCII Class. Ads via › › standard input and standard output Some hooks get control data via a command-line argument (argv) Hooks can be written in any language (scripts, binaries, whatever you want) so long as you can read/write Stdin/out www. cs. wisc. edu/Condor 5

What hooks are available? › Hooks for fetching work (startd): h. FETCH_JOB h. REPLY_FETCH h. EVICT_CLAIM › Hooks for running jobs (starter): h. PREPARE_JOB h. UPDATE_JOB_INFO h. JOB_EXIT www. cs. wisc. edu/Condor 6

HOOK_FETCH_JOB › Invoked by the startd whenever it wants to try to fetch new work h. Fetch. Work. Delay expression › Stdin: slot Class. Ad › Stdout: job Class. Ad › If Stdout is empty, there’s no work www. cs. wisc. edu/Condor 7

HOOK_REPLY_FETCH › Invoked by the startd once it decides › › what to do with the job Class. Ad returned by HOOK_FETCH_WORK Gives your external system a chance to know what happened argv[1]: “accept” or “reject” Stdin: slot and job Class. Ads Stdout: ignored www. cs. wisc. edu/Condor 8

HOOK_EVICT_CLAIM › Invoked if the startd has to evict a › › › claim that’s running fetched work Informational only: you can’t stop or delay this train once it’s left the station Stdin: both slot and job Class. Ads Stdout: ignored www. cs. wisc. edu/Condor 9

HOOK_PREPARE_JOB › Invoked by the condor_starter when it › first starts up (only if defined) Opportunity to prepare the job execution environment h. Transfer input files, executables, etc. › Stdin: both slot and job Class. Ads › Stdout: ignored, but starter won’t › continue until this hook exits Not specific to fetched work www. cs. wisc. edu/Condor 10

HOOK_UPDATE_JOB_INFO › Periodically invoked by the starter to › let you know what’s happening with the job Stdin: slot and job Class. Ads h. Job Class. Ad is updated with additional attributes computed by the starter: • Image. Size, Job. State, Remote. User. Cpu, etc. › Stdout: ignored www. cs. wisc. edu/Condor 11

HOOK_JOB_EXIT › Invoked by the starter whenever the › job exits for any reason Argv[1] indicates what happened: h“exit”: Died a natural death h“evict”: Booted off prematurely by the startd (PREEMPT == TRUE, condor_off, etc) h“remove”: Removed by condor_rm h“hold”: Held by condor_hold www. cs. wisc. edu/Condor 12

HOOK_JOB_EXIT … › “HUH!? ! condor_rm? What are you talking about? ” h. The starter hooks can be defined even for regular Condor jobs, local universe, etc. › Stdin: copy of the job Class. Ad with extra attributes about what happened: h. Exit. Code, Job. Duration, etc. › Stdout: ignored www. cs. wisc. edu/Condor 13

Defining hooks › Each slot can have its own hook ”keyword” h. Prefix for config file parameters h. Can use different sets of hooks to talk to different external systems on each slot h. Global keyword used when the per-slot keyword is not defined › Keyword is inserted by the startd into its copy of the job Class. Ad and given to the starter www. cs. wisc. edu/Condor 14

Defining hooks: example # Most slots fetch work from the database system STARTD_JOB_HOOK_KEYWORD = DATABASE # Slot 4 fetches and runs work from a web service SLOT 4_JOB_HOOK_KEYWORD = WEB # The database system needs to both provide work and # know the reply for each attempted claim DB_DIR = /usr/local/condor/fetch/db DATABASE_HOOK_FETCH_WORK = $(DB_DIR)/fetch_work. php DATABASE_HOOK_REPLY_FETCH = $(DB_DIR)/reply_fetch. php # The web system only needs to fetch work WEB_DIR = /usr/local/condor/fetch/web WEB_HOOK_FETCH_WORK = $(WEB_DIR)/fetch_work. php www. cs. wisc. edu/Condor 15

Semantics of fetched jobs › Condor_startd treats them just like any other kind of job: h. All the standard resource policy expressions apply (START, SUSPEND, PREEMPT, RANK, etc). h. Fetched jobs can coexist in the same pool with jobs pushed by Condor, COD, etc. h. Fetched work != Backfill www. cs. wisc. edu/Condor 16

Semantics continued › If the startd is unclaimed and › › fetches a job, a claim is created If that job completes, the claim is reused and the startd fetches again Keep fetching until either: h. The claim is evicted by Condor h. The fetch hook returns no more work www. cs. wisc. edu/Condor 17

Limitations of the hooks › If the starter can’t run your fetched job because your Class. Ad is bogus, no hook is invoked to tell you about it h. We need a HOOK_STARTER_FAILURE › No hook when the starter is about to evict you (so you can checkpoint) h. Can implement this yourself with a wrapper script and the Soft. Kill. Sig attribute www. cs. wisc. edu/Condor 18

Job Router › Automated way to let jobs run on a wider array of resources h. Transform jobs into different forms h. Reroute jobs to different destinations www. cs. wisc. edu/Condor 19

What is “job routing”? original (vanilla) job Universe = “vanilla” Executable = “sim” Arguments = “seed=345” Output = “stdout. 345” Error = “stderr. 345” Should. Transfer. Files = True When. To. Transfer. Output = “ON_EXIT” routed (grid) job Job. Router Routing Table: Site 1 … Site 2 … Universe = “grid” Grid. Type = “gt 2” Grid. Resource = “cmsgrid 01. hep. wisc. edu/jobmanager-condor” Executable = “sim” Arguments = “seed=345” Output = “stdout” Error = “stderr” Should. Transfer. Files = True When. To. Transfer. Output = “ON_EXIT” final status www. cs. wisc. edu/Condor 20

Routing is just site-level matchmaking › With feedback from job queue • number of jobs currently routed to site X • number of idle jobs routed to site X • rate of recent success/failure at site X › And with power to modify job ad • change attribute values (e. g. Universe) • insert new attributes (e. g. Grid. Resource) • add a “portal” grid proxy if desired www. cs. wisc. edu/Condor 21

Configuring the Routing Table › JOB_ROUTER_ENTRIES • list site Class. Ads in configuration file › JOB_ROUTER_ENTRIES_FILE • read site Class. Ads periodically from a file › JOB_ROUTER_ENTRIES_CMD • read periodically from a script • example: query a collector such as Open Science Grid Resource Selection Service www. cs. wisc. edu/Condor 22

Syntax › List of sites in new Class. Ad format [ Name = “Grid Site 1”; … ] [ Name = “Grid Site 2”; … ] [ Name = “Grid site 3”; … ] … www. cs. wisc. edu/Condor 23

Syntax [ Name = “Site 1”; Grid. Resource = “gt 2 gk. foo. edu”; Max. Idle. Jobs = 10; Max. Jobs = 200; Failure. Rate. Threshold = 0. 01; Job. Failure. Test = other. Remote. Wall. Clock. Time < 1800 Requirements = target. Want. Job. Router is True; delete_Want. Job. Router = true; set_Periodic. Remove = Job. Status == 5; ] www. cs. wisc. edu/Condor 24

What Types of Input Jobs? › Vanilla Universe › Self Contained (everything needed is in file transfer list) › High Throughput (many more jobs than cpus) www. cs. wisc. edu/Condor 25

Grid Gotchas › Globus gt 2 hno exit status from job (reported as 0) › Most grid universe types hmust explicitly list desired output files www. cs. wisc. edu/Condor 26

Job. Router vs. Glidein › Glidein - Condor overlays the grid hjob never waits in remote queue hjob runs in its normal universe hprivate networks doable, but add to complexity hneed something to submit glideins on demand › Job. Router hsome jobs wait in remote queue (Max. Idle. Jobs) hjob must be compatible with target grid semantics h simple to set up, fully automatic to run www. cs. wisc. edu/Condor 27

Job Router Hooks › Truly transform jobs, not just reroute them h. E. g. stuff a job into a virtual machine (either VM universe or Amazon EC 2) › Hooks invoked like startd ones www. cs. wisc. edu/Condor 28

HOOK_TRANSLATE › Invoked when a job is matched to a › › › route Stdin: route name and job ad Stdout: transformed job ad Transformed job is submitted to Condor www. cs. wisc. edu/Condor 29

HOOK_UPDATE_JOB_INFO › Invoked periodically to obtain extra › › information about routed job Stdin: routed job ad Stdout: attributes to update in routed job ad www. cs. wisc. edu/Condor 30

HOOK_JOB_FINALIZE › Invoked when routed job has › › completed Stdin: ads of original and routed jobs Stdout: modified original job ad or nothing (no updates) www. cs. wisc. edu/Condor 31

HOOK_JOB_CLEANUP › Invoked when original job returned to › › schedd (both success and failure) Stdin: Original job ad Use for cleanup of external resources www. cs. wisc. edu/Condor 32

Power Management › Hibernate execute machines when not › › needed Condor doesn’t handle waking machines up yet Information to wake machines available in machine ads www. cs. wisc. edu/Condor 33

Configuring Power Management › HIBERNATE h. Expression evaluated periodically by all slots to decide when to hibernate h. All slots must agree to hibernate › HIBERNATE_CHECK_INTERVAL h. Number of seconds between hibernation checks www. cs. wisc. edu/Condor 34

Setting HIBERNATE › HIBERNATE must evaluate to one of these strings: h“NONE”, “ 0” h“S 1”, “STANDBY”, “SLEEP” h“S 2”, “ 2” h“S 3”, “RAM”, “MEM” h“S 4”, “DISK”, “HIBERNATE” h“S 5”, “SHUTDOWN” › These numbers are ACPI power states www. cs. wisc. edu/Condor 35

Power Management on Linux › On linux, theses methods are tried in order for setting power level: hpm-UTIL tools h/sys/power h/proc/ACPI › LINUX_HIBERNATION_METHOD can be set to pick a favored method www. cs. wisc. edu/Condor 36

Sample Configuration Should. Hibernate = ((Keyboard. Idle > $(Start. Idle. Time)) && $(CPUIdle) && ($(State. Timer) > (2 * $(HOUR))) HIBERNATE = if. Then. Else( $(Should. Hibernate), “RAM”, “NONE” ) HIBERNATE_CHECK_INTERVAL = 300 LINUX_HIBERNATION_METHOD = “/proc” www. cs. wisc. edu/Condor 37

Dynamic Slot Partitioning › Divide slots into chunks sized for › › matched jobs Readvertise remaining resources Partitionable resources are cpus, memory, and disk www. cs. wisc. edu/Condor 38

How It Works › When match is made… h. New sub-slot is created for job and advertised h. Slot is readvertised with remaining resources › Slot can be partitioned multiple times › Original slot ad never enters Claimed state h. But may eventually have too few resources to be matched › When claim on sub-slot is released, resources are added back to original slot www. cs. wisc. edu/Condor 39

Configuration › Resources still statically partitioned › between slots SLOT_TYPE__PARTITIONABLE h. Set to True to enable dynamic partition within indicated slot www. cs. wisc. edu/Condor 40

New Machine Attributes › In original slot machine ad h. Partitionable. Slot = True › In ad for dynamically-created slots h. Dynamic. Slot = True › Can reference these in startd policy expressions www. cs. wisc. edu/Condor 41

Job Submit File › Jobs can request how much of partitionable resources they need hrequest_cpus = 3 hrequest_memory = 1024 hrequest_disk = 10240 www. cs. wisc. edu/Condor 42

Dynamic Partitioning Caveats › Cannot preempt original slot or group of sub-slots h. Potential starvation of jobs with large resource requirements › Partitioning happens once per slot each negotiation cycle h. Scheduling of large slots may be slow www. cs. wisc. edu/Condor 43

Concurrency Limits › Limit job execution based on admindefined consumable resources h. E. g. licenses › Can have many different limits › Jobs say what resources they need › Negotiator enforces limits pool-wide www. cs. wisc. edu/Condor 44

Concurrency Example › Negotiator config file h. MATLAB_LIMIT = 5 h. NFS_LIMIT = 20 › Job submit file hconcurrency_limits = matlab, nfs: 3 h. This requests 1 Matlab token and 3 NFS tokens www. cs. wisc. edu/Condor 45

New Variable Substitution › $$(Foo) in submit file h. Existing feature h. Attribute Foo from machine ad substituted › $$([Memory * 0. 9]) in submit file h. New feature h. Expression is evaluated and then substituted www. cs. wisc. edu/Condor 46

More Info For Preemption › New attributes for these preemption expressions in the negotiator… h. PREEMPTION_REQUIREMENTS h. PREEMPTION_RANK › Used for controlling preemption due to user priorities www. cs. wisc. edu/Condor 47

Preemption Attributes › Submitter/Remote. User. Prio h. User priority of candidate and running jobs › Submitter/Remote. User. Resources. In. Use h. Number of slots in use by user of each job › Submitter/Remote. Group. Resources. In. Use h. Number of slots in use by each user’s group › Submitter/Remote. Group. Quota h. Slot quota for each user’s group www. cs. wisc. edu/Condor 48

Thank You! › Any questions? www. cs. wisc. edu/Condor 49