NEEShub Simulation Capabilities February 17 2012 Webinar http

NEEShub Simulation Capabilities February 17, 2012 Webinar http: //nees. org/resources/4079 George E Brown, JR. Network for Earthquake Engineering Simulation Gregory Rodgers Ph. D. NEESComm IT Purdue University, West Lafayette, IN Post-webinar updates

Webinar Introduction Audience Simulation tool developers and NEES power users who: • have a very large simulations or many simulations in excess of 30 minutes run time. • need to script parameter sweeps • run a structure analysis with a large suite of ground motions. Prerequisite An understanding of command line interfaces such as Linux bash Summary • This webinar will introduce advanced users to new NEEShub capabilities in the area of simulation and batch processing. Power users often write a script to orchestrate a set of simulation runs to cover many different test cases. Recent batch processing services have been added to NEEShub to make this easy and to provide access to large scratch space. Upon completion of this webinar, a user will be able to write scripts to submit one or more jobs to multiple execution venues to utilize high performance computing resources available to NEES

Agenda HOUR 1 • • • How Simulation fits into the NEES Cyber. Infrastructure Introduction to the linux workspace tool on NEEShub Manual execution (command line) of applications. Manual execution (command line) of the opensees simulator Use of the new batchsubmit command to run opensees Use of batchsubmit to run other applications The batchstatus command Demonstration of how HOME directory space is linked to scratch space. Advanced batchsubmit options and scripting the execution of batchsubmit. HOUR 2 (advanced) • • How to build a bash command file including editors available on Linux Simple parallel execution (The --ncpus argument to batchsubmit) Parallel opensees (how to modify sequential input to be parallel input) How to use batchsubmit for other venues. Overview of various NEES execution High Performance Computing (HPC) venues: They are local hub execution, osg, hansen, steele, kraken, and ranger, How does the openseeslab user interface use batchsubmit Advanced batchsubmit options review Scratch cleanup algorithm

NEES Cyber Infrastructure A. Site Operations Tools site/personal data Scratch Space F. Spreadsheet DBs NSF Xsede Open Science Grid Purdue Hansen Project Editor C. Cloud / Simulation Environment E. EOT Collaboration NE Hub Tool Sessions B. NEESHub Web Server Resources ch Sy n Group Space N Web Browser PE Personal Space om ls st oo Cu S t W ro ne es Experiment Data NEES Web Services Server D. The NEES Project Warehouse

Introduction to the linux workspace tool on NEEShub. • Start a workspace from this page http: //nees. org/resources/workspace Click “Launch” • You must be part of a special group. If you are not in this group, open a ticket stating that you need workspace access, provide justification, and we will add you to the group. • The window can be resized and popped out of the browser. • Multiple terminals can be opened in the same window. • A workspace session is persistent. You can leave the browser and you can get back to existing workspace from myneeshub page at any time. • This session can also be shared with other users or administrators.

Execution of Applications from the command line • Simple utilities date Print the date env List environment variables ls (and ls –l) Show list of files (long list) cd Change working directory pwd Show working directory mkdir make a directory rm (rmdir) Remove a file (directory) cat Write contents of file on the screen cp Copy a file man <command> Show help about a command (man pages) exit Teminate your session Use Arrow keys get previous commands

Putting commands in a script file • A list of commands can be put into a script – Avoid retyping – Loop through commands – To make executable use command: chmod 755 <filename> • 3 important scripting languages to consider bash Tcl/Tk Python linux commands, also csh The language for opensees Advanced high performance language

Manual Execution of Open. Sees • Opensees tcl prompt verses Linux command prompt • Start opensees with a tcl prompt (no argument) • Start opensees to execute a file of tcl commands. ( one argument) • The binary Open. Sees verses the wrapper shell called opensees <input TCL file> The spelling of the Open. Sees binary is Open. Sees, but opensees is a wrapper to call Open. Sees that sets up the environment correctly.

High Volume Batch on NEEShub • Consistent and asynchronous submission to multiple venues: local, osg, steele, hansen, kraken. The last three are part of the new xsede system that replaces Teragrid. • Asynchronous: job is submitted without waiting for job to complete before returning control to submitter. • Your run directories in $HOME/scratch will be symbolic links to a large (>30 TB) shared space. Runs will be compressed or purged with a cleanup algorithm as needed. • Only user will have access to run directories

batchsubmit • The batchsubmit command is a wrapper around any command to execute an asynchronous batch job. batchsubmit <batchsubmitoptions> command <command options> • batchsubmitoptions begin with a double dash. • batchsubmit prints one line of output: the name of the newly created directory where BOTH job input is located and output will be found. • The help for batchsubmit gives an example of how to run opensees batchsubmit –h | more batchsubmit date batchsubmit opensees /apps/demo/sine. tcl batchsubmit –appdir /apps/openseesbuild/osg Open. Sees /apps/demo/sine. tcl batchsubmit –jo btype sine –onlyinfile opensees /apps/demo/sine. tcl

Input Processing • Default: The first argument after the application command is considered an input file. All files from this directory are copied to the scratch run directory. Two other options: --onlyinfile Only copy the input file --rcopyindir Recursive copy all files and directories from the same directory as the inputfile. • Note: input file not allowed to be home directory unless –onlyinfile specified. You should create a directory for your opensees tcl file. Recommend a dir for each simulation. • What if you have an application where the first argument is NOT the input file (unlike opensees)? --infilearg Indicates which argument is the input file --infile Use this file as an input file where this file is implied by application command hence not one of its arguments.

batchsubmit files/dirs • Job input exists in new scratch directory upon completion of the batchsubmit command. One scratch directory for each batchsubmit command (each job). The directory name has this template. $HOME/scratch/<jobtype>/<jobname>/ • Job output exists when the job is completed. – You will get an email when job starts and when job completes unless you specify –nonotify • Review the various output files generated in a job run directory <jobname>. stdout <jobname>. stderr The run directory @STATUS joblog. born_on_date Standard output. What would be printed to screen Standard error. Same directory name where the input file was found. Note: your input file is in this directory. Interesting info about the environment job was run Statistics recorded about this job Used for scratch cleanup.

Job lifecycle • System uses the file @STATUS to store the job status. • States: Presubmit – only for remote venues Submitted – Waiting to start. Only for remote venues. Started – application is running. Remote venues will actually update this file Completed – All results are returned. Deleted – Job has been removed from the shared scratch space but your scratch directory still shows it. Saved – Job was moved to your HOME directory. Symbolic link to shared scratch space is gone. Job is taking up your quota when it is saved.

batchstatus and batchcancel • Other batchsubmit utilities batchstatus – shows the status of each of your jobs. batchcancel – Cancel a job. This command is not released yet. batchsave – Remove a job from scratch space and save it to your HOME directory space.

NEEShub Disk Space HOME space Groups Space Scratch space Warehouse /home/neeshub/<youruserid> /data/groups/<groupname> $HOME/scratch /nees/home/<PROJ-DIR> • Use of synchronees to upload and download between your workstation and NEEShub spaces • Advice: Use relative names for input and output files so your job can run on venues other than “local” webdav • NEEShub data locations: HOME Space Synchronees workstation data Group Space NE batchsubmit Scratch Space

Advance batchsubmit options --wait Only for venues local and osg. This option will hold the completion of batchsubmit until the job is COMPLETED. Standard output and standard error will be printed on the screen. --appdir A Directory containing the application with bin and lib subdirs. The app_command must be in appdir/bin subdirectory. Both bin and lib directories are sent to execution machine for every run. So be careful not to specify a large installed application directory. This option eliminates need to install apps on venues other than local. See your application provider. --envars List of environment variables separated by commas. Only specify names here, values must be set before calling the batchsubmit command thus allowing special characters. For local execution, all environment variables are commuted.

HOUR 2 Agenda • Simple parallel execution (The --ncpus argument to batchsubmit) • Parallel opensees (how to modify sequential input to be parallel input) • How to use batchsubmit for other venues. • Overview of various NEES execution High Performance Computing (HPC) venues: Local Use for testing small jobs less than 4 hours osg Use for many moderate size jobs. hansen Use for large parallel jobs Steele Use for many parallel jobs kraken and ranger (pending) ncpus<16 ncpus=1 ncpus<=48 ncpus<=8 • Advanced batchsubmit options and scripting the execution of batchsubmit. • Building bash scripts to save typing. • Scratch cleanup algorithm

Simple Parallel Execution --ncpus <value> • The above options will cause your application command to execute <value> times in parallel. Example: batchsubmit –ncpus 4 date • What good is it to run the same thing ncpus times? None, unless your application is aware that it is running in parallel. • A parallel aware application will only do 1 Nth the amount of work, knowing that the other processors will do the other parts of the work. • It is not hard to make your application become parallel aware especially with a scripting language like TCL.

Simple Parallel Execution Example: Run the same model through 27 ground motions. We want to divide the ground motions among 8 processors, PID = 0. . 7 P 0 P 1 P 2 P 3 P 4 P 5 P 6 P 7 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 /* If PID is processor number, then this can be run on all 8 processors */ For count = 0 to 26 if (count % 8) == PID then /* % gets remainder from division */ Do analysis for ground motion #count. else skip end

set pid [get. PID] set num. P [get. NP] set count 0; source Read. Record. tcl set g 384. 4 foreach scale. Factor {0. 25 0. 75 1. 0} { foreach g. Motion [glob -nocomplain -directory GM *. AT 2] { if {[expr $count % $num. P] == $pid} { source model. tcl source analysis. tcl set ok [do. Gravity] load. Const -time 0. 0 if {$ok == 0} { set g. Motion. Name [string range $g. Motion 0 end-4 ] Read. Record. /$g. Motion. Name. AT 2. /$g. Motion. Name$scale. Factor. dat d. T n. Pts time. Series Path 1 -file. Path $g. Motion. Name$scale. Factor. dat -dt $d. T -factor [expr $g*$scale. Factor] if {$n. Pts != 0} { recorder Envelope. Node -file $g. Motion. Name$scale. Factor. out -node 3 4 -dof 1 2 3 disp do. Dynamic $d. T $n. Pts file delete $g. Motion. Name$scale. Factor. dat if {$ok == 0} { puts "$g. Motion. Name with factor: $scale. Factor OK" } else { puts "$g. Motion. Name with factor: $scale. FActor FAILED" } } else { puts "$g. Motion - NO RECORD" } } wipe } incr count 1; } } Yellow highlighted code is possible in Open. Sees. MP You can remove the yellow and run in Open. Sees But it will take much longer. The value of num. P will be the –ncpus value provided to batchsubmit

How to use batchsubmit for other execution venues --venue hansen | steele | osg Future values will include kraken and ranger. Note: The batchsubmit options --nn and --ppn are not yet functional. In the future, this will allow extremely large values of --ncpus. -- ncpus will be the product of --nn and --ppn. --mpiargs This option specifies additional arguments to mpirun. Wrap these arguments in single quotes. Typically no additional arguments mpi agruments are needed.

Sample parallel jobs To save typing, I created the following scripts /apps/demo/bin/ex 1 /apps/demo/bin/ex 2 The above will just print the batchsubmit examles but not run them. The following scripts will print and run the commands /apps/demo/bin/ex 1. sh /apps/demo/bin/ex 2. sh Lets take time to study these examples.

Venue guidelines Venue ----local osg hansen steele Guidance -------------------Use for testing small jobs less than 4 hours Use for many moderate size jobs. Use for large parallel jobs Use for many parallel jobs --ncpus -------ncpus<16 --ncpus=1 --ncpus<=48 --ncpus<=8 • Future venues to include kraken and ranger. • Xsede (formerly teragrid) venues are steele, kraken, and ranger. Xsede and hansen use PBS for job submission. PBS jobs submission is automated by batchsubmit. • This batchsubmit option can change the pbs queue --xdqueue The default queue for steele is "standby". The default queue for hansen is "nees".

Advanced batchsubmit options --jnpref Job name prefix, default is "job". Blanks not allowed Environment variable JNPREF will also override this. Try "export JNPREF="run_" before batchsubmit. --jobname Specify jobname and override autoincrement generated jobname. Recommend not to use this to avoid jobname collisions. However, if a collision occurs with an existing scratch dir, batchsubmit will create a new directory. --xdqueue Queue for xsede machines (steele or hansen) The default queue is "standby". The default queue for hansen is "nees".

Building bash scripts • Commands can be stored in a file and these files can be executed • File can be “ sourced” or executed. • Recommend you store your personal scripts in $HOME/bin • Text Editors available on NEEShub gedit nano vi

Scratch Cleanup Algorithm 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. IF used < 75% THEN EXIT report no activity required Delete all jobs > 1 yr old , log action FOR ACTION = compress, archive (phase 1 , phase 2) | FOR T=6 m, 5 m, 4 m, 3 m, 2 m, 4 w, 3 w (pass 1, pass 2, …) | | FOR X= 5, 10, 20, 40, ALL | | | Calculate set of top X users of scratch space | | | FOR SIZE=128 G, 32 G, 8 G, 2 G, 512 M, 128 M, 32 M | | FOREACH rundirectory | | | IF rundirectory size>SIZE AND | | | rundirectory is owned by X AND | | | rundirectory lifetime >T | | | THEN ACTION rundirectory, log action | | | | IF used < 50% THEN | EXIT report SIZE, X, T, A thresholds IF used > 50% THEN report policy failure and revise policy Values in red are policy parameters that can be revised by management as needed

GC Algorithm Lemmas • • No jobs < 3 weeks old will ever be deleted or compressed without a policy change. Very small jobs < 32 MB compressed will be never be deleted by the system. – Worst case: 500, 000 32 MB jobs would consume 50% of a 32 TB scratch. • • • Process largest to smallest jobs for a fixed set of users and older than a specific age (inner loop) Process sets of large users with jobs older than a specific age (middle loop) Outer loop – Pass 1 Process jobs > 6 months – Pass 2 Process jobs > 5 months … • No jobs are deleted until all jobs >3 weeks old and > 32 MB have been compressed. Compression is phase 1, deletion is phase 2. • Example report stream: – – – – Day 1 : Day 2 : Day 3 : Day 4 : Day 5 : Day X-1 : Day. X : No activity, 74% used >2 GB, Top 10 users, >3 months old, compressed 50% used >32 MB, All users, > 3 weeks old, compressed 50% used (closest call to deletion) > 32 GB, Top 5 users, >2 months old, deleted 45% used No activity, 65% used >2 GB, All users, > 3 weeks old, deleted 50% used (close to policy failure) >32 M, All users, > 3 weeks old deleted, 60% used , POLICY FAILURE Policy parameters need adjustment

Topics Not covered in this webinar • Use of batchsubmit to build User Interface • Use of pegasus for workflow management – This is in development and test. A single pegaus job can submit many jobs that have inter-job dependencies. • Creation of appdir for portable applications. Only functional appdir today is /app/openseesbuild/osg • Modification of Open. Sees source to create personal copy of Open. Sees with custom materials and models. – Process in development with Prof. Elwood’s graduate student.