Using the ARCS Grid and Compute Cloud Jim

Using the ARCS Grid and Compute Cloud Jim Mc. Govern

Some Housekeeping First thing – get quota in the ARCS Compute Cloud. By registering with ARCS Access Service, you will automatically be given quota in the ARCs Compute cloud. Quest. Net 2011

Compute Resources used by Researchers may access a range of resources to meet their compute needs. For example Their own PC Their own University servers or HPC A regional HPC facility, e. g. QCIF Facilities at another university (possibly national) remote merit-based allocation reserved for their discipline or project. E. g. NCI, VLSCI, QFAB. Quest. Net 2011

National HPC Resources Quest. Net 2011

Problem: Multiple Systems Quest. Net 2011 More places to submit jobs Even more to learn (submission systems, quirks, etc) Need accounts set up at each site Firewall dramas, etc

Problems associated with Access to Diverse, Remote Compute Resources Different submission systems Different accounts and authentication Different administration systems/ people to deal with. Different file systems and difficulty in sharing output or using shared input. Need for support/ capability Quest. Net 2011

Solution: Grid - Unified Submission System Use Globus Quest. Net 2011 One submission system to understand Handles local quirks automagically A kind of single login (via certificates) However. . .

Example Job Script /usr/bin /whoami Quest. Net 2011

Better Solution: Grisu Single Grisu core with access to all resources Client machine connects to this core to manage all aspects of job creation, submission, and result collection Provides a simplified API for the Grid Quest. Net 2011 Reduces firewall issues Shibboleth logins

Even Better Solution: The ARCS Compute Cloud App 1 App 2 App n Quest. Net 2011 Access via applications (“application as service”) One account gives access to all Shibboleth login Minimal input with standardised implementations

ARCS Compute Cloud Provides a web-based Graphical User Interface for job submission Same interface for all systems, even though target is different. Uses AAF for authentication – use your own institution's credentials Is integrated with ARCS Data Fabric Provides support resources Quest. Net 2011

ARCS Compute Cloud Provides useful resources across many HPC resources that any one can access. Provides Application as a Service: access package, hides Compute Resource Selects best location to run jobs from those available. Minimises user input Can free up desk top (e. g. rendering, statistical analysis) Quest. Net 2011

Demonstration: Running an application through ARCS Compute Cloud Quest. Net 2011

Re-cap on demonstration Single Web-based GUI for HPC job management Input from Data Fabric/Desk top Output to Data Fabric Login with your home institution's credentials Ability to use defaults only. Quest. Net 2011

Run an application through ARCS Compute Cloud Go to ARCS Web site and select the “Services Login” Login using your Id. P Select Compute Cloud Select New Job Select Underworld application from list on LHS Select Data Fabric for input files. First underworld file is primary input file, and second is additional input file. Quest. Net 2011 Submit Job

Questions? Quest. Net 2011

Extending the ARCS Grid and Compute Cloud Jim Mc. Govern Quest. Net 2011

Accessing other resources on the ARCS Grid The ARCS Compute Cloud provides a useful facility for those who want to use a standard available application in a standard way. The ARCS facilities also support: Roll your own code New applications (new templates) Restricted resources or applications Quest. Net 2011 Building your own workflows:

Accessing other resources on the ARCS Grid Roll your own code: - Java template - Python Scripts - Custom Scripts Quest. Net 2011

Accessing other resources on the ARCS Grid What resources are available on the Grid? See http: //nf. nci. org. au/facilities/software_category. php This provides all information, links to HPC providers Quest. Net 2011 .

Accessing other resources on the ARCS Grid Building new templates Create a text file of properties https: //projects. arcs. org. au/trac/grisu/wiki/Grisu. Cloud. Service/Job. Template Submit to ARCS Place in a particular repository folder – e. g. under Earth Sciences with. template extension Automatically put in Web interface (Compute Cloud) Produces RSL that is translated to local submission system Quest. Net 2011

Accessing other resources on the ARCS Grid http: //projects. arcs. org. au/trac/grisu/wiki/Grisu. Cloud. Service https: //projects. arcs. org. au/trac/grisu/wiki/Grisu. Cloud. Service/Job. Template Quest. Net 2011

Building New Templates commandline = Underworld ${file} application = Underworld application. Version = 1. 4. 1 = Generic = Quest. Net 2011

Building New Templates [cpus] - type = Cpus [jobname] title = CPUs type = Jobname size = 100 x 100 default. Value = underworld Default. Value = 4 title = Jobname - size = 2000 x 70 [walltime] type = Walltime title = Walltime default. Amount = 10 default. Unit = hours size = 200 x 100 #[email] #type = Email #size = 400 x 100 Quest. Net 2011

Accessing other resources on the ARCS Grid Restricted Applications and Facilities: – – Licensed software Restricted allocations • Merit allocations Quest. Net 2011

Accessing other resources on the ARCS Grid Create a VO that embodies particular accounts. (approved by sys admins) Map shib credentials to those VOs. E. g. – – – QCIF users Fluent users NCI/QFAB/VLSCI users Quest. Net 2011

Accessing other resources on the ARCS Grid Supporting your own work flows/ interfaces: – BES Standard interface • – RESTful interface • – Web services that allow management of activities including processes that run compute jobs Easier to invoke web services suitable for quick easy development with Python clients for example SLCS Delegation Service • Quest. Net 2011 Use Shib credentials in your apps

Accessing other resources on the ARCS Grid Adding your own resources: – – – Add your facility/ resource (Condor) to Grid Documentation Assistance Set up and test Globus gateways Create MDS (Information Service) entries Quest. Net 2011

Non-Standard use Summary The compute cloud can be tailored for: compile and run apps new templates particular sets of users (licensees/ authorised users) particular compute resources Specific work flows and interfaces Add your resource to grid Quest. Net 2011

How to get started? Standard Applications: Register with ARCS and you will have access to these. For other applications, projects or work flows, call the ARCS help Desk and we will work with you to set up what is needed. Quest. Net 2011

Questions? Quest. Net 2011