VEIGA Very Easy Interface to Grid Applications formerly

VEIGA: Very Easy Interface to Grid Applications (formerly known as "Styx Grid Services") Jon Blower Technical Director Reading e-Science Centre jdb@mail. nerc-essc. ac. uk http: //www. resc. rdg. ac. uk

Overview of VEIGA • Aim is to make running applications on a Grid as easy as running local applications • Apps can be run from remote locations using the same command as for running the app locally • Simple workflows can be created using simple shell scripts – Data can be streamed directly from app to app • Can run applications on Condor and Globus resources • Secure

Example: GULP • General Utility Lattice Program (Julian Gale, IC) – Used extensively in mineral sciences (e. g. NERC e-Minerals project) • Takes a single input file (input. txt) describing a crystal lattice • Outputs a single file (gulp_0001. txt) with bond lengths, Helmholtz free energy etc • Having deployed GULP on a VEIGA server, it is run by clients like so: – veigarun gulp • Or can create simple wrapper script so that we can just type: – gulp

How it works • • GULP is installed on the VEIGA server holds an XML description of the GULP program: • •

• veigarun is a completely generic VEIGA client program and performs the following tasks: – Downloads the XML description of GULP – Uses it to parse the command line to find out what input and output files to expect (in case of GULP, these happen to have fixed names) – Uploads input file(s) to VEIGA server – Starts GULP, which runs on the VEIGA server – Redirects the standard streams (stdin, stdout, stderr) – downloads the output file(s) as they are produced (doesn't wait for finish)

Some Benefits • Very easy to deploy and run applications in the VEIGA environment – See http: //www. resc. rdg. ac. uk/veiga • • Useful when application is tied to a particular platform but you want to run it from elsewhere and pretend it's local Can easily substitute the "Grid-enabled" GULP for your local GULP – There is a GUI for GULP that wraps the GULP executable. Can simply point this to the remote GULP instead of a local installation: no change to GUI code • Can use the remote application in place of local applications in shell scripts – Easy to create "workflows" – Can use all shell constructs such as pipes etc and combine with local apps – Can arrange for data to take most direct route between servers • Can do computational steering – Simplementation but it works • Custom clients and server plug-ins can be created using a Java API – Documentation on this is not great yet… – WSRF bindings have been created (Andrew Harrison, Cardiff): VEIGA app modelled as WS-Resource

VEIGA and Condor Worker nodes • If VEIGA server is a Condor submit host, can delegate jobs to a Condor pool • Helps to load balance – Not everything runs on VEIGA server • Multiple jobs can be run with a single command: – Upload directories instead of single files: – myprog –i inputs –o outputs – Easy high-throughput computing • Demo VEIGA Server + Condor VEIGA client

VEIGA and Globus myproxy server NGS • – E. g. NGS • VEIGA server VEIGA can be used as a portal to Globus resources Users log on to VEIGA with My. Proxy user/password – VEIGA retrieves proxy certificate • Helps with Globus firewall problem – VEIGA server takes pain on behalf of client • VEIGA client VEIGA server used as staging post for files – Could transfer files direct from client to destination but would require client to have Grid. FTP or similar

Some technical points of (possible) interest • VEIGA uses the Styx protocol for distributed systems – JStyx – Java implementation of Styx developed at Re. SC – http: //jstyx. sf. net – Styx is ~20 years old • Single server process handles all traffic – File upload/download, progress monitoring, control messages – Styx is a multiplexing protocol – Can tunnel traffic through SSH • • Client-server connections are persistent Servers only need one incoming port open through firewall Clients need no incoming ports open VEIGA only requires a JRE – includes all other dependencies – Whole system is a 3 MB download

Security • VEIGA can be set to permit access to registered users only – Contains set of usernames/passwords. Passwords stored as secure hash, not plain text – No client certificates! (my bank does not use client certs) • All traffic can be encrypted using SSL – Server has a certificate • Or, everything can be tunnelled down SSH – Because VEIGA is a single server process with single port – Keeps some sysadmins happy • For access to GSI-protected resources, users can log into a VEIGA server with My. Proxy password – Then VEIGA retrieves proxy certificate for them – VEIGA behaves like a Grid portal

Possible test case: GCEP • Grid for Coupled Ensemble Prediction • Run ensembles of climate simulations and get stats of results • Use resources at Reading, RAL, British Antarctic Survey (BAS), NGS • Need to share results among project partners and allow partners to submit jobs – Need a resource broker (even if very simple) • Can we do the whole thing using SSH (and VEIGA)? – No Globus, no GSI, no proxy certs (except for NGS access) – SSH very well understood and trusted by all relevant sysadmins and already installed – Shared file system with sshfs (or SRB if not) – Give me a good reason why not!

Conclusions • VEIGA favours simplicity over efficiency – A simple tool for simple jobs • Users already know how to use it! – Remote apps can be run just like local ones – Doesn't feel like job submission • • Simple workflows are created with shell scripts Can use VEIGA as an easy-to-use "portal" to Condor and Globus – Client interface is identical in all cases • Could use same principles to launch jobs using AHE / Grid. SAM – VEIGA client does not know about the back end – VEIGA as AHE client plug-in? • See http: //www. resc. rdg. ac. uk/veiga – (as of 4/5/06) – Contains tutorials, links to papers etc – Current release (0. 2. 0) does not contain everything I've talked about today – next release (0. 3. 0)!