http europlanet-ri eu Summary of distributed tools of

http: //europlanet-ri. eu Summary of distributed tools of potential use for JRA 3 Dugan Witherick HPC Programmer for the Miracle Consortium University College London JRA 3 Workshop @ UCL: 22 -24 April 2010

Talk Outline JRA 3 Workshop @ UCL: 22 -24/04/2010 2

Ø Summarise the Goals of JRA 3 Tasks 3 and 4 l Task 3: Distributed Numerical Modelling Laboratory (DNML) l Task 4: Distributed Data Analysis Laboratory (DDAL) What is IDIS? Ø What are other people doing (two examples)? Ø A possible solution from the Virtual Observatory Ø l Who’s using Virtual Observatory Technology l The Universal Worker Service JRA 3 Workshop @ UCL: 22 -24/04/2010 3

Summary of Tasks 3 and 4 Goals (a reminder) JRA 3 Workshop @ UCL: 22 -24/04/2010 4

Ø Task 3: l The main goal of this task is to bring together the numerical simulation efforts currently underway in Europe and to create a coordinated European multidisciplinary modelling facility for Planetary Science: a Distributed Numerical Modelling Laboratory (DNML). l …. feeding into the catalogue part of SA IDIS. Ø Task 4: l The main goal of this task is to bring together expert teams in data analysis with their present research infrastructures and create a coordinated European interdisciplinary data analysis facility for Planetary Science: a Distributed Data Analysis Laboratory (DDAL). l …. feeding into the catalogue part of SA IDIS. Ø The main technical requirements for these tasks could be satisfied with the same solution JRA 3 Workshop @ UCL: 22 -24/04/2010 5

Integrated and Distributed Information Service (IDIS) JRA 3 Workshop @ UCL: 22 -24/04/2010 6

Ø What is the purpose of IDIS? l The task of IDIS is to provide an easy-to-use web-based platform to give access to available data, initiate new research activities needed to interpret observations or to solve open questions, locate teams and laboratories with special knowledge needed, and exploit synergies between space-based missions and capabilities of groundbased observatories. l A set of tools for describing, accessing and combining information and data from different sources will be developed, with the objective to offer a Virtual Observatory like access to many data essential for planetary research from European and non-European sources. Ø Any solution we come up with must fit in with the existing plans of IDIS JRA 3 Workshop @ UCL: 22 -24/04/2010 7

What are other people doing? (two examples) JRA 3 Workshop @ UCL: 22 -24/04/2010 8

Ø Automated Multi Dataset Analysis (AMDA) l A generic Webtool for Space Physics data o automated event search and characterisation o catalogue generation and exploitation o automated database conditional extraction o access to remote Data Centers Web portal located at http: //cdpp-amda. cesr. fr/ Ø The service can be reached and used from other sites according to the Virtual Observatory paradigm. Ø Not so much a distributed data analysis tool but a single tool linked into the Virtual Observatory Ø JRA 3 Workshop @ UCL: 22 -24/04/2010 9

JRA 3 Workshop @ UCL: 22 -24/04/2010 10

Ø Community Coordinated Modelling Centre CCMC l http: //ccmc. gsfc. nasa. gov/ l Focus on Space Weather Ø Models are run through an automated request system: l Instant Run – results are provided immediately or l Run on Request – user is informed by email when run has completed o These longer running models are added to a queue to run on their Beowulf Cluster Capable of running more than one model/analysis tool Ø Not really distributed Ø l Appears that all work is being completed at one site JRA 3 Workshop @ UCL: 22 -24/04/2010 11

JRA 3 Workshop @ UCL: 22 -24/04/2010 12

JRA 3 Workshop @ UCL: 22 -24/04/2010 13

Remember IDIS (Distributed is a keyword) JRA 3 Workshop @ UCL: 22 -24/04/2010 14

JRA 3 Workshop @ UCL: 22 -24/04/2010 15

A Possible Solution from the Virtual Observatory JRA 3 Workshop @ UCL: 22 -24/04/2010 16

Ø Common Execution Controller (CEC) l Web service for running various asynchronous applications. l Part of the Common Execution Architecture (CEA) suite of software from Astrogrid. l Designed to wrap existing applications so that they are presented as IVOA standard web services. o Commandline applications o Existing web form based HTTP applications o Custom written Java applications l Suitable for applications that: o have no graphical interface o need no user interaction once launch l Supports the following standards: UWS, VOSI, General Security, Registry, Delegation, VOSpace JRA 3 Workshop @ UCL: 22 -24/04/2010 17

Ø The Universal Worker Service (UWS) l An International Virtual Observatory Specification l http: //www. ivoa. net/Documents/UWS/ l The Universal Worker Service (UWS) pattern defines how to manage asynchronous execution of jobs on a service l Has features over the previous standards of o Control for job lifetime o Access to user sandbox (in order to inspect intermediate files l There are UWS deployments that do not use the CEA o Not necessary to use CEA JRA 3 Workshop @ UCL: 22 -24/04/2010 18

Ø Virtual Observatory Registry l Essentially the “Yellow Pages” or Telephone Directory for services l User client (web portal or client tool) is used to search the registry o Information about the resource is returned § Brief summary § Contact Name § Inputs for the model/tool § Location for the model/tool l Two types of registry o Local registry which lists the services installed locally o Full registry harvests the local registries to find out what they have installed o Users search the full registry JRA 3 Workshop @ UCL: 22 -24/04/2010 19

CEA Service Running H 3 PCOOL UCL Publishing Registry EMDAF Full Registry CEA Service Running VOISE CEA Service Running Other Codes Graz Publishing Registry Codes Running at Graz Other Publishing Registries Codes running at Other Local Registries JRA 3 Workshop @ UCL: 22 -24/04/2010 20

IDIS Full Registry Virtual Observatory Full Registry EMDAF Full Registry NVO (STSci) Publishing Registry Vizier Publishing Registry VAMDC Publishing Registry JRA 3 Workshop @ UCL: 22 -24/04/2010 21

Ø Advantages to this plan l Allows us to take advantage of the extensive effort made in developing standards o Saves us a lot of effort l We can use/modify existing VO compliant tools o Tool for OPerations on Catalogues And Tables o Aladin. VO – VO compliant image analysis tool o VOSpec – spectral analysis tool l Possible access to workflow tools l Provides us with VO access to other data/tools l Some IDIS data sources are already published to the VO Ø Disadvantages to this plan l Unclear whether this would be supported by IDIS l The VO has had its own problems which we may end up getting caught in l This may not be appropriate for fulfilling our deliverables JRA 3 Workshop @ UCL: 22 -24/04/2010 22