Open DSP — Condor to DRMAA layer DRMAA

Open. DSP - Condor to DRMAA layer (DRMAA to Condor ; -) Krzysztof Kurowski Poznan Supercomputing and Networking Center Poland European Condor Week , Milan, Italy 26 -29 June 2006

Introduction to PSNC • PSNC was officially established as a non-profit institution affiliated to the Institute of Bioorganic Chemistry Polish Academy of Science in 1993 • A few years later, a critical mass has been achieved among PSNC and MANs in Poland… and today: • PSNC is an SUN Excellence Center in Portals, New Generation Networks and Grid Technologies and well known worldwide research lab • PSNC has been actively involved in 20 EUfunded IST projects • PSNC is a technology and knowledge transfer center to academia as well as to business (spin-off, start-up, …) • PSNC is an operator of PIONIER: Polish Optical Internet European Condor Week , Milan, Italy 26 -29 June 2006

PIONIER Strategic Research Objectives • Construction of the broadband optical Internet, • Development and verification of pilot services and applications for the Information Society, creating a base for new developments in: e. Science, e. Learning, e. Health, government and local administration as well as industry and business services, • Creation of competitive conditions for active software development for new applications and services in the Information Society. European Condor Week , Milan, Italy 26 -29 June 2006

PIONIER: the current status You are here European Condor Week , Milan, Italy 26 -29 June 2006

Grid R&D at PSNC European Condor Week , Milan, Italy 26 -29 June 2006

i. Grid 2005 Grid. Lab demo Gridge middleware Displaying Computing Nodes Extracting Encoding Merging Computing Nodes Data flow Computing Node European Condor Week , Milan, Italy 26 -29 June 2006 Data flow

New Generation Mutimedia Services and Portals • Virtual Laboratory is a distributed workgroup environment, with the main task of providing a remote access to the various kind of rare and expensive scientific laboratory equipment and distributed computational and data resources. • i. TV platform (TV/multimedia on demand) enables secure and scalable digital content provisioning, which paves a new path for large scale interactive services deployment used by various e. Science communities. • e. Health Telemedicine project provides a secure web collaboration platform for surgery and on-line consultation in the hospital treatment process together with mobile services for remote and easy -to-use access to medical databases from mobile devices and PDAs. • EU e. Health ACGT project brings together internationally recognized leaders, including PSNC, in their respective fields, with the aim to deliver to the cancer research community an integrated Clinico-Genomic ICT environment enabled by a powerful Grid infrastructure. European Condor Week , Milan, Italy 26 -29 June 2006

New Generation Mutimedia Services and Portals European Condor Week , Milan, Italy 26 -29 June 2006

Ideas behind DRMAA • • • Existing approaches (e. g. Globus) in a relatively simple, script-based and often insecure way try to integrate capabilities provided by Distributed Resource Management (DRM) systems with both end users and administrators applications. DRMAA is a GGF API specification for "the submission and control of jobs to one or more Distributed Resource Management (DRM) systems". Providing a single interface to DRM resources, DRMAA substantially eliminates a problem currently facing commercial software developers - proprietary interfaces which make it difficult to integrate applications with DRM systems. DRMAA substantially accelerates the adoption of direct interfaces to DRM systems, where adoption of proprietary interfaces has traditionally been low. DRMAA provides a standard on which people can create new applications using a "write-once" interface and makes grid deployment and management easier for systems administrators. European Condor Week , Milan, Italy 26 -29 June 2006

DRMAA in DRMs European Condor Week , Milan, Italy 26 -29 June 2006

What is possible now? • Currently, for each end user a physical local account must be set up in advance to allow him access to operations provided by DRMAA. • End user has to login first to a submission or execution DRMs host before any routine from DRMAA is invoked. • It is possible to use DRMAA APIs only locally. • Communication processes between end users' applications using DRMAA are not encrypted and usually separated from the Internet by firewalls and appropriate Intranet configuration. • Administrators have limited or no control over DRMAA routines used by end users. European Condor Week , Milan, Italy 26 -29 June 2006

Customers and Enterprises needs… Currently, customers main concerns are security and interoperability. This is the reason why I asked about DRMAA and Web Services. I have no doubt that your open. DSP will address a lot of our interoperability issues. This is great stuff! Melvin Koh Chee Kian Asia Pacific Science & Technology Center Sun Microsystems Inc. European Condor Week , Milan, Italy 26 -29 June 2006

What is still missing? • Reference implementations of the DRMAA specification available today (e. g in Sun Grid Engine v 6 and Condor) allow much better integration of applications with DRMs, but … • In addition to DRMAA APIs, consistent, remote and multi-user access, and policy-based job control over various DRMs are needed. • Based on our experiences and best practice efforts, we have taken seriously security (Authentication Authorization Accounting), performance, interoperability, easy of use, and flexibility requirements in order to design a set of modules based on DRMAA and its current implementations to provide multi-user access to operations on jobs in DRMs, (in particular in SGE and Condor). • Support for standards accepted and widely used by customers (Web Services, SAML, etc. ) in order to allow interoperability between business applications and DRMs. • Thus, one of the main motivations for designing and implementing DRMAA Service Provider (DSP) is a lack of such solutions today (existing ones do not meet mentioned requirements). European Condor Week , Milan, Italy 26 -29 June 2006

Open. DSP architecture Open DRMAA Service Provider (open. DSP) and its modules Existing DRMAA routines provided by DRMs European Condor Week , Milan, Italy 26 -29 June 2006

Release notes (Open. DSP 0. 1) - Built-in support of DRMAA job submission, control, monitoring and error handling over DRMs - Authentication and encryption mechanisms based on SSL or GSI - Successfully tested with external services based on Globus Toolkit 2. 4. x, 3. 2. x and 4. 0 - Authorization callouts and external access control list - External accounting and auditing database available - Stage-in/out operations supported over DIME/MIME attachments - JSDL 1. 0 POSIX based job description - Successfully tested with the following resource/cluster management systems (where is Condor? ): SGE (SGEv 6 u 4, SGE v 6 u 5, SGEv 6 u 7 recommended) over the following OS systems: - Linux x 86, kernel 2. 4, glibc >= 2. 2 - Linux AMD 64 (Opteron), kernel 2. 4, glibc >= 2. 3 - Solaris (Sparc) 9 64 -bit - Solaris (x 86) 9 - Apple Mac OS/X - Java, . NET and C-based example clients and SDKs available - Distributed under the GPL license European Condor Week , Milan, Italy 26 -29 June 2006

Collaboration with Condor On Jun 21, 2006, at 3: 39 PM, Peter Troeger wrote: >> Hi Becky, >> >> I received a big improvement patch for the Condor DRMAA library by a >> research project in Poland. The people are interested when they could >> see the changes in the official Condor release. Is there a timeframe >> for a new DRMAA library version ? >> >> Best regards, >> Peter. That's great! I'll add the patch to the next Condor release. Is the patch currently in Source. Forge, or should I obtain it from another location? I'll pull the latest files when I hear from you as to where they are. Thanks, Becky European Condor Week , Milan, Italy 26 -29 June 2006

Collaboration with Condor >> The Condor people will provide your patched >> version with Condor 6. 7. 21. Since 6. 7. 20 >> is out since yesterday, I expect the next >> version in 4 -6 weeks at the earliest. Would that >> be enough for you ? Once a new Condor version is released please visit: http: //sourceforge. net/projects/opendsp European Condor Week , Milan, Italy 26 -29 June 2006

What’s next? Fed. Stage Computing Provider • Fed. Stage Computing Provider is a generic job execution, monitoring and management HPC service provider accessible through a well-defined Web Service interface based on DRMAA specification and its reference implementations supported by local cluster/resource management systems, such as Sun Grid Condor • • • Engine, or PBS/Torque. Remote Interfaces: Web Service over SSL, GSI and SSL/SAML 2. 0 with JSDL 1. 0 support AAA Module: Tested and integrated with internal and external AAA modules (Fed. Stage Identity Manager and Fed. Stage Authorization Manager) Distributed Resource Manager: Sun Grid Engine (SGEv 6 u 4, SGE v 6 u 5, Condor SGEv 6 u 7, SGEv 6 u 8), (6. 7. 21 and higher), PBS/Torque(2. 0 and higher) Computing Node: Solaris (Sparc), Solaris (x 86), Free. BSD, Linux x 86, Linux AMD 64, 9/10 Mac X OS, Windows 2000/XP, Available: commercial license and open source under GPL (Open. DSP project) European Condor Week , Milan, Italy 26 -29 June 2006

Fed. Stage: a new wave of HPC solutions? • Done from scratch based on our experiences and lessons learned • Interoperability, Peformance, Security are the key issues and therefore Fed. Stage AAA-based solutions are of primary importance for the cross-organization sharing of applications, data, and computational services in a collaborative business environment today. • Fed. Stage products follow Liberty Alliance and WS-I • W 3 C standards: XML, SOAP • OASIS standards: WS-Security, SAML 2. 0 WS-Notification, WS-Reliability, • GGF standards(recommendations): DRMAA, JSDL • IETF standards: SSL/TLS European Condor Week , Milan, Italy 26 -29 June 2006

Fed. Stage: a new wave of HPC solutions? • • • The convergence of Web Services, Grid technologies and SOA/ESAs that currently occurs among enterprise, media, mobile and Internet domains requires a standard and secure approach for virtualization of IT resources and technology independent interoperability across platforms, services and systems Fed. Stage products are loosely coupled services (SOA) which may be individually useful, or they can be composed to be used in specific business processes. Fed. Stage products may be completely self-contained, or they may depend on the availability of other enterprise services. Fed. Stage products capabilities as well as communication and data channels among them and consumers are protected by transport and message level security mechanisms. To deliver the best-quality software to the market we combine innovations with best practices, and thus follow: – – industry specifications, security standards and recommendations, pure Web Services requirements, WS-* extensions. European Condor Week , Milan, Italy 26 -29 June 2006

Enterprise Access Management And Control Products: Fed. Stage Identity Manager Fed. Stage Authorization Manager Enterprise Infrastructure Service Providers: Fed. Stage Computing Provider Fed. Stage Storage Provider Fed. Stage Notification Provider Official release: 3 Q 2006 European Condor Week , Milan, Italy 26 -29 June 2006

How old is this picture? European Condor Week , Milan, Italy 26 -29 June 2006

Collaboration with Condor • • • Peter Troeger Dan Bradley Becky Gietzel Alan De Smet R. Kent Wenger European Condor Week , Milan, Italy 26 -29 June 2006

Thank you! krzysztof. kurowski@man. poznan. pl European Condor Week , Milan, Italy 26 -29 June 2006