Component-Based Portals for Grid Computing Marlon Pierce Community Grids Lab Indiana University OGCE Consortium
NSF NMI Project for Reusable Portal Components: Who We Are • University of Chicago – Gregor von Laszewski • Indiana University – Marlon Pierce, Dennis Gannon, Geoffrey Fox, and Beth Plale • University of Michigan – Charles Severance, Joseph Hardin • NCSA/UIUC – Jay Alameda, Joe Futrelle • Texas Advanced Computing Center – Mary Thomas OGCE Consortium
What Is Grid Computing? • Grid Computing provides an overlay infrastructure that can be used to bind computing and data resources from multiple organizations into “virtual organizations”. – Security, information services, resource access protocols, file transfer, etc. • Open Grid Services Architecture recasts Grid capabilities as Web Services – WSDL descriptive conventions, advanced features for transient services, etc. – Service hosting environments manage service lifecycles, interactions with requestor agents. • But what about the clients? – And what about user centric services? OGCE Consortium
Towards A Common Grid Client Hosting Environment Grid portal background and emerging common frameworks OGCE Consortium
What Is a Computing Portal? • Browser based user interface for accessing grid and other services – – – “Live” dynamic pages for accessing grid services Use(d) Java/Perl/Python COGs Manage credentials, launch jobs, manage files, etc. Hide Grid complexities Can run from anywhere Unlike user desktop clients, connections go through portal server, so overcome firewall/NAT issues • Combine “Science Grid” with traditional web capabilities – Get web pages for news feeds – Post and share documents – And other more traditional web page features • Customizable interfaces and user roles/views OGCE Consortium
Let 10000 Flowers Bloom • Many portal projects have been launched since late ’ 90 s. – Hot. Page from SDSC, NCSA efforts, DOD, DOE Portals, NASA IPG – 2002 Special Issue of Concurrency and Computation • Continue to be important component of many large projects – NEESGrid, DOE Sci. DAC projects, NASA, NSF, many international efforts • Global Grid Forum’s Grid Computing Environments Research Group – Community forum OGCE Consortium
Three-Tiered Architecture Portal User Interface Grid and Web Protocols JDBC, Local, or Remote Connection Portal Client Stub Database Service Database Portal Client Stub Grid Resource Broker Service HPC or Compute Cluster Portal Client Stub Information and Data Services Grid Information Services, SRB Three-tiered architecture is accepted standard for accessing Grid and other services OGCE Consortium
Problem with Portals • GCE revealed two things – Everyone was doing the same thing • Not quite, but significant • Everyone builds secure logins, remote file manipulation, command execution, access to info servers. • Everyone would at least like support for multiple user roles (administrators, users) and customization – No one could share components with other groups • No well defined way of sharing UI components or making services interoperate. • No well defined interfaces to portal services. • A research opportunity! – Two levels of integration: user interfaces and services • Our challenges – Stop reinventing things and provide ways for groups to reuse components. – Provide a portal marketplace for competing (advanced) services. – Provide APIs for service integration OGCE Consortium
A Solution based on components • A software component is object defined by – A precise public interface – A semantics that includes a set of “standard” behaviors. • A Software component architecture is: – A a set of rules for component behavior & – A framework in which components can be easily installed and interoperate. • The component architecture of choice for the Portal community is the one based on portlets – Java components that generate content, make local and remote connections to services. – Portal containers manage portlet lifecycles OGCE Consortium
A Portlet Approach to Grid Services • A Portlet is a portal server component that provides basic services rendered in a user-configurable window in a portal pane. Event and logging Services Portal Server Portlet 1 Portlet 2 Portlet 3 Portlet 4 Portlet 5 Portlet 6 My. Proxy Server Metadata Directory Service(s) Application Factory Services Messaging and group collaboration Directory & index Services OGCE Consortium
The Grid Portal • Provides Portlets for – Management of user proxy certificates – Remote file Management via Grid FTP – News/Message systems • for collaborations – – Grid Event/Logging service Access to OGSA services Access to directory services Specialized Application Factory access • Distributed applications • Workflow – Access to Metadata Index tools • User searchable index OGCE Consortium
OGCE Foundations: Portal Containers and Grid Access OGCE Consortium
Portlet Component and Container Technologies • Jakarta Jetspeed – Open source Java portlet project – Jetspeed is both a framework and reference implementation – Defines portlets, portal service APIs (login, authorization, customization, etc. ) • CHEF from University of Michigan – Uses Jetspeed as a framework • Reimplements many of the core classes – Basis for UM Course. Tools – NEESGrid portal – CMCS Portal OGCE Consortium
Background • CHEF is organized around groups of users • Portals in CHEF are group based (a group can consist of only one person!) • A user sees the Portals for each group of which that person is a member • The Portal is a collection of Portal pages • Each Portal page contains one of more teamlets OGCE Consortium
CHEF Architecture Configurable as to what implementation provides what service Services API Services Persistent System-wide Multiple implementations of services Teamlets: Written in JAVA Responsible for GUI Operate in the context of a session. Rely on services for any persistent or “cross-user” information. Portal Engine: Jetspeed Velocity CHEF Web Server: Tomcat Turbine Servlets: Access services outside of the portal engine: Access. Servlet and Web. Dav. Servlet Non-HTTP Components (i. e. E-Mail) OGCE Consortium
What is a Teamlet? • A teamlet is a portal-like presentation of information and possible user actions • It can be placed in multiple places with a portal in across multiple portals; each placement is independent • Each placement is configurable • Each placement belongs to a portal; and is therefore associated with a group OGCE Consortium
Design Process - Elements • The design of a teamlet consists of three elements – A service (the Java class or classes that implement the interface to a source/store of information) – An action (the Tool in CHEF; one of more Java classes that present information to the user and respond to user actions) – The GUI (usually a set of Velocity templates) OGCE Consortium
Java Co. G Kit • Provides interfaces to elementary Grid functionality – Copy a file from here to there – Execute a remote job on the Grid – Authenticate to the Grid • Provides interfaces to more advanced Grid functionality such as simple job queues and task graphs • Provides a convenient API level interface that protects you from many changes in the Grid such as GT 1. x to GT 2. x to GT 3. x to GT 4. x OGCE Consortium
What does the user see? Portal Interface Portlet Java Co. G Kit High-Level Java Co. G Kit Low-Level GT 2 GT 3 interface GT 4 interface Condor SSH OGCE Consortium
Portal Capabilities A survey of current portal capabilities. OGCE Consortium
User Portlets Portal Capabilities Description Grid Proxy Certificate Manager Get My. Proxy certs after logging in. Schedule Interactive individual and group calendars Discussion Persistent topic-based discussion for groups Chat Live chat services and interfaces Document managers WEBDav based document system for group file sharing MDS/LDAP Browsers Basic Globus MDS browsing and navigating Grid. Context Portlets Access context services for managing metadata GRAM Job Submission Run simple executables on remote hosts Grid. FTP Upload, download, crossload remote files. GPIR Portlets View, interact with HPC status, job, etc information. Anabas Access to Anabas shared display applets Newsgroups and citation portlets Post topics to newsgroup, manage group references and citations with access controls
Grid Portlet Examples • We’ll next overview several portal capabilities. • Jetspeed/CHEF acts as a clearing house for portal capabilities – User interface components can be added in well defined ways. – First level of integration • All Grid access goes through the Java COG. OGCE Consortium
Example Capability: Portals for Users User “Beth” • The My. Proxy Manager – The user contacts the portal server and asks it to do “grid” things on behalf of the user. – To make this possible the server needs a “Proxy Certificate” • The user has previously stored a proxy cert in a secure My. Proxy Server stored with a temporary password. • User give the portal server the password and the portal server contacts the proxy server and loads the proxy. • The portal server will hold the proxy for the user for a “short amount of time” in the user’s session state. 1. Load my Proxy Certificate! Portal Server 2. Give me Beth’s proxy certificate My. Proxy Portlet COG 3. I am Beth’s Proxy My. Proxy Server OGCE Consortium
Example Capability: File Management User “Beth” • Grid FTP portlet– Allow User to manage remote file spaces – Uses stored proxy for authentication – Upload and download files – Third party file transfer • Request that Grid. FTP server A send a file to Grid. FTP server B • Does not involve traffic through portal server Portal Server Grid. FTP portlet Java COG Grid. FTP Server A Grid. FTP Server B OGCE Consortium
Example Capability: Grid Context Service • User’s want to be able to use the portal to keep track of lots of things – Application and experiment records • File metadata, execution parameters, workflow scripts – “Favorite” services • Useful directory services, indexes, links to important resources – Notes and annotations • “Scientific Notebooks” OGCE Consortium
XDirectory: A Grid Context Service • XDirectory is itself a Grid Service that is access by the portal. – An index over a relational database – Each node is either a “directory node” or a leaf. – Leaf nodes are xml elements which contain metadata as well as html annotations. OGCE Consortium
Portlet Interfaces to Grid Context Services • A Remote Service Directory Interface – Holds references and metadata about application services. • User selects interface to application service from the directory browser. • Examples: (near completion) – Select a link to a Dagman document and invoke the Condor service on the script. – Same for Grid. Ant/Ogre or BPEL workflow script. – Factory services for any grid apps that have interactive user interfaces. Portal Server Remote Service Directory Service Remote Grid Application Service OGCE Consortium
Example Capability: Topic Based Messaging Systems • Indiana University has implemented a XML metadata system based on messages. • Newsgroups – Topic based posting and administration • Citation/reference browsers – Topic based, export/import bibtex • Portlets sit atop JMS-based message system. OGCE Consortium
OGCE Consortium
User Privileges for Group Channels • Users request access to specific topics/channels. – Granted by administrator for that topic • Can request – Read/write by browser – Read/write by email (newsgroups) – Receive/don’t receive attachments. • Topic admin can edit these requests. • Super admins can manage administrators to topics OGCE Consortium
GPIR Data • Load - aggregated CPU • Downtime data for a machine – Jobs: aggregated queue • MOTD • Nodes: job usage for each machine node • NWS: based on VO and Click model • Grid Monitoring – Based on TACC GMS System – Custom providers – Plans to include MDS 3. 0 and INCA data uderway • Expanding to include: – – – queuing system application profiles performance data Application profiles Doc links • Model allows generic inclusion of any XML data from any recognized source – Need schema – Need query OGCE Consortium
Grid Portal Information Repository (GPIR 1. 1) OGCE Consortium
GPIR Components • Web Services Ingestor – Web Services Ingestor and clients – XML Schemas - can be changed • Data Repository – Local Cache – Archival --> Postgre. SQL • Web Service Query – retrieve data – XML Queries – Retrieving current snapshot and archived data • Clients – – Grid. Port services Portal/Web Interface (Portlets, servlets, JSP) Command line Any that speak web services OGCE Consortium
Future Capabilities OGCE Consortium
Major Theme: Grid Application Support • Current portal’s job submission capabilities are vanilla – Type desired machine, executable, output file – Generates RSL, runs command • Actual job management requires more – Integration of information, scheduling services, file transfers, job sequencers, events OGCE Consortium
Capability: Job Sequencer Portlets " xsi: schema. Location ="http: //grids. tacc. utexas. edu/schemas/sequencer/job. Sequence C: DOCUME~1MaytalDesktopMaytalWorkGP-IRX~1motd. xsd"> < <Status>New</Status> <Step> <Status>Unscheduled</Status> <Type> CSFJob</Type> <Parameter name="job. Factory. Service. Handle ">http: //129. 116. 218. 36: 15080/ogsa/services/metascheduler/J ob. Factory. Service</Parameter> <Parameter name="queue">normal</Parameter> <Parameter name="executable"> pam</Parameter> <Parameter name="arguments">-g 1 mpichp 4_wrapper /home/monitor/mpi_jobs/mpimd_5</Parameter> <Parameter name="directory">/home/monitor/ mpi_jobs</Parameter> <Parameter name="count">4</Parameter> <Parameter name=" td. In">/dev/null</Parameter> s <Parameter name="std. Out">/home/monitor/ pi_jobs/tomislav. Sequencer. Job. Out m </Parameter> <Parameter name="std. Err">/home/monitor/ pi_jobs/tomislav. Sequencer. Job. Err m </Parameter> </Step> <Status></Status> <Type>Grid. FTP</Type> <Parameter name="rom. Host">[Previous]</Parameter> f <Parameter name="o. Host">blanco. tacc. utexas. edu: 2811</Parameter> t <Parameter name="from. File. Full. Name">/home/monitor/ pi_jobs/tomislav. Sequencer. Job. Out m </Parameter> <Parameter name="to. File. Full. Name">/home/monitor/ pi_jobs/tomislav. Sequencer. Job. Out. Copied m </">/hom e/monitor/mpi_jobs/tomislav. Sequencer. Job. Err </Parameter> <Parameter name="to. File. Full. Name">/home/monitor/ pi_jobs/tomislav. Sequencer. Job. Err. Copied m </Parame ter> </Step> </Job. Sequence > User uses Portal to generate XML description of sequence. Currently, sequence steps can consist of File Transfers and Job Submissions to the CSF meta scheduler GPIR The XML is then decomposed and persisted to GPIR where the status information of each step in the sequence and of the sequence as a whole can be stored Sequencer Grid. Port returns a Sequence ID to the Portal immediately and then begins executing the Sequence to completion or to error. Status information can be obtained at any time with the Sequence ID OGCE Consortium
Capability: Community Scheduling Framework Portlets User Workstation User Portal Grid. Port Visualization System Bandera CSF Blanco Buda CSF Use Case • Researcher submits job through User Portal • User Portal uses Grid. Port to – – – • • authenticate user optionally make advanced reservation to visualization system submit job to CSF selects compute cluster with best fit and forwards job Gridport sends results to visualization system OGCE Consortium
O. G. R. E. —A Job Management Engine • See Thursday Demo • O. G. R. E. = Open Grid Computing Environments Runtime Engine • What Ant lacked, but we needed: • Broader conditional execution, • Ant: based on write-once String properties. • A general “loop” structure for Task execution. • Data-communication between Tasks (and with their containers). • Specialized tasks • File reading and writing • Local and remote file management (gridftp) • Web service related tasks • Event- and process-monitoring-tasks OGCE Consortium
Data and Metadata Management • When the job is through… • Simulations, experiments generate both data and metadata – Metadata includes from code input parameters, host machines, data formats, owners of data, generators of data, … • NEESGrid metadata system will be integrated into the portal release. • Another example of integrated Grid services – Grid. FTP, CAS and other services OGCE Consortium
Metadata Repository Capabilities • Data store – Files – Logical naming – Format translation • Metadata store – Structured (RDF-like schemas) – Random-access (tuple store) – Version control • Archiving – Mass store – “nar” archive format • Security – Single signon – Secure reliable file transfer with Grid. FTP – Authorization via CAS • Grid service interfaces – NFMS: NEESgrid File Management Service – NMDS: NEESgrid Metadata Service – Repo. service (Façade) – Secure remote access by applications OGCE Consortium
Repository architecture repository user GSDL, Grid. FTP HTTPS Java API, Grid. FTP JDBC, File I/O Portal Repo browser File xfer servlet CAS Repo. service (Façade) CAS DB NMDS DB NFMS File system Grid. FTP OGCE Consortium
Access Grid and Related Portlets
Architectural Upgrades Portlet standards, service managers, event standards OGCE Consortium
“A Bag of Portlets…” • Portlet/container systems provide a simple level of user interface integration. – A clearing house for pluggable components of all sorts • User interfaces are actually to a diverse set of backend services. – A mixture of UIs to Web services, grid services, communication/collaboration services, …. • We are a portlet marketplace… • But we need closer integration OGCE Consortium
OGCE Initial Architecture Proxy Portlets Teamlets Java COG API HTTP Service API Java Co. G Kit Remote Interfaces Portal Local Portlets Grid Protocols Grid Services GRAM, MDS-LDAD My. Proxy Grid Services Co. G Stubs Other Services SOAP CHEF Services Jetspeed Internal Services Initial architecture aggregates multiple services into a single portal using portlet containers OGCE Consortium
Integration Points and Service Abstractions • Internal portal service abstractions – Service layer abstractions to define how to interact with in-memory proxy certificates. • Authorization – Internal and external roles need to be integrated. • Events – Share events between services – Job submissions should automatically update the calendar operation, for example OGCE Consortium
Portal Grid Service Stubs Portlets and Teamlets Service API Local Portal Services Remote Content Services Grid. Port Toolkit Tera. Grid Integrated Architecture HTTP Grid Services Java Co. G Kit Web Services Remote Content Servers Jetspeed Internal Services Diagram demonstrates how existing software projects (such as Grid. Port) can be adapted to support NMI Portals software system OGCE Consortium
Portlet Standards • Current portal uses Jetspeed portlet API • Other portlet systems available – Websphere->Grid. Sphere • Portlet standard: JSR 168 – A common API for all next generation portlet systems. – Compliant portlet components may be shared between systems. • Open Source Implementation (Pluto) is available – We will be adopting this, will be part of our SC 2004 release – Will leverage education portal work from CHEF team. OGCE Consortium
OGCE Portals in Action Some early applications OGCE Consortium
New Starts: Tera. Grid Portal • Access to Tera. Grid Services – Version 0: Collecting Initial Services • Public Information about Resources • Private Information for the developers. – Version 1: A User centered portal (Q 2 2004) • Hotpage/Gridport style access to user accounts, credentials, job submission & management. – Version 2: Portals for Science Collaborations (Q 3 2004) • Shared spaces, whiteboards, AG access, group authorization, shared application services OGCE Consortium
LEAD Portal OGCE Consortium
OGCE Collaboratory We Can’t Do It All We’ll Take Credit For It, Though OGCE Consortium
OGCE Collaboratory • Hopefully, we have convinced you to not rebuild portals from scratch. – Time to use pluggable components in consistent frameworks. • Our award is not just to release our own software. – We want to foster the portal community – Contributed third party components will be sought. • Initial contributions will be from similar projects – CMCS and other Sci. DAC projects are closely allied OGCE Consortium
Additional Information • OGCE Web site: www. ogce. org – Download the portal software – Join news lists, get announcements • OGCE Demo Portal: www. collab-ogce. org – See our demo Thursday night • Contact us – marpierc@indiana. edu OGCE Consortium
Скачать презентацию Component-Based Portals for Grid Computing Marlon Pierce Community
1a20c81d71964ec765b363963e434c87.ppt