Development Implementation of an Inter-institutional Multi-purpose Grid

Development & Implementation of an Inter-institutional Multi-purpose Grid SURAgrid at Internet 2 Members’ Meeting, September 2005 • Mary Fran Yafchak, SURA • Jim Jokl, University of Virginia • Ashok Adiga, Texas Advanced Computing Center • Art Vandenberg, Georgia State University

Presentation agenda • • About SURAgrid - Mary Fran Yafchak SURAgrid auth. N/auth. Z - Jim Jokl SURAgrid portal - Ashok Adiga SURAgrid applications - Art Vandenberg This is a living, breathing project. Exchange of ideas encouraged throughout! Southeastern Universities Research Association

About SURAgrid • A “beyond regional” initiative in support of SURA regional strategy “Mini-About” SURA – SURA region: 16 states & DC, from Delaware to Texas – SURA membership: 62 research universities, mostly within the region – SURA mission: Foster excellence in scientific research, strengthen capabilities, provide training opportunities • Evolved from the NMI Testbed Grid project, an outgrowth of SURA’s management of the NMI Integration Testbed Program – http: //www 1. sura. org/3000/NMI-Testbed. html Southeastern Universities Research Association

SURAgrid Goals SURAgrid: Organizations collaborating to bring grids to the level of seamless, shared infrastructure Goals: ðTo develop scalable infrastructure that leverages local institutional identity and authorization while managing access to shared resources ðTo promote the use of this infrastructure for the broad research and education community ðTo provide a forum for participants to gain additional experience with grid technology, and participate in collaborative project development Southeastern Universities Research Association

SURAgrid Participants • • • University of Alabama at Birmingham* University of Alabama in Huntsville* University of Arkansas* University of Florida* George Mason University* Georgia State University* Great Plains Network University of Kentucky* University of Louisiana at Lafayette* Louisiana State University* University of Michigan Mississippi Center for Super. Computing Research* • • • University of North Carolina, Charlotte North Carolina State University* Old Dominion University* University of South Carolina* University of Southern California Southeastern Universities Research Association (SURA) Texas A&M University* Texas Advanced Computing Center (TACC)* Texas Tech Tulane University* Vanderbilt University* University of Virginia* *SURA member Southeastern Universities Research Association

Current Status • Grid-Building Ø Themes: heterogeneity, flexibility, interoperability, scalability Ø More – Ashok Adiga, Texas Advanced Computing Center • Inter-institutional Auth. N/Auth. Z Ø Themes: maintain local autonomy; leverage enterprise infrastructure Ø More – Jim Jokl, University of Virginia • Application Development Ø Themes: immediate benefit to applications; apps drive development Ø More – Art Vandenberg, Georgia State University • Project Planning Ø Provided and facilitated by SURA Ø All meeting notes available on SURAgrid Web site Ø First in-person meeting held Sept. 7 & 8, 2005 Southeastern Universities Research Association

In the coming months… • Will continue evolving key areas – Grow and solidify grid infrastructure – Continue expanding and exploring auth. N/auth. Z – Build application set; help to grid-enable new applications • More formal work on organizational definition – Charter, policy, governance • Pursuit of funding opportunities and additional collaboration – Four collaborative proposals submitted this summer; more anticipated in the next six months – Some areas of interest: scalable mechanisms for shared, dynamic access; interoperability in grid products; grid-enabling applications; grids for education; broadening participation; support and management of large-scale grid operations Southeastern Universities Research Association

Jim Jokl, University of Virginia SURAgrid auth. N/auth. Z

SURAgrid Authentication • Goal – Develop a scalable inter-campus solution • Preferred mechanisms – Leverage campus middleware activities • Researchers should not need to operate their own authentication systems • Use local campus credentials inter-institutionally – Rely on existing higher education interinstitutional authentication efforts Southeastern Universities Research Association

Inter-campus Globus Authentication • Globus uses PKI credentials for authentication • Leverage native campus PKI credentials on SURAgrid – Users do all of their work using local campus PKI credentials • How do we create the inter-campus trust fabric? • Standard inter-campus PKI trust mechanisms include – Operating a single Grid CA or trusting other campus CAs – Cross-certification and Bridge PKIs • How well does Globus operate in a bridged PKI? – Open. SSL PKI in Globus is not bridge-aware – Known to work from NMI Testbed project • Decision: intercampus trust based on a PKI Bridge – Leverage EDUCAUSE Higher Education Bridge CA (HEBCA) when ready Southeastern Universities Research Association

Background: Cross-certification • Top section – Traditional hierarchical validation example • Bottom section – Validation using cross certification example – UVA signed a certificate request from the UAB CA – UAB signed a certificate request from the UVA CA – This pair of cross certificates enables each school to trust certs from the other using only their own root as a trust anchor – An n 2 problem Southeastern Universities Research Association I: UAB S: UAB I: UVA S: UVA I: UAB S: User-2 I: UVA S: User-1 I: UAB S: UAB I: UAB S: UVA I: UVA S: User-1 I: UVA S: UVA Cross Certs I: UVA S: UAB I: UAB S: User-2

Background: Bridged PKI Bridge CA Cross-certificate pairs Campus A Mid-A Campus B Campus n Mid-B User A 1 User B 1 User A 2 User B 1 Southeastern Universities Research Association • Used to enable trust between multiple hierarchical CAs • Generally more infrastructure than just the cross-certificate pairs • Typically involves strong policy & practices • Solves the n 2 problem • For SURAgrid we preload cross-certs

SURAgrid Authentication Schematic Campus F Grid E’s PKI SURAgrid Bridge CA F’s PKI Campus E Grid Cross-cert pairs D’s PKI A’s PKI Campus A Grid B’s PKI Campus B Grid Southeastern Universities Research Association C’s PKI Campus C Grid Campus D Grid

SURAgrid Authentication Status • SURAgrid Bridge CA – Off-line system – Used Linux and Open. SSL to build bridge • Cross-certifications with the bridge complete or in progress for 8 SURAgrid sites • Several more planned in near future • SURAgrid Bridge Web Site • Interesting PKI issues discussed in paper Southeastern Universities Research Association

Higher Education Bridge Certification Authority (HEBCA) • A project of EDUCAUSE – Implement a bridge for higher education based on the Federal PKI bridge model – Support both campus PKIs and sector hierarchical PKIs – Cross-certify with the Federal bridge (and others as appropriate) • Should form an excellent permanent trust fabric for a bridge-based Grid Southeastern Universities Research Association

Model SURAgrid Authentication Campus F Grid E’s PKI HEBCA F’s PKI Campus E Grid Cross-cert pairs D’s PKI A’s PKI Campus A Grid B’s PKI Campus B Grid Southeastern Universities Research Association C’s PKI Campus C Grid Campus D Grid

Bridge to Bridge Context • A federal view on how the inter-bridge environment is likely to develop SAFE • FBCA – Federal Bridge • SAFE – Pharmaceutical • HEBCA – Higher Ed • Commercial - aerospace Commercial and defense • Grid extensible across PKI bridges? Southeastern Universities Research Association FBCA HEBCA Others

SURAgrid Auth. N/Auth. Z Status • Bridge CA and cross-certification process – Forms the basic Auth. N infrastructure – Builds a trust fabric that enables each site to trust the certificates issued by the other sites • The grid-mapfile – Controls the basic (binary) Auth. Z process – Sites add certificate Subject DNs from remote sites to their grid-mapfile based on email from SURAgrid sites Southeastern Universities Research Association

SURAgrid Auth. Z Development • Grid-mapfile automation – Sites that use a recent version of Globus will use a LDAP callout that replaces the grid-mapfile – For other sites there will be some software that provides and updates a grid-mapfile for their gatekeeper Southeastern Universities Research Association

SURAgrid Auth. Z Development • LDAP Auth. Z Directory – Web interface for site administrators to add and remove their SURAgrid users – Directory holds and coordinates • • • Certificate Subject DN Unix login name (prefixed by school initials) Allocated Unix UID (high numbers) Some Unix GIDs? (high numbers) Perhaps SSH public key, perhaps gsissh only Other (tbd) – Reliability • Replication to sites that want local copies Southeastern Universities Research Association

SURAgrid Auth. Z Development • Sites contributing non-dedicated resources to SURAgrid greatly complicate the equation • We will provide a code template for editing grid-mapfiles to manage SURAgrid users • Publish our LDAP schema – Sites may query LDAP to implement their own SURAgrid Auth. Z/Auth. N interface Southeastern Universities Research Association

Likely SURAgrid Auth. Z Directions and Research – User directory or directory access • • • Group management Person attributes VO names Store per-person, per-group allocations Integrate with accounting Local and remote stop-lists – Resource directory • Hold resource usage policies • Time of day, classifications, etc – Mapping users to resources within resource policy constraints – We’ll learn a lot more about what is actually required as we work with the early user groups Southeastern Universities Research Association

Ashok Adiga, Texas Advanced Computing Ctr. SURAgrid portal

Configuring SURAgrid nodes • SURAgrid supports dedicated & non-dedicated nodes – Common software stack being defined for dedicated nodes – Non-dedicated nodes support basic grid services • Job & data management • Authentication • Resource monitoring • Simple process to add resources to the grid – Install Globus (GRAM & gridftp) – Cross sign CA certificates with Bridge CA – Install GPIR perl provider scripts on resource and add resource description to User Portal. Southeastern Universities Research Association

Motivation for User Portals • Make joining the SURAgrid easier for users • Single place for users to find user information and get user support • Certain information can be displayed better in a web page than in a command shell • Allow novice users to start using grid resources securely through a Web interface • Increase productivity of SURAgrid researchers – do more science! Southeastern Universities Research Association

What is a Grid User Portal? • In general, a portal is a gateway to a set of distributed services accessible from a Web browser • Provides – – – Aggregation of different services as a set of Web pages Single URL Single Sign-On Personalization Customization Southeastern Universities Research Association

Characteristics of a User Portal • A User Portal can include the following services: – Documentation Services – Notification Services – User Support Services • • Allocations Accounts Training Consulting Southeastern Universities Research Association

User Portal Characteristics (cont’d)l – Collaborative Services • Calendar • Chat • Resource sharing – Information Services • Resource • Grid-wide – Interactive Services • Manage Jobs & Data • Doesn’t replace the command shell but provides a simpler, alternative interface Southeastern Universities Research Association

Service Aggregation Documentation User Guides Interactive Job Submission File Transfer Notification User News GSI User Support Consulting HTTP/SSL/SOAP User Portal HTTP/SSL Client Browser Southeastern Universities Research Association Collaborative Calendar Chat Information Resource Grid

Portal build using Grid. Port 4 • Developed at TACC and San Diego State University • Includes: – Portal framework-independent “portlets” • • Expose backend services as customizable web interfaces JSR 168 Portlet Standard Install into Grid. Sphere by default Small changes would allow portlets to run in any JSR-168 compliant portal framework – u. Portal, Web. Sphere, Jetspeed, etc. – Portal services • Services that run in the same web container as portlets • Provide portlet cohesion and portal framework independent support for portals • Interface to grid technologies – GRAM, Grid. FTP, My. Proxy, WSRF, science applications • Notable Technologies – Spring framework (portal services); Hibernate O/R mapping (persistence); Tomcat Southeastern Universities Research Association

SURAgrid Portal • • Single sign-on to access all grid resources Documentation tab has details on: – Adding resources to the grid – Setting up user ids and uploading proxy certificates Southeastern Universities Research Association

Information Services • Resource – State information about individual resources • Queue, Status, Load, OS Version, Uptime, Software, etc. . • Grid – Grid-wide network performance – Aggregated capability • GPIR information Web Service – Collects and provides information above Southeastern Universities Research Association

Resource Monitoring http: //gridportal. sura. org/gridsphere? cid=resource-monitor

Interactive Services • Security – Hidden from the user as much as possible • File Management – Upload – Download – Transfer between resources • Job Submission to a single resource • Job Submission to a grid meta-scheduler (future) • Composite Job Sequencing (future) Southeastern Universities Research Association

Proxy Management • Upload proxy certificates to My. Proxy server • Portal provides support for selecting a proxy certificate to be used in a user session Southeastern Universities Research Association

File Management • List directories, Move files between grid resources, Upload/download files from local machine Southeastern Universities Research Association

Job Management • • • Submit Jobs for execution on remote grid resources Check status of submitted jobs Cancel and delete jobs. Southeastern Universities Research Association

Future Directions • User Portal currently offers basic user, informational and interactive services. – Build on other services such as user support • Need to expand services as grid grows – Resource broker to automatically select resource for job execution – Workflow support for automation and better utilization of grid resources – Reliable file transfer services • Build customized application portlets Southeastern Universities Research Association

Art Vandenberg, Georgia State University Applications on SURAgrid

SURAgrid Applications • Need applications to inform and drive development • Want to be of immediate service to real applications • Believe in grids as infrastructure – but not “if you build it they will come”… • Identifying & Fostering Applications Southeastern Universities Research Association

Proposed Application Process • Continuing survey of applications – Catalog of Grid Applications; similar agency and partner databases; survey of SURA membership • Identify target applications – Region significance, multi-institutional, intersection other e-Science – Illustrating grid benefits • Test it – Globus, auth. N-Z/Bridge. CA, compilers, portal… and more • Implementation options – 1) Immediate deployment – 2) Demonstration deployment opportunities – 3) Combined with proposal development Southeastern Universities Research Association

Catalog of Grid Applications • • http: //art 11. gsu. edu: 8080/grid_cat/index 5. jsp Researchers of grid, grid potential applications Initial intent just to see who's doing what Potentially larger resource (collaboration, regional perspective, overall trends) • 21 sites, 530+ researchers • Current focus: – Automated maintenance – Improved search, browse Southeastern Universities Research Association

Identify an Applications Base • Build from application activities already underway in SURAgrid • Integrate with regional strategy (SURA HPC-Grid Initiatives Planning Group) • Apply additional resources – Seeking additional collaboration, external funding • Achieve critical mass within a critical timeframe • Seek FUNDING Southeastern Universities Research Association

SURAgrid Applications • • • Multiple Genome Alignment (GSU, UAB, UVA) Task Farming (LSU) Muon Detector Grid (GSU) BLAST (UAB) ENDYNE (TTU) SCOOP/ADCIRC (UNC, RENCI, MCNC, SCOOP partners, SURAgrid partners) • … Potential applications… Southeastern Universities Research Association

Multiple Genome Alignment. GSU, UAB, U. Virginia, U. Southern Ca. Seq 1 -2 Seq 3 -4 Seq 5 -6 Sequences 1 -6 Sequences 7 -12 • Demoed March 2005 SURA IT Comm (used Bridge. CA) • SMP cluster UAB grid SURAgrid • Iteratively advance understanding (algorithm, UAB grid, Bridge CA, multiple clusters…) • USC baseline testing Mar-Jun 2005

Task Farming. Louisiana State U. • • • Demo Nov SC 2004; Mar 2005 - SURA IT Comm (Bridge. CA) Pluggable components to use different technologies Application independent = no need to recompile Grid enabled, supports task scheduling HTTP interface: monitor progress, steer individual TFM

Muon Detector Grid. Georgia State University • Demoed Internet 2 Fall 2004 • Detector grid: collectors, compute & data grid • Grid infrastructure addresses digital divide

BLAST- U. Alabama at Birmingham • Nearing SURAgrid deployment • Database search application for protein and nucleotide sequences • Globus: job staging, submission, retrieval • ncbi. BLAST for computation • Pubcookie initial login, myproxy grid login • Simplified web interface • Sequence database pre-staged on nodes

ENDYNE- Texas Tech Left: Simulation of the H+* + C 2 H 2 reaction, CS END grid Right: CS wave packets trajectory on X-Z plane predicting reaction • • Run on SURAgrid, September 2005 Electron Nuclear Dynamics simulations Trajectory calculation in quantum phase space Using grid enables real-time solutions

SCOOP/ADCIRC- UNC, RENCI, MCNC, SCOOP Partners, SURAgrid Participants • • • SURA program to create infrastructure for distributed Integrated Ocean Observing System (IOOS) in the southeast – Shared means for acquisition of observational data – Enables modeling, analysis and delivery of real-time data SCOOP will serve as a model for national effort http: //www 1. sura. org/3000/3300_Coastal. html • SCOOP/ADCIRC: forecast storm surge • • • 1: resource selection (query MDS) 2: build package (application & data) 3: send package to resource (gridftp) 4: run adcirc in mpi mode (globus rsl & qsub) 5: retrieve results from resource (gridftp)

SCOOP/ADCIRC… Left: ADCIRC max water level for 72 hr forecast starting 29 Aug 2005, driven by the "usual, alwaysavailable” ETA winds. Right: ADCIRC max water level over ALL of UFL ensemble wind fields for 72 hr forecast starting 29 Aug 2005, driven by “UFL always-available” ETA winds. Images credit: Brian O. Blanton, Dept of Marine Sciences, UNC Chapel Hill

SCOOP/ADCIRC Results SURAgrid U. Kentucky (CCS-UKY, 48 CPU/230 Gflops/48 G RAM, 500 G Disk) • • • • • • -rwx------rwx------rwx------rw-------rw-------rw-rw-r-- 1 howard 1 howard 1 howard howard howard 1458444 Sep 14 13: 39 adcirc. x 12 Sep 14 13: 39 adcpost. inp 843813 Sep 14 13: 39 adcpost. x 29 Sep 14 13: 39 adcprep. inp 1150926 Sep 14 13: 39 adcprep. x 915 Sep 14 13: 39 execute_parallel_bundle. sh 3042520 Sep 14 13: 39 fort. 14 64545 Sep 14 13: 39 fort. 15 19804050 Sep 14 13: 39 fort. 22 1444457 Sep 14 16: 17 fort. 61 -rw-rw-r-- 1 howard 202457 Sep 14 16: 17 fort. 62 Results stored in fort. 61 - 64 -rw-rw-r-- 1 howard 105626297 Sep 14 16: 18 fort. 63 -rw-rw-r-- 1 howard 169753697 Sep 14 16: 19 fort. 64 -rw------- 1 howard 1257568 Sep 14 13: 39 fort. 68 -rw-rw-r-- 1 howard 1326004 Sep 14 13: 40 fort. 80 -rw------- 1 howard 3940266 Sep 14 13: 40 metis_graph. txt -rwx------ 1 howard 1802370 Sep 14 13: 39 padcirc. x -rw-rw-r-- 1 howard 403 Sep 14 13: 39 pbs_sub-howard -rw-r--r-- 1 howard 1028 Sep 14 13: 39 pbs_sub-howard. e 125698 -rw-r--r-- 1 howard 91 Sep 14 13: 39 pbs_sub-howard. o 125698 drwxrwxr -x 2 howard 4096 Sep 14 13: 41 PE 0000 drwxrwxr -x 2 howard 4096 Sep 14 13: 41 PE 0001 drwxrwxr Directories created by job -x 2 howard 4096 Sep 14 13: 41 PE 0002 drwxrwxr -x 2 howard 4096 Sep 14 13: 41 PE 0003

Potential applications… • Turbomachinery Flow Field Simulation (Dr. Alan Shih, UAB) • Computational fluid dynamics (Tulane, TACC, UAB…) Southeastern Universities Research Association

Potential apps & grants… • NSF CRI proposal pending (SURAgrid team) – Improved 2 D gel statistical analysis • Dr. Alan Shih, ME, Dr. Sreelatha Meleth, Dept. Med. , U. Alabama at Birmingham – Asynchronous iterative algorithms • Dr. Jim Browne, CS, U. Texas at Austin – Protein structure prediction • Dr. Yi Pan, CS, Dr. Robert Harrison, CS/Biol, Georgia State U. – Configurable grid testbed • Dr. Ashok Adiga, Dr. Warren Smith, Texas Advanced Computing Center – Application discovery and knowledge management • Dr. Vijay Vaishnavi, CIS/CS, Art Vandenberg, Georgia State U. Southeastern Universities Research Association

Applications drive infrastructure • • • Contributed nodes Defining software stack (evolving) Bridge CA Portal Policy, meta-scheduling CHALLENGE: Pragmatic Managed Experimental Production Southeastern Universities Research Association

Challenges • Essentially: more applications • Meeting broad objectives of SURAgrid in context of size & diversity of SURA • Busy people, multiple priorities, tight resources • “Application implementation template” • Finding convergence of interests • Collaboration is key Southeastern Universities Research Association

SURAgrid Summary • Fulfilling SURA mission to foster excellence in scientific research, strengthen capabilities, provide training opportunities • Evolving beyond regional initiative • Growing infrastructure moving to production • Identifying applications & participants • Collaborative research activities – Auth. N/Z, Portals, Applications, Metascheduling – Grid middleware services – Funding opportunities Southeastern Universities Research Association

Additional questions or comments? For more information: http: //www 1. sura. org/3000/SURAgrid. html