Status of European Data Grid Charles Loomis CNRS LAL

Status of European Data. Grid Charles Loomis CNRS/LAL Nordu. Grid Workshop May 23, 2002 C. Loomis – Status of European Data. Grid – May 23, 2002 – 1

Introduction & Outline European Data. Grid v 3 -year EU-funded project v. Goals: — develop grid middleware — deploy onto working testbed — demonstrate grid technology with working applications v. Strong application component unique! Current Software v Machine Tour v Status Testbed v Deployed software v Present & Future Sites Near-term Developments v EDG v 1. 2 v Latest Globus Release v EDG License Longer-term Developments v Testing & Support Infrastructure v Enhanced EDG Features v Interoperability Further Information C. Loomis – Status of European Data. Grid – May 23, 2002 – 2

User Interface Lightweight access to grid Services: v. Access from Laptop v. User. Interface (CLI) v. No host certificate needed. v. Globus GSI v. Some question about CRLs. vglobus-url-copy (client) Limitations v. Cannot run ftp daemon here. v. Development libraries — Broker. Info — Replica Catalog APIs — GDMP client interface C. Loomis – Status of European Data. Grid – May 23, 2002 – 3

Resource Broker Finds resources, submits & tracks jobs: Services: v. Heavyweight machine. v. Resource Broker v. Talks to RC and MDS. v. Job. Submission Service v. Acts as users’ network presence. v. Talks to proxy server. Bottleneck v. Can replicate, but enough? — Condor-G below v. Information Index v. Logging & Bookkeeping v. GSI-ftp daemon C. Loomis – Status of European Data. Grid – May 23, 2002 – 4

Computing Element Accepts & Executes Jobs: v. Gatekeeper — acts as public interface to computing resources v. Worker Node(s) Services: v. Gatekeeper v. GSI-ftp daemon v. GIIS/GRIS — provides all software needed for applications — accessible via batch system • PBS, LSF, … C. Loomis – Status of European Data. Grid – May 23, 2002 – 5

Storage Element Generic interface to storage: v. Gatekeeper — should go away v. GSIFTP v. RFIO Services: v. Gatekeeper v. GDMP v. GSI-ftp daemon v. RFIO daemon C. Loomis – Status of European Data. Grid – May 23, 2002 – 6

Replica Catalog Provides information about replicas: v. Catalog Service — accessed via RB or directly Services: v. LDAP v. GIIS/GRIS C. Loomis – Status of European Data. Grid – May 23, 2002 – 7

Authorization/Authentication System All based on GSI (PKI): Services: v. Certification Authorities v. LDAP for VO servers v. Virtual Organization Servers vvarious SW for CA’s vmkgridmap generation software C. Loomis – Status of European Data. Grid – May 23, 2002 – 8

Software Distribution & Installation Storage: v. Package repository v. CVS server Distribution v. HTTP downloads vwget with rpm lists vmost primitive link in chain Installation v. LCFG (LCFG-lite) v. Only works for RH 6. 2 C. Loomis – Status of European Data. Grid – May 23, 2002 – 9

Software on “Production” Testbed Stopped work on 1. 1 -series to focus on 1. 2. v. Deployed v 1. 1. 4+patches version not uniform v. Significant functionality missing for applications. —Replica Management —Access to mass storage. v. Difficult for middleware to support this version. Testbed works, but… v. Known stability problems: —Information Index dies regularly. —Broker needs to be restarted often. v. Support limited —Maintenance reduced to life support. —Effort for new sites limited to “available effort. ” C. Loomis – Status of European Data. Grid – May 23, 2002 – 10

Production Testbed Sites Production Sites Site Location Catania (I) CC-IN 2 P 3 Lyon (F) CERN Geneva (CH) v. Typically few to 10’s of machines. CNAF Bologna (I) v. LCFG for Install. & Config. Imperial College London (UK) MSU Moscow (Russia) NIKHEF Amsterdam (NL) Padova (I) RAL Rutherford (UK) Torino (I) v. Most have dedicated hardware. —Lyon running on main batch system. —Lyon again exception. Limitations to Expansion v. Info. systems unreliable. —manual reg. not scalable or dynamic v. How to add countries w/o CA? —OK for users (CNRS CA) —Not OK for host certificates. Croatia Taiwan United States C. Loomis – Status of European Data. Grid – May 23, 2002 – 11

EDG Release 1. 2 New Features in 1. 2 Release (a 10) v. Replica Management API —first implementation has limited API v. Access to Mass Storage Systems —authorization linked to user account mapping v. Auto-resubmission of failed jobs. —will help with stability problems (but is not a solution!) Current Problems v. GASS cache file locking problems (failed job submissions) v. Open. LDAP timeout (II hangs; complete loss of MDS information) v. FTree interfering with gatekeeper. (Causes crashes; failed submissions) C. Loomis – Status of European Data. Grid – May 23, 2002 – 12

Expected Schedule 13 14 15 16 17 ITeam at CERN May 20 1. 2 alpha 22 23 28 Refine alpha 21 29 30 GASS/MDS Prbs. 27 JJ/Ingo Tests 3 <1% error rate 4 June 11 ESRIN Demo 17 General Deployment 18 24 Test 3 Sites 31 App. Testing 5 6 12 13 Core Site Deployment 19 20 1. 3 code license info 14 21 App. Testing 10 1. 2 beta RAL/CNAF Deployment Decision 7 C. Loomis – Status of European Data. Grid – May 23, 2002 – 13

Upgrade to Latest Globus Release EDG Globus beta-21 is based on first Globus 2 beta. v. Includes some patches for security. v. Some EDG-specific patches. v(Larger changes for EDG 1. 2. ) Upgrade to current Globus 2 release depends on: v. Desire of the applications groups —Only known critical problem is with file transfers >20 min. v. Whether it contains fixes for GASS/MDS problems. v. When EDG software for release 1. 2 is deemed stable. EDG 2. 0 release in fall will be based on Globus 2! v. OGSA being evaluated, but no whole-scale move yet. v. Some new EDG software functions as “Web Service” C. Loomis – Status of European Data. Grid – May 23, 2002 – 14

Testing & Support Testing Group v. Goal: Intensive testing of releases v. Provide framework for: — unit tests — integration tests — stress tests Support Infrastructure v. Provide email-based support for both end-users and system administrators. — ITeam and other experts — New system administrator group v. Tracking & follow-up of problems. v. Provide material for objective evaluation of software for EU-review. v. Create “knowledge base” for FAQs and typical problems. v. Use tests for: v. Interact with LCG and Cross. Grid to share the support effort. — check of quality of software — verification of functionality — check configuration of new sites v. Has started with EDG 1. 2 (a 10). System in place shortly; fully functional for Testbed 2. — should have feedback for EDG 1. 2 deployment decision C. Loomis – Status of European Data. Grid – May 23, 2002 – 15

EDG Software License v. EDG software license will be in BSD family (see EDG website): v. Open. Source license. v. Developments may be put back into code base. v. Allows commercial use of code. v. Standard license for most Grid-projects —Exception: Class. Ads, Condor-G will be LGPL. v. EDG audit of external packages: v. Necessary to ensure we can apply our own license. v. Necessary to ensure that we properly attribute other groups’ work. v. Need to be especially careful with GPL code. —Ensure that core functionality consistent with license. —LCFG will likely be GPL license rather than the EDG license. C. Loomis – Status of European Data. Grid – May 23, 2002 – 16

Release Schedule Release Date 1. 1 1. 2 July 31 2. 0 v. Provide intermediate checks on progress. May 31 1. 4 v. Keep developments compatible. March 31 1. 3 Moved to iterative releases: Jan. 31 Sept. 30 v. Allow applications to evaluate functionality. v. Not all intermediate releases will be deployed! Release 2. 0 is hard deadline; others somewhat flexible. Details in “Release Plan” document on web site, highlights… C. Loomis – Status of European Data. Grid – May 23, 2002 – 17

Release 1. 2 General Data Management (WP 2) v. Emphasis on stability. v. Replica Manager (first impl. ) v. Deploy as production release. v. GDMP 3. 0 Globus Fabric Management (WP 4) v. Uses first Globus 2 beta (beta-21) v. Updated LCFG v. Plus EDG patches. v. EDG Gatekeeper (LCAS) Storage Element (WP 5) Workload Management (WP 1) v. Proxy renewal for long jobs. v. Auto-resubmission of failed jobs. v. Access to existing data in MSS. Networking (WP 7) v. Publish network data into MDS. C. Loomis – Status of European Data. Grid – May 23, 2002 – 18

Release 1. 3 General v. Autobuild all EDG packages. v. Copyright and license for code. Globus v. Update to latest Globus 2 release Grid Mon. /Info. Services (WP 3) v. R-GMA deployed in parallel with MDS Fabric Management (WP 4) v. EDG Job. Manager Storage Element (WP 5) v. RFIO with GSI Workload Management (WP 1) v. C APIs v. MPICH support. v. Prototype Grid. FTP with MSS access. Networking (WP 7) v. Network cost function. Data Management (WP 2) v. Replica Manager v. Replica Location Service (giggle) C. Loomis – Status of European Data. Grid – May 23, 2002 – 19

Release 1. 4 General Grid Mon. /Info. Services (WP 3) v. Support RH 6. 2, RH 7. 2 v. Better integration of R-GMA. v. GLUE Schema v. Unified (GLUE) schema. v. New authorization scheme. Fabric Management (WP 4) v. Kick. Start translator. Workload Management (WP 1) v. Interactive jobs. v. Job dependencies. v. Triggered file transfers. Data Management (WP 2) v. Replica Manager with Optimiser v. Spit. Fire beta release. v. Monitoring & Alarms. v. Condor supported. Storage Element (WP 5) v. Disk. Manager for disk-only SE. Testbed (WP 6) v. New authorization scheme. Networking (WP 7) v. Publication of network metrics. C. Loomis – Status of European Data. Grid – May 23, 2002 – 20

Release 2. 0 General v. Support RH 6. 2, RH 7. 2, Solaris? Grid Mon. /Info. Services (WP 3) v. R-GMA Web. Services Fabric Management (WP 4) Workload Management (WP 1) v. Job checkpointing. v. Accounting. v. Advance reservation. Data Management (WP 2) v. Full integration of components. v. HLD templates. v. Credential service (LCMAPS). Storage Element (WP 5) v. Disk. Manager access to all HSM. v. Reservation, pinning, quotas. Testbed (WP 6) v. Laptop based UI machine. Networking (WP 7) v. Network cost for all sites. C. Loomis – Status of European Data. Grid – May 23, 2002 – 21

Interoperability v. Working with Gri. Phy. N, PPDG, i. VDGL, Data. Tag, Cross. Grid, v. First concrete example is GLUE schema. v. Places for conflict: v. Information systems v. Agreed interfaces C. Loomis – Status of European Data. Grid – May 23, 2002 – 22

Further Information Interesting web sites: v. EDG: http: //www. eu-datagrid. org/ —general information about EDG project —links to all work package web sites v. WP 6: http: //marianne. in 2 p 3. fr/ —support information (contacts, bug reporting, documentation, mailing lists) —meeting agenda/minutes —links to source code in CVS; packages in package repository Bleeding-edge information: vhep-proj-grid-integration-team@cern. ch v. Warning: this is a high-volume list! C. Loomis – Status of European Data. Grid – May 23, 2002 – 23