Enabling Grids for E-scienc. E INFN-GRID Release Cristina Aiftimiei (cristina. aiftimiei@pd. infn. it) Simone Dalla Fina (simone. dallafina@pd. infn. it) - on behalf of INFNGrid Release Team - EGEE Admin Tutorial, Bologna, June 6 -8, 2007 V INFNGrid Workshop Padova, 18 – 20 December 2006
Outline Enabling Grids for E-scienc. E • • • • Why do we need it Customizations Short history g. Lite Release INFN-GRID Release Repositories Packages lists and metapackages Quattor ig-yaim Documentation Support The release-testbed People Conclusions V INFNGrid Workshop Padova, 18 – 20 December 2006
What does it mean release? Enabling Grids for E-scienc. E • HUGE collection of packages (~ O(1000)) – Middleware (services, client tools, ) – External dependencies (libraries, . . . ) • The packages are mainly developed by EGEE JRA 1 but also by other projects (VDT, Globus, . . . ) • Organized in profiles (profile = “node type” = “grid element”) – Each profile is (normally) installed into one node and is composed by one or more services: Examples of profiles: Computing. Element (CE), Workload Management System (WMS) Examples of services running on a WMS: glitenetworkserver, glite-logging-and-bookeeping, etc. . . – Each service has one or more daemons V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID Release - why? Enabling Grids for E-scienc. E • The production infrastructure is used by other projects/experiments – Babar, Virgo, CDF, ARGO, Zeus, . . . • Additional configuration for the middleware is defined once, at ROC level, to reduce misconfiguration risks: – More VOs: VO servers, poolaccounts, add VOMS certificates – MPI (requested by non-HEP sciences), additional Grid. ICE config (monitor Wns), . . . • Deploy additional middleware in a non intrusive way: – Since Nov. 2004 VOMS, now in g. Lite; DGAS (Data. Grid Accounting System); Network. Monitor (monitor network connection metrics) • Of course 100% compatibility is mandatory V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID customizations Enabling Grids for E-scienc. E • Additional VOs (~20) • Grid. ICE on WN • Preconfigured support for MPI – WN without home shared but with ssh hostbased authentication • DGAS: accounting – New profile (HLR server) + additional packages on CE and WN • Grid. ICE: monitoring – new versions + automatic configuration • Quattor (collaboration with CNAF-T 1) • Dynamic Information-Providers for LSF – corrected configuration, new vomaxjobs • NTP • . . . V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID customizations (II) Enabling Grids for E-scienc. E – Work-in-progress: cream. CE patched My. Proxy (long-live proxy delegation with voms extensions) transparent support to MPICH and MPICH-2 AMGA Web Interface GSAF (Grid Storage Access Framework) Secure Storage System g. Lite for Windows with torque/maui support (Roberto Barbera's group middleware developments) V INFNGrid Workshop Padova, 18 – 20 December 2006
Short History Enabling Grids for E-scienc. E LCG EGEE II LCG 1. 0 2003 LCG 2. 0 2004 2005 1. 0 2. 0 g. Lite 3. 0 2006 2007 2008 3. 0 INFN-GRID V INFNGrid Workshop Padova, 18 – 20 December 2006
The g. Lite release Enabling Grids for E-scienc. E • Integration, testing and certification are performed by SA 3 in order to produce the release • Currently there are two types of releases: – “Big”: changes on almost all profiles (general upgrade) – “Small”: upgrades targeted only on a few profiles (bugfixes, ad-hoc improvements) • g. Lite deployment: Nicholas Thackray • Yaim development/deployment: Robert Harakaly V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID: release structure Enabling Grids for E-scienc. E • 3 areas: – Middleware installation: packages and metapackages – Middleware configuration: yaim – Documentation: installation, upgrade and release notes V INFNGrid Workshop Padova, 18 – 20 December 2006
Repositories Enabling Grids for E-scienc. E • The packages are distributed in apt repositories available via HTTP • For each release, we create an apt repository, with our customizations – http: //grid-it. cnaf. infn. it/apt/ig_sl 3 -i 386 • It is complementary to the g. Lite repositories collecting different types of packages: – Middleware http: //glitesoft. cern. ch/EGEE/g. Lite/APT/R 3. 0/rhel 30/ – Security http: //linuxsoft. cern. ch/LCG-CAs/current/ V INFNGrid Workshop Padova, 18 – 20 December 2006
Packages lists and metapackages Enabling Grids for E-scienc. E • Installation of the middleware is based on metapackages • Our metapackages are supersets of the g. Lite ones: – INFNGRID metapackage = g. Lite metapackage + INFNGRID additional rpms • INFN-GRID metapackages have the ig_ prefix (e. g. : glite-UI => ig_UI) V INFNGrid Workshop Padova, 18 – 20 December 2006
Metapackages management process Enabling Grids for E-scienc. E • 1: starting from g. Lite lists: http: //glite. web. cern. ch/glite/packages/R 3. 0/deployment update INFN-GRID lists (mantained in SVN repo) • 2: once the lists are ok, generate a first version of INFN-GRID metapackages to test them • 3: install and/or upgrade the metapackages on the release testbed • 4: if there are errors, correct and goto step 2: • 5: publish the new metapackages on the official repositories so they are available for everybody • 6: send the “official” announcement for sites to upgrade V INFNGrid Workshop Padova, 18 – 20 December 2006
ig-yaim Enabling Grids for E-scienc. E • YAIM (Yet Another Installation Manager) - Semiautomatic tool for installation and configuration of the middleware • glite-yaim is the “standard” yaim to configure g. Lite • We create a new package (ig-yaim) that provides additional scripts and configuration files to add/override features on top of standard glite-yaim – Additional functions or functions that override existing ones. Both are stored in functions/local instead of functions/ – e. g to configure NTP, AFS, LCMAPS gridmapfile/groupmapfile, . . – More poolaccounts => ig-users. def instead of users. def – More configuration parameters => ig-site-info. def instead of siteinfo. def • Both packages are needed!! V INFNGrid Workshop Padova, 18 – 20 December 2006
ig-yaim - essentials Enabling Grids for E-scienc. E • YAIM is not more than, just a set of bash scripts and function. YAIM is distributed in rpm form, it usually resides in /opt/glite/yaim. • The configuration variables: – The configuration is stored in a directory structure which will be extended in the near future. Currently the following files are used: site-info. def, groups. conf, users. conf, and the vo. d directory. • IMPORTANT: The configuration files which are coming with the YAIM rpm are just examples! Please review them and edit your own! V INFNGrid Workshop Padova, 18 – 20 December 2006
ig-yaim - essentials Enabling Grids for E-scienc. E #. /ig_yaim --help Usage: . /ig_yaim <action> <parameters> Actions: -i | --install : Install one or several meta package. Compulsory parameters: -s, -m -c | --configure : Configure already installed services. Compulsory parameters: -s, -t -r | --runfunction : Execute a configuration function. Compulsory parameters: -s, -f Optional parameters : -n -h | --help : This help Specify only one action at a time ! Parameters: -s | --siteinfo: : Location of the site-info. def file -m | --metapackage : Name of the metapackage(s) to install -n | --nodetype : Name of the node type(s) to configure -f | --function : Name of the functions(s) to execute V INFNGrid Workshop Padova, 18 – 20 December 2006
ig-yaim - essentials Enabling Grids for E-scienc. E Examples: Installation: . /ig_yaim -i -s /root/siteinfo/site-info. def -m ig_SE_dpm_mysql Configuration: . /ig_yaim -c -s /root/siteinfo/site-info. def -t ig_SE_dpm_mysql Running a function: . /ig_yaim -r -s /root/siteinfo/site-info. def -n ig_SE_dpm_mysql -f config_mkgridmap • Configuring multiple node type or installing multiple meta-packages you have to define them repetedly on the command line, for example: . /ig_yaim -i -s /root/siteinfo/site-info. def -m ig_SE_dpm_mysql -m ig_BDII V INFNGrid Workshop Padova, 18 – 20 December 2006
Quattor Enabling Grids for E-scienc. E • quattor is a system administration toolkit providing a powerful, portable and modular toolsuite for the automated installation, configuration and management of clusters and farms, used at CERN and other Tier 1 sites (CNAF) • All quattor template lists are provided for each release – http: //grid-it. cnaf. infn. it/index. php? quattor&type=1 V INFNGrid Workshop Padova, 18 – 20 December 2006
Documentation Enabling Grids for E-scienc. E • Documentation is published at each release – Release notes, upgrade and installation guides: http: //grid-it. cnaf. infn. it/? siteinstall http: //grid-it. cnaf. infn. it/? siteupgrade http: //grid-it. cnaf. infn. it/? releasenotes • written in La. Te. X and published in html, pdf and txt • Additional informations about Upgrades, various Notes are published also in wiki pages: – https: //grid-it. cnaf. infn. it/checklist/modules/dokuwiki/doku. php? id=rel: updates – https: //gridit. cnaf. infn. it/checklist/modules/dokuwiki/doku. php? id=rel: hlr_server_installation_a nd_configuration – https: //igrelease. forge. cnaf. infn. it/doku. php? id=start V INFNGrid Workshop Padova, 18 – 20 December 2006
Support Enabling Grids for E-scienc. E • 1 st line of support for INFN Production Grid sites managers for middleware installation and configuration • Support for INFN-GRID customizations (Grid. ICE, DGAS. . . ) • Interface with developers • Problems related to g. Lite middleware routed to SA 3 experts using Savannah and GGUS – Collaboration with the glite deployment and yaim development teams – Request for new features, bugs, support, suggestions, . . . V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID Release Testbed essentials Enabling Grids for E-scienc. E • Goals: – Certificate that the INFN-GRID middleware customizations work and are deployable – Simulate different configurations coupling differente services – Run the current production release to validate any urgent update • Virtualization benefits for certification: – Quick installation/reinstallation from scratch – Good heterogeneous environment V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID Release Testbed setup Enabling Grids for E-scienc. E • Now passing to a completly virtualized testbed – Using XEN 3. 0. 3 on SLC 4/FC 5 distros • Installed 4 mini-testbeds including all INFN-GRID customized profiles – Running current production middleware release – Evaluating real resource requirements for different services (some profiles have particular load, e. g. WMS) V INFNGrid Workshop Padova, 18 – 20 December 2006
INFN-GRID Release Testbed map Enabling Grids for E-scienc. E Mini-testbed LSF (cert-06) – ig_CE_LSF – (cert-01) – ig_glite_CE_LSF– (cert-02) – ig_WN_LSF – (cert-03) – ig_WN_LSF – (cert-05) Mini-testbed Torque (cert-09) ig_CE_torque – (cert-10) ig_glite_CE_torque (cert-11) – ig_WN_torque (cert-16) – ig_WN_torque (cert-21) – Mini-testbed SE (cert-13) ig_SE_classic + ig_MON (cert-17) ig_SE_dpm (cert-18) ig_SE_dcache (cert-19) ig_SE_storm (cert-20 - not yet) Mini-testbed Services (cert-14) – ig_UI (cert-22) – ig_RB (cert-23) – ig_WMSLB (cert-24) – ig_HLR + ig_BDII (cert-25) V INFNGrid Workshop Padova, 18 – 20 December 2006
Conclusions Enabling Grids for E-scienc. E • INFN-GRID release team is dedicated to the test and integration of additional middleware and configurations • It is a stable point of reference for deployment troubleshooting • Middleware previously integrated in INFN-GRID was added to the lcg/glite distributions – (VOMS, DGAS, Gridice, . . . ) • Other projects install the INFN-GRID release – GILDA, EUMEDGrid, EUIndia. Grid V INFNGrid Workshop Padova, 18 – 20 December 2006
People Enabling Grids for E-scienc. E • https: //forge. cnaf. infn. it/projects/igrelease/ – – – Cristina Aiftimei (INFN-Padova) Sara Bertocco (INFN-Padova) Luciana Carota (INFN-CNAF) Simone Dalla Fina (INFN-Padova) Saverio Da Ronco (INFN-Padova) V INFNGrid Workshop Padova, 18 – 20 December 2006
Скачать презентацию Enabling Grids for E-scienc E INFN-GRID Release Cristina
c1d0d5137b33b80e7f95696163251bb4.ppt