Gestion des jobs grille CMS and Alice Artem

Gestion des jobs grille CMS and Alice Artem Trunov CMS and Alice support

CMS and Alice site roles • Tier 0 – Initial reconstruction – Archive RAW + REC from first reconstruction – Analysis, detector studies, etc • Tier 1 – – – Archive a fraction of RAW (2 nd copy) Subsequent reconstruction “Skimming” (off AOD) Archiving Sim data produced at T 2 s Serve AOD data to other T 1 and T 2 s Analysis • Tier 2 – Simulation Production – Analysis

CMS Tools • Bookkeeping tools – – • Production tools – – • Manage scheduled or personal production of MC, reconstruction, skimming Prod. Agent Transfer tools – – • Description of datasets, provenance, production information, data locatoin DBS/DLS (Data Bookkeeping Service/Data Location Service) All scheduled intersite transfers Phedex User Tools – – For end user to submit analysis jobs. CRAB UI Oracle FTS srmcp UI Web. Interface CRAB Phedex DBS/DLS Prod. Agent Oracle LFC

Bookkeeping Tools • Centrally set up database • Has different scopes (“local”, “global”), to differentiate official data from personal/group data. • Use Oracle server at CERN. • Data location service uses either Oracle or LFC. • Very complex system

Production Agent • Managing all production jobs at all sites • This is a set of daemons running on top of LCG UI by operators at their home institution. – usually a dedicated machine due to low submission rate and large CPU consumption per summision. – Operators certificates are used to submit jobs – Operators have VOMS role cmsprod • Instances of it are used by production operators – 1 for OSG + few for EGEE sites • Merging production job output to create a reasonable size files. • Registering new data in Phedex for transfer to the final destination – MC data produced at T 2 to T 1. • All prod info goes to DBS • User make MC production requests via the web interface and also can track their requests.

Phedex • • • CMS data transfer tool The only tool that is required to run at all sites – Work in progress to ease this requirement – with SRM, fully remote operation is possible. However local support is still need to debug problems. Set of site-customizable agents that perform various transfer related tasks. – – – • • • download files to site produce SURL of local files for other sites to download follow migration of files to the MSS staging of files to the MSS removing local files Uses ‘pull’ model of transfers, i. e. transfers are initiated at the destination site by Phedex running at this site. Uses Oracle at CERN to keep it’s state information Can use FTS to perform transfer, or srmcp – Or another mean, like direct gridftp, but CMS requires SRM at sites. One of the oldest and stable SW component of CMS – Secret of success: development is carried out by the CERN+site people who are/were involved in daily operations Uses someone’s personal proxy certificate

CRAB • Main CMS user tool to submit analysis jobs • Users specifies a data set to be analyzed, his application and job configuration. • CRAB: – – – locates the data with DLS splits jobs submits jobs to the Grid tracks jobs collects results • Can be configured to upload job output to some Storage Element with srm or gridftp.

“Last mile” tools • • working with ntuples, custom skims etc Not well developed! People are by now on their own. CRAB is not scaling to well down to end-user analysis – too rigid, requires maintaining bookkeeping infrastructure service etc. – Not convenient to physicists • User questionnaire on CRAB (D-CMS group) – 6 – use it • 3 – like it • 2 – report problems – 3 - don’t use it • 1 never used Grid • 1 avoids using Grid and Crab for problems • 1 need to run on unofficial data • PROOF – is of interests to CMS. There are groups setting up PROOF at their institutions – access to CMS data structures via a loadable library. – working on a plugin for LFN lookup in DBS/DLS and PFN conversion.

Other CMS tools • Software installation service – In CMS software is installed in VO area via grid jobs by software managers who have VOMS role cmssgm. – Also cmssgm run SW monitoring service – SW installation uses apt, rpm and scram. – Base releases are required to be installed at sites to run jobs – User bring only his own compiled code on top of the base release via grid sandbox. • Dashboard – Collecting monitoring information (mostly jobs) and presenting through a web interface. – There is an effort to make use of Dashboard for all CMS monitoring – It is now beeing adopted by some other experiments – Dashboard relies on various grid components to get the state of a job, and thus it is usually inaccurate by ~15% • ASAP – Automatic submission of analysis jobs using web interface. – Seems not to be used anymore. Used to give user an option to use his software release in a for of DAR (distribution archive).

Alice services - Ali. En • • “Alice Environment’ - independent implementation of GRID middleware. Central services: – Job Queue – File catalog – Authorization • Site services include: – Computing element • With LCG/g. Lite support – Storage element – xrootd protocol – Pack. Man - SW installation – File Transfer Deamon • with FTS support – Cluster. Monitor – Monal. Lisa agent • Services run at each site on a standard VO Box, optimized for low maintenance and unattended operations. – central and site services are managed by a set of experts including developers and regional responsible experts

Alice Services at a site LCG CE Ali. En Central Services Configuration, JA submit, follow Monitoring Reports get jobs Alien site services Xrootd cluster Install software VO Box Operator gsissh login $VO_ALICE_SW_DIR

Job management • Job Agent model – Alien CE (a site service) submits a job agent (‘pilot job’) to this site queue via local batch interface or LCG interface – Once such job agent starts on a WN, it contacts the central database (Job Queue, a certral service) and pick up real tasks in form of Alien JDL. • Software needed by jobs (as specified in Alien JDL) is installed via Pack. Man site service. • User input files are downloaded from remote xrootd servers to the local scratch area. • Disadvantages of agent model in Alien – More agents are submitted then there are real jobs, and many job agents are exiting with out a real task after few minutes.

User tools • Full featured user login shell – alien shell – in fact is the only official way to submit jobs in ALICE. – the same ALIEN tools are used by production and users. • Data storage – global logical alien name space for production and home dirs accessible in the shell – backend is xrootd.

PROOF • Main end-user analysis tool in ALICE • PROOF is a part of ROOT and gives a mechanism to parallelize analysis of ROOT trees and ntuples. – Requires high speed access to data, not often possible in current setups • WN and server bandwidth limitation – WN: 1 GB per rack -> 1. 5 MB/s per cpu core with (4 cores * 24 WN/rack) – Servers: usually have 1 Gb/s, but disk subsystem may perform better. – Deployment model endorsed by PROOF team: • Large dedicated clusters with locally pre-loaded data – Not preferable at the moment at the large busy centers. • Integration with Alien via own TFile. Adapter – so a user gives it’s LFN, which is automatically converted to PFN by the adapter. • So far only at CERN – CAF. • At CC – work in progress (see next slide).

PROOF at CC-IN 2 P 3 • • • PROOF agents are run on the xrootd cluster and take advantage of the following: free cpu cycles due to low overhead of xrootd zero cost solution - no new hardware involved Direct access to data on disk, not using bandwidth 1 GB node interconnect when inter-server access is required. transparent access to full data set stored at our T 1 for all experiments via xrootd-dcache link deployed on this xrootd cluster and dynamic staging management of infrastructure conveniently fit into existing xrootd practices this setup is more close to possible 2008 PROOF solution because of 1 GB node connection and large "scratch" space. this is a kind of setup that T 2 sites may also considers to deploy GSI auth Remote session HPSS IFZ XROOTD analysis pool worker PROOF Master VO BOX Local User Session

See also • Thursday afternoon session on analysis centers.