Скачать презентацию ATLAS Production Kaushik De University of Texas At Скачать презентацию ATLAS Production Kaushik De University of Texas At

90b9e689d168eed58c379df46021b8c6.ppt

  • Количество слайдов: 40

ATLAS Production Kaushik De University of Texas At Arlington LHC Computing Workshop, Ankara May ATLAS Production Kaushik De University of Texas At Arlington LHC Computing Workshop, Ankara May 2, 2008

Outline q Computing Grids q Tiers of ATLAS q Pan. DA Production System q Outline q Computing Grids q Tiers of ATLAS q Pan. DA Production System q MC Production Statistics q Distributed Data Management q Operations Shifts q Thanks to T. Maeno (BNL), R. Rocha (CERN), J. Shank (BU) for some of the slides presented here Kaushik De May 2, 2008 2

EGEE Grid + Nordugrid - NDGF (Nordic countries) Kaushik De May 2, 2008 3 EGEE Grid + Nordugrid - NDGF (Nordic countries) Kaushik De May 2, 2008 3

OSG – US Grid Kaushik De May 2, 2008 4 OSG – US Grid Kaushik De May 2, 2008 4

Tiers of ATLAS q 10 Tier 1 centers q Canada, France, Germany, Italy, Netherlands, Tiers of ATLAS q 10 Tier 1 centers q Canada, France, Germany, Italy, Netherlands, Nordic Countries, Spain, Taipei, UK, USA q ~35 Tier 2 centers q Australia, Austria, Canada, China, Czech R. , France, Germany, Israel, Italy, Japan, Poland, Portugal, Romania, Russian Fed. , Slovenia, Spain, Switzerland, Taipei, UK, USA q ? Tier 3 centers Kaushik De May 2, 2008 5

Tiered Example – US Cloud BNL T 1 MW T 2 UC, IU IU Tiered Example – US Cloud BNL T 1 MW T 2 UC, IU IU OSG NE T 2 BU, HU SW T 2 UTA, OU UTA DPCC SLAC T 2 OU Oscer GL T 2 UM, MSU Tier 3’s Wisconsin UC Teraport UTD LTU SMU Kaushik De May 2, 2008 6

Data Flow in ATLAS Kaushik De May 2, 2008 7 Data Flow in ATLAS Kaushik De May 2, 2008 7

Storage Estimate Kaushik De May 2, 2008 8 Storage Estimate Kaushik De May 2, 2008 8

Production System Overview Panda Prod. DB submit send jobs DQ 2 approved register files Production System Overview Panda Prod. DB submit send jobs DQ 2 approved register files Clouds Tasks requested by Physics Working Group Kaushik De CA, DE, ES, FR, IT, NL, TW, UK, US (NDGF coming) May 2, 2008 9

Panda q PANDA = Production ANd Distributed Analysis system q Designed for analysis as Panda q PANDA = Production ANd Distributed Analysis system q Designed for analysis as well as production q Project started Aug 2005, prototype Sep 2005, production Dec 2005 q Works both with OSG and EGEE middleware q A single task queue and pilots q Apache-based Central Server q Pilots retrieve jobs from the server as soon as CPU is available low latency q Highly automated, has an integrated monitoring system, and requires low operation manpower q Integrated with ATLAS Distributed Data Management (DDM) system q Not exclusively ATLAS: has its first OSG user CHARMM Kaushik De May 2, 2008 10

Panda Cloud Tier 1 job input files storage output files job Tier 2 s Panda Cloud Tier 1 job input files storage output files job Tier 2 s input files output files Kaushik De storage May 2, 2008 11

Panda/Bamboo System Overview DQ 2 Panda server Prod. DB bamboo job LRC/LFC send log Panda/Bamboo System Overview DQ 2 Panda server Prod. DB bamboo job LRC/LFC send log pull https pilot job submit site A pilot End-user Kaushik De logger site B job https http submit condor-g Autopilot Worker Nodes May 2, 2008 12

Panda server clients https DQ 2 Panda server LRC/LFC Panda. DB Apache + gridsite Panda server clients https DQ 2 Panda server LRC/LFC Panda. DB Apache + gridsite logger pilot q Central queue for all kinds of jobs q Assign jobs to sites (brokerage) q Setup input/output datasets q q Create them when jobs are submitted Add files to output datasets when jobs are finished q Dispatch jobs Kaushik De May 2, 2008 13

Bamboo prod. DB Bamboo cx_Oracle Panda server https Apache + gridsite cron https q Bamboo prod. DB Bamboo cx_Oracle Panda server https Apache + gridsite cron https q Get jobs from prod. DB to submit them to Panda q Update job status in prod. DB q Assign tasks to clouds dynamically q Kill TOBEABORTED jobs A cron triggers the above procedures every 10 min Kaushik De May 2, 2008 14

Client-Server Communication q HTTP/S-based communication (curl+grid proxy+python) q GSI authentication via mod_gridsite q Most Client-Server Communication q HTTP/S-based communication (curl+grid proxy+python) q GSI authentication via mod_gridsite q Most of communications are asynchronous q Panda server runs python threads as soon as it receives HTTP requests, and then sends responses back immediately. Threads do heavy procedures (e. g. , DB access) in background better throughput q Several are synchronous Panda Client Python obj serialize (c. Pickle) Python obj deserialize (c. Pickle) Kaushik De Request HTTPS User. IF mod_python (x-www-form -urlencode) mod_deflate Response Python obj May 2, 2008 15

Data Transfer DQ 2 Panda submitter q Rely on ATLAS DDM submit Job q Data Transfer DQ 2 Panda submitter q Rely on ATLAS DDM submit Job q Panda sends requests to DDM q DDM moves files and sends notifications back to Panda subscribe T 2 for disp dataset q Panda and DDM work data transfer asynchronously callback q Dispatch input files to T 2 s and aggregate output files to T 1 pilot q Jobs get ‘activated’ when all input files get Job are copied, and pilots pick them up run job q Pilots don’t have to wait for data finish Job arrival on WNs q Data-transfer and Job-execution can run in parallel add files to dest datasets data transfer Kaushik De callback May 2, 2008 16

Pilot and Autopilot (1/2) q Autopilot is a scheduler to submit pilots to sites Pilot and Autopilot (1/2) q Autopilot is a scheduler to submit pilots to sites via condor-g/glidein Pilot Gatekeeper Job Panda server q Pilots are scheduled to the site batch system and pull jobs as soon as CPUs become available Panda server Job Pilot q Pilot submission and Job submission are different Job = payload for pilot Kaushik De May 2, 2008 17

Pilot and Autopilot (2/2) q How pilot works q Sends the several parameters to Pilot and Autopilot (2/2) q How pilot works q Sends the several parameters to Panda server for job matching (HTTP request) § § § q CPU speed Available memory size on the WN List of available ATLAS releases at the site Retrieves an `activated’ job (HTTP response of the above request) § activated running Runs the job immediately because all input files should be already available at the site q Sends heartbeat every 30 min q Copy output files to local SE and register them to Local Replica Catalogue q Kaushik De May 2, 2008 18

Production vs Analysis q Run on same infrastructures Same software, monitoring system and facilities Production vs Analysis q Run on same infrastructures Same software, monitoring system and facilities q No duplicated manpower for maintenance q q Separate computing resources Different queues different CPU clusters q Production and analysis don’t have to compete with each other q q Different policies for data transfers Analysis jobs don’t trigger data-transfer § Jobs go to sites which hold the input files q For production, input files are dispatched to T 2 s and output files are aggregated to T 1 via DDM asynchronously § Controlled traffics q Kaushik De May 2, 2008 19

Current Pan. DA production – Past Week Kaushik De May 2, 2008 20 Current Pan. DA production – Past Week Kaushik De May 2, 2008 20

Pan. DA production – Past Month Kaushik De May 2, 2008 21 Pan. DA production – Past Month Kaushik De May 2, 2008 21

MC Production 2006 -07 Kaushik De May 2, 2008 22 MC Production 2006 -07 Kaushik De May 2, 2008 22

ATLAS Data Management Software - Don Quijote q The second generation of the ATLAS ATLAS Data Management Software - Don Quijote q The second generation of the ATLAS DDM system (DQ 2) q q DQ 2 developers M. Branco, D. Cameron, T. Maeno, P. Salgado, T. Wenaus, … Initial idea and architecture were proposed by M. Branco and T. Wenaus q DQ 2 is built on top of Grid data transfer tools q Moved to dataset based approach § Datasets : an aggregation of files plus associated DDM metadata Datasets is a unit of storage and replication § Automatic data transfer mechanisms using distributed site services § § Subscription system § Notification system q Current Kaushik De version 1. 0 May 2, 2008 23

DDM components DDM end-user tools (T. Maeno, BNL) (dq 2_ls, dq 2_get, dq 2_cr) DDM components DDM end-user tools (T. Maeno, BNL) (dq 2_ls, dq 2_get, dq 2_cr) DQ 2 dataset catalog Local File Catalogs Kaushik De File Transfer Service DQ 2 Subscription Agents DQ 2 “Queued Transfers” May 2, 2008 24

DDM Operations Mode üAll Tier-1 s have predefined (software) channel with CERN and with DDM Operations Mode üAll Tier-1 s have predefined (software) channel with CERN and with each other. üTier-2 s are associated with one Tier-1 and form the cloud üTier-2 s have predefined channel with the parent Tier-1 only. US ATLAS DDM operations team : BNL GLT 2 MWT 2 NET 2 SWT 2 WISC H. Ito, W. Deng, A. Klimentov, P. Nevski S. Mc. Kee (MU) C. Waldman (UC) S. Youssef (BU) P. Mc. Guigan (UTA) Y. Wei (SLAC) X. Neng (WISC) NG RAL CNAF SARA LYON Cloud lpc Tokyo TWT 2 CERN T 3 grif PIC ASGC LYON Beijing ASGC Cloud Melbourne TRIUMF FZK lapp Romania BNL T 1 -T 1 and T 1 -T 2 associations according to GP . ATLAS Tiers associations T 1 T 2 Kaushik De T 3 BNL Cloud NET 2 GLT 2 wisc MWT 2 VO box, dedicated computer to run DDM services SWT 2 May 2, 2008 25

Activities. Data Replication q Centralized and automatic (according to computing model) q Simulated data Activities. Data Replication q Centralized and automatic (according to computing model) q Simulated data § AOD/NTUP/TAG (current data volume ~1. 5 TB/week) § § § Validation samples § q Replicated to BNL for SW validation purposes Critical Data replication § Database releases § § replicated to BNL from CERN and then from BNL to US ATLAS T 2 s. Data volume is relatively small (~100 MB) Conditions data § q BNL has a complete dataset replicas US Tier-2 s are defined what fraction of data they will keep – From 30% to 100%. Replicated to BNL from CERN Cosmic data § § BNL requested 100% of cosmic data. Data replicated from CERN to BNL and to US Tier-2 s q Data replication for individual groups, Universities, physicists q Kaushik De Dedicated Web interface is set up May 2, 2008 26

Data Replication to Tier 2’s Kaushik De May 2, 2008 27 Data Replication to Tier 2’s Kaushik De May 2, 2008 27

You’ll never walk alone Weekly Throughput 2. 1 GB/s out of CERN From Simone You’ll never walk alone Weekly Throughput 2. 1 GB/s out of CERN From Simone Campana Kaushik De May 2, 2008 28

Subscriptions q Subscription q Request for the full replication of a dataset (or dataset Subscriptions q Subscription q Request for the full replication of a dataset (or dataset version) at a given site q Requests are collected by the centralized subscription catalog q And are then served by a site of agents – the site services q Subscription on a dataset version q One time only replication q Subscription on a dataset q Replication triggered on every new version detected q Subscription closed when dataset is frozen Kaushik De May 2, 2008 29

Site Services q Agent based framework q Goal: Satisfy subscriptions q Each agent serves Site Services q Agent based framework q Goal: Satisfy subscriptions q Each agent serves a specific part of a request q q q q q Fetcher: fetches up new subscription from the subscription catalog Subscription Resolver: checks if subscription is still active, new dataset versions, new files to transfer, … Splitter: Create smaller chunks from the initial requests, identifies files requiring transfer Replica Resolver: Selects a valid replica to use as source Partitioner: Creates chunks of files to be submitted as a single request to the FTS Submitter/Pending. Handler: Submit/manage the FTS requests Verifier: Check validity of file at destination Replica Register: Registers new replica in the local replica catalog … Kaushik De May 2, 2008 30

Typical deployment q Deployment at Tier 0 similar to Tier 1 s q LFC Typical deployment q Deployment at Tier 0 similar to Tier 1 s q LFC and FTS services at Tier 1 s q SRM services at every site, including Tier 2 s Central Catalogs Site Services Kaushik De May 2, 2008 31

Interaction with the grid middleware q File Transfer Services (FTS) q One deployed per Interaction with the grid middleware q File Transfer Services (FTS) q One deployed per Tier 0 / Tier 1 (matches typical site services deployment) q Triggers the third party transfer by contacting the SRMs, needs to be constantly monitored q LCG File Catalog (LFC) q One deployed per Tier 0 / Tier 1 (matches typical site services deployment) q Keeps track of local file replicas at a site q Currently used as main source of replica information by the site services q Storage Resource Manager (SRM) q Once pre-staging comes into the picture Kaushik De May 2, 2008 32

DDM - Current Issues and Plans q Dataset deletion q q q Non trivial, DDM - Current Issues and Plans q Dataset deletion q q q Non trivial, although critical First implementation using a central request repository Being integrated into the site services q Dataset consistency q q q Between storage and local replica catalogs Between local replica catalogs and the central catalogs Lot of effort put into this recently – tracker, consistency service q Prestaging of data q q Currently done just before file movement Introduces high latency when file is on tape q Messaging q More asynchronous flow (less polling) Kaushik De May 2, 2008 33

ADC Operations Shifts q ATLAS Distributed Computing Operations Shifts (ADCo. S) q World-wide shifts ADC Operations Shifts q ATLAS Distributed Computing Operations Shifts (ADCo. S) q World-wide shifts q To monitor all ATLAS distributed computing resources q To provide Quality of Service (Qo. S) for all data processing q Shifters receive official ATLAS service credit (OTSMo. U) q Additional information q http: //indico. cern. ch/conference. Display. py? conf. Id=22132 q http: //indico. cern. ch/conference. Display. py? conf. Id=26338 Kaushik De May 2, 2008 34

Typical Shift Plan q Browse recent shift history q Check performance of all sites Typical Shift Plan q Browse recent shift history q Check performance of all sites q File tickets for new issues q Continue interactions about old issues q Check status of current tasks q Check all central processing tasks q Monitor analysis flow (not individual tasks) q Overall data movement q File software (validation) bug reports q Check Panda, DDM health q Maintain elog of shift activities Kaushik De May 2, 2008 35

Shift Structure q Shifter on call q Two consecutive days q Monitor – escalate Shift Structure q Shifter on call q Two consecutive days q Monitor – escalate – follow up q Basic manual interventions (site – on/off) q Expert on call q One week duration q Global monitoring q Advice shifter on call q Major interventions (service - on/off) q Interact with other ADC operations teams q Provide feed-back to ADC development teams q Tier 1 expert on call q Very important (ex. Rod Walker, Graeme Stewart, Eric Lancon…) Kaushik De May 2, 2008 36

Shift Structure Schematic by Xavier Espinal Kaushik De May 2, 2008 37 Shift Structure Schematic by Xavier Espinal Kaushik De May 2, 2008 37

ADC Inter-relations Production Alex Read Central Services Birger Koblitz Operations Support Pavel Nevski Tier ADC Inter-relations Production Alex Read Central Services Birger Koblitz Operations Support Pavel Nevski Tier 1 / Tier 2 Simone Campana Kaushik De Tier 0 Armin Nairz DDM Stephane Jezequel ADCo. S Distributed Analysis Dietrich Liko May 2, 2008 38

Kaushik De May 2, 2008 39 Kaushik De May 2, 2008 39

Kaushik De May 2, 2008 40 Kaushik De May 2, 2008 40