Metadata and COMA Elizabeth Gallas Oxford ATLAS Weekly

Metadata and COMA Elizabeth Gallas Oxford. ATLAS Weekly / Open EB Meeting October 18, 2011

Outline § § Overview of ATLAS Metadata Interfaces What is COMA ? Principles Tour VERY quick tour today…hope you will try yourself! § Summary and Plans Conclusions § Backup Slides: Links § Thanks to many experts, developers and also many of you who post questions / use cases to e-groups !!! June 2011 Elizabeth Gallas - COMA 2

ATLAS Metadata Application Overview § Subsystem specific metadata applications. . . driven by use cases/workflow § § § Trigger: Trigger. Tool … Web: trigconf and trigger timelines Geometry DB: Detector Description Browser Conditions DB: § § § run. Query (Run information from Conditions DB) ATLAS WEB DQ COOL Tag Browser Data Summary Reports (Luminosity, Beam) Beam Spot Summary GANGA and PAthena Panda / monitor DQ 2 Client Tag Collector – software releases. . . Dedicated Metadata Catalogs § TAGs – event level (file and relational formats) § § § TAG Catalog AMI – Dataset, processing … other metadata § § i. ELSSI and Suite of TAG Services And the AMI Suite of services COMA – Conditions and configuration June 2011 Elizabeth Gallas - COMA Connected in many ways (and with links to many above applications) … Some mentioned today in the course of “COMA” topic discussed today 3

? coma ? Many definitions: Medicine: deep sleep … a state of extreme unresponsiveness Optics: aberration inherent to certain optical designs or due to imperfection in the lens or other components… Astronomy: luminous cloud surrounding the frozen solid nucleus in the head of a comet … ATLAS: COMA is an ~ acronym for a metadata catalog and its related services: Conditions / Configuration Metadata for ATLAS June 2011 Elizabeth Gallas - COMA 4

What is COMA ? § COMA: “Conditions/Configuration Metadata for ATLAS” § § COMA Components: § § § Originally: built to support dynamic queries of TAG DB Evolved: into a standalone system with interfaces Now part of general effort to consolidate/relate ATLAS Metadata Relational Database § Contains extracted, refined, reduced, and derived information from system specific data sources (e. g. Conditions DB, …) plus information from non-database sources. Interfaces § A set of Browsers and Reports Main Documentation https: //twiki. cern. ch/twiki/bin/viewauth/Atlas/Conditions. Metadata § COMA Portal (grid certificate in browser is required) https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html June 2011 Elizabeth Gallas - COMA 5

COMA Interfaces: General Principles § Report or Browse the COMA catalogue: § Originally developed to check COMA loading and do cross checks § Evolved to current form: on request from experts and users § Logical division of information into separate but cross linked reports § Include links to more details (COMA links and External links) § § run. Query, Luminosity, AMI, Trigger, DQ … Provide reports [distinctive from, complimentary to] other systems § (Avoid duplication of effort) § Display COMA ‘derived’ information § Not available in other systems Note: All the COMA Reports use GET methods: § Inputs are not complex, user can modify URL to get report they want, put links into TWiki or other documentation https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § June 2011 Elizabeth Gallas - COMA 6

COMA Interfaces Portal https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html Tour stops today DB mtg Weds June 2011 Elizabeth Gallas - COMA 7

Introduction: ATLAS Data Periods § A Data Period is a set of ATLAS Runs grouped for a purpose § § Before 2011, Data Periods were § § Defined by Data Preparation Coordinators Used in ATLAS data processing, assessment, and selection … Each Period uniquely defined with a combination of § Project name (i. e. ‘data 10_7 Te. V’) § Period name (i. e. ‘C 1’, ‘C 2’, ‘C’, ‘All. Year’ …) Described on TWiki page § https: //twiki. cern. ch/twiki/bin/view/Atlas. Protected/Data. Periods Stored in a file based system § Edited by hand by Data Prep Coordination (experts) § Structure evolved over last year with experience This experience valuable to decide/define long term solution Early in 2011: Data Periods stored in the COMA DB § Thanks: Beate (Data. Prep Coordinator), AMI team, me. June 2011 Elizabeth Gallas - COMA 8

Menu Purpose: Show Data. Prep defined Periods giving links to reports of general info about their Runs. Choose the Period of interest: By Year E. G. all ‘ 2010’ By Project e. g. ‘data 10_7 Te. V’ By specific Period or Group Click on the project and then Period of interest Wildcards can be entered in this optional section, then click on Submit button “highlight” expanding sections https: //atlas-tagservices. cern. ch/RBR/r. BR_Period_Report. php Elizabeth Gallas - COMA 9 June 2011

Period Description Report Example for data 10_7 Te. V Header: Input criteria; Links: Contact, Documentation Links: to COMA & run. Query multi-Run Reports for chosen Period (COMA link: next slide) Members of data 11_7 Te. V. M are M 1, M 2, M 3, M 4 -/+ highlighted links: These sections expand to. June 2011 show period members Period “Status”: M 1 is ‘frozen’, M 2 -M 4 are ‘locked’ … M ‘unlocked’ (M 5 is on-going) Elizabeth Gallas - COMA 10

COMA Multi-Run Report: Purpose: show basic information for all runs matching the input criteria Run links: COMA, run. Query, and Lumi. Data. Summary Single Run report (COMA link: next slide) Links in Table column headers: Short description of column SMK links: COMA Trigger Super Master Key Report Hover any link: Indicates what will happen https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? fnt=data 10_7 Te. V&pn=Vd. M June 2011 Elizabeth Gallas - COMA 11

COMA Single Run 180776 Overview https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=180776 General Run info: § § Project, LB range, Date … Events, HLT Release … Links to COMA reports § COMA Master Key report (no time to show today) Links to external reports § HLT release, AMI, Lumi. DS, run. Query, Trigger, PSEvol Yellow sections expand: § § § June 2011 Elizabeth Gallas - COMA Data Periods Active Streams AMI tags (TAG production) Shortcut find runs w/AMI tag Prescale Evolution PS Key changes Trigger section All HLT signatures All LVL 1 items 12

COMA Single Run -/+AMITag Section Recent data 11_7 Te. V M 4 Period Run 190933 Shows: § All physics streams § § “First Pass” – first processing § § Including debug ! New columns appear for reprocessing passes AMI tag used to process data § Link: COMA multi-Run Report § § “? ”: link to AMI § § Description of AMI tag Launch i. ELSSI ! § § § June 2011 All runs processed w/this tag For this Run/Stream/AMItag Take a quick look at this run with the ELSSI TAG browser !!! See if events pass your trigger in the debug stream … Elizabeth Gallas - COMA 13

COMA Single Run -/+Trigger Section Start to see here some COMA “derived” info COMA divides HLT trigger signatures: “Physics”: complete EF-L 2 -L 1 chains § “Commissioning”: has missing level(s) COMA looks at each LB range: 1. Decides if each chain is “active” based on prescales, passthrough, rerun in every LB range of Prescale evolution 2. For “active” chains via prescale, calculates an “Aggregate prescale” (EF*L 2*L 1 prescale) in each LB range 3. Determines Min, Max of Agg. PS over the Run for each chain 4. Determines Run and LB-wise PS, PT, RR Flags § Open this section in next slide… June 2011 Elizabeth Gallas - COMA 14

COMA single-Run Report: “Physics” Trigger Section Choose (using j. Query. Column. Filters) Chains: EF*JPSI and Stream: egamma re ate p eg Aggr ) S, RR -wise T, P Run gs(P Fla and nge e Ra scal COMA Chain Report: All Runs w/ this chain/item COMA Prescale Report: LB-wise PS info for this Run/Signature (next slide) June 2011 Elizabeth Gallas - COMA 15

COMA Prescale Report (Run 180776, EF_e 9_tight_e 5_tight_Jpsi) Run-wise Summary: § Includes links to COMA reports: § § Run, Master Key, Chain and Prescale Reports (each level) Associated L 2, L 1 names Aggregate PS Range: 4 to 5 Flags: § § § Partially active via PS Never passthrough (L 2 or EF) Never Rerun LB Range Summary table: § LB-wise Prescale Evolution § Link: Trigger DB Prescale Reports (useful cross check !) § Prescales: § § L 1, L 2, EF and Aggregate PS Chain Inactive from § LB 1 -99 and 114 -134 (LB ranges in COMA always inclusive) Next slide: COMA Chain Report for EF_e 9_tight_e 5_tight_Jpsi June 2011 Elizabeth Gallas - COMA 16

RBR Chain Report (EF_e 9_tight_e 5_tight_Jpsi) Configuration Summary: Shows where this element is configured: § Super Master Key(s), § Runs (Summary) Period Evolution: Shows chain/item bit, version evolution for “Physics” chains during Period Runs Activation Summary: Shows the Runs in which this element is “active” separated by: § “Physics” EF-L 2 -L 1 signatures § § Open these sections in next slides… June 2011 Via prescale Via pass through Via rerun “Commissioning” signatures § § § Elizabeth Gallas - COMA Via prescale Via pass through Via rerun 17

COMA Chain Report (EF_e 9_tight_e 5_tight_Jpsi) Expand Run-wise Activation … 70 Period Runs where this chain is “active” “Physics” EF-L 2 -L 1 signatures Active via Prescale Runs in Data Periods Table Shows (Run Count): § § § June 2011 Periods, Link: Run, SMK Reports Level bit assignments Link to: Chain/Item Reports (3) Range of Aggregate Prescale while chain is active via prescale in Run Links: COMA Prescale Report (3) Elizabeth Gallas - COMA 18

COMA Chain Report (EF_e 9_tight_e 5_tight_Jpsi) Period Evolution Section … § For “Physics” chains in Period Runs § Separate table for each EF-L 2 -L 1 signature at each beam energy Each Table Shows: § Row-wise: Distinct set of bit and chain/item versions Columns: § Bit assignments § Chain version (links to Trig diff) § Chain Report Links § Range of Agg. PS, SMK, Run, Period, Date, HLTRelease oerg omasz, J o. T Thanks t ul discussions sef r many u fo June 2011 Elizabeth Gallas - COMA 19

COMA Chain Wildcard Report (input: “EF_g 10%”) Purpose: See all the names matching a pattern or Find exact name from part of the name Report: Displays chain/item names matching the input string … text size proportional to occurrence in SMK § In Period Runs and in All Runs June 2011 Elizabeth Gallas - COMA 20

Summary and Plans § COMA – an integral part of ATLAS Metadata infrastructure § Essential to ATLAS event-level metadata decoding § Ideally placed to provide links and interface to other metadata § § § Primary source for “ATLAS Data Periods” § § Trigger experts recommend COMA Trigger/Prescale Report usage: from ~200 to over 5000 pages viewed/month. § § Periods in Lum, DQ, Run Summary, AMI reports comes from COMA Reports feature “derived” information not available elsewhere § § Special relationship to AMI (and TAG catalog) Launch i. ELSSI to take a quick look at any Run Peaked in July as users did final preparations for summer conferences Current efforts: § COMA Database content growing § § COMA Report and Browser development § § Watch use cases to identify new areas to focus growth keep pace with content, improve functionality and usability Beyond COMA: Interface development § June 2011 Connecting COMA to other parts of the infrastructure Elizabeth Gallas - COMA 21

COMA Conclusions § This is an evolving system … information in the system is growing based on information available and use cases § § § We want to insure the Metadata is § § § complete enough to satisfy use cases while reflecting accurately its limitations Interfaces are being constructed to use selection syntax, criteria, and communication in common use in ATLAS § § Adding more dimensions to the Conditions data § With suitable relationships to facilitate queries Making that criteria available in dynamic useable interfaces This facilitates cross checks with other systems Continuous process: talking with various experts to ensure § data integrity, completeness, compatibility w/other systems … Very positive feedback so far … more always welcome … hn-atlas-physics. Metadata@cern. ch June 2011 Elizabeth Gallas - COMA 22

Backup June 2011 Elizabeth Gallas - COMA 23

Link Summary Main TWiki Documentation: https: //twiki. cern. ch/twiki/bin/viewauth/Atlas/Conditions. Metadata COMA Report Documentation: https: //atlas-tagservices. cern. ch/RBR/r. BR_Documentation. html COMA Portal (grid certificate in browser is required) https: //atlas-tagservices. cern. ch/tagservices/Run. Browser/index. html Trigger General Meeting (June 8, 2011): “User Guide to Trigger/Prescale information in COMA Reports” https: //indico. cern. ch/material. Display. py? contrib. Id=0&material. Id=slides&con f. Id=125723 § Data sources, Business rules and Caveats; Form input; Example links; details of Data Period & derived quantities; Data Periods TWiki (revised): https: //twiki. cern. ch/twiki/bin/view/Atlas. Protected/Data. Periods June 2011 Elizabeth Gallas - COMA 24

COMA Report(s) Example Links COMA single-Run Report (Run 161562): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=161562 § General Run information, AMI tags, DQ, Prescale Evolution, Triggers § Trigger tables show derived Run-integrated prescales, passthrough COMA Trigger Super Master Key Report (Key 877): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? smk=877 § Runs using this SMK, physics/commissioning chains, Level 1 items COMA Chain Report (Chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 11_etcut § § Summary: SMK, Runs, Signature appearances of this chain Runs with this chain ‘active’ COMA Chain Wildcard Report (chain name matching string EF_g 1%): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? cn=EF_g 1% § Summary: SMK, Runs with this chain: Configured, Active COMA Prescale Report (Run 162882 Prescales for chain EF_g 11_etcut): https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. php? runs=162882&cn=EF_g 11_etcut § Derived chain activity over the Run and LB-wise prescales June 2011 Elizabeth Gallas - COMA 25

Data Periods: Links to Reports and Services Revised TWiki page: https: //twiki. cern. ch/twiki/bin/view/Atlas. Protected/Data. Periods § Interactive USERS COMA Data Period Documentation Interface § https: //atlas-tagservices. cern. ch/RBR/r. BR_Period_Report. php § § Comments: hn-atlas-physics. Metadata@cern. ch. Programmatic USERS For systems needing period info: run. Query, beamspot, Data Quality, …, “Data Period Services” provided via py. AMI: § § http: //ami. in 2 p 3. fr/opencms/AMI/www/Client/Data. Periods_py. AMI. pdf § Comments: AMI / Tag_Collector Team. Data Preparation EXPERTS: Entry Interface: § https: //ami. in 2 p 3. fr/AMI/servlet/net. hep. atlas. Database. Bookkeeping. AMI. Servlet. Command? link. Id=1479 § June 2011 Comments: AMI / Tag_Collector Team. Elizabeth Gallas - COMA 26

COMA run. Browser. Report Index page § § INDEX: https: //atlas-tagservices. cern. ch/RBR/run. Browser. Report. html GUIDE: https: //atlas-tagservices. cern. ch/RBR/r. BR_Documentation. html “Index” page: § Enter one/more input quantity, § IE: a Run number § IE: a Chain Name § Click on submit (or return key) Links at bottom of every page: Reports use “GET” methods: Modify the URL to change input Reports contain links No need to use index page for every query Links lead you to other COMA reports other external reports AMI, run. Query, Trigger. DB… June 2011 Elizabeth Gallas - COMA 27