US CMS US ATLAS ITR pre- proposal Globally

US CMS + US ATLAS ITR (pre-)proposal “Globally Enabled Analysis Communities” Sound Bites: “Dynamic workspaces” “Private Grids to support scientific analysis communities” “Build autonomous communities operating within global collaborations” “Empower small groups of scientists to profit from and contribute to international big science” “Democratization of Science via new technologies” What is it? Well, we will have to define it in the proposal, due in March 2003 Lothar A T Bauerdick Fermilab 3/18/2018 1

Berkeley Workshop Nov 2002 -- The Global Picture Development of A Science Grid Infrastructure (L. Robertson) LHC applications middleware R&D (CS) bio applications other science applications science …. . end-to-end hep middleware end-to-end bio middleware end-to-end middleware Application/grid interface Advanced Middleware requirements defined by current projects Hardening/Reworking of basic middleware prototyped by current projects middleware engineering Lothar A T Bauerdick Fermilab operation of base grid infrastructure • ~ 4 -5 core centres (including the LCG Tier 1 s) other grid nodes for physics, • information service, catalogues, . . biology, medicine, …. • coordination, operations centre, . . • call centre, user support, training, . . 3/18/2018 2

… and the “missing pieces” Transition to Production Level Grids (middleware support, error recovery, robustness, 24 x 7, monitoring and system usage optimization, strategy and policy for resource allocation, authentication and authorization, simulation of grid operations, tools for optimizing distributed systems) Globally Enabled Analysis Communities (WG 2) Enabling Global Collaboration (a medium ITR? ) Lothar A T Bauerdick Fermilab 3/18/2018 3

The goal: Provide individual physicists and groups of scientists capabilities from the desktop that allow them: è participate as an equal in one or more To “Analysis Communities” è Full representation in the Global Experiment Enterprise è on-demand receive whatever resources To and information they need to explore their science interest while respecting the collaboration wide priorities and needs. Lothar A T Bauerdick Fermilab 3/18/2018 4

Assumptions of the ITR pre-proposal èThe Project will provide some of the missing capabilities for ATLAS and CMS data analysis systems. èProject will be managed as part of US CMS and US ATLAS S&C projects. èAn existing robust, fully functional core Grid Infrastructure for work flow management, data management (data, meta-data, provenance), and security management. èProject will deploy incremental capabilities for experiment use throughout the lifetime. èUp front experiment-wide management and oversight will ensure appropriateness and buy-in. Lothar A T Bauerdick Fermilab 3/18/2018 5

Constraints of the ITR pre-proposal è may actually not get accepted. . . It è Five year development and deployment program, can’t start before end of 2003 è total possible funding is $15 M over 5 years the (including all overheads that is “only” some ten FTE!) è Many (>12? ) institutions/universities want to participate in the project è Development teams include Computer Scientists (Information Technologists? ) and Physicists è Funding comes from Computer Science department in the NSF, and reviewers are mainly Computer Scientists. Lothar A T Bauerdick Fermilab 3/18/2018 6

Physics Analysis in CMS The Experiment controls and maintains the global enterprise: Hardware: Computers, Storage (permanent and temporary) Software Packages: physics, framework, data management, build and distribution mechanisms; base infrastructure (operating systems, compilers, network, grid); Event and Physics Data and Datasets Schema which define: meta-data, provenance, ancillary information (run, luminosity, trigger, Monte-Carlo parameters, calibration etc) Organization, Policy and Practice Analysis Groups - Communities are of 1 to many individuals Each community is part of the Enterprise : è Is assigned or shares the total Computation and Storage è Can access and modify software, data, schema (meta-data) è is subject the overall organization and management Each community has local (private) control of è Use of outside resources e. g. local institution computing centers è Special versions of software, datasets, schema, compilers è Organization, policy and practice We must be able to reliably and consistently move resources & information in both directions between the Global Collaboration and the Analysis Communities can share among themselves. Lothar A T Bauerdick Fermilab 3/18/2018 7

Environment for CMS (LHC) Distributed Analysis on the Grid Dynamic Workspaces - provide capability for individual and community to request and receive expanded, contracted or otherwise modified resources, while maintaining the integrity and policies of the Global Enterprise. Private Grids - provide capability for individual and community to request, control and use a heterogeneous mix of Enterprise wide and community specific software, data, meta-data, resources. Lothar A T Bauerdick Fermilab 3/18/2018 8

The Global Community C C St C So. So X Sc. Sc C X Sc St CC St Sc So So So X X Information flow Lothar A T Bauerdick Fermilab C C St a private grid/analysis community X - physicist So - software C- compute St- storage Sc - schema/information - desktop 3/18/2018 9

Technologies to be Developed The CS and IT part of the proposal! Lothar A T Bauerdick Fermilab 3/18/2018 10

Infrastructure to support Private “Community Grids” Meta-data to describe and manage private grids. Tools for information and data transfer and communication between communities Synchronization and validation tools between the community grids and the global enterprise. Application/user interfaces for management, administration and operation of set of private grids within an enterprise. Lothar A T Bauerdick Fermilab 3/18/2018 11

Infrastructure to support dynamic workspace capabilities Rapid response reconfiguration and administration tools. Enterprise wide integrity and validation tools across all private grids. Application/user interfaces for the distributed request and control of extensions and contractions of private grids. Lothar A T Bauerdick Fermilab 3/18/2018 12

De-centralized, multi-tiered schema evolution and synchronization Mechanisms to support parallel evolution and resynchronization of decentralized heterogeneous local and enterprise schema. Application interfaces for definition, modification of and access to local and enterprise schema. Lothar A T Bauerdick Fermilab 3/18/2018 13

HEP specific developments: User Interfaces: èDescribe, modify, control and access all physics analysis data and meta-data èControl and manage analysis processes èRequest and use private community grids Integration with physics codes: èDefinition of architecture into which ITR deliverables will fit èIntegration activities to include, test and validate ITR deliverables interactively. Lothar A T Bauerdick Fermilab 3/18/2018 14

Architecture and Work Plan Architecture is very important for developing the work plan --> Proposal!! è Blueprint RTAG continuation w/r to Grid interfaces? è Computing model, work loads, data flows? We need to discuss and agree on a system architecture and components from the end-to-end physicist user to underlying grid infrastructure assumed to be in place; We need to have a project work plan (concrete set of tasks) in place with deliverables to make people realize this is a project to deliver working production systems for the experiments to use as the project proceeds; We need to negotiate and agree on some deliverables/components from other projects e. g. from LCG Applications Area; and to understand the time line in terms of what capabilities would be available when; We need to upfront explicitly identify who is going to do test cases of running analyses with these tools to provide feedback to development effort --> link to data challenges Lothar A T Bauerdick Fermilab 3/18/2018 15

LCG Application Domain LCG Applications Area Blueprint: Blueprint Scope 4) Physics data management: … Provide data management services meeting the scalability requirements of the LHC experiments, including integration with large-scale storage management and the Grid… Schema evolution , propagation, merging relevance to grid interfaces and capabilities? it might for example be useful to look at extending it Lothar A T Bauerdick Fermilab 3/18/2018 16

This ITR should be of direct benefit for CMS Each analysis group/physicist will be able to perform local analyses which can be reliably and quickly validated and trusted by the collaboration on request. The experiment will be able to demonstrate and compare methods and results reliably and improve the turnaround time to physics publications. The experiment will be able to quickly respond to and decide upon new requests from analysis groups/physicists for resources, with minimal perturbation to the rest of the collaboration The experiment will have an established infrastructure for evolution and extension for its long life. There will be a lowering of the intellectual cost barrier for new physicists and researchers to contribute. We will enable small groups to perform reliable exploratory analyses on their own. There will be an increased potential for individual or small community analyses and discovery. Individuals and groups will be assured they are using a well defined set of software and data. Lothar A T Bauerdick Fermilab 3/18/2018 17

This may all “seem easy” but ask any physicist doing analysis on a large experiment today… And the LHC is 10 times larger in all dimensions. Lothar A T Bauerdick Fermilab 3/18/2018 18