CHEP 2000 Session F Data Analysis Technology and

CHEP 2000 Session F: Data Analysis: Technology and Presentation Convenors: Francois Etienne - IN 2 P 3, Atlas, Ruth Pordes, FNAL Thank You to all the Speakers for Providing me with their Transparencies to use. 29 Oral presentations; 1 Poster

Outline • Quick Review of Topics/Titles • More Detail for Each Category • Summary

Numerical Library Software - 2 papers • Development of Numerical Library Software in Java – KEK – Initial Work • Scientific Toolkit for Java – Phenix – Really an Analysis Framework • (poster paper not in this session on COLT - Cern/IT) • All in Java

Electronic Document Management systems - 4 papers from 2 systems • Java-based User Interfaces for CERN's Engineering Data Management System • Integrated Information Management for TESLA • Asset Management for Cavities and Computers at DESY • Experience with Document Management at DESY

Singletons: • Bringing Mathematical Formatting to the Web - An Overview of the Technology and SLAC's Experience • Multi-threaded, Discrete Event, Simulation of Distributed Computing Systems – CMS • CORBA Scripting in HEP and Beyond. • ATLAS DAQ back-end Software § § Integrated toolset, mainly Open Source Second use of this will be an interesting test. Defining deliverables of “pre-design” phase. How to move to an Open Source development environment for critical piece of Experiment architecture?

Event Displays - 3 talks • Atlantis an Event Display for ATLAS: When displaying 6000 hits recorded by only 7 layers, can we recognize tracks? • WIRED - World-Wide Web Interactive Remote Event Display • Hep. Rep: a Generic Interface Definition for HEP Event Display Representables

General Analysis Packages/Frameworks • Java Analysis Studio • ROOT at RUN II • Open Scientist • A Comparison of Data Analysis Packages (Root and IDL)

Experiment Analysis Packages • A Graphics Sampler for the D 0 Experiment • The CMS IGUANA Project • Object Oriented Data Analysis in the DELPHI Experiment • Distributed BELLE Analysis Framework • New Data Analysis Environment in H 1

General Analysis Components • Abstract Interfaces for Data Analysis – AIDA – Cern/IT • A Histogramming package in Java - KEK • Lightweight Histogram Interface Layer – from NERSC/LBL

Software Process and Configuration Management - 5 talks all from Europe. . • An Evaluation of Tools for Static Checking of C++ Code • Impact of Software Review and Inspection • C++ Code Analysis: an Open Architecture for the Verification of Coding Rules • CMT: a Software Configuration Management Tool • Use of a Configuration Management Tool in LHCb Software

What there was Not • Software Development Methodologies and Case • Tools. 3 -D and Virtual Reality. Overall Impression • Continuation of directions and initiatives from last Chep.

F 19 Math. ML Interface for Abstracts in SLAC Spires • Situation today (finally): § § method for defining a mathematical expression in terms of its presentation and semantics Math. ML is a standardized instance of XML • Used “IBM Tech. Explorer Hypermedia Browser” § § § A Web browser plug-in Dynamically formats and displays documents coded with TEX/LATEX and/or Math. ML Beta version applied to existing Spires Te. X files works. • Try it and feed back comments. . .

F 19 - Tool for Simulation of Complex Distributed Data Handling Systems • Technology – in house developed JAVA toolkit. Process driven not event driven architecture § Full presentation and easy editing of parameters § encourage many different models and users § Immediate publication of results on the web § Enable physicists to read, analyse and feedback. Alternative architectures and designs can be commented with quantitative and analytic input Technology is ready to be tested for “Simulator Friendliness” … § Volunteers to model the existing Babar and Run II systems would indeed be useful. § • •

F 359, F 65, F 66, - DESY Integrated Information Management • Reminded us of the importance of need for and time required to § Peoples Use and Access Patterns change after they have experience with a working system § Current and Future Business Processes significantly affect design § Users expect latest Technology (currently video, mobile computing etc) • Demonstrated Success of collaborative development of vendor tool for HEP needs

Current Tuovi Status & Statistic • 20 projects with: - 6 GB data vault - 2400 project nodes - 18600 files - 3300 registrated user

Cern EDMS Extensions • Import/Export Interface - define an exchange data • format in XML. § Conversion of metadata to and from various formats e. g. Excel, Html, Databases, Text Processors etc. in use by several LHC groups: § § § LHC Baseline and ABS ATLAS: Import/Export of ABS Structures CMS: ECAL ABS ALICE: Documents and Files check-in Documents Management in the LHC Design office (in progress)

Histogramming Packages • Histogrammer in JAVA - Successful interfacing of Java and C++ objects for in a simple Histogrammer application • Lightweight C++ Histogram Interface - Break between Statistical Data Gathering, from analysis and visualization tasks. Define Standard API and Support Object Structures. § How does it relate to transient Cern/HTL implementation

Analysis and Event Display Toolkits and Interfaces • • WIRED Hep. Rep AIDA Java Analysis Studio Scientific Toolkit for Java Open Scientist ROOT IDL - through comparison with ROOT

Wired • Rewrite since Last CHEP • In Use by CHORUS, Ba. Bar, DELPHI etc. • Lacks a Scripting Language. • Ba. Bar Use shows up Need to Think before Defining what Data gets Transported between User Client and Data Server – don’t want a “Packet per Point”

Hep. Rep • Experience with Wired at Ba. Bar led to need to layer interface between Client and Display so as to § reduce granularity in getting remote information § increase ability to for Client/Users to easily add objects and views • Hep. Rep defines a general API for serving Representables, not Physics Objects • “Serving Representables keeps the detailed reconstruction code, swimmers and detector models on the server side where they belong. • Spatial information is assembled and shipped in an efficient manner, avoiding the overhead of too many individual method calls. ”

General Comments • Transport the Physics Analysis Code to the Data - JAS – No Answer Yet on Whether this will Work Well. • WIRED and JAS both Talking about Scripting Languages – no decision yet. • Once Package in Use and Responding to User Use Needs, More Core Development More Difficult • Integration with C++ Analysis Algorithms and Data Adaptors Not Here Yet.

AIDA • Part of Next LHC++ Analysis Toolkit – the Start. • Hep. Vis ’ 99 established Collaboration for Definition of Abstract Interface for Data Analysis • Interface Architected to Support Scripting and GUI Equivalently. • Goal of Adaptors/Interfaces to Hep. Tuple, JAS, Openscientist, Root etc.

JAS • Extensions since last CHEP: § § JEditor Bean Using Swing JEditor. Pane JAS now supports HTML display XML support - JASHist bean directly supports reading/writing XML Servlet Support • Continued use in Ba. Bar Online and LCD Simulations • Packaged and made Available in Open. Source Paradigm - Contributions starting e. g. 3 D Classes from Fermilab.

Scientific Toolkit for Java • Explained was a Separate Project to “Learn Technology”. § Use of Java. Help § Based on. Java 2 D (JDK 1. 2) • Generic Swing Console for those Interpretive Languages not supported by Swing. • Anticipate Discussing Collaboration with / Contributions to JAS. • Starting New Development Allowed use of Latest Generation Technologies

Open Scientist • Work Continuing for Experiments – LHCb, Virgo, Auger etc. • Concept of “Hub” as Integration Layer. What is boundary/layering between Frameworks and Utilities Do we need More Concepts Here? • Worklist includes Integration of CINT and Extensions to Free Open Inventor Implementation.

ROOT - Run II Perspective • Progress since Run II Adoption of ROOT with Caveats. • Priorities from 1999 Workshop been in Large Part Addressed – Robustness, CINT Extensions, Precompilation of user classes. • Tutorials and Education Popular with Run II Users. • To meet Local Needs currently Building 23 Flavors. • Collaboration Works with Technical Commitment • Second ROOT Workshop last week at Cern resulted in New Worklist - now posted – main extension Self Describing Data format.

Comparison of ROOT and IDL: • IDL command language § § Concatenate Sets of Files of ntuples - chain command. Read in and Use Variables from an “ntuple”File Local Extension. Event Selection Cuts on Several Variables Plot - Static or “Liveplot”. • Large ntuple problem addressed by “mapped files”. • Needs test drive for “large scale analysis”.

Experiment Specific Event Displays and Analysis

Mature Experiments Retooling Technology for Analysis – Delphi and H 1 reported • Issues include: § § Maintenance – Technical and Resources; Support of Multiple Generations of Technology. • OO, ROOT, C++ - Speed and Usability Benefits • Encourage New Collaborators • Allow and Provide for Curation of the Data

Atlas Event Display - Atlantis • Applying Hans Drevermanns Algorithms to New/Future Experiment § “V-plot” § Z finder § Hit filtering • Plan Recode from Fortran to Java.

Experiment Event Displays – D 0 • Suite of Different Tools based on Legacy and New Packages. • New Tools based on Open Inventor. Will Evolve. • Finding very useful for algorithm development and starting for analysis of processed Monte Carlo data

D Graphics Sampler Rest of D 0 software D Graphics HEPVis Open Inventor Xi Graphics Exceed 3 D Mesa Open. GL

CMS - IGUANA • On the way to being a Fully Featured Data Presentation • • and Manipulation Framework. Will be used in CMS Simulation and Test Beam Analysis this Year. Pragmatic Approach to getting Prototypes and Early Production Releases. Based on Open Toolkits and Open Inventor. “Sharing experience may be more important than sharing code”- aligned with LHC++ strategy.

IGUANA Prototypes: Objectivity Browsers and tor Persistent Histogramsnand Tags Plotters ve r , & rer lo xp orer E ep xpl H a) RIS E 6( I o PIn vent E ) H en. In if b 6 ( Op Mot & • IGUANA Objectivity browsing (d) 6 wt Q Qt & • ) 6(c lot ci. P Qt) S / t (X y vit ti ) ec r bj se uple A O aly o nt N n UA ag a ilar t IG t im s is g (ta

BELLE – BASF Analysis Framework • Currently supports Single SMP Parallel Processing of Events for Users. • d. BASF Extension being Designed to Support Network Connected Clusters of PCs. Package of Daemons for § Job Management § Data I/O § Resource Management § Monitoring. • Resource Managers Dynamically Reconfigures – Increase, Decrease - Number of Computing Servers Running Jobs.

HEP Configuration Management Tools • Report from 2 users of CMT – Build and Release System. • Poster papers on SRT Rewrite, NT Extensions • From Marjorie’s Plenary Talk: “Everyone now using them, can’t we use the same one…”

Packages & versions Free action onto the package editor Package areas browser jcmt Configuration monitor Requirements editor

F 63 - C++ Code Rules and Checking • C++ Coding Standard – 180 Rules for Naming, Coding and Style of Varying Severity for Each Experiment. • Evaluation of Commercial tools for Code Rule Checking. Tested on Geant 4 and Atlas Libraries. • Out of 5 tools considered 2 Meet the Basic Needs – about 75% of total: Code. Wizard and QA C++. • Now Traditional Questions on: “What does it take to Decide? ” “Will IT/CD Support It? ”

F 361 - C++ Code Rules and Checking Alice • Development of Code Analysis Tools - Parser, • • Formatter, Checker Limitations in Checking Complete C++ syntax e. g. Templates, Namespace Successfully used on ALICE code

F 331 - Atlas Software Review and Inspections • Movement in HEP from Fagan Inspections (Fermilab SDSS/DART/Run II) to Gilb Inspection Method • Bodes Well that this is Part of Formal Development of Proposed Open. Source Project. .

F 119 - Applications using CORBA Script – continuation of work reported at Chep ‘ 98. • Interpreted Scripting Language which can interact with CORBA objects through IIOP and has dynamic access to any CORBA object. • OMG Standard • Interface Definitions Stored in Repository • Applications in Babar Data Handling System

Summary – Technology Java, XML, CORBA • JAVA § Mainstream for New Presentation Applications. § Not Mainstream for Data Analysis and Reconstruction. § Work Not yet Done to Integrate with C++ Algorithm Modules and Frameworks. § Work only just Beginning on HENP Utility Libraries.

Summary – Physics Analysis Packages • Not much development using Commercial Analysis Tools. • ROOT most Generally Used; Extensions still being requested to meet Experiment Long Term Needs. Open Source Collaboration without Open Source licence? • JAS, Open. Scientist, WIRED, progressing steadily. • Question still Open as to How or Whether any of the Existing Analysis Technologies will Successfully meet Needs of >500 Physicists doing Analysis of Babar and Run II over the next 10 Years.

Summary – Development Environment Technology • Experiments looking for more and better tools for Configuration Management but § no widespread adoption yet of single solution § no work yet on significant extensions to the services provided § no new work to look at commercial tools • Attendees very interested in Code Quality and Checking Tools and Process – but if the number of talks is an indication of the relative investment. .

Summary - misc • Mature Experiments Taking Time to Upgrade Analyses to New Technologies to Support Ongoing Analysis and Provide Tools for Newcomers – § Reminder this will Happen for Current and Future Experiments. • (Where did Virtual Reality go? )