New Developments in Data Analysis Tools The Anaphe

New Developments in Data Analysis Tools: The Anaphe project 8 th Topical Seminar on Innovative Particle and Radiation Detectors Siena, 21 -24 October 2002 Lorenzo Moneta CERN IT/API Lorenzo. Moneta@cern. ch Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002

Outline z Introduction z AIDA (Abstract Interfaces for Data Analysis) yconcept and design z Anaphe yarchitecture ycomponents yuser examples z Summary and Conclusions Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 2

Introduction z Complexity of detectors and huge amount of data produced y Impose strong requirements on computing systems and their software to reconstruct and analyze the data z Need a long term vision ATLAS y Technology changes y Maintenance z Use of modern techniques: y Rigorous software engineering y OO programming y Importance of modular software xsee I. Papadopoulos ’s talk on Monday y Role of abstract interfaces CMS (component based programming) z Example: success of Geant 4 Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 LHCb 3

Abstract Interfaces z An Abstract Interface (Class) specifies a protocol how clients may access and manipulate a component z Defines no implementation but only functionality z Essential element of OO to achieve a modular design: y Clean separation of specification and implementation y Clean separation of components y Components can be upgraded or replaced without effecting usage ( plug in /out model) Interfaces are the communication protocol of the bus SW bus C 1 Lorenzo Moneta, CERN IT/API C 2 C 3 C 4 Siena, 21 -24 October 2002 components 4

What is AIDA ? z AIDA : Abstract Interfaces for Data Analysis z Open source project with the goal to define abstract interfaces for common physics analysis objects y Histograms, ntuples, functions, fitter, plotter, tree and data storage z Defines a common XML format for data exchange z Allows multiple implementations and multiple languages y C++, Java and Python z Exist three AIDA implementations: y. Anaphe (CERN) in C++ y. JAS (SLAC) in Java y. Open. Scientist (Orsay) in C++ Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 5

AIDA z User code sees only the abstract interfaces y. Implementation can be selected at run time without any change to the code (loading shared libraries) Anaphe User program (E. g. Geant 4) A I D A Open. Scientist JAS Other Tools? Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 A I D A F I L E 6

AIDA History z AIDA started in 2000 by defining a common interfaces for histograms z First end-user release (v. 2. 2) end of 2001 z New AIDA release 3. 0 in October 2002 y large improvement in functionality (fitter and plotter) y New Anaphe release 5. 0. 0 implementing AIDA 3. 0 y JAS and Open. Scientist releases expected soon z Geant 4 adopted AIDA for analysis z AIDA is used also within Gaudi (SW framework used by LHCb, ATLAS and HARP) z Recommended for adoption by LHC Computing Grid project (LCG) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 7

Example of AIDA z Histogram interfaces IHistogram: IHistogram 1 D interface Common functionality for all histograms (like entries, label, dimension, ) IHistogram 1 D IHistogram 2 D IHistogram 3 D Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 8

AIDA implementations z JAS (Java Analysis Studio) y jas. freehep. org/ y Analysis tools developed a SLAC written in Java y Easy to use and robust, multi platform, flexible and easy extendable y Large user community (Ba. Bar, GEANT 4 through AIDA ) z Open. Scientist y http: //www. lal. in 2 p 3. fr/Open. Scientist y Modular tool developed by G. Barrand (Orsay) y Collections of various C++ packages (histogramming, visualisation, storage) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 9

Anaphe z Anaphe : Analysis for Physics Experiments z An project in CERN IT division to provide a modular OO/C++ alternative to CERNLIB z Provides libraries for y Histograms and Ntuples y Plotting and visualisation y Fitting and Minimisation y Management and storage y Interactive analysis using Python (Lizard) z Try to use standards wherever possible z Try to re-use existing class libraries Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 10

Layered Architecture of Anaphe z Basic functionalities (histograms, fitting, etc. ) are available as individual C++ class libraries (components) z A thin wrapper layer implementing AIDA using the component libraries è Easy to adapt to changes in interfaces due to user request (e. g. adding functionality) z A developer interfaces level extending the AIDA interfaces y More efficient (extra functionality is needed internally) y Maintain insulation è Easy to replace a component without affecting usage z User sees only top level (AIDA) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 11

Anaphe Architecture AIDA interfaces IHistogram IPlotter IFitter developer interf. IDev. Histogram IDev. Plotter IDev. Fitter wrapper layer AIDA Plotter AIDA Fitter Grace Plotter FML Basic components Histo library Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 12

Anaphe Components Lizard Analyzer Interactive Commands Histograms NTuples Fitting Plotting Functions Data. Point. Set Histogram Library Ntuples Library Fitting and Minim. Plotter Store libraries + other utility libs Python / SWIG XML parser Qt Grace NAG-C Objectivity User’s C++ code optional Abstract types Anaphe Implementations AIDA (Abstract Interfaces for Data Analysis) Lorenzo Moneta, CERN IT/API CLHEP CERNLIB Hep. ODBMS HEP implementations non-HEP components commercial components Siena, 21 -24 October 2002 14

Anaphe History z LHC++ project started in 1997 z HEP foundation libraries developed 1997 -2000 z Anaphe started in 2000 with first version of Lizard (interactive python component) z Production version Summer 2001 z Major re-design in 2002 to integrate with AIDA y AIDA 2. 2 compliant version Summer 2002 z New version October 2002 implementing AIDA v 3. 0 y New Wrappers for AIDA y Improved Histograms and Ntuples libraries y New Plotter library based on Grace y Introduction of XML store Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 15

Histograms and Tuples libraries z Histogram Library y Based on ideas of previous library (HTL) developed for LHC++ y High performance y Histograms (up to 3 D) and profiles y Unbinned histograms (clouds) z NTuple Library y Raw and column wise (Hbook type) y Nested ntuples (ntuple in ntuple) x Event/track/hits z Data Point Set (Vector of Points) y Simple container for n-dimensional measurement points (values and positive/negative errors) y Functionality to have operations (add/subtract) on different sets Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 16

Plotter libraries z Qplotter y used by old Anaphe versions y library based on Qt Free 3 ( C++ & GUI open source library) z GRACE Plotter y introduced in latest release z Based on GRACE y a open source graphics package under GPL license y Very high quality graphics and powerful (publication quality plots) y Convenient point and click user interface y Flexible and easily extendable y Easy integration in Anaphe Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 17

Fitting library z Fitting and minimization library (FML) y Flexible OO library. y Using minimization engine based on NAGC/MINUIT but easy extendable to others y Powerful: 2, binned and unbinned maximum likelihood fits y Plug-in mechanism to load user functions y Implement new AIDA 3. 0 interfaces (lots of new functionality) y Integrated with all data sets (histograms, data points, ntuples) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 18

Management and persistency z Hide implementations from user y Use factories to create objects (Histograms, Ntuples, …. ) z Objects are managed in a tree-directory structure y Support for Unix-like directory and commands (ls, cp, mv, …) z Tree hides store details from the user y User chooses store type at run time (when creating the tree) z Multi store types functionality y can run with two different store type at the same time ! z Support in Anaphe for three store types: y XML (compress and uncompress) defined within AIDA x Possible to exchange files with other AIDA implementations (JAS) y Hbook (only histograms and Tuples) y Objectivity using HEPOBDMS z Easy extendable to new types Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 19

Interactivity: Lizard z Lizard : Python environment for interactive analysis y Unified user interface at top level y AIDA types and methods mapped into Python commands xuse SWIG to generate the mapping from the C++ classes y User modules can be plugged in as required y Analyzer module provides on-the-fly compilation and running of user code C 1 z Python as scripting language: y Easy to use y Object Oriented language y Maps well to C++ and Java y Huge user base with lots of free software (networking, GUI, OS, scientific etc ) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 C 2 Lizard C 3 C 4 C 5 C++ component libraries 20

Example of Lizard z Lizard code example in Python: y Creating an Histogram, filling, fitting and saving the result in an XML store # create the tree with an XML store tree=tf. create ("my. Example. xml", "XML", 0, 1) # create histogram (first factory) hf = af. create. Histogram. Factory(tree) h 1 = hf. create. Histogram 1 D(“ 1", "Gaussian Distribution", 100, 0. , 100. ) # filling for i in range(0, 10000) : xval = 1. *random. gauss(45. , 10. ) h 1. fill(xval, 1. ) #fitting – create first fitter from Factory fitter = fitter. Factory. create. Fitter(“Chi 2”, ””) fit. Result = fitter. fit(h 1, ”G”) #save all in XML file tree. commit() Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 21

Anaphe Users z Users from HEP and non HEP community z Geant 4 has adopted AIDA as a tool-independent analysis standard z Anaphe is used in GEANT 4 y In the advanced examples (ATLAS and CMS calorimeter test beam simulations) y And in analysis of underground, astroparticle experiments and even in medical applications (radiotherapy) y Adopted for GEANT 4 test and validation process z Running of Anaphe in a distributed environment (GRID) è See next talk of J. Moscicki z Interest in AIDA also from LHC Computing Grid project (LCG) Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 22

Example of Anaphe users Courtesy of ESA Astrophysics Bepi Colom bo UK Dark Matter LISA Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 ESA mission to Mer spectrum from a Marssimulant Anaphe Brachyterapy 23

Summary z Anaphe is a layered set of loosely coupled C++ components for data analysis, plus an interactive Python framework (Lizard) y. Easy to use y. Applicable to different environment z HEP-specific parts written in-house z Developed and maintained by CERN IT z Committed to AIDA compliance y. Following LCG recommendation z Open to new requirements from experiments Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 24

References z. For documentation, downloads and more information y. AIDA: xhttp: //aida. freehep. org/ y. ANAPHE: xhttp: //cern. ch/anaphe zor send mail to xanaphe-editors@cern. ch Lorenzo Moneta, CERN IT/API Siena, 21 -24 October 2002 25