Design Development and Evolution of the ROOT system

Design, Development and Evolution of the ROOT system Rene Brun February 21 2010 ACAT 2010 Jaipur ROOT: design, development, evolution 1 René Brun, ACAT 2010, Jaipur

plan ROOT in 2010 packages users How did we reach this situation? after-PAW fights Objectivity era and big confusion about object persistency LCG project and problems with the committee approach the ramping and consolidation years Directions ROOT: design, development, evolution 2 René Brun, ACAT 2010, Jaipur

Software hierarchy End user Analysis software 0. 1 MLOC 2 MLOC Experiment Software 2 MLOC Frameworks like ROOT, Geant 4 OS & compilers Hardware Hardware ROOT: design, development, evolution 20 MLOC Networking 3 René Brun, ACAT 2010, Jaipur

ROOT today a quick review ROOT: design, development, evolution 4 René Brun, ACAT 2010, Jaipur

15 years of ROOT after PAW GEANT 3 Objectivity era LCG projects start PI, SEAL, POOL ROOT mainstream PAW-like ROOT 1995 ROOT: design, development, evolution 2000 2005 5 2010 René Brun, ACAT 2010, Jaipur

Modular structure? Nice and modular building. But can we use only what we need? ROOT: design, development, evolution 6 René Brun, ACAT 2010, Jaipur

Project Structure xrootd NET HIST I/O Trees VMC reflex pyroot TMVA TGEO CORE CINT PROOF g 3 root Tfluka g 4 root MATH Roofit GRAF 3 D GRAF 2 D GUI EVE Roo. Stats ROOT: design, development, evolution 7 René Brun, ACAT 2010, Jaipur

framework Like in the Albert Hall Museum, we created base classes (the CORE), doors and windows (IO) and utility libraries on top. Can be seen as a mess or a very functional building. ROOT: design, development, evolution 8 René Brun, ACAT 2010, Jaipur

Interpreters In this talk, I have no time to discuss issues about interpreters like CINT and Pyroot, or the internals of the reflexion system (see my slide about LLVM later). CINT and its reflexion system is the cornerstone of ROOT for the IO system and scripts. CINT was not designed to be a full C++ interpreter. We came several years ago with ACLIC that automatizes the interface with the compiler. We strongly recommend the use of ACLIC for medium/large size scripts. LLVM will simplify even more the problem once the system will be operational. ROOT: design, development, evolution 9 René Brun, ACAT 2010, Jaipur

Interpreter & Compiler integration execute file script. C root >. x script. C execute function Do. Something root > Do. Something(…); root >. x script. C++ compile file script. C and execute it root >. x script. C+ compile file script. C if file has been modified. execute it g. ROOT->Process. Line(“. L script. C+”); g. ROOT->Process. Line(“Do. Something(…)”); ROOT: design, development, evolution 10 same from compiled or interpreted code René Brun, ACAT 2010, Jaipur

ROOT Math/Stat Libraries The new CERNLIB/MAT HLIB High Level Analysis Libraries Roo. Stats Roo. Fit TMVA Statistical Libraries Fitting and Minimization With better algorithms Quadp Minuit 2 Linear Fitter TMinuit TFumili Linear Algebra SMatrix Physics Vectors Gen. Vector TMatrix TVector 3/TLV Math. Core Limit classes MLP Extra Libraries Unuran FFTW Spectrum Foam Math. Mor e Random Numbers Math Interfaces Fitting Classes Extra algorithms Basic algorithms TComplex Extra Math functions Y TRandom GSL ROOT: design, development, evolution René Brun, ACAT 2010, Jaipur 11

ROOT 2 -D graphics based on 10 years experience with PAW and 15 years of operation with ROOT itself. zillions of tiny new features always requested by users works in batch (canvas. root, canvas. pdf, canvas. gif, . . ) object editors C++ script generated from canvas (TObject: : Save. Primitive) ROOT: design, development, evolution 12 René Brun, ACAT 2010, Jaipur

2 -D Graphics New functions added at each new release. Always requests for new styles and new ways to display data. ps, pdf, svg, gif, Move to GL ? jpg, png, c, root, etc ROOT: design, development, evolution 13 René Brun, ACAT 2010, Jaipur

ROOT: design, development, evolution 14 René Brun, ACAT 2010, Jaipur

Image processing ROOT: design, development, evolution 15 René Brun, ACAT 2010, Jaipur

GUI Portable GUI with all widgets found in modern UI toolkits. Many enhancements in the GUI classes: browser, html browser, TABs, EVE widgets, Fit. Panel. GUI builder with C++ code generator. Session recorder tool (class TRecorder) can store and replay a GUI session: All mouse events Keyboard input, including macro execution QT interfaces: a big pain, difficult to maintain with the successive versions of Qt. ROOT: design, development, evolution 16 René Brun, ACAT 2010, Jaipur

GUI ROOT: design, development, evolution 17 René Brun, ACAT 2010, Jaipur

GUI One of the most demanding areas Must follow technology and not standard systems (gks -> phigs >Motif ->Qt ->? ) GUI based on Open. GL too ! Plug-ins must work on the web (but 2 -d and 3 -d graphics too!) GUI builder extensions (code generators) GUI must be scriptable and theme-able HELP + language internalisation Context menus help access to documentation ROOT: design, development, evolution 18 René Brun, ACAT 2010, Jaipur

ROOT 3 -D graphics Open. GL based: graphics accelerators detector geometry visualization event displays (EVE) interaction between GL view and original objects (picking/editing) ROOT: design, development, evolution 19 René Brun, ACAT 2010, Jaipur

3 -D Graphics T P ad th e G L v i e w e r ROOT: design, development, evolution 20 Highly optimized GL views in in Detector hits Mathematical objects René Brun, ACAT 2010, Jaipur

Graphics (2 D-3 D) TH 3 TGLParametric “LEGO” ROOT: design, development, evolution “SURF” 21 TF 3 René Brun, ACAT 2010, Jaipur

The Geometry Package TGEO The TGeo classes are now stable (thanks ALICE). Can work with different simulation engines (G 3, G 4, Fluka) (See Virtual Monte Carlo) G 3 ->TGeo, G 4 ->TGeo, TGeo >GDML Used in online systems and reconstruction programs Built-in facilities for alignment Impressive gallery of experiments (35 detectors in $ROOTSYS/test/stress. Geometry) ROOT: design, development, evolution 22 René Brun, ACAT 2010, Jaipur

Open. GL see $ROOTSYS/tutorials/geom ROOT: design, development, evolution 23 René Brun, ACAT 2010, Jaipur

Event Display: EVE a GL-based package for event displays. Developed in collaboration with ALICE (Ali. Eve) and CMS (Fire. Works). Provides all the GUI widgets, browsers, GL infrastructure (far better than the old Open. Inventor). Used now by many experiments (see eg FAIRROOT, ILCROOT) to display raw data, MC events or detector oriented visualization. ROOT: design, development, evolution 24 René Brun, ACAT 2010, Jaipur

PROOF Massive data analysis with parallel systems. Now in production in many places. Growing success. A lot of developments for a better integration with dataset management systems and to support priority and accounting management. PROOF-LITE is a mini version to run on a multi-core. ROOT: one core only with PROOF-LITE with PROOF ROOT: design, development, evolution 25 René Brun, ACAT 2010, Jaipur

IO and Trees They are the most important components of the system because they have a deep impact on the experiment frameworks and their data handling strategy. Shaping these 2 systems has been a long process. ROOT: design, development, evolution 26 René Brun, ACAT 2010, Jaipur

Memory <--> Tree Each Node is a branch in the Tree Memory 0 1 T. Get. Entry(6) 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 T. Fill() T ROOT: design, development, evolution 27 René Brun, ACAT 2010, Jaipur

Automatic branch creation from object model branch buffers float a; int b; double c[5]; int N; float* x; //[N] float* y; //[N] Class 1 c 1; Class 2 c 2; //! Class 3 *c 3; std: : vector; std: : vector; TClones. Array *tc; ROOT: design, development, evolution 28 René Brun, ACAT 2010, Jaipur

branches created from a dynamic list Each named member of a TList Is a top level branch a a a b c d e ROOT: design, development, evolution 29 René Brun, ACAT 2010, Jaipur

Top level branches created by user Top level branches Are created manually a b c d e ROOT: design, development, evolution 30 René Brun, ACAT 2010, Jaipur

Branches created from STL collections Use split=199 to reduce The number of branches In case of polymorphic collections std: : vector split = 0 split >0 std: : vector split <100 std: : vector split =199 std: : vector TClones. Array ROOT: design, development, evolution split >0 31 René Brun, ACAT 2010, Jaipur

Trends in Tree structure Maximum split level, ie more branches is better eg, Atlas 9785, CMS 1436, LHCb 1167, Alice >1000 Many branches is good for the future because branch reading can be executed in parallel. many branches is better because the access granularity is better for fast selections. ROOT: design, development, evolution 32 René Brun, ACAT 2010, Jaipur

Object. Wise/Member. Wise Streaming 3 modes to stream an object a b c d i bject-w o er ff to a bu se member-wise to a ea ch a 1 a 2. . anb 1 b 2. . bnc 1 c 2. . cnd 1 d 2. . dn a 1 a 2…an m em be rt ROOT: design, development, evolution buffer a 1 b 1 c 1 d 1 a 2 b 2 c 2 d 2…anbncndn oa bu b 1 b 2…bn ffe r c 1 c 2…cn 33 d 1 d 2…dn member-wise gives better compression René Brun, ACAT 2010, Jaipur

Supported Platforms Linux (RH, SLC 4, SLC 5, Suse, Debian, Ubuntu) gcc 3. 4, …, gcc 4. 4 (32 and 64 bits) icc 10. 1, icc 11. 1 Mac. OS X (Intel/PPC, 10. 4, 10. 5, 10. 6) gcc 4. 0. 1, gcc 4. 2. 1, gcc 4. 4. 1, llvm-g++ icc 10. 1, icc 11. 1 Windows (XP, Vista) Support for many compilers adds robustness at a minor development cost. It facilitates the migration to new compiler versions. VC++7. 1, VC++9 Cygwin gcc 3. 4, 4. 3 Solaris + Open. Solaris CC 5. 2 gcc 3. 4, gcc 4. 4 ROOT: design, development, evolution 34 René Brun, ACAT 2010, Jaipur

Distribution Stats More than 20000 users 35000 FTP-only distributions of binaries per month. Many more from SVN source or AFS access ROOT: design, development, evolution 35 René Brun, ACAT 2010, Jaipur

How did we reach the current situation Rien ne sert de courir, il faut partir a point Jean de la Fontaine ROOT: design, development, evolution 36 René Brun, ACAT 2010, Jaipur

one cannot forget the past Systems like ROOT and G 4 have been strongly influenced par their predecessors, by politics, by sociology. Like Jumbo jets, large systems that required a lot of manpower evolve with time with the pros and cons of their inertia. Once a system is massively used, people expect backward and forward compatibility. However ROOT in 2010 is very much different of ROOT in 1995 or 2000 or 2005 ROOT: design, development, evolution 37 René Brun, ACAT 2010, Jaipur

Software in HEP most systems used over 2, 3 or 4 decades ROOT---========== JAS--===----…. . . geant 4 ----========== piaf==== paw---==========---------…. . zebra---=========------------……. . geant 3 ===========---------………. . . geant 1, 2 ==-hbook================------------…………. . . cernlib ========-------------………………. . . . 1970 1980 ROOT: design, development, evolution 1990 2010 38 René Brun, ACAT 2010, Jaipur

Main parameters who shaped ROOT Long experience with GEANT and PAW Reuse of components Concept of packages within a framework Optimal for batch, optimal for interactivity User support quasi online ROOT: design, development, evolution 39 René Brun, ACAT 2010, Jaipur

CERNLIB 1970: main. f links with cernlib. a where cernlib is a collection of independent subroutines. 1975: subroutines are grouped into packages identified by the letters of the alphabet, eg Y 250 1980: cernlib accepts large packages, eg hbook, minuit, gd 3 1985: cernlib is a collection of large packages, mathlib, kernlib, packlib, graflib, pawlib, geant 3 ROOT: design, development, evolution 40 René Brun, ACAT 2010, Jaipur

hbook : 1973 20 xy first large package in cernlib histogram-oriented system including visualization (from line-printer to graphics) and persistency. interface with Minuit and Fumili interface with zbook in 1975, then zebra in 1983 extended in 1985 to support row-wise ntuples and columnwise ntuples in 1990. the heart of PAW ROOT: design, development, evolution 41 René Brun, ACAT 2010, Jaipur

geant 3 : 1980 - 20 xy successor of geant 1 (1975), geant 2(1978) includes a powerful geometry package used to model all existing HEP detectors. interfaces with external hadronic shower packages like tatina, gheisha, fluka. includes visualisation: detector cross-sections, event displays. includes IO for geometry and events (zebra) ROOT: design, development, evolution 42 René Brun, ACAT 2010, Jaipur

PAW : 1984 20 xy started by a committee (design for the past) quickly reshaped in 1985 with KUIP, HIGZ, ntuples query processor: ntuple/plot 10. px analysis patterns indicates row-wise column-wise storage and access. developments using zebra RZ IO system more and more graphics options ROOT: design, development, evolution 43 René Brun, ACAT 2010, Jaipur

PIAF : 1992 -1994 PAW Parallel Interactive Analysis Facility very important step to understand the issues with parallelism with the IO sub system and the implications on the users analysis code. very important to understand the balance between CPU, disk IO, network IO. ROOT: design, development, evolution 44 René Brun, ACAT 2010, Jaipur

software crisis 1992 -1994 It was clear in 1992 that our Fortran-based systems will be impacted by new languages and new techniques. Many people thought that Fortran was the future. MOOSE project investigating languages (C++, Eiffel) and techniques (UML, Rose, OO-design) commercial software or home-grown dilemma OODBMS at the horizon ROOT: design, development, evolution 45 René Brun, ACAT 2010, Jaipur

software crisis 1992 -1994 -more many computer scientists in the discussion most of them with no experience existing experience with large software neglected as well as feedback coming from experienced physicists west coast syndrome Babar as guinea pig (C++ and OODBMS) Revolution, not Evolution ROOT: design, development, evolution 46 René Brun, ACAT 2010, Jaipur

Towards Objectivity 1994 -2000 Gurus predict that OODBMS will replace RDBMS systems in 2000. Post-mortem influence of defunct SSC dream of a central data base accessed world-wide at the bit level for writing and reading. Don’t worry about containers structure, CPU and network performance; the technology will solve your problems. Most physicists trust gurus for reaching Nirvana ROOT: design, development, evolution 47 René Brun, ACAT 2010, Jaipur

1994 -2000 Lost in the clouds before the grid and the clouds Experienced physicists busy with LHC exp design. It took many years (of frustration) to move them from good old Fortran to the magic of OO and C++ catastrophic LCB (LHC Computing Board) committee with always the same referees with no experience in the new technology. As a consequence the LCB has no choice but trusting the gurus. Despite early signs of problems, gurus and referees persist in their recommendations. ROOT: design, development, evolution 48 René Brun, ACAT 2010, Jaipur

1995 -1998 ROOT is born in the meadows While many people were actively working in committees and expecting magic from Objectivity, a small group of anarchists is exiled in the CERN 2 meadows. free of committees and not spending time only with UML diagrams , but concentrating their work on key issues and delivering frequent releases of prototypes tested in real life experiments (eg NA 49). While the first ROOT prototype was a C++ version of PAW, it was rapidly realized that the key problem was with a general object persistency solution much different than the Objectivity solution. ROOT: design, development, evolution 49 René Brun, ACAT 2010, Jaipur

Original Objectives in 95 Continue the tradition started with CERNLIB to develop packages targeted to the needs of our experiments and with a long life time. Eg hbook (36 years), geant 3 (28), paw(25): still in use! Focus on data analysis (PAW in the oo world) Open Source model to get contributions and feedback as soon as possible. First ROOT prototype for NA 49 in 1995, rapidly followed by many more adopters. Bottom-up (user-driven) instead of top-down (committee) approach. Users workshops. ROOT: design, development, evolution 50 René Brun, ACAT 2010, Jaipur

ROOT in 1995 During the first 3 months of 1995 we prototyped several versions of the histograming system, including a version based on templates. In October 1995 we presented a first prototype with histograms, functions, graphics (a la PAW). This prototype included a command line interface supporting very primitive calls to ROOT classes member functions. This prototype generated a lot of attention but also fears. ROOT: design, development, evolution 51 René Brun, ACAT 2010, Jaipur

ROOT in 1996 Instead of our very basic interpreter, we decided to use CINT developed since 1991 by Masa Goto from HP Japan. We used CINT not only as an interpreter, but also as a parser for the ROOT header files to generate automatically a dictionary such that: one could call class member functions from C++ interpreted scripts one could stream objects to a buffer with explicit streamers. ROOT version 1 included also many more classes and better graphics. ROOT: design, development, evolution 52 René Brun, ACAT 2010, Jaipur

ROOT in 1997 The IO Streamer functions generated by rootcint were quite simple to understand were efficient. However, we realized soon that classes were evolving. We had to introduce class versions and the possibility for users to write their own custom streamers where they could take into account the class schema evolution. At the same time, we made rootcint more general such that it could parse many more classes, in particular non. ROOT classes developed by our experiments guinea-pigs. ROOT: design, development, evolution 53 René Brun, ACAT 2010, Jaipur

ROOT in 1998 We released version 2 with more features (IO, math, graphics and more sophistication in Trees, our main container for event data. In fall 1998 ROOT was selected by FNAL & RHIC as: the persistency solution the data analysis solution This decision generated a big chock-wave in the establishment. It also generated more work for us. ROOT: design, development, evolution 54 René Brun, ACAT 2010, Jaipur

ROOT in 1999 The ROOT Streamers were very efficient. However the library containing the Streamers was required when reading the data sets. While this requirement was not a problem when reading complex object collections and the classes used to write were necessary in the read phase, it was a serious drawback for files with simple Trees like PAW ntuples. At this point we implemented an automatic system to stream data using the information in the dictionary instead of using the generated C++ Streamers ROOT: design, development, evolution 55 René Brun, ACAT 2010, Jaipur

Streamer. Info With the automatic streaming, we also saved to the ROOT file the meta data information describing all classes such that ROOT files are now self-describing. With time and in the following years we continued to optimize this system: to support the full C++ language in general. to support more sophisticated cases of class schema evolution. ROOT: design, development, evolution 56 René Brun, ACAT 2010, Jaipur

ROOT vs Objectivity In 99. 99% cases people write a file once and read it many times. No need for a locking process when reading. ROOT files are self-described and independent of the location where they are processed. ROOT files are compressed. Typical compression factors for LHC experiments is between 3 and 5. ROOT files are machine independent ROOT: design, development, evolution 57 René Brun, ACAT 2010, Jaipur

ROOT vs Objectivity 2 ROOT does not stream transient data members ROOT can stream collections member-wise, optimizing both speed and disk space. ROOT can read files containing different class versions: automatic class schema evolution. ROOT has a complete query/visualization system. ROOT files can be processed in parallel with PROOF ROOT was processing files orders of magnitude faster ROOT has a powerful caching mechanism allowing to read files very efficiently in wide area networks ROOT: design, development, evolution 58 René Brun, ACAT 2010, Jaipur

Directions ROOT: design, development, evolution 59 René Brun, ACAT 2010, Jaipur

Our Challenges Keep a stable core, improving robustness At the same time follow the technology changes Distributed and parallel computing Multi-core systems, GPU processing (Open. CL) New version of C++, etc CINT to LLVM Improve algorithms (quality and speed) Improve doc , user interface, user support Add support for new communities, LC, etc And, of course, provide a robust and efficient data analysis tool for the LHC experiments. ROOT: design, development, evolution 60 René Brun, ACAT 2010, Jaipur

Problem decomposition Will have to deal with many shared libs Only a small fraction of code used ROOT: design, development, evolution 61 René Brun, ACAT 2010, Jaipur

Some Facts 100 shared libs 2000 classes 10 shared libs 200 classes ROOT in 2010 ROOT in 1995 PAW model Plug-in manager ROOT: design, development, evolution 62 René Brun, ACAT 2010, Jaipur

Executable module in 2000 many_x. f -> many_x. o + some_libs. a Input. dat (free format) + many_libs. so -> x. exe Zebra file RFIO x. exe Output. log Objectivity? ROOT ? ROOT: design, development, evolution 63 René Brun, ACAT 2010, Jaipur

Executable module in 2010 Shared libs dynamically loaded/unloaded u. so by the plug-in manager a. so ROOT files Dcache castor d root x b. so Config. C (interpreter) x. exe Output. log LAN ROOT files ROOT: design, development, evolution Oracle Mysql 64 René Brun, ACAT 2010, Jaipur

Executable module in 2020 ? Local shared libs dynamically Compiled/loaded/unloaded http from a source URL u. cxx http b. cxx a. cxx ROOT files WAN he Cac y x Pro er ag man ROOT files ROOT: design, development, evolution x. exe Config. C (interpreter) Multi-threaded Core executor Output. log ROOT buffers local cache Oracle Mysql 65 René Brun, ACAT 2010, Jaipur

The LLVM system We are having big hopes with LLVM (open source project with people from Apple, Adobe, Google, etc). LLVM is a gcc compatible compiler, parser and JIT (Just In Time) compiler. Our idea is that the interpreter will be the compiler itself (C++0 x compliant) and offering many additional possibilities like quality and speed for important classes like TFormula, TTree. Formula, etc We have a working prototype today, but a fully operational use of LLVM in ROOT is probably 2 years away. ROOT: design, development, evolution 66 René Brun, ACAT 2010, Jaipur

Design for Parallelism The GRID is a parallel engine. However it is unlikely that you will use the GRID software on your 32 -core laptop. There is a lot of work to be done within ROOT to use parallelism internally or making ROOT-based applications more easily parallelizable. Think Top->Down and Bottom->Up Fine grain Coarse grain: job, event, track ROOT: design, development, evolution vectorization 67 René Brun, ACAT 2010, Jaipur

Towards a distributed system 5, 000 physicists 100, 000 computers in 1000 locations LAN N WA ROOT: design, development, evolution 68 René Brun, ACAT 2010, Jaipur

GRID: Users profile Few big users submitting many long jobs (Monte Carlo, reconstruction) They want to run many jobs in one week Many users submitting many short jobs (physics analysis) They want to run many jobs in one hour or less ROOT: design, development, evolution 69 René Brun, ACAT 2010, Jaipur

Difficult to fit all sizes Like the Taj Mahal, ROOT has many different facets and the visitor may have different impressions depending on his viewpoint and time of day. We try to do our best to satisfy very heterogeneous users. Some use ROOT as a framework, others as a simple tool, others as a set of libraries. ROOT: design, development, evolution 70 René Brun, ACAT 2010, Jaipur

Taj Mahal & ROOT The Taj Mahal combines elements from Persian, Indian, Islamic architectural styles, an integrated complex of structures. Building began around 1632 and was completed around 1653, and employed thousands of artisans and craftsmen. ROOT combines ideas and techniques that proved to be successful in the geant 3, zebra and PAW eras. Building began in 1995 and in 2010 is still under construction by many motivated people. ROOT: design, development, evolution 71 René Brun, ACAT 2010, Jaipur

Many Thanks I have been pleased to follow the ACAT meetings since the beginning in Lyon (1989 ? ) Many thanks to my colleagues organizing this event. This is my last talk at ACAT as ROOT project leader. I take this opportunity to thank all my colleagues who are working hard in this project and I wish them a healthy future. ROOT: design, development, evolution 72 René Brun, ACAT 2010, Jaipur

Input/Output: Major Steps User written streamers filling TBuffer member-wise streaming for STL collections streamers generated by rootcint automatic streamers from dictionary with Streamer. Infos in self-describing files generalized schema evolution parallel merge member-wise streaming for TClones. Array ROOT: design, development, evolution 73 René Brun, ACAT 2010, Jaipur

Major Technical Steps I/O based on dictionaries in memory Graphics based on TVirtual. X CINT Reflex From SEAL to ROOT GUI based on signal/slo ts Math. Libs Trees automatic split Roo. Fit TMVA PROO F GL/EVE CINT 7 LLVM CMZ…………………CVS……………………………. SVN…………… ROOT: design, development, evolution 74 René Brun, ACAT 2010, Jaipur

Testing, Nightly builds Several thousand tests run every night on many platforms. Code checking with a coding convention rule checker Code coverage to be improved, but already much better than any other package reviewed. Frequent valgrinding and profiling ROOT: design, development, evolution 75 René Brun, ACAT 2010, Jaipur

PROOF-LITE PROOF optimized for single many-core machines Zero configuration setup. No config files and no daemons Workers are processes and not threads for robustness Like PROOF it can exploit fast disks, SSD’s, lots of RAM, fast networks and fast CPU’s Once your analysis runs on PROOF Lite it will also run on PROOF (with exactly the same user code) ROOT: one core only ROOT: design, development, evolution with PROOF-LITE 76 René Brun, ACAT 2010, Jaipur

Executable module in 1990 many_x. f -> many_x. o + many_libs. a -> x. exe Input. dat (free format) x. exe Output. log portable Zebra file ROOT: design, development, evolution 77 René Brun, ACAT 2010, Jaipur