ETICS build and test system Marian ZUREK on

ETICS build and test system Marian ZUREK on behalf of ETICS Team www. eu-etics. org ICT-RI- ETICS build and test system

Contents • • • ICT-RI- Introduction ETICS Features Architecture or “how we do it”? ETICS test scenario ETICS Portal Command Line Interface Some facts and numbers Future Plans Conclusions Q&A ETICS build and test system 2

ETICS in a nutshell • ETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software • EU-founded Project started in January 2006 foreseen to last at least till February 2010. Future plans to transform into European sustainable infrastructure. • Goal is to extend and improve automated tools for building and testing as already developed since the beginning ICT-RI- ETICS build and test system 3

The ETICS Partners ICT-RI- ETICS build and test system 4

ETICS Features Summary • We provide service for build and test purposes • We manage automatically dependencies providing you with rich library of external dependencies you may require for building your software (4 Suite, axis, Condor, VDT, jdk, etc. ) • We offer infrastructure covering large number of hardware/OS combinations • We produce native packages for you in a automated way (rpms, tar, msi, debs) and store them in the ETICS repository together with rich build and test logs • We offer the possibility of checking your code against coding conventions, run your tests, check documentation • We offer the possibility of tests in the root enabled environment ICT-RI- ETICS build and test system 5

ETICS Features Summary • We offer the build from sources or using pre-build binary packages • We offer the extensibility mechanism based on plugin to perform code analysis: Users are able to contribute with their plugins or use already existing set: Py. Unit, Junit, Cpp. Unit, emma, jdepend, gcov, slocccount, IPv 6 compliance, bug tracking connectors, etc. • We offer support for the co-scheduled distributed multi-node test. How? ICT-RI- ETICS build and test system 6

Architecture or how we do it? ETICS Portal (My. ETICS, Configuration, Administration, Repository, Requests) Web Service Repository Service Via browser Report Build/Test Project Artefacts DB DB Metronome Execution Engine Clients WNs ICT-RI- Metronome Client Wrapper ETICS Infrastructure ETICS build and test system 7

ETICS test scenario • We offer support for the co-scheduled distributed multinode test. How? • A) You need to define in ETICS system a configuration describing: (1) software you want to test and its dependencies, (2) number and type of platforms to run (3) flow of actions (script) for each node including synchronisation constrains if needed • B) Submit it: ETICS processes the configuration information, generates NMI scripts and using Condor launches the execution of your test scenario on the testbed. • C) Analyse results: The produced artefacts (logs, reports) are stored in the ETICS repository • Thanks to the persistent storage you are able to repeat your test scenario at any time (B and C of course as A is already in the configuration database) ICT-RI- ETICS build and test system 8

ETICS Portal • ETICS Portal is a concentration point of ETICS Web Applications (WA): – My. ETICS - start-up page: latest news and announcements, user rights, access to recent build, useful links – Configuration WA to manage configurations – Administration WA to manage access rights – Repository WA to access build/test reports and build artefacts (packages) – Requests WA to submit an external dependency request • The Portal offers secure (X. 509) role-based access to the ETICS Services. ICT-RI- ETICS build and test system 9

ETICS Portal ICT-RI- ETICS build and test system 10

ETICS Portal ICT-RI- ETICS build and test system 11

Web Application ICT-RI- ETICS build and test system 12

Web Application ICT-RI- ETICS build and test system 13

Configuration WA • The Web Application (WA) offers to users the possibility of: – browsing configurations – editing configurations – static and dynamic dependencies definition on the top of dependency scope (build-time, run-time or both) – remote build and test submission of configurations ICT-RI- ETICS build and test system 14

Repository WA • Repository WA offers the possibility of: – – ICT-RI- Accessing build products (packages), logs and build/test results Storing permanently and/or temporarily the artefacts Manage (upload/download) the artefacts Build the customized and federated repositories (including connectors for popular repositories like Source. Forge) ETICS build and test system 15

Repository ICT-RI- ETICS build and test system 16

Reports ICT-RI- ETICS build and test system 17

Administration WA - Security Access to ETICS services is based on X 509 certificates and fine-grained user roles: • Developers: can do local builds of project, subsystems and components • Integrators: can create and run integration configurations • Testers: can register and submit tests attached to configurations • Release Managers: can create and publish configurations, create release notes and other documentation • Project Managers: can create and manage projects, assign permissions, set policies, etc The administration part is fully delegated to project owners/administrators, so no contact with ETICS support is needed. ICT-RI- ETICS build and test system 18

Command Line Interface ICT-RI- ETICS build and test system 19

Command Line Interface • All operation possible with WA can be performed using the ETICS CLI • One can use the API for scripting complex/test scenarios • Security based on X. 509 certificates and roles • One can build locally – The same operations performed on the local node will be executed remotely – context swapping using the concept of workspaces (sandboxes) to facilitate working on multiple builds/tests in an isolated environment ICT-RI- ETICS build and test system 20

Some facts • 28 projects already registered (some very active, some performing experiments) • More than 220 users registered • More than 70’ 000 jobs executed on the ETICS production nodes • Around 200 test/builds per day • Our biggest client is EGEE with g. Lite software stack (around 380 modules) • Continuous integration is the key aspect for early faults/bugs detection ICT-RI- ETICS build and test system 21

ETICS sites • CERN – SLC 3, SLC 4, RHES, Debian, FC (32 and 64 -bits), Win. XP, Mac OSX • Univ. of Wisconsin, Madison, USA – HP HPUX, IBM AIX, OSF 1, SGI Irix, Sun Solaris, Linux (several flavours) • INFN, Bologna, Italy – Cent. OS, SLC 3, SLC 4 (32 - and 64 -bits) Job migration between CERN, Uo. W and INFN ICT-RI- ETICS build and test system 22

Future plans • Extension of the ETICS services to more infrastructures • Federation and customization of resources (use of custom VMs, different types of repository) • Promotion of the proper software certification process that project might want to adopt in order to assess their products - we have developed the Grid QCM which allows to measure in the practical way the quality of distributed software projects ICT-RI- ETICS build and test system 23

Conclusions • Collaboration with Uo. W Condor+NMI B&T Lab was very successful, constructive and essential to the success of the ETICS Project. • In just 2 years ETICS became the default test and build infrastructure for many projects in Europe. Thank You !!! ICT-RI- ETICS build and test system 24

Q&A http: //eu-etics. org etics-support@cern. ch https: //etics. cern. ch/etics. Portal http: //etics. cern. ch/nmi Marian. ZUREK@cern. ch ICT-RI- ETICS build and test system 25