INFSO-RI-223782 Support for IPv 6 in ETICS EGEE

INFSO-RI-223782 Support for IPv 6 in ETICS EGEE’ 08 Conference, Istanbul, 22 -26 September 2008 Marian ZUREK CERN - ETICS Marian. ZUREK@cern. ch

Contents • • • INFSO-RI-223782 General overview Who/what we are: Partners/Architecture ETICS Infrastructure IPv 6 compliance (IPv 6 plugin) IPv 6 Testbed Examples of usage ETICS plugin framework Future work Conclusions 2

General Overview INFSO-RI-223782 • ETICS stands for E-infrastructure for Testing, Integration and Configuration of Software • ETICS started in January 2006 and ended in December 2007. It had a very successful review exceeding all stated objectives • ETICS 2 started in March 2008 (ranked first in the proposal selection process) and it’s supposed to run until February 2010 (mostly with EC funding) and then on its own • ETICS is not ‘just’ a build system, it’s a complete infrastructure for building, testing, configuring and managing software projects 3

Partners INFSO-RI-223782 4

Architecture INFSO-RI-223782 5

What ETICS is INFSO-RI-223782 • It’s a software engineering management system • It’s a build and test infrastructure • It provides tools and resources to configure, manage and analyse build and test runs • It provides a common interface to diverse projects to facilitate knowledge sharing and operations management • It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information • It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions • It’s multi-platform and independent from any specific build or test tool 6

ETICS Users INFSO-RI-223782 • EGEE uses ETICS within four Activities: NA 4, JRA 1, SA 2, SA 3 • JRA 1 and SA 3 are the main ETICS users • SA 2 has implemented with ETICS the IPv 6 compliance analysis of g. Lite code and the distributed IPv 6 experimental testbed where the IPv 6 version of BDII (and more) has been tested • NA 4 is using ETICS for experimenting with a few projects like Gridway, mpi, SAGA and the panc quattor compiler 7

The ETICS Infrastructure INFSO-RI-223782 • The ETICS Infrastructure is currently based on three resource pools managed by Condor, at CERN, Uo. W and INFN • There about 450 cores available in the three sites with more than 40 different types of platforms • At CERN in particular there are 120 cores and 12 different platforms (a platform is a combination of operating system, CPU architecture and compiler, ex: slc 4_ia 32_gcc 36, deb 4_x 86_64_gcc 412, etc) • Additionally various nodes are attached to the CERN pool from third-party projects or organizations for specific use (4 D Soft Ltd for DILIGENT in Hungary, GARR for EUChina. Grid and EGEE/SA 2, IN 2 P 3 for EGEE/SA 2) 8

The ETICS Infrastructure INFSO-RI-223782 9

The CERN Resource Pool INFSO-RI-223782 10

More about ETICS … INFSO-RI-223782 Please join us on Thursday, the 25 th of September in the Malazgirt Hall between 16: 00 and 19: 00. 11

IPv 6 support in ETICS INFSO-RI-223782 • As a consequence of task defined in SA 2/EGEE (EU requirement) for • assuring grid interoperability • integrate sites/countries lacking public IPv 4 address • ETICS has been contacted as a partner in this task 12

Under the hood or what’s needed for IPv 6 tests INFSO-RI-223782 A dedicated Condor configuration has been adopted to assure the proper match making for jobs requesting IPv 6 resources (not visible to the user) host_network_stack = "IPv 6” STARTD_EXPRS = $(STARTD_EXPRS) host_network_stack START = (job_network_stack =? = $(host_network_stack)) “extra” part in the submision engine (not visible to user) append_requirements = ( host_network_stack =? = "IPv 6" ) ++job_network_stack = "IPv 6” 13

Is your code IPv 6 compliant? INFSO-RI-223782 For assessing the source code compliance a ETICS IPv 6 Plugin has been developed and is present in the main-stream ETICS distribution Language coverage: Survey under preparation for the most common languages: C, C++, Java, Python, PERL Disclaimer: The passive check of the source code by the means of IPv 6 plug-in presents assessment of the IPv 6 compliance. Only the run-time tests could confirm the 100% compliance of your code. More on that - please see talks by Mario & Etienne. Please keep that in mind when watching coming slides. ETICS@PSM - 3 July 2008 14

Is your code IPv 6 compliant? INFSO-RI-223782

Running tests on IPv 6 enabled resources INFSO-RI-223782

INFSO-RI-223782 Where to look for g. Lite IPv 6 compliance reports https: //eticsrepository. cern. ch: 8443/repository/download/volatile/gli te_3_1_0/Overall. Reports/org. glite/3. 1. 01/slc 4_x 86_64_gcc 346/reports. tar. gz$reports/ipv 6/inde x. html 17

IPv 6 compliance report - ETICS INFSO-RI-223782

IPv 6 compliance report - g. Lite INFSO-RI-223782

IPv 6 compliance report - g. Lite/LCG-DM INFSO-RI-223782

Plugins and Metrics Collectors INFSO-RI-223782 • The ETICS system is plugin-based (like for example ECLIPSE) • It provides a core set of tools (like IPv 6) and a published specification for developing additional plugins • Plugins are essentially thin wrappers around existing or custom tools providing very specific functionality like packaging, static or dynamic testing, standards compliance testing, service installation and management, reporting, etc • Plugins can publish information in the ETICS system in the form of metrics, which can then be used to do data mining or trend analysis using the available ETICS reporting tools • Should another plugin be needed - it can be easily added to ETICS 21

INFSO-RI-223782 Examples of metrics collectors Metrics Type Programming languages/ technologies Complexity static Java Python Javancss JCcn. Plugin Py. Complexity. Plugin. py Design quality static Java Jdepend JDepend. Plugin N of potential bugs static C/C++ Python Perl PHP Java Flawfinder, RATS PMD Findbugs CFlawfinder. Plugin CPy. Php. Rats. Plugin JPmd. Plugin JFindbugs. Plugin N of potential bugs dynamic C/C++ Valgrind CValgrind. Plugin static All SLOCCount. Plugin Coverage dynamic Java Emma Cobertura JUnitemma. Plugin JCobertura. Plugin Unit tests success rate dynamic Java Python JUnit Py. Unit JUnit. Plugin JUnitreports. Plugin. py Py. Unit. Plugin. py static IPv 6 WSI dynamic C/C++ Java Lines of code Compliance with standards Profiling Tool ETICS Plugin IPv 6 Plugin WSInteroperability. Plugi n Jrat Valgrind JRat. Plugin CValgrind. Plugin 22

Fixing your code – what it takes INFSO-RI-223782 IPv 6 related issues could be quickly and easily fixed • JAVA applications are already IPv 6 compliant (proper system parameters must be set) - examples: /proc/sys/net/ipv 6/bindv 6 only if set to 1 will not accept IPv 6 calls • e. g. only changes in 9 (nine) lines were necessary to make BDII IPv 6 compliant 23

An example IPv 6 programming: a dual stack server BDII INFSO-RI-223782 • The first version, an IPv 6 server only, that opens one socket using IPv 4 mapped address, is easier to develop: • Two files and few lines were patched • Only 4 lines in “bdii-fwd” were patched: [root@quarks sbin]# diff bdii-fwd. map. . /sbin. sav/bdii-fwd 10, 11 d 9 < use Socket 6; < use IO: : Socket: : INET 6; # UREC/CNRS EGEE-SA 2 xj 61 c 59 < $proxy_server = IO: : Socket: : INET 6 ->new(@proxy_server_config) > $proxy_server = IO: : Socket: : INET->new(@proxy_server_config) 169 c 167 < $remote_server = IO: : Socket: : INET 6 ->new(@rs_config) > $remote_server = IO: : Socket: : INET->new(@rs_config) 24

An example IPv 6 programming: a dual stack server BDII • Only INFSO-RI-223782 5 lines in “bdii-update”: [root@quarks sbin]# diff bdii-update. map. . /sbin. sav/bdiiupdate 17 d 16 < use IO: : Socket: : INET 6; # UREC CNRS EGEE SA 2 xj 416 c 415 < my $s = IO: : Socket: : INET 6 ->new(@port 2 skip); > my $s = IO: : Socket: : INET->new(@port 2 skip); 445 c 444 < my $guard = IO: : Socket: : INET 6 ->new(@port 2 keep); > my $guard = IO: : Socket: : INET->new(@port 2 keep); 570 c 569 < my $s = IO: : Socket: : INET 6 ->new(@port 2 skip); > my $s = IO: : Socket: : INET->new(@port 2 skip); 639 c 638 < system("$bdii_fwd_prog --local : : --service $bdii_port_read" > system("$bdii_fwd_prog --local 0. 0 --service $bdii_port_read" 25

Lesson learned INFSO-RI-223782 Taking into account the site policies, building the Condor pool out of geographically dispersed Worker Nodes isn’t a trivial task as it may appear. Numerous issues have been identified and adressed: • Reverse DNS: Solved by hard-coding the IP addresses in the Condor config files • Firewall policies (procedures, transitional openings) • Condor: Job Match-making: partial match was causing the ETICS production pool throughput degradation • Power-cuts, etc. Mail, IM, SMS, phone were very useful 26

Latest development INFSO-RI-223782 I am leaving floor to Etienne who will be talking about extension to IPv 6 code checker. We call it “LD_PRELOAD ipv 6 checker”. The discussion will cover technique used and also summarise its applicability. 27

Not enough? INFSO-RI-223782 A dedicated tutorial is planned to happen during the JRA 1 All hands meeting. For those not attending - please refer to our talks and references listed in handouts ETICS@PSM - 3 July 2008 28

INFSO-RI-223782 Next Steps • Keep the testbed operational • Store working configurations tests/deployments inside ETICS and setup automated • Make sure the ETICS service/infrastructure helps you in achieving your (IPv 6 and other) goals (= provide continuous support to the community ) • Your ideas – how would you like to proceed with your IPv 6 compliance development and test • we have a manually setup testbed – do we keep it operational or rather deploy the whole infrastructure automatically using ETICS • Suggestions, ideas, complains …. . 29

Conclusions INFSO-RI-223782 • ETICS in collaboration with GARR and UREC is providing the infrastructure enabling the community to perform the IPv 6 tests • Automated deployment mechanisms should be established to facilitate IPv 6 compliance validation test • We plan to continue the collaboration 30

INFSO-RI-223782 The End Q&A 31

Benefits of Using ETICS INFSO-RI-223782 • Common set of interfaces and tools to different projects • Simplifies working with different projects and transferring knowledge among technical people • Configuration information is stored in a single place with a common schema and preserved while people or projects change • Multi-platform support allow building and testing the same code-base on as many platforms as needed with negligible additional effort • The resource infrastructure allows running several builds and tests without having to set up dedicated machines • All machines are configured in the same way or use virtual images in order to guarantee reproducibility of results • Standard build and test tools are available out-of-the-box and can be applied in the same way to different build/test runs and different projects over time 32

Benefits of Using ETICS (2) INFSO-RI-223782 • The built-in packaging system abstracts the package information and automatically builds the appropriate package format for the different platforms (tarballs, RPMS, debs, MSIs, etc). No need to maintain separate scripts • Reporting tools are available out-of-the-box to control the software evolution, pinpoint issues and quickly solve problems • APIs and plugins allow to integrate the tools into existing development processes and extend them with custom tests or reports as needed • New plugins and tools developed by a project can be of benefit to all other projects (no duplication of effort) • ETICS provides a large set of external libraries in source format or already precompiled for many platforms (the ‘externals’ project). Components can set dependencies on these libraries and automatically configure their workspace. 33

The ‘Distributed Testing’ Feature INFSO-RI-223782 • One of the last features to be added, still in experimental mode • It allows designing complex tests involving several services and test applications to be deployed on separate nodes • ETICS analyses the definition and deploys the services on the necessary nodes • A synchronization mechanism orchestrate the start/stop of services and the execution of the test applications • At the end the results are collected and reported as a single report • It is not yet usable by ‘any user’, it requires some deep knowledge of the system to be tested • It has been successfully used in a number of cases by the DILIGENT project • It has the strong prerequisite that the services to be deployed have to be managed without user intervention, which is not always the case 34

How to fix your code INFSO-RI-223782 • Small hint here …. • This will be covered in a short tutorial by Mario • Another presentation will be given during JRA 1 All Hand meeting in …. dates/place TBConfirmed 35