Condor for building with NMI and ETICS Marian

Condor for building with NMI and ETICS Marian ZUREK www. eu-etics. org Europeean Condor Week 2006 Milano, June 26 -29, 2006

Overview • • • What is ETICS What is NMI Where is Condor NMI architecture NMI job Hello. World example ETICS Testbed What we do using NMI & Condor g. Lite testing requirements Conclusions Wish list Q&A Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 2

What is ETICS • ETICS stands for e-Infrastructure for Testing, Integration and Configuration of Software • It’s an SSA • Vision: A dependable, reliable, stable grid infrastructure requires high-quality, thoroughly tested, interoperable software middleware and applications • Mission: Provide a generic service that other projects can use to efficiently and easily build and test their grid and distributed software. • Set up the foundations for a certification process to help increasing the quality and interoperability of such software Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 3

The ETICS Consortium Build system, software configuration, service infrastructure, dissemination, EGEE, g. Lite, project coord. The Condor batch system, distributed testing tools, service infrastructure, NMI Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 4

ETICS Service Overview Web Application Web Service Via browser Report DB Command. Line tools Project DB Build/Test Artefacts NMI Scheduler Clients WNs Continuous Builds NMI Client Wrapper ETICS Infrastructure Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 5

Browsing a project Module Details Project Structure Module Configurations Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 6

Browsing configurations Module Details Configuration Browser Conf Details Sub-Configurations Commands Properties Environment Dependencies Build Schedules Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 7

Command line tools • Command Line Interface (CLI) is developped along with the Web. Application and offers the same functionality as Web. Application (they use the same underlying web service libraries) • Written in Python – – – Using ZSI CVS-style command syntax etics-* (etics-get-project, etics-checkout, …) • The CLIs are directly used for NMI/Condor job/tests submission • etics-get-project Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 8

List of Commands: Submit • etics-remote-build [options] –c <configuration-name> -p <platform-namelist> -m <project-name> – It performs a remote build for an ETICS project – The remote build consists on submitting a request to an external build system that uses the ETICS infrastructure to build – By default, the command remotely builds the current project by using the current configuration and platform. – It returns an ID that will be used to get the status build – -p, --platforms – It specifies the list of platforms where to build – -m, --project – It specifies the project name to build – -c --configuration – It specifies the configuration name to be used to build – --all-platforms – It means that the remote build is performed on all platforms Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 9

CLI - example What is under the hood ? • cvs co org. etics (etics-get-project, etics-checkout) • . /configure; make dist | test | install (etics-build) Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 10

What is Condor and NMI • NMI is a multi-platform facility designed to provide (automated) software building and testing services for a variety of (grid) computing projects. • NMI is a layer on the top of Condor to abstract the typical complexity of the Build and Test process • Condor is offering mechanisms and policies that support High Throughput Computing (HTC) on large collections of distributed computing resources Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 11

NMI architecture Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 12

NMI job • a set of user commands/scripts representing the workflow of the build or test producing the output (optional) Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 13

NMI job Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 14

Hello. World. nmi Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 15

Hello. World. cvs Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 16

Dedicated UI WN Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 17

ETICS testbed • etics. cern. ch – Official ETICS submission node - production host – 2000+ jobs (as of 24 th June 2006) • etics-test. cern. ch, etics-01. cnaf. infn. it – test submission nodes • etics-dev. cern. ch – development node – non-stable • etics-db. cern. ch – My. SQL database host Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 18

etics. cern. ch pool • ia 32 – – – – 6 x SLC 3 2 x SLC 4 1 x RHES v 3. 0 1 x Debian v 3. 1 1 x FC 3 1 x FC 4 1 x Win. XP • ia 64/x 86_64 – – – 2 x SLC 3 1 x SLC 4 1 x PPC Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 19

Production • Operational setup – WNs status page http: //etics. cern. ch/nmi/? page=pool/index http: //etics-test. cern. ch/nmi/? page=pool/index – Job submission status page http: //etics. cern. ch/nmi/? page=results/overview http: //etics-test. cern. ch/nmi/? page=results/overview Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 20

Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 21

Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 22

• What we use NMI & Condor for ? Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 23

What we want to do • our final goal is to test the current g. Lite components with ETICS. We’d like to : – automatically deploy the full set of g. Lite services on different platforms – automatically have our full set of tests executed periodically – set explicit dependencies for the tests on other tests or specific middleware components: – « execute test B only is test A was successful » – « install component C 1 and C 2 if required by test A » – – – easily access the test results easily debug test failures easily reproduce a test result from a specific configuration ==> for that we need reliably scheduling and executing infrastructure Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 24

g. Lite testing requirements • Dependency on UI (User Interface) – UI installation required for each test – Time impact --> dedicated UI Worker Node – append_requirements = ( Machine == “lxb 1115. cern. ch”) • Proxy forwarding – Once proxy exists on the submit node NMI/Condor can forward it to the target WN • Freeze option – Shoud the test fail I want this node to leave condor pool and I want to debug the failing component(s) ==> we need very flexible environment Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 25

Root-enabled installations/tests • Some g. Lite tests require root environment • System test require services persistency – Some services after installation should remain operational for the predefined period of time • Configuration of the Condor running in the priviliged mode – sudo, STARTd configuration – ==> not the preferred operational mode, but possible (very close to the production quality) Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 26

Public/private pools • A project wants to use ETICS and brings in its private nodes and wants its full power to be private • Steering the jobs to this node, preventing from others landing there • ==> already implemented and operational Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 27

EGEE 2 and Other Collaborations • The ETICS services are intended to provide from the start the integration, testing and QA tools for EGEE/EGEE 2, Diligent and other partner projects • Part of the infrastructure and tools, the NMI infrastructure, are the same already used by project like Condor and VDT at the Uo. W. It also provides a direct bridge between EGEE 2 and OSG • Other projects developing software and in particular applications providers are more than welcome to use the services and tests their products with ETICS • One of the major goals of ETICS is to strengthen the interoperability and quality of middleware and applications developed for the Grid and other complex distributed environments. Everybody’s collaboration is welcome and needed. Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 28

Conclusions • We need robust components – We need reliable scheduling and execution environment for the complex workflows – We have many spread resources (CERN, INFN, Madison, …) and want to access them transparently • Collaboration with UW – Very good support from the Condor Team (Peter COUVARES and Anatholy KARP) – Bi-weekly technical meetings • We are on the track Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 29

Wish list … • • • Co-scheduling/parallel system tests Persistency Virtualisation Light dynamic Class. Ad publish mechanism Easy mechanism for match-making investigation – condor_q -better-analyse – Quill++ ? • How about your wishes for ETICS? Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 30

Want to learn more ? http: //www. eu-etics. org etics-support@cern. ch Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 31

Certification question Standards, QA, interoperability, … ETICS In what sense software is “good”? ETICS PAB User communities The software adheres to agreed standards, it executes in the expected way, satisfied a number of qualitative and quantitative requirements, etc, etc Condor. WEEK’ 2006, Milano, Marian. ZUREK@cern. ch 32