1 Software Engineering in the Trilinos Project Then

1 Software Engineering in the Trilinos Project: Then, Now and To Come Michael A. Heroux Sandia National Laboratories Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC 04 -94 AL 85000.

Trilinos Contributors (past 3 years) Chris Baker Developer of Anasazi, RBGen Mike Heroux Trilinos Project Leader Lead Developer of Epetra, Aztec. OO, Ross Bartlett Kokkos, IFPACK, Tpetra Lead Developer of MOOCHO, Stratimikos, RTOp, Developer of Amesos, Belos, Epetra. Ext Thyra Developer of Rythmos Ulrich Hetmaniuk Developer of Anasazi Pavel Bochev Lead Developer Intrepid Robert Hoekstra Lead Developer of Epetra. Ext Paul Boggs Developer of Epetra Developer of Thyra Russell Hooper Erik Boman Developer of NOX Lead Developer Isorropia Developer Zoltan Vicki Howle Lead Developer of Meros Todd Coffey Lead Developer of Rythmos Jonathan Hu Developer of ML Karen Devine Lead Developer Zoltan Joe Kotulski Lead Developer of Pliris Clark Dohrmann Lead Developer of CLAPS Rich Lehoucq Developer of Anasazi and Belos Carter Edwards Lead Developer phd. Mesh Kevin Long Developer of Thyra Michael Gee Developer of ML, Moertel, NOX Roger Pawlowski Lead Developer of NOX Bob Heaphy Lead developer of Trilinos SQA Michael Phenow Developer Zoltan Trilinos Webmaster Developer Web. Trilinos Eric Phipps Lead developer Sacado Developer of LOCA, NOX 2 Dennis Ridzal Lead Developer of Aristos, Intrepid Marzio Sala Lead Developer of Didasko, Galeri, IFPACK, Web. Trilin Developer of ML, Amesos Andrew Salinger Lead Developer of LOCA, Capo Paul Sexton Developer of Epetra and Tpetra Bob Shuttleworth Developer of Meros. Chris Siefert Developer of ML Bill Spotz Lead Developer of Py. Trilinos Developer of Epetra, New_Package Ken Stanley Lead Developer of Amesos and New_Package Heidi Thornquist Lead Developer of Anasazi, Belos, RBGen and Teuch Ray Tuminaro Lead Developer of ML and Meros Jim Willenbring Developer of Epetra and New_Package. Trilinos library manager Alan Williams Lead Developer Isorropia, FEI Developer of Epetra, Epetra. Ext, Aztec. OO, Tpetra

3 Take Home Messages § We (the Trilinos team) are not experts, but committed to improvement. § Quality matters. § New practices and tools can help us.

4 Background/Motivation

5 Target Problems: PDES and more… Circuits PDES Inhomogeneous Fluids And More…

6 Target Platforms: Any and All (Now and in the Future) § Desktop: Development and more… § Capability machines: w Redstorm (XT 3), Clusters w Roadrunner (Cell-based). w Large-count multicore nodes. § Parallel software environments: w MPI of course. w UPC, CAF, threads, vectors, … w Combinations of the above. § User “skins”: w C++/C, Python w Fortran. w Web, CCA.

7 Motivation For Trilinos § Sandia does LOTS of solver work. § When I started at Sandia in May 1998: w Aztec was a mature package. Used in many codes. w FETI, PETSc, DSCPack, Spooles, ARPACK, DASPK, and many other codes were (and are) in use. w New projects were underway or planned in multi-level preconditioners, eigensolvers, non-linear solvers, etc… § The challenges: w Little or no coordination was in place to: • • Efficiently reuse existing solver technology. Leverage new development across various projects. Support solver software processes. Provide consistent solver APIs for applications. w ASCI (now ASC) was forming software quality assurance/engineering (SQA/SQE) requirements: • Daunting requirements for any single solver effort to address alone.

8 Evolving Trilinos Solution § Trilinos 1 is an evolving framework to address these challenges: w w Fundamental atomic unit is a package. Includes core set of vector, graph and matrix classes (Epetra/Tpetra packages). Provides a common abstract solver API (Thyra package). Provides a ready-made package infrastructure (new_package): • • Source code management (cvs, bonsai). Build tools (autotools). Automated regression testing (queue directories within repository). Communication tools (mailman mail lists). w Specifies requirements and suggested practices for package SQA. § In general allows us to categorize efforts: w Efforts best done at the Trilinos level (useful to most or all packages). w Efforts best done at a package level (peculiar or important to a package). w Allows package developers to focus only on things that are unique to their package. 1. Trilinos loose translation: “A string of pearls”

9 Evolving Trilinos Solution physics § § Beyond a “solvers” framework Natural expansion of capabilities to satisfy application and research needs L(u)=f discretizations Math. model Lh(uh)=fh Numerical math Convert to models that can be solved on digital computers Numerical model Algorithms uh=Lh-1 fh Find faster and more efficient ways to solve numerical models methods Time domain Space domain Automatic diff. Domain dec. Mortar methods solvers Linear Nonlinear Eigenvalues Optimization core Petra Utilities Interfaces Load Balancing Algorithms § computation Discretization methods, AD, Mortar methods, …

10 Trilinos Package Concepts Package: The Atomic Unit

11 Trilinos Packages § Trilinos is a collection of Packages. § Each package is: w Focused on important, state-of-the-art algorithms in its problem regime. w Developed by a small team of domain experts. w Self-contained: No explicit dependencies on any other software packages (with some special exceptions). w Configurable/buildable/documented on its own. § Sample packages: NOX, Aztec. OO, ML, IFPACK, Meros. § Special package collections: w w Petra (Epetra, Tpetra, Jpetra): Concrete Data Objects Thyra: Abstract Conceptual Interfaces Teuchos: Common Tools. New_package: Jumpstart prototype.

12 Trilinos Package Summary Objective Rythmos Automatic Differentiation Sacado Mortar Methods Moertel Epetra, Jpetra, Tpetra Thyra, Stratimikos, RTOp Load Balancing Zoltan, Isorropia Py. Trilinos, Web. Trilinos, Star-P, For. Trilinos C++ utilities, (some) I/O Teuchos, Epetra. Ext, Kokkos, Triutils Iterative (Krylov) linear solvers Aztec. OO, Belos, Komplex Direct sparse linear solvers Amesos Direct dense linear solvers Epetra, Teuchos, Pliris Iterative eigenvalue solvers Solvers Time Integration “Skins” Core phd. Mesh, Intrepid Abstract interfaces Methods Meshing & Spatial Discretizations Linear algebra objects Discretizations Package(s) Anasazi ILU-type preconditioners Aztec. OO, IFPACK Multilevel preconditioners ML, CLAPS Block preconditioners Meros Nonlinear system solvers NOX, LOCA Optimization (SAND) MOOCHO, Aristos

13 Characterizing the Trilinos “Project” § Not a “project” but an infrastructure to support inter-related projects: A project of projects. § Package participation is voluntary: w w Framework must be attractive (and continue to be). Requirements are few, opportunities are many. Package team decides what and when. Opt-out is always an option. § Package autonomy is carefully guarded: w Even if redundant development occurs. w Decision-making pushed to lowest (best) level. § Participation is attractive: w Increasing infrastructure capabilities. w Access to many other packages.

14 Implications § Package teams must see value of SE tools and processes. w Won’t adopt something because SE theory says it’s good. § Might adopt something “imposed” on them: w But use will be temporary. w Ultimately a negative impact. § New tools and processes adopted incrementally: w Package teams busy with algorithm development. w Open to learning SE as time permits. § Prefer established tools to bleeding edge: w Package teams busy with algorithm development.

15 Trilinos Strategic Goals § Scalable Computations: As problem size and processor counts increase, the cost of the computation will remain nearly fixed. § Hardened Computations: Never fail unless problem essentially intractable, in which case we diagnose and inform the user why the problem fails and provide a reliable measure of error. § Full Vertical Coverage: Provide leading edge enabling technologies through the entire technical application software stack: from problem construction, solution, analysis and optimization. § Grand Universal Interoperability: All Trilinos packages will be interoperable, so that any combination of packages that makes sense algorithmically will be possible within Trilinos and with compatible external software. § Universal Accessibility: All Trilinos capabilities will be available to users of major computing environments: C++, Fortran, Python and the Web, and from the desktop to the latest scalable systems. § Universal Solver RAS: Trilinos will be: w Integrated into every major application at Sandia (Availability). w The leading edge hardened, efficient, scalable solution for each of these applications (Reliability). w Easy to maintain and upgrade within the application environment (Serviceability). Algorithmic Goals Software Goals

16 Then When Trilinos was unknown

Important Early SE Decisions 2000 -2002 § Package Autonomy: w Carefully monitored by Tammy Kolda. w Meeting NOX needs kept us on track. § Self-contained OO Data Model: w Critical to decoupling packages. w Participation in Equation Solver Interface (ESI) Forum. w Petra heavily influenced by Alan Williams, Robert Clay OO design. § Public visibility: w w Public repository: Improves SW quality (ad hoc code reviews). Use of mail lists to (over) communicate. Regular phone meetings. Tammy Kolda and Paul Sery critical here. § Focus on working with customer: w Problem-driven, close interaction. w Roger Pawlowski integrating NOX into many codes. 17

Starting Points uid=26437(mheroux) … CVSROOT loginfo, 1. 2, 1. 3 Fri Dec 14 15: 26: 06 PST 2001 Update of /usr/local/CVSROOT In directory iterative: /tmp/cvs-serv 5952 Modified Files: loginfo Log Message: Fixed mail alias on loginfo Epetra-Checkins Archives Starting: Sat Sep 28 07: 19: 50 2002 Ending: Sat Sep 28 08: 37 2002 Messages: 4 [Epetra-Checkins] Trilinos/packages/epetra/src classic. Makefile mheroux@skip. sandia. gov Last message date: Sat Sep 28 08: 37 2002 Archived on: Sat Sep 28 01: 39: 00 2002 18

19 Process Improvement § Process Improvement: Tradition for Trilinos. § Yearly rollout of new goals. § Discussed and decided during Trilinos User Group meeting (Nov 6 -8 this year). § Here are some past years.

20 Major Framework Themes: FY 05/06 § Trilinos Package Architecture: w Continue refinement of new_package. w Explicitly define Trilinos compatibility. w Resolve the abstract interface issue. § Software Quality: w w Expand use and ease-of-use of test harness. Identify metrics and automate capture and display. Establish a life-cycle model (hybrid agile/unified process? ). Customize the ASC SQP to our environment. § Packages: w Foster new package development. w Manage the growth. Issue: complexity of package coupling. w Harden our mature packages. § Transition to post-delivery maintenance: w Organizational issue: Tough to solve.

21 Major Themes for FY 06/07 Framework § Take Steps toward dynamic package addition. w In light of more outward focus: (Sci. DAC, Boeing, ? ? ? ) w Trilinos-compatibility definition or something similar. § Define SW Lifecycle(s) and begin formalized efforts. w Need something for external audits. w Agile vs. UP vs. hybrid.

FY 07/08 Themes: Framework § Trilinos Level II Milestone. w Demonstrate Use of Full Vertical capabilities in Charon. § Joint licensing and copyright of development with other organizations. w Other DOE Labs, international orgs, private companies. § Package Autonomy (reacting to rapid growth): w 18 to 27 to ? ? w Guarding against incidental coupling. w Revisiting the location of “skins”. § Stratimikos: Uniform access to many packages. § Access from Fortran. w DOE Science Users, DOD Users. w Stratimikos focus. § Split of User vs Developer tools. w http: //trilinos. sandia. gov w http: //software. sandia. gov 22

23 Remainder of “Then” § § Use of serious configuration tools. Active, well used website. Responsive support of framework. Much more… § Attitude of Trilinos contributors: w True desire to work together. w Focused on problem solving. w The primary reason for success.

24 Now: Current Tools & Practices

25 Software Goals § Quality: w Everybody wants it, but expensive in time and money. w No longer sufficient to say “Trust us, we do good work. ” § Modularity/Interoperability: w Tight cohesion within a package. w Loose coupling across packages. § Scalability: w Make the cost of adding/supporting a new package as low as possible. w Shield the user from this complexity as much as possible. § Efficient use of Experts’ Time: w Provide as much support to package teams as possible. w Provide defaults with ‘opt-out’ clauses. § Accessibility and Support: Get the software out there.

26 Software Principles § Package Orthogonality: w Trilinos “string-of-pearls”. w Asynchronous, flexible package maturation process. § Global Services: w Framework: Established tools and processes. w New. Package package. § Tight collaboration: w Environment that fosters interaction: • Across package teams. • With Users and Clients. w Tools for automated communication of events. § Iterative Development: User access to multiple SW branches. § Process Improvement: w Record what we do and don’t do but want to do. w Track progress.

27 Software Tools and Practices § § § § Source Management: CVS & Bonsai. Communication channels: Email lists, regular phone meetings. Documentation: Custom docs, tech reports, journals, Doxygen. Configuration management: Autoconf, Automake. Information distribution: Website. Testing: Test Harness. Release process: Formal, checklist driven.

28 Development Practices and Tools What We Do and Use

29 Source Management § Source management is a first order activity: w Required for any other process. w Useful for personal files too! § CVS: w Tried and True. w No User Training Required. Free. Ubiquitous. w Not the fullest in features anymore. § Subversion (svn) clearly a good replacement at some point. w Already used by related projects. w Not yet for Trilinos: Bonsai. § Other Options: GIT, Mercurial, …

Bonsai “blame mode” 30

Mailman: Extensive use for many types of communication 31

32 Issue Tracking § Bugzilla: w Normal everyday issues. w Release Process “metabugs”. w Performance testing records. w Completed release checklists. w 2458: Metabug blocked by all normal bugs to be fixed before release. w 2447: Trilinos level release bug blocked by Epetra release bug.

33 Sometimes we have a little fun!

34 Reference Documentation § Automatically generate twice a day from repository. § Avid fans of Doxygen: w Dimitr van Heesch http: //www. stack. nl/ ~dimitri/doxygen/

Building Trilinos 35

Simple Sample Scripts. . /configure 36 • Builds serial version of default packages. • All 3 rd party libraries (BLAS, LAPACK) must be in lib search p . . /configure --prefix=/home/mheroux/Trilinos/EPETRA_OPT_SERIAL CXXFLAGS="-O 3“ --disable-default-packages --enable-epetra • Builds serial version of Epetra only with some optimization • All 3 rd party libraries (BLAS, LAPACK) must be in lib searc • “make install” will put /include/*. h and /lib/*. a in specified d • Builds MPI version of Epetra only with specified MPI en • All 3 rd party libraries (BLAS, LAPACK) must be in lib se • “make install” will put /include/*. h and /lib/*. a in specified. . /configure --enable-mpi --with-mpi=/usr/MPICH/SDK. gcc --prefix=/home/mheroux/Trilinos/EPETRA_MPI --with-mpi-cxx=g++ --with-mpi-cc=gcc --with-mpi-f 77=g 77 --with-mpi-libs=-lmpich --disable-default-packages --enable-epetra

More Complex Sample Scripts 37 Most found in Trilinos/sample. Scripts . . /configure --host=powerpc-ibm-aix 5. 1. 0. 0 • --enable-teuchos-extended --disable-new-package Configure/make on some platform --enable-aztecoo-azlu complex. --enable-anasazi • We maintain a directory of sample --enable-epetraext-transform scripts to guide installers. --enable-epetraext-transform-tests --enable-amesos --with-ml_teuchos --with-ml_anasazi --with-ml_external_mpi_functions --disable-examples --with-blas="-lessl -L/sierra/Release/lapack/3. 0/lib/dbg_dp_ibm -lblas" --with-lapack="-lessl -L/sierra/Release/lapack/3. 0/lib/dbg_dp_ibm -llapack" --enable-mpi --with-ldflags="-L/sierra/Release/y 12 m/1. 00/lib/dbg_dp_ibm -ly 12 m" --prefix=/sierra/Release/Trilinos/3. 1. 1/install_ibm --with-mpi-incdir=. --with-mpi-libs="-binitfini: poe_remote_main -lmpi_r -lvtd_r" --with-ar="ar -X 64 csrv" CXX="mp. CC -q 64" CC="mpcc -b 64 -q 64" F 77="mpxlf -b 64 -q 64" CXXFLAGS="-O 3 -w -qnofullpath -qlanglvl=ansi" CCFLAGS="-O 3 -w" FFLAGS="-O 3 -w" CPPFLAGS= LDFLAGS= LIBS=-lm FLIBS="-lxlf 90 -lxlopt -lxlf -lxlomp_ser"

38 Website § § § http: trilinos. sandia. gov Developer content on software. sandia. gov. Always looking to improve layout, content. Site was recently redesigned.

Test Harness § Tests detected in repository. § Presence of test definition file triggers harness run. § Form of def file: # # This group defines # CATEGORY-1 and # CATEGORY-2 tests. # (CATEGORY-1, CATEGORY-2) { NAME-A = VALUE-A; NAME-B = VALUE-B; NAME-C = VALUE-C; } # # This group defines # CATEGORY-3 tests. # (CATEGORY-3) { NAME-A = VALUE-A; NAME-B = VALUE-B; NAME-C = VALUE-C; } 39

Test Harness Example 40

Release Process § Checklists Drive the Release Process. § Quality and Stability of the Release has improve dramatically with each additional list. § Let’s Look at one. 41

Package-level Release Process Checklist 42

43 Four Parts: Plan, Do, Check, Act § § Plan: Do: Check: Act: Determine what you will do. Do it. Confirm results. Take action to complete process. Also known as the Deming Cycle (W. Edwards Deming)

44 Part 1: Plan

Part 2: Do 45

Part 3: Check 46

Part 4: Act 47

48 History of a Package Maturation Process Example: ML Use of Tools Asynchronicity

History of ML Maturation Process Step Package added to CVS: Import existing code or start with new_package. Example ML CVS repository migrated into Trilinos (July 2002). Mail lists, Bugzilla Product, Bonsai database created. ml-announce, ml-users, ml-developers, ml-checkins, mlregression @software. sandia. gov created, linked to CVS (July 2002). Package builds with configure/make, Trilinoscompatible ML adopts Autoconf, Automake starting from new_package (June 2003). Epetra objects recognized by package. ML accepts user data as Epetra matrices and vectors (October 2002). Package accessible via Thyra interfaces. ML adaptors written for TSFCore_Lin. Op (Thyra) interface (May 2003). Package uses Epetra for internal data. ML able to generate Epetra matrices. Allows use of Aztec. OO, Amesos, Ifpack, etc. as smoothers and coarse grid solvers (Feb. June 2004). Package parameters settable via Teuchos Parameter. List ML gets manager class, driven via Parameter. Lists (June 2004). Package usable from Python (Py. Trilinos) ML Python wrappers written using new_package template (April 2005). Startup Steps Maturation Steps 49

50 Day 1 of Package Life § § CVS: Each package is self-contained in Trilinos/package/ directory. Bugzilla: Each package has its own Bugzilla product. Bonsai: Each package is browsable via Bonsai interface. Mailman: Each Trilinos package, including Trilinos itself, has four mail lists: w package-checkins@software. sandia. gov • CVS commit emails. “Finger on the pulse” list. w package-developers@software. sandia. gov • Mailing list for developers. w package-users@software. sandia. gov • Issues for package users. w package-announce@software. sandia. gov • Releases and other announcements specific to the package. § New_package (optional): Customizable boilerplate for w Autoconf/Automake/Doxygen/Python/Thyra/Epetra/Test. Harness/Website

51 Maturation Jumpstart: New. Package § New. Package provides jump start to develop/integrate a new package § New. Package is a “Hello World” program and website: w Simple but it does work with autotools. w Compiles and builds. § New. Package directory contains: w w w Commonly used directory structure: src, test, doc, example, config. Working Autoconf/Automake files. Documentation templates (doxygen). Working regression test setup. Working Python and Thyra adaptors. § Substantially cuts down on: w Time to integrate new package. w Variation in package integration details. w Development of website. NOTE: New. Package can be used independent from Trilinos

52 Software Quality

53 SQA/SQE § Software Quality Assurance/Engineering is important. § Not sufficient to say, “We do a good job. ” § Trilinos facilitates SQA/SQE development/processes for packages: w 10 of 30 ASC SQE practices are directly handled by Trilinos (no requirements on packages). w Trilinos provides infrastructure support for the remaining 20. w Trilinos Dev Guide Part II: Specific to ASC requirements. w Trilinos software engineering policies provide a ready-made infrastructure for new packages. w Trilinos philosophy: Few requirements. Instead mostly suggested practices. Provides package with option to provide alternate process.

Trilinos Service SQE Practices Impact 54 Yearly Trilinos User Group Meeting (TUG) and Developer Forum: Once a year gathering for tutorials, package feature updates, user/developer requirements discussion and developer training. — All Requirements steps: gathering, derivation, documentation, feasibility, etc. — User and Developer training. Monthly Trilinos leaders meetings: Trilinos leaders, including package development leaders, key managers, funding sources and other stakeholders participate in monthly phone meetings to discuss any timely issues related to the Trilinos Project. —Developer Training. —Design reviews. —Policy decisions across all development phases. Trilinos and package mail lists: Trilinos lists for leaders, announcements, developers, users, checkins and similar lists at the package level support a variety of communication. All lists are archived, providing critical artifacts for assessments and audits. —Developer/user/client communication. —Requirements/design/testing artifacts. —Announcement/documenting of releases. Trilinos and Trilinos 3 PL source repositories: All source code, development and user documentation is retained and tracked. In addition, reference versions of all external software, including BLAS, LAPACK, Umfpack, etc. are retained in Trilinos 3 PL. —Source management. —Versioning. —Third-party software management. Bugzilla Products: Each package has its own Bugzilla Product with standard components. —Requirements/faults capturing and tracking. Trilinos configure script and M 4 macros: The Trilinos configure script and related macros supportable installation of Trilinos and its packages —Portability. —Software release. Trilinos test harness: —Pre-checkin and regression testing. Trilinos provides a base testing plan and automated testing across multiple —Software metrics. platforms, plus creation of testing artifacts. Test harness results are used to derive a variety of metrics for SQE.

55 Software Lifecycles

56 (Typical) Project Lifecycle Consider this lifecycle Project Conception Research & Development Production Support & Maintenance End of Life

Scientific Research and Life Cycle Models § Life Cycle Models are generally developed from the point of view of business software. § Little consideration is given to algorithmic development. § Traditional business execution environment is traditional mainframe or desktop, not parallel computers. § Traditional development “techniques” are assumed. 57

Research Software needs a different model § Research should be “informal”: w Allow external collaborators, students, post-docs, etc. w Allow changes of direction without seeking permission w Should use modern software development paradigms • i. e. Lean/Agile methods w Must be verified more than validated § Production code must: w w w Have formality appropriate to risks, Be Complete (documentation, testing, …), Be “user proofed”, Be platform independent (as necessary), Be validated not just verified. 58

59 “Promotional” Model Phase k • Lower formality • Fewer Artifacts • Lean/Agile Promotional Event Phase k+1 • Higher formality • Sufficient Artifacts • Bullet proof • Maintainable

60 Trilinos Software Lifecycle Model

61 Lifecycle Models § A major component of any software project. § Exists, whether formal or ad hoc. § Trilinos model: w Really a meta-model. w Attempts to captures the reality of our software engineering environment.

62 Trilinos Software Environment § Many formal software lifecycle models exist. § Trilinos environment seems somewhat unique: (When compared to commercial environments, or not? ) § On one hand: Tasked to develop algorithms and software that are leading-edge, with the goal of solving problems that were previously intractable. § On the other: Required to deliver software that can eventually be used to certify critically important engineering systems.

63 Further Complexities § Trilinos composed of packages: w Self-contained pieces of software developed by semiindependent small teams. w Each package matures at its own pace: • Typically evolving from small algorithms study. • Becoming a widely-used piece of software. • Embedded in multiple applications. § Requirements for rigor change as a particular Trilinos package matures.

64 Trilinos Lifecycle Phases § Three phases: w Research. w Production Growth. w Production Maintenance. § Each phase contains its own lifecycle model. § Promotional events: w Required for transition from one phase to next. w Signify change in behaviors and attitude. § Phase assigned individually to each package.

65 Lifecycle Phase 1: Research § Conducting research is the primary goal. § Producing software is potentially incidental to research. § Any software that is produced is typically a “proof of concept” or prototype. § Software that is in this phase may only be released to selected internal customers to support their research or development and should not be treated as production quality code.

66 Phase 1 Required Practices § The research proposal is the project plan. § Software is placed under configuration control as needed to prevent loss due to disaster. § Peer reviewed published papers are primary verification and validation. § The focus of testing is a proof of correctness, not software. § Periodic status reports should be produced, presentation is sufficient. § A lab notebook, project notebook, or equivalent is the primary artifact.

67 Phase 1 Remarks § Phase 1 practices are common to efficient research in general. § Research phase software: w Need not be written in the “target” language. w Nor support all target machines. w Usually has limited error checking and recovery. § (Software) risk is low (primarily technical, not mitigated formality processes. § Level of formality is low.

68 Phase 2: Production Growth Goals: 1. Elevate package to releasable product. 2. Satisfy the Advanced Scientific Computing (ASC) Software Quality Plan, at a minimum. 3. Make software product suitable for use by highly skilled users.

69 Phase 1 2 Promotion Event Risk Assessment: 1. What are the package’s primary technical and project management risks? 2. How can these risks be mitigated? Gap Analysis: 1. Which practices and processes must be added or improved to get the package into a releasable state? 2. What special actions or training will be required? 3. What is the target date for complying with the level of practices and processes required for release? Promotion Decision: 1. Considering the results of the risk assessment, gap analysis, and other data, will the package be promoted to Phase 2? 2. What is the target date for releasing the package?

70 Phase 2 Required Practices (Most Important) 1. Agile methods (with associated lifecycles) are encouraged, for example the practices and processes promoted by Extreme Programming. 2. All essential ASC SQE practices performed at an appropriate level (predetermined during promotion event from the research phase). 3. Artifacts should naturally “fall out” from SQE practices and periodic status reviews and management reports. 4. Process improvement and metrics are appropriate.

71 Phase 2 Remarks § Phase may be cyclic (spiral, etc. ) as new algorithms become incorporated. § Software may not yet support all intended missions or platforms. § Risk level is medium: w Technical risks are reduced. w Total risk is more project management oriented such as schedule, budget, staffing, etc. ). § Default level of formality is medium.

72 Phase 3: Production Maintenance § Goal: Robust software suitable for typical end uses. § At this point: w Requirements and prototype software foundation are stable. w However, Agile methods no long sufficient: • Product maintenance during the coming decades of software use where typically only adaptive maintenance • Response to computer system changes. § More complete set of artifact is required. § Software itself will require changes to improve maintainability. § In extreme case: May make sense to rewrite large portions.

73 Phase 2 3 Promotional Event Risk Assessment: 1. What are the package’s primary technical and project management risks? 2. How can these risks be mitigated? Gap Analysis: 1. Which practices and processes must be added or improved to get the package into a maintenance ready state? 2. What is the target date for complying with the required level of practices and processes?

74 Phase 2 3 Promotional Event (cont) Promotion Decision: • What is the medium to long-term funding outlook for the package? • Who is going to provide long-term maintenance services for the project? (One or more of the original developers, or a different group? ) • If funding is not likely to be available for future maintenance, current customers should be notified so they have a chance to offer continued funding if it is in their best interest. A list of these customers should be produced. • If the customer base of the package is small or the package has been replaced with another code, the development team may consider retiring the code rather than moving to the third development phase. Any such decision should be approved by customers and management. • Considering the answers to the above questions, and other available data, will the package be promoted to Phase 3? • What is the target date (if any) for turning the package over to the long-term maintenance team?

75 Phase 3: Required Practices 1. After achieving maintenance ready status, the package may (as determined during the promotion event) be handed over to another party during this phase for continued support and development. 2. If the code is transferred to a different party, the ownership of the design is not necessarily transferred. The design owner must attend meetings concerning requirements changes and potential design changes. 3. A widely-accepted lifecycle model such as the Waterfall or Unified Process methods is used. 4. End of life planning is a key component during this phase. In particular, the software must have good compliance with SQE practices and internal documentation must be formal (UML is suggested). 5. SQE practice compliance and solid documentation will help to ensure a successful transfer of the code, and must be completed whether a transfer is currently planned or not.

76 Phase 3 Remarks § The risk level is low (almost totally project management risks, which can be mitigated by appropriate process formality). The default level of formality is high (particularly if the project may be handed off to another party). § This phase is untested so far. § Moving to this phase is expensive.

77 Exceptional Cases § Isolated Lower Phase: w Multiple techniques: • Subdirectories: All lower phase software is contained in speciallydesignated subdirectories and is only activated by special compilation procedures. • Interface adapters: Lower phase software is self-contained in new class files and accessed via polymorphism of abstract interface mechanisms or similar techniques that are not necessary for basic operation. • Conditional compilation directives: discouraged in general. § Externally-developed Packages: w By default in Phase 1: Must go through same process. w BLAS/LAPACK different: Certified as part of dependent-package process.

78 Usage Status § § § Lifecycle Model newly define: Gives us a target. 25% of package firmly in Phase 1. 75% of packages on the way to Phase 2. None are in Phase 3 is thus just speculation at this point.

79 To Come Opportunities and Challenges

80 Themes for FY 08/09 § § § Redefinition of Trilinos scope beyond solvers. Next steps in packaging and distribution. Continued outreach to other communities Rethinking source management. Formalizing App-Trilinos relationship. Post-delivery maintenance improvements.

81 Scope of Trilinos § Addition of Sacado, Zoltan, FEI, Intrepid, phd. Mesh: Not solvers. § Framework support natural. § Rephrasing of project goals, descriptions underway. § Grouping of packages into meta-packages: At least conceptually.

82 Packaging and Distribution § Mac and Windows are ever more popular development environments. § Goal: Provide click-install capabilities for Mac OS, MS Visual Studio, Linux COE.

83 Outreach § Trilinos packages part of Sci. DAC: w ITAPS, CSCAPES, TOPS-2. w Opportunity to serve broader DOE community. § Trilinos popular in universities: w Single largest sector of users. § Trilinos part of several industrial efforts. w Improves capabilities. w Amortizes costs over broader funding sources. § Elevates certain activities: w Fortran accessibility. w Packaging & distribution.

84 Source Management § Think of repository as a database. § Logical collections gathered dynamically. § Consider use of multiple source management tools: w Local vs. global management. w Fully distributed. § Certainly svn is option, but looking at all options.

85 App-Trilinos Relationship § How should apps to interact with Trilinos? § Many facets: w w Interfaces: concrete vs. abstract. Testing: How frequently, who does it. Distribution: Trilinos separate or integrated. … § Goal: Any two applications should differ only in: w Essential needs due to formulation. w Specific uses of enabling technologies.

Post-delivery Maintenance and Support § Long-term support for software that is: w Heavily used but, w Not actively developed, is a chronic issue across the DOE complex. § BLAS, LAPACK, Scalapack, Aztec… § Organizational structures do not reflect overall cost of software: w 60%-70% in post-delivery maintenance. w We are research focused. § Working on plan to use matrixed org. 86

87 Take Home Messages § We (the Trilinos team) are not experts, but committed to improvement. w Regular assessment of opportunities helpful. w Team sees past adoption successful, willing to try more. § Quality matters. w We want our software to be trusted and reliable. § New practices and tools can help us. w Open Source tool sets are truly impressive. w More opportunities are possible.

88 Trilinos Availability/Information § Trilinos and related packages are available via LGPL. § Current release (8. 0) is “click release”. Unlimited availability. § Trilinos Awards: w w 2004 R&D 100 Award. SC 2004 HPC Software Challenge Award. Sandia Team Employee Recognition Award. Lockheed-Martin Nova Award Nominee. § More information: w http: //trilinos. sandia. gov w http: //software. sandia. gov w Additional documentation at my website: http: //www. cs. sandia. gov/~mheroux. § 5 th Annual Trilinos User Group Meeting: November 6 -8, 2007 at CSRI/90 (This room) Sandia National Laboratories, Albuquerque, NM, USA.