Forum of the John von Neumann Society Budapest

Зарегистрируйтесь, чтобы просмотреть полный документ!

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Software Engineering Evidence Dieter Rombach (rombach@informatik. uni-kl. de) Technical University of Kaiserslautern Computer Science Department Software Engineering Chair Kaiserslautern, Germany wwwagse. informatik. uni-kl. de Fraunhofer Institute for Experimental Software Engineering (IESE) Kaiserslautern, Germany www. iese. fhg. de 0

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Contents • Experimental Software Engineering • Software Engineering Evidence • Towards a Theory of Software Engineering Evidence • Empirical Methods • Existing Body of Knowledge • Experimental Software Engineering in Kaiserslautern • Practical Examples • Agenda for Research, Tech Transfer & Teaching • Outlook 1

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering (1 of 3) • Computer Science is one of the scientific base disciplines for the “engineering of large (software) systems” Systems Engineering Mechanical Engineering Software Engineering Physics Computer Science Mathematics Economics … Psychology 2

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering (2 of 3) • All research in computer science & software engineering should be application oriented – basic research has a longer time-scale than applied research – non-application oriented research is only justified in natural sciences! Which testing technique – application needs (in engineering) are defined via G == f (P, C) promises/guarantees • at least constraints (e. g. , staff experience levels, engineering 95 defect detection development technology such as OO/UML) effectiveness for large OO/UML-based systems? • goals (e. g. , reliability of >= 0. 99) 3

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering (3 of 3) Experimental SE • Software Engineering comprises – (formal) methods (e. g. , Experimental Software modeling techniques, Engineering Formal description languages) – system technology (e. g. , Process System - recognizes the nature architecture, modularization, of our field Technology Theory OO, product domain) (engineering in a human-based design lines) – process technology (e. g. , - applies empirical studies in order to life-cyle models, processes, determine engineering management, measurement, effects (f) organization, planning QS) in such a design domain Empiricism – empiricism (e. g. , experimentation, experience capture, experience reuse) Methods 4

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Software Engineering Evidence (1 of 2) • Empirical studies aim to capture quantitative evidence regarding (P) – product characteristics (definition, behavior) “What is the complexity of a product? ” • “What is the performance of a system? ” • G == f (P, C) – process characteristics (definition, behavior) • “What is the inherent degree of parallelism? ” • “How much effort does it take? ” – process-product relationships • “How does design complexity affect test effort? ” • Issues – How deterministic are studies? – How easy/hard is it to test results via replication? 5

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Software Engineering Evidence • Traditional (quantitative) empirical evidence – controlled experiments (variation in G == f (P, C) C is controlled) – case studies (C is a constant – reflecting some environment) Statistical Significance increases • Questionnaires, Action Research, …. (mostly qualitative) • Expert consensus (like in medicine) Practical Acceptance increases Scientists (aiming at testable cause-effect relations) prefer controlled expriments! 6 Practitioners (aiming at low-risk technology infusion) prefer case studies & expert consensus

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Towards a Theory of SE Evidence (1 of 3) <var> • Elements of Evidence (G == f (P, C)) – Review Technique P has effectiveness (80%, 50%) for (experienced, inexperienced) developers – Life cycle model P consumes resources (effort distribution model 1/model 2) for (large/small x functional/OO design based) projects • G: any quality, cost & time model (e. g. , reliability, effectiveness) • P: any SE process, method or technique (e. g. , testing, design) • C: any project environment characteristic that defines the validity of G All evidence should be packaged like (G x P x C ) • f: mapping from (P, C) onto G in order to empirically (because lack engineering • ==: mapping is definedbe accessible for of physical laws!) • <var>: decisions! some measure of certainty (e. g. , +- X%) This is also important for scientific publications! 7

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Towards a Theory of SE Evidence (2 of 3) • Aggregation (vertical: P constant) – to increase significance (i. e. reduce <var>) – to increase generality, i. e. variation of C : = C 1 x C 2 x C 3 x C 4 by increasing the range of each Ci • Significance increase – experiment replication. Problem (C coverage): Complexity (e. g. , inspection area) – formal aggregation depends on measure for significance (e. g. # replications are incremented) Maturing (observations -> laws -> theories) based on • Variation increase – empirical variation (e. g. , applications, experiences, …) experiment studies ALLONE takes too long! – complexity: simple coverage for 5 variables with 4 values each requires “ 4 to the power of 5” = 1024 experiments? ? ? Expert consensus „when oberservations are considered laws“ – new hidden context variables HC appear in meta analysis! is necessary! • E. g. , (G 1, P, C) & (G 2, P, C) (G 1/G 2, P, C x (HV)) 8

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Towards a Theory of SE Evidence (3 of 3) • Aggregation (horizontal: P increases) – to scale up to “larger” processes P (e. g. , Cleanroom software development process) – perform controlled experiments in “key elements” (e. g. , unit inspections vs. testing) – perform integration case studies – acceptance of scaled-up evidence must be confirmed by expert consensus (organization or community) Complexity Problem (P size): Performing controlled experiments on large processes is practically impossible! Expert consensus is required to accept evidence! 9

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Empirical Methods (1 of 3) • Science in general involves – modeling of software product & process artifacts – empirical validation of hypotheses regarding their characteristics & behavior in testable form • Empirical foundation includes methods for There existsrelating goals to measurements (GQM) a comprehensive body of empirical methods! • piggy-bagging empirical studies on real projects (QIP) (ISERN Workshops & • organizing empirical observations for reuse (EF) • ISESE activities such as experimental design, data specific Conference in Los Angeles, analysis week of 16 August 2004) – importance of combining quantitative & qualitative analysis • No excuse for not applying it! 10

Forum of the John von Neumann Society Empirical Methods (2 of 3) Generic Form of Empirical SE Contributions: G == f (P, C, HC) Budapest, Hungary, 10 March 2004 • For non-deterministic processes • Function f can only be determined empirically • Variance of function f depends on – sufficient # data points – inclusion of relevant C – stability of non-included C • important use of qualitative analysis: assure that there are no non-identified context factors • Variance may increase when – repeated in/included with new environment – reason: hidden context factor (i. e. , factor which was constant in one or both environents) becomes influential – consequence: New empirical cycle OR keep two different domains (i. e. identical recommendation exists for product lines!) <var> - G: any quality aspect such as What is method, equivalent to physical laws? our - P: process, technique, reliability or maintainability tool under investigation - congitive „laws“? - HC: hidden context factorsempirically validated? - that only show up at certain levels of - always context dependent? generalization - C: context factors such as experience, lifecycle model 11

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Empirical Methods (3 of 3) • Physics offers laws for electrical eng. – precise – not circumventable Physical laws • Computer Science & …. offer “laws” for SE – empirically precise Cognitive Laws – circumventable (e. g. , you may increase the complexity of any system and it still may work! is this really true? – not if one includes evolution! • what defines bounds? – models that capture the negative consequences if you exceed complexity bounds • 12 (Com <= Co Test_Effort <= To ! Test_Effort > To)

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Existing Body of Knowledge • There exists more knowledge than we typically recognize – mostly in terms of context-specific empirical observations – rarely in terms of generalized “laws” • There exist already more empirical “laws” than we typically recognize – book (Endres/Rombach, Addison, 2003) – inspections – design principles • More studies need to be done – repeat (with variation) – generalize 13

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Fraunhofer IESE Series Editorial Board includes - Dave Parnas - Dieter Rombach - Ian Sommerville - Marv Zelkowitz Handbook capturing existing body of knowledge Students can learn about existing body of knowledge Practitioners can avoid negligance of due dilligance 14 Additions are welcome for next edition of book

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering in Kaiserslautern (1 of 8) • Empirical study driven software engineering research requires a laboratory environment for – in vitro: controlled student experiments – In vivo: case studies on real industry projects • “Mother” of all lab environments – NASA SEL (GSFC, CSC & UMD) combination of stake-holders • long-standing trusting relationship • sustained accomplishments None ofreduction of cost (50%) & increase in quality (no slipping the SEL success stories would have – defects) happened without empirical studies! – sustained via empirically modeled “laws” – regular use of “formal” methods (e. g. , functional semantics in Cleanroom) – overcoming of development “factoids” (e. g. , testing is best defect detection approach) • 15

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering in Kaiserslautern (2 of 8) Fraunhofer IESE SME‘s 1 RL State 2 1 2 SW&Sys. Eng Large Comp‘s RL (Bosch) Univ. Kaiserslautern DFG Res. Institutes 16

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Experimental Software Engineering in Kaiserslautern (3 of 8) • Experimentation (EXP) (Experimental designs, QIP/GQM/EF, Analysis techniques) • Requirements and Usability Engineering (RUE) (Requirements Engineering, Usability Engineering) • Component-based Software Engineering (CBE) (Component-based software development mainly in the field of embedded and realtime systems) • Software Product Lines (SPL) (Architecture recovery, Architecture development & evaluation, Scoping & modeling of domains & decisions) 17

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 ESE in Kaiserslautern (IESE, Research Competences) (4 of 8) • IT Security & Safety (ITS) (Security Assessments, EF for Security, Automated checking of infrastructures) • Quality and Process Engineering (QPE) (Goal-oriented measurement GQM, Descriptive process modeling, Goal-oriented product/process assessment, Subcontracting, COTS Acquisition, Risk Management) • Systematic Learning & Improvement (SLI) (QIP-based SPI planning & setup, EF buildup and Organizational Learning) • Certifiable Education & Training in SE (CET) (Evaluation and Certification, Technology-enabled Learning, Simulation-based learning and decision support) 18

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 ESE in KL (IESE, Maintain intellectual Control over System Complexity) (5 of 8) Cost Avoiding deterioration of system complexity SE principles & over time yields linear technical engineering cost curve! processes : actual cost curve : desired cost curve Product Functions (Releases) over time 1 2 3 19

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 ESE in KL (IESE, Maximize Software Reuse) (6 of 8) Cost Product Line Reuse Exploiting reuse potential with proper apriori scoping (tool support) yields constant cost curve! : PL cost curve : desired cost curve Product Functions (Releases) over time 1 2 3 20

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 ESE in KL (IESE, Achieve Predictability for Software Development) (7 of 8) – Cost X Management Principles X X X : maximum oscillation Measurement-based project management models yield small bandwidth prediction capabilities! X X : desired cost curve X 1 2 Product Functions (Releases) over time (Size) 3 21

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 ESE in KL (IESE, Business Areas) (8 of 8) Reliable Software for Embedded Systems • • • Secure Software for Infrastructure Facilities and Services • Telecommunication • Providers • Telematic Service Providers Flexible Software for IT-based Business Processes • Commerce (B 2 B, B 2 C) • Medical Information Systems • Financial Service Providers / Insurance • e. Government Software-based Products and Services • Software Companies • Continuing Education & Training Service Providers • Consulting Companies Automotive Industry and Railroad Mechanical Engineering Air and Space Industry Industrial and Consumer Electronics Chemical and Pharmaceutical Industry Medical Technology 22

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Practical Examples (Inspections @ NASA) (1 of 4) • NASA SEL Experience (see Basili, JSS, 1997) – stepwise abstraction code reading vs. testing (Basili/Selby, TSE, 1987) • controlled experiment at UMD & NASA/CSC • effectiveness & cost (SAR > testing) • self-assessment (SAR > testing) – stepwise abstraction code reading in regular SEL project • case study at NASA/CSC • SAR did not show any benefits • diagnosis: People did stewise abstraction code reading not as well as they should have as they believed that testing would make up for their mistakes – Cleanroom vs. standard SEL software development • controlled experiment at UMD • more effective application of reading, less effort and more schedule adherence – stepwise abstraction code reading in SEL Cleanroom projects • case study(ies) at NASA/SEL • improved failure rates (- 25%) and productivity (+30%) 23

Forum of the John von Neumann Society Industrial Success Stories Research - Studies - Methods NASA/SEL: code inspections are effective were (- 30/50% not lived! code def) Budapest, Hungary, 10 March 2004 ……. Defect Selby Study Cleanroom Cost study (inspection > Study (Basili (Boehm) testing) et al) Allianz: requirem. inspections are effective (- 50 % def) PBR studies (MD & IESE) Functional Stepwise Ab- Cleanroom Perspective 24 Semantics straction Reading Process based Read. (Mills) (Mills, Basili) (Mills et al) (Basili et al)

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Practical Examples (Inspections @ IESE) (3 of 4) • Further IESE work on inspections – investigation of effects in OO/UML environment (Laitenberger) • • defined PBR for OO/UML (packaging of reading unit across views controlled experiments – students at UKL (SE class) – PBR of requirements spec (UML) vs CBR – effectiveness & cost (PBR > CBR) • replication of existing (see NASA/SEL) studies in varying contexts (application domains, technology domains) – variation of existing studies to address new questions • optimal effort for preparation phase in inspection process (exists as demonstrated at Bosch; is used to manage inspection process) • Industrial relevance – helped establish inspections with sustained success in several companies (e. g. , Allianz, Bosch) – focus on inspections (with measurement-based feedback) matures development organizations (e. g. , Bosch unit with inspections went from CMM 1 to CMM 3 in one step!) 25

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Practical Examples (Design & Documentation @ IESE) (4 of 4) • IESE Studies on OO/UML (Briand, Bunse, Daly) – operationalized good design principles such as coupling, information hinding & cohesion – hypotheses: • • #1: “Good” OO designs are better understood – measured by the correctness of answers to a set of questions #2: Impact analysis on “good” OO designs is performed beter and faster – measured by the time & correctness of all changes to perform a set of given change requests – controlled experiments at UKL – 2 systems (“good”, “bad”); 2 x 2 factorial design – results • all results significantly in favor of “good” design • students made important self-experience regarding a set of engineering principles 26

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Agenda (for Research) (1 of 3) • SE Research results require “some form of evidence” – notations, techniques, methods & tools w/o evidence arenot accepted (e. g. , Ph. D theses) Partners for EU-No. E – collaboration with SE/CS experts on „SE Evidence“ • Current research focus (SE techniques) (SE, ESE)? – notations like Java, UML, … – architecture (specifically “product line architectures”) – processes such as inspection & testing, design requirements FROM – quality (reliability, security, safety) perspective – efficiency (cost, time) & effectiveness (“dependable SE”) • Current research focus (ESE methods) – Theory for “Evidence” Partners for EU-STREP on „Theory for Evidence“? 27

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Agenda (for Tech Transfer) (2 of 3) • Apply “ESE” as transfer vehicle to create sustained improvements • Use empirical studies to – evaluate major process-product relations prior to offering to industry (e. g. , in vitro controlled experiments) – method prototyping: Evaluate new methods together with industry experts in order to provide ROI potential insight for decision makers (e. g. , Ricoh, Bosch, German Telecom) – motivate candidate pilot project (developers & managers) with semi-controlled training experiment – evaluate pilot project (in vivo case studies) in order to adapt & motivate – continuously evaluate wide-spread use in order to motivate & optimize Without empirical evidence, no human-based process is lived! This has contributed to the grwoing gap between research & practice! 28

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Agenda (for Teaching & Training) (3 of 3) • Learning is based on • • • reading doing experiencing • Human-based engineering activities depend heavily on the latter approaches to learning • Teaching must reflect this – first analysis, then construction – perform “self-experience” experiments • At Technical University of KL/CS department – 1 st semester: NO programming (just reading & changing) – SE experiments (final UG class) • • • #1: Unit inspection more efficient than testing #2: Traceable design documentation reduces effort & risk of change #3. Informal (req) documents can be inspected efficiently (> 90%) – practical semester-long team projects with “data collection & process improvements” 29

Forum of the John von Neumann Society Budapest, Hungary, 10 March 2004 Outlook • SE has the chance to become a respected engineering discipline – automotive companies have more software than hardware engineers (since 2000) – mature software engineering includes empiricism (to create evidence) – system & service engineering require mature software engineering • Techncial University of Kaiserslautern – has leading laboratory settings for empirically driven software engineering research – is ranked #6 in “Software & Systems Eng Research” world-wide by JSS (12/03) – maintains international network (USA, Asia, Europe) – looks for international partners in SE & ESE (provided they accept experimental framework!) THANK YOU for your attention! QUESTIONS? 30

Скачать презентацию Forum of the John von Neumann Society Budapest

a6ee5eb6753639f4ca1a1ef7d78b4f57.ppt

Количество слайдов: 31