Osaka University An Experimental Comparison of Checklist-Based Reading

Osaka University An Experimental Comparison of Checklist-Based Reading and Perspective-Based Reading for UML Design Document Inspection Giedre Sabaliauskaite* Fumikazu Matsukawa** Shinji Kusumoto** Katsuro Inoue** *Graduate School of Engineering Science, Osaka University **Graduate School of Information Science and Technology, Osaka University

Osaka University Contents l Background l Research Goals l Experiment description Reading techniques Experimental design Data analysis Results l Conclusions and Further Work 2

Osaka University Software Inspection l Software inspection is an effective and efficient method to detect defects in early stages of software development life-cycle l Software inspection usually consists of several activities [1] Planning Defect detection Defect collection Key activity – defects are detected Reading techniques are used to guide inspectors Defect correction [1] O. Laitenberger, J. M. De. Baud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5 -31. 3

Osaka University Reading techniques l Ad hoc and Checklist-based Reading are the most popular [1] l Ad hoc: no guidance for inspectors l Checklist-based Reading (CBR): Checklist l Scenario-based Reading is a more recent approach: Scenario l Perspective-based Reading (PBR) [2] is one of Scenario-based approaches Software product is inspected from different perspectives (designer, tester, etc. ) Provides inspectors with different Scenario for each perspective l We analyzed CBR and PBR inspection techniques [1] O. Laitenberger, J. M. De. Baud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5 -31. [2] V. R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, M. V. Zelkowitz, “The Empirical Investigation of Perspective. Based Reading”, Empirical Software Engineering: An International Journal, vol. 1 , no. 2, 1996, pp. 133 -164. 4

Osaka University Related research l Several works are done in the area of Unified Modelling Language (UML) diagram inspection l One of the works is described in [1] Experimental comparison of CBR and PBR 18 subjects (practitioners) and 2 software systems UML diagrams inspected – Class, State, Sequence, Collaboration Results - PBR 3 -person teams exhibited higher effectiveness and lower cost per defect than CBR l Authors of different studies came to the teams conclusion that UML inspections need to be further investigated [1] O. Laitenberger, C. Atkinson, M. Schlich, K. El Emam. An experimental comparison of reading techniques for defect detection in UML design documents. The Journal of Systems and Software, 53: 183 -204, 2000. 5

Osaka University Research Goals l Conduct UML diagram inspection experiment in Osaka university Language used during experiment - Japanese l Compare Reading techniques CBR and PBR with respect to Defect detection effectiveness Cost per defect Time spent on inspection l Compare both individual inspector results and team results l Acquire experience for future inspection 6

Osaka University Structure of CBR and PBR techniques, used during experiment l CBR Diagram-specialized Checklist developed Included 20 questions l PBR l Three perspectives: User, Designer, Implementer l Three scenarios (one for each perspective) developed l Scenarios consisted of several steps, each of them included Explanations Tasks to perform Questions to answer 7

Osaka University Assignment of UML diagrams l All inspectors were given system requirement description and Use-Case diagrams l Defects were injected into Activity, Class, Sequence and Component diagrams Document PBR perspective Semina Hospit r al CBR Design Implemen User system er ter System requirement description 1 1 Use-Case diagrams 1 1 Activity diagrams 8 7 Class diagrams 1 1 Sequence diagrams 12 7 Component diagrams 1 1 8

Osaka University Experiment design l Subjects 59 third year students of the Software Development course, Osaka University l Objects Seminar information system ( 24 pages ) Hospital information system ( 18 pages ) PBR User Designer Implementer CBR Seminar system 7 6 6 11 Hospital system 7 6 6 10 9

Osaka University Defects l Three types of defects inserted into UML diagrams Syntactic – missing or unnecessary information Semantic – misrepresentation of a concept, or unclear representation Consistency – inconsistency between UML diagrams l In total, 15 defects inserted into the each software system 4 into Activity diagrams 3 into Class diagrams 5 into Sequence diagrams 3 into Component diagrams 10

Osaka University Experimental Hypotheses l H 01: subjects who use PBR technique should spend less time on inspection l H 02: subjects who use PBR should have higher cost per defect l H 03: individual defect detection effectiveness should be different between CBR and PBR inspectors l H 04: team defect detection effectiveness should be different between CBR and PBR inspector teams 11

Osaka University Data analysis Two types of analysis l Individual data analysis Defect detection effectiveness Time spent on inspection Cost per defect l Team data analysis Subjects assigned into 3 -person virtual teams Effectiveness of CBR and PBR teams compared 12

Osaka University Percentage of inspectors, who have detected defects relevant to different perspectives Both CBR and PBR exhibited similar results 13

Osaka University Defect detection effectiveness (according to defect types) Syntactic defects: CBR more effective Semantic and Consistency defects: PBR more effective 14

Osaka University Defect detection effectiveness (according to UML diagram types) Class Diagrams: CBR more effective Activity and Component Diagrams : PBR more effective Sequence Diagrams: CBR and PBR exhibited similar 15

Time spent on inspection, Cost per defect and Defect detection effectiveness Reading Osaka University Variables Time spent on inspection Cost per defect Effectiveness technique CBR PBR CBR Mean SD 62. 9 51. 3 6. 2 10. 2 70. 2 11. 7 15. 1 1. 6 3. 5 11. 5 PBR 69. 1 15. 3 CBR: 39% (4 min / PBR: 18% (11 min) Effectiveness defect) lower cost per less time spent on ≈PBR defect inspection Independent samples t-test • Significant difference in Effectiveness Inspection Time and Cost Mann-Whitney test 16 • No significant difference in Effectiveness

Osaka University Virtual teams l. Subject assignment into 3 -person virtual teams Any three CBR reviewers are included into team One reviewer using each of the three PBR perspectives included l. Grouping virtual teams into team statistically independent groups CBR groups: 3 teams included (11 inspectors for Seminar system, 10 subjects for Hospital system) PBR groups: 6 teams included (7 Users, 6 Designers, 6 Implementers for each systems) CBR group C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 9 PBR group U 1 D 1 I 1 U 2 D 2 I 2 U 3 D 3 I 3 U 4 D 4 I 4 U 5 D 5 I 5 U 6 D 6 I 6 17

Osaka University Comparisons between CBR and PBR teams C 4 C 5 C 6 C 7 C 8 C 9 CBR group 1 C 2 C 5 C 6 C 7 C 8 C 10 D 1 I 1 U 2 D 2 I 2 U 6 D 6 l Seminar system PBR group 1 U 1 D 1 I 6 U 2 D 2 l Hospital system I 1 … … U 6 D 6 I 5 … CBR group 2 Number of comparisons: I 6 C 3 C 4 U 1 … C 3 … C 2 PBR … C 1 Comparisons CBR PBR group 2 … 18

Osaka University Team data analysis (1/2) l CBR and PBR teams were compared with respect to Team defect detection effectiveness l Number or comparisons, in which either CBR or PBR was more effective, or both techniques showed similar effectiveness Software system Team defects detection effectiveness CBR>PBR PBR>CBR CBR=PBR Seminar 54 753 326 688 585 743 712 544 449 600 Hospital 10 160 482 176 8 064 149 760 CBR teams more effective than PBR teams 19

Osaka University Team data analysis (2/2) l T-test with significance level of 2. 5% was used to evaluate in which comparisons there was a significant difference between CBR and PBR team effectiveness l Two types of comparisons made All defects included Syntactic defects omitted Defect detection effectiveness Software system CBR > PBR > CBR = PBR All defects included Seminar 7631557968 0 0 Hospital 7233873848 0 0 Syntactic defect omitted Seminar 6968160000 0 0 Hospital 85122000 0 0 Comparison type CBR teams more effective than PBR teams 20

Osaka University Experiment results l Individual inspector results Inspectors who use PBR spend on average 18% less time on inspection Cost per defect of Inspectors who use PBR is on average 39% higher There is no statistically significant difference in defect detection effectiveness between individual CBR and PBR inspectors, however on average • • PBR is more effective for Semantic and Consistency defects PBR is more effective for Activity and Component diagrams CBR is more effective for Syntactic defects CBR is more effective for Class diagrams l Three-person virtual team results CBR teams exhibit higher defect detection effectiveness than PBR teams 21

Osaka University Threats to validity l Internal validity Selection: experiment was a mandatory part of the course Instrumentation: objects which we used could have influence l External validity Students were used as subjects Design documents were similar to those which are used in practice, but the size of systems in industry is usually larger l There were threats to internal and external validity, but they were not considered large in 22

Osaka University Conclusions and Further research l UML diagram inspection experiment was conducted l The results of the experiment indicate that There is no statistically significant difference in effectiveness of CBR and PBR individual inspectors CBR virtual teams are more effective than PBR virtual teams l Future research will be directed to further investigation of UML diagram inspection A replication of experiment was conducted in July 2002, during which real team meeting were performed More detail analysis of both individual and team data 23

Osaka University Software Engineering Laboratory http: //iip-lab. ics. es. osaka-u. ac. jp/ 24

Osaka University ADDITIONAL SLIDES: 26 -32 25

Osaka University Experiment Variables and Hypotheses Independent variables Null Hypotheses l Reading technique (CBR and PBR) l For individual inspectors Dependent variables l Number of defects found l Time spent on inspection l Defect detection effectiveness l Cost per defect H 01: PBR Time > CBR Time H 02: PBR Cost < CBR Cost H 03: PBR Effectiveness = CBR Effectiveness l For 3 -person virtual teams H 04: PBR Effectiveness = CBR Effectiveness 26

Osaka University Information systems inspected l Seminar information system System supports the process of a company which organizes seminars Includes activities from seminar planning to issuing the qualification certificate Designs relationships among personnel of seminar company, participants, lecturers, etc. l Hospital information system System supports the process of a hospital Includes activities such as patient registration, medical examination, treatment, prescription of the medicines, etc. Designs the relationships among the personnel of a hospital, patients, pharmacists, etc. 27

Osaka University Collected data l Two types of data collected during experiment Defect data Time data 　　 Number of inspecto rs 　 Inspection time (min) Defects Number of detectabl e defects Average number of defects detected max/mi n Average max/min PBR User 4. 43 6/3 60. 43 90 / 46 PBR Designer 6 6 5. 00 6/4 65. 50 80 / 51 PBR Implementer 6 9 6. 50 9/5 76. 67 95 / 40 11 15 10. 55 13 / 8 74. 64 90 / 62 PBR User Hospital informatio n system 7 CBR Seminar informatio n system 7 7 7 4. 43 6/3 48. 29 70 / 25 PBR Designer 6 6 3. 83 5/3 59. 17 73 / 30 PBR Implementer 6 9 6. 33 7/5 63. 30 77 / 44 28

Osaka University CBR Checklist CHECKLIST Locate the following diagrams: Class Diagrams, Activity Diagrams, Sequence Diagrams and Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form. … SEQUENCE DIAGRAMS 10. Aren’t there any inconsistencies among Sequence Diagram, Class Diagram and Requirements Specification? yes no 11. Are all the necessary Objects and Messages defined? yes no 12. Is every Class from Class Diagram included in any of Sequence Diagrams and vice versa? yes no 13. Is every Use Case from Use-Case Diagram implemented in at least one of Sequence Diagrams? yes no 14. Are there no redundant elements (Objects of Messages) in Sequence Diagram? yes no … 29

Osaka University PBR Scenario DESIGNER SCENARIO Assume that you are the Designer of the system. The concern of the designer is to ensure that the designer’s needs are completely satisfied in the following documents: Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designer’s point of view. Please perform tasks following Step 1 through Step 3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form. Step 3 Locate the Class Diagram and Sequence Diagrams Make a list of all Objects included in Sequence Diagrams and answer the following questions. 3. 1. Are all the Objects of Sequence Diagrams defined in Class Diagrams? Are all Classes of Class Diagrams defined in Sequence Diagrams? 3. 2. Are all the Messages between objects Sequence Diagram defined as Methods of the corresponding Class in Class Diagram? 3. 3. Are there no redundant elements in Sequence diagrams? 30

Osaka University Defect registration form 31

Osaka University Software development process l The main steps of software development process 1. 2. 3. 4. Development of Use-case diagrams Describing system activities in Activity diagrams Defining static structure of the system in Class diagrams Modelling dynamic aspects of the system in Sequence diagrams 5. Detailed description of object states in Statechart diagrams 6. Development of the Component diagrams l User’s perspective in our experiment covered the second, and partially the third and the fourth steps l Designer’s perspective covered the third and the fourth steps l Implementer’s perspective covered the sixth step, and 32