Test process analysis of Gateway GPRS Support Node

Test process analysis of Gateway GPRS Support Node Author: Jorma Tuominen Supervisor: Professor Sven-Gustav Häggman Nokia Networks

Agenda • Introduction • • • GGSN in GPRS and 3 G packet core network GGSN in Nokia Intelligent Content Delivery system Software development process and testing phases • Research problem • Research method • Results • Conclusions 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 2 (15)

GPRS and 3 G packet core network Mobile ISP News Trading Sports Games WAP MMS BTS NMS BSC CG LIG 2 G SGSN IP backbone BS RNC FW GGSN 3 G SGSN FW BG Inter-PLMN network 17. 1. 2006 Intranet services Internet Jorma Tuominen Test process analysis of Gateway GPRS Support Node Corporate customer 3 (15)

Nokia Intelligent Content Delivery System ICD CG GTP’ OSC LDAP NSM GGSN LDAP user data TA user data CA user data RADIUS or DIAMETER user data + RADIUS 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 4 (15)

SW development process and testing phases 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 5 (15)

Research problem Testing requires tradeoffs between cost, time and quality. How available testing time and resources can be used as effectively as possible to find and correct the critical defects from the product before it is released? 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 6 (15)

Research method • Participating in GGSN system testing • Literature study of GPRS and 3 G packet core network elements and protocols • Literature study of software process and software testing • Collecting and analyzing testing metrics • Defect analysis • Risk analysis • Planning risk based testing 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 7 (15)

Testing metrics • Testing effort metrics • Distribution of working hours by test phases • Defect metrics • • Number of discovered defects per week Percentage of “correction not needed” defect reports • Test case metrics • • Number of test cases Percentage of not relevant test cases • Testing effectiveness metrics • • • 17. 1. 2006 Hours needed to discover one defect Number of test cases needed to discover one defect Defect Removal Effectiveness Jorma Tuominen Test process analysis of Gateway GPRS Support Node 8 (15)

Testing metrics examples • Distribution of working hours by test phases (lessons learned) • Number of discovered defects per week (real-time metrics) 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 9 (15)

Defect analysis • Search for the error prone functionality areas • • • Defects discovered in the earlier product releases Defects discovered in the earlier testing phases Defects discovered by the customers • Can be done after product release or during testing • Pareto 80 -20 phenomena • • 17. 1. 2006 Many software phenomena follow a Pareto distribution: 80% of contribution comes from 20% of the contributors [Boehm 1989] Examples: • 20% of the software modules contribute 80% of the errors. • 20% of the errors cause 80% of the down time Jorma Tuominen Test process analysis of Gateway GPRS Support Node 10 (15)

Risk analysis • System requirements must be prioritised • What is the likelihood that this feature will fail to operate? • What would be the impact on the user if this feature will fail to operate? • The likelihood of failure depends on the following: • • • error proneness of the requirement newness of the requirement complexity of the requirement • Likelihood of failure: 1 = very high, . . . 5 = very low • Impact on the user: 1 = severe, . . . 5 = negligible • Risk priority = (Likelihood of failure) x (Impact on the user) 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 11 (15)

Risk based testing Risk 100% Risk based testing Traditional testing 50% E 17. 1. 2006 Jorma Tuominen E 2 1 Testing effort Test process analysis of Gateway GPRS Support Node 12 (15)

Results • Requirements have been divided into three groups: • High, medium and low risk requirements • Test case titles will be planned for each requirement • Test cases will be designed and executed as follows: • • • High risk requirements: Medium risk requirements: Low risk requirements: 100% of test cases 70% of test cases 30 % of test cases • Test design and execution effort can be reduced about 30%. • Effect on the quality of the product can not be estimated yet, because the product is not yet released. 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 13 (15)

Conclusions and future work • The availability of the real-time testing metrics should be improved. • Co-operation between the product development, the testing and the customer support should be strengthened. • The most important future work is to harmonize the usage of requirement management database, test case database and defect tracking database. 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 14 (15)

Thank You! Questions? 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 15 (15)

Additional slides 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 16 (15)

Cost of defects Phase where defect is discovered Requirements Relative cost to correct a defect 1 Design 3 -6 Coding 10 Development testing 15 -40 Acceptance testing 30 -70 Production 40 -1000 Phase where defect is discovered Relative cost to correct a defect Definition High-level design 5 Code 10 Unit test 15 Integration test 22 System test 50 Post delivery Jorma Tuominen 2 Low-level design 17. 1. 2006 1 100+ Test process analysis of Gateway GPRS Support Node 17 (15)

Early involvement of the testers • Important aspects of early involvement of the testers are: “Testers need a solid understanding of the product so they can devise better and more complete test plans, designs, procedures, and cases. Early testteam involvement can eliminate confusion about functional behavior later in the project lifecycle. In addition, early involvement allows the test team to learn over time which aspects of the application are the most critical to the end user and which are the highest-risk elements. This knowledge enables testers to focus on the most important parts of the application first, avoiding over-testing rarely used areas and under-testing the more important ones. ” [Dustin 2003]. 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 18 (15)

Thoughts about testing • “Program testing can be used to show the presence of bugs, but never to show their absence!” [Dijkstra 1969]. • Testing completion criteria: Risk of stopping testing = Risk of continuing testing • “In most cases ‘what’ you test in a system is much more important than ‘how much’ you test” [Craig 2002]. • “Prioritise tests so that, when ever you stop testing, you have done the best testing in the time available” [ISEB 2003]. • No risk No test 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 19 (15)

References • Myers G. The art of software testing, USA, John Wiley & Sons, ISBN 0 -471 -04328 -1, 1979, 177 p. • Hetzel B. Complete Guide to Software Testing, USA, John Wiley & Sons, ISBN 0 -471 -565679, 1988, 284 p. • Craig R. , Jaskiel S. Systematic Software Testing, USA, Artech House Publishers, ISBN 158053 -508 -9, 2002, 536 p. • Dijkstra E. W. Structured programming. Proceedings of the Second NATO Conference on Software Engineering Techniques, Rome Italy, NATO, 1969, pp. 84 -88. • Dustin E. Effective Software Testing, USA, Addison-Wesley, ISBN 0 -201 -79429 -2, 2003, 271 p. • Bender R. Requirements Based Testing presentation. Proceedings of Star. East 2005, Software testing analysis & review conference, USA Orlando, Bender RBT Inc. 2005, 232 p. • Beizer B. Black-Box Testing: Techniques for Functional Testing of Software and Systems, USA, John Wiley & Sons, ISBN 0 -471 -12094 -4, 1995, 294 p. • Koomen T. , Pol M. Test Process Improvement: A practical step-by-step guide to structured testing, Great Britain, Addison-Wesley, ISBN 0 -201 -59624 -5, 1999, 218 p. • Hutcheson M. Software Testing Fundamentals: Methods and Metrics, USA, John Wiley & Sons, ISBN 0 -471 -43020 -X, 2003, 408 p. • Boehm B. W. Software Risk Management, USA, IEEE Press, ISBN 0 -8186 -8906 -4, 1989, 495 p. • Grove Consultants, ISEB testing foundation course material, Great Britain, Grove Consultants, 2003, 98 p. 17. 1. 2006 Jorma Tuominen Test process analysis of Gateway GPRS Support Node 20 (15)