Use Case-Based Acceptance Testing of a Large Industrial

Use Case-Based Acceptance Testing of a Large Industrial System: Approach and Experience Report Dr. S. Roubtsov, Ir. P. Heck TAIC PART 2006 Cumberland Lodge, Windsor August, 31 2006 La. Qu. So is an activity of Technische Universiteit Eindhoven

Outline l E-Ticketing System for the Netherlands l La. Qu. So Assignment l Field Survey l Approach l Test Preparation l Test Execution l Conclusion: Lessons Learned Copyright © La. Qu. So Eindhoven 2006

Dutch E-Ticketing System: overview Initialization & Personalization Centre Central Processing System of Public Transport Organization Central Clearing House System Bank E-ticket Front-end Device Copyright © La. Qu. So Eindhoven 2006 Station Processing System - main transaction flows

Dutch E-Ticketing System: architecture Central Back Office Central Clearing House Clearing Operator (CO) & Card Issuer (CI) Bank E-Ticketing System Initialization and Personalization Center La. Qu. So Assignment Field Survey at TLS Level 4 CPS PTO 1 … … Approach SPS Test Preparation Test Execution … SPS … Front-end devices … Conclusion … CPS PTO n … … SPS … Level 3 SPS … Front-end devices … E-tickets - smart cards card usage transaction flow blacklists bank interface files Copyright © La. Qu. So Eindhoven 2006 settlement reports Level 2 Level 1 Level 0

Involved Parties E-Ticketing System La. Qu. So Assignment Field Survey at TLS Approach Test Preparation Test Execution Conclusion Customer: l Trans Link Systems (TLS): NS (national railway), Connexxion, GVB (Amsterdam), RET (Rotterdam), HTM (The Hague) ~90% public transport market Supplier: l East West Consortium (EW): Accenture, Thales, Vialis, MTR & Octopus Cards Ltd Hong Kong prototype system Supplier was responsible for Site Acceptance Testing Copyright © La. Qu. So Eindhoven 2006

La. Qu. So Assignment E-Ticketing System La. Qu. So Assignment Field Survey at TLS Approach Test Preparation Test Execution Conclusion To take part in SAT of CBO (Level 4): l to participate in SAT planning l to validate completeness and correctness of requirements coverage by tests l to assess test specification and reporting documents l to witness tests and evaluate their results Copyright © La. Qu. So Eindhoven 2006

Field Survey: User Requirements Sources E-Ticketing System l Contract La. Qu. So Assignment l Business rules document l Conceptual design l High level system design l Processes supporting the system Field Survey at TLS Approach Test Preparation Test Execution Conclusion l l Some of the documents were not finalized Requirements were not listed formally Copyright © La. Qu. So Eindhoven 2006

Requirements for Testing l Use E-Ticketing System La. Qu. So Assignment Field Survey at TLS l System and processes had to be tested separately (two testing teams) Approach Test Preparation l No connection with levels 0 to 3 during testing (use of simulators) Test Execution Conclusion case-based approach l Compliance with IEEE 829 standard Copyright © La. Qu. So Eindhoven 2006

Results of Field Survey l Testing E-Ticketing System La. Qu. So Assignment Field Survey at TLS Approach Test Preparation Test Execution Conclusion approach needed to be elaborated l User requirements had to be retrieved from the project documentation and listed formally l Changes in documentation would lead to multiple iterations of requirements adjustment and tracing Copyright © La. Qu. So Eindhoven 2006

Approach: Problem statement E-Ticketing System La. Qu. So Assignment Field Survey at TLS Approach Little data in literature as to how to apply use case based approach to large scale systems n use cases would contain very coarse steps: how to convert them into precise test cases? Test Preparation n almost all triggers and some of the steps were at levels 0 to 3 (out of scope): how to adjust them to level 4? n could be too many test cases with similar control flows: how to reuse test procedures? Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006

Approach: Three-Level Specification l Test E-Ticketing System La. Qu. So Assignment Scenarios: end-to-end use case scenarios Field Survey at TLS Approach Test Preparation l Test Scripts: procedures describing how to perform testing of relevant steps of test scenarios Test Execution Conclusion l Test Cases: instances of test scripts with test data Copyright © La. Qu. So Eindhoven 2006

Test Scenario Example Buy PTO-IO product ID: TS. COP. 08 Actors: Customer, Product Retailer (PR), Product Owner (PO), Card Issuer (CI), Clearing Operator (CO) Preconditions (Input): E-Ticketing System La. Qu. So Assignment Field Survey at TLS Card master ID is created during the card production Card is authorized and usable in the system Product profile is registered and valid (effective and not expired) in level 4 CCHS Main scenario Approach 1. Customer requests for PTO-I/O product and fills in a form for certain products. 2. PR (POST) performs the following checks before product loading: • product availability, for example, effective date of product profile; • card type support, for example, P-Card only; • customer availability, for example, age limitation. Customer pays for the product. PR loads the product in the card: • logs in as operator under “commercial shift” on the POST terminal; • holds card in the reader and select the preferred screen; • flags the required product; • accepts transaction. PR (system) sends “Product Sales” transaction to CO via Level 3 CPS. CO (system) performs day-time transactions validations. CO (system) forwards the validated transaction to corresponding CI and Cross Selling Product Sale (CSPS) transaction to PO (CSPS transaction means that PR is not equal to PO. As a result, PO doesn’t have the sale information. In order to forward such transactions to corresponding PO, CO consolidates and generates transaction files for PO): • CO consolidates “Not on us” product transactions and generates product transaction files for individual PO periodically; • PO retrieves consolidated product transaction files from file server of CO. CI (system) performs day-time transaction validations. CI (system) updates Card Master. PO retrieves consolidated product transaction files from file server of CO System before End-Of-Day process. Test Preparation 3. 4. Test Execution Conclusion 5. 6. 7. 8. 9. 10. Copyright © La. Qu. So Eindhoven 2006

Test Scenario Example (Cont. ) Post-conditions After point 3, product is loaded in the card and it can be used if product is effective. E-Ticketing System Product transactions file is stored in CCHS before forwarding to individual PO during end of day process. Card Master is updated. La. Qu. So Assignment Variations 1. Buy single PTO product (No cross selling in this case). 2. Buy / load product at TVM (selling machine). (Only certain products, for which no form has to be filled in, can be bought at TVM. ) Field Survey at TLS Approach Exceptions 1. CO day-time transaction validation fails (exceptions are captured in the exception report). 2. CI day-time transaction validations fails (exceptions are captured in the exception report). 3. Transactions are not received by CO. 4. Wrong product loaded on the card. 5. Quality problem with the card. 6. Card is blacklisted. 7. Device is blacklisted. 8. Sent transactions are not received by Card Master. 9. Sent transactions are not correctly updated in Card Master. 10. Product loading fails because of malfunctioning device (POST / TVM). 11. Product loading fails because of faulty cards. Test Preparation Test Execution Conclusion Related test scenarios 1. Buy a card with PTO product (preloaded). 2. Perform Day-time validation (CO/CI). Copyright © La. Qu. So Eindhoven 2006

Relations between Test Scenarios E-Ticketing System La. Qu. So Assignment Field Survey at TLS

Test Script Items La. Qu. So Assignment Field Survey at TLS Approach Test Preparation Test Execution Conclusion ID l Reference to the parent test scenario/variation/exception l Description of (”how to do”) each step of the parent main scenario / variation / exception l Input data types l Pass and fail criteria l E-Ticketing System Copyright © La. Qu. So Eindhoven 2006

Test Case Items E-Ticketing System l ID La. Qu. So Assignment l Reference Field Survey at TLS Approach Test Preparation to the parent test script l Steps from the parent test script l Input data values for each step l Expected results for each step Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006

Requirements Coverage: Traceability Model E-Ticketing System La. Qu. So Assignment Field Survey at TLS

Documentation Traceability Model E-Ticketing System La. Qu. So Assignment Requirement has to be mentioned in at least one document Field Survey at TLS Approach Test Preparation Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006

Test Scenarios Reviewing Error type E-Ticketing System Effect on test scripts/cases La. Qu. So Assignment Missing preconditions Missing input data Field Survey at TLS Missing steps in scripts Conditionals (”if”. . . ”then”. . . ”else”) Missing scripts and/or cases presented by alternative paths Actor of a step is unclear Impossible to tell user input from system reaction Missing postconditions Missing/wrong pass/fail criteria Missing variations/exceptions Missing scripts and/or cases Approach Test Preparation Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006

Test Scripts/Cases Reviewing Test Scenario Name Test Scenario ID La. Qu. So Assignment Field Survey at TLS S t e p TS. COP. 08 Test Script ID E-Ticketing System Buy PTO – IO product TSC. COP. 08 Main scenario O p s L 0 / 3 Op s L 4 T L S O p s L 4 E W S y s t e m Approach 2 Ö Test Preparation Test Execution 3 Ö 4 Ö Test Script Description Test Data Specification Actions Ö 6 a Ö 7 Ö 8 9 Ö Ö 10 Ö Elaborate steps Test data and/or definitions IDs of test and values scripts reused by this one 5 6 Copyright © La. Qu. So Eindhoven 2006 Expected Results L 4 Conclusion steps of test scenario Script ID system SAT related steps Output data and pass criteria Ö Test Cycle ID Testing procedure plus input data 1 L 0 / 3 S y s t e m reference to test scenario

Test Execution: System Testing l IPC (Card Initialization & Personalization Center) E-Ticketing System La. Qu. So Assignment n incomplete coverage of ‘Dutch delta’ u ‘ghost’ cards u data input validation failure u Field Survey at TLS Approach Test Preparation Test Execution Conclusion many issues, e. g. : l CCHS (Central Clearing House) n Rigid sets of test data were generated by simulators. Tests were very well rehearsed. Few minor bugs were found Copyright © La. Qu. So Eindhoven 2006

Test Execution: Operations Testing l IPC E-Ticketing System La. Qu. So Assignment n Field Survey at TLS data gaps in supporting forms, reports, etc. u poorly defined interfaces between EW and TLS processes u Approach Test Preparation Test Execution Conclusion System tests were reused with emphasis on supporting processes l CCHS n Free format (negative, boundary, etc. ) testing through user interface u more than 130 issues (system and operations, about 30% of critical and high severity) Copyright © La. Qu. So Eindhoven 2006

Test Results l During 3 months all critical and high level issues, as well as a half of the medium ones were resolved l 94% of the listed requirements were covered. The rest (network, security, bank interface, etc. ) could not be tested within the level 4 in isolation l E-Ticketing System The pilot system (all levels) was launched in April 2005 in Rotterdam region La. Qu. So Assignment Field Survey at TLS Approach Test Preparation Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006

Conclusion: Lessons Learned l l Three level use case based approach is feasible for large systems SAT has to be performed independently of the supplier. The tester should be either the customer or a third party l Doing acceptance testing rely on people who is going to run the system l La. Qu. So Assignment Use case based approach is suitable for SAT l E-Ticketing System Modern requirement-management tools can improve requirements traceability and SAT efficiency drastically Field Survey at TLS Approach Test Preparation Test Execution Conclusion Copyright © La. Qu. So Eindhoven 2006