cb1cc925e6f5932fdc71decc0091ebb1.ppt
- Количество слайдов: 68
Requirements Engineering Lecture 11 Acceptance Testing Jerzy. Nawrocki@put. poznan. pl www. cs. put. poznan. pl/jnawrocki/re quire/ J. Nawrocki, Acceptance Testing Copyright, 2003 Jerzy R.
Introduction Test-driven development Acceptance tests: customer Automated accept. testing A. testing: customer+tester J. Nawrocki, Acceptance Testing Progress measurement based
Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing J. Nawrocki, Acceptance Testing
What is test automation? 1. Buy a test execution tool. That’s simple! 2. Record the manual tests. 3. Play the tests back whenever you want to. „Just as there is more to software design than knowing a programming language, there is more to automating testing than knowing a testing tool. ” J. Nawrocki, Acceptance Testing -- M. Fewster & D. Graham
Test activities Identify test conditions (‘what’ to test) and prioritize Design test cases (‘how’ to test) Build test cases (scripts, data etc. ) Execute test cases Compare test outcomes to expected outcomes J. Nawrocki, Acceptance Testing
Quality attributes of a test case • How effective in detecting defects? • How exemplary? (the more exemplary, the less test cases needed) • How economic? • How evolvable? (maintenance effort) J. Nawrocki, Acceptance Testing
Test automation effects Effective Evolvable Economic Exemplar y J. Nawrocki, Acceptance Testing
Example test conditions Item Price Quantit [zl] y Ros 2 5 e Bag 3 1 • Order created for a single item (VB 10) • Order quantity of 100 (VB 23) • Order cancelled (V 8) J. Nawrocki, Acceptance Testing
An example test case Prerequisites: • logged into the purchase order system as a data entry clerk; • database system must contain the standard Data Set; Ste Input Exp. out Cond. • no other new purchase order activity on the system. p 1 2 3 Create a new order Order confirmation for any one standard message displayed order, setting quantity to 100 Confirm the order Purchase order printed Print a new orders New orders report printed showing just this one new order J. Nawrocki, Acceptance Testing VB 10 VB 23
Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing J. Nawrocki, Acceptance Testing
The Scribble application 1 Netherlands Initial state New item 1 Netherlands added – end 2 Belgium 1 Belgium 2 Netherlands Enter: Belgium Sorted 1 Belgium 2 Denmark 3 Netherlands New item added in order 1 Denmark 2 Netherlands Old item deleted J. Nawrocki, Acceptance Testing Enter: Denmark Enter: 1
Manual test for Scribble countries. d cm Belgium Netherland s UK Add 2 new items to the sorted list. Move an item (the list gets unsorted). Add an item to the unsorted list. countries 2. d cm J. Nawrocki, Acceptance Testing Delete an item. Try to delete an invalid item (position)
The screen shots – Starting the Scribble File List J. Nawrocki, Acceptance Testing
The screen shots – Openning a file Scribble File List Ope n New Clos e Print Exit Scribble File List File Name countrie List Sorted s Countries on our mailing list are: Belgium Netherlands UK
J. Nawrocki, Acceptance Testing
The screen shots – Adding an item (Sweden) Scribble File List File Name countrie List Sorted New s List Countries on our mailing list are: Sort Add List Belgium Item Del Netherlands Item UK Mov Scribble
List File Item File Name countrie List Sorted s Countries on our mailing list are:
Belgium Netherlands Sweden UK
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Sorted s Countries on our mailing list are:
Belgium Netherlands UK Add Item
OK Sweden| Cance l
The screen shots – Adding an item (USA) Scribble File List File Name countrie List Sorted New s List Countries on our mailing list are: Sort Add List Belgium Item Del Netherlands Item Sweden Scribble Mov UK Item File List
File Name countrie List Sorted s Countries on our mailing list are:
Belgium Netherlands Sweden UK USA
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Sorted s Countries on our mailing list are:
Belgium Netherlands Sweden Add Item UK OK
USA| Cance l
The screen shots – Moving an item Scribble File List File Name countrie List Sorted New s List Countries on our mailing list are: Sort List Belgium Add Netherlands Mov Item Sweden Item Scribble UK Del File List Item USA File Name countrie List Unsorte
s d Countries on our mailing list are:
UK Belgium Netherlands Sweden USA
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Sorted s Countries on our mailing list are:
1 Belgium 2 Netherlands Move Item 3 Sweden Fro 4 4 UK OK m 5 USA To 1 Cance
l
The screen shots – Adding an item (Norway) Scribble File List File Name countrie List Unsorte New s d List Countries on our mailing list are: Sort Add UK List Item Del Belgium Scribble Item Netherlands File Mov List Sweden File Name countrie List Unsorte Item USA s d Countries on our mailing list are:
UK Belgium Netherlands Sweden USA Norway
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Unsorte s d Countries on our mailing list are:
UK Belgium Netherlands Add Item Sweden OK Norway| USA Cance
l
The screen shots – Deleting a valid item Scribble File List File Name countrie List Unsorte New s d List Countries on our mailing list are: Sort Del UK List Add Item Belgium Scribble Mov Item Netherlands File List Item Sweden File Name countrie List Unsorte USA s d Countries on our mailing list are: Norway
UK Netherlands Sweden USA Norway
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Unsorte s d Countries on our mailing list are:
1 UK 2 Belgium 3 Netherlands Delete Item 4 Sweden OK From Position 5 USA Cance 6 Norway 2 l
The screen shots – Deleting an invalid item Scribble File List File Name countrie List Unsorte New s d List Countries on our mailing list are: Sort Del UK List Add Item Netherlands Scribble Mov Item Sweden List File Item USA File Name countrie List Unsorte Norway s d Countries on our mailing list are:
Delete Error 1 UK 2 Netherlands Position Not 3 Sweden Valid 4 USA 5 Norway OK
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Unsorte s d Countries on our mailing list are:
1 UK 2 Netherlands 3 Sweden Delete Item 4 USA OK 5 Norway From Position Cance
7 l
The screen shots – Closing Scribble File List File Name countrie List Unsorte Open s d New Countries on our mailing list are: Close UK Print Netherlands Exit Sweden USA Norway
J. Nawrocki, Acceptance Testing Scribble File List File Name countrie List Unsorte s d Countries on our mailing list are:
Close 1 UK 2 Netherlands Save the i 3 Sweden document? 4 USA OK No 5 Norway
The screen shots – Exiting Scribble File List Ope n New Clos e Exit Print J. Nawrocki, Acceptance Testing
What was the test input? 1 Move mouse to Scribble icon 2 Double click J. Nawrocki, Acceptance Testing
What was the test input? Scribble File List J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu
What was the test input? Scribble File List J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click
What was the test input? Scribble File List Ope n New Clos e Print Exit J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option
What was the test input? Scribble File List Ope n New Clos e Print Exit J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click
What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm J. Nawrocki, Acceptance Testing
What was the test input? 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click J. Nawrocki, Acceptance Testing
What was the test input? Scribble File List File Name countrie List Sorted s Countries on our mailing list are: Belgium Netherlands UK
J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
What was the test input? Scribble File List File Name countrie List Sorted s Countries on our mailing list are: Belgium Netherlands UK
J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
What was the test input? Scribble File List File Name countrie List Sorted New s List Countries on our mailing list are: Sort Add List Belgium Item Del Netherlands Item UK Mov
Item J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
What was the test input? Scribble File List File Name countrie List Sorted New s List Countries on our mailing list are: Sort Add List Belgium Item Del Netherlands Item UK Mov
Item J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
What was the test input? Scribble File List File Name countrie List Sorted s Countries on our mailing list are: Belgium Netherlands UK Add Item
OK Sweden| Cance l J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
What was the test input? Scribble File List File Name countrie List Sorted s Countries on our mailing list are: Belgium Netherlands UK Add Item
OK Sweden| Cance l J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing J. Nawrocki, Acceptance Testing
The manual test process • Unscripted manual testing: 1. Identify conditions to test 2. Think up specific inputs 3. Enter the inputs 4. Checked it worked OK J. Nawrocki, Acceptance Testing
The manual test process • Unscripted manual testing • Vague manual scripts: 1. Read what to do 2. Think up specific inputs 3. Enter the inputs 4. Checked it worked OK J. Nawrocki, Acceptance Testing
Vague manual script 1 Run up Scribble File menu displayed 2 Open file with sorted List displayed File contents 3 Add some items to List Items added in order 4 Move an item unsorted Item moved, List now 5 Add an item List 6 Delete item form list Item added at end of Item deleted 7 Use invalid position number J. Nawrocki, displayed Acceptance Testing Error message
The manual test process • Unscripted manual testing • Vague manual scripts • Detailed manual scripts: 1. Read what to do 2. Enter the inputs 3. Checked it worked OK J. Nawrocki, Acceptance Testing
Detailed manual script 1 Click on Scribble icon Scribble opened, File menu enabled 2 Move mouse to File menu, click Options available: Open, New, Exit 3 Move mouse to Open option, Filenames displayed including click countries. dcm 4 Move mouse to countries. dcm Text plus List of 3 countries: Belgium, Netherlands, UK, File and List menus displayed. . . J. Nawrocki, Acceptance Testing. . .
Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing J. Nawrocki, Acceptance Testing
Manual or Automatic Testing? The amount of effort to automate a test case: 2 – 10 times the effort required to run the test manually (!) J. Nawrocki, Acceptance Testing 1 Move mouse to Scribble icon 2 Double click 3 Move mouse to File menu 4 Click 5 Move mouse to Open option 6 Click 7 Move mouse to countries. dcm 8 Double click 9 Move mouse to List menu
Introduction – What is Rational Robot? Robo t Baselin e J. Nawrocki, Acceptance Testing Baselin e
Introduction – Rational Robot and test-first coding Remarks: • When you start testing an application is ready. • Rational Robot is for regression testing, not for test-first coding. • The only chance for test-first coding is SQABasic. J. Nawrocki, Acceptance Testing
Testing HTML Applications Start. Browser [ URL$, ] [ Window. Tag= Name$ ] Web page address Identifies this instance of the browser • Starts the browser, • enables HTML testing, and • loads a page if specified. Start. Browser "http: //www. cs. put. poznan. pl/jnawrocki/models/", "Window. Tag= Parent" J. Nawrocki, Acceptance Testing
Testing HTML Applications Browser action$, rec. Method$, parameters$ Back, Forward, New. Page, Set. Frame, Close. Win, . . HTMLTitle=$, Name=$, "". . • Performs an action on a Web browser Browser New. Page, "" Wait for the topmost frame J. Nawrocki, Acceptance Testing Wait= % default time is 30 s
Testing HTML Applications HTML action%, rec. Method$, parameters$ Click, . . HTMLTitle=$, Name=$, "". . Coords= x, y • Performs a mouse action on an HTML tag. HTML Click, "Name= Autor", "Coords= 10, 12" J. Nawrocki, Acceptance Testing
SQABasic HTMLDocument Click, "Type=HTMLDocument; Index=1", "Coords=25, 14 " Result = HTMLTable. VP (Compare. Data, "Type=HTMLTable; Index=1", "VP=Weryfikacja_ daty") Test. DatastoreDefault. Test. Script. DatastoreTMS_Scriptvptest 7. Wer J. Nawrocki, Acceptance Testing yfikacja_daty. base. grd
An alternative to Rational Robot http. Unit Based on j. Unit For programmers! J. Nawrocki, Acceptance Testing
Data-driven scripts Data-driven scripting = Test inputs stored in a separate (data) file. countries, Sweden, USA, countries 2 countries, France, Germany, test 2 countries, Austria, Italy, test 3 J. Nawrocki, Acceptance Testing
Data-driven scripts Data-driven scripting = Test inputs stored in a separate (data) file. Control Open. File ‘Scribble. Data’ For each record in Scribble. Data Read INPUTFILE Read NAME 1 Read NAME 2 Read OUTPUTFILE Call Scribble. Open (INPUTFILE) Focus. On ‘Scribble’ Select. Option ‘List/Add Item’ Focus. On ‘Add Item’ Type NAME 1 Left. Mouse. Click ‘OK’. . . End. For script Scribble. Dat USA, countries 2 a countries, Sweden, countries, France, Germany, test 2 countries, Austria, Italy, test 3 J. Nawrocki, Acceptance Testing
A more sophisticated data-driven script Open. File ‘Scribble. Data’ Control Read INPUTFILE script Call Scribble. Open (INPUTFILE) Go to next record (ie row) For each record in Scribble. Data Scribble. Dat Read ADDNAME a If ADDNAME <> Blank Then { countries Focus. On ‘Scribble’ Select. Option ‘List/Add Item’ Swede Focus. On ‘Add Item’ n Type ADDNAME Left. Mouse. Click ‘OK} USA Read MOVEFROM 4 1 Read MOVETO. . . Norwa End. For y J. Nawrocki, Acceptance Testing 2 7
Data-driven scripts Additional automated regression test cases Software under test Key automated regression test J. Nawrocki, Acceptance Testing cases
Keyword-driven scripts Keyword-driven script = A data-driven scripts augmented with keywords representing user actions. SQABasic? ? ? Test Scribble. Open countries file Add. To. List Sweden USA Save. As countries 2 Control For each script TEST_ID Open. File TEST_ID For each record in test file Read KEYWORD Call KEYWORD End. For Close. File TEST_ID End. For J. Nawrocki, Acceptance Testing Supporting Scribble. Open scripts. . . Add. To. List. . . Save. As. . .
Shared scripts Shared script = A script used (shared) by more than one test case. SQABasic Call Scribble. Open (‘countries’) Focus. On ‘Scribble’ Select. Option ‘List/Add Item’. . . Call Scribble. Save. As (‘countries 2’) J. Nawrocki, Acceptance Testing
Automating test execution Automated scripts are not like manual scripts Don’t automate testing by simply recording Automated execution with manual verification? How much shoul be compared (whole, minimum, something between)? Dynamic comparison and post-execution comparison Automated comparison messages have to be manually checked While tests fail the second time? J. Nawrocki, Acceptance Testing
Presentation Contents Testing Life Cycle The Scribble Application The Manual Test Process Test Automation Use-Case-Based Acceptance Testing J. Nawrocki, Acceptance Testing
Test activities Identify test conditions (‘what’ to test) and prioritize Design test cases (‘how’ to test) Build test cases (scripts, data etc. ) Execute test cases Compare test outcomes to expected outcomes J. Nawrocki, Acceptance Testing
Test activities Identify important scenarios (criticality/frequency) and prioritize Instantiate the selected scenarios (1 -> *) Implement (scripts, data etc. ) Execute test cases Compare test outcomes to expected outcomes J. Nawrocki, Acceptance Testing
Use case goal levels Summar y Level Book trip Book flight User Goal Level Book hotel Book trip Book flight Reserve seat Find J. Nawrocki, Acceptance Testing flight Book hotel Find hotel Subfuncti on Level Reserve room
Test activities Identify important scenarios (criticality/frequency) and prioritize Instantiate the selected scenarios (1 -> *) Implement (scripts, data etc. ) Execute test cases Compare test outcomes to expected outcomes J. Nawrocki, Acceptance Testing
Test coverage Get. Software G 1: 0 Select. AProject G 1: 0 Get. Info. About. Projects G 1: 0 Register. ACustomer G 1: 1 Z 1: 1 Submit. AProposal G 1: 0 R 1: 0 U 1: 0 Scenari o J. Nawrocki, Acceptance Testing Total calls
Use-Case Example Success Guarantees: Requestor has goods, correct budet Guarantees ready do be debited. Trigger: Requestor decides to buy something. Trigger Main Success Scenario 1. Requestor: Initiate a request. Requestor 2. Approver: Check money in the budget, check price of Approver goods, complete request for submission. 3. Buyer: Check contents of storage, find best vendor for Buyer goods. 4. Authorizer: Validate Approver’s signature. Authorizer. . . Extensions 1 a. Requestor does not know vendor or price: leave those J. Nawrocki, Acceptance Testing parts blank and continue.
Test scenario Aktor: Reprezentant klienta Cel: Zarejestrowanie organizacji klienta w bazie danych SDS Pochodzi z: przypadku użycia Register. ACustomer [Reg. Cust] Autor: J. Nawrocki; Zmodyfikowano: 28. 11. 2003 Scenariusz: Główny scenariusz – G 1 Scenariusze pomocniczych przypadków: Warunki wstępne: Firma X nie jest zarejestrowana jako reprezentant klienta. Punkt wejścia: Przeglądarka internetowa (Internet Explorer 5. 0) Krok Pro 1 Akcja aktora Spodziewany rezultat Otwórz stronę SDS-u. Strona SDS jest otwarta. 1 Otwórz formularz rejestracji. 2 Wprowadź dane [Cust. Info] i wyślij je. Formularz otwarty [Req. Cust. Form]. Zostaje wyświetlony komunikat “Registration confirmed” [Req. Cust-Confirm]. Firmy J. Nawrocki, Acceptance Testing X
Test scenario – Screen shots Reg. Cust. Form J. Nawrocki, Acceptance Testing Reg. Cust. Confirm
Summary SDS test development: 1. Use cases 2. Manual test scenarios 3. Automated test scripts 4. Test cases sharing the scripts http. Unit or Rational Robot? J. Nawrocki, Acceptance Testing
Questions? J. Nawrocki, Acceptance Testing


