Testing Techniques Conformance Testing Methodology and Framework ISO

Testing Techniques Conformance Testing Methodology and Framework ISO IS-9646

Overview l Conformance Testing Methodology and Framework ISO IS-9646 Literature: “An Overview of OSI Conformance Testing” © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 2

Overview: This Lecture l Protocol conformance testing: why and what l Standardization l Overview Standard IS-9646: Conformance Testing Methodology and Framework n n n n Conformance: static and dynamic Options, protocol implementation conformance statement Test generation: test purposes, abstract-, executable tests Test methods, test architectures, Test classification and -structuring Test notation Test implementation and -execution © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 3

Types of Testing © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 4

Protocol Conformance Testing distributed systems exchange of information rules for communication : protocol standardized protocol specifications conforming implementations testing for protocol conformance © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 5

Protocol Conformance Testing International Standard ISO IS-9646 : Conformance Testing Methodology and Framework l How to test protocol entity implementations for conformance with respect to their specifications l Originally for OSI systems l Now more generally applied l Defines test language TTCN l Standard accepted by ITU-T, ETSI, ATM-Forum, …… © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 6

Protocol Conformance Testing l Starting point : a correct and valid (standardized) protocol specification l Original goal : standardization of test suites in TTCN n consistency between specification and test suite ? l Conformance testing interoperability testing © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 7

The Conference Protocol © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 8

The Conference Protocol join leave send receive CPE CPE Conference Service UDP Layer © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 9

Conference Protocol Test Architecture Tester Tor. X A UT-PCO = C-SAP CPE = IUT U-SAP B C LT-PCO UDP Layer © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 10 LT-PCO

Protocol Conformance Testing vs. Interoperability Testing (N)-ASPs (N)-service provider (N)-Protocol Entity (N)-PDUs protocol (N)-Protocol Entity (N-1)-ASPs (N-1)-service provider © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 11

Protocol Conformance Testing vs. Interoperability Testing l Protocol Conformance Testing: n correct protocol (design and validation) n conforming protocol entities (testing) interoperability l Interoperability testing n test directly whether each pair of protocol entities can work together (i. e. deliver (N)-service) l Internet: n interoperability testing with (standard) reference entities © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 12

Standard Conformance Testing l Testing by: n manufacturers n users organizations n telecom operators n independent test laboratories l Standard IS-9646 for n certification of implementations n accreditation of testing processes and test laboratories n international acceptance of test results © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 13

ISO IS - 9646 l Part 1 : Introduction and general concepts l Part 2 : Abstract test suite specification l Part 3 : TTCN Tree and Tabular Combined Notation l Part 4 : Execution of tests l Part 5 : Requirements on test laboratories and administrative conformance assessment process © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 14

standard protocol specification test generation implementation process standard conformance test suite protocol implementation IUT test implementation executable test suite test execution process of conformance testing © Ed Brinksma/Jan Tretmans verdict TT 2004, ISO Conformance 15

Conforming Implementation l Specifications in natural language l What is a conforming implementation ? l “ … exhibits conformance if it complies with the conformance requirements of the applicable specification standard …” l Satisfaction of conformance requirements l Conformance requirements : “the implementation shall do …. . “ “the implementation shall not do …. . ” l Complication : options © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 16

Conforming Implementation l (P)ICS (Protocol) Implementation Conformance Statement : n list of implemented options n supplied by manufacturer l (P)ICS proforma n fill-in questionnaire with all possible options n attachment to protocol standard n to be completed by manufacturer to make PICS l Restrictions on combinations of implemented options n documented in standard in static conformance requirements © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 17

Example : Static Conformance Requirement l Feature A is optional l Feature B is optional l But A may only be implemented if B is also implemented © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 18

Conforming Implementation l Requirements on dynamic behaviour: dynamic conformance requirements l Messages : n structure n coding l Ordering : n sequence of sent and receive messages n relation between information content of different messages l Example: n When in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase. © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 19

Conforming Implementation l A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the PICS l Conformance testing: n checking static conformance by reviewing n checking dynamic conformance requirements by testing © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 20

Test Generation dynamic conformance requirement test purpose(s) generic test case abstract test case executable test case execution © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 21

Test Generation l Test purpose : precise description (natural language) of what is to be tested in order to decide about satisfaction of a particular conformance requirement l Example: n When in state Enable, upon receiving a Data-Request, the implementation shall send a Data-PDU or it shall enter the abort phase. n Check whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. (but implementation may also enter the abort phase!) © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 22

Test Generation l Generic test case: operationalization of test purpose; description of actions to be performed to reach test purpose l Example: n Check whether in state Enable, after receiving a Data-Request, a Data-PDU is sent. n Go to state Enable, send a Data-Request, and observe a Data-PDU, and if not check that the implementation starts entering the abort phase © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 23

Test Generation l Abstract test case: take test method into account l Test method ( test architecture, test environment ) : abstract description of environment in which testing will be performed, in terms of PCO - Points of Control and Observation l Example: Do Connect-Request, check Connect-Response (now in state Enable), send Data-Request observe Data-PDU or observe Start-Abort © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 24

Test Methods Local test method UT PCO test coordination (N)-ASPs IUT PCO (N)-PDUs (N-1)-ASPs LT © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 25

Test Methods Distributed test method test system SUT test coordination UT LT PCO (N)-ASPs (N)-PDUs PCO IUT (N-1)-ASPs service provider © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 26

Test Notation l Abstract test cases are written in standardized notation : TTCN : Tree and Tabular Combined Notation l Specification language for Abstract Test Cases l Behaviour specified as tree of sequences of possible events l Data specified in tables or in ASN. 1 l ISO IS-9646 Part 3 l With: input ? , output ! (from tester perspective!) , timers / time out, ordering of events, default behaviour, alternative behaviours, tree attachment (=subroutines), parallel component testers, verdicts: pass, fail, inconclusive © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 27

Example : TTCN Test Case Dynamic Behaviour Test Case Name: Conn_Estab Group: transport/connection Purpose: Check connection establishment with remote initiative behaviour constraints +preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance verdict pass fail inconclusive fail 28

Classification of Tests l Basic interconnection tests l Capability tests l Behaviour tests l Conformance resolution tests © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 29

Structuring of Test Suites Test suite Test group Test case Test step Test event PCO ? X © Ed Brinksma/Jan Tretmans PCO ! A TT 2004, ISO Conformance 30

Test Implementation l Transform abstract test suite into executable test suite by n compilation n interpretation n a test slave …. . l Select test cases from abstract test suite based on PICS l Incorporate (P)IXIT (Protocol) Implementation e. Xtra Information for Testing : n address information n timer information n specific parameter values n …. . l PIXIT proforma supplied by test laboratory © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 31

Test Execution l Execute / interpret executable test suite l Collect and analyse the results based on verdicts pass, fail, inconclusive of each test case execution l Collect all results of n static conformance review n dynamic testing into Protocol Conformance Test Report © Ed Brinksma/Jan Tretmans TT 2004, ISO Conformance 32

standard protocol specification implementation process dynamic conformance req’s test purposes static conformance req’s generic test suite PICS proforma PIXIT proforma (standardized) abstract test suite protocol implementation IUT Basic interconn. capability behaviour PICS PIXIT test selection static conformance review test implementation test execution analysis of results executable test suite verdict test report © Ed Brinksma/Jan Tretmans certification TT 2004, ISO Conformance 33 standardized test methods test notation