Скачать презентацию Testing C 2 Interoperability Advancements in Testing of Скачать презентацию Testing C 2 Interoperability Advancements in Testing of

565699307ee4273d485103212a5a415f.ppt

  • Количество слайдов: 30

Testing C 2 Interoperability Advancements in Testing of the MIP Baseline 3 Solution Nico Testing C 2 Interoperability Advancements in Testing of the MIP Baseline 3 Solution Nico Bau, Michael Gerz & Michael Glauer FGAN FKIE 13 th ICCRTS Bellevue, WA June 17 th-19 th, 2008 FGAN Research Institute for Communication, Information Processing and Ergonomics

Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics

Goals n The aim of the Multilateral Interoperability Programme (MIP) is to achieve international Goals n The aim of the Multilateral Interoperability Programme (MIP) is to achieve international interoperability of Command Control Information Systems (C 2 IS) at all levels from corps to battalion, or lowest appropriate level, in order to support multinational (including NATO), combined and joint operations and the advancement of digitization in the international arena. n Voluntary Project of 26 Nations & NATO ACT u u u Technical specifications based on consensus System-independent capabilities No common funding FGAN Research Institute for Communication, Information Processing and Ergonomics

The MIP Solution Source: MIP FGAN Research Institute for Communication, Information Processing and Ergonomics The MIP Solution Source: MIP FGAN Research Institute for Communication, Information Processing and Ergonomics

MIP Baseline 3 – JC 3 IEDM n JC 3 IEDM = Joint Command, MIP Baseline 3 – JC 3 IEDM n JC 3 IEDM = Joint Command, Control, and Consultation Information Exchange Data Model n MIP Baseline 3 available as draft specification u Current version of the data model: JC 3 IEDM 3. 1 c n Entity relationship model 271 Entities, 2005 attributes, » 10. 000 fixed domain values + Semantic definitions + Business Rules (Integrity & transformation rules) u Valid attribute value combinations u Valid associations u Modeling of geometry of objects and actions u Mapping onto APP 6 a symbols u FGAN Research Institute for Communication, Information Processing and Ergonomics

MIP Baseline 3 – Data Exchange Mechanism n Automatic node identification u Broadcast mechanism MIP Baseline 3 – Data Exchange Mechanism n Automatic node identification u Broadcast mechanism n Publish/Subscribe technology u u Partial replication of JC 3 IEDM data base content Based on subscriptions to operational information groups Taking into account • Referential integrity • Semantic completeness DEM exchange format coupled with relational DB schema FGAN Research Institute for Communication, Information Processing and Ergonomics

Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics

Testing the MIP Solution n Without MTRS u u u International test sessions in Testing the MIP Solution n Without MTRS u u u International test sessions in Greding (Germany) • Bilateral/Multilateral C 2 ISs alternatively take role of • System Under Test • Test System Semi-formal test documents n Problems u u C 2 ISs do not support all test phases • Specification, preparation, execution, evaluation Testing (esp. test setup and coordination) is expensive Legacy systems are not suitable as unbiased testers Test specifications are subject to interpretation FGAN Research Institute for Communication, Information Processing and Ergonomics

Bilateral MIP Interoperability Tests National Test Operator A National C 2 IS A controls Bilateral MIP Interoperability Tests National Test Operator A National C 2 IS A controls & observes Interpretation MIP Test Documents National MIP Gateway Test Coordination (Chat) LAN / Internet Assessment National MIP Gateway Interpretation controls & observes National C 2 IS B FGAN National Test Operator B Research Institute for Communication, Information Processing and Ergonomics Test Result MIP Test Documents

MTRS MIP Conformance Tests MTRS Test Cases controls & observes MIP Gateway Test Manager MTRS MIP Conformance Tests MTRS Test Cases controls & observes MIP Gateway Test Manager Interpretation Test Results User Data LAN / Internet Java RMI National MIP Gateway controls & observes National C 2 IS FGAN controls & observes National Test Operator Research Institute for Communication, Information Processing and Ergonomics MIP Test Documents

Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI SLT 1 Common information model C 2 IS Commander Staff officer Mapping MCI Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP

Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI SLT 1 Common information model C 2 IS Commander Staff officer Mapping MCI Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP

Test Script Data SLT 1 (Protocol Tests) 00000022 T 1|449000001|180550001 00000093 T 2|D 1|{10003008130201000001|BLK Test Script Data SLT 1 (Protocol Tests) 00000022 T 1|449000001|180550001 00000093 T 2|D 1|{10003008130201000001|BLK 3 SLT 1 DEM A RESP ORG OIG_1|GLBSGN|BLK 3 SLT 1 DEM A RESP ORG|0} 00000022 T 1|180550001|449000001 00000093 T 2|D 1|{10003008140301100003|BLK 3 SLT 1 DEM B RESP ORG OIG_3|GLBSGN|BLK 3 SLT 1 DEM B RESP ORG|0} 00000031 T 2|D 2|{10003008130201000001||H} 00000510 T 2|D 4|10003008130201000001|0|3085|x? ½VÛr? ý ¾ Ã? } Äx? SE h§yê? ? ü{Á? ? u@jlëøà? . . . 00000031 T 2|D 2|{10003008140301100003||H} 00000237 T 2|D 4|10003008130201000001|0|826|x? ? Q? 0 Do$³ X? z · õÓª ¡¨ wo¢Öb J 0ÈËÎÎdó. . . 01 testcase MSLT 1 MTRS_M 101 { 02 request("Please start your C 2 IS. " ); 03 //send our own DEM Connection Information 04 Own. Connection. Information own. DCI = get. Own. Connection. Information(); 05 [!] send(own. DCI) to validator; 06 07 //now we want to receive a reply 08 [? ] receive(Connection. Info dci) from validator in 15000 { 09 //check whether all fields are set correctly: 10 assert. Equals("Node. Id", get. Sut(). get. Node. Id(), dci. get. Node. Id()); 11 assert. Equals("Ip. Address", get. Sut(). get. Ip. Address(), dci. get. Ip. Address()); 12 assert. Equals("Tcp. Port", get. Sut(). get. Tcp. Port(), dci. get. Tcp. Port()); 13 if (dci. get. Scope() == Scope. ANNOUNCE) repeat; 14 } 15 } FGAN Research Institute for Communication, Information Processing and Ergonomics

SLT 1 (Protocol Tests) FGAN Research Institute for Communication, Information Processing and Ergonomics SLT 1 (Protocol Tests) FGAN Research Institute for Communication, Information Processing and Ergonomics

Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI Common information model C 2 IS Commander Staff officer Mapping Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP

„Test Script“ Data SLT 2 (DB to DB) X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. Boundary. Org/TC „Test Script“ Data SLT 2 (DB to DB) X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. Boundary. Org/TC 1201_01|1090301120100001|0}}10000254{{10903003212010000001|AFLGEO|1090301120100001|0}}10000076 {{10903011212010000001|OR|BLK 3 SLT 2 DEMARESPORG|1090301120100001|0}{10903011212010000002|FE|Feature/TC 1201_01|109030 1120100001|0}}10000051{{10903012212010000002|CF|1090301120100001|0}}10000049{{10903011212010000002|CF|10903011 20100001|0}}10000261{{10903012212010000001|10903003212010000001|1090301120100001|0}}10000094{{1090301221201000 0001|GVTORG|NO||MECHINFCOY|1090301120100001|0}}10000085{{10903011212010000001|UN|1090301120100001|0}}1000025 8{{10903003212010000001|FRA|1090301120100001|0}}10000032{{10903011212010000002|NOS|1090301120100001|0}}100002 01{{10903012212010000001|MILORG||1090301120100001|0}}10000128{{10903011212010000001|BLK 3 SLT 2 COYA||1090301120100001|0}}10000035{{10903012212010000002|BDYOR|1090301120100001|0}}10000137{{1090300821200001|1090301121201000 0001|INCLDE|1090301120100001|0}{1090300821200001|10903011212010000002|INCLDE|1090301120100001|0}}10000157{ {10903016212010000001|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10 903016212010000002|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903 016212010000003|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}{10903016 212010000004|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10903016212 010000005|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIASST|1090301120100001|0}{10903016212010 000006|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903016212010000 010|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}}10000154{{1090301621 2010000001|20070528112900. 000||1090301120100001|0}{10903016212010000002|20070528112900. 000||1090301120100001|0} {10903016212010000003|20070528112900. 000||1090301120100001|0}{10903016212010000004|20070528112900. 000||… --@0200_01 INSERT INTO CONTXT_ELMT_STAT (contxt_id, contxt_elmt_ix, contxt_elmt_stat_ix, cat_code, effctv_dttm, estblng_org_id, creator_id, update_seqnr) VALUES (10903008202000000001, 1090002000000001, 'ADDITN', '20070920100000. 000', 10903011202000000001, 0); Identify records that shall -- Link the elements to the OIG: HOSTILITY INSERT INTO CONTXT_ELMT (contxt_id, contxt_elmt_ix, rptd_id, creator_id, update_seqnr) (not) be replicated VALUES (1090300820200001, 10900020000003, 1090301620200004, 1090301120200001, 0); --@0200_01 INSERT INTO CONTXT_ELMT_STAT(contxt_id, contxt_elmt_ix, contxt_elmt_stat_ix, cat_code, effctv_dttm, estblng_org_id, creator_id, update_seqnr) VALUES (1090300820200001, 10900020000003, 10900020000007, 'ADDITN', '20070920100000. 000', 10903011202000000001, 0); -- CONTEXT part --@!!0200_01 INSERT INTO CONTXT(contxt_id, cat_code, name_txt, security_clsfc_id, creator_id, update_seqnr) VALUES (1090300820200002, 'OIG', 'OIG_Subgroup_0200_2', NULL, 1090301120200001, 0); FGAN Research Institute for Communication, Information Processing and Ergonomics

SLT 2 (DB to DB) FGAN Research Institute for Communication, Information Processing and Ergonomics SLT 2 (DB to DB) FGAN Research Institute for Communication, Information Processing and Ergonomics

Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange MCI Common information model C 2 IS Commander Staff officer Mapping Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP

SLT 3 (C 2 IS Level) „Test Script“ Data =? X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. SLT 3 (C 2 IS Level) „Test Script“ Data =? X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. Boundary. Org/TC 1201_01|1090301120100001|0}}10000254{{10903003212010000001|AFLGEO|1090301120100001|0}}10000076 {{10903011212010000001|OR|BLK 3 SLT 2 DEMARESPORG|1090301120100001|0}{10903011212010000002|FE|Feature/TC 1201_01|109030 1120100001|0}}10000051{{10903012212010000002|CF|1090301120100001|0}}10000049{{10903011212010000002|CF|10903011 20100001|0}}10000261{{10903012212010000001|10903003212010000001|1090301120100001|0}}10000094{{1090301221201000 0001|GVTORG|NO||MECHINFCOY|1090301120100001|0}}10000085{{10903011212010000001|UN|1090301120100001|0}}1000025 8{{10903003212010000001|FRA|1090301120100001|0}}10000032{{10903011212010000002|NOS|1090301120100001|0}}100002 01{{10903012212010000001|MILORG||1090301120100001|0}}10000128{{10903011212010000001|BLK 3 SLT 2 COYA||1090301120100001|0}}10000035{{10903012212010000002|BDYOR|1090301120100001|0}}10000137{{1090300821200001|1090301121201000 0001|INCLDE|1090301120100001|0}{1090300821200001|10903011212010000002|INCLDE|1090301120100001|0}}10000157{ {10903016212010000001|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10 903016212010000002|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903 016212010000003|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}{10903016 212010000004|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10903016212 010000005|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIASST|1090301120100001|0}{10903016212010 000006|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903016212010000 010|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}}10000154{{1090301621 2010000001|20070528112900. 000||1090301120100001|0}{10903016212010000002|20070528112900. 000||1090301120100001|0} {10903016212010000003|20070528112900. 000||1090301120100001|0}{10903016212010000004|20070528112900. 000||… MIP 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 MTRS prompt 'Please create a friendly unit with some location. '; oig FRDNEU { UNIT unit; OBJ_ITEM_LOC obj. Item. Loc. Old; GEO_POINT geo. Point. Old; obj. Item. Loc. Old -> unit; obj. Item. Loc. Old -> geo. Point. Old; } prompt 'Please change the location of the unit. '; oig FRDNEU { new OBJ_ITEM_LOC obj. Item. Loc. New; new GEO_POINT geo. Point. New; obj. Item. Loc. New -> unit; obj. Item. Loc. New -> geo. Point. New; assert: geo. Point. New. lat_coord <> geo. Point. Old. lat_coord OR geo. Point. New. long_coord <> geo. Point. Old. long_coord; } Research Institute for Communication, Information Processing and Ergonomics FGAN

SLT 3 (C 2 IS Level) Generated automatically, displays received/expected data FGAN Research Institute SLT 3 (C 2 IS Level) Generated automatically, displays received/expected data FGAN Research Institute for Communication, Information Processing and Ergonomics

Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics

Status Quo – System Level Tests n Available test cases u u 107 formal Status Quo – System Level Tests n Available test cases u u 107 formal test cases for SLT 1 (protocol stack) 166 test scripts for SLT 2 (replication mechanism) > 240 MTRS test scripts under development for SLT 3 (C 2 IS-JC 3 IEDM mapping) Compare to 17 SLT 1 and 20 SLT 2 test cases in MIP Baseline 2! FGAN Research Institute for Communication, Information Processing and Ergonomics

Usage Statistics (1) n MTRS used for 20 systems of 17 nations + NATO Usage Statistics (1) n MTRS used for 20 systems of 17 nations + NATO n Tremendous testing activity … even during weekends! u u u 100 - 300 tests each day (with a peak of 500+ tests) More than 26, 400 test runs as of 05 May 2008 SLT 1: 99, 9% completed SLT 2: about 75% completed SLT 3: starting in June 2008 FGAN Research Institute for Communication, Information Processing and Ergonomics

Usage Statistics (2) Total testing time: 4191 hours ≈ 524 man days FGAN Research Usage Statistics (2) Total testing time: 4191 hours ≈ 524 man days FGAN Research Institute for Communication, Information Processing and Ergonomics

Usage Statistics (3) FGAN Research Institute for Communication, Information Processing and Ergonomics Usage Statistics (3) FGAN Research Institute for Communication, Information Processing and Ergonomics

Usage Statistics (4) n Test executions Verdict Count Pass 11864 401: 14 h 2: Usage Statistics (4) n Test executions Verdict Count Pass 11864 401: 14 h 2: 02 min Fail 9646 471: 56 h 2: 56 min Inconclusive 4893 476: 16 h 5: 50 min 26403 1349: 27 h 3: 04 min Total Duration Avg. Duration n Test execution takes about 32% of total test time n Successful tests are executed fastest FGAN Research Institute for Communication, Information Processing and Ergonomics

Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics

Summary (1) n MIP Test Reference System u u u Functional black box testing Summary (1) n MIP Test Reference System u u u Functional black box testing Tests conformance to MIP specification Automated test execution n Flexible Test Execution u u 24 x 7 Availability over the Internet No need for test coordination Support for multilateral tests • Different test configurations thanks to modular test gateways Parallel use by multiple C 2 IS without interference n Simplified Test Evaluation u Error analysis by means of detailed logs n MIP Project Management Group "strongly MIP Test Reference System" FGAN supports the goal and use of Research Institute for Communication, Information Processing and Ergonomics

Summary (2) n Client-Server Architecture with alternative MTRS clients u u u Graphical user Summary (2) n Client-Server Architecture with alternative MTRS clients u u u Graphical user interface Java API for test automation on the nations’ side MTRS Website incl. test repository, ticketing system… https: //trac. fkie. fgan. de/MTRS n MIP Baseline 3 Roadmap u u System Level Test started in September 2007 Expected to be completed by January 2009 n Significant improvement to MIP test specification process u u u All MIP test documents put under version control Significant increase in quantity and quality of tests Tool support to keep test artifacts consistent FGAN Research Institute for Communication, Information Processing and Ergonomics

Thank you very much for your attention FGAN Research Institute for Communication, Information Processing Thank you very much for your attention FGAN Research Institute for Communication, Information Processing and Ergonomics