Testing C 2 Interoperability Advancements in Testing of the MIP Baseline 3 Solution Nico Bau, Michael Gerz & Michael Glauer FGAN FKIE 13 th ICCRTS Bellevue, WA June 17 th-19 th, 2008 FGAN Research Institute for Communication, Information Processing and Ergonomics
Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics
Goals n The aim of the Multilateral Interoperability Programme (MIP) is to achieve international interoperability of Command Control Information Systems (C 2 IS) at all levels from corps to battalion, or lowest appropriate level, in order to support multinational (including NATO), combined and joint operations and the advancement of digitization in the international arena. n Voluntary Project of 26 Nations & NATO ACT u u u Technical specifications based on consensus System-independent capabilities No common funding FGAN Research Institute for Communication, Information Processing and Ergonomics
The MIP Solution Source: MIP FGAN Research Institute for Communication, Information Processing and Ergonomics
MIP Baseline 3 – JC 3 IEDM n JC 3 IEDM = Joint Command, Control, and Consultation Information Exchange Data Model n MIP Baseline 3 available as draft specification u Current version of the data model: JC 3 IEDM 3. 1 c n Entity relationship model 271 Entities, 2005 attributes, » 10. 000 fixed domain values + Semantic definitions + Business Rules (Integrity & transformation rules) u Valid attribute value combinations u Valid associations u Modeling of geometry of objects and actions u Mapping onto APP 6 a symbols u FGAN Research Institute for Communication, Information Processing and Ergonomics
MIP Baseline 3 – Data Exchange Mechanism n Automatic node identification u Broadcast mechanism n Publish/Subscribe technology u u Partial replication of JC 3 IEDM data base content Based on subscriptions to operational information groups Taking into account • Referential integrity • Semantic completeness DEM exchange format coupled with relational DB schema FGAN Research Institute for Communication, Information Processing and Ergonomics
Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics
Testing the MIP Solution n Without MTRS u u u International test sessions in Greding (Germany) • Bilateral/Multilateral C 2 ISs alternatively take role of • System Under Test • Test System Semi-formal test documents n Problems u u C 2 ISs do not support all test phases • Specification, preparation, execution, evaluation Testing (esp. test setup and coordination) is expensive Legacy systems are not suitable as unbiased testers Test specifications are subject to interpretation FGAN Research Institute for Communication, Information Processing and Ergonomics
Bilateral MIP Interoperability Tests National Test Operator A National C 2 IS A controls & observes Interpretation MIP Test Documents National MIP Gateway Test Coordination (Chat) LAN / Internet Assessment National MIP Gateway Interpretation controls & observes National C 2 IS B FGAN National Test Operator B Research Institute for Communication, Information Processing and Ergonomics Test Result MIP Test Documents
MTRS MIP Conformance Tests MTRS Test Cases controls & observes MIP Gateway Test Manager Interpretation Test Results User Data LAN / Internet Java RMI National MIP Gateway controls & observes National C 2 IS FGAN controls & observes National Test Operator Research Institute for Communication, Information Processing and Ergonomics MIP Test Documents
Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI SLT 1 Common information model C 2 IS Commander Staff officer Mapping MCI Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP
Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI SLT 1 Common information model C 2 IS Commander Staff officer Mapping MCI Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP
Test Script Data SLT 1 (Protocol Tests) 00000022 T 1|449000001|180550001 00000093 T 2|D 1|{10003008130201000001|BLK 3 SLT 1 DEM A RESP ORG OIG_1|GLBSGN|BLK 3 SLT 1 DEM A RESP ORG|0} 00000022 T 1|180550001|449000001 00000093 T 2|D 1|{10003008140301100003|BLK 3 SLT 1 DEM B RESP ORG OIG_3|GLBSGN|BLK 3 SLT 1 DEM B RESP ORG|0} 00000031 T 2|D 2|{10003008130201000001||H} 00000510 T 2|D 4|10003008130201000001|0|3085|x? ½VÛr? ý ¾ Ã? } Äx? SE h§yê? ? ü{Á? ? u@jlëøà? . . . 00000031 T 2|D 2|{10003008140301100003||H} 00000237 T 2|D 4|10003008130201000001|0|826|x? ? Q? 0 Do$³ X? z · õÓª ¡¨ wo¢Öb J 0ÈËÎÎdó. . . 01 testcase MSLT 1 MTRS_M 101 { 02 request("Please start your C 2 IS. " ); 03 //send our own DEM Connection Information 04 Own. Connection. Information own. DCI = get. Own. Connection. Information(); 05 [!] send(own. DCI) to validator; 06 07 //now we want to receive a reply 08 [? ] receive(Connection. Info dci) from validator in 15000 { 09 //check whether all fields are set correctly: 10 assert. Equals("Node. Id", get. Sut(). get. Node. Id(), dci. get. Node. Id()); 11 assert. Equals("Ip. Address", get. Sut(). get. Ip. Address(), dci. get. Ip. Address()); 12 assert. Equals("Tcp. Port", get. Sut(). get. Tcp. Port(), dci. get. Tcp. Port()); 13 if (dci. get. Scope() == Scope. ANNOUNCE) repeat; 14 } 15 } FGAN Research Institute for Communication, Information Processing and Ergonomics
SLT 1 (Protocol Tests) FGAN Research Institute for Communication, Information Processing and Ergonomics
Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange SLT 2 MCI Common information model C 2 IS Commander Staff officer Mapping Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP
„Test Script“ Data SLT 2 (DB to DB) X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. Boundary. Org/TC 1201_01|1090301120100001|0}}10000254{{10903003212010000001|AFLGEO|1090301120100001|0}}10000076 {{10903011212010000001|OR|BLK 3 SLT 2 DEMARESPORG|1090301120100001|0}{10903011212010000002|FE|Feature/TC 1201_01|109030 1120100001|0}}10000051{{10903012212010000002|CF|1090301120100001|0}}10000049{{10903011212010000002|CF|10903011 20100001|0}}10000261{{10903012212010000001|10903003212010000001|1090301120100001|0}}10000094{{1090301221201000 0001|GVTORG|NO||MECHINFCOY|1090301120100001|0}}10000085{{10903011212010000001|UN|1090301120100001|0}}1000025 8{{10903003212010000001|FRA|1090301120100001|0}}10000032{{10903011212010000002|NOS|1090301120100001|0}}100002 01{{10903012212010000001|MILORG||1090301120100001|0}}10000128{{10903011212010000001|BLK 3 SLT 2 COYA||1090301120100001|0}}10000035{{10903012212010000002|BDYOR|1090301120100001|0}}10000137{{1090300821200001|1090301121201000 0001|INCLDE|1090301120100001|0}{1090300821200001|10903011212010000002|INCLDE|1090301120100001|0}}10000157{ {10903016212010000001|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10 903016212010000002|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903 016212010000003|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}{10903016 212010000004|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10903016212 010000005|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIASST|1090301120100001|0}{10903016212010 000006|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903016212010000 010|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}}10000154{{1090301621 2010000001|20070528112900. 000||1090301120100001|0}{10903016212010000002|20070528112900. 000||1090301120100001|0} {10903016212010000003|20070528112900. 000||1090301120100001|0}{10903016212010000004|20070528112900. 000||… --@0200_01 INSERT INTO CONTXT_ELMT_STAT (contxt_id, contxt_elmt_ix, contxt_elmt_stat_ix, cat_code, effctv_dttm, estblng_org_id, creator_id, update_seqnr) VALUES (10903008202000000001, 1090002000000001, 'ADDITN', '20070920100000. 000', 10903011202000000001, 0); Identify records that shall -- Link the elements to the OIG: HOSTILITY INSERT INTO CONTXT_ELMT (contxt_id, contxt_elmt_ix, rptd_id, creator_id, update_seqnr) (not) be replicated VALUES (1090300820200001, 10900020000003, 1090301620200004, 1090301120200001, 0); --@0200_01 INSERT INTO CONTXT_ELMT_STAT(contxt_id, contxt_elmt_ix, contxt_elmt_stat_ix, cat_code, effctv_dttm, estblng_org_id, creator_id, update_seqnr) VALUES (1090300820200001, 10900020000003, 10900020000007, 'ADDITN', '20070920100000. 000', 10903011202000000001, 0); -- CONTEXT part --@!!0200_01 INSERT INTO CONTXT(contxt_id, cat_code, name_txt, security_clsfc_id, creator_id, update_seqnr) VALUES (1090300820200002, 'OIG', 'OIG_Subgroup_0200_2', NULL, 1090301120200001, 0); FGAN Research Institute for Communication, Information Processing and Ergonomics
SLT 2 (DB to DB) FGAN Research Institute for Communication, Information Processing and Ergonomics
Testing the MIP Solution Effective C 2 for international operations OLT SLT 3 Shared tactical picture Mapping Automated information exchange MCI Common information model C 2 IS Commander Staff officer Mapping Common information model C 2 IS Staff officer Commander MIP Solution • Specifications • SOPs • Documents Collaboration Common understanding FGAN Research Institute for Communication, Information Processing and Ergonomics Source: MIP
SLT 3 (C 2 IS Level) „Test Script“ Data =? X 0110000082{{10903012212010000001|OR|NO|MECHANISEDINFANTRYCOMPANY|1090301120100001|0}{10903012212010000002|FE|NO|Ctrl. F eat. Boundary. Org/TC 1201_01|1090301120100001|0}}10000254{{10903003212010000001|AFLGEO|1090301120100001|0}}10000076 {{10903011212010000001|OR|BLK 3 SLT 2 DEMARESPORG|1090301120100001|0}{10903011212010000002|FE|Feature/TC 1201_01|109030 1120100001|0}}10000051{{10903012212010000002|CF|1090301120100001|0}}10000049{{10903011212010000002|CF|10903011 20100001|0}}10000261{{10903012212010000001|10903003212010000001|1090301120100001|0}}10000094{{1090301221201000 0001|GVTORG|NO||MECHINFCOY|1090301120100001|0}}10000085{{10903011212010000001|UN|1090301120100001|0}}1000025 8{{10903003212010000001|FRA|1090301120100001|0}}10000032{{10903011212010000002|NOS|1090301120100001|0}}100002 01{{10903012212010000001|MILORG||1090301120100001|0}}10000128{{10903011212010000001|BLK 3 SLT 2 COYA||1090301120100001|0}}10000035{{10903012212010000002|BDYOR|1090301120100001|0}}10000137{{1090300821200001|1090301121201000 0001|INCLDE|1090301120100001|0}{1090300821200001|10903011212010000002|INCLDE|1090301120100001|0}}10000157{ {10903016212010000001|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10 903016212010000002|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903 016212010000003|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}{10903016 212010000004|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OITYPE|1090301120100001|0}{10903016212 010000005|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIASST|1090301120100001|0}{10903016212010 000006|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OISTAT|1090301120100001|0}{10903016212010000 010|1|REP|NO||A|20070528112900. 000||RDABST|||10903011212010000001|OIHSTS|1090301120100001|0}}10000154{{1090301621 2010000001|20070528112900. 000||1090301120100001|0}{10903016212010000002|20070528112900. 000||1090301120100001|0} {10903016212010000003|20070528112900. 000||1090301120100001|0}{10903016212010000004|20070528112900. 000||… MIP 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 MTRS prompt 'Please create a friendly unit with some location. '; oig FRDNEU { UNIT unit; OBJ_ITEM_LOC obj. Item. Loc. Old; GEO_POINT geo. Point. Old; obj. Item. Loc. Old -> unit; obj. Item. Loc. Old -> geo. Point. Old; } prompt 'Please change the location of the unit. '; oig FRDNEU { new OBJ_ITEM_LOC obj. Item. Loc. New; new GEO_POINT geo. Point. New; obj. Item. Loc. New -> unit; obj. Item. Loc. New -> geo. Point. New; assert: geo. Point. New. lat_coord <> geo. Point. Old. lat_coord OR geo. Point. New. long_coord <> geo. Point. Old. long_coord; } Research Institute for Communication, Information Processing and Ergonomics FGAN
SLT 3 (C 2 IS Level) Generated automatically, displays received/expected data FGAN Research Institute for Communication, Information Processing and Ergonomics
Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics
Status Quo – System Level Tests n Available test cases u u 107 formal test cases for SLT 1 (protocol stack) 166 test scripts for SLT 2 (replication mechanism) > 240 MTRS test scripts under development for SLT 3 (C 2 IS-JC 3 IEDM mapping) Compare to 17 SLT 1 and 20 SLT 2 test cases in MIP Baseline 2! FGAN Research Institute for Communication, Information Processing and Ergonomics
Usage Statistics (1) n MTRS used for 20 systems of 17 nations + NATO n Tremendous testing activity … even during weekends! u u u 100 - 300 tests each day (with a peak of 500+ tests) More than 26, 400 test runs as of 05 May 2008 SLT 1: 99, 9% completed SLT 2: about 75% completed SLT 3: starting in June 2008 FGAN Research Institute for Communication, Information Processing and Ergonomics
Usage Statistics (2) Total testing time: 4191 hours ≈ 524 man days FGAN Research Institute for Communication, Information Processing and Ergonomics
Usage Statistics (3) FGAN Research Institute for Communication, Information Processing and Ergonomics
Usage Statistics (4) n Test executions Verdict Count Pass 11864 401: 14 h 2: 02 min Fail 9646 471: 56 h 2: 56 min Inconclusive 4893 476: 16 h 5: 50 min 26403 1349: 27 h 3: 04 min Total Duration Avg. Duration n Test execution takes about 32% of total test time n Successful tests are executed fastest FGAN Research Institute for Communication, Information Processing and Ergonomics
Outline n Introduction to MIP u u Goals & Organization MIP Solution of Baseline 3 n MIP Tests u u u MIP Interoperability & Conformance Tests MIP Test Reference System SLT 1 / SLT 2 / SLT 3 n Status Quo & Usage Statistics u u MIP System Level Tests Use of the MTRS n Summary FGAN Research Institute for Communication, Information Processing and Ergonomics
Summary (1) n MIP Test Reference System u u u Functional black box testing Tests conformance to MIP specification Automated test execution n Flexible Test Execution u u 24 x 7 Availability over the Internet No need for test coordination Support for multilateral tests • Different test configurations thanks to modular test gateways Parallel use by multiple C 2 IS without interference n Simplified Test Evaluation u Error analysis by means of detailed logs n MIP Project Management Group "strongly MIP Test Reference System" FGAN supports the goal and use of Research Institute for Communication, Information Processing and Ergonomics
Summary (2) n Client-Server Architecture with alternative MTRS clients u u u Graphical user interface Java API for test automation on the nations’ side MTRS Website incl. test repository, ticketing system… https: //trac. fkie. fgan. de/MTRS n MIP Baseline 3 Roadmap u u System Level Test started in September 2007 Expected to be completed by January 2009 n Significant improvement to MIP test specification process u u u All MIP test documents put under version control Significant increase in quantity and quality of tests Tool support to keep test artifacts consistent FGAN Research Institute for Communication, Information Processing and Ergonomics
Thank you very much for your attention FGAN Research Institute for Communication, Information Processing and Ergonomics
Скачать презентацию Testing C 2 Interoperability Advancements in Testing of
565699307ee4273d485103212a5a415f.ppt