NIST HL 7 V 2 IHE-PCD Pre- and

NIST HL 7 V 2 IHE-PCD Pre- and Connectathon Test Tools (2011 -2012 Cycle 6) John J. Garguilo and Sandra Martinez National Institute of Standards and Technology October 11 th 2011 (Web. Ex 2 – 3 PM) Contact: john. garguilo@nist. gov

NIST IHE-PCD V 2 Test Effort NIST Team Members • • John Garguilo (john. garguilo@nist. gov, 301 -975 -5248) Sandra Martinez (sandra. martinez@nist. gov, 301 -975 -3579) Julien De. Shayes (julien. deshayes@nist. gov Guest Researcher) Jing Gao (jing. gao@nist. gov Guest Researcher) NIST HL 7 V 2 Tool Web sites Instance Test Environment: http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Con/ New – Isolated Test Environment: http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Pre. Con/ 2

IHE-PCD Testing – Key Objectives • Increase test comprehensiveness & quality • Support both conformance & interoperability testing • Support for Pre- & Virtual-connectathons, actual Connectathon & enable year round testing • Remain in alignment with IHE-PCD integration profile development road map • Establish single framework for PCD covering increasing complexity and technologies over next 5 years • Coordinate with IHE “Gazelle Project” • Generate work products that companies can use in their regulatory submissions 3

Advantages of NIST Tooling • Simplicity of use, quick feedback (via reports) – 24 hour x 365 day availability – Test Management Capability – easily record your test results • Viewable by IHE-PCD Test Manager (aka Manny Furst) • Tooling focused on IHE-PCD – including: – – Current profiles, including recently adopted CPs, Framework, Supplement, and Trial Implementation Documentation PCD Test cases – pre-loaded to match/be synchronized with Gazelle Level of rigor matches specifications (reference standards and TF docs) • No Conformance Profile needed – Already Integrated into the NIST tooling… • No stand-alone application installation needed (NIST web site interface) • MDC nomenclature/terminology included [h. RTM] • Support of NIST team and synergy w/ IHE-PCD group 4

Tooling Status • HL 7 V 2 Validation (IHE-PCD) (Required for Cycle 5, 2010 -11) – Instance-type Environment (at message level) / Pre and Connectathon Setting – http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Con/ – http: //hit-testing. nist. gov: 8080/HL 7 Web/ (general use site – domain agnostic) • New for Cycle 6 (2011 -12) – – – Isolated-type Environment / Pre-Connectathon Setting Scenario based Actor centric One System Under Test (SUT) http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Pre. Con/ 5

• • NIST IHE-PCD Test Tool: Actors / Transactions Supported Test Tool required to fulfill 2011 -12 V 2 IHE-PCD Pre-Connectathon testing requirements Results are stored in the NIST IHE-PCD tool; The IHE project manager will review the results IHE-PCD Profile/HL 7 Version 2. 6 Transaction Actors (Source / Receiving) Device Enterprise Communication – DEC - DEC w/ WCM (option) PCD-01 – Communicate PCD Data Reporter / Consumer Device Enterprise Communication, Subscribe to Patient Data - DEC SPD (Option) PCD-02 – Communicate PCD Data Reporter / *Filter / Consumer *Not supported for in ‘Isolated’ testing at this time, ‘Instance’ only Point of Care Infusion Verification – PIV PCD-03 – Communicate Infusion Order Programmer / Consumer Alarm Communication Mgmt - ACM w/ WCM (option) PCD-04 – Report Alarm Reporter / Manager Implantable Device Cardiac Observation – IDCO PCD-09 – Communicate IDC Observation Reporter / Consumer Infusion Pump Event Communication - IPEC PCD-10 – Communicate Infusion Pump Event Reporter / Consumer 6

Things to know • If port or IP is wrong, the test system times out > 2 mins • If NIST Test System tool acts as a reporter and sends to an incorrect port or IP – the message will go into cyber-space… • If NIST Test System tool acts as consumer, the parsed message will determine IP and port from reporter known issues/problems/ect… • When on vendor is registering multiple applications, the user must create multiple accounts (e. g. , Co_1_Device, Co_2_Device) • The test system infuses “bad, but not too bad” messages so processing can continue; same when TS is consumer (e. g. , if MSH. 9 - Message Type, MSH. 11 - HL 7 Version, MSH. 12 – are incorrect a ack with “AR’ with no furthering processing by the consumer) 7

Issues • IPEC, WCM actors are not yet in Test System -Coming soon! • IPEC, WCM related tests have been identified but not completed – NIST is working w/ IHE-PCD Integration Profile groups to finalize (more discussion at F 2 F – agenda item) • In process of adding descriptions to test case documentation in NIST Test System • In the process of reconciling the test cases between Gazelle and NIST (Tool Index) • User’s Guide not completed - will be available on the site under “Documentation” (within ‘Documentation’ tab) 8

NIST PCD Test Tool: Support • IHE-PCD Google Group – – http: //groups. google. com/group/ihe_pcd_testing Preferred method of communication for discussing common issues Go to the site and request membership Administrative (only) issues can be sent directly to john. garguilo@nist. gov sandra. martinez@nist. gov or julien. deshayes@nist. gov • Issues Tracker (suggested, not yet created) – http: //code. google. com/pcdtool/issues/list – Google utility to report issues, bugs, and enhancement suggestions 9

Troubleshooting • Site is unavailable – Typically means we are redeploying the application to make updates and the tool – We will announce redeployments – The downtime will be in the neighborhood of 10 -15 minutes – Please refresh the browser following the redeployment – Server is controlled by central computing facility and may be unavailable • Site appears to be down – Restart in a new browser window • Site has timed out – This will happen if the tool is open in an inactive browser window – To correct, refresh the browser 10

Troubleshooting (continued) • No response from console log – You may have sent to a destination that we could not process (i. e. , an incorrect receiving name) – We will send back an acknowledge/error over the network – On the console log nothing will happen – You need to check the destination on your end – If you determined that this was the problem you can resend a properly addressed message; we’ll still be listening up to the session timeout • Completed report is missing – When we discover an error in the test case (i. e. , the test case is wrong) we fix it – Tests that passed using an incorrect test are no longer valid – This should not occur often 11

NIST Testing Strategy

NIST Testing Strategy Test Environments • Instance Testing – Conformance (e. g. , against HL 7 2. x or CDA) • Implementation conforms to Spec. on which it is based • IHE Model: ~Virtual and Pre-Connectathon • Isolated System Testing – Includes Instance Testing Activities – Protocol Conformance – Functional Behavior Conformance • Features and Operational behavior correspond to Specs. • IHE Model: ~Virtual and Pre-Connectathon • Peer-to-Peer System Testing – Includes Isolated System Testing Activities – Interoperability Testing • Testing complete application environment • May include interacting w/ Database, using Network Communications, or interacting w/ other hardware, apps, or systems if appropriate • IHE Model: ~Connectathon 13

IHE-PCD 2011 Pre-Connectathon NIST Instance Testing Support NIST V 2 Testing Tool is available for message validation using the ‘instance testing’ environment: Services HL 7 V 2 Message Validation • • Test Management Report Test Artifacts Conformance Profile HL 7 Tables ‘Device’ Test Agents ISO/IEEE 11073/Rosetta Terminology Registry/ Repository HL 7 V 2 Message Validation Test Case Web Application Client Results HL 7 V 2 Message Validation Report User Test Execution Test Harness (Java Code) http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Con/ 14

NIST Testing Strategy Test Environments • Instance Testing – Conformance (e. g. , against HL 7 2. x or CDA) • Implementation conforms to Spec. on which it is based • IHE Model: ~Virtual and Pre-Connectathon • Isolated System Testing – Includes Instance Testing Activities – Protocol Conformance – Functional Behavior Conformance • Features and Operational behavior correspond to Specs. • IHE Model: ~Virtual and Pre-Connectathon • Peer-to-Peer System Testing – Includes Isolated System Testing Activities – Interoperability Testing • Testing complete application environment • May include interacting w/ Database, using Network Communications, or interacting w/ other hardware, apps, or systems if appropriate • IHE Model: ~Connectathon 15

IHE-PCD 2011 Pre-Connectathon NIST Isolated Testing Support NIST V 2 Testing Tool is available for message validation using the ‘isolated testing’ environment: Services HL 7 V 2 Message Validation HL 7 V 2 Message Generation IHE-PCD DOC Test Agent IHE-PCD DOR/DOF Test Agent IHE-PCD IOC Test Agent IHE-PCD IOR Test Agent IHE-PCD AM Test Agent IHE-PCD AR Test Agent IHE-PCD IDCC Test Agent Test Management IHE-PCD IDCR Test Agent Web Application Client IHE-PCD Client Test Scenario Vendor Results HL 7 V 2 Message Validation Reports Test Execution Test Harness (Java Code) Test Artifacts • Conformance Profiles • HL 7 Tables • Validation Context Files • Generation Context Files Report Router/Logger/Proxy System Under Test http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Pre. Con/ 16

Test Environment Message Validation NIST V 2 Testing Tools: IHE-PCD • Validation of IHE-PCD message(s) and corresponding HL 7 Profile(s) • Syntax and Semantic Content Validation – Against HL 7 V 2 message (e. g. , PCD-01) • Message structure (e. g. , MSH, PID, PV 1, OBR, NTE, {{OBX}, OBX, …}) – Against HL 7 profile • (Msg_type^Event_type^ e. g. , ORU^R 01^…) – Against HL 7 and/or user provided tables (value sets) • Example of user provided table is RTM for Ref_IDs, Units, etc. – Against ‘validation context’, including specific values • Defined in XML (e. g. , specific test case values) 17

NIST V 2 Testing Tools and Services Testing Validation Types • Validation against ‘failure types’: – VERSION*: The version in the message and in the profile should match. – MESSAGE_STRUCTURE_ID*: The message type (MSH. 9 element) in the profile and in the message should match. – MESSAGE_STRUCTURE: The message should have a valid message structure (correct usage, correct cardinality, and correct element name). – USAGE: R elements should be present; X elements should not be present in the message. – CARDINALITY: Elements should be present at least the minimum times and at most the maximum times specified in the profile. It should also take into account the usage of the element (X element with a minimum of 4 should not be present in the message). – LENGTH: The value of the element should have a length equal or less than the value specified in the profile. – DATATYPE: For the datatype NM, DTM, SI and TM, the value of the element should match the regular expression defined in the standard. – DATA: The value of the element should match a constant specified in the profile, a value set specified in a table, a value or a regular expression specified in the message validation context. – MESSAGE_VALIDATION_CONTEXT*: This is a user input error when the location specified in the message validation context can't be found in the message. – TABLE_NOT_FOUND*: This is a user input when a table can't be found in the table files (Table. Profile. Document). – AMBIGUOUS_PROFILE*: The profile should not be ambiguous. 18

Overview of the NIST HL 7 V 2 IHE-PCD Pre-Connectathon Test Tools

NIST HL 7 V 2 IHE-PCD Test Tool: Access • Web-based application (no downloads or installations needed) – ‘Isolated’ Site: http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Pre. Con/ – ‘Instance’ Site: http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Con/ • Tool may be used in Anonymous Mode or Registered Mode – Anonymous Mode (“Guest” Users) • Does not require user registration and may be used to conduct ad-hoc system testing – Registered Mode • • Required for completing the Pre-Connectathon -- Could be used for participating in the Connectathon Required to save Pre-Connectathon test results Test reports are made available to the IHE project manager 20

NIST V 2 HL 7 IHE-PCD Test Tool: Operational Process INTERACTION/REPORTS MANUAL OR AUTOMATED SUT END-USER (VENDOR) Web Application Interface MESSAGES (TEST OBJECTS) STIMULUS OR RESPONSE (MESSAGES) NIST IHE-PCD HL 7 v 2/v 3 TEST TOOL (via the communication protocol currently only MLLP) SYSTEM UNDER TEST (SUT) SPECIFICATIONS (test material that defines test assertions) V 3 – Future Work 21

Registration, Login, and Steps for using the NIST Test System

Steps for Completing Test Cases 1. Register/log-in 2. Select HL 7 Version (v 2 only, V 3 Future add on) and Actor 3. Select Test Case 4. Set Configuration Information 5. Run Test 6. View/Save/Submit Reports 23

Registration/Log-in • To register, click-on Register in the upper-right corner of the tool’s home page – – – – Name Organization Name Email Address Login Information Application Name (MSH 3. 1) Facility Name (MSH 4. 1) IP Address(es) and Port Number(s) • NIST Administrator will confirm log-in credentials • Log-in is required to save test results, submit results to the IHE project manager, and gain credit for test completion NIST IHE-PCD Test Tool *Push/Pull to Gazelle IHE Project Manager can view database of results IHE Gazelle * not yet implemented 24

Registration/Log-in Fill out the User Information section completely. Users unable to be verified by the NIST administrators will have accounts deleted. Document v 2 information as appropriate for your case studies. Fields exist for multiple IP Addresses and Port Numbers if needed, based on your system configuration (future place holder). V 3 Actor Information not used currently (future place holder). 25

Viewing Account Information To view information about the currently logged in user, go to the User Account page. The Actor Information section will display the server configuration fields documented during registration. The Responder Information section will display as a reference the unique client configuration information, which the test tool dynamically generates. 26

System Configuration Information (Dialog Box) NIST Test System Configuration User must supply the “Vendor Configuration and include all four of these fields. 27

NIST PCD Test Tool: Support • IHE-PCD Google Group (suggested, not yet created) – – http: //groups. google. com/group/ihe_pcd_testing Preferred method of communication for discussing common issues Go to the site and request membership Administrative (only) issues can be sent directly to john. garguilo@nist. gov sandra. martinez@nist. gov or julien. deshayes@nist. gov • Issues Tracker (suggested, not yet created) – http: //code. google. com/pcdtool/issues/list – Google utility to report issues, bugs, and enhancement suggestions 28

Summary • New ‘isolated’ system available and required for IHE-PCD cycle 6 (Fall Pre-Connectathon 2011) – – – Potential use in 2012 January Connectathon (simulate actors) Isolated-type Environment Scenario based Actor centric One System Under Test (SUT) • System will be “official” upon Steve Moore’s announcement of Gazelle ready (registration and test cases) – Start testing after this week’s F 2 F • Let’s work collaboratively to advance rigor… – – Establish Google group for sharing conversations/questions/issues… Contact Manny Furst for IHE-PCD Procedural issues… Contact NIST for test tooling issues… Record and document any findings/inconsistencies/errors and work with PC/TC and Integration Profile sub-groups - Develop CPs as needed… 29

URLs and References • Additional Tutorial Reference – Last year’s ‘instance’ test system tutorial recording (Nov 2010) – https: //ihe. webex. com/ihe/lsr. php? AT=pb&SP=MC&r. ID=45696462&r. Key= 120 ce 82 d 2491 d 896 – This year’s tutorial recording (Oct 2011) • Current NIST V 2 HL 7 IHE-PCD test sites ‘Instance’ Test Environment: • http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Con/ – ‘Isolated’ Test Environment (New for Pre-Connectathon this Fall): • http: //hit-testing. nist. gov: 13100/PCD-HL 7 Web. Pre. Con/ • NIST Healthcare & Medical Device Communication Testing Web sites – www. nist. gov/healthcare – www. nist. gov/medicaldevices 30

Thank You! • Questions • Discussion • NIST Looks forward to advancing device testing rigor… … with your help!… 31