Image processing algorithm regression testing framework Soumik Ukil

Testing Ground Rules § Objectives: § Testing is the process of executing a program with the intent of finding errors § A good test case is one that has a high probability of finding an error § A successful test is one that uncovers an error § However, testing cannot show the absence of defects

§ Unit Testing: § § Testing to determine that individual program modules perform to specification Basic units of software tested in isolation § Regression Testing: § § Selective retesting to detect faults introduced during modification of system Should be performed after any changes in software like bug fixes

Motivation n Projects like NETT and BRP n n Complex algorithms which run on hundreds of datasets Source is being constantly modified to fix bugs etc Manual testing infeasible Need automated regression testing

Testing Strategies n Dynamic Analysis n n n Exercise the software in its execution environment Black Box n Treat the system as black -box, ie no knowledge of internal structure n Only need to know legal input and expected output White Box n Depends on Internal structure of program n Test all paths through code

Testing Strategies § Static Analysis § § § Code Reviews Walkthroughs Inspections

Tests on image data § Gold-standard data used as basis for correctness of output § Black box tests: § § § Already know 'correct' output Generate data from algorithm Define tests that compare the two sets of data

Tests on image data n Generic Tests: n n n Check output is of correct datatype Check image dimensions Specific tests: n n n Depends on application being tested For airway tree validation we may want to compare distance between branchpoints For Lung Segmentation we need to compare volumes and/or distance between contours

Objectives of test framework n Different applications: n n n Regression testing after any changes Validation of data Flexible: n n Testers only plug in specific tests Test data generation and reporting of test results taken care by framework

Implementation n Built on top of Py. Unit: n Part of standard Python library for Python 2. 1 and later n Based on JUnit, a proven testing architecture n Allows creation of user defined tests, aggregation into suites, running tests in textual or GUI mode

Writing tests with Py. Unit: § Basic building blocks called Test Cases § § § Fixtures § § § A set-up and tear down method for each test case Many different test cases can use the same fixture Test Suites § § § Created by deriving from base class unittest. Test. Case An instance of a Test. Case class is an object that can completely run a single test method Test case instances can be grouped together according to the features they test All tests can be executed together as part of a suite Test Runner § § A class whose instances run tests and report results Can be used in text or GUI mode

Example: class Lung. Seg. Test. Case(unittest. Test. Case): def initialize(self, fname 1, fname 2): A = Ana. File() result # Anafile object for reading images self. data, hdr=A. read(fname 1) # segmentation self. ref_data, ref_hdr = A. read(fname 2) # reference mask labels def set. Up(self): self. labels=[20, 30] # left and right lung def tear. Down(self): self. data = None self. ref_data= None def test 1(self): ## test method names begin 'test*' """Test to see if mask image datatype is uint 8. """ self. assert. Equals(self. data. typecode(), 'b') def test 2(self): """Test to check that correct labels are present in mask file. """ for n in self. labels: errormsg="label" + str(n) + "missing" self. assert. Not. Equals(sum(ravel(equal(self. data, n))), 0, errormsg)

Application dependent tests: n For lung segmentation we use an area overlap measure to test correctness: n Check that value is above a given threshold to 'pass' test Similar tests can be defined with distance measures between contours User has to plug-in appropriate tests n n

Flow diagram:

Sample configuration file: [DEFAULT] # Directory of gold-standard data Refdir =. . /Validation. Data #Directory of data generated by algorithm Maskdir =. . /New. Data # Output file extension Mask. Ext = mask. img # Name of logfile directory Logdir =. . /logfiles [ALGORITHM] # Dir. of raw image data to run algorithm on Inputdir = /home/xyz/lung/Data # Path to executable for algorithm Execpath =. . /lung # Input file extension Input. Ext = img

Reporting test results: n Textual output directed to log files Success: ----------------------------------Ran 6 tests in 88. 337 s OK Failure: FAIL: Left lung pixel count test (slice by slice). -----------------------------------Traceback (most recent call last): File "lungsegtests. py", line 101, in test 6 self. fail. If(fail==1, errormsg) File "unittest. py", line 264, in fail. If if expr: raise self. failure. Exception, msg Assertion. Error: significant segmentation mismatch for left lung on slices: [458, 462]

Status: n Testing for Lung segmentation/smoothing has been implemented with the following tests: n Type checking of output masks Checking that all expected labels are present n Slice by slice area comparison for both lungs n n n Can be used by segmentation algorithms on ANALYZE images which produce labeled masks Framework can be extended to handle other formats like airway tree definitions etc

References: 1. http: //pyunit. sourceforge. net 2. http: //www. xprogramming. com/testfram. htm Documentation and source code: CVS repository: pulmonary/Test. Frame. Work