Segmentation of Medical Images László Nyúl Department of

Segmentation of Medical Images László Nyúl Department of Image Processing and Computer Graphics University of Szeged nyul@inf. u-szeged. hu 1

X-ray radiography 2

Computerized Tomography (CT) 3

Magnetic Resonance Imaging (MRI) 4

Single Photon Emission CT (SPECT) Positron Emission Tomography (PET) 5

Ultrasound Imaging 6

Cryo-section Photographs 7

Thermographic Images 8

Range Images Reflection image Range image 9

Purpose of 3 D Imaging l IN: multiple multimodality images (CT, MR, PET, SPECT, US, …) l OUT: information about an object/object system (qualitative, quantitative) 10

Sources of Images l l 2 D: digital radiographs, tomographic slices 3 D: a time sequence of 2 D images of a dynamic object, a stack of slice images of a static object 4 D: a time sequence of 3 D images of a dynamic object 5 D: a time sequence of 3 D images of a dynamic object for a range of imaging parameters (e. g. , MR spectroscopic images of heart) 11

Operations l Preprocessing: for defining the object information Visualization: for viewing object information Manipulation: for altering object information Analysis: for quantifying object information l The operations are independent l l l 12

Preprocessing Operations l Volume of interest (VOI) converts a given scene to another scene of smaller scene domain (ROI) and/or intensity range (IOI) l Filtering converts a given scene to another scene by suppressing unwanted information and/or enhancing wanted information 13

Preprocessing Operations l Interpolation converts a given scene to another scene of specified level and orientation of discretization l Registration converts a given scene/structure to another scene/structure by matching it with another given scene/structure l Segmentation converts a given set of scenes to a structure/structure system 14

Image Segmentation l l Purpose: to extract object information from scenes and represent it as a structure/structure system Consists of l Recognition l l l Delineation l l Determine roughly the objects’ whereabouts in the scene humans >> computer algorithms Determine the objects’ precise spatial extent and graded composition computer algorithms >> humans Manual delineation specifying graded composition is impossible Needed for most (3 D) imaging operations 15

Challenges in Medical Imaging l Subject of imaging l l Side effects, health hazards of the acquisition l l Human beings Contrast agents Radiation Invasive techniques Data handling l Privacy 16

Challenges in Medical Imaging l Image processing l l l Grey-level appearance of tissues Characteristics of imaging modality Geometry of anatomy l l l Organs are of different size and shape Normal vs. diseased Objects may change between acquisitions Automated processing is desirable Evaluation l No ground truth available! 17

Limitations of Acquisition Techniques l Resolution l l l l Spatial Temporal Density Tissue contrast Noise distribution, shading Partial volume averaging Artifacts Implants 18

Noise and Sampling Errors 19

Different Tissue Contrast 20

Artifacts 21

Applications of Image Segmentation in Medicine l l l Visualization, qualitative analysis Quantitative analysis Neurological studies Radiotherapy planning Diagnosis Research Implant design Image guided surgery Surgical planning and simulations Therapy evaluation and follow up … 22

Brain and the Ventricles 23

Regions to Segment l Target regions for l l quantification and measurements radiation treatment needle insertion, biopsy surgical resection l Regions to avoid by l l radiation needle drill surgical knife 24

Computer Aided Diagnosis (CAD) l l The computer can store, process, compare, and present (visualize) data The computer may even make suggestions The physician has to make the final judgment! 25

Image Segmentation l Consists of l Recognition l l Delineation l l l humans >> computer algorithms >> humans Manual delineation specifying graded composition is impossible Aim: exploit the synergy between the two (humans and computer algorithms) to develop practical methods with high l PRECISION: reliability/repeatability l ACCURACY: agreement with truth l EFFICIENCY: practical viability 26

Approaches to Recognition l Automatic l l l Knowledge- and atlas-based artificial intelligence techniques used to represent object knowledge Preliminary delineation needed to form object hypotheses Map geometric information from scene to atlas 27

Approaches to Recognition l Human assisted l Often a simple human assistance is sufficient as a recognition aid: l l l Specification of “seed” points in the object Indication of a box enclosing the object Click of a mouse button to accept a real object or reject a false object 28

Approaches to Delineation l Boundary-based l l l Region-based l l l Work with boundaries (contours, surfaces) Output boundary description of objects Work with regions (pixels, voxels and patches) Output regions occupied by objects Hybrid l Combine boundary-based and region-based methods 29

Approaches to Delineation l Hard (crisp) l l Each voxel in the output has a label of belonging either to object or background Fuzzy l Each voxel in the output has a membership value in both object and background 30

Boundary-based Segmentation Methods l l l Align model boundary with object boundary using image features (edges) Requires initialization near solution to avoid becoming stuck in local minima Pixel information inside the object is not considered 31

32

Pre-processing (creating boundaries) l Feature detection l l l l Points Lines Edges Corners Junctions Contour following Edge linking Canny edge-detector 33

Iso-surfacing l l Produce a surface that separates regions of intensity > threshold from those < threshold Digital surfaces l l l Voxels Voxel faces Polygonal elements 34

Gradient operators 35

Creating Edges From Image Gradient 36

Degree of “edginess” 37

Hough Transform l l l Locate curves described by a few parameters Edge points are transformed into the parameter space and a cumulative map is created Local maximum corresponds to the parameters of a curve along which several points lie Straight lines Circles 38

Parameterization of a Line 39

Detecting Lines via Hough Transform 40

Detecting Lines via Hough Transform 41

Detecting Circles via Hough Transform 42

Live Wire Segmentation of the Knee and the Ankle l l l Live wire Live lane Live wire 3 D 43

Deformable Boundaries l l Active/dynamic contour, snake Active surface Active shape model (ASM) Active appearance model (AAM) l Aim: minimize an energy functional with internal and external energy content l Challenges l l Tuning the effects of the energy components Handling topology changes during evolution 44

Active Contour shrink wrap balloon 45

Gradient Vector Field 46

Active Contour with GVF 47

Deformable Surfaces in 3 D l Transition from 2 D to 3 D is not trivial! l 2 D contour as polygon (vertices, edges) 3 D surface as polyhedron (vertices, edges, faces) l l Self-crossing Topology changes l Topology adaptive snakes (T-snakes) l 48

Affine cell image decomposition (ACID) 49

Active Surface Segmentations of the Liver and the Right Kidney 50

Level-set and Fast Marching Methods l Contour/surface is represented as the zero level set of some evolving implicit function 51

Evolving Level Set Functions 52

Segmenting Vessels and Gray Matter using Level-sets 53

Region-based Segmentation Methods l l l Image pixels are assigned to object or background based on homogeneity statistics Advantage is that image information inside the object is considered Disadvantage is lack of provision for including shape of object in decision making process 54

Thresholding l General form: T = T{ x, A(x), f(x) } l l l l Global: T = T{ f(x) } Local: T = T{ A(x), f(x) } Adaptive / dynamic: T = T{ x, A(x), f(x) } Single threshold Band thresholding Hysteresis thresholding Dozens of strategies for determining thresholds 55

Thresholding Original image Global thresholding 56 Local thresholding

Hounsfield Unit Ranges for CT 57

Fuzzy Image Processing l Why Use Fuzzy Image Processing? 58

Membership functions 59

„young person” 60

„cold bear” 61

Typical shapes of membership functions 62

Set and its complement 63

General Structure of Fuzzy Image Processing 64

Representing “dark graylevels” with sets 65

Histogram Fuzzification with Three Membership Functions 66

Fuzzy thresholding 67

Example of fuzzy thresholding 68

Thresholding Using Fuzziness 69

Measures of Fuzziness l Linear index of fuzziness for an image of size Mx. N: l Quadratic index of fuzziness: l Fuzzy entropy: 70

Clustering Techniques 71

Clustering using two features 72

k-nearest neighbors (k. NN) l Training: identify two sets of voxels XO in object region and XNO in background l For each voxel v in input scenes, find its location P in feature space Find k voxels closest to P from sets XO and XNO If a majority of those are from XO, v belongs to object, otherwise to background l l l Fuzzy k. NN is possible l if m out of k nearest neighbors of voxel v belongs to object, than we can assign (v)=m/k as the membership of v in the object 73

74

75

Region Growing 1. 2. 3. 4. Specify a (set of) seed voxel(s) in the object and put them in a queue Q. Specify criteria C for inclusion of voxels (such as thresholds on voxel intensity and/or mean intensity and/or variance of growing region) If Q is empty, stop, else take a voxel v from Q and output v Find those neighbors X of v in scene which were not previously visited and satisfy C Put X in Q and go to Step 2. 76

Watershed Algorithm 77

Using Markers to Overcome Over-segmentation 78

Markov Random Fields (MRF) l MRF can be used to model l l nonlinear interaction between features spatial and temporal information Cliques Statistical processes 79

Artificial Neural Networks (ANN) 80

Object Characteristics in Images l Graded composition heterogeneity of intensity in the object region due to heterogeneity of object material, blurring, noise, and background variation caused by the imaging device l Hanging-togetherness (Gestalt) natural grouping of voxels constituting an object a human viewer readily sees in a display of the scene in spite of intensity heterogeneity l Graded composition and hanging-togetherness are fuzzy properties 81

Fuzzy Sets and Relations l Fuzzy subset: l Membership function: l Fuzzy relation: l Fuzzy union and intersection operations (e. g. , max and min) Similitude relation: reflexive, symmetric, transitive l 82

Fuzzy Digital Space l Fuzzy spel adjacency: how close two spels are spatially. Example: l Fuzzy digital space: l Scene (over a fuzzy digital scene): 83

Fuzzy Connectedness l Fuzzy spel affinity: how close two spels are spatially and intensitybased-property-wise (local hanging-togetherness) l l l Path (between two spels) Fuzzy k-net Fuzzy k-connectedness (K) c d 84

Fuzzy Connected Objects l Binary relation K l Fuzzy k-component of strength x Fuzzy k x object containing o l Very important property: robustness l 85

Fuzzy Connectedness Variants l Multiple seeds per object l Scale-based fuzzy affinity l Relative fuzzy connectedness Iterative relative fuzzy connectedness Vectorial … fuzzy connectedness l l 86

Scale-based Fuzzy Affinity l spatial adjacency homogeneity object feature object scale l global hanging-togetherness l l l 87

Scale As Used in Fuzzy Connectedness l l l “Scale” is the size of local structures under a pre-specified regionhomogeneity criterion. In an image C at any voxel c, scale is defined as the radius r(c) of the largest ball centered at c which lies entirely within the same object region. The scale value can be simply and effectively estimated without explicit object segmentation. 88

Applications with Fuzzy Connectedness Segmentation l MR l l MRA l l l Detecting microcalcifications Craniofacial 3 D imaging l l Detecting and quantifying cancer, cyst, polyp Detecting and quantifying stenosis and aneurism Digitized mammography l l Kinematics studies Measuring bone density Stress-and-strain modeling CT soft tissue (fat, skin, muscle, lungs, airway, colon) segmentation l l Vessel segmentation, artery-vein separation CT bone (skull, shoulder, ankle, knee, pelvis) segmentation l l Brain tissue segmentation Brain tumor quantification Image analysis in multiple sclerosis and Alzheimer’s disease Visualization and surgical planning Visible Human Data 89

Brain Tissue Segmentation (SPGR) A C D F J B G H K L M 90

MS Lesion Quantification (FSE) A B C D E F G H 91

MTR Analysis 92

Brain Tumor Quantification A B C D 93

MRA Vessel Segmentation A B C D E F 94

Hybrid Segmentation Methods l l Combine boundary-based and region-based methods Each well understood Utilize strengths of both, reduce exposure to weakness of either Advantage l l More reliable When the region-based method is trapped in a local minimum, the boundary-based method can drive it out 95

Fuzzy connectedness with Voronoi diagram l l Use fuzzy connectedness to generate statistics for homogeneity operator in a color space (e. g. , RGB, HCV) Run Voronoi Diagram-based algorithm in multiple color channels Identify connected components Use deformable model to determine final (3 D) boundary 96

Slides with hybrid segmentation examples borrowed from lecture notes of Celina Imielinska (Columbia University) 97

Examples: ________________________ Understanding Visual Information: Technical, Cognitive and Social Factors 98

Method I: Gray Matter GM WM ________________________ Understanding Visual Information: Technical, Cognitive and Social Factors 99

Fuzzy Connectedness with Voronoi Diagram Fuzzy Connectedness: Fuzzy map Binary image from which we generate homogeneity statistics Voronoi Diagram with yellow boundary Voronoi regions Final boundary: a subgraph of the 100 Delaunay triangulation

Fuzzy Connectedness with Voronoi Diagram Fuzzy Connectedness: Fuzzy map Voronoi Diagram with yellow boundary Voronoi regions Binary Image (homogeneity statistics) Final Boundary: a subgraph of Delaunay Triangulation (click to play movie) 101

102

Segmentation of Brain Tissue 103

Hybrid Segmentation: Visible Human Male: Kidney Input data Result: FC Columbia University Result: FC/VD Hand Segmentation Result: FC/VD/CC/DM 104

Hybrid Segmentation: Visible Human Male: Kidney Result: FC/VD/CC Columbia University Result: FC/VD/CC/DM Hand Segmentation 105

Useful Techniques 106

Scale-space and Multi-level/ Multi-scale Techniques l l l Gaussian Pyramid Reduction of data Gain in processing time Gain in robustness Gain in accuracy 107

Details at Different Scales 108

Details at Different Scales 109

Template Matching with Cross-correlation 110

Other Useful Techniques l Morphological operators l l Dilation, erosion, opening, closing Cavity filling Connected component labeling Distance transform l l l Euclidean City block / Manhattan 3 -4 -5 chamfer 111

Evaluation of Image Segmentation Methods 112

Measures and Figures of Merit l The method’s effectiveness can be assessed by several sets of measures l l Precision (reliability) Accuracy (validity) Efficiency (practical viability in terms of the time required) In fact, effectiveness should be assessed by all measures, since one measure by itself is not always meaningful 113

Precision l Three types of precision is usually measured l l l Intra-operator precision Inter-operator precision Repeat-scan precision For each test, volume difference and overlap agreement may be measured For repeat-scan overlap measurement, registration of the two scenes is necessary 114

Accuracy l l Segmentation results of a method are usually compared to some surrogate truth since real truth is rarely available Measures for comparison l l Absolute values (number of voxels, volumes) l True positives (TP) l False positives (FP) l False negatives (FN) Relative values (comparable among different studies) l TP volume fraction (TPVF) l FP volume fraction (FPVF) l FN volume fraction (FNVF) 115

Efficiency l l Unfortunately, this sort of evaluation is often neglected Possible measures l l l Running time (wall clock time) l highly dependent on what type of hardware the program is running on Amount of necessary human interaction l number and length of interactive sessions How convenient it is for the operator l the way the human input is required 116

Motto: “There is no silver bullet” l Whatever technique you choose you have to tailor it to the particular application context l This usually means not only setting parameters but also designing new algorithms built from existing ones, combining different pre- and post-processing techniques with robust algorithms, sometimes even combining several segmentation algorithms to achieve the goal, designing workflows, user interfaces, and validation methods 117