Content-based image retrieval and Computer-aided diagnosis systems Paulo

Content-based image retrieval and Computer-aided diagnosis systems Paulo Mazzoncini de Azevedo Marques - Ph. D (pmarques@fmrp. usp. br) Science of Images and Medical Physics Center School of Medicine of Ribeirão Preto University of São Paulo

DIAGNOSIS Signal Detection Theory – Decision Matrix The Essential Physics Of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott Williams Wilkins, Philadelphia, USA, 2002.

DIAGNOSIS PERFORMANCE MEASUREMENTS True Positive Fraction (TPF) TPF = TP/(TP+FN) Sensitivity = TP/(TP+FN) = TPF Specificity = TN/(TN+FP) = (1 -FPF) False Positive Fraction (FPF) FPF = FP/(FP+TN) Accuracy = (TP+TN)/(TP+TN+FP+FN)

DIAGNOSIS PERFORMANCE MEASUREMENTS ROC curves (receiver operating characteristic) Az The Essential Physics of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott Williams Wilkins, Philadelphia, USA, 2002. sortit ion CAD /CB IR

Computer-aided Diagnosis Definition: (CAD) A diagnosis made by a radiologist using the output of a computerized scheme for automated image analysis as a diagnostic aid (second opinion). K. Doi - Computerized Medical Imaging and Graphics 31 (2007) 198– 211 With CAD, the performance by computers does not have to be comparable to or better than that by physicians, but needs to be complementary to that by physicians (synergy). Nishikawa RM - Applied Radiology, Suplement November 2001: 14 -16

CAD TYPES OF AID n Computer-aided Detection (CADe) – usually confined to marking suspicious structures and sections – Initially approved by FDA-USA in 1998 for mammography

CAD TYPES OF AID n Computer-aided Diagnosis (CADx) – usually focused on to classify detected structures or regions (more academic).

CAD KNOWLEDGE INVOLVED n Computer Vision (quantitative features extraction) – Preprocessing (noise reduction and enhancement) – Segmentation (regions, edges, structures) – Structure/ROI Analyze (form, size and location, texture, topology) n Artificial Intelligence (classification) – Features selection – Classification

CAD- EXAMPLE CAD in Orthopedic Radiology: Quantitative Evaluation of Vertebral Morphometry Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa, Rangaraj M. Rangayyan, Paulo M. Azevedo-Marques School of Medicine of Ribeirão Preto, University of São Paulo, Ribeirão Preto, São Paulo, Brazil Department of Electrical & Computer Engineering, University of Calgary, Alberta, Canada

$Vertebral fractures are important indicators of osteoporosis. Insufficiency fractures of the vertebrae are usually$

Vertebral fractures are important indicators of osteoporosis. Insufficiency fractures of the vertebrae are usually seen as a partial collapse of the vertebral body. Both semi-quantitative and quantitative analysis of spinal and vertebral deformities could assist in the diagnostic decision-making process and in guiding therapeutic procedures.

Grading of Vertebral Fractures (Genant) Genant HK et al. Journal of Bone and Mineral Research, 8: 1137– 1148, 1993. Manual quantitative morphometric analysis is labor-intensive and subject to inter-observer and intra-observer variability

CAD - Pipeline Image Acquisition (film digitization) Vertebral Plateau Segmentation (Gabor Filters and ANN) Vertebral Morphometry (vertebral height measurement) Genant Grading Analysis of Vertebral Height (rule-based classification)12 12

Marking Reference Points Five points, P 1–P 5, were manually marked near the middle of the intervertebral spaces spanning the range of L 1–L 4 by using a pointer. The distances between the points were calculated automatically: D(1, 2), D(2, 3), D(3, 4), and D(4, 5). Using 75% of each distance measure, the corresponding line joining the manually marked points was shifted in either direction along its perpendicular to create a quadrilateral for each vertebra.

Segmentation F S Segmentation is based on the detection and characterization of oriented edges using Gabor filters and classification using a neural network. F. J. Ayres and R. M. Rangayyan. Journal of Electronic Imaging, 16(2): 023007: 1– 12, 2007. Each image was filtered with a bank of 180 Gabor filters (sinusoidally modulated Gaussian functions) in steps of 1 degree Width = 4 pixels and elongation factor = 8. For each pixel, the magnitude response and angle of the Gabor filter providing the highest output were used to compose a Gabor magnitude image and an orientation field.

Result of Gabor Filters original image Gabor magnitude response coherence image

Manual Delineation of Vertebral Plateaus 5 -pixel thick lines drawn for L 1 -L 4

Detection of Vertebral Plateaus with a Neural Network Pixels in regions corresponding to L 1 -L 4 were obtained from the original image, the Gabor magnitude response, and the coherence image for analysis using a logistic sigmoid neural network. A leave-one-out training and testing procedure was used. The output of the neural network for each pixel was used to label the pixel as belonging to a vertebral plateau or not.

Detection of Vertebral Plateaus with a Neural Network Original image Output of neural network Manual annotation

Vertebral Morphometry convex skeleton hull skeleton apply 19 skeleton to plateaus remove spurs

Measurement of Vertebral Height Measures of height obtained for a normal vertebral body Measures of height obtained for an abnormal vertebral body

$Initial Results of CAD Results of computer-aided grading of vertebral fracture using the method$

Initial Results of CAD Results of computer-aided grading of vertebral fracture using the method proposed by Genant. Values along the main diagonal correspond to correct classification by the CAD (86%).

Content-Based Image Retrieval Definition: CBIR Content-based image retrieval (CBIR), also known as query by image content (QBIC) is the application of computer vision techniques to the image retrieval problem, that is, the problem of searching for similar images in large databases. Content-based means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags, and/or descriptions associated with the image. Mu ller H. et al. International Journal of Medical Informatics (2004) 73, 1— 23

CBIR Framework Extracted Features qu ery ob jec t ID of retrieved features Color Query texture Features of the query object by Similarity Module Features Extraction shape . . . Similar Features + distances + Images ID Indexing Query Image feedback structure Similar Images

Computer Vision Feature Extraction Original Image n Feature Vector X 1 X 2. . . XN Image Processing Techniques – Feature Extraction n Feature Vector (based on shape, texture, color or others techniques)

Similarity Searches Data Domains – MAM-Metric Access Methods n Multi-dimensional Domains n Adimensional Domains – Fingerprints, words and so on. n Example – mvp-tree, M-tree, Slim-Tree (query by example)

Similarity Searches Metric Space n n Metric Space is a pair: M=(D, d) where: – D is the characteristic domain of objects – d is a metric distance function. Properties of the distance function d(): – symmetry: n d(x, y) = d(y, x) – non-negativity: n 0 < d(x, y) < , x y e d(x, x) = 0 – triangle inequality: n d(x, y) d(x, z) + d(z, y) Where x, y e z are objects of D Minkowski Function

Similarity Searches Query Definitions n Range Query: “Find all the images that are within 10 units of distance from image 1. ” n Nearest Neighbor Query (k-NN): "Find the 5 nearest images to image 1”

CBIR PERFORMANCE MEASUREMENTS Precision X Recall curves

CBIR- EXAMPLE Content-based retrieval of color images of dermatological ulcers. Silvio Moreto Pereira, Marco Andrey C. Frade, Rangaraj M. Rangayyan, Paulo M. Azevedo-Marques School of Medicine of Ribeirão Preto, University of São Paulo, Ribeirão Preto, São Paulo, Brazil Department of Electrical & Computer Engineering, University of Calgary, Alberta, Canada

Dermatological Ulcers may appear on the legs due to chronic diseases such as diabetes and venous insufficiency. Visual assessment of pathological regions and evaluation of macroscopic features are used for the diagnosis of skin lesions in clinical practice. The appearance of a lesion provides important clues regarding the diagnosis, severity, and prognosis. The red-yellow-black-white (RYKW) model of tissue composition is useful as a descriptive tool.

Ulcer Tissue Types Granulation (red) Scar or necrosis (black) Fibrin (yellow) Mixed

Imaging of Ulcers 32

Representation of Color Images Each color image was represented using the standard representations as • [red, green, blue] or RGB, • [hue, saturation, intensity] or HSI, and • L*u*v*.

Segmentation of Ulcer Images Original image Hue-saturation histogram Red regions Yellow regions S>0. 4 and S>0. 2 and H 300º to 0 to 30º H 30º to 90º Black regions Ulcer regions S<0. 2 and I<0. 25*max

Features Extraction Multispectral cooccurrence matrices (CCMs) obtained from the RGB, HS, u*v*, and a*b* components. a total of 111 statistical features were extracted from the R, G, B, H, S, u*, v*, and b* components to characterize each color image

KNN Based Retrieval using Cosine Distance

CAD-CBIR/PACS INTEGRATION Visualization Imaging Workstation Modality Speech Recognition PACS RIS DB DICOM HL-7 High-Speed DICOM Network HIS/MIS Firewall RAID Archive Web-based RIS/PACS/EMR PACS – Picture Archiving and Communication System

PACS AND IMAGING INFORMATICS: Basic Principles and Applications - H. K. Huang, New Jersey - USA, 2004

Example of CAD/PACS integration framework: – Communication services (DICOM functionalities) – Image-processing pipeline (CAD-CBIR server) Azevedo Marques PM et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180 -S-181.

Example of CAD-PACS integration cores. put("normal", Color. WHITE); cores. put("ground-glass", Color. BLUE); cores. put("reticular-linear", Color. GREEN); cores. put("micronodules", Color. RED); cores. put("honeycombing", Color. YELLOW); cores. put("emphysematous", Color. MAGENTA); cores. put("consolidation", Color. CYAN); Azevedo Marques PM, et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180 -S-181.

Example of CAD/CBIR-PACS integration CAD scheme using CBIR approach. Bin Zheng. Computer-Aided Diagnosis in Mammography Using Content based Image Retrieval Approaches: Current Status and Future Perspectives. Algorithms. 2009 June 1; 2(2): 828– 849. Example of applying a CAD scheme using CBIR approach to detect and classify a suspicious breast mass region. A suspicious mass is automatically detected by CAD scheme and queried by the observer (pointed by the arrow). In CAD workstation, the mass region segmentation (boundary contour), 12 CBIR-selected similar ROIs, and both detection and classification scores are displayed. Among the 12 similar ROIs, 8 depict malignant masses (marked by Red frame), 2 depict benign masses (marked by Green frame), and 2 depict CAD-cued false-positive regions (marked by Blue frame).

CONCLUSION Computer-aided diagnosis has become a part of clinical work in the detection of breast cancer by use of mammograms, but is still in the infancy of its full potential for applications to many different types of lesions obtained with various modalities. Content-based image retrieval is an alternative and complementary approach for image retrieval based on key-words and metadata. Initial results are very promising about using CBIR as a diagnostic support tool In the future, it is likely that CAD and CBIR schemes will be incorporated into PACS CAD and CBIR will be employed as useful tools for diagnostic examinations in daily clinical work.

THANK YOU! pmarques@fmrp. usp. br