Task 3 Semi-Automatic System for Pollen Recognition

Task 3 : Semi-Automatic System for Pollen Recognition Partners: – REA (Barcelona) – REA (Cordoba) – LASMEA (Clermont-Ferrand) – INRIA (Sophia-Antipolis) 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Global architecture schema 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Material and Methods • System Hardware: – – – Light microscope 3 axes micro-positionning device CCD colour camera Image acquisition card PC computer • Pollen slides are prepared by technicians – Pollen grains are sampled using Hirst traps – Pollen grains are coloured with fuchsine (4µg/100 ml) 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

System Hardware A light microscope is driven automatically by a computer 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

3 mm 13 mm slide Adhesive (from hirst sampler) label 25 mm (slide width) Slide preparation for automatic analysis 75 mm (slide length) • Choice of final coloration: 4µg/100 ml 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Man-Machine Interface The system can work in supervised or automatic mode 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Man-Machine Interface • Sweeping scheme acquisition and convertion • Slide information acquisition (for slide information automated management) 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Pollen detection and extraction • Sweep and scan the slide for pollen grains (x, y) • Focus on the objects of the scene (z) – Optimisation of a sharpness criterion (SML) • Detect pollen grains in the scene (2 D) – Colour coding: principal component analysis on RGB – Split-and-merge segmentation with markovian relaxation – Pollen detection by colour analysis (luminance/chrominance) • Extract pollen grains (3 D) – 3 D digitisation from the central image 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Pollen Grain Detection and Localisation • Automated sweeping and localisation (x 20 or x 40 lens) • Fully automated or semi-automated extraction of pollen grains (x 40 or x 60 lens) Segmentation Localisation Automatic pollen grain detection 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Pollen Grain Extraction • 3 D acquisition of pollen grains – set of images at different depths For each grain • 100 optical sections • step = 0. 5 microns Features may appear on different heights 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Slide Supply: REA to LASMEA Number of slides for development: • Reference pollen slides: 74 • Aerobiological slides: 37 Number of slides for validation: • Reference pollen slides: 24 • Aerobiological slides: 12 TOTAL: 147 * Reference pollen slides: real pollen grains without dust or pollution * Aerobiological slides: real pollen grains from Hirst traps (true conditions) 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the image acquisition module Validation with reference slides: Pollen Type Poaceae Cupressaceae Parietaria Olea Total pollen analysed 273 325 89 - Number of pollen located 259 291 89 - Nº pollen not located 14 34 0 - Percentage of location 94, 9 % 89, 5 % 100 % 0% 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the image acquisition module Validation with aerobiological slides: Btu 1101 d 2 Btu 2100 d 3 Example of good results Example of bad results • True positive detection (pollen detected) : 80% • False positive detection (non-pollen detected) : 25% 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Global architecture schema 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Goal for recognition • Identify the 4 ASTHMA types from other pollen types • Similar in appearance • Flowering at the same time of the year Detect true positive • Do not identify another type as an ASTHMA type Avoid false positive 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Main Pollen Types Studied and Similars Cupressaceae Olea Parietaria Poaceae Populus Brassicaceae Fraxinus Ligustrum Phillyrea Salix Broussonetia Morus Urtica membranacea Celtis Coriaria 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Reference / Aerobiological images Reference images: Aerobiological images: 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Knowledge acquisition General knowledge acquisition Specific knowledge acquisition 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Palynological Knowledge • The system tries to mimic the palynologists • Pollen knowledge is used to identify each grain • Knowledge sources from – Palynology – Aerobiology 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Second module: Pollen recognition • First step: coarse classification – Global measures on the grain (2 D) • Size, colour (RGB), shape, convexity, . . . – Sampling date (external data for flowering season) – First estimations of possible types (sorted hypothesys list) 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Global measures & flowering information 1) Global measures – taken on the central image of the grain – colour, size, shape, … 2) Flowering information – taken from pollinic calendar 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Second module: Pollen recognition • Second step: fine classification – Search specific characteristics (3 D) – Need specific knowledge about pollen types – Driven by the first hypotheses (test only most possible types) – Refine until no ambiguity remains 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Search pollen characteristics – 3 D analysis is made to find possible locations – Search is done on isolated pollen masks Focus analysis vs Image number Image Full Grain Interior Exine 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Search of specific characteristics Example: Poaceae pore Image 35 /100 02/10/2001 Image 50 / 100 Image 65 / 100 Task 3: Semi-automatic System for Pollen Recognition Image 80 / 100

Search of specific characteristics Example: Cupressaceae cytoplasm Above central image Below central image Sum of bright regions 02/10/2001 Sum of dark regions Task 3: Semi-automatic System for Pollen Recognition

Search of specific characteristics Example: Olea reticulum – Network located on the external surface of the grain – Detection steps: • Check if the grain is reticulated • Localise the reticulum (3 D) • Analyse the reticulum 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Aerobiological images: dust removal 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Sequence Supply: LASMEA to INRIA Number of sequences for development: • Reference pollen sequences: 350 • Aerobiological sequences: 140 Number of sequences for validation: • Reference pollen sequences (set 1) 150 • Reference pollen sequences (set 2) 85 • Aerobiological sequences: 211 TOTAL: 02/10/2001 936 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module • Problems with reference validation data sets: Development set 02/10/2001 Validation set 2 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module • Calibration of the system has changed !! – Calibration is not robust enough – Need an automatic calibration method • New steps for validation – Reference images • leave-one-out on the development set • tests on validation sets with different configurations – Aerobiological images • training on reference images / test on aerobiological images 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module • Validation using leave-one-out method – Testing using the development set • Learn parameters on all but one pollen grain • Test one pollen grain • Loop for each pollen grain – Simulation of the sampling date (flowering - Bellaterra) • Results: – Cupressaceae: 94. 4% – Olea: 100% - Parietaria: 100% - Poaceae: 100% – 5 classes (ASTHMA types / others): 99. 7 % – 32 classes (all pollen types): 77. 7 % 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module Tests with validation sets using different configurations 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module • Tests with validation sets using different configurations – Reduce the colour dependency of the system – Simulation of the sampling date (flowering - Bellaterra) • Good results for Olea, Cupressaceae and Poaceae – Not with the same test – No general configuration to recognise all – Several Poaceae grains are recognised as Cupressaceae • Difficult to recognise Parietaria – It was configured to be recognised using colour – Hard to recognise with only size + flowering information 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the recognition module • Validation using aerobiological images – Problems: • Work with incomplete data (occlusions by dust and particles) • Training was done using reference images (different conditions)!! • Results: – Cupressaceae: 69. 2% – Olea: 25% - Parietaria: 0% - Poaceae: 20. 7% – 5 classes (ASTHMA types / others): 29. 6 % 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the semi-automatic system for pollen recognition • Acquisition module – Excellent results, except for Olea – Interface should be improved for non-experts in computer science like palynologists • Recognition module – Shows the feasibility of pollen recognition – Further investigations are needed to be more robust 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

Validation of the semi-automatic system for pollen recognition • Prospects: – Calibration • automatic procedure to ensure standard conditions – Data set (training) not representative enough • reference / aerobiological • need complete data sets for all pollen types (not only 4 types) – Algorithms are too much colour-dependant • find and develop new algorithms more robust 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition

New European project proposal • Information Society Technologies Programme • Call in Cognitive Vision • Deadline: 17 october 2001 • Proposal: CATEGORIES • Cognitive Vision: Automated Techniques for Effective Goal-based Object Reasoning in Intelligent Embedded Systems • Participants: • Computer science labs: INRIA, COGS, LASMEA • Industrial: TIMEAT • End-users group: UCO, UAB, UHU, ISAO, HNO, PRU, FIN • Coordinator: ERCIM 02/10/2001 Task 3: Semi-automatic System for Pollen Recognition