Part 1 Ideas for themes topics misconceptions 1 Biometrics

Part 1 Ideas for themes/topics/misconceptions 1. Biometrics – Reality and Myths 2. Biometrics in Real Life 3. Biometrics –privacy

Hollywood Face Recognition Common misconceptions – 100% match to any image at any angle – Instantly recognize any person – Tied into a “super database” that knows who everyone is – Available to and in use by law enforcement Movie scene (downloaded from www pubic domain)

Hollywood DNA Misconceptions – Access to a super database that has everyone’s DNA – Automatically and rapidly processes a sample Movie scene (Pubic domain)

Hollywood Fingerprints Common misconceptions – instant match – Only one fingerprint is enough – Available to use at any location Screenshot from “Man in black” movie (Pubic domain)

Face Recognition Today • Today’s Reality – Affected by lighting, angle, quality of captured image – Requires a “high-end” computer for real-time face capture/processing – Many are stand-alone systems – Being evaluated, not deployed Ft. Lauderdale Airport, Florida Identi. Face system interface

Face Recognition Today • Today’s Reality – Varying confidence of match depending on application – Multiple unique and proprietary image formats make sharing hard – Intelligence images not available to local law enforcement or corrections – Data sharing across jurisdictions is a problem 100 known images in the database Identiface

Face Recognition Today • Face Recognition Vendor Test 2002 and 2006 provides independent government evaluations of commercially available and mature prototype face recognition systems. • Results available at http: //www. itl. nist. gov/iad/894. 03/f ace/face. html • • FRVT 2002 and 2006 evaluated performance on: High resolution still imagery (5 to 6 mega-pixels) 3 D facial scans Multi-sample still facial imagery Pre-processing algorithms that compensate for pose and illumination

Fp. VTE Fingerprint Vendor Technology Evaluation • The Fingerprint Vendor Technology Evaluation (Fp. VTE) 2003 is an independently administered technology evaluation of fingerprint matching, identification, and verification systems. • Assessed the capability of 18 vendors fingerprint systems to meet requirements for large-scale and small-scale real applications. • Consists of multiple tests performed with combinations of fingers and different types and qualities of operational fingerprints • Conducted by the National Institute of Standards & Technology (NIST) between October and November 2003 on behalf of U. S. Department of Justice. • Report made public in June 2004 at http: //Fp. VTE. nist. gov

Fp. VTE Fingerprint Vendor Technology Evaluation Some of the Results • Systems that performed most accurately were developed by NEC, SAGEM, and Cogent • The variables that had the largest effect on system accuracy were the number of fingers used and fingerprint quality. • Different systems were distinguished by how they performed across the spectrum from good to bad (performance separation was really on “bad” quality).

Applications: Biometrics in Schools • National News Reports • • Interior System Exterior System Eleven, Single-Eye LG Electronics Iris. Access 2200 Iris Recognition Cameras were Evaluated – 6 cameras within closed areas in 3 schools – 5 cameras were located outdoors with fabricated protective closures Unsuccessful attempts mostly due to camera capture errors (16%) and access attempts by unknown users (5. 8%) Issues remaining: – Tailgating (accepted users holding door open for others) – Ability to Capture Iris Outdoors (lighting conditions)

Applications: Biometrics in Correction Facilities • Demonstration and Assessment of Facial Recognition Technology at Prince George’s County Correctional Facility (Visionics corporation) • Visionics (now Identix) system installed based on results of FRVT 2000 • Required re-work of room lighting, addition of camera lights, and training of staff and system users. • Interfaced with Staff and Volunteer Access Control System to verify identity of staff and volunteers upon entry and exit from the facility • Augments manned access control station

PRIVACY: Video surveillance When you walk into a building from a parking lot? When you shop at your favorite store? Go to your bank?

Privacy: Video surveillance When you buy gas for your car? Pay at a toll booth? Video surveillance is a daily fact of life. Current motivation is mostly to avoid theft in commerce.

Summary: • Biometric Technology is concerned with representation, storage, matching, synthesis and visualization of biometric information. • Tremendous advance has been achieved over the last few years in both fundamental theoretical development, matching and synthesis, as well as biometric hardware and software products. • Many misconceptions remain, and are supported by mass media and newspapers • There are real privacy concerns with video surveillance technology, which is not biometrics.

Part 2 Ideas for biometric theme – definitions, variety of resources 1. Biometrics Identifiers - classification 2. Biometric market share 3. Biometric – how to make choice

Biometric identifiers From G. Bromba research

Biometric Market Share

Comparison of biometric techniques

Palm

Hand vein

Facial Thermogram

Ear print

Retina

Which Biometric is the Best? Universality (everyone should have this trait) Uniqueness (everyone has a different value) Permanence (should be invariant with time) Collectability (can be measured quantitatively) Performance (achievable recognition accuracy, resources required, operating environment) • Acceptability (are people willing to accept it? ) • Circumvention (how easily can it be spoofed? ) • • •

Selecting a Biometric Selecting the ‘right’ biometric is a complicated problem that involves more factors than just accuracy. It depends on cost, error rates, computational speed, reliability, privacy and easy of use.

Part 3 Historical artifacts 1. History of fingerprints 2. First automated computer systems 3. Historical artifacts 4. Fingerprint devices in real life 5. Optional – sensing devices 6. Face recognition – methods, illusions, image synthesis 7. Other biometric devices

Cartoon (copyrighted)

History of fingerprints • Use of fingerprints to associate a person with an event or transaction can be traced to ancient China, Babylonia and Assyria as early as 6, 000 BC. 28

Archaelogical remains 29

History of fingerprints • 1750 B. C. - people in Babylon used fingerprints to sign their identity on clay tablets • 300 B. C. -Emperors of China used personalized clay seals • In 1686, Marcello Malpighi, an anatomy professor at the University of Bologna, wrote in a paper that fingerprints contained ridges, spirals and loops. • In 1856, Sir William Herschel, a British Magistrate in Jungipoor, India, used fingerprints (actually palmprints) to certify native contracts. 30

History of fingerprints • During the 1870 s, Dr. Henry Faulds, a British surgeon in Japan, after noticing finger marks on ancient pottery, studied fingerprints, recognized the potential for identification, and devised a method for classifying fingerprint patterns. • 1880 -Faulds published an article in "Nature, " discussing fingerprints as a means of personal identification. He is also credited with the first fingerprint identification of a greasy fingerprint left on an alcohol bottle. 31

32

History of fingerprints • In 1880, Alphonse Bertillon, a Paris police department employee and son of an anthropologist, developed a system of anthropometry as a means for classifying criminals and used this system to identify recidivists. Anthropometry (a system of cataloging an individual's body measurements such as height, weight, lengths of arm, leg, index finger etc. ) was shown to fail in a famous case at Leavenworth Prison, where two prisoners, both named William West, were found to have nearly identical measurements even though they claimed not to be biologically related. 33

Alphonse Bertillon 34

History of fingerprints • Francis Galton, an anthropologist, began a systematic study of fingerprints as a means of identification in the 1880 s. In 1892, he published the first book on fingerprints. • In 1897, Sir Edward Henry, a British police officer in India, established a modified fingerprint classification system using Galton's observations. This system was ultimately adopted by Scotland Yard in 1901 and is still used in many Englishspeaking countries. 35

Historical Overview • Henry Faulds, William Herschel and Sir Francis Galton proposed quantitative identification through fingerprint and facial measurements in the 1880 s. • Edmond Locard introduced using biometrics in forensic identification in 1920 s. 36

History of fingerprints • 1924 -an act of U. S. Congress established the Identification Division of the FBI (Federal Bureau of Investigation) with a database of 810 000 fingerprint cards • Most of the early fingerprint identification systems were put into place in major metropolitan areas or as national repositories. Juan Vucetich established a fingerprint file system in Argentina in 1891, followed by Sir Edward Henry in 1901 at Scotland Yard in England. 37

Classification: Manual Card Files In the Henry classification system, numerical weights are assigned to fingers with a whorl pattern. A bin number, based on the sum of the weights for the right hand sum of the weights for the left hand is computed to generate 1, 024 possible bins. Letter symbols are assigned to fingers: capital letters to the index fingers and lower-case letters to other fingers. These are combined with the numeric code to further subdivide the 1, 024 bins. Each of these pattern groupings defines bins into which fingerprint cards with the same pattern group are placed. A bin might be a folder in a file drawer or several file drawers if it contains a common pattern group and the file is large. 38

Classification • Fingerprint patterns comprise of loops (left or right), whorls and arches. The patterns are differentiated based on the presence of zero, one or two delta regions. A delta region is defined by a tri -radial ridge direction at a point. Arch patterns have no delta, loops have one delta, and whorls have two deltas. Some of the common fingerprint types. The core points are marked with solid white circles while the delta points are marked with solid black circles. 39

Matching Fingerprint matching prior to automation involved the manual examination of the so-called Galton details (ridge endings, bifurcations, lakes, islands, pores etc. collectively known as "minutiae"). Prior to the late 1960 s, neither the available computer systems that could display fingerprint images for comparison were affordable, nor a significant number of digital fingerprint images were available for display. Consequently, the comparison was manual, requiring a magnification glass for comparing the features of the many candidate prints manually retrieved from the database files. 40

Early Automation Efforts • US NBS/NIST Research: In the mid-1960 s, the National Institute of Standards and Technology (NIST) initiated several research projects to automate the fingerprint identification process. The efforts were supported by the Federal Bureau of Investigation (FBI) as part of an initiative to automate many of the processes in the Bureau. • Royal Canadian Police: By the mid-1960 s, the fingerprint collection of the Royal Canadian Mounted Police (RCMP) had grown to over a million tenprint records. The videofile system was operational until the mid-1970 s, when the RCMP installed the first automated fingerprint identification system (AFIS). 41

Early Automation Efforts • FBI: In the USA, at about the same time that the RCMP and the UK Home Office were looking for automation technologies, the FBI was investigating the possibilities for automating the fingerprint identification operations. In the mid-1960 s, the FBI signed research contracts with 3 companies to build a working prototype for scanning FBI fingerprint cards, completed by the early 1970 s. • United Kingdom: In the UK, over about the same time-scale as the FBI, the Home Office was working within its own Scientific Research and Development Branch (SRDB). • Japan: The Japanese National Police (JNP) who had a fingerprint file of over six million records, also initiated study of the automation possibilities. 42

Further Development • During the 1970 s, the FBI contracted with a number of organizations as well as developed their own research organization to manage the numerous projects that lead the way to the Integrated Automated Fingerprint Identification System (IAFIS). • The transition to a large-scale imaging application environment provided enormous challenges for everyone at that time, but it was especially challenging for the FBI to implement a system to manage up to 35, 000 image-based transactions per day. • The efforts put into AFIS interoperability by NIST under the FBI sponsorship resulted in an ANSI/NIST standard for data interchange. This standard was initially crafted in mid-1980, is updated every 5 years and defines data formats for images, features and text. 43

Early Devices • The FBI initiated a research program to build an engineering model of a scanner that could sample an object area of 1. 5 X 1. 5 in at 500 pixels per inch (DPI), with an effective sampling spot size of 0. 0015 in, signal-to-noise ratio (S/N) in excess of 100: 1 and digitized to at least 6 bits (64 gray levels). • In the late 1960 s, these requirements could only be met by a system that used a cathode ray tube and a precision deflection system, an array of tubes that measure and reflected light, and amplifier-digitizer to convert the electrical signal into a digital value for each pixel. • There were relatively few scanning devices by the late 1970 s that met the technical characteristics requirements of 500 dpi, a 0. 0015 inch effective sample size, greater than 100 S/N noise ratio and a 6 bit dynamic range. • It ten more years the scan quality standards were set by IAFIS, which is the current benchmark for scanning of fingerprint images, 44 requiring 200 or more gray levels.

Historical Overview Early Computerized Systems • Speaker and fingerprint recognition systems were first AUTOMATED systems in 1960. The potential for application of this technology to high-security access control, personal locks and financial transactions were recognized in the early 1960 s. • The 1970 s saw development and deployment of hand geometry systems, the start of large-scale testing and increasing interest in government use of these "automated personal identification" technologies. There are currently 180 readers used by about 18, 000 enrolled users. • Retinal and signature verification systems came in the 1980 s, followed by the face systems. • Iris recognition systems were developed in the 1990 s. 45

Biometrics examples Modern and Privacy Biometric measures can be used in place of a name, Social Security number or other form of identification to secure anonymous transactions. Walt Disney World sells season passes to buyers anonymously, then uses finger geometry to verify that the passes are not being transferred. (Image of pass to Disney World) The real fear is that biometric measures will link people to personal data, or allow movements to be tracked. After all, credit card and phone records can be used in court to establish a person's activities and movements. Phone books are public databases linking people to their phone number. These databases are accessible on the Internet. “Reverse" phone books also exist (a name from a phone number). Unlike phone books, databases of biometric measures cannot generally be reversed to reveal names from measures because biometric measures, although distinctive, are not unique. 46

Modern Examples 47

Biometricsapplications Everyday and Privacy • Samples of these documents can be requested: • Five US states have electronic fingerprint records of social service recipients (Arizona, California, Connecticut, New York and Texas). • Six states (California, Colorado, Georgia, Hawaii, Oklahoma and Texas) maintain electronic fingerprints of all licensed drivers. • Nearly all states maintain copies of driver's license and social service recipient photos. • FBI and state governments maintain fingerprint databases on convicted felons and sex offenders. • Federal government maintains hand geometry records on those who have voluntarily requested border crossing cards. 48

• Interesting facts Biometric and Privacy • Like telephone and credit card information, biometric databases can be searched outside of their intended purpose by court order. • Unlike credit card, telephone or Social Security numbers, biometric characteristics change from one measurement to the next. • Unlike more common forms of identification, biometric measures contain no personal information and are more difficult to forge or steal. • Biometric measures can be used in place of a name or Social Security number to secure anonymous transactions. • Some biometric measures (face images, voice signals and "latent" fingerprints left on surfaces) can be taken without a person's knowledge, but cannot be linked to an identity without a pre-existing database. • Searching for personal data based on biometric measures is not as reliable or efficient as using better identifiers, like legal name or Social Security number. • Biometric measures are not always secret, but are sometimes publicly observable and cannot be revoked if compromised. 49

OPTIONAL: Fingerprint Sensing: optical FTIR Frustered Total Internal Reflection (FTIR): The oldest and most used livescan technique. Reflection allows the ridges (appear dark) to be discriminated from the valleys (appear bright). Because devices sense a 3 D surface, they cannot be easily deceived by a photograph or printed image of a fingerprint. 50

Fingerprint Sensing: FTIR with a sheet prism 51

Optical Fibers 52

Optical Sensors 53

Ultrasound sensors 54

Solid State Sensing Thermal Electric Field Piezoelectric 55

Really rare examples of fingerprints 56

Face Recognition (copyrighted cartoon)

Imaging

Face tracking in digital cameras Foto. Nation Face Tracker http: //www. fotonation. com/index. php? module=product&item=23

Tracking through background Cha Zhang (Microsoft Research) – my colleague- uses background segmentation for face identification and tracking

Psychological Aspects of Face Perception and Recognition Visually Derived Semantic Categories of Faces Humans can recognize faces along a number of dimensions referred as • “visually derived semantic categories”: race, sex, and age. • personality characteristics (visually specified, albeit abstract, categories): face that looks “generous”, or “extroverted” An intriguing aspect of this phenomenon is that making such judgments actually increases human accuracy when compared to making physical feature-based judgments (e. g. , nose size). In contrast to the information needed to specify facial identity, visually derived semantic categorizations are based on the features a face shares with an entire category of faces. There has been less research on the perception of visually derived semantic categories than on the face recognition.

Thatcher illusion. The inverted face appears normal. Upright, however, the configural distortion is evident. The illusion illustrates the limits of configural processing for atypical views of the face.

Modeling expression This system (FACS) is purely descriptive and includes no inferential labels. By converting codes to EMFACS or similar systems, face images may be coded for emotion-specified expressions as well as for more molar categories of positive or negative emotion. 64

Computationally-derived information in images and three dimensional head models that specifies the gender of a face.

SUMMARY part 3 Although many recent advances and successes in automatic facial expression analysis have been achieved, many questions remain open. Some major ones are: How do humans correctly recognize facial expressions? Is it always better to analyze finer levels of expression? Is there any better way to code facial expressions for computer systems? How do we obtain reliable ground truth? How do we recognize facial expressions in real life? Ho do we best use the temporal information? How may we integrate facial expression analysis with other modalities? Which role psychology plays in the process? What impact quality of devices has on the result of recognition?

Other technologies – signature authentication

Digitizing Tablet

Iris synthesis An ocularist's approach to human iris synthesis. Lefohn et. al. 2003. Used with permission.

Summary: In BT lab, we are trying to find answer on those and other questions related to multi-modal biometric and intelligent data representation and processing. Asimo, Honda robot – one example of attempt to make robot resemble humans in both appearance and action.