Intelligent Control Systems Laboratory Preliminary Study on Vision-based

Intelligent Control Systems Laboratory Preliminary Study on Vision-based Pen-and-Ink Drawing by a Robotic Manipulator Yan Lu, Josh Lam, Yeung Yam The Chinese University of Hong Kong Dept. of Mechanical and Automation Engineering The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Outline Ø Motivation Ø Background Ø Methodology Ø Demonstration Ø Summary The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Motivation Pen-and-Ink is a traditional form of art, which is still popular today. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Motivation Skill-dependent Time-consuming The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Background A. Robot Platform Ø 5 -DOF manipulator Ø 0. 1 mm movement resolution Ø Visual feedback by CCD camera The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Background B. Pen-and-Ink Properties 1. Stroke is monochromatic; 2. Stroke contributes to both tone and texture; 3. Strokes work collectively, that is, no single one is of critical importance. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology Main challenge Only monochromatic strokes can be used to convey three key aspects of an image: ü structural contents ü tone– brightness and darkness ü and textures The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology A. Structural Contents Step 1. Outline extraction– Canny edge detection Step 2. Structural significance of edges– measured by the “lifetime” in the Gaussian scale space of the original image (Orzan et al. 2007) http: //artis. imag. fr/Publications/2007/OBBT 07/ Step 3. Outline delineation by strokes of different widths according to structural significance The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Original image The Chinese University of Hong Kong Structural significance

Intelligent Control Systems Laboratory Methodology B. Tone Expression The key is to place a stroke at a right location, based on the rule that strokes should be placed evenly: close together in dark areas, and widely spaced in bright areas. To accomplish this hatching process, we need the vision system to provide visual feedback. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology B. Tone Expression (Cont’d) Step 1. Capture the current drawing by camera, blur it by an average filter, transfer it to top view by homography, denoted as Iop; Step 2. Compute the difference between current image and object image: Idf=Ioj-Iop The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology B. Tone Expression (Cont’d) Step 3. Determine the importance image as: Where i denotes the i -th stroke, and a is a parameter, empirically set as 0. 1~0. 5. Step 4. Place the next stroke at the point where the value of the importance image is the largest Step 5. Repeat 1~4 steps The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology B. Tone Expression (Cont’d) Example of determining stroke position Object image Ioj The Chinese University of Hong Kong Current image Iop Importance image Iip

Intelligent Control Systems Laboratory Methodology C. Texture Representation Strokes should be appropriately orientated to represent desirable textures. A natural way is to orientate a stroke according to the gradient direction of the original image. However, in practice, image gradient directions are often too noisy to be used directly. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology C. Texture Representation (Cont’d) Problem illustration Original image The Chinese University of Hong Kong Local texture By gradient

Intelligent Control Systems Laboratory Methodology C. Texture Representation (Cont’d) Since strokes work collectively, a local texture should be expressed by a group strokes with consistent orientations. We propose to select a set of “strong points” whose gradients are large enough to be believed, and calculate stroke direction by interpolation using a few nearest “strong points”. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology C. Texture Representation (Cont’d) Stroke Orientation at p Where pi is one of the n strong points nearest to p, θ(pi) is the gradient direction at pi , and d(p, pi) denotes the distance between p and pi. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology C. Texture Representation (Cont’d) Strong points The Chinese University of Hong Kong By gradient interpolation

Intelligent Control Systems Laboratory Methodology D. Stop Criterion A criterion function is proposed to terminate the hatching process automatically as follows: The first component is a descending function wrt. stroke number i, while the second is an ascending one. They work together to determine the stop point. The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Methodology D. Stop Criterion (Cont’d) The hatching process stops at the minimum point of the criterion function, which can be tuned by b. b=1. 0, i=1000 The Chinese University of Hong Kong b=0. 9, i=1400

Intelligent Control Systems Laboratory Methodology Overview The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Demonstration i=0 i=1026 The Chinese University of Hong Kong i=300 i=700 i=1283 i=1502

Intelligent Control Systems Laboratory The Chinese University of Hong Kong

Intelligent Control Systems Laboratory Summary We have presented a robotic system for automated Pen-and-Ink rendering Ø Structural contents delineated according to structural significance of edges Ø Tone expression based on visual feedback Ø Texture representation by interpolating local gradient directions Ø Automatic stop using a criterion function The Chinese University of Hong Kong

Intelligent Control Systems Laboratory The Chinese University of Hong Kong