Stochastic Paintbrush Image Transformation

Images can be interpreted in several ways by decomposition into basic functions: strokes, fractals, wavelets, etc...Each of these is natural in some sense: strokes are good representations of letters or shapes, fractals originate from the self-similarity, wavelets are useful for multi-scale representation. When observing an image, our visual system is searching for small image-fragments to describe the scene: contours described by lines and curves, patches, simple textures, strokes. How can we interpret an image? In the biological sense, it is not an easy question. Brain and eye-researchers are looking for the right answer. However, there is an offering answer for this question from an artistic point of view: ask a talented painter and he/she will give a painted interpretation of the world: the scene as the artist sees it. Such an image is made of brush-strokes of different sizes and colors put on the screen one after the other in a sequence by the painter. Small articles are elaborated with fine brushes, while plain surfaces are painted with greater strokes.

Stochastic Paintbrush Transformation (SPT) is a new method to simulate such a painting process. It is based on a random search to insert brush-strokes into a generated image at decreasing scale of brush-sizes, without predefined models or interaction. The main goal of the method is to transform the image into a representation that is very similar to the human sensation of classical artistic images. When defining what we hope from a new process that follows the main features of representational painting, we can describe the main concepts of the new algorithm:

• It should have sharp edges at any level of image-construction;
• There are no fine details below a limit;
• There are sharp details at the finest level as well;
• From a given distance the image should give the same visual scenery as the original.

Other image-painting methods of computer graphics are model- or edge-controlled, or they require human interaction. Unlike these methods, the present stochastic searching algorithm can find any small details (up to the smallest brush-size) without additional processing.

The novelty of our approach is two-fold. First, SPT is a completely automatic (no human intervention, no pre-processing) image-painting algorithm. Second, SPT is constructed in such a way that it provides a multi-scale representation of an image which is close to the human sensation of paintings. In other words, it provides an interpretation of an image. Although this interpretation is quite specific (series of brush-strokes), it can be used in many areas of image processing. For this purpose, we would like to emphasize two important properties of SPT:

1. The original image can be described with the parameters of the consecutive paintbrush strokes. The resulting parameter-series can be used for some moderate compression as well.
2. There is a well-defined scale-space line: First large details, then finer details are proceeded. In this aspect, SPT is similar to the anisotropic diffusion that is based on scale-space theory. However, there is a very important difference between the two scale-space approaches: Anisotropic diffusion enhances the main edges but smoothes the others, while the present paintbrush stroke-rendering has sharp edges at any stage.

The painting process can be applied for scale-space image representation, segmentation and contour detection, image representation for retrieval purposes. Other possible application areas are animated movie generation of painted images or the analysis of real paintings to find their style of construction when considering different brush-stroke techniques.

Figure: Paintbrush transformation example. Effect of bigger and smaller brush sizes.

Original	Big brush	Fine brush

The overall goal of this project is to investigate both theoretical and applied aspects of this new algorithm. A conference paper [1] has been published by our collaborative partners which describes the basic SPT algorithm and its application to image compression. In the current implementation, it is basically a sequential multi-scale image decomposition method, based on simulated rectangular-shaped paintbrush strokes. The resulting images look like good-quality paintings with well-defined contours, at an acceptable distortion compared to the original image (see Fig. ). However, the algorithm takes about 30-60 minutes on a Pentium III PC when generating a 512×512 image, there is no theoretical framework in which the method could be further enhanced, and there is a lot more application that we could explore. Thus, we propose the following research directions:

Image Similarity Measure

SPT provides a description of an image via parameter-series of brush-strokes. This information could be used to capture the (low-level) content of the original image. We will extract various statistics of paintbrush strokes and define a similarity measure based on them. Such measures can be used in an image indexing and retrieval system. Current brush parameters include color, size, and orientation. Therefore not only the classical color or texture content can be measured but it also provides some information about the structure of the image: fine/coarse details, typical orientations, etc...Spatial distribution of brush-parameters may provide further information about higher level content: presence of objects, composition. The method will be tested on a large collection of images.

Image Segmentation

Herein, we will concentrate on the scale-space properties of SPT. Since it enhances the main features, preserves sharp edges and dynamic of colors (see Fig. ), we expect good results in image segmentation. First, an image segmentation algorithm based on the current SPT algorithm will be developed. The algorithm will be tested on several real and synthetic images and results must be compared with those obtained via classical segmentation methods. We consider several ways to improve segmentation quality: use non-rectangular brush-strokes, optimize brush selection (size, shape) for segmentation. Another challenging problem is to detect textured regions. Since multi-resolution approaches are broadly used in extracting texture features (eg. Gabor filters and wavelets) and mathematical morphology tools (granulometry and pattern spectra) have also been efficient in that area, we believe that SPT could be applied successfully to texture detection.

Theory

SPT needs a theoretical framework in which we can provide a mathematical model of the algorithm. This would help us to better understand the behavior of the method and to come out with further enhancements. In particular: reducing the computing cost of the algorithm, dealing with textures (currently SPT is based only on color), effect of different brush-shapes, alternative brush-selection strategies. We think that the method could be formulated in the framework of scale-space theory, mathematical morphology, Markov random fields (MRF) and Monte Carlo simulation.

Collaboration

We are working on this project with Tamas Szirányi, Laszlo Czuni and Zoltan Toth at Image Processing and Neurocomputing Department of the Veszprém University, Veszprém, Hungary.

Bibliography

1

T. Sziranyi and Z. Toth, ``Random paintbrush transformation,'' in 15th ICPR, (Barcelona, Spain), 2000.