Research Group on Visual Computation

Multicue Image and Video Segmentation: a Multi-layer MRF Framework

Icon of project
  • Members: Zoltan Kato
  • Funded by:
    • FRC, National University of Singapore,
    • Hungarian Scientific Research Fund (OTKA),
    • Janos Bolyai Research Fellowship of the Hungarian Academy of Sciences
  • Partners:
    • Ting Chuen Pong (Computer Science Department of the Hong Kong University of Science & Technology),
    • Guo Qiang Song (National University of Singapore)
  • Lifetime: 1999 Jan - 2007 Jan

Description

The human visual system is not treating different features sequentially. Instead, multiple cues are perceived simultaneously and then they are integrated by our visual system in order to explain the observations. Therefore different image features has to be handled in a parallel fashion. In this project, we attempt to develop such a model in a Markovian framework. The model has a multi-layer structure (see image on the left showing the layers in the case of motion and color based segmentation): Each feature has its own layer, called feature layer, where an MRF model is defined using only the corresponding feature. A special layer is assigned to the combined MRF model. This layer interacts with each feature layer and provides the segmentation based on the combination of different features. Unlike previous methods, our approach doesn’t assume common boundaries for different features. The uniqueness of the proposed method is the ability to detect boundaries that are visible only in one of the features.

The proposed model consists of 3 layers. At each layer, we use a first order neighborhood system and extra inter-layer cliques. The image features are represented by multivariate Gaussian distributions. The combined layer only uses the features indirectly, through inter-layer cliques, thus the model is not fusioning the feature data directly, rather it combines label proposals coming from the individual feature layers. The combined layer model also estimates the number of classes and chose those label pairs which are actually present in the input image.

The proposed algorithm has been tested on real video sequences using motion and color features as well as on color textured images using color and texture features. Experimental results demonstrate that the proposed method is quite powerful in combining different features in order to detect boundaries visible only in one of them.

Results

First, we present some segmentation examples using color and texture features. More results are available here.

 

Segmentation results on color textured images

Below, we show some segmentation results based on combined color and motion features. On the sythetic example, we also show detected occlusion boundaries in black color.

 

Segmentation results on real video sequences

 

Comparison with [Khan & Shah, CVPR 2001]
Mother & daughter sequence
Multi-layer model
Khan & Shah, CVPR 2001

 

Result on a synthetic sequence

 

Publications to cite:
  1. Csaba Benedek, Tamas Sziranyi, Zoltan Kato, Josiane Zerubia, A Multi-Layer MRF Model for Object-Motion Detection in Uregistered Airborne Image-Pairs, In Proceedings of International Conference on Image Processing, IEEE, vol. VI, San Antonio, Texas, USA, pp. 141-144, 2007. [bibtex]
  2. Csaba Benedek, Tamas Sziranyi, Zoltan Kato, Josiane Zerubia, Detection of Object Motion Regions in Aerial Image Pairs with a Multilayer Markovian Model, In IEEE Transactions on Image Processing, IEEE, vol. 18, no. 10, pp. 2303-2315, 2009. [bibtex]
  3. Zoltan Kato, Ting Chuen Pong, Guo Qiang Song, Multicue MRF Image Segmentation: Combining Texture and Color, In Proceedings of International Conference on Pattern Recognition, IEEE, vol. 1, Quebec, Canada, pp. 660-663, 2002. [bibtex]
  4. Zoltan Kato, Ting Chuen Pong, Guo Qiang Song, Unsupervised segmentation of color textured images using a multi-layer MRF model, In Proceedings of International Conference on Image Processing, vol. I, Barcelona, Spain, pp. 961-964, 2003. [bibtex]
  5. Zoltan Kato, Ting Chuen Pong, A Multi-Layer MRF Model for Video Object Segmentation, In Proceedings of Asian Conference on Computer Vision (P. J. Narayanan, Shree K. Nayar, Heung-Yeung Shum, eds.), Springer, vol. 3852, Hyderabad, India, pp. 953-962, 2006. [bibtex]

Hichem Abdellali has been awarded the Doctor of Philosophy (PhD.) degree...

2022-04-30


Hichem Abdellali has been awarded the KÉPAF Kuba Attila prize...

2021-06-24