ABSTRACT: In this paper we study the feasibility of developing a search engine capable of retrieving images from a granite image database based on a query image that is similar to the intended targets. The main focus was on the determination of the set of colour and/or texture features which yields highest retrieval accuracy. To assess the performance of the considered image descriptors we created a granite image database, formed by images recorded at our laboratory as well as taken from the Internet. Experimental results show that colour and texture features can be successfully employed to retrieve granite images from a database. We also found that improved accuracy is achieved by combining different colour and texture feature sets through classifier fusion schemes.

KEYWORDS: granite, visual appearance, colour, texture, image retrieval systems, CBIR.

RESUMEN: En este artículo estudiamos la viabilidad de desarrollar un buscador para bases de datos de imágenes de granito que realice las búsquedas basándose en un criterio de similitud visual con la imagen que define la consulta. El estudio se centra en la determinación del conjunto de características de color y/o textura que proporciona una recuperación más exacta. Para evaluar las prestaciones de los descriptores de imagen considerados, hemos creado una base de datos de imágenes de granito, formada tanto por imágenes grabadas en nuestro laboratorio como por imágenes encontradas en internet. Los resultados experimentales muestran que las características de color y textura se pueden emplear con éxito en la búsqueda de imágenes de granito en una base de datos. Los resultados obtenidos también muestran que combinando diferentes características de color y textura mediante esquemas de fusión de clasificadores, la recuperación de imágenes mejora.

PALABRAS CLAVE: granito, apariencia visual, color, textura, recuperación de imágenes, CBIR.

Manufacturing of granite slabs comprises visual inspection tasks at different production stages. Grading (i.e., grouping products into lots of similar visual properties) and defect detection (such as stains, veins, etc.) are quality control procedures routinely performed in granite industry. Visual inspection is also useful in the commercialization stage, when we have to search for tiles of a given visual appearance in order to replace broken pieces or to extend previous supplies. These tasks are usually carried out by a human expert, who subjectively assesses the visual properties of the granite slabs based on his own skills and experience. This qualitative and non-repetitive inspection often fails to comply with customer specifications. As a consequence complaints and legal claims may arise. In order to avoid these issues, granite industry is highly concerned with the development of an automated computer vision system for comparing and searching granite slabs in a quantitative, reliable and reproducible manner, on the basis of a criterion of visual similarity.

All these problems belong to the area of computer vision, which can be defined as the branch of artificial intelligence and image processing concerned with computer processing of images from the real world. For a comprehensive review on this subject, interested readers are referred to the book by Sonka et al. [1].

Three most prominent branches of image analysis have emerged so far, namely: classification (IC), segmentation (IS) and content-based retrieval (CBIR). A wide variety of applications has also been reported: classification has been applied to automatic characterization of minerals contained in coal [2]; segmentation has been used in industrial applications such as the detection of mature fruit in coffee harvesting [3] or faulty pieces in the granite industry [4], and content-based image retrieval has been employed for quality control purposes in the production of semiconductors [5], paper [6] and many other products.

In this paper we study the feasibility of developing a search engine capable of retrieving images from a granite image database based on a criterion of visual similarity. The industrial interest of the proposed CBIR system is two-fold. On the one hand such a search engine would provide a fast, easy and efficient means to catalogue granite images; on the other hand, it would make it possible to sell granite products through the Internet.

Based on the above summarized motivations, we are concerned, in this paper, with the problem of evaluating which set of colour and/or texture features would yield the best performance in terms of retrieval accuracy. We also consider the effects of combining colour and texture features through suitable fusion schemes. The experimental results show that the last approach outperforms the methods based on colour or texture features alone.

The remainder of the paper is organized as follows: section 2 provides a general description of CBIR systems; section 3 presents the colour and texture descriptors considered in this work; section 4 describes our proposal for automated granite image retrieval together with the experimental activity; section 5 presents the results and discussion followed by the conclusions (section 6).

Image retrieval systems aim at searching digital images in large databases [7]. Two main approaches exist: those that rely on textual metadata and those based on the image content. In text-based systems, images are described through textual annotations (keywords, labels, etc.). Due to the intrinsic difficulty in converting the visual content of an image into words, there is a semantic gap

between the system and the user [8]. To overcome this issue, the concept of content-based image retrieval (CBIR) has been proposed. As stated by Datta et al. [9], CBIR is “any technology that in principle helps to organize digital picture archives by their visual content”. In a CBIR system, the visual content of an image is represented through a suitable feature vector. Such features, which are extracted using image processing techniques, are not affected by the intrinsic subjectivity of textual descriptors [10]. The most common implementation of CBIR is query by image: the user submits an example, and the system searches for the most similar images in the database. For CBIR to provide a ranked set of the most relevant images, we first need to extract suitable features from the images, and then we have to define a proper distance in the selected feature space that measures the similarity between the query image and the other images in the database.

Most commonly the image features used in CBIR applications are colour, texture, shape and spatial layout [11]. CBIR systems often use more than one type of features [12]. This is the case of commercial systems such as QBIC by IBM, NeTra, IRIS, CORE and VisualSEEK. Two out of the set of four features mentioned above, namely shape and spatial layout, are not so relevant in granite retrieval, since it is widely accepted that most of the visual content of a granite image can be described in terms of colour and texture [4]. Based on this assumption we have only considered colour and texture features in this paper.

Colour and texture are two different but complementary visual stimuli. Colour is related to the spectral content of the image, whereas texture refers to the variation of the intensity in a neighbourhood of pixels. As used herein, the term “spectral content” refers to the energy distribution in the visible region of the electromagnetic spectrum. In this section we describe the main aspects of both types of stimuli, and present the colour and texture features that we considered for the implementation of the CBIR system. Comparative results are presented in section 5.

3.1 Colour features
Colour has been extensively used in image processing [13]. Most commonly the colour content of an image is conveyed by three-channel digital images, such as the RGB images used in our experiment. Colour-based features are invariant to translation and/or rotation of the pixels in an image, and only slightly dependent on the viewing angle. However, their effectiveness drops drastically in case of varying illumination. Colour features can be grouped into two main categories, namely histogram-based methods and colour statistics.

Histogram-based methods rely on the probability distribution of the colours of a predefined palette. This approach was originally introduced by Swain and Ballard [14], who proposed the joint 3D colour histogram. Marginal histograms have also been used as colour features: in this case the probability distribution of colours is considered separately for each channel, irrespective of channel interactions. In [15] Pietikäinen et al. compared the performance of the joint 3D colour histogram with three marginal histograms in the classification of printed colour paper.

The term colour statistics refers to global statistical parameters (such as mean value, standard deviation, median, centiles, etc.) which are computed directly from the colour images. In this framework Kukkonen et al. [16], proposed the use of the mean values of the R, G, and B colour channels to classify ceramic tiles. Niskanen et al. applied colour centiles (i.e. intensity values of each colour channel below which a certain percent of pixels falls) to wood inspection [17]. Other features of this group are the soft colour texture descriptors reported in [18,19]. The chromaticity moments proposed by Paschos [20] also fall in this group. This approach consists in calculating a set of moments (up to 10) from the 2D chromaticity histogram. In the original formulation the chromaticity moments are not invariant to image dimension. This makes the method inapplicable in CBIR. In order to cope with this problem we introduced in our experiments a normalized version of the method. Last, López et al. [21] proposed various combinations of statistical descriptors computed from the RGB and the CIELAB spaces. The entire set includes mean, standard deviation and average deviation of each channel and two blocks of marginal histogram moments from the 2nd to the 5th degree and from the 6th to the 10th degree respectively. The authors achieved high classification accuracy in surface grading of decorated ceramic tiles with this approach.

The main advantage of these methods is that the dimension of the feature vector is usually low, unlike histogram-based methods. As a consequence, the computational overhead is reduced, which makes these techniques particularly well suited for real-time applications.

3.2 Texture features
Texture analysis has been traditionally performed by extracting features from gray-scale images, and hence disregarding colour information [22]. Many approaches to texture analysis have been proposed in literature. In the following paragraphs we briefly describe the methods used in this paper.

The Coordinated Clusters Representation (CCR) is a method based on global binarization of the input image. In order to preserve textural information, care must be taken in the computation of an adequate threshold. This model represents textures through the probability of occurrence of the 512 elementary binary patterns (texels) that can be defined in a 3×3 binary window [23].

The Local Binary Patterns (LBP) are closely related to the CCR [24]. The main difference with respect to the CCR texture model is that binarization is local, since in the LBP the gray level of the central pixel in a 3×3 neighbourhood is used as local threshold. In this method there are 256 elementary binary patterns. In addition we also considered an improved version of the LBP (ILBP) that takes the mean gray-scale value of the neighbourhood as threshold [25].

Gabor filters have been used extensively in texture analysis. They have important relations with the vision system of mammals. The design of a Gabor filter bank involves the selection of a proper set of values for central frequency, orientation and smoothing parameters [26]. The possible combinations of the various parameters provide different tesselations of the frequency domain and determine how the filter bank performs a localized and oriented frequency analysis of a two-dimensional signal. Feature extraction based on Gabor filters is accomplished by computing the mean and the standard deviation of the transformed images corresponding to each filter of the bank.

The Gray Level Co-occurrence Matrices (GLCM) introduced by Haralick [27], are based on the joint conditional probability that a pair of pixels separated by a given displacement vector have a certain gray-scale value. For each displacement vector the corresponding co-occurrence matrix is computed. Subsequently, suitable statistical descriptors (such as homogeneity, contrast, correlation, variance, entropy, energy, etc.) are extracted from each co-occurrence matrix.

The ranklets are a non-parametric texture analysis method. They are defined for gray-scale images by splitting a variable-sized square cluster of pixels into two subsets with the same cardinality, this pair of subsets being defined differently for the horizontal, vertical and diagonal directions, and by counting how many pixels of one subset have a higher gray-scale value than all the pixels of the other subset [28].

Finally, it is worth mentioning that texture features cannot be considered, in general, invariant to changes in viewpoint, scale and rotation angle. On the contrary some of them (such as LBP and ranklets) are by definition invariant to any monotonic change in the illumination intensity of the input image.

3.3 Combining colour and texture features
Since colour and texture contribute to determine the visual appearance of a material in a different way, it makes sense trying to join them together. There is a wide variety of ways to combine different sets of features into a hybrid model: concatenation [17,29,30], joint distribution [31] and fusion of classifiers [32, 33]. Herein we adopted the latter approach. We used two well established techniques: majority voting and weighted majority voting [32].

In order to evaluate the effectiveness of colour and texture features, both separately and jointly, we developed an experimental CBIR system (figure 1) for granite images. As a first step we created a database of 24 images which were recorded in controlled laboratory conditions. This means that illumination, viewpoint, zoom and distance between camera and tile are maintained constant during the image acquisition process. The images belong to the six following granite classes: Azul Platino, Bianco Cristal, Giallo Napoletano, Giallo Ornamentale, Giallo Santa Cecilia and Rosa Porriño. Four images of different tiles represent each class. The tiles of the same class have very similar visual properties. In addition the database contains 282 images of granite tiles taken from the Internet, which correspond to 30 commercial granite classes, including the six classes mentioned above.

Figure 1. Flowchart diagram of the proposed CBIR system

As a second step we implemented different search engines based on colour and texture features separately, and on various combinations of colour and texture features through fusion of classifiers.

The CBIR task consisted in submitting a query image to the system, and retrieving from the database a set of three images sorted in descending order of similarity. We picked one query image from each of the six groups of images acquired in the laboratory. The “ground truth” of the experiments has been established a priori by a group of human subjects, who sorted the images of each group in descending order of similarity with respect to the query images. Two different distance measures have been considered: the Manhattan (L1) distance and the Euclidean (L2) distance.

In order to estimate the effectiveness of each method we used two figures of merit, namely: precision (P) and average rank (A), which may be expressed as:

where N_c is the number of relevant images (i.e. retrieved images which are in the ground truth), N_g is the number of images which form the ground truth (herein N_g = 18) and r_i is the rank of the i-th relevant image. To compute this index, the retrieved images are sorted by their distances to the query image in ascending order. Smaller distances correspond to higher ranks and vice versa. The rank represents a relative measure of the perceptual similarity between query and retrieved images. The average rank allows one to better assess retrieval performance of different features that yield the same precision values.

The results of the experiments are summarized in tables 1 and 2. Figure 2 shows the ground truth images used in the experiment and the retrieval results of the fusion of three different sets of features. If weconsider each feature space separately (table 1) we can appreciate, on average, the better performance of colour features over gray-scale texture features. However it is fair to recognize that the good performance of the colour features is due, to a great extent, to the fact that the best matching images were acquired in the same controlled lab environment than the query images. It is well-known that the performance of a CBIR system is strongly dependent on the image acquisition conditions since noise factors such as variable illumination, usually degrade retrieval accuracy. Nevertheless the requisite of invariable viewing and illumination conditions can be easily complied with in a granite processing factory through the use of affordable machine vision equipment. Another important outcome is that fusing different features markedly improves the retrieval accuracy. We tested different combination strategies (table 2), namely: fusion of all the colour features (row 1); fusion of all the texture features (row 2); fusion of all the features (row 3); fusion of the best texture features (row 4); fusion of the best colour features with the best texture features (rows 5 to 8) and fusion of the best colour features (row 9). It turns out that fusing all the individual colour and gray-scale texture features gives the best performance (100% precision). The fusion of the two best colour features also gives a precision of 18 out of 18. In both cases we achieved a high success rate, irrespective of either the chosen distance or the voting system. This suggests that fusing colour and texture features is a robust approach to granite image retrieval. The results show that the weighted voting scheme slightly outperforms the non-weighted one. From table 2 we also note that the effect of the considered distances on the performance is very similar, and therefore we cannot draw significant conclusions about the influence of the similarity measure on the retrieval accuracy.

In this paper we presented an automatic search engine to perform queries in a database of granite images based on the visual content. Our objective was to determine the feature set which gives the highest retrieval accuracy in this domain of application, assuming that colour and texture are the two most significant features in the visual appearance of granite. An extensive experimental campaign has been carried out to compare several fusion schemes of colour and gray-scale texture features. The results show that the retrieval accuracy can be as high as 100 % when colour and texture features are used jointly. As one could expect, fusion of colour and texture improves the results obtained by colour or texture alone. Obviously, when comparing the results, computational complexity issues should also be kept in mind. However, the main goal of our paper was to assess the retrieval accuracy attainable through different colour and texture features rather than to evaluate practical aspects of the implementation. This is the reason why the software we developed for this study was made with an emphasis on short development time and high flexibility, irrespective of the computing speed.

As a final conclusion, we could say that the introduction of CBIR systems in the natural stone industry would provide an easier, faster and more efficient way to catalogue granite images and/or to sell granite products using the Internet.

[1] SONKA, M., HLAVAC, V., BOYLE, R. Image Processing, Analysis, and Machine Vision, 3rd edition. Thomson Engineering, 2007.         [ Links ]
[2] LÓPEZ, J., BRANCH, J. W. Comparación de modelos de clasificación automática de patrones texturales de minerales presentes en los carbones colombianos. Dyna, 146, 115–124, 2005.         [ Links ]
[3] MONTES, N., OSORIO, G, PRIETO, F., ANGULO, F. La visión artificial aplicada al proceso de producción del café. Dyna, 133, 41–49, 2001.         [ Links ]
[4] SONG, K. Y., KITTLER, J. PETROU, M. Defect detection in random colour textures. Image and Vision Computing, 14, 667–683, 1996.         [ Links ]
[5] TOBIN, K. W., KARNOWSKI, T. P., FERRELL, R. K. Image retrieval in the industrial environment. Proceedings of SPIE, 3652, 184–192, 1999.         [ Links ]
[6] MALDONADO, J. O. Estudio de métodos de indexación y recuperación en bases de datos de imágenes. PhD Thesis, Universidad del País Vasco, 2008.         [ Links ]
[7] LIU, Y., ZHANG, D., LU, G. MA, W. A survey of content image retrieval with high-level semantics. Pattern Recognition, 40, 262–282, 2007.         [ Links ]
[8] SMEULDERS, A. W. M., WORRING, M., SANTINI, S., GUPTA, A., JAIN, R. Content-Based Image Retrieval: at the End of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (12), 1349–1380, 2000.         [ Links ]
[9] DATTA, R., JOSHI, D., LI, J., WANG, J. Z. Image Retrieval: Ideas, Influences and Trends of the New Age. ACM Computing Surveys, 40 (2), Article 5, 2008.         [ Links ]
[10] BIZETTO, O. A., DA SILVA, R. Color descriptors for Web image retrieval: a comparative study. Proceedings of the XXI Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'08), 163–170, 2008.         [ Links ]
[11] HUNG, K.-H., AW-YONG, M. A Content-based Image Retrieval System Integrating Color, Shape and Spatial Analysis. Proceedings of the 2000 IEEE International Conference on Systems, Man, and Cybernetics, 2, 1484–1488, 2000.         [ Links ]
[12] CHORAS, R. S., ANDRYSIAK, T., CHORAS, M. Integrated color, texture and shape information for content-based image retrieval. Pattern Analysis and Applications, 10, 333–343, 2007.         [ Links ]
[13] SCHETTINI, R., CIOCCA, G., ZUFFI, S. A survey of methods for colour indexing and retrieval in image databases. In Color Imaging Science: Exploiting Digital Media, John Wiley 2001.         [ Links ]
[14] SWAIN, M. J., BALLARD, D. H. Color indexing. International Journal of Computer Vision, 7, 11–32, 1991.         [ Links ]
[15] PIETIKÄINEN, M., NIEMINEN, S., MARSZALEC, E., OJALA, T. Accurate color discrimination with classification based on features distributions. Proceedings of the 13th International Conference on Pattern Recognition, Vienna ( Austria ), 3, 833–838, 1996.         [ Links ]
[16] KUKKONEN, S., KÄLVIÄINEN, H., PARKKINEN, J. Color features for quality control in ceramic tile industry. Optical Engineering, 40, 170–177, 2001.         [ Links ]
[17] NISKANEN, M., SILVÉN, O., KAUPPINEN, H. Color and texture based wood inspection with non-supervised clustering. Proceedings of the 12th Scandivanian Conference on Image Analysis, Bergen ( Norway ), 336–342, 2001.         [ Links ]
[18] PRATS-MONTALBÁN, J. M., LÓPEZ, F., VALIENTE, J. M., FERRER, A. Multivariate statistical projection methods to perform robust feature extraction and classification in surface grading. Journal of Electronic Imaging, 17:031106–1–10, 2008.         [ Links ]
[19] LÓPEZ, F., VALIENTE, J. M., MONTALBÁN, J. M., FERRER, A. Performance evaluation of soft color texture descriptors for surface grading using experimental design and logistic regression. Pattern Recognition, 41, 1744–1755, 2008.         [ Links ]
[20] PASCHOS, G. Fast color texture recognition using chromaticity moments. Pattern Recognition Letters, 21, 837–841, 2000.         [ Links ]
[21] LÓPEZ, F., VALIENTE, J. M., BALDRICH, R., VANRELL, M. Fast surface grading using color statistics in the CIE Lab space. Lecture Notes in Computer Science, 3773, 13–23, 2005.         [ Links ]
[22] PETROU, M., GARCÍA-SEVILLA, P. Image Processing. Dealing with Texture. Wiley Interscience, 2006.         [ Links ]
[23] KURMYSHEV, E. V., SÁNCHEZ, R. E. Comparative experiment with colour texture classifier using the CCR feature space. Pattern Recognition Letters, 26 (9), 1346–1353, 2005.         [ Links ]
[24] MÄENPÄÄ, T. The local binary pattern approach to texture analysis – extensions and applications. PhD Thesis, University of Oulu, 2003.         [ Links ]
[25] JIN, H., LIU, Q., TONG, X. Face Detection Using Improved LBP Under Bayesian Framework. Proceedings of the 3rd International Conference on Image and Graphics, 306–309, 2004.         [ Links ]
[26] MANJUNATH, B. S., MA, W. Y. Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18 (8), 837–842, 1996.         [ Links ]
[27] HARALICK, R. M., SHANMUGAN, K., DINSTEIN, I. Textural features for image classification. IEEE Transactions on Systems, Man and Cybernetics SMC-3(6), 610–621, 1973.         [ Links ]
[28] BIANCONI, F., FERNÁNDEZ, A., GONZÁLEZ, E., ARMESTO, J. Robust colour texture features based on ranklets and discrete Fourier transform. Journal of Electronic Imaging, 18, 043012–1–8, 2009.         [ Links ]
[29] DRIMBAREAN, A., WHELAN, P. F. Experiments in colour texture analysis. Pattern Recognition Letters, 22, 1161–1167, 2001.         [ Links ]
[30] MONADJEMI, A., THOMAS, B., MIRMEHDI, M. Speed v. accuracy for high resolution colour texture classification, Proceedings of the 13th British Machine Vision Conference, Cardiff, (UK), 143–152, 2002.         [ Links ]
[31] CHATZICHRISTOFIS, S., BOUTALIS Y. FCTH: Fuzzy color and texture histogram – a low level feature for accurate image retrieval, Proceedings of the 9th International Workshop on In Image Analysis for Multimedia Interactive Services (WIAMIS '08), 191–196, 2008.         [ Links ]
[32] KUNCHEVA, L. I. Combining Pattern Classifiers. Methods and Algorithms. Wiley Interscience, 2004.         [ Links ]
[33] LEPISTÖ, L., KUNTTU, I., VISA, A. Classification of natural rock images using classifier combinations, Optical Engineering, 45: 097201–1–7, 2006.         [ Links ]