ARTÍCULO ORIGINAL

Detection and Classification of Non-Proliferative Diabetic Retinopathy using a Back-Propagation Neural Network

Detección y Clasificación de Retinopatía Diabética No Proliferativa usando una Red Neuronal de Retropropagación

Jesús Salvador Velázquez-González¹, Alberto Jorge Rosales-Silva^1*, Francisco Javier Gallegos-Funes¹, Guadalupe de Jesús Guzmán-Bárcenas²

¹ Escuela Superior de Ingeniería Mecánica y Eléctrica Unidad Zacatenco, Instituto Politécnico Nacional. Av. Luis Enrique Erro s/n, Unidad Profesional Adolfo López Mateos, Zacatenco, Delegación Gustavo A. Madero. C.P. 07738. México D.F., México.

² Centro Interdisciplinario de Ciencias de la Salud Unidad Santo Tomas, Instituto Politécnico Nacional. Av. de los Maestros s/n casi esquina, Calzada de los Gallos Col., Santo Tomás, Delegación Miguel Hidalgo. C.P. 11340. México D.F., México.

* Corresponding author: Alberto Jorge Rosales Silva, e-mail: arosaless@ipn.mx

(Received February 12, 2014; accepted October 20, 2014)

Abstract

One of the most serious complications of type 2 Diabetes Mellitus (DM) is the Diabetic Retinopathy (DR). DR is a silent disease and is only recognized when the changes on the retina have progressed to a level at which treatment turns complicate, so an early diagnosis and referral to an ophthalmologist or optometrist for the management of this disease can prevent 98% of severe visual loss. The aim of this work is to automatically identify Non Diabetic Retinopathy (NDR), and Background Retinopathy using fundus images. Our results show a classification accuracy of 92%, with sensitivity and specifity of 95%.

Keywords: Diabetic Retinopathy, early diagnosis, automatically identification, fundus images

Resumen

Una de las complicaciones más graves de la Diabetes Mellitus tipo 2 es la Retinopatía Diabética (RD). La RD es una enfermedad silenciosa y solo es reconocida por el portador cuándo los cambios en la retina han progresado a un nivel en el cual el tratamiento se complica, por lo que el diagnóstico oportuno y la remisión al oftalmólogo u optometrista para el manejo de esta enfermedad pueden prevenir el 98% de la pérdida visual grave. El objetivo de este trabajo es identificar de manera automática la No Retinopatía Diabética (NRD) y la Retinopatía de Fondo, utilizando imágenes del fondo de ojo. Nuestros resultados muestran una efectividad del 92%, con una sensitividad y especificidad del 95%.

Palabras clave: Retinopatía Diabética, diagnóstico temprano, identificación automática, imágenes de fondo

Introduction

The DM is defined as a set of chronic and degenerative disorders that involves alterations in the metabolism of carbohydrates, lipids, and proteins, as a consequence of a decreasing in the production of the hormone insulin for the β cells from the pancreas, and a resistance to the hormone's action in the different tissues [1]. One of the most serious complications of the DM is the DR [2], which is the main cause of worldwide blindness in the economically active population, because it affects people between 20 to 74 years old [3, 4]. Two types of clinical DR exist: Non-Proliferative Diabetic Retinopathy (NPDR), also called Background Retinopathy and Proliferative Diabetic Retinopathy (PDR), as shown in figure 1. Unfortunately, the DR is commonly detected only in advanced stages (PDR), with unfavorable forecast even with the right treatment. So, a timely diagnosis in the first stages of the NPDR namely: mild (level 1), moderate (level 2) and severe (level 3), and besides the proper canalization with the visual health specialists for the opportune treatment can control the severe visual loss in ranges of 98% of the cases [5]. The diagnosis emitted by the visual health specialist, based on the observation of the retinal damage or retinal lesions in fundus images (Figure 2) has an approximately precision of 90% [6]. The presence of those lesions in various degrees determines whether the NPDR is 'mild', 'moderate' and 'severe'.

The first injuries clinically detectable of the DR are the microaneurysms [8], and they are focal dilations of the walls of the retinal capillaries, which appear as red small points [9] with diameters between 10 and 100 micron. The hard exudates are deposits of lipids, and they are the result of leakage of blood fluids of the microaneurysms, and the capillary that surrounds the macular area. If the lipid is extended to the macular zone, the vision may be seriously compromised [10]. The hard exudates have a yellowish appearance, they don't have form or size defined and they are located in every part of the retina [8, 10].

A Computer-Aided Diagnosis System (CADx System) is defined as the combination of digital image processing techniques and intelligent methods such as Artificial Neural Networks (ANN) and Fuzzy Logic (FL), employed to improve the diagnosis made by medical interpretation, and providing a more efficient result diagnosis [11]. With the proposed CADx System, there exist the opportunity to analysis the digital fundus images, and assists in the visualization and quantification of the anatomic structure and the presence of anatomic-pathologist alterations such as blood vessel's segmentation, microaneurysms and hard exudates.

There have been some research investigations to detect and classify the NDR and the stages of NPDR. In [12] is proposed the use of a Back-Propagation Neural Network (BPNN) to classify NPDR. The network reached a sensitivity of 88.4% and a specificity of 83.5% for the DR detection (where sensibility is the probability to classify correctly a sick individual, and the specificity is used to diagnose the health of the individuals). The image pre-processing methods used for each of the detected features (blood vessels, exudates and haemorrhages) are pre-processing filters: Median smoothing was used for the detection of exudates and haemorrhages. This was a 9 point neighbourhood median filter. For each pixel in this filter, the nearest 8 neighbouring pixels were compared and the median value substituted into the centre pixel. Maximum median filtering was used for exudates where the maximum neighbourhood value was substituted, and the minimum median filtering was used for haemorrhages where the minimum pixel value was used. For the detection of blood vessels, a 9 point averaging filter was first used followed by a Sobel edge detection filter. In [13] is proposed a novel computer-based image analysis method that is being developed to assist and automate the diagnosis of retinal disease. They use feature description models and perceptual organization for low level analysis and spatial relationships and clinical metadata to extract semantic information in a higher level analysis. The sensitivity and accuracy for NPDR ranged from 75% to 94.7%. In [14] is proposed an automated system based on ANN for eye disease classification. Abnormal fundus images from four different classes namely NPDR, Central retinal vein occlusion (CRVO), Choroidal neovascularization membrane (CNVM) and central serous retinopathy (CSR) are used in this work. A suitable feature set is extracted from the pre-processed images and fed to the classifier. Classification of the four eye diseases is performed using the supervised ANN namely BPNN; the network achieved a sensitivity of 84% and a specificity of 97.3%. The results are compared with the statistical classifier namely minimum distance classifier to justify the superior nature of the ANN based classification. In [15] is proposed an automated DR diagnosis system used to detect various lesions of the retina, i.e. exudates, microaneurysms and hemorrhages, using mathematical morphological operators and filters, genetic algorithm and Fuzzy clustering. They did not present the sensitivity and specificity of their work, but they obtained an interesting conclusion; There are certain features present in the normal physiology of the retina which have to be differentiated from the abnormal pathology, e.g. optic disc has the same pixel brightness as the exudates and thus has to be localized before establishing the presence of the exudates. Similarly, the blood vessel and fovea region have to be subtracted from the retinal image before diagnosing microaneurysms and hemorrhages, which are one of the principles of our work. In [16] is proposed an automated detection of DR for early diagnosis using Feature Extraction and Support Vector Machine, with an average accuracy of 93%. The SVM classifier was trained through supervised learning for the features extracted to classify the retinal images. In [17] is proposed a method for the Retinal image analysis through efficient detection of exudates and recognizes the retina to be normal or abnormal. 110 images were trained and tested in order to extract the exudates and blood vessels. In this system, they used the Probabilistic Neural Network (PNN) for training and testing the pre-processed images. There is 98% accuracy in the detection of the exudates in the retina. In [18] is studied the effectiveness of two non-stereoscopic digital 50 degree photograph of each eye (one centered on the fovea with the nasal edge of the optic disc at the edge of the photograph and a nasal field with one disc diameter at the temporal edge of the optic disc) in the grading of DR, in comparison to 35-mm color slides. Two-field digital non-stereoscopic retinal photographs and two-field 35-mm retinal photographs were made at the same time from patients visiting a DR outpatient clinic. The digital images were stored integrally (TIFF-file) and in a compressed way (JPEG-file). Two ophthalmologists assessed the photographs in a masked fashion. The sensitivity for the detection of vision-threatening DR using the JPEG-stored images was 0.72-0.74, and the specificity was 0.93-0.98. The sensitivity for vision-threatening retinopathy detection using the integrally stored images was 0.86-0.92, the specificity was 0.93. They concluded that, the compression of the digital images seems to have some adverse effect on the detection of DR.

Material and Proposal Methodology

The data set was downloaded from TECHNO-VISION [19] and consists in 216 RGB color retinal images, (54 images without DR or normal, and 54 for each stage of NPDR). The images were taken by TOPCON TRC NW6 fundus camera interfaced to a computer, stored in 24-bit TIFF format with an images size of 3504 x 2336 pixels, 2304 x 1536 pixels and 1440 x 960 pixels. For this work, 143 samples were used for training and the remaining 73 samples were used for testing the ANN.

Preprocessing

Segmentation

Next step is to segment the image obtained in homogeneous regions to an automatic analysis and recognition. The general algorithms used in the development of the proposal CADx System are well-known in the literature [21-24]. Binarization: defined by, where is a threshold established by Otsu's algorithm that changes [23]. Edge detection [20, 24], to make this operation, was employed Canny's algorithm [25]. Mathematical morphology: were used Dilation and Erosion [20]. Equations applied to closure is A●B = (AB)B and to opening is: A○B = (AB)B.

Features extraction

Consist in extract relevant information of the features and regions of interest (ROIs) from the retinal images, which will be part of the input layer of the ANN (see Decision making). In the proposed CADx System the ROIs are Blood vessels (BV), microaneurysms (µAns) and hard exudates (HE), while the texture (homogeneity and entropy) is a feature of interest in the fundus image. A brief description of those extractions is given below.

Border Formation

Two methods were implemented as shown in figure 3. The border contains noise if the output's pixel is different from the reference binary pixel ''1'' (white). When the both pixels are binary 1, the resulting image is the circular border.

Blood Vessels (BV)

Figure 4 shows the block diagram of the developed algorithm for the blood vessels segmentation in a RGB color fundus image. The Logical AND operation, achieved in the noisy and optical disk images, is applied to obtain the final segmentation of the BV. The first one is the image of the BV with noise presence (see Figure 5i) and the second one with the BV and the Optical Disk (OD) (see Figure 6e). The OD is represented as a black color circle (Figure 5d, 5e and 5f); the anterior is achieved by finding the maximum brightness value for every one of the 720 columns of the image. The elimination of the OD (Figure 5g) is obtained subtracting the equalized image (Figure 5e) from the resulting image in the morphological opening (Figure 5f), after this procedure, the image is binarized (Figure 5h) and is realized a new morphological opening procedure to eliminate the noise in the image (Figure 5i).

The AND logic is applied to mark out the similar pixels of the two images. The output pixel is registered as binary 1 (white) when the both pixels of the images are binary 1 (white). The resulted image is a higher quality BV image (Figure 6f).

Microaneurysms

Figure 7 shows the microaneurysms (µAns) segmentation with noise presence. The grayscale image was used to detect the circular border and the OD mask, the green component of a color image was used in the rest of the algorithm. The procedure is as follows: the morphological opening of the BV elimination. The first step in the elimination process of the BVs is achieved realizing a morphological opening at the image, computing the second contrast enhancement and then applying a Logical AND operation using the previous image and the image without HE. To eliminate the HE, is necessary to apply a Logical AND operation using the image binarized after the first contrast enhancement and the resulting image of the µAns segmentation with noise presence. To achieve this, we execute a Logical AND operation using the resulting images of the edge detection after the first contrast enhancement and the morphological opening of the edge detection before the contrast enhancement. The second one is the OD mask, and the third one is the morphological opening of the edge detection without any contrast enhancement.

Figure 8 shows the microaneurysms (µAns) segmentation, where Fig. 8a) illustrates the binarized image, the Fig. 8b) indicates hard exudates suppression, finally in the Fig. 8f) is present the final segmentation of the microaneurysms, isolating efficiently the blood vessels.

Hard Exudates

The block diagram of figure 9 shows the process to obtain the hard exudates.

We implement the contrast enhancement applied to the image improvement before using the Canny's algorithm to detect the outlines of the image. After, we implemented the morphological gradient to obtain the circular border of the fundus image. The morphological closing was the method used in the BV suppression, which consists of the dilation followed by the erosion. The function of dilation expands the area of the hard exudates, while the erosion function removes the BV as shown in Fig. 10d).

The next step consists in detecting the location of the hard exudates, and eliminates all the objects foreign to them. To eliminate the OD, the image is binarized without the BV and the obtained images are subtracted from the OD mask. To eliminate the circular border, the image is subtracted without OD, with the image of the circular edge in a fundus image, and the result of these differences is applied to the method of morphological closing, afterward is binarized and inverted the values. Finally, the logical operation AND is implemented between the resulting image obtained from the morphological closing and the binarized one. Figure 11 shows the results obtained in every one of the segmentation processes of the hard exudates.

Area of segmented features

Texture analysis

The conventional systems of texture analysis can be grouped in three categories: structural, statistics and spectral [3]. We propose the use of statistical algorithms based on the relationship between intensity pixel values [26]; the measures include the homogeneity and the entropy on the Gray Level Co-occurrence Matrix (GLCM) [22]. Figure 12 presents the block diagram of the texture analysis proposed.

The homogeneity is the measurement of the closeness of the distribution of elements in the Grey Level co-occurrence Matrix (GLCM) to the GLCM diagonal, and returns a value between 0 and 1. The homogeneity can be mathematically written as , where describes the repetition frequency of a and b pixels in the window, separated a distance d in the direction φ [20]. On the other hand, the entropy is the statistical measure of the randomness of the grayscale image's texture and it is defined as . Here, the adaptive histogram equalization is applied twice to enhance the contrast and texture of the green channel of the fundus image.

Decision making

The diagnosis of the NDR and NPDR stages are achieved through the analysis of fundus images by means of the BPNN shown in figure 13 [10]. The input layer is composed of five neurons, which correspond to the number of characteristics employed for the detection and classification of the NDR and NPDR stages, which are blood vessels, microaneurysms, hard exudates, homogeneity and entropy. The hidden layer is formed by two layers with ten neurons each, the number of neurons is determined using the theorem of Kolmogorov [23]. The output's layer will classify four classes as NDR (Normal), Light NPDR, Moderate NPDR and Severe NPDR.

The network was trained using a supervised learning method with a given set of training data of 143 fundus images and then tested with 73 samples. During the training phase, each output of the BPNN is a value in the range [0,1], whereas the 'desired' output value is either [0,1]. The mean square error of the BPNN was 0.001 and it was obtained after 290 iterations (Figure 14). The convergence value indicates that the maximum error accepted for every sample must be less than 0.1%.

Experimental Results

The features such as blood vessels, microaneurysms, hard exudates areas, homogeneity and entropy values were extracted using the above algorithms. The parameter values obtained are shown in table 1.

The p-value can be obtained using ANOVA (analysis of variance between groups) test [27]. The result of the ANOVA test for the perimeter and area of different types of images is shown in Table 2. It can be seen from Table 2 that our features are clinically significant (p<0.01). Figure 15 shows the final segmentation of the blood vessels, microaneurysms and hard exudates using four fundus images in RGB color space in TIFF format; one of this fundus images does not have DR and the remaining three present some stage of the NPDR. These results were obtained after analyzing the 73 fundus images, where 19 are without DR and 54 possess NPDR.

By other hand, these results can be presented in a quantitative way through quality indexes employed in the evaluation of the medical diagnosis systems, which are the sensibility and the specificity [13]. The Sensibility [28], mathematically is expressed by (Eq. 2):

The Specificity [28], mathematically is expressed by (Eq. 3):

Table 2 shows the relationship between the results of diagnostic tests and the presence or absence of an illness. The total number of samples employed was 73 images.

Sensibility, specificity and error tests were conducted to avoid the BPNN (see Table 3).

Result Discussions

From the segmented images, the area of blood vessels, microaneurysms and hard exudates are calculated by finding the total number of blood vessels, microaneurysms and hard exudates respectively. While, the set of features which provides more meaningful information for classification are extracted from the selected cluster using GLCM. The features extracted from the selected clusters are Homogeneity and Entropy, where: Homogeneity is related to the closeness of the distribution of elements in the GLCM to the GLCM diagonal and range = [0 1] is measured by homogeneity. Homogeneity is 1 for a diagonal GLCM. So, in a homogeneous image, there are very few dominant gray-tone transitions; hence this image will have fewer entries of large magnitude. So, the energy of an image is high when the image is homogeneous. Entropy is the randomness of a gray level distribution. The entropy is high if the gray levels are distributed randomly throughout the image.

Thus, the total number of features used in this work is five from two different categories. These features are found to be more suitable for medial image processing. In order to know the behavior of the BPNN in every stage of the NPDR, the error was computed in percentages. Table 4 presents the error percentages corresponding to every one of the NPDR stages.

An error of 0% was obtained in the light NPDR, which indicates that this CADx System is capable of classifying this stage. Also, an error of 5% was obtained in the moderate NPDR, while that in the classification of the severe NPDR was of 3%. These results show that in some cases the CADx System was unable to classify accurately the stage of the patient really sick; this is due to the similarities that exist between the characteristics that define both stages.

The CADx System shown in this work is based on basic methods of image processing, which allows to detect and diagnose, in an automatic and rapid manner (less than 1 minute), the NDR and the NPDR stages, with good results and low computational cost against the other methods used. Table 5 shows the comparative results between the proposed algorithm and others found in the literature review.

Conclusions

Diabetic Retinopathy is a condition where the retina is damaged due to fluid leaking from the blood vessels into the retina. In extreme cases, the patient will become blind. Therefore, early detection of the Diabetic Retinopathy is crucial to prevent blindness. In this paper, a Computer Aided Diagnosis System was developed to analyze digital RGB color fundus images for three features of Non-Proliferative Diabetic Retinopathy: blood vessels, microaneurysms and hard exudates and two features of the image: homogeneity and entropy. The system proposed demonstrated a classification accuracy of 92%, sensitivity and specificity of 95%. These results revealed that the CADx System proposed can help the ophthalmologist to detect Diabetic Retinopathy at the early stages, or provide a second opinion on the ophthalmologist to provide a better accurate diagnosis.

Finally, the accuracy of the system could be further improved using more input features, taking more retinal images under uniform lighting conditions and applying more robust algorithms such as Fuzzy Logic, High Order Spectra, Watershed Transformation, among others.

Acknowledgments

This work is supported by the Instituto Politécnico Nacional de México (IPN) and CONACyT.

References

1. Organización Mundial de la Salud. Diabetes. Nota descriptiva N.° 312. Available on: http://www.who.int/mediacentre/factsheets/fs312/es/index.html Accessed: October 9, 2014.         [ Links ]

2. R. Frank. ''Diabetic Retinopathy''. The New England Journal of Medicine. Vol. 350. 2004. pp. 48-58.         [ Links ]

3. D. Fong, L. Aiello, T. Gardner, G. King, G. Blankenship, et al. ''Retinopathy in Diabetes''. Diabetes Care. Vol. 27. 2004. pp. 584-587.         [ Links ]

4. D. Browning. Diabetic Retinopathy: Evidence Based Management. 1^st ed. Ed. Springer. New York, USA. 2010. pp. 31-61.         [ Links ]

5. L. Verma, G. Prakash, H. Tewari. ''Diabetic Retinopathy: Time for Action. No complacency please''. Articles from Bulletin of the World Health Organization. Vol. 80. 2002. pp. 419-420.         [ Links ]

6. S. Le, E. Lee, R. Kingsley, Y. Wan, D. Russell, R. Klein, A. Warn. ''Comparison of Diagnosis of Early Retinal Lesions of Diabetic Retinopathy Between a Computer System and Human Experts''. Arch. Ophthalmol. Vol. 119. 2001. pp. 509-515.         [ Links ]

7. M. El-Bab, N. Shawky, A. Al-Sisi, M. Akhtar. ''Retinopathy and risk factors in diabetic patients from Al-Madinah Al-Munawarah in the Kingdom of Saudi Arabia''. Clinical Ophthalmology. Vol. 6. 2012. pp. 269-276.         [ Links ]

8. A. Khurana. Comprehensive Ophthalmology. 4^thed. Ed. New Age International (P) Ltd., Publishers. New Delhi, India. 2007.         [ Links ]

9. A. Fleming, S. Philip, K. Goatman, J. Olson, P. Sharp. ''Automated microaneurysm detection using local contrast normalization and local vessel detection''. IEEE Transactions in Medical Imaging. Vol. 25. 2006. pp. 1223-1232.         [ Links ]

10. T. Walter, J. Klein, P. Massin, A. Erginay. ''A contribution of image processing to the diagnosis of diabetic retinopathy - detection of exudates in color fundus images of the human retina''. IEEE Transactions on Medical Imaging. Vol. 21. 2002. pp. 1236-1243.         [ Links ]

11. M. Giger, N. Karssemeijer, S. Armato. ''Computer-Aided Diagnosis in Medical Imaging''. IEEE Trans. Med. Imag. Vol. 20. 2001. pp. 1205-1208.         [ Links ]

12. G. Gardner, D. Keating, T. Williamson, A. Elliot. ''Automatic detection of diabetic retinopathy using an artificial neural network: a screening tool''. British Journal of Ophthalmology. Vol. 80. 1996. pp. 940-944.         [ Links ]

13. E. Chaum, T. Karnowski, V. Govindasamy, M. Abdelrahman, K. Tobin. ''Automated Diagnosis of Retinopathy by content-based image retrieval''. The Journal of Retinal and Vitreous Diseases. Vol. 28. 2008. pp. 1463-1477.         [ Links ]

14. J. Anitha, D. Selvathi, D. Hemanth. Neural Computing Based Abnormality Detection in Retinal Optical Images. Proceedings of the IEEE International Advance Computing Conference (IACC). Patiala, India, 2009. pp. 630-635.         [ Links ]

15. N. Singh, R. Chandra. ''Automated Early Detection of Diabetic Retinopathy Using Image Analysis Techniques''. International Journal of Computer Applications. Vol. 8. 2010. pp. 18- 23.         [ Links ]

16. D. Selvathi, N. Prakash, N. Balagopal. ''Automated Detection of Diabetic Retinopathy for Early Diagnosis using Feature Extraction and Support Vector Machine''. International Journal of Emerging Technology and Advanced Engineering. Vol. 2. 2013. pp. 103-108.         [ Links ]

17. R. Radha, B. Lakshman. ''Retinal Image Analysis Using Morphological Process and Clustering Technique''. Signal and Image Processing: An International Journal (SIPIJ). Vol. 4. 2013. pp. 55-69.         [ Links ]

18. C. Stellingwerf, P. Hardus, J. Hooymans. ''Assessing Diabetic Retinopathy using two-field digital photography and the influence of JPEG-compression''. Documenta Ophthalmologia. Vol. 108. 2004. pp. 203-209.         [ Links ]

19. MESSIDOR, TECHNO-VISION. MESSIDOR: methods to evaluate segmentation and indexing techniques in the field of retinal ophthalmology. 2014. Available on: http://messidor.crihan.fr/index-en.php Accessed: October 9, 2014.         [ Links ]

20. R. Gonzalez, R. Woods, S. Eddins. Digital Image Processing using MATLAB. 1^st ed. Ed. Gatesmark Publishing. Knoxville, USA. 2009. pp. 334-358.         [ Links ]

21. P. Qiu. Image Processing and Jump Regression Analysis. 1^sted. Ed. John Wiley & Sons, Inc. New Jersey, USA. 2005. pp. 2-205.         [ Links ]

22. J. Nayak, P. Bhat, R. Acharya, C. Lim, M. Kagathi. ''Automated Identification of Diabetic Retinopathy Stages Using Digital Fundus Images''. J. Med. Syst. Vol. 32. 2008. pp. 107-115.         [ Links ]

23. N. Otsu. ''A Threshold Selection Method from Gray-Level Histograms''. IEEE Transactions on Systems, Man and Cybernetics. Vol. 9. 1979. pp. 62-66.         [ Links ]

24. F. Cui, L. Zou, B. Song. Edge Feature Extraction Based on Digital Image Processing Techniques. Proceedings of the IEEE Int. Conference on Automation and Logistics. Qingdao, China. 2008. pp. 2320-2324.         [ Links ]

25. J. Canny. ''A Computational Approach to Edge Detection''. IEEE Transactions on Pattern Analysis and Machine Intelligence. Vol. 8. 1986. pp. 679-698.         [ Links ]

26. L. Boroczky, P. Cremonesi, N. Scarabottolo. Texture Analysis for Image Processing on General-Purpose Parallel Machines. Proceedings of the International Symposium on Parallel Architectures, Algorithms and Networks. Budapest, Hungary. 1994. pp. 17-24.         [ Links ]

27. Centre for Innovation in Mathematics Teaching (CIMT). Analysis of variance (Anova). Available on: http://www.cimt.plymouth.ac.uk/projects/mepres/alevel/fstats_ch7.pdf, Accessed: November 9, 2014.         [ Links ]

28. A. Akobeng. ''Understanding diagnostic test 1: sensitivity, specificity and predictive values''. Foundation Acta Pædiatrica/Acta Pædiatrica. Vol. 96. 2006. pp. 338-341.         [ Links ]