Issuu

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017), pp. 69-78 http//dx.doi.org/10.14257/ijsip.2017.10.9.07

A New Algorithm for Improving Basic Model Based Foreground Detection Using Neutrosophic Similarity Score 1

Keli Hu* and En Fan Department of Computer Science and Engineering, Shaoxing University, Shaoxing 312000, Zhejiang, China *ancimoon@gmail.com Abstract Foreground detection is a task for detecting the moving objects in the scene like in video surveillance. Several basic background models are often used due to their high efficiency. However, their results are not good when there exists noisy information generated by the bad weather, camera jitter, etc. Neutrosophic set (NS) is as a new branch of philosophy dealing with the origin, nature and scope of neutralities. It has an inherent ability to handle the indeterminant information like the noise included in images and video sequences. Till now, NS has been successfully applied into many computer vision research fields, such as image segmentation and skeleton extraction. In this paper, we utilize the NS to improve the robustness of the basic background model. First, a sequence of background subtraction images is calculated. Second, such a sequence is represented in the NS domain via three membership subsets T, I, and F. Finally, a neutrosophic similarity score (NSS) is defined to highlight the moving objects on the image plane. Then the Otsu’s method is employed to determine an optimized value for extracting the area of the moving objects. Experiments are conducted on a variety of realworld video sequences, and the experimental results demonstrate that the proposed approach can extract the moving objects robustly. Keywords: foreground detection, background model, neutrosophic set, neutrosophic similarity score

1. Introduction Analysis of video sequences is an active research field. Detecting moving objects is the first step for the applications in this research area [1, 2], like video surveillance, optical motion capture. The purpose of such a task is to separate the moving objects called “foreground” from the static information called the “background”. The method of background subtraction is mainly used for foreground separation, and lots of efforts have been made in recent years. A mixed Gaussians model for moving objects detection was proposed in [3], each pixel was modeled as a mixture of Gaussians (MOG) and an online approximation was used to update the model. Due to the performance of the mixed Gaussians in background subtraction, many works were proposed to improve the original MOG [2, 4, 5]. For video surveillance, imprecision and uncertainties are always generated in the whole process of background subtraction. Thus, fuzzy concepts are introduced into the task of foreground detection [6]. A fuzzy model was formulated in [7] to deal with decision problems arising in the foreground detection process. In [8], the authors proposed a novel background modeling method for motion detection in dynamic scenes, based on Type-2 Fuzzy Gaussian Mixture Model and Markov Random Field. The algorithm performs well when there exist dynamic

Received (July 2, 2017), Review Result (August 10, 2017), Accepted (August 30, 2017)

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

backgrounds as waving trees and water rippling. A comprehensive survey on fuzzy concepts used in background subtraction steps were given in [6]. Neutrosophic set (NS) [9] is as a new branch of philosophy to deal with the origin, nature and scope of neutralities. It has an inherent ability to handle the indeterminate information like the noise included in images [10-13] and video sequences. Till now, NS has been successfully applied into many computer vision research fields, such as image segmentation [10-13] and skeleton extraction [14]. Algorithms for clustering also employ the NS theory. For instance, to improve the performance of the method of c-means clustering, indeterminate factors were considered by using a NS model [15]. For the applications of the NS theory in image segmentation domain, a specific neutrosophic image was usually computed [10-13] to conquer the noise interference. In addition, many kinds of NS criteria for handling the noise disturbance are proposed [10-13, 16]. In [16], multiple criteria were proposed for solving the thresholding problem on a static image, and a neutrosophic image was built before calculating the optimal threshold . It has been proved that such a method has greatly improved the performance of image thresholding when confronting the noise problem. However, to our own knowledge, though the NS theory has shown its advantage for solving the noise problem in applications like image segmentation and skeleton extraction, information between frames of a video sequence has not been considered in the NS domain. It i s meaningful to introduce the NS theory into the background modelling problem to handle those imprecision and uncertainties when determine the foreground pixel through the information from several consecutive frames. In this work, we propose a robust foreground detection method based on the basic background model and the NS theory. First, three kinds of NS based criteria are proposed. Then a neutrosophic background subtraction image is calculated using a weighted similarity degree measurement in NS domain [17]. Finally, the method for determining a more reliable threshold is proposed to separate the foreground from the background. The remainder of this paper is organized as follows. In Section 2, main steps of our algorithm are first given, and then the details of the proposed algorithm are illustrated in the following subsections. Experimental evaluations and discussions are presented in Section 3, and Section 4 is the conclusion.

2. Our Method The main steps of our method for foreground extraction are summarized in Figure 1. The third step plays as the core of the method. Generally speaking, the major task of the method is to build a neutrosophic image based on the basic foreground separation model, and then determine a proper threshold to separate the moving objects from the background. Details of each step are given in the following subsections.

Get a new frame

Basic model

Calculating the background subtraction image

Neutrosophic theory

Calculating the Neutrosophic background subtraction image

Otsuâ€™s method

Foreground separation

Figure 1. Main Steps of the Proposed Algorithm

2.1. Calculating Background Subtraction Image

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

The basic model for foreground separation is applied for generating the background subtraction image. Suppose It is the frame of one video sequence at time t, Bt is the background model calculated at time t, where (1) Bt  (1   ) Bt 1   I t where Bt-1 is the previous background model at time t-1, α∈(0,1), it determines the update rate of the background model. Simply, the foreground area is calculated as 1 if I t (x)  Bt (x)  T (2) Ft   else 0 where It(x) and Bt(x) is the intensity at the pixel location x, corresponding to the frame and the background model respectively. If the difference between It(x) and Bt(x) is larger than the threshold T, the pixel x is then judged as an element of the foreground objects. St = |It-Bt| is called the background subtraction image here. As seen in Equation (2), the element in St with a higher value is more likely to be judged as foreground. The performance of such a foreground extraction method greatly depends on the choice of the parameter T. In this work, instead of finding the proper thresholding value through the original background subtraction image, a neutrosophic subtraction image is calculated and employed, to deal with the challenges of sequence noise, bad weather, camera jitter, changing illumination, etc. 2.2. Neutrosophic Similarity Score (NSS) A neutrosophic set with multiple criteria can be expressed as follows: Let A = {A 1, A 2, …, A m } be a set of alternatives and C = {C1, C2, …, Cn } be a set of criteria. Then the character of the alternative A i (i = 1, 2…m) can be represented by the following information:





Ai  C j , TC j ( Ai ), IC j ( Ai ), FC j ( Ai ) C j  C i  1,2...m, j  1,2...n

(3)

where TCj(Ai), ICj(Ai), FCj(Ai)∈[0,1]. TCj(Ai) denotes the degree to which the alternative Ai satisfies the criterion Cj , ICj(Ai) indicates the indeterminacy degree to which the alternative A i satisfies or does not satisfy the criterion Cj, FCj(Ai) indicates the degree to which the alternative A i does not satisfy the criterion Cj. A method for Multicriteria decision-making based on the correlation coefficient under single-valued neutrosophic environment is proposed in [17]. The similarity degree between two elements Ai and Aj is defined as: TCk ( Ai )TCk ( Aj )  I Ck ( Ai ) I Ck ( Aj )  FCk ( Ai ) FCk ( Aj ) (4) SCk ( Ai , Aj )  TCk 2 ( Ai )  I Ck 2 ( Ai )  FCk 2 ( Ai ) TCk 2 ( Aj )  I Ck 2 ( Aj )  FCk 2 ( Aj ) Assume the ideal alternative





A*  C j ,TC j ( A* ), IC j ( A* ), FC j ( A* ) C j  C i  1,2...m, j  1,2...n

Then the similarity degree between any alternative A i and the ideal alternative A* can be calculated as TCk ( Ai )TCk ( A* )  ICk ( Ai ) I Ck ( A* )  FCk ( Ai ) FCk ( A* ) (5) SCk ( Ai , A* )  TCk 2 ( Ai )  ICk 2 ( Ai )  FCk 2 ( Ai ) TCk 2 ( A* )  I Ck 2 ( A* )  FCk 2 ( A* ) Suppose wk∈[0,1] is the weight of each criteria Ck, then the weighted correlation coefficient between an alternative A i and the ideal alternative A * is defined by n TCk ( Ai )TCk ( A* )  ICk ( Ai ) ICk ( A* )  FCk ( Ai ) FCk ( A* ) * (6) W ( Ai , A )   wk k 1 TCk 2 ( Ai )  ICk 2 ( Ai )  FCk 2 ( Ai ) TCk 2 ( A* )  ICk 2 ( A* )  FCk 2 ( A* )

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

2.3. Calculating the Neutrosophic Background Subtraction Image In this work, we try to translate the original background subtraction image St into the neutrosophic set domain. Let SNSt be the corresponding neutrosophic image. Surpose U is a universe, FP is a forground pixel set in U. A pixel in SNSt is denoted as PNS(T, I, F), T, I, F∈[0, 1] which means it is T true belonging to the FP set, I indeterminate, and F false. According to the above definition, a pixel in SNSt is interpreted as PNSt (x)  (Tt (x), I t (x), Ft (x)) Tt(x), It(x), Ft(x) represent memberships belonging to the foreground pixel set FP, indeterminate set and background pixel set, respectively. When considering the criteria, a pixel in SNSt can interpreted as PNSt (x)  (C j ,Tt (x), It (x), Ft (x)) Three criteria are proposed. At the intensity criteria Cg, the corresponding Tt(x), It(x), Ft(x) are defined as t S (x)  Smin (7) TCtg (x)  t t t Smax  Smin SD(x)  SDmin (8) I Ct g (x)  SDmax  SDmin

FCtg (x)  1  TCtg (x)

(9)

where St(x) is the intensity value of the pixel x in the original background subtraction image, Stmax and Stmin is the maximal and minimal intensity value among all of the pixels in St, respectively. SD(x) is the standard deviation of the corresponding intensity values in the past N background subtraction images, which can be represented as SV (x)=stddev(St  N 1 (x), St  N  2 (x), St (x)) (10) For Equation (8), SDmax is the maximal standard deviation value, and SDmin is the minimal one. The ideal alternative A * is set to (Cg, 1, 0, 0) under this criteria. From the Equation (5), the similarity degree between PNSt and the ideal alternative under the intensity criteria is calculated as TCg ( Ai )TCg ( A* ) * (11) SCg ( P(x), A )  TCg 2 ( Ai )  I Cg 2 ( Ai )  FCg 2 ( Ai ) Mean filtering is known as a simple but effective procedure for image smoothing and noise reduction. It is employed as the second criteria for judging the similarity degree, we call this criteria mean criteria. Under this criteria, the corresponding Tt(x), It(x), Ft(x) are defined as t w/2 w/2 Sm (x)  Smmin 1 (12) TCtm (x)  t t , Smt (x)  St ( x  m, y  n)   t Smmax  Smmin w  w m w/2 n w/2 SDm(x)  SDmmin I Ct m (x)  (13) SDmmax  SDmmin

FCtm (x)  1  TCtm (x)

(14) t

where w in Equation (12) is the width of the mean filter window, Sm max and Sm min is the maximal and minimal intensity value after smoothing, respectively. SD(x) is the corresponding standard deviation after smoothing. According to the structure of any object, pixels nearby the pixel which is considered to be an element of the foreground with a high probability are also more likely to be parts of the foreground. Due to this, regional maxima is applied as the third criteria, then the corresponding Tt(x), It(x), Ft(x) are represented as

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

TCtma (x) 

t Smat (x)  Smamin t t Smamax  Smamin

(15)

Smat (x)  max St ( x  m, y  n) | m, n  (  w / 2, w / 2) I Ct ma (x) 

SDma (x)  SDmamin SDmamax  SDmamin

(16)

FCtma (x)  1  TCtma (x)

(17)

where SDma(x) is the corresponding standard deviation based on regional max. As the ideal alternative defined for the intensity criteria, the ideal alternative at the mean criteria and the regional maxima criteria are defined as ( Cm , 1, 0, 0) and (Cma, 1, 0, 0), respectively. Thus, by applying Equation (6), the weighted similarity degree of a pixel x can be generated as W ( P(x), A* )  wg

TCg ( Ai )TCg ( A* ) TCg 2 ( Ai )  I Cg 2 ( Ai )  FCg 2 ( Ai )

 wma

 wm

TCm ( Ai )TCm ( A* ) TCm 2 ( Ai )  I Cm 2 ( Ai )  FC 2 ( Ai )

TCma ( Ai )TCma ( A* ) TCma 2 ( Ai )  I Cma 2 ( Ai )  FCma 2 ( Ai )

(18)

 wg SCg ( P(x), A* )  wm SCm ( P(x), A* )  wma SCma ( P(x), A* )

where wg+wm+wma =1. Then the weighted similarity for a pixel x is employed as the value of the corresponding pixel for the neutrosophic background subtraction image. 2.3. Extracting Foreground Objects Based on NSS Once the neutrosophic background subtraction image is constructed, the only task left is to find an optimal threshold based on such a neutrosophic image. In this work, the Otsu’s method [18] is applied for tackling this problem. For the completeness of this work, details of the procedure are presented below: (1) Selecting one value as the initial threshold TNSS. (2) Using this threshold to separate the pixels of the neutrosophic background subtraction image SNSt into two classes C1 and C2 . For example, C1 contains all the pixels who are smaller than TNSS. Then we can calculate the variance of SNSt by  = 1 p1   2 p2 where σ1 and σ2 are the variance of the similarity degrees of the pixels belonging to C1 and C2 , respectively. p 1 and p 2 are probability of the corresponding pixels. (3) Try another value as the threshold until find the minimal σ, and the corresponding value is considered as the optimal threshold. After finding the optimal threshold, the foreground can be extracted by using Equation (2).

3. Experiments We tested our algorithm on several challenging real-world video sequences. For comparison, we implemented two algorithms. Both algorithms employ the basic model introduced in section 2.1. The only difference is on the selection of the threshold T. The first one uses a fixed threshold, called FTFD for short. The other one employs the Otsu’s method to find the optimal threshold based on the original background subtraction image at each frame, we call this algorithm OTFD for short. To gauge absolute performance, we also compare our results to the Euclidean distance [19] algorithm, we call this algorithm EDFD here.

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

3.1 Setting Parameters The update rate α for our method, FTFD and OTFD are all set to 0.01. For Equation (12) and (15), the width of the filter window is set to 5. For the weight of each criteria, wg, wm, wma are set to 0.5, 03, 0.2 in Equation (18), respectively. Finally, all parameters were kept constant for all experiments. 3.2. Foreground Extraction Results Figures 2–4 show screen captures for some of the testing sequences. The second column of each figure shows the background subtraction image (BSIMG) at each time. Below is a more detailed discussion of the foreground extraction results.

Figure 2. Screenshots of Foreground Separation Results of the Wetsnow Sequence with the Challenge of Bad Weather wetsnow sequence: this sequence highlights the challenges of bad weather, slow movement and foreground disturbance. As shown in Figure 2, when the water droplets flow along the mirror of the camera, our method can successfully handle this, while the other three methods fail to filter out such a kind of foreground disturbance, see frame #235, #462. From frame #235 to frame #610, compared to our results, the separation results of other algorithms are noisy. As can be seen in

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

frame #786, there is only one person appears in the camera view and moves very slowly. The person only occupies a small area of the camera view, which leads to the failure of the OTFD. In frame #1015, before the person disappears from the view, a totally wrong threshold is employed by the OTFD to separate the foreground from the background. The EDFD also produces some noise due to the slow motion.

Figure 3. Screenshots of Foreground Separation Results of the Blizzard Sequence with the Challenge of Bad Weather blizzard sequence: this sequence presents the challenges of bad weather, similar background, tinny camera jitter and small foreground area. Figure 3 shows some screenshots of foreground separation results of this sequence. As seen in frame #878 and #886, some noisy foreground are included in the results for OTFD, due to the small camera jitter. The EDFD gives an imperfect foreground, while our method outperforms others. In frame #814, due to the similar background, only the lamps of those two cars are detected as foreground. Noise is included in the results of OTFD on account of the small foreground area. As seen in Figure 2, our method performs the best. The FTFD sometimes perform worse due to the fixed threshold. badminton sequence: challenge of camera jitter is presented in this sequence. As shown in Figure 4, our method performs the best, and the EDFD performs the

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

second best. Due to the serious camera jitter, edges jitter in each frame lead to the wrong separation of the foreground. When the camera stabilize for a while, the disturbance of the edges in each frame are successfully filtered out, see frame #1036.

Figure 4. Screenshots of Foreground Separation Results of the Badminton Sequence with the Challenge of Camera Jitter As seen in Figure 2-4, our algorithm performs well due to the usage of the three kinds of criteria in NS domain. The standard deviation employed by the calculation of the indeterminacy degree I contributes a lot for the performance of our method when handling the problem of camera jitter or noise pixels.

4. Conclusion In this paper, we propose a new algorithm for foreground separation. The background subtraction image is achieved by employing a basic background model. Then three criteria are proposed, and a neutrosophic background subtraction image is calculated by using the metric of neutrosophic similarity score. Finally, an optimal threshold can be found based on such a neutrosophic image. Experimental results have shown that our method performs well when handling challenges like bad weather and camera jitter. In future, it will be our primary mission to try to further improve the robustness of foreground detection by introducing the neutrosophic scheme into a more robust background model.

Acknowledgements This work is supported by the public welfare technology application research project of Zhejiang province under Grant No. 2016C31082.

References [1]

Bouwmans, T., 'Traditional and recent approaches in background modeling for foreground detection: An overview', Computer Science Review, vol. 11â€“12, no. 0, pp. 31-66, (2014).

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)

[2]

[3] [4]

[5] [6] [7]

[8] [9] [10] [11]

[12] [13] [14] [15] [16]

[17]

[18] [19]

Thierry, B., B. Fida El, and V. Bertrand, 'Background Modeling using Mixture of Gaussians for Foreground Detection - A Survey', Recent Patents on Computer Science, vol. 1, no. 3, pp. 219-237, (2008). Stauffer, C. and W.E.L. Grim tive background mixture models for real-time tracking'. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Collins, Colorado, pp. 22-29, (1999). Elgammal, A., R. Duraiswami, et al., 'Background and foreground modeling using nonparametric kernel density estimation for visual surveillance', Proceedings of the IEEE, vol. 90, no. 7, pp. 1151-1163, (2002). Gutchess, D., M. Trajkovic, et al. 'A background model initialization algorithm for video surveillance'. IEEE International Conference on Computer Vision. Vancouver, pp. 733-740, (2001). Bouwmans, T., 'Background subtraction for visual surveillance: A fuzzy approach', Handbook on Soft Computing for Video Surveillance, pp. 103-134, (2012). Maddalena, L. and A. Petrosino, 'A fuzzy spatial coherence-based approach to background/foreground separation for moving object detection', Neural Computing and Applications, vol. 19, no. 2, pp. 179-186, (2010). Zhao, Z., T. Bouwmans, et al., 'A fuzzy background modeling approach for motion detection in dynamic backgrounds', Multimedia and Signal Processing, pp. 177-185, (2012). Smarandache, F., A unifying field in logics. Neutrosophy: neutrosophic probability, set and logic, American Research Press, Rehoboth, (1999). Guo, Y. and A. Şengür, 'A novel image segmentation algorithm based on neutrosophic similarity clustering', Applied Soft Computing Journal, vol. 25, pp. 391-398, (2014). Anter, A.M., A.E. Hassanien, et al., 'Neutrosophic Sets and Fuzzy C-Means Clustering for Improving CT Liver Image Segmentation', Advances in Intelligent Systems and Computing, vol. 303, pp. 193-203, (2014). Karabatak, E., Y. Guo, and A. Sengur, 'Modified neutrosophic approach to color image segmentation', Journal of Electronic Imaging, vol. 22, no. 1, (2013). Zhang, M., L. Zhang, and H.D. Cheng, 'A neutrosophic approach to image segmentation based on watershed method', Signal Processing, vol. 90, no. 5, pp. 1510-1517, (2010). Guo, Y. and A. Sengur, 'A novel 3D skeleton algorithm based on neutrosophic cost function', Applied Soft Computing Journal, vol. 36, pp. 210-217, (2015). Guo, Y. and A. Sengur, 'NCM: Neutrosophic c-means clustering algorithm', Pattern Recognition, vol. 48, no. 8, pp. 2710-2724, (2015). Guo, Y., A. Şengür, and J. Ye, 'A novel image thresholding algorithm based on neutrosophic similarity score', Measurement: Journal of the International Measurement Confederation, vol. 58, pp. 175-186, (2014). Ye, J., 'Multicriteria decision-making method using the correlation coefficient under single-valued neutrosophic environment', International Journal of General Systems, vol. 42, no. 4, pp. 386-394, (2013). Otsu, N., 'A threshold selection method from gray-level histograms', Automatica, vol. 9, no. 1, pp. 6266, (1975). Benezeth, Y., P. M. Jodoin, et al., 'Comparative study of background subtraction algorithms', Journal of Electronic Imaging, vol. 19, no. 3, pp. 033003-033003-12, (2010).

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 10, No. 9 (2017)