Issuu

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 07 Issue: 01 | Jan 2020

p-ISSN: 2395-0072

www.irjet.net

Driver Drowsiness Detection System Using Computer Vision Aditya Ranjan1, Karan Vyas2, Sujay Ghadge3, Siddharth Patel4, Suvarna Sanjay Pawar5 1,2,3,4B.E. Student, Dept. of Computer Engineering, Sinhgad College of Engineering, Pune, Maharashtra, India 5Professor, Dept. of Computer Engineering, Sinhgad College of Engineering, Pune, Maharashtra, India

---------------------------------------------------------------------***---------------------------------------------------------------------Abstract - A driver drowsiness detection system is proposed that involves detection of driver drowsiness by use of an algorithm. For detection of drowsiness, the most relevant visual indicators that reflect driver's condition is the eye behavior. The facial algorithm employed makes use of an eye aspect ratio and physical landmark measurements. Landmark detectors used in the algorithm demonstrate robustness against varied head orientations, facial expressions and lighting conditions. The proposed real time algorithm will estimate eye aspect ratio that measures eye open level in each video frame. It perceives eye blink pattern as EAR values. By doing so, potential drowsiness is detected. A large number of road mishaps occur due to drivers falling asleep due to exhaustion or long-haul driving and negligence. The proposed system under development can help prevent the same by providing non-invasive and easy to use specialized devices.

localization and detection model can provide results with an average error below five percent of interocular distance. Recent approaches provide even better real-time performance and accuracy [10]. Thus, we hereby propose an efficient approach to detect driver drowsiness that can detect human eye blinks using a recent facial landmark detection. Eye Aspect Ratio is a single scalar quantity that is derived from the landmarks to reflect level of eye opening. Further, each sequence of input video clip is used to obtain estimates of eye-opening level. Eye blinks are positively detected with the help of a Support Vector Machine classifier trained on blinking and nonblinking patterns of humans. Model presented in [12] is similar to the proposed method which can help in drowsiness detection of driver. However, it requires on an average, five seconds per frame for segmentation and eye open level is normalized by estimated statistics. Thus, the presented model is capable of only offline processing. Proposed driver drowsiness detection method can run in real time with negligible extra cost of eye opening using linear SVM and facial landmarks.

Key Words: HOG Histogram of Gradients, EAR Eye Aspect Ratio, SVM Support Vector Machine. 1. INTRODUCTION Monitoring eye blinking pattern is primordial in driver awareness systems that are developed to monitor human operator drowsiness [5,11]. System can warn a user who infrequently blinks for a long time to prevent dry eye [13,6,7].

1.1 LITERATURE SURVEY 1. PHYSIOLOGICAL CHARACTERISTICS SENSING The most accurate method is based on human physiological activity [1]. The sensing technique can implement in two ways: first is measuring of changes in physiological signals, such as brain waves, heart rate, and eye blinking; and the second one is measuring the physical changes which can be sagging posture, leaning of the driver’s head and the open/closed states of the eyes [1]. Though this technique is most accurate, it is not that realistic in nature, since sensing the electrodes would have to be attached directly into the body of the driver, and hence it can be annoying and distracting to the driver. In addition, long time driving would result in perspiration on the sensors, which would be diminishing their ability to monitor accurately.

Existing systems are currently either active or passive. Active systems provide higher reliability with use of specialized, intrusive and expensive devices, e.g. eye observation using infrared and special close up cameras [4,8]. Multiple methods tend to provide the feature of automatic detection of eye blinks in a video clip which employ use of motion estimation in eye region. Viola-Jones detector can be used for facial and eye detection. Further on, eye motion is studied by optical flow [6] and adaptive thresholding with frame-by-frame intensity differentiation. Lastly, whether the eye is being covered by eyelids or not is decided. A major drawback of many of the existing approaches is they usually implicitly require high amount of complex device setup in virtue of motion dynamics, head orientation, image illumination and resolution, etc. Heuristic approaches based on image intensity are sensitive irrespective of their high real-time performance.

2. VEHICLE RESPONSE SENSING Driver and vehicle operations can be constantly monitored by steering wheel movement, acceleration and braking patterns, linear speed of vehicle, lateral acceleration and displacement [3]. The suggested technique is a non-intrusive way of drowsiness detection but is generally limited to vehicle type and driver's driving style.

Robust landmark detectors can be used to detect facial characteristic points in a human face image. Such detection models can be trained on “in-the-wild datasets” and are robust in dealing with varied illumination, head orientation and facial expressions. Typically, a state-of-the-art landmark

Impact Factor value: 7.34

ISO 9001:2008 Certified Journal

Page 1555

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 07 Issue: 01 | Jan 2020

p-ISSN: 2395-0072

www.irjet.net

different pattern of blinking. The blink patterns differ mostly in the speed of closing and the speed of opening, a degree of squeezing an eye and in duration. We propose to exploit facial landmark detection model for localization of the eyes and its contours.

3. MONITORING DRIVER RESPONSE The approach used in drowsiness detection involves monitoring the driver response. This requires periodic need of input from driver to the system to indicate alertness. This technique involves a major problem in terms of tiring and annoying the driver of the vehicle at regular intervals 2. PROPOSED SYSTEM

Fig -3: Facial Landmarks Facial landmark detection (as seen in figure 3) is performed on input video sequence, frame by frame. It makes use of dlib, a machine learning toolkit and OpenCV with python bindings. Facial landmark detection can be used to locate and represent human facial features such as: Nose, Eyes, Jawline, Eyebrows, Mouth.

Fig -1: Proposed System Overview

Facial landmark detection is currently applied for head pose estimation, face alignment and swapping, blink detection and more. Detecting facial features in a human face is a subset problem of shape prediction. A shape predictor is given an input image which then localizes regions of interest along the shape. In facial landmark detection, we detect important facial structures on a face using shape prediction models. Any typical facial landmark detection approach has two significant steps i.e. 1. Localization of face in input data (image/video). 2. Detection of important facial structures on the face region of interest.

Fig -2: Proposed Hardware Placement Diagram The system being proposed is a driver drowsiness detection system that can be placed in a vehicle. Figure 1 explains the proposed system overview and figure 2 showcases the proposed placement of required hardware to practically implement the said system.

Typically, the first step is achieved by applying a pre-trained H.O.G. and Linear S.V.M. object detector primed specifically for face detection tasks. Likewise, a face bounding box is generated i.e. (x,y) coordinates of the face in input data (video/image). The second step involves the process of localization and labelling of facial regions like nose, mouth, jaw, left eye, right eye, left eyebrow and right eyebrow.

Blinking is a process of fast closing and reopening of a human eye. Each and every individual has a somewhat

Impact Factor value: 7.34

ISO 9001:2008 Certified Journal

Page 1556

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 07 Issue: 01 | Jan 2020

p-ISSN: 2395-0072

www.irjet.net

Facial landmark detection can be implemented by using a training image set of labelled facial landmarks. The set is manually labeled by marking (x,y) coordinates of regions of interest. It will also make use of priors i.e. distance probability between input pair of pixels.

Next step to end goal is to detect and count eye blinks in input video stream for drowsiness detection. Typical approach to counting blinks involve localization of eyes, thresholding to determine white region of eyes and determining if the said region disappears for a certain period of time thus indicating a blink.

After this stated process, an ensemble of regression trees can then be trained to locate facial landmarks using pixel intensities and not feature extraction. This results in realtime and high-quality predictions.

The proposed drowsiness detection system uses an elegant and reliable solution of eye aspect ratio which is based on distance ratio between facial landmarks of the eyes. E.A.R. provides an easy, fast and efficient eye blink detection method. For blink detection, we focus on indices of two set of facial structures i.e. the eyes.

The pretrained facial landmark detection model of dlib library can localize sixty-eight (x,y) coordinates, each representing a facial structure. Index locations of each region of interest is shown in figure 4. Points shown in figure 4 are part of iBUG 300-W dataset on which dlib facial landmark detector is trained on. The dlib library also provides a minor modified version of H.O.G. and linear S.V.M. method for object detection. Input video from a camera looking over the driver is processed into number of frames based images and rescales them to a width of 500 pixels. Gray scaling is performed on input frames. Face bounding boxes are generated on each image frame. The availability of generated face coordinates allows localization of each region of the face. Each part of the face can be accessed individually to extract eye region by NumPy array slicing.

Fig -5: Eye Landmark Points

By using zero indexing with python, we can obtain localization information of eye region which has (according to figure 4):

Each eye is depicted by six (x,y) coordinates, beginning with left corner of eye and plotting clockwise around the eye region as shown in figure 5. The relation between the width and height of the plotted coordinates is called the E.A.R.

Fig -6: Eye Aspect Ratio (E.A.R.) Equation The relation has six points represented by p1, p2, p3, p4, p5 and p6 which in turn are two-dimensional eye landmark locations. The numerator of E.A.R. equation computes vertical eye landmark distance while the denominator computes horizontal eye landmark distance. There exist two sets of vertical and one set of horizontal points. E.A.R. value remains approximately constant when the driver's eye is open but drastically falls to zero value when a blink is detected. It is partially person and head pose insensitive. Aspect ratio of the open eye features a small variance among individuals and it's fully invariant to a consistent scaling of the image and in-plane rotation of the face. As blinking of eyes is a synchronous process, the EAR of both eyes is averaged. Hence, by use of E.A.R., we cut down the need of an image processing technique and rely on E.A.R. to detect eye blink.

Fig -4: Facial Landmark Index Locations 1. Right eyebrow through 17 to 22 2. Left eyebrow through 22 to 27 3. Right eye through 36 to 42 4. Left eye through 42 to 48

Impact Factor value: 7.34

ISO 9001:2008 Certified Journal

Page 1557

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 07 Issue: 01 | Jan 2020

p-ISSN: 2395-0072

www.irjet.net 3. PROPOSED SYSTEM RESULTS

Fig -7: E.A.R. Value Graph In figure 7, in the top left image, the eye seems to be completely open hence, E.A.R. should be larger and constant over time. In case of a blink, like in the top right image, E.A.R. rapidly approaches zero value. The bottom graph plots E.A.R. values over a period of time for a video clip. E.A.R. remains constant and then drastically drops close to zero value and then recovers back thus depicting a single eye blink. A blink or eye closure can be registered using thresholding by value of 0.3 for E.A.R. value which can turn out to suit many applications. We also set the number of successive frames that must have an E.A.R. value below 0.3 to register drowsiness. This can be ideally set to 48 to help detect driver drowsiness. The thresholding values used can be modified as per other implementation needs.

Fig -8: Normal Operation (Eyes are open)

Live video feed can be taken from a camera mounted near or on dashboard of car and pointing to the vehicle's driver. It can be a USB camera or Raspberry Pi camera module. We can average the E.A.R. values obtained of each eye to obtain more precise blink detection. Low E.A.R. values along with duration for which it remains below threshold value can together help detect drowsiness in vehicle drivers. If E.A.R. values indicate that the driver's eye has been closed or near-close for a significant amount of time then the system can sound an alert to wake up the driver. The E.A.R. is closely monitored for scenarios where the value drops low but does not recover back up again, thus implying drowsiness and that the driver has closed his/her eyes.

Fig -9: Drowsiness detected and alerted (Eyes are closed) 4. CONCLUSIONS

The system can be modified to detect just one or many faces at a single go along with drowsiness detection for each face. The drowsiness detection seems to provide robust results in a variety of environmental conditions like driving under the sunlight and in low or artificial lighting.

Proposed system will be efficient in driver drowsiness detection by providing reliably precise enough estimation of level of eye openness. This is due to robustness towards low image resolution and improper head orientation, low illumination and varied facial expressions, etc. Proposed alert system can be used in real time due to a very negligible performance cost experienced in facial landmark detection and can also be made to use of linear SVM. Fixed blink duration is assumed even though everyone’s blink duration lasts differently. The assumption approach can be replaced by an adaptive approach. EAR is estimated from two-dimensional data which cannot account for out of plane head orientation. This can be corrected by estimating EAR using three-dimensional data (position and orientation)

Impact Factor value: 7.34

ISO 9001:2008 Certified Journal

Page 1558

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 07 Issue: 01 | Jan 2020

p-ISSN: 2395-0072

www.irjet.net

of three-dimensional landmark model. In the current system, failure to detect eyes in input video sequence is alerted to the driver to correct or switch to a suitable sitting posture.

[12] Z. Yan, L. Hu, H. Chen, and F. Lu. Computer vision

The proposed system can further be extended by adding more safety features and functionalities to the current system to help prevent road accidents. Possible future revisions of the system can include the addition of features like traffic light detection, speed limit zone detection and speed tracking and traffic violation detection.

[13] T. Soukupova and J. Cech. Real-Time Eye Blink Detection

syndrome: A widely spreading but largely unknown epidemic among computer users. Computers in Human Behaviour, (24):2026–2042, 2008. 1 using Facial Landmarks. In 21st Computer Vision Winter Workshop Luka Cehovin, Rok Mandeljc, Vitomir ˇ Struc (eds.) ˇ Rimske Toplice, Slovenia, February 3–5, 2016 AUTHORS

REFERENCES [1]

W.W. Weirwille, “Overview of Research on Driver Drowsiness Definition and Driver Drowsiness Detection,” 14th International Technical Conference on Enhanced Safety of Vehicles, pp 23-26, 1994.

[2]

A. Kircher, M. Uddman, and J. Sandin, “Vehicle control and drowsiness,” Swedish National Road and Transport Research Institute, Tech. Rep. VTI-922A, 2002.

[3]

M. Eriksson and N.P. Papanikolopoulos, “Eye-tracking for Detection of Driver Fatigue”, IEEE Intelligent Transport System Proceedings, pp 314-319, 1997.

[4]

L. M. Bergasa, J. Nuevo, M. A. Sotelo, and M. Vazquez. Real-time system for monitoring driver vigilance. In IEEE Intelligent Vehicles Symposium, 2004. 1

[5]

T. Danisman, I. Bilasco, C. Djeraba, and N. Ihaddadene. Drowsy driver detection system using eye blink patterns. In Machine and Web Intelligence (ICMWI), Oct 2010. 1, 6, 7

[6]

M. Divjak and H. Bischof. Eye blink-based fatigue detection for prevention of computer vision syndrome. In IAPR Conference on Machine Vision Applications, 2009. 1

[7]

T. Drutarovsky and A. Fogelton. Eye blink detection using variance of motion vectors. In Computer Vision ECCV Workshops. 2014. 1, 2, 5, 6, 7

[8]

Medicton group. The system I4 Control. http:// www.i4tracking.cz/. 1

[9]

S. Ren, X. Cao, Y. Wei, and J. Sun. Face alignment at 3000 fps via regressing local binary features. In Proc. CVPR, 2014. 2

ADITYA RANJAN

KARAN VYAS

SUJAY GHADGE

SIDDHARTH PATEL

PROF. SUVARNA SANJAY PAWAR

[10] A.

Sahayadhas, K. Sundaraj, and M. Murugappan. Detecting driver drowsiness based on sensors: A review. MDPI open access: sensors, 2012. 1

[11] F. M. Sukno, S.-K. Pavani, C. Butakoff, and A. F. Frangi.

Automatic assessment of eye blinking patterns through statistical shape models. In ICVS, 2009. 1, 2

Impact Factor value: 7.34

ISO 9001:2008 Certified Journal

Page 1559