Human Driver’s Drowsiness Detection System

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 06 | Jun 2022 www.irjet.net p ISSN: 2395 0072

Human Driver’s Drowsiness Detection System

1,2,3,4 Student, Department of Computer Science and Engineering, Jain University, Bengaluru, India 5Guide, Department of Computer Science and Engineering, Jain University, Bengaluru, India ***

Abstract An Advancements in technology and artificial intelligence in the past few years have led to improvements in driver monitoring systems. Many studies have collected real driver drowsiness data and applied machine learning algorithms to enhance the performance of these systems. This paper presents a review report on the project to develop a system for driver drowsiness detection to prevent accidents caused by driver fatigue. This paper contains reviews of recent systems using different methods to detect drowsiness. In this paper, the proposed system captures video of the driver's face to detect drowsiness and alert an alarm if needed. A machine learning algorithm was applied to the model to evaluate the accuracy of this approach. Real world implementation of the project gives an idea of how the system works and what can be done to improve the accuracy of the overall system. Furthermore, the paper highlights the observation, accuracy, and challenges of the system.

Keywords: Accidents, Drowsiness Detection, Eyes Detection, Fatigue, Machine Learning, Yawn

1. INTRODUCTION

We often hear of drunken driving, no seat belt, speeding, harsh weather, and mechanical failures. But one of the biggestandyetoftenunrecognizedhumanerrorsisdrowsy driving. A major problem not only in India but across the globe. The risk,danger,and often tragic resultsofdrowsy drivingarealarmingindeed.Lackofsleepisthemainculprit. And add to it the catalyst agents like medications, alcohol andsleepdisorders,andsleepinessgetsaggravated.Many drivers in the country sacrifice sleep, an often overlooked anddangerousbehaviourthatresultsinthemajorityofthem being sleep deprived while behind the wheel every day. There is no official count of lives lost in drowsy driving relatedcrashesinourcountry.Fallingasleepatthewheelis suicidal.Itisnotonlydangeroustothedriverbutallother roadusers. Inthisprojectwearegoingtobuildadetection systemthatusesOpenCVtocapturedrivers’facesusingEye Aspect Ratio (EAR), Mouth Aspect Ratio (MAR) and a machinelearningmodeltodetectdrowsiness

2. PROPOSED SYSTEM

Accordingtocurrenttechnology,monitoringdriverswhile driving is quite complex computation and expensive equipment, and it is also not comfortable to wear while driving.Forexample,EEG,andECGtocheckthefrequency

and rhythm of the heart. As a new solution, a drowsiness detectionsystemwhichusesacameraplacedinfrontofthe driverismoresuitabletobeusedbutthephysicalsignsthat willindicatedrowsinessneedtobelocatedfirsttocomeup withadrowsinessdetectionalgorithmthatisdependableand accurate.Lightingintensityandtheorientationofthedriver aretheproblemsthatoccurduringthedetectionofeyesand mouthregions.

So,inthisproject,weproposeamethodtocapturedrivers’ faces using a webcam or a small camera and analyze each frameofthevideowearegettingtodetectdrowsiness.

3. OBJECTIVE

Tosuggestawaytodetectfatigueanddrowsiness whiledriving

Toinvestigatesthephysicalchangesoffatigueand drowsiness

To develop a system that uses the closing of eyes andyawningtodetectfatigueanddrowsiness.

To provide an alert (sound) when drowsiness occurs

4. SYSTEM IMPLEMENTATION

Inourproject,implementationisdoneintwoparts.Thefirst partisbuildingamodeltodetectthestatusoftheeyes.The nextpartisdetectinghowmanyyawnsarecaptured.Since theclosureofeyesandyawncountareimportantaspectsof drowsiness.

WewillbeusingPython,TensorFlowandOpenCVtobuild thisdetectionsystem.OpenCVwillbeusedtomonitorthe driverusingawebcamandthefeedwillbetransferredinto ourmachinelearningmodeltodetectthestatusoftheeyes.

Face landmarks:

DLibisacross platformsoftwarelibraryoriginallywritten in C++ that contains machine learning algorithms. Even though itisa C++library,manyofitstoolscanbeused in python. It is majorly used for face detection and facial landmarkdetection.TheDLiblibrarycanbeusedtodetecta faceinanimageandthenfindsixty eightfaciallandmarkson thedetectedface.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 06 | Jun 2022 www.irjet.net p ISSN: 2395 0072

Figure 1: Facial landmarks

Sometimeswedonotneedtouseallthelandmarkpointsbut for our project, we are going to use eyes and mouth landmarkpoints.

Detection of Eyes:

At first, the video is captured using a webcam or a small camera.Fromthatvideo,thefaceofthedriverisdetected using the Haar cascade algorithm and then the eyes are detected.ThenweusedourCNNmodeltodetectthestatus oftheeyewithanaccuracyof98%.HaarCascadeClassifier is used for detection of the face and EAR is used on both eyes.EARisdefinedastheproportionbetweentheheight andthewidthoftheeyebasedonitslandmarks.IftheEAR goes below a threshold value i.e., closing of eyes for a particular period an alert in the form of an alarm will be heardbythedriver.

5. ARCHITECTURE

Figure 2: EAR

Yawn Count:

Themouthisrepresentedbyeightlandmarkpoints.Atany pointoftimewhenapersonopenshis/hermouthtoyawn, thedistancebetweentheupperandlowerlandmarkpoint increases. The proportion between upper and lower lip distancetotheleveldistancebetweenthecornerofthelips isusedtodetermineMAR.Wewillassignathresholdyawn count.Iftheperson’syawncountismorethanthethreshold countanalertintheformofanalarmwillbeheardbythe driver.

Detecting stage:

In the initial detecting stage, the face of the person is detected and creates a square boundary as shown in the figurebyenclosingthecorepartofdrowsinesslikeeyesand mouth inside the boundary and for this we will be using Haarcascadebygoogletodetecttheface.

Tracking stage:

Afterdetectingstagetherewillbetrackingstagewherethe coreregionofdrowsiness(eyes,mouth)willbetrackedby usingtwoparameters MAR(MouthAspectRatio),EAR(Ear AspectRatio)fortrackingwhetherthepersonisclosingeyes than a particular ratio as well as the person is opening mouth than a particular ratio in order to detect the drowsiness.

Warning stage:

Afterdetectingtheeyesandmouthifthepersonistracked as yawning or sleeping the model will provide a warning alarmthatwillbejusttoalertthedrowsinesstomakethe driverengageindriving.

Alert stage:

In this final stage, If the yawning and sleeping count goes more than a particular count there will be an alertness to stopdriving.Inthisstageinsteadofgivingasmallalertthe alarmwillbecontinuouslybeepinguntilthedriverstopsthe car.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 06 | Jun 2022 www.irjet.net p ISSN: 2395 0072

6. SEQUENCE DIAGRAM

Smallcameraorwebcam:

Figure 5: Sequence Diagram

Training stage:

Inthetrainingstage,thedataset(MRLeye)containingthe imagesofclasses(open,close)eyeswillbepre processedby resizing,convertingtograyscaleetc.Asthebetterprediction isoccurredwhenwehaveagreaternumberofsamplessoto occupy more number of samples, we will be doing data augmentation, by using image data generator we will be generating more samples that contains (rotated, flipped, invertedetc.)samplesoftheoriginalimagesowhiletraining themachinelearnsbetter.Movingfurtherweareusingapre trained model (sequential) that contains pre trained parametersthatcanbedirectlyusedinourmodelinorderto use that we will be importing the model and after adding layersthatisneededforourclassification,wewillbefitting the model. After training the model will be saved to the systemin.h5format.

Testing stage:

Inthetestingstage,wewillbeusingOpenCVforrealtime face detection using video streaming. For classification of drowsinesswe will beimportingthetrained.h5 model as wellaswewillbeusingHaarcascadeclassifierfortheface detection and to create boundary, moving further the ROI like eyes and mouth will be extracted for capturing drowsiness feature and if the drowsiness is captured through video the warning alarm will be given and if the countofdetectionwillbeincreasedthestoppingalarmwill begiventothedriverinordertostopdriving.

7. HARDWARE DETAILS

LaptoporPC:

Figure 6: Laptop

Figure 7: Webcam

8. SOFTWARE DETAILS

Python 2.7 andaboveis requiredaswe are using python as our language to implement, and we recommend the latest version 3.7 for the TensorFlowenvironment. 

OpenCVisahugeopen sourcelibraryforcomputer vision,machinelearning,andimageprocessing.By using it, one can process images and videos to identifyobjects,faces,oreventhehandwritingofa human. 

Anaconda navigator: we need an “anaconda navigator”fortheimplementationoftheprojectand it is used for launching a variety of python interpretersand pythonIDE.And werecommend thelatestversion. 

Jupyter notebook we need to install the latest version of “Jupyter notebook” as a python interpreter,oryoucanuse“visualstudiocode,”or GoogleColabwhichisacloud basedonlinepython interpreterbygoogle.

9. OUTPUT

IftheEARgoesbelow0.25andtheyawncountgoesabove4, thepersonissaidtobe in drowsiness state. The detection was done based on whetherthepersonwaswearingspectaclesornot.

Figure 8: With spectacles output 1

Thepersonhasyawnedfourtimesasofrightnow.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 06 | Jun 2022 www.irjet.net p ISSN: 2395 0072

First implementation: CNN(InceptionV3)

Final implementation: CNN(Sequential)

Figure 9: With spectacles output 2

Thepersonhasyawnedmorethanfourtimes.Analertwill displayalongwithanalarmringingstatingthatthe“Subject isyawningalot”.

Figure 12: Model Accuracy Graph

11. CONCLUSIONS

In the end, we were able to detect the drowsiness of the driverinreal time.Thisisdonebyinstallingawebcamora smallcamerainfrontofthedriver.ACNNmodelwasapplied tothereal timevideotodetectthestatusofeyeswith98% accuracy. If the eyes were closed for some frames in the videothedriverwillhearanalertsound.

Figure 10: With spectacles output 3

The EAR has gone below 0.25 stating that the person is sleeping.Adrowsinessalertwillbedisplayedalongwithan alarmringing.

Similarly, an algorithm was created using Haar Cascade Classifier to detect yawn counts based on MAR (Mouth AspectRatio).Analertsoundwasheardiftheyawncount goesabovethethresholdcount(i.e.,fouryawncountsinour method).

Theory 1: OpenCV,CNN.

Theory 2: CNN,LOP.

Theory 3: OpenCV,DLib,EAR,SVM.

Theory 4: MinMaxScalar,CNN.

Our Model: CNN(InceptionV3,Sequential),OpenCV,EAR, MAR.

Figure 11: With spectacles output 4

10. OUTPUT ANALYSIS

Togetabetterunderstandingofthebehaviourofthesystem, weinvestigatedtheresultsofthemosteffectivemodelinthe previousroundoftesting.

Whenwestartedthisproject,wethoughtofgettingabetter orbetteraccuracycomparedtotheotherstudies.Wethen implementedtheCNN model onthe dataset,and wewere providedwiththebestaccuracyamongallwhichis98%for drowsinessdetection.

Figure 13: Comparison Accuracy Graph

2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 06 | Jun 2022 www.irjet.net p ISSN: 2395 0072

12. FUTURE SCOPE

The future work on this project can focus on the automaticallyzoomingofthecameraontotheeyesoncethey arelocalized.Algorithmscanbeimprovedtodetecteyesand mouth.Lightingchangesmustbeencounteredtogetbetter detection.Bettervideoqualitytogetdetectionfasterasthe numberofpixelsincreases, thedetectionwill beaccurate. And finally, this system can be connected to the speed control of the car. So, when the system detects the drowsiness state of the driver, it can alert the driver with soundwhilesimultaneouslyreducingthespeedofthecar.

Forthefurtherproceedingofthisproject,Aprototypemodel canbedevelopedtostopthecarwhenthedrivercompletely goes under sleep. This can be developed by using IOT deviceslikeRaspberrypi,Arduinoandsensorslikesteering anglesensor,Otherthangettingdrowsinessthepersoncan gounderpanicattack,faint,dizzywherethedriverwon'tbe abletogetthecontrolofthecarinthatcaseitisnecessaryto develop a model that stops the car by detecting the face, handanglepositiononsteeringwheel.

Inthefuture,thepopulationandtechnologygrowalongwith that the usage of a variety of vehicles will be increasing, whichwillincreasemoreandmoretrafficviolations,Soto stop this a system is very important. The model that we createdaswellastheproposedfuturemodelprototypewill provideamilestoneintheindustryofartificialintelligence andautomobilesinordertoreducethetrafficaccidentsand violations.

13. REFERENCES

[1] V. Chaudhary, Z. Dalwai and V. Kulkarni, “Intelligent Distraction and Drowsiness Detection System for Automobiles,”inIEEE,04August2021.

[2] F. You, X. Li, Y. Gong, H. Wang and H. Li, “A Real time Driving Drowsiness Detection Algorithm With Individual DifferencesConsideration,”inIEEEAccess,2019.

[3]G.Roshini,Y.Kavya,R.Hareesh,M.SumaandN.Sunny, “Driver Distraction and Drowsiness Detection System,” in IEEE,04August2021.

[4] P. Bajaj, R. Ray, S. Shedge, S. Jaikar and P. More, “SynchronousSystemforDriverDrowsinessDetectionUsing Convolutional Neural Network, Computer Vision and Android Technology,” in IEEE, Coimbatore, India, 03 June 2021.

[5]Flores,MarcoJavier,JoséMaríaArmingolandArturode la Escalera. “Driver drowsiness detection system under infrared illumination for an intelligent vehicle.” Iet Intelligent Transport Systems 5 (2011): 241 251., “Driver drowsinessdetectionsystemunderinfraredilluminationfor anintelligentvehicle,”vol.5,2011.

[6]AvigyanSinha;RPAneesh;SaradaKGopal,“Drowsiness Detection System Using Deep Learning,” in IEEE, 4 June 2021.

[7]RatebJabbara,KhalifaAl Khalifaa,MohamedKharbechea, Wael Alhajyaseen,Mohsen, “Real timeDriverDrowsiness DetectionforAndroidApplication,”inThe9thInternational Conference on Ambient Systems, Networks, and Technologies,2018.

[8]CharlotteJacobédeNauroisa,ChristopheBourdina,Anca Stratulatb, Emmanuelle Diazb, Jean Louis Vercher, “Detection and prediction of driver drowsiness using artificialneuralnetwork,”inElsevier,2017.

[9] F. Friedrichs and B. Yang, “Camera based drowsiness reference for driver state classification under real driving conditions,”in2010IEEEIntelligentVehiclesSymposium, 2010.

[10] K. Satish, A. Lalitesh, K. Bhargavi, M. S. Prem and T. Anjali,“DriverDrowsinessDetection,”inIEEE,01September 2020.

[11]ChallaYashwanth,“DriverDrowsinessDetectionSystem Based on Visual Features,” in IEEE, Greater Noida, Uttar Pradesh,India.

[12]MohammadAminAssari;MohammadRahmati,“Driver drowsinessdetectionusingfaceexpressionrecognition,”in IEEE,2February2012.

2022, IRJET | Impact Factor value: 7.529 | ISO 9001:2008 Certified Journal