Gesture Controlled Virtual Mouse Using Computer Vision

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:13Issue:04|Apr2026 www.irjet.net p-ISSN:2395-0072

Gesture Controlled Virtual Mouse Using Computer Vision

Manikanta Gugulothu1 , Meenakshi Sakre2 , Rakshitha Vennamaneni3, Raheem Mohammad4, Mr. M. RamaMohan5

1-4 Undergraduate Students, Department of Information Technology, Vidya Jyothi Institute of Technology, Hyderabad, Telangana, India

5 Assistant Professor, Department of Information Technology, Vidya Jyothi Institute of Technology, Hyderabad, Telangana, India ***

Abstract - This paper presents the development of a gesture-controlled virtual mouse using computer vision techniques. The system enables users to control a computer cursor using hand gestures captured through a standard webcam, eliminating the need for a physical mouse. The proposed approach uses MediaPipe for hand landmark detection and OpenCV for real-time video processing. By tracking 21 key points on the hand, the system interprets different gestures and maps them to common mouse operations such as cursor movement, clicking, dragging, and scrolling. During development, emphasis was placed on achieving smooth cursor control and maintaining reliable performance under normal lighting conditions. The system provides a simple and contactless method of interaction, makingit usefulinenvironments where touch-free operation is preferred. In addition to improving user convenience, the system can also assist individuals with physical limitations. Sinceitrequiresonlyawebcamandnoadditionalhardware,it offers acost-effectivealternativetotraditionalinputdevices. Overall, the proposed system demonstrates a practical approach to gesture-based human-computer interaction.

Key Words: Gesture Recognition, Computer Vision, Virtual Mouse, Human-Computer Interaction, MediaPipe, OpenCV, Hand Tracking, Touchless Interaction

1.INTRODUCTION

Inrecentyears,touch-freeinteractionhasgainedsignificant attentioninthefieldofmoderncomputing.Insteadofrelying only on traditional input devices such as a mouse or keyboard, there is a growing interest in interacting with systemsusingnaturalhumangestures.Withtheincreasing demandforintuitiveanduser-friendlyinterfaces,gesturebasedsystemsarebecominganimportantareaofresearch inhuman-computerinteraction.Althoughconventionalinput devices are widely used, they have certain limitations. In environments such as hospitals or shared workspaces, continuousphysicalcontactwithdevicescanraisehygiene concerns.Inaddition,peoplewithphysicaldisabilitiesmay find it difficult to use standard input hardware. These challengesmotivatedustoexploreatouchlessinteraction approachusingcomputervision.Inthiswork,wefocuson

developingavirtualmousethatcanbecontrolledusinghand gestures.Theideaistouseawebcamtocapturereal-time handmovementsandconvertthemintocursoractionssuch asmovement,clicking,dragging,andscrolling.Thisprovides amorenaturalwayofinteractingwiththecomputerwithout requiringanyphysicaldevice.Toimplementthissystem,we usedMediaPipeandOpenCV.MediaPipeisusedfordetecting andtrackinghandlandmarks,whileOpenCVhandlesimage processing and real-time video analysis. During development, we found that combining these two frameworksprovidesagoodbalancebetweenaccuracyand performance under normal conditions. Apart from improving user convenience, touch-free systems can also enhanceaccessibilityandmaintainbetterhygieneincertain environments. Such systems can be useful in applications likevirtualreality,gaming,assistivetechnologies,andsmart environments. Overall,this projectdemonstratesa simple andcost-effectiveapproachtobuildingagesture-controlled virtual mouse using computer vision, making humancomputerinteractionmorenaturalandflexible.

2. LITERATURE SURVEY

[1]Kumaretal.proposedahandgesturerecognitionsystem usingConvolutionalNeuralNetworks(CNNs)forclassifying static gestures such as open palm, fist, and pointing. The model achieved high accuracy due to strong feature extraction capabilities of CNNs. However, the system requires high computational resources, including GPU support,andstruggleswithdynamicgesturesandocclusion scenarios.

[2]Pateletal.introducedareal-timehandtrackingsystem using MediaPipe, which detects 21 hand landmarks for gesturerecognition.Thesystemperformsefficientlyevenon low-end devices and supports real-time applications. However, its accuracy decreases under poor lighting conditionsandcomplexbackgrounds.

[3] Patel et al. presented a vision-based hand gesture recognition system using OpenCV techniques such as contourdetection,backgroundsubtraction,andconvexhull analysis.Theapproachissimpleandsuitableforreal-time applications with low computational cost. However, it is sensitivetonoiseandlightingvariations,resultinginlimited accuracy.

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:13Issue:04|Apr2026 www.irjet.net

[4]Singhetal.developedareal-timegesture-basedvirtual mousesystemusingMediaPipeforhandtrackingandrulebasedgesturemappingformouseoperationssuchascursor movement and clicking. The system provides lightweight and real-time performance without requiring additional hardware.However,itsupportsalimitedsetofgesturesand showsreducedaccuracyduringfasthandmovements.

[5]Kumaretal.proposedahandgesturerecognitionsystem using machine learning classifiers such as Support Vector Machines (SVM) and K-Nearest Neighbors (KNN). The system uses extracted hand features like finger distances andanglesforclassification,achievingbetteraccuracythan traditionalmethods.However,itrequiresalargeamountof training data and involves higher computational cost. Despitetheseadvancements,existingsystemsfaceseveral challengessuchassensitivitytolightingconditions,reduced performanceindynamicenvironments,limitedgesturesets, and dependency on high computational resources. To overcometheselimitations,theproposedsystemintegrates MediaPipe and OpenCV to achieve accurate, real-time gesturerecognitionwithminimalhardwarerequirements, enabling efficient cursor control and improved user interaction. To address these challenges, the proposed systemintroducesagesture-controlledvirtualmouseusing computervisiontechniquesthatcombinestheefficiencyof MediaPipeforhandtrackingwiththeflexibilityofOpenCV forreal-timeprocessing.Thesystemisdesignedtoperform cursor movement and mouse operations such as clicking, dragging,andscrollingusingsimplehandgestures,ensuring a balance between accuracy, performance, and costeffectiveness.

3.PROPOSED MODEL

Theproposedsystemisagesture-controlledvirtualmouse thatenablesuserstointeractwithacomputerwithoutusing aphysicalmouse.Inthiswork,weuseastandardwebcamto capturereal-timevideoinput,whichisthenprocessedusing computervisiontechniquestointerprethandgestures. Thesystemcontinuouslycapturesvideoframesthroughthe webcam.TheseframesareprocessedusingtheMediaPipe frameworktodetectandtrackhandlandmarks.MediaPipe provides21keypointsonthehand,whichmakesitpossible toaccuratelytrackfingerpositionsandmovementsinreal time.

Basedontheselandmarks,differentgesturesarerecognized using simple predefined conditions. OpenCV is used for imageprocessingandhandlingvideoframes,whichhelpsin maintainingsmoothandefficientreal-timeperformance. The detected hand movements are mapped to screen coordinates to control the cursor. Different gestures are assigned to perform mouse operations such as cursor movement,leftclick,rightclick,dragging,andscrolling.For instance,themovementoftheindexfingerisusedtocontrol thecursor,whilespecificcombinationsoffingersareusedto performclickactions.

p-ISSN:2395-0072

During implementation, we observed that small hand movements could cause unwanted cursor fluctuations. To addressthis,smoothingandthresholdingtechniqueswere applied to improve cursor stability and reduce noise. The gesture recognition logic was also designed to work reasonablywellundermoderatelightingvariations. Theoverallsystemisdividedintomultiplestages,including video acquisition, hand detection, landmark extraction, gesture recognition, and action mapping. This modular approachmakesthesystemeasiertounderstandandallows futureimprovements.

Theproposedmodeliscost-effectiveanddoesnotrequire any additional hardware apart from a webcam. It can be useful in applications where touch-free interaction is preferredandcanalsoassistuserswithphysicallimitations. Withfurtherimprovements,thesystemcanbeextendedto areas such as virtual reality, gaming, and smart environments.

4.METHODOLOGY

The proposed gesture-controlled virtual mouse system followsastructuredworkflowconsistingoffivemainstages: video acquisition, hand detection, landmark extraction, gesturerecognitionandmapping,andcursorcontrol.This step-by-stepapproachhelpsinachievingaccurateandrealtimetouch-freeinteractionwiththecomputer.

-1: SystemArchitecture

4.1 Video Acquisition

Theprocessbeginswithcapturingreal-timevideoinput usingastandardwebcam.Thecameracontinuouslycaptures frames, which are then processed by the system. These frames provide the necessary input for detecting and trackinghandmovements.

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr2026 www.irjet.net p-ISSN:2395-0072

4.2 Hand Detection

Inthisstage,MediaPipeisusedtodetectthepresenceof ahandineachvideoframe.Thesystemidentifiesthehand regionandseparatesitfromthebackground.Duringtesting, we observed that the detection works well under normal lightingconditions,althoughperformanceslightlydecreases inlow-lightenvironments.

4.3 Landmark Extraction

Once the hand is detected, MediaPipe extracts 21 key landmark points representing different parts of the hand, including finger joints and tips. These landmarks provide precisepositionalinformation,whichisusedtounderstand fingermovementinrealtime.

4.4 Gesture Recognition and Mapping

Thesystemusespredefinedrulestorecognizedifferent handgesturesandmapthemtomouseoperations.Anopen palm gesture is used to initialize the system. Cursor movement is controlled using the index finger, while different finger combinations are used for actions such as clickinganddragging.Forexample,bringingtheindexand middlefingersclosetogethertriggersaclickevent,whilea closed hand gesture is used for dragging. These gestures were selected based on trial and testing to ensure better reliabilityduringusage.

4.5 Cursor Control and Action Mapping

Inthefinalstage,therecognizedgesturesareconverted into corresponding mouse actions. The movement of the index finger controls the cursor position on the screen. Differentgesturestriggeroperationssuchasleftclick,right click, dragging, and scrolling. OpenCV is used for screen mapping and to ensure smooth cursor movement. During implementation, smoothing techniques were applied to reducejitterandimproveoveralluserexperience.

5.RESULT AND DISCUSSION

Theproposedgesture-controlledvirtualmousesystemwas developedusingPythonandimplementedwiththehelpof OpenCV, MediaPipe, and PyAutoGUI libraries. The system wastestedinrealtimeunderdifferentconditionstoevaluate itsperformance,reliability,andusability.Duringtesting,we considered key performance factors such as accuracy, responsetime,framerate,andoverallsystemstability.We observedthatthesystemperformsconsistentlyacrossmost gesturesandprovidessmoothcursorcontrolduringnormal usage. Under standard lighting conditions, the system achievedanaccuracyofaround94%to96%.Amongallthe gestures, cursor movement showed the highest accuracy, mainly because it relies on continuous tracking of finger

landmarks. On the other hand, gestures like dragging and scrollingshowedslightlyloweraccuracy,astheydependon multiplefingerpositionsandtheircoordination.Intermsof responsiveness, the system showed an average response timebetween25msand45ms,whichallowsnearreal-time interaction without noticeable delay. The frame rate was observed to be between 22 and 28 frames per second, providing smooth visual feedback and stable cursor movement. We also noticed that system performance dependsonenvironmentalconditions.Itworksbestinwelllitenvironmentswithmoderatehandmovement,whilelow lightingandveryfastgesturescanslightlyreduceaccuracy.

Table-1: PerformanceEvaluationofGestureOperations

Gesture Accuracy(%) Response Time(ms)

CursorMovement 96 25 LeftClick 95 35 RightClick 94 40 Drag 93 45 Scrolling 92 45

Thesystemperformedwellunderdifferentconditionsbut showed slight accuracy reduction in low light. Moderate handmovementproducedthebestresults.

Volume:13Issue:04|Apr2026 www.irjet.net p-ISSN:2395-0072

-6: Drag

TheinitializationandhanddetectionprocessisshowninFig -2.CursormovementusinghandgesturesisillustratedinFig -3.TheleftclickoperationisdemonstratedinFig-4,while therightclickoperationisshowninFig-5.Dragoperation usinggesturesispresentedinFig-6.Thesefiguresconfirm the practical implementation of the system and show its real-time functionality. During testing, we observed that environmental conditions have a noticeable impact on system performance. The system works best in well-lit environmentswithminimalbackgroundinterference,where handlandmarksareclearlydetected.Inlow-lightconditions or cluttered backgrounds, a slight drop in accuracy was observedduetodifficultyindetectingprecisehandfeatures. User behaviour also affects performance. Moderate and controlledhandmovementsresultinsmootherinteraction andbetteraccuracy.Incontrast,veryfastorabruptgestures can introduce minor delays due to processing limitations. Whencomparedtotraditionalinputdevices,theproposed systemoffersa flexible and contactlesswayofinteraction withoutrequiringadditionalhardware.Althoughaphysical mouse provides slightly higher precision, the proposed system performs well for general-purpose tasks while offering advantages such as improved hygiene and accessibility.Overall,theresultsindicatethatthesystemis efficient,responsive,andsuitableforreal-timeapplications. ThecombinationofMediaPipeandOpenCVprovidesreliable gesture recognition while keeping computational requirementslow,makingthesystempracticalforeveryday use.

6.CONCLUSION

Thisworkpresentsapracticalapproachforenablingtouchfreeinteractionwithacomputerusinghandgestures.Inthis project,wedevelopedasystemthatallowsuserstocontrol cursor movement and perform mouse operations using a standard webcam, without relying on traditional input devices.Thismakestheinteractionmoreintuitiveanduser-

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:13Issue:04|Apr2026 www.irjet.net

friendly. The system integrates MediaPipe for hand landmark detection and OpenCV for real-time processing, which helps in accurately tracking hand movements. Differentgesturesaremappedtooperationssuchascursor movement,clicking,dragging,andscrolling.Theuseoffinger positionand distance-basedcalculationsprovidesreliable gesture recognition while maintaining real-time performance.Fromtheexperimentalresults,weobserved that the system performs well under normal conditions, providinggoodaccuracywithminimaldelay.Itworksbestin well-litenvironmentswithmoderatehandmovementsand does not require any additional hardware, making it accessible to a wide range of users. The proposed system reducesdependencyonphysicalinputdevicesandoffersa hygienicandcontactlessmethodofinteraction.Italsoadapts reasonablywelltodifferentuserswithoutrequiringcomplex calibration. However, some limitations were identified duringtesting.Thesystemshowsreducedperformancein low-lightconditionsandduringveryfasthandmovements. In addition, complex backgrounds can affect detection accuracy. These challenges highlight the need for further improvements in robustness. Overall, the system demonstratesa practical alternative to traditional mousebasedinteractionandshowsthepotentialofgesture-based interfaces in modern computing. With further enhancements, it can be extended to applications such as virtual reality, gaming, smart systems, and assistive technologies.

7.FUTURE SCOPE

Thecurrentsystemprovidesagoodfoundationfortouchfree human-computer interaction, but there are several areas where further improvements can be made. One important area is improving system performance under different environmental conditions. During testing, we observed that low lighting and complex backgrounds can affect detection accuracy. Future work can focus on enhancing robustness using improved image processing techniques.Anotherpossibleimprovementistheintegration of machine learning or deep learning models for gesture recognition.Thiscanhelpthesystemadapttodifferentusers and support a wider range of dynamic and personalized gestures.Thesystemcanalsobeextendedtosupportmultihand detection and more complex gesture combinations, which would enable advanced functionalities. In addition, optimizing the system for higher frame rates and lower latency can further improve real-time performance, especiallyforfasthandmovements. The proposedsystem also has strong potential applications in the healthcare domain, where touch-free interaction can help reduce contamination risks. It can also be integrated with virtual reality and augmented reality systems to provide more immersive interaction. Furthermore, the system can be applied in areas such as gaming, robotics, and assistive technologies for individuals with physical disabilities. In future, deploying the system on mobile and embedded

p-ISSN:2395-0072

platforms can improve portability and make it more accessible.Withtheseenhancements,thesystemcanevolve into a more efficient and scalable solution for nextgenerationhuman-computerinteraction.

8.REFERENCES

[1]N.SharmaandA.Gupta,“ARealTimeAirMouseUsing VideoProcessing,”InternationalJournalofAdvancedScience andTechnology,vol.29,pp.4635–4646,2020.

[2]P.MishraandK.Sarawadekar,“FingertipsDetectionin EgocentricVideoFramesUsingDeepNeuralNetworks,”in Proc. International Conference on Image and Vision ComputingNewZealand(IVCNZ),2019,pp.1–6.

[3] S. Shriram, B. Nagaraj, J. Jaya, S. Shankar, and P. Ajay, “DeepLearningBasedReal TimeAIVirtualMouse System UsingComputerVision,”JournalofHealthcareEngineering, 2021.

[4]“Virtual Mouse Using MediaPipe and OpenCV,” InternationalJournalofResearchPublicationandReviews, 2024.

[5]“RealTimeHandGestureControlledVirtualMouseusing MediaPipe,”InternationalJournalofAdvancedScienceand ResearchManagement,2023.

[6] R. Matlani, R. Dadlani, S. Dumbre, S. Mishra, and A. Tewari, “Virtual Mouse using Hand Gestures,” in Proc. International Conference on Technological Advancements andInnovations(ICTAI),2021,pp.340–345.

[7]“Gesture Controlled Virtual Mouse with Voice Automation,”InternationalJournalofEngineeringResearch andTechnology,2023.

[8] T. Grzejszczak, R. Molle, and R. Roth, “Tracking of Dynamic Gesture Fingertips Position in Video Sequence,” ArchivesofControlSciences,2020.

[9]A.Kumar,A.Chaudhary,A.Singhal,A.Dev,andA.Jaiswal, “GestureandVoiceControlledVirtualMouse:AReview,”in Proc.InternationalConferenceonDisruptiveTechnologies (ICDT),2024,pp.876–879.

[10] G. Sung et al., “On Device Real Time Hand Gesture Recognition,”arXiv,2021.

[11]A.Morajkar,A.M.James,M.Bagwe,A.S.James,andA. Pavate, “Hand Gesture and Voice Controlled Mouse,” AdvancedEngineeringDays,vol.6,pp.127–131,2023.

[12] S. R. Chowdhury, S. Pathak, and M. D. A. Praveena, “GestureRecognitionBasedVirtualMouseandKeyboard,”in

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:13Issue:04|Apr2026 www.irjet.net p-ISSN:2395-0072

Proc.InternationalConferenceonTrendsinElectronicsand Informatics(ICOEI),2020,pp.585–589.

[13] V. V. Reddy, T. Dhyanchand, G. V. Krishna, and S. Maheshwaram,“VirtualMouseControlUsingColoredFinger TipsandHandGestureRecognition,”inProc.IEEEHYDCON, 2020,pp.1–5.

[14]R.Titlee,A.U.Rahman,H.U.Zaman,andH.A.Rahman, “ANovel Design ofan Intangible Hand GestureControlled ComputerMouseUsingVisionBasedImageProcessing,”in Proc.InternationalConferenceonElectricalInformationand CommunicationTechnology(EICT),2017,pp.1–4.

[15]F.Zhangetal.,“MediaPipeHands:OnDeviceRealTime HandTracking,”arXiv,2020.

[16] K. H. Shibly, S. K. Dey, M. A. Islam, and S. I. Showrav, “Design and Development of Hand Gesture Based Virtual Mouse,” in Proc. International Conference on Advances in Science,Engineering andRobotics Technology(ICASERT), 2019,pp.1–5.

[17]J.Kotti,B.Padmaja,andD.Deepa,“EnhancingGesture Controlled Virtual Mouse and Virtual Keyboard Using AI Techniques,”JournalofMobileMultimedia,vol.20,no.2,pp. 473–493,2024.

[18] V. V. Reddy, T. Dhyanchand, G. V. Krishna, and S. Maheshwaram,“VirtualMouseControlUsingColoredFinger TipsandHandGestureRecognition,”IEEEHYDCON,2020.

[19] D. Kalaivani et al., “Enhancing Accessibility Through Gesture Based Human Computer Interaction: A Virtual Mouse Approach,” in Proc. ICDSMLA, Lecture Notes in ElectricalEngineering,vol.1528,Springer,2026.

[20] D. Sanka et al., “AI Virtual Mouse: Revolutionizing HumanComputerInteraction,”inIntelligentCommunication, Control and Devices, Lecture Notes in Networks and Systems,vol.1164,Springer,2025.