Virtual Mouse Using Hand Gesture with Voice Assistant

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

Virtual Mouse Using Hand Gesture with Voice Assistant

Arya Prashant Ghadge, Tanvi Prasad Chavan, Prof. Mohan Bonde

1,2 Dept. of Data Science and EngineeringUsha Mittal Institute of TechnologyMumbai,India

Assistant Prof Dept. of Data Science and EngineeringUsha Mittal Institute of TechnologyMumbai,India

ABSTRACT-Human Computer Interaction (HCI) is evolving toward more natural and contactless interaction methods to improve usability, accessibility, and safety. Traditional input devices like mouse and keyboard require continuous physical contact, which may not be suitable in certainenvironmentsorforuserswithphysicallimitations.

This paper proposes a multimodal system, Virtual Mouse UsingHandGesturewithVoiceAssistantthatenableshandsfree computer control using real-time hand gesture recognition and voice commands. The system uses MediaPipeandOpenCVtodetecthandmovementsforcursor control, clicking, and system operations, while a voice assistant allows users to perform tasks such as opening applicationsandsearchinginformation.

By integrating gesture and voice interaction into a unified framework, the system ensures efficient and seamless operation. The proposed solution is cost-effective, easy to implement, and suitable for practical real-world applications.

Keywords – virtual mouse, hand gesture recognition, voice assistant, human computer interaction, mediapipe, opencv, multimodal interface, accessibility.

I. INTRODUCTION

Human Computer Interaction is central to the design of contemporary computing systems, due to its direct influenceonhowefficientlyandcomfortablyusersinteract with digital environments. While the classical mouse and keyboardarestillthemostpopularinputdevices,theyalso involvecontinuousphysicalcontactandfinegrainedmotor action.Sucharequirementislimitingforpatientswhoare motor impaired and may be less appropriate in situations (hospitals, laboratories, public kiosks) where maintaining physicalhygieneisparamount.

Latest developments in computer vision and speech processing have made it possible to consider alternative interaction paradigms based on hand gestures or voice commands. Gestures provide an intuitive way to manipulate visual elements, while voice interfaces allow users to give high-level commands or access information. Mostavailablesolutions,however,treatgesture-basedand

voice-based interaction as separate modalities, limiting their real-world applicability. A multimodal interaction mechanism can address this limitation by integrating continuous gesture control with command-driven voice interaction.

In this paper, a consolidated multimodal system called Virtual Mouse Using Hand Gesture with Voice Assistant is introduced. The system enables real-time cursor manipulation through hand movements while also allowing system-level and application-level operations throughsynchronizedvoicecommands.

II. RELATED WORK

A lot of studies look at how gestures can control virtual mice, some relying on older image processing tools while others apply newer machine learning ideas.[4][5] Lately, using MediaPipe for hand landmark detection has gained attention due to its efficiency and ability to track hands accurately in real time on everyday hardware.[1] Still, certain projectsrelyondifferentdeeplearning setupslike CNNs or RNNs to recognize moving hand patterns.[1][4] Even so, ways like these often lead to sharper recognition results.Still,theytendtodrainsystemresources.[1]

Nowhere is the shift clearer than in tools shaped by talk and touch. Instead of relying on just one method, people explorecombinationslikemovinghandswhilespeakingto createsmootherexperiences.[2][3]Yethere'sacatch,most setupstodaystilldependonbulkyhardwaresuchasdepth cameras.[4]Thatgearaddsexpenseandmakesinstallation tougher.So,whileearlyversionswork,theyoftencomeata steeppriceinsetuphassle.

What makes the suggested system different is how it combines gestures and voice control within one smooth interface,allpoweredbyanordinarywebcam.Ratherthan keeping these inputs apart, it brings them together into a unified experience that feels natural and immediate. A broad set of hand movements allows fine-grained interaction, while voice commands are understood to trigger more complex tasks. At its core, a smart coordination system keeps everything aligned so both inputs behave as one logical input. This setup blocks

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net

clashing commands while keeping hand gestures and spoken control working together making things feel smoother,morehuman.

III. PROPOSED SYSTEM

A. SYSTEM OVERVIEW

Theproposedsystemconsistsofthreemainmodules:

1.Handgesturerecognitionbasedvirtualmousemodule

2.Voiceassistantandcommandinterpretationmodule

3.Controlandcoordinationmodule

A camera continuously captures live input to track hand movements and perform cursor operations such as movement, clicking, and scrolling. Simultaneously, voice commandsareprocessedtoperformtaskssuchasopening applications and browsing. A centralized coordination system ensures synchronization between gesture and voice inputs, preventing conflicting actions and enabling seamlessinteraction.

B. System Architecture

TheproposedVirtualmouseusinghandgesturewithvoice assistantfollowsamodularlayeredarchitecturetosupport real-timeprocessingandmultimodalinteraction.

1. User Interface Layer:

The user interface provides a graphical interface that displays user commands, assistant responses, and system status. Commands are forwarded to the Python backend using Eel functions, acting as a communication bridge betweentheuserandthesystem.

2. Interaction Processing Layer:

Thislayerconsistsoftwomajorsubsystems.

a) Hand Gesture Processing Subsystem

The system captures video frames using OpenCV and processesthemusingMediaPipetodetecthandlandmarks. A geometric approach is used to recognize gestures based

p-ISSN: 2395-0072

on landmark positions. A stabilization mechanism is appliedtoreducenoiseandimproveaccuracy.

b) Command Interpretation Subsystem

The system processes user commands using rule-based matching. It supports application control, web navigation, conversational responses, and system operations. Knowledge queries are handled through a lightweight interface.

3. Control and Coordination Layer:

This layer manages synchronization between gesture and voice inputs using a shared execution flag. It controls the activation of the gesture engine and ensures that actions areexecutedonlywhenenabled.

4. Execution Layer:

This layer performs system actions such as cursor movement, mouse operations, drag-and-drop, system control, application launching, web browsing, and keyboardautomation.Itensurespropermappingbetween recognizedinputsandsystemresponses

IV. Research Methodology

This section is about how we designed and implemented themultimodalinteractionframeworkthatweproposed.

A. Software Libraries and Tools:

We used the following libraries and tools to make this work:

• OpenCVtocapturevideoandprocessframes

• MediaPipetodetecthandlandmarksintime

• PyAutoGUI to automate the mouse and control the cursor.

•NumPyandmathfunctionstodogeometriccalculations.

•pyttsx3togivetext-to-speechfeedback

•Eel framework to make the user interface and communicatewiththebackend

•WikipediaAPItoanswerknowledge-basedquestions.

•OS and web browser libraries to do system and web things

•Multithreadingtorunmodulesatthesametime

B. HAND GESTURE RECOGNITION METHOD:

The hand gesture recogniton module enables users to interact with the system using hand movements. A webcam continuously captures live video frames, which are processed using OpenCV. The frames are flipped horizontally for natural interaction and converted to RGB formatforprocessingusingMediaPipeHands.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net

MediaPipe detects hand landmarks in real time and identifies the presence of one or two hands. A geometric analysis is applied to recognize gestures based on the relativepositionsanddistancesbetweenlandmarks.

To improve accuracy and reduce false detections, a stabilization mechanism is applied where a gesture is considered valid only if it is consistently detected across multipleframes.

The system supports the following operations:

•Cursormoment.

•Leftclick

•Rightclick

•Doubleclick

•Draganddrop.

•Volumecontrol.

•Brightnesscontrol

Each recognized gesture is mapped to a corresponding system action using automation functions. A neutral hand stateismaintainedtopreventunintendedoperations.

C. Voice Assistant and Command Processing Method:

The voice assistant operates as a text-based command processing system that receives user input through a graphicalchatbotinterfaceandprocessesitusingaPython backendviatheEelframework.

Commands are parsed using a rule-based intent recognition approach. Each input is preprocessed and matched with predefined patterns to determine the appropriateaction.

The assistant supports multiple categories of operations, including system control, application launching, web navigation, and information retrieval. Once a command is recognized, the corresponding task is executed, and feedbackisprovidedusingtext-to-speechsynthesis.

D. Mechanism of Multimodal Coordination:

Thereisa sharedexecution statevariable,whichindicates whether the gesture recognition engine is running. This is controlled by a central coordination mechanism issuing commandsthroughtheuserinterfaceofthechatbot.

The gesture controller checks whether it is executing beforeeveryframeandwillnotperformanycursoraction if gesture recognition is disabled. This technique allows switchingbetweeninteractionmodeswithoutaccidentally movingwhenswitchingthemode.

E. WORKING:

The Virtual Mouse Using Hand Gesture with Voice Assistant system operates in real time by processing both hand gestures and user commands simultaneously. The system integrates gesture recognition and command

p-ISSN: 2395-0072 © 2026, IRJET | Impact Factor value: 8.315 | ISO 9001:2008

processingtoperform systemoperationswith lowlatency andsmoothinteraction

1 Hand Gesture Recognition Working:

The system captures live video input using a webcam and detects hand movements using MediaPipe. Hand landmarksaretrackedandanalyzedtorecognizegestures, which are mapped to corresponding system actions such as cursor movement, clicking, and scrolling. The system ensures real-time response by continuously processing framesandexecutingactionswithoutdelay

1.1 Landmark extraction and hand detection:

VideoframesarecapturedandprocessedusingMediaPipe Hands to detect hand landmarks. The system identifies multiple landmark points and determines hand orientation, enabling accurate tracking using a standard RGBcamerawithoutrequiringspecializedhardware.

1.2. Gesture Classification:

Gestures are recognized using geometric relationships between hand landmarks, including distances and relative positions. To improve reliability, a gesture is confirmed only when it is consistently detected across multiple frames, reducing the effect of noise and unintended movements

1.3. Action Mapping and Execution:

Recognized gestures are mapped to predefined system actions using automation functions. The system performs cursor control, clicking, dragging, and system adjustments such as volume and brightness. Execution occurs only whenthegesturesystemisactive,ensuringcontrolledand stableinteraction

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net

2. Voice Assistant Working:

The voice assistant works alongside gesture control by processing user commands through a chatbot interface. Commands are sent to the backend, preprocessed, and matched using rule-based logic. Based on the identified intent, the system performs actions such as application control, web navigation, and information retrieval. Responses are generated using text-to-speech and displayedontheinterfacetoprovidefeedbacktotheuser

3. Dataset Description:

p-ISSN: 2395-0072

Our proposed system does not rely on custom collected dataset as it utilizes pre trained MediaPipe Hands model. Theunderlyingtrainingdatasetisalarge-scaleproprietary dataset composed of real-world and synthetic hand images.Thedatasetcoversdiversehandposes,skintones, lightingconditionsandbackgrounds.

Eachhandsampleisannotatedwithtwenty-onelandmark points representing the wrist and finger joints. Synthetic handimagesgeneratedusingthree-dimensionalmodelling are included to improve robustness against occlusion, extremeposesandperspectivevariations.

Data augmentation techniques such as rotation, scaling, flipping and illumination adjustment are applied during training of the model. The availability of a robust pretrained model allows the proposed system to perform accurate real-time hand tracking without collecting or trainingacustomdataset.

V. IMPLEMENTATION

Thissectiondescribestheimplementationoftheproposed systemandhowthearchitectureisrealizedinpractice.

A. Backend Framework and Execution Environment:

The entire system is implemented in Python and executed on a Windows platform. Independent modules handle gesture recognition, assistant logic and user interface communication. Multithreading enables concurrent executionofthegestureengineandchatbotinterface.

B. Graphical User Interface Integration:

The graphical interface is embedded into the application using the Eel framework. The backend exposes Python functions to the interface, allowing bidirectional communication.

User messages are added to the chat window and processedbythebackend.Systemresponsesandexecution feedbackaredisplayedinrealtime

C. Voice Assistant and Command Handling Implementation:

Theassistantoperatesasatext-basedintelligentcommand processor. Incoming commands are stored in a queue and processedsequentially.

Each command is analyzed using rule-based matching. Supported commands include greeting, assistant identification, date and time queries, web access, application launching, gesture engine control, copy and pasteoperationsandprogramtermination.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

Spoken feedback is generated using an offline text-tospeechengine.

D. Gesture Recognition Engine Implementation:

The gesture recognition engine runs as an independent thread. It continuously captures frames from the webcam andprocessesthemusingMediaPipe.

For each detected hand, landmark positions are extracted and passed to the gesture recognition logic. The system supportsbothsingle-handanddual-handinteractionusing majorandminorhandidentification.

E. Cursor Control and Motion Stabilization:

Cursorpositioniscomputedusingastablelandmarkpoint onthehand.Adampeningmechanismisappliedusing the difference between current and previous positions to reducejitterandsuddencursorjumps.Thescalingfactoris adjusteddynamicallytoensuresmoothmotion.

F. Gesture-to-Action Mapping:

Thefollowinggesturemappingsareimplemented:

•V-shapedfingerpostureforcursormovement.

•Fistgesturefordrag-and-drop.

•Two-fingerclosedgesturefordoubleclick

•Indexfingergestureforrightclick

•Middlefingergestureforleftclick

• Pinch gesture using the major hand for system volume control.

•Pinchgestureusingthemajorhandforscreenbrightness control

Multiple item selection is supported through continuous drag-basedinteraction.

G. System Brightness and Volume Control:

System brightness is controlled using a dedicated brightness control interface. Pinch displacement magnitude is converted into incremental brightness changes.

Systemvolumecontrolisimplementedusingtheoperating system audio endpoint interface. The volume level is updatedsmoothlybasedonpinchgesturemovement.

H. Application and System Control Implementation:

The assistant supports direct execution of system applications through operating system calls. The supportedapplicationsinclude:

•Notepad.

•Calculator.

•CommandPrompt

•MicrosoftWord

•MicrosoftExcel

•WhatsApp

Webbrowsingandsearchoperationsareexecutedthrough thedefaultsystembrowser.

I. Keyboard Automation:

Keyboardautomationisimplementedtosupport copyand paste operations using simulated control key combinations. This enables text manipulation without physicalinputdevices.

J. Concurrent Execution and Thread Management:

The chatbot interface and gesture recognition engine operate in separate threads. A shared execution flag ensuresthatthegestureenginecanbestartedandstopped safelywithoutinterruptingthechatbotprocess. This design improves responsiveness and ensures stable systembehaviour.

VI. RESULTS AND DISCUSSION

1. Testing and Validation :

Theproposedsystemwastestedundervaryingconditions, including different lighting environments, backgrounds, and hand positions, to evaluate its performance and adaptability. The gesture recognition module demonstrated smooth cursor movement and accurate detection of actions such as clicking and dragging under stable lighting conditions. However, slight accuracy degradationwasobservedinlow-lightconditionsorwhen handocclusionoccurred.

The voice assistant module showed reliable performance withfastresponsetimesandaccuratecommandexecution. The integration of gesture and voice inputs ensured smooth interaction, even when one modality was temporarily unavailable.The system was further evaluated based on responsiveness and consistency during continuous usage. Gesture-based actions exhibited low latency, enabling near real-time interaction, while command processing remained efficient. The system maintained stable performance over extended periods, indicating its reliability. Additionally, the multimodal approach improved usability, flexibility, and user comfort byreducingdependencyonphysicalinputdevices.

© 2026, IRJET | Impact Factor value: 8.315 | ISO 9001:2008 Certified Journal | Page300

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net

2. Results:

The results demonstrate that the proposed system achieves low-latency and reliable real-time interaction. The gesture recognition module provides accurate cursor control and stable execution of actions, while voice commands enable efficient system-level operations withoutinterruptinggestureinput.

Theintegrationofgestureandvoiceinteraction,supported by a coordination mechanism, prevents conflicting actions and ensures organized execution. The system operates effectively on standard hardware with low computational requirements,makingitcost-effectiveandpractical. Although the system performs well under normal lighting conditions,itsperformanceisaffectedinlow-lightornoisy

p-ISSN: 2395-0072

environments,indicatingscopeforfurtherimprovementin robustness.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

VII. LIMITATIONS

There are shortcomings to the proposed approach that provide room for improvement: the current gesture recognition module is constrained to the finite set of predetermined hand gestures that limit the range of possible interactions for the user. Environmental conditions, such as poor lighting or complex background environments, have been reported to affect system performance, as they impact gesture detection accuracy andreliability.

VIII. CONCLUSION

This research presented a virtual mouse using hand gestureswithavoiceassistantwhichaimedatmultimodal human-computer interaction system that allows full interaction with a computer without physical contact and combinesareal-timehandgesturerecognitionenginewith asmartvoice-basedassistant.Theinterfaceusesagesture recognition engine based on a webcam and a conversationalassistant.

Usingacentralizedcoordinatortosynchronizegestureand voice input and resolve conflicts, this system has been shown to exhibit stable, low-latency performance, and provide,gestureandvoice,inputasareliable,low-costand accessible alternative to desktop input mechanisms such asmiceandkeyboards.

In sum, the system has proven to be a solid platform for future clever interaction systems that are able to exploit visual and auditory information in a lightweight manner usingoff-the-shelfhardware.

Future iterations of the system could potentially be made smarter and more dynamic by using a machine learning based adaptive gesture learning that allows the system to learn user specific gestures and improve the system accuracy in the process. Expanding the voice assistant to support multiple languages and personalized commands wouldconsiderablyincreaseitsusabilityfordiverseusers.

Future extensions of this work could include multi-user gesturetrackingformulti-userinterfaces,theuseofclever personal assistantsandsmartdevices,andtheapplication of the system to IoT-based systems for home automation, assistive technologies and smart work environments. This would improve the robustness, scalability and usability of thesystemindifferentapplicationdomains.

ACKNOWLEDGMENT

Theauthorswouldliketoexpresstheirsinceregratitudeto the Department of Data Science, Usha Mittal Institute of Technology and SNDT Women’s University for providing

thenecessaryfacilitiesandacademicsupport.Theauthors alsothanktheirprojectguideforcontinuousguidanceand encouragementduringthedevelopmentofthiswork.

REFERENCES

[1] O.-J. Yaseen et al., “Next-Gen Dynamic Hand GestureRecognition:MediaPipe,Inception-v3and LSTM-Based Enhanced Deep Learning Model,” Electronics,2024.

[2] Y. Bawa et al., “Virtual Mouse using Gesture Recognition and Voice Control,” IRE Journals, 2024.

[3] K. Kavyasree et al., “Hand Glide: GestureControlled Virtual Mouse with Voice Assistant,” IJRASET,2024.

[4] M. R. Visavarapu et al., “Gesture Control Mouse,” EasyChairPreprint,2024.

[5] K. A. Jacob et al., “Gesture Recognition Based Virtual Mouse Using OpenCV and Python,” AIP ConferenceProceedings,2024.

[6] Google, “MediaPipe Hands,” Google Developers, 2023.

[7] F. Chollet, “Deep Learning with Python,” Manning Publications,2018.

[8] D. Zhang, “Hand Gesture Recognition Using Computer Vision,” IEEE Transactions on Pattern AnalysisandMachineIntelligence,2021.

[9] S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics,2007.

[10] R. W. Picard, “Affective Computing,” MIT Press, 1997.

[11] Kendon, “Gesture: Visible Action as Utterance,” CambridgeUniversityPress,2004.

[12] T. Starner and A. Pentland, “Real-Time American Sign Language Recognition from Video Using HiddenMarkovModels,”MITMediaLab,1995.