Lyra: A Multimodal Intelligent Desktop Interaction System

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072

Lyra: A Multimodal Intelligent Desktop Interaction System

Siya Vaity1 , Prachi Das2 , Shrot Maurya3,Manthan Bid4

1 Artificial Intelligence and Machine Learning, Viva Institute Of Technology

2 Artificial Intelligence and Machine Learning, Viva Institute Of Technology

3 Artificial Intelligence and Machine Learning, Viva Institute Of Technology

4Artificial Intelligence and Machine Learning, Viva Institute Of Technology

Abstract - Human-computer interface is taking a new directiontowards morenaturalandtouchlessinterface.Inthis paper, the author describes the Lyra, the intelligent desktop assistant that combines the voice recognition and real-time handgesturecontroltoallowoperatingthecomputerwithout hands.Thesystemenablesuserstoopenapplications,browse, handle files, type, manage windows and carry out mouse actions by voice command and gesture detected by the webcams. Lyra is a blend of speech to speech processing, command interpretation, computer vision and system automationasameanstodevelopaneffectiveandconvenient interaction model. The suggested system will enhance the productivity, user ease of access due to their physical disabilities, and facilitate hygienic touchless computing conditions. Real world experimental use has shown to have smooth cursor control, command recognition and low system response

Key Words: Voice Assistant, Gesture Recognition, Human–Computer Interaction, Computer Vision, Automation, AI Assistant

1. INTRODUCTION

Traditionalcomputerinterfacereliesonkeyboards andpointers.Thesetoolsareeffectivebutconstrainthe naturalcommunicationprocessandtheymightnotfit intheaccessibility-orientedortouchlessenvironments.

The recent developments in the field of Artificial Intelligence (AI), speech recognition, and computer vision have allowed more intuitive interaction mechanisms.ThispaperpresentstheconceptofLyra,a multi-modal AI assistant integrating the use of voice commands with the use of gestures to manage a desktop system. This is aimed at offering an uninterrupted, smart and natural interface that minimizeshardwareinputdevicephysicaladdiction.

Thekeycontributionsofthisworkare:

 Desktopautomationthroughvoice.

 A hand gesture control mouse and window controlsysteminreal-time.

 Combinationofspeechandvisionmodulesto oneassistant.



Safety and stability systems to avoid unintentionalactionsofthesystem.

1.1 Voice Assistant Module

Thismoduleworkswithspeechrecognitionandautomation ofthesystem.

Components:

 Speechrecognitionengine

 Matchingandinterpretationunitofcommand.

 Weblauncherandapplication.

 Searchmoduleoffilesandfolders.

 Automatedtypingcontroller.

 Filtertoavoidinterferenceofcriticalsystems.

1.2 Gesture Control Module

Hand tracking and gesture recognition This module is a camera-basedapplicationthattrackshandsandrecognizes gestures.

Components:

 Webcamcapturesystem

 Handlandmarkidentificationalgorithm.

 Logicofgestureclassification.

 Layersofwebsocketcommunication.

 SystemactionexecutorusingPython Thetwomodulescommunicatewiththeoperatingsystem toruncommandon-the-fly

2. Voice Assistant Functionalities

2.1 Wake and Sleep Mechanism

Lyra starts with preprogrammed wake words and sleeps afternotbeingused,whichmaximizesconsumption.

2.2 Website and Application Control.

Userscanalsoopentheinstalledapplicationsandwebsites throughthenaturalvoicecommands.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072

2.3 File and Folder Management

The assistant locates popular directories and opens requestedfilesorfolders.

2.4 Information Services

Lyrahastimeanddateinformationthatisinrealtime.

2.5 Web Search and Media Playback.

The voice commands start web searches and playback of musicprovidedthroughYouTube.

2.6 Automated Typing

Theassistanttypedwhatwasdictatedintoactivetextfields.

3. Gesture Control System

Thegesturesystemcanallowthefullcontrolofthemouse andwindowswithoutphysicaldevice.

3.1 Cursor Movement

Thepalmcentermappingandmotionsmoothingenablesthe cursortobecontrolledbyanopenpalmgesture.

3.2 Scrolling

Fingerpatternsenableoperatingfingerscrollingsmoothlyin anadaptivemannerbothuphillanddownhill.

3.3 Window Management

Themovementsofthewindowandsafewindowclosingare madepossiblebyspecifiedgestures.

3.4 Stability Enhancements

ExponentialMovingAverage(EMA)smoothing.

Dead-zonefiltering

Cursorfreezeduringclick Adaptivespeedcontrol

Themethodsminimizejitterandenhanceprecision.

4. Communication Framework.

Thegesturemoduleisworkingwith:

 HTTPServertoservegesturesinterface.

 WSServertotransmitreal-timedata.

 PythonBackendtodoactionattheOSlevel. Thisguaranteeslow-latencyandon-goingcommunication betweenparts.

5. Performance Optimizations

Itimprovesperformanceinthesystemby:

 Motionsmoothingalgorithms.

 Dynamicspeedscaling

 Autocalibration

 Camerawindowpersistence

 Portconflictprevention

Thesearedonetoprovidestabletrackingandhassle free controlofthesystem.

6. Future Scope

 Thefuturedevelopmentcanbe:

 Multi-languagesupport

 Offlinespeechrecognition

 AdaptivegesturelearningthatisbasedonAI.

 IoTintegration

 Commandcustomizationonapersonallevel.

The diagram represents the overall workflow and architecture of a smart laptop/desktop assistant that operatesusinghandgestures,voicecommands,andsystem controls. The system is divided into multiple functional layersthatworktogetherthroughacentralcontroller.

1. Startup and Initialization Phase

The system begins with a Startup Sequence, which initializesthecorecomponents.Thisisfollowedby:

 StartHTML Input– activates browser-based input interfaces

 Initialize Gesture Module – prepares gesture detectionmechanisms

 Activate Logical Core – enables voice processing andintentanalysis

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 02 | Feb 2026 www.irjet.net p-ISSN: 2395-0072

This phase ensures all modules are ready before user interactionbegins.

2. HTML Input Layer

The HTML Input Layer acts as the primary data acquisitionlayer.Itcollects:

 Webcamvideoinputfordetectinghandgestures

 Microphoneaudioinputforvoicecommands

Theseinputsarecontinuouslyfedintothesystemasraw inputdata.

3. Gesture Processing Module

The Gesture Processing unit interprets visual data receivedfromthewebcam.Itperforms:

 Hand tracking to detect hand position and movement

 Gesturerecognitiontoidentifypredefinedgestures

 Action command generation based on recognized gestures

Theresultinggestureactionsareforwardedtothecentral controller.

4. Logical Core

TheLogical Coreis responsibleforintelligentdecisionmaking.Itskeyfunctionsinclude:

 Voiceprocessing(speechrecognition)

 Intentanalysistounderstandusercommands

 System control logic to determine appropriate actions

 It sends command signals to the main controller afterprocessinguserintent.

5. UI / Main Controller (Central Hub)

TheUI/MainControlleractsastheheartofthesystem.It

 Receives command signals from both the Gesture ProcessingModuleandtheLogicalCore

 Providesassistantstatusandfeedbacktotheuser

 Controlsoperatingsystemfunctionssuchasmouse movement, keyboard input, and application management

This module coordinates all interactions and executes system-levelactions.

6. System Actions and Shutdown Process

Oncecommandsareexecuted:

SystemActionsareperformed(openingapps,controlling OS,etc.)

Ifrequired,thesystementersaShutdownProcess,which includes:

 Shutdownsequence

 Errorhandlingmechanismstomanageunexpected failures

This ensures a safe and controlled termination of the system.

7. CONCLUSIONS

Lyra shows how voice AI and gesture recognition can change the interaction at the desktop to a touchless experience. The system is a system that is based on the combinationofspeechautomation,computervisionandrealtime processing to form an intelligent human-computer interaction framework. Lyra has good prospects of being accessible,productiveaswellascomputingnext-generation environments.

ACKNOWLEDGEMENT

The author would also like to mention the help of academic materials and open-source technologies which helpedinthecreationoftheLyraassistantsystem.

REFERENCES

[1] S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics,vol.37,no.3,pp.311–324,May2007.

[2] Z. Zhang, “Hand Gesture Recognition Based on ComputerVision,”inProc.InternationalConferenceon Artificial Intelligence and Pattern Recognition, 2019, pp.45–50.

[3] R. Szeliski, Computer Vision: Algorithms and Applications,London,U.K.:Springer,2011.

[4] D. Jurafsky and J. H. Martin, Speech and Language Processing,3rded.,vol.1,PearsonSeriesinArtificial Intelligence,2020.

[5] T. Starner, “Wearable Computer Vision System for Gesture Recognition,” U.S. Patent 6 577 742, Jun. 10, 2003.