International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 05 | May 2025
p-ISSN: 2395-0072
www.irjet.net
Smart Hands-Free Interaction System Using Eye Tracking and Whisper AI for Multimodal Human-Computer Interaction KEERTHI P1, MONICA M2, SIVARANJANI S3, SOWMIYA P4 1,2,3,4Computer Science and Engineering ,Government College of Engineering Srirangam,
Trichy ,Tamil Nadu -620 012
5Asst.Prof Bhuvaneswari P , Dept. of Computer Science and Engineering, Government college of Engineering
Srirangam, Tamil Nadu, India ---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract - Human-Computer Interaction (HCI) is becoming
Eye-tracking technology has arisen as a significant tool for sidestepping classical input constraints. Through the detection and interpretation of the direction of a user's gaze, it is made possible to direct cursor movement on a screen without physical contact so that users can accomplish pointing tasks by merely glancing at various regions of the display. Also, blink detection can be used to simulate mouse clicks, which eliminates the requirement for physical mouse inputs. Although eye-tracking systems are very effective for cursor control, they frequently face limitations when performing more complicated tasks or giving commands.
more intuitive, intelligent, and accessible by doing away with physical input devices. This project puts forward a completely hands-free HCI system that incorporates Blue Eyes Technology (BET) along with Computer Vision and Voice Recognition for facilitating natural interaction for users, particularly those who have physical impairments. The system utilizes MediaPipe, FaceMesh for real-time eye tracking and blink detection, supporting precise gaze-based cursor movement and click simulation using Convolutional Neural Networks (CNN) and binary classification. A regression model is used for estimating the gaze direction accurately. Concurrently, Whisper AI, a powerful transformer-based voice recognition model, is used for voice command transcription, enabling users to perform system-level operations like application control, web browsing, and file manipulation through voice alone. This multimodal system—incorporating visual and auditory feedback—provides effortless interaction with computing environments without the need for hand gestures. The architecture is lightweight and deployable on low-resource devices, enhancing accessibility on both personal and professional levels. The solution proposed is improving HCI by offering a more intelligent, adaptive, and inclusive interface.
In order to provide a solution for such limitations, our project puts forth voice recognition as an additional modality. By pairing speech-based command processing with gaze-based cursor control, the system empowers users to execute a broad variety of activities—ranging from launching applications, browsing, and interacting with system components—to being completely hands-free. This multimodal HCI system utilizes Blue Eyes Technology, MediaPipe FaceMesh for facial landmark detection, and Whisper AI, a transformer-based speech recognition model, to provide an accessible and intuitive interface. The system is not only geared to improve the accessibility of the physically impaired, but also experts, teachers, and anyone engaged in hands-occupied environments requiring an effective hands-free interface. For the physically disabled, people with amputation or restricted motion in limbs, the system is a useful addition to conventional input devices, facilitating their interaction with digital systems with greater inclusiveness and ease. In addition, professionals in active working environments—e.g., surgeons, lab technicians, or field operatives—may find value in the system's capacity to execute intricate operations without pauses in their work processes. Teachers, for instance, may manage presentations or educational resources without physically being in contact with their computers, creating a better learning environment while staying involved with students.
Key Words: Human-Computer Interaction, Blue Eyes Technology, Eye Tracking, Blink Detection, Voice Recognition, MediaPipe, Whisper AI, CNN, Gaze Estimation, Hands-Free System.
1.INTRODUCTION Human-Computer Interaction (HCI) is central to filling the gap between humans and machines, facilitating easy communication and task performance. Conventional HCI approaches are based on physical input devices like keyboards and mice, which are not always within everyone's reach, especially those with physical disabilities like amputees. This necessitates the need for alternative interaction models that provide equal usability and accessibility. To fill this void, cutting-edge technologies such as Blue Eyes Technology, computer vision, and voice recognition provide exciting possibilities, allowing people to engage with digital systems more naturally and automatically.
© 2025, IRJET
|
Impact Factor value: 8.315
With the combination of eye-tracking and voice recognition, the system generates a smooth and smart HCI interface that enables users to interact with and control digital spaces with ease. The integration minimizes cognitive and physical load, which decreases mentally demanding tasks and makes them
|
ISO 9001:2008 Certified Journal
|
Page 17