International Research Journal of Engineering and Technology (IRJET)
e-ISSN: 2395-0056
Volume: 12 Issue: 12 | Dec 2025
p-ISSN: 2395-0072
www.irjet.net
Visual Speech Recognition through Lip Movements Shivam Singh1, Sahil Kumar2, Swati Mishra3, Sourabh Patel4, Prof. Rajeev Raghuwanshi5 1Student, Department of CSE-AIML, Oriental Institute of Science and Technology, Bhopal, India 2Student, Department of CSE-AIML, Oriental Institute of Science and Technology, Bhopal, India 3Student, Department of CSE-AIML, Oriental Institute of Science and Technology, Bhopal, India
4Student, Department of CSE-AIML, Oriental Institute of Science and Technology, Bhopal, India 5Professor, Department of CSE-AIML, Oriental Institute of Science and Technology, Bhopal, India ---------------------------------------------------------------------***---------------------------------------------------------------------
Abstract – Visual speech recognition, commonly known as
of a speaker’s mouth region. Despite its practical importance, visual speech recognition remains a challenging task due to several factors, including subtle and rapid lip movements, similarities between visual speech units (visemes), speaker variability, lighting conditions, and the absence of explicit boundaries between spoken characters or words.
lip reading, aims to interpret spoken language by analyzing movements of the lips without relying on acoustic signals. This capability becomes particularly important in noisy environments and for hearing-impaired individuals, where audio-based speech recognition systems often fail. Recent advances in deep learning have significantly improved the feasibility of end-to-end visual speech recognition by enabling models to learn both spatial and temporal patterns directly from video data. This paper presents an end-to-end sentencelevel lip-reading system based on the LipNet architecture. The proposed approach directly maps sequences of lip-region video frames to corresponding text sentences without requiring handcrafted visual features or explicit frame-to-character alignment. The model integrates spatiotemporal convolutional neural networks for visual feature extraction, bidirectional recurrent layers for temporal sequence modeling, and Connectionist Temporal Classification for alignment-free decoding. By jointly optimizing all components, the system effectively captures complex lip motion dynamics and longterm contextual dependencies. The proposed framework demonstrates the potential of deep learning-based visual speech recognition systems to achieve accurate and robust performance in real-world scenarios.
Early approaches to lip reading relied on handcrafted visual features such as lip contours, geometric shape descriptors, and motion-based parameters, which were subsequently classified using traditional machine learning techniques like Hidden Markov Models or Support Vector Machines. Although these methods achieved limited success, they typically required complex preprocessing pipelines and precise temporal alignment between visual frames and speech units. As a result, their performance and generalization capability were often constrained. Recent advances in deep learning have significantly transformed the field of visual speech recognition. Convolutional neural networks have demonstrated strong capabilities in learning discriminative visual features directly from raw images, while recurrent neural networks are effective in modeling temporal dependencies within sequential data. Building upon these developments, end-toend architectures have emerged that jointly learn feature extraction and sequence modeling without the need for manual intervention.
Key Words: Lip Reading, Visual Speech Recognition, Deep Learning, LipNet, Spatiotemporal CNN, Bi-GRU, CTC Loss.
LipNet represents a major advancement in this direction by introducing a fully end-to-end deep learning framework for sentence-level lip reading. The model integrates spatiotemporal convolutional neural networks to capture both spatial and temporal characteristics of lip movements, bidirectional recurrent networks to model long-range contextual dependencies, and Connectionist Temporal Classification loss to enable training without explicit frameto-character alignment. This unified design allows the system to directly map variable-length video sequences to textual sentences.
1. INTRODUCTION Speech perception in humans is inherently multimodal, relying on both auditory input and visual cues from a speaker’s facial movements. Among these visual cues, lip movements play a critical role in enhancing speech comprehension, particularly in noisy environments or situations where audio signals are degraded or unavailable. For individuals with hearing impairments, lip reading serves as a primary means of understanding spoken communication. These observations have motivated extensive research into automatic lip reading, also referred to as visual speech recognition (VSR).
In this paper, we focus on the design and explanation of a LipNet-based visual speech recognition system. The proposed approach highlights the effectiveness of deep learning techniques for decoding speech from visual input alone and demonstrates their potential for applications in
Automatic lip reading aims to transcribe spoken language by analyzing visual information extracted from video sequences
© 2025, IRJET
|
Impact Factor value: 8.315
|
ISO 9001:2008 Certified Journal
|
Page 1022