Sign Language Recognition using MediaPipe and Machine Learning

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 p-ISSN: 2395-0072

Volume:13 Issue: 03| Mar 2026 www.irjet.net

Sign Language Recognition using MediaPipe and Machine Learning

1Prof. Dr. B. Geetha Vani, 2K. Chandrakanth, 3S. Anusha, 4B. Harish

1Professor in Dept. of CSE, G. Pulla Reddy Engineering College, Kurnool, AndhraPradesh,India, 2,3,4Undergraduate student in Dept.ofCSE,G. Pulla ReddyEngineering College,Kurnool, Andhra Pradesh, India

Abstract - Sign language recognition systems play a vital role in improving communication for hearing and speechimpaired individuals; however, real-time gesture recognition is affected by variations in hand positioning, background noise, and lighting conditions. In this work, MediaPipe is used for hand landmark detection along with a Random Forest classifier for real-time gesture recognition. A workflow is designed to collect data, extract normalized landmarks, and train the model. The system is deployed through a web-based interface that converts hand signs representing alphabets (A–Z) and digits (0–9) into text with word and sentence formation. Experimental results show reliable performance and improved accuracy, demonstrating the suitability of the proposedsystem for assistive communication applications.

Key Words: Sign Language recognition, hand gesture detection, MediaPipe, Random Forest, real-time system, assistive communication

1. Introduction

Signlanguagerecognitionsystemsarewidelyusedtoassist communication for hearing and speech-impaired individuals in various real-world environments. However, real-time hand gesture recognition systems often suffer from variations in hand positioning, background noise, and poor lighting conditions. These limitations arise due to camera quality, environmental factors, and diverse hand orientations. As a result, the accuracy of gesture interpretation is frequently insufficient for reliable communication, highlighting the need for effective gesture recognitiontechniques.

Several approaches have been explored to improve the performance of gesture recognition systems, ranging from traditional image processing techniques to machine learningand deeplearning-based methods.Inrecentyears, handtrackingframeworkscombinedwithmachinelearning algorithms have shown significant improvements over conventional approaches. In particular, landmark-based detectionmethodshavegainedattentionduetotheirability to capture precise hand features. Prior studies have demonstrated the effectiveness of such approaches for recognizing hand gestures, while also highlighting challengesrelatedtodatasetvariability,lightingconditions, andcomputationalefficiency.

Recent advancements in hand tracking technologies have led to the development of efficient frameworks capable of detecting detailed hand landmarks in real time. Among these, MediaPipe has gained attention for its ability to accurately detect and track hand movements under varying conditions. Unlike traditional methods that rely heavily on raw images, MediaPipe extracts structured landmark features, making the system more robust andefficient.Motivatedbythesecapabilities,thisworkdevelopsa real-time sign language recognition system using MediaPipe for feature extraction and a Random Forest classifier for gesture classification. The system is deployed through a web-based interface, focusing on real-time performance and practical usabilityratherthancomplexmodelarchitectures.

2. Related Work

Sign language recognition and hand gesture detection have been extensively studied to improve communication systems for hearingandspeech-impairedindividuals.Earlyapproachesrelied on traditional image processing techniques such as skin color detection, contour extraction, and thresholding; however, these methodsoftenfailedtoaccuratelycapturecomplexhandgestures under varying lighting and background conditions. With the advancement of machine learning, landmark-based and feature extraction techniques have demonstrated significant improvements in gesture recognition tasks by learning patterns fromstructuredhandrepresentations.

More recently, deep learning-based methods such as ConvolutionalNeuralNetworks(CNNs)havebeenintroducedfor hand gesture recognition due to their ability to automatically learn spatial features from images. CNN-based models have shown promising results in recognizing complex gestures and improving classification accuracy. Several studies have explored the use of deep learning for sign language recognition, highlighting their effectiveness in handling large-scale datasets while also noting challenges related to high computational cost, large training data requirements, and real-time implementation constraints.

In the context of eal-time gesture recognition, existing works haveprimarilyfocusedontrainingcomplexdeeplearningmodels or using hybrid approaches combining image processing and neuralnetworks.

Volume:13 Issue: 03| Mar 2026 www.irjet.net

In contrast, this work emphasizes a lightweight and practical approach by utilizing MediaPipe for efficient handlandmarkdetectionandaRandomForestclassifier forgestureclassification.

While existing studies have demonstrated the effectiveness of of deep learning models for gesture recognition, most approaches focus on improving accuracy through complex architectures or large datasets. These methods often require significant computationalresourcesandarenotalwayssuitablefor real-time deployment. In contrast, the proposed work adopts an application-oriented approach by directly using normalized hand landmarks with a machine learning classifier, enabling efficient real-time recognitionwithoutheavycomputationalrequirements.

3. Methodology

Theoverallworkflowbeginswiththeacquisitionofhand gesture images captured using a webcam for different classes representing alphabets (A–Z) and digits (0–9). These images represent real-time conditions, including variations in hand positioning, background noise, and illumination differences.Sincethe data iscollected from real-world environments, it reflects practical challenges encounteredingesturerecognitionsystems.

After acquisition, each image undergoes processing using MediaPipe Hands before being passed to the classification model. The processing stage includes detection of hand landmarks and extraction of normalizedcoordinatefeaturesbysubtractingminimum x and y values. This normalization ensures consistency acrossdifferenthandpositionsandscales,improvingthe robustnessofthesystemundervaryingconditions.

The feature representation is constructed using the extracted hand landmarks, where each gesture is represented as a set of normalized (x, y) coordinates. These features are used to train a Random Forest classifier, which learns patterns corresponding to different gestures. The trained model is capable of performing real-time classification and predicting the corresponding alphabet or digit based on the detected handgesture.

To evaluate enhancement performance, the dataset is divided into training and testing sets using a standard split. The model performance is assessed using metrics such as accuracy, precision, recall, and F1-score. Additionally, the system is deployed through a webbased interface that enables real-time gesture recognition.

p-ISSN: 2395-0072

Figure - 2 : Architecture of Gesture Recognition

4.Experimental Setup

The experimental evaluation of the proposed sign language recognition system was conducted using a collection of hand gesture images captured through a webcam. The dataset was designed to reflect real-time conditions rather than controlled laboratory environments. A set of images was collected for each classrepresentingalphabets(A–Z)anddigits(0–9).The data collection process involved capturing multiple samples per class under different orientations and lightingconditionstoensurevariabilityandrobustness.

The dataset includes a wide range of variations such as different hand positions, background conditions, and illuminationlevels,whicharecommonlyencounteredin real-world scenarios. It also incorporates variations in gesture orientation and scale, which can affect recognitionaccuracy.

The model architecture utilizes a landmark-based feature extraction framework using MediaPipe Hands for real-time gesture detection. In this approach, the hand tracking module focuses on identifying 21 key landmark points representing finger joints and palm structure. These landmarks capture meaningful spatial relationships of the hand. The normalization processplaysacrucialroleinreducingvariationscausedbyhand position and scale differences, enabling consistent feature representationacrossdifferentinputs.

To further utilizes a landmark-based feature extraction framework using MediaPipe Hands for real-time gesture detection. In this approach, thehand tracking module focuses on identifying21keylandmarkpointsrepresentingfingerjointsand palm structure. These landmarks capture meaningful spatial relationships of the hand. The normalization process plays a crucial role in reducing variations caused by hand position and scale differences, enabling consistent feature representation acrossdifferentinputs.

Each gesture sample in the dataset is processed individually through the detection and classification pipeline. The system records several evaluation parameters including accuracy, precision, recall, and F1-score tomeasure the effectiveness ofthe recognition process. In addition toquantitative evaluation, realtime testing is performed using the web-based interface, where users interact with the system by performing gestures. This evaluation allows assessment of the model’s performance under practicalconditionsanddemonstratesitseffectivenesscompared totraditionalgesturerecognitionapproaches.

5. Evaluation Metrics

Dueto theabsence ofstandardized benchmark datasets for realtime hand gesture recognition under varying conditions, evaluation accuracy is computed using classification-based performancemetrics.Theeffectivenessoftheproposedsystemis assessed by comparingpredicted gesture labels withactual class labelsobtainedfromthedataset.

Accuracyiscalculatedastheratioofcorrectlypredictedgestures tothetotalnumberofsamples.Itprovidesanoverallmeasure of the model’s performance in correctly classifying hand gestures. However, accuracy alone may not fully represent performance when class distributions vary, making additional metrics necessaryforcomprehensiveevaluation.

Volume:13 Issue: 03| Mar 2026 www.irjet.net

Similarly, precision and recall are computed to measure the model’s ability to correctly identify gesture classes. Precision evaluates how many predicted gestures are actually correct, while recall measures how effectively themodelidentifiesallrelevantgestureinstances.These metrics provide insight into classification reliability underdifferentconditions.

In addition to accuracy, precision, and recall, the F1scoreiscalculatedastheharmonicmeanofprecisionand recall.Thismetricprovides a balancedevaluation ofthe model’s performance, especially when dealing with variations in gesture recognition. These evaluation metrics collectively enable effective assessment of the proposedsystemunderreal-timeconditions.

6. Result and Analysis

The experimental results demonstrate noticeable improvements in real-time hand gesture recognition accuracy and prediction consistency. The proposed system effectively identifies hand gestures representing alphabets and digits under varying conditions. Compared to traditional image-based approaches, the landmark-basedmethodprovidesstablepredictionsand reduceserrorscausedbybackgroundnoiseandlighting variations.

Handregionsandfingerpositionsareaccuratelydetected using MediaPipe, enabling precise classification of gestures. In several cases, gestures that are visually similar are correctly distinguished based on landmark features.

Quantitative evaluation indicates strong performance in terms of accuracy, precision, recall, and F1-score. The model achieves reliable classification results across different gesture classes. As expected in real-time systems,slightvariationsin handpositioningmayaffect predictions,butoverall performanceremainsconsistent duetonormalizedfeaturerepresentation.

The system demonstrates robustness in handling variations in illumination, orientation, and background conditions.Theuseofnormalizedlandmarkcoordinates reduces dependency on image-specific features, improvinggeneralizationcapability.

Real-time testing through the web-based interface confirms that the system can effectively convert hand gestures into text, allowing users to form words and sentencesinteractively,therebyenhancingtheusability.

p-ISSN: 2395-0072

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 p-ISSN: 2395-0072

Volume:13 Issue: 03| Mar 2026 www.irjet.net

1 - Execution Time Table

Performance Table

7. Discussion

The results suggest that the proposed sign language recognition system is well-suited for real-time gesture-to-text conversion underpracticalconditions.

However, certain limitations were observed. Variations in hand positioning, lighting conditions, and background complexity can occasionally affect prediction accuracy. Gestures with similar finger configurations may sometimes lead to minor misclassifications.

Additionally, since the evaluation relies on classification-based metrics rather than controlled benchmark datasets, the reported performance reflects practical system behavior rather than ideal conditions. Future studies may incorporate larger and more diversedatasetsevaluation.

8. Conclusion

Thisstudypresentedapracticalsignlanguagerecognitionsystem based on MediaPipe and a Random Forest classifier, aimed at enabling real-time conversion of hand gestures into text. The primary objective of the work was to develop an efficient and reliable system capable of recognizing hand signs representing alphabets (A–Z)anddigits (0–9)underreal-world conditions. To achieve this, the proposedsystemincorporates a structured data collection process followed by feature extraction using normalizedhandlandmarksandclassificationthrougha machine learningmodel.Thesystemisfurtherintegratedintoaweb-based interfacetosupportreal-timeinteractionandusability.

The results obtained from the experiments demonstrate that the proposed gesture recognition system achieves reliable performance in terms of accuracy and real-time responsiveness. The system effectively detects hand landmarks and classifies gestures with minimal computational complexity. The ability to convertgesturesintowordsandsentencesenhancesitspractical applicabilityinassistivecommunication.Theseimprovementsare particularlyvaluableinreal-worldscenarios where seamless and efficientinteractionisrequired.Bothqualitativeobservationsand quantitative evaluation indicate that the proposed system is capable of providing accurate and consistent gesture recognition undervaryingconditions.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 p-ISSN: 2395-0072

Volume:13 Issue: 03| Mar 2026 www.irjet.net

Overall, the proposed system demonstrates that realtime hand gesture recognition using MediaPipe and machinelearningtechniquescanplayanimportantrole in improving assistive communication systems. By converting hand gestures into meaningful text, the framework contributes to enhancing accessibility and interaction for hearing and speech-impaired individuals, supporting the broader field of humancomputer interaction and gesture-based communication.

Future research may focus on several possible improvements to further enhance the system’s performance and applicability. One potential direction isthedevelopmentofdeeplearning-basedmodelssuch as Convolutional Neural Networks to improve recognition accuracy for complex gestures. Another important extension would involve expanding the dataset with more diverse hand gestures, varying backgrounds, and different lighting conditions to improve generalization. Additionally, integrating the systemwithreal-timevideoprocessingpipelines.

References

1) Zhang, Y., & Jiang, X. (2024). "Recent Advances on Deep Learning for Sign Language Recognition." ComputerModelinginEngineering&Sciences.

2) Ansar, H., Al Mudawi, N., Alotaibi, S. S., Alazeb, A., Alabdullah, B. I., Alonazi, M., & Park, J. (2023). "Hand Gesture Recognition for Characters Understanding Using Convex Hull Landmarks and GeometricFeatures."IEEEAccess.

3) Rajalakshmi, E., Elakkiya, R., Subramaniyaswamy, V., Alexey, P., Mikhail, G., Bakaev, M., Kotecha, K., Gabralla, L. A., & Abraham, A. (2023). "MultiSemantic Discriminative Feature Learning for Sign Gesture Recognition Using Hybrid Deep Neural Architecture."IEEEAccess.

4) Rokade, Y. I., & Jadav, P. M. (2017). "Indian Sign Language Recognition System." International JournalofEngineeringandTechnology.

5) Reshna, S., & Vidhya, K. V. (2023). "Recognition of Indian Sign Language using Hand Gestures and FacialExpressions."InternationalConferencePaper.

6) Sneha,B.S.,Sowmya,R.,Srilakshmi,T.M.,Bhat,S.,&Reddy, S. (2022). "Sign Language Recognition System Using Indian Sign Language." International Journal of Creative Research Thoughts(IJCRT).

7) Shinde,A.,&Kagalkar,R.(2014)."SignLanguageRecognition for Deaf Sign User." International Journal for Research in AppliedScience&EngineeringTechnology.

8) Goyal, S., Sharma, I., & Sharma, S. (2013). "Sign Language Recognition System For Deaf And Dumb People." InternationalJournalofEngineeringResearch&Technology.

9) Singha,J.,&Das,K.(2015). "AutomaticIndianSignLanguage Recognition for Continuous Video Sequence." ADBU Journal ofEngineeringTechnology.

10) Rokade, Y. I., & Jadav, P. M. (2017). "Indian Sign Language Recognition System using Vision-Based Approach." InternationalJournalofEngineeringandTechnology.