Bridging the Communication Gap: A Sensor-Based System for Sign-to- Speech and Speech-to-Text Convers

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

Bridging the Communication Gap: A Sensor-Based System for Sign-toSpeech and Speech-to-Text Conversion

Yigithan OZER, Galip ERGUN, Goktug BAY, Hatice OKUMUS

Karadeniz Technical University, Turkiye, Department of Electrical and Electronics Engineering, Faculty of Engineering, 61080, Trabzon, Turkiye, ***

Abstract - Hearing and speech-impaired individuals use country-specific sign languages to communicate effectively both with one another and with others in their daily lives. However, many people in society lack sufficient knowledge of sign language. This limitation makes it difficult for hearingimpaired individuals to build social relationships, advance in education, and succeed in professional life. Feeling excluded from society can also lead to a loss of self-confidence. These communication barriers restrict their ability to fully express themselves and actively participate in social life. In this study, a glove is designed that converts sign language gestures into speech and simultaneously converts spoken language from non-disabled individuals into text. The goal of this design is to enable real-time, two-way communication between hearing and speech-impaired individuals and those who do not know sign language, thereby supporting their more active inclusion in society. In the first phase, flex sensors placed on the fingers will be used to detect the hand movements of hearing and speech-impaired individuals and convert these gestures into speech. These sensors will accurately capture hand and finger movements, interpret sign language gestures through a microcontroller, and convert them into audible output via a speaker. In the second phase, the speech of non-disabled individuals will be detected using a microphone and speech recognition software, converted into text, and displayed on a screen. Through this study, hearing and speech-impaired individuals will not only be passive listeners but will also be able to actively participate in conversations and express their thoughts instantly.

Key Words: Hearing impaired people; Sign Language; Arduino Nano; Flex Sensor; Gloves

1. INTRODUCTION

Technological advancements not only make human life easierbutalsoenhancethequalityoflifeforindividualswith disabilities,allowingthemtointegratemoreeffectivelyinto society. In recent years, newly developed technological devicesandsoftwarehavemadesignificantcontributionsto overcomingcommunication barriersfacedby people with disabilities.Suchtechnologicalsolutionsfacilitatethedaily lives of hearing-impaired individuals and enable them to participatemoreactivelyinsociallife.Varioussoftwaretools andassistivehardwaresystemsprovidesubstantialbenefits in reducing or even eliminating disabilities [1]. In the

literature,manystudieshavebeenconductedtoimprovethe livesofhearing-impairedindividuals.Patiletal.designeda device that enables deaf and mute individuals to communicatewithnormal-hearingpeoplebyconvertingsign languageexpressionsintoaudiblecommands.Thisdeviceis adataglovemountedonthehand,equippedwithfivesmall, low-power, three-axis ±3 g accelerometer sensor units (ADXpL335)attachedtothefingertips.Sincetheoutputof these sensors is in analog form, an Arduino Mega microcontrollerwasfoundtobesuitableforinterpretingthe signals.Theentiresystemwasintegratedandverifiedusing theArduinoIDE[2].Extensiveexperimentswerecarriedout with numerous volunteers who could fluently perform multiplehandmovementsasspecifiedintheMarathiSign Language (MSL) for the Marathi alphabet. The real-time performance of the system was evaluated in terms of its accuracy and precision in recognizing Marathi alphabets, establishingacorrelationbetweenhuman-generatedsigns andthosestoredinthesystem[3].Mathewetal.developeda deviceaimedatenhancingthemobility,independence,and safetyofhearing-impairedindividualswhilecrossingstreets. The device detects approaching vehicles and provides vibration-based alerts to help users perceive cars coming frombehind.Anultrasonicsensorandacameraareusedto identifyobjectsintheenvironment.Inaddition,thedevice can connect to smartphones via Bluetooth and employs a piezoelectric transducer to convert sound signals into vibrations, thereby improving the efficiency of alarm and communicationsystems[4].Navaitthipornetal.designeda glovethatconvertssignlanguagegesturesintobothspeech and text, addressing communication challenges faced by hearing-impaired individuals. In this project, flex sensors measure the bending of fingers, while the GY-521 module detectstheorientationandmovementofthehand.Thedata from the sensors are transmitted to the Arduino IDE, converted into alphabetic text, and then synthesized into speech [5]. Similarly, Rewari et al. used flex sensors and gyroscopes to detect finger movements and hand angles, converting these gestures into sound through a microcontroller [6]. Chin et al. developed a wearable assistivedevicethatenableshearing-impairedindividualsto recognizewarningsoundsfromvehiclesontheroad[7].For

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

this purpose, an EfficientNet-based and fuzzy-ranking ensemble model was proposed and integrated into an ArduinoNano33BLESensedevelopmentboard.Thesound filesweretakenfromtheCREMA-Ddataset[8]andalargescale sound dataset containing sirens from emergency vehicles [9], comprising a total of 8,756 audio samples. Theseincludefourtypesofvocalizationsandthreetypesof siren sounds. The sound signals were converted into spectrogramsusingshort-timeFouriertransform(STFT)for feature extraction. When one of the three siren sounds is detected,thewearabledeviceprovidesavibrationalertand displaysacorrespondingmessageonanOLEDpanel.Inthe device developed by Yağanoğlu and Köse, sound data collectedbyamicrophoneareprocessedinaRaspberryPi environmentthroughaUSBsoundcard.Oncereceived,the soundsareclassifiedusingvarioustechniquestodetermine their types. The system identifies environmental sounds such as doorbells, alarms, telephone rings, horns, brakes, dogs, human voices, and other noises. For each sound, a uniquevibrationpatternisdesignedandtransmittedtothe uservia a vibration motor, providingtactile feedback that canbeperceivedthroughtouch[10].Asakuradevelopedan augmented reality-based system that converts household sounds into visual information for hearing-impaired individuals. In this system, environmental sounds are continuously captured using a microphone. The sound pressure waveform recorded by an omnidirectional microphoneconnectedtoasoundlevelmeteristransferred toalaptopthroughanaudiointerfaceforprocessing.These sound

waveformsarethenanalyzedbyamachine-learning-based classifierandsimultaneouslyconvertedintospectrograms. Throughaugmentedrealityglasses,userscanviewnotonly the classification results but also real-time visual representations of the detected sounds [11]. WHK et al. developed an application designed to support hearingimpairedelementaryschoolstudentsinSriLanka.Thestudy consistsofthreemaincomponents:asoundrecognitionand classification system, Android software that translates gestures in Sinhala Sign Language (SSL) into text, and a mobileapplicationthatperformsemotionrecognitionand text-to-speechconversioninSinhala.Thesoundrecognition systemrapidlydetectspotentialhazardstoensurethesafety ofhearing-impairedchildrenathome.TheAndroidsoftware captures SSL gestures through the device’s camera and represents them as text, while the mobile application identifies emotions from facial expressions using ConvolutionalNeuralNetworks(CNNs)[12].Baltacıoğluet al. designed a wristband for parents who are hearingimpaired or have hearing loss. The main objective of this study is to analyze and differentiate between silent environments,speechsounds,andbabycriesbyusingsound intensitydistributiontocreateawarningsystem.Thesound data were collected and analyzed to develop a vibrationbasedalertmechanism.Thedatafromthebaby’sroomwere transmittedviaradiofrequenciestotheparent’swristband. An analysis of 20-second sound samples revealed that speechsoundsincreasedbyapproximately75%compared tosilentenvironments,whilebabycriesshoweda102.5%

Fig- 1. Thegeneralblockdiagramofthesystem

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

increase. These results demonstrate that the system can effectivelyfunctionasawearabletechnology[13].Quiapiet al.,intheirproject,designedasystemthatconvertscaptured American Sign Language gestures into simple English sentencesusingasensor-basedglove[14].Whenexamining previousstudies,itisobservedthatmostofthemofferonesidedsolutions.However,theproposedstudyaimstodesign a glove that enables two-way communication by both convertingsignlanguageintospeechandtransformingthe speech of non-disabled individuals into text. In the first phase of the glove’s design, flex sensors attached to each fingerdetectthehandmovementsofhearing-andspeechimpairedusers.Theanalogsignalsfromthesesensorsare processed by an Arduino Nano microcontroller, which interprets the gestures and converts them into corresponding audio outputs. In the second phase, the speech of non-disabled individuals is transformed into writtentextusingsmartphone-basedSpeech-to-Text(STT) technology. This text is then wirelessly transmitted to the Arduino-connected display, where it is instantly shown in realtime.

2. PROPOSED SYSTEM

In this study, a smart glove that detects the hand movementsofhearingandspeech-impairedindividualsand converts them into both auditory and visual outputs was developed. The system employs flex sensors to measure finger movements. The analog data received from the flex sensors were processed using an Arduino Nano microcontroller.Eachfingermovementwasdefinedbasedon specificthresholdvalues,andthesysteminstantlyrecognized thecorrespondingwordsassociatedwiththesemovements.

Theidentifiedwordswereplayedasaudiofilesthrougha

data in real time and displayed it on the TFT LCD screen mountedontheglove.Inthisway,spokeninformationcould beinstantlyfollowedintextform.Thegeneralblockdiagram ofthesystemispresentedinFig-1.

Each piece of equipment and component used in the development of this study is given in detail in Table-1. ArduinoNanowaschosenduetoitslowcostandcompact design, which allowed the overall size of the project to be reducedandportabilitytobeachieved.Iflargerboards(such as the Arduino Uno or Mega) had been used instead, the project would have required more space and power.

Table -1: Listoftheequipmentand componentsusedinthedevelopmentof theglove.

Component Functionality

Arduino Nano To process data received from sensors, control operations, and provide appropriate feedback.

Flex Sensor To detect finger movements. Speaker To audibly transmit data obtained from sensors and provide instant feedback to the user.

LCD TFT Display To display the text form of the speech from a non-disabled individual.

Li-Po Battery To supply power to the system.

Micro SD Card Module To interface the SD card with the circuit.

SD Card To store audio data. HC-06 Bluetooth Module To provide wireless (Bluetooth) communication between the Arduino and other devices.

PAM8403 Audio Amplifier To increase the output sound level of the speaker.

Fig- 2. RepresentationofFingerPositionsCorrespondingtoWords

speaker andsimultaneouslydisplayed on a TFT screen withoutdelay.Thespeechofnon-disabledindividualswas converted into text using the Speech-to-Text (STT) technologycommonlyemployedinsmartphones.Thistext datawastransmittedwirelesslytotheArduinoviatheHC-06 Bluetoothmodule.TheArduinoprocessedthereceivedtext

Additionally, these boards are more expensive and less suitableintermsofportability.ALi-Pobatterywasusedto meettheproject’spowerrequirements.Li-Pobatteriesare preferred power sources in portable projects due to their highenergydensityandlightweightstructure.Theselected

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

battery has sufficient capacity to ensure continuous operationofthesystem.Moreover,therechargeablenature ofLi-Pobatteriesallowsforlong-termuseoftheproject.Flex sensorswereemployedtodetectfingermovements.These sensors measure the degree of finger bending, enabling accurate detection of the user’s gestures. Their sensitivity allowsthesystemtoprovidepreciseandreliablefeedback

A. Recognition of Sign Language Gestures

In the recognition of sign language gestures, five flex sensorswereusedtomeasurethedegreeoffingerbending. Each flex sensor generates analog signal values corresponding to finger movements, and these values are calibrated at specific intervals. The finger positions corresponding to each word were modeled using the minimumandmaximumthresholdvaluesobtainedfromthe sensor readings. Some example finger positions correspondingtospecificwordsareshowninFig-2.

Forinstancethesystemrecognizesthegesturegivenin Fig-3asthe“GoodMorning”commandwhenthereal-time sensorreadingsfallwithinthepredefinedthresholdranges givenbelow.Oncethegestureisdetected,thesystemtriggers the audio file named gunaydin.wav stored on the SD card, providingauditoryfeedbackthroughthespeaker.

Minimum Threshold Values: {160,-1,160,160,160}

Maximum Threshold Values: {500,159,500,500,500}

Inthecodestructure,themeasurementrangeofeachof thefiveflexsensorswasdefinedseparatelyforeach word, andtheincomingsensorvalueswerecheckedtodetermine whethertheyfellwithintheseranges.Forexample,forthe word“Hello”,allsensorvaluesmustbewithintherangeof 160to250,whereasfortheword“Hi”,thefirstsensorvalue isexpectedtobewithintherangeof0–159,whiletheother sensorsoperatewithinhigherranges.Athresholdvalueof-1 indicatesthatthereisnominimumlimitforthatparticular sensor,andonlythemaximumthresholdisevaluated.This threshold-based approach numerically defines the unique fingerpositionscorrespondingtoeachword.Thereal-time dataobtainedfromtheflexsensorsarecomparedwiththese predefinedvalueranges.Ifthesensorreadingsfallwithinthe

specified intervals, the system correctly identifies the correspondingword.Thus,theanalogdataprovidedbythe flexsensorsserveasaneffectivemeasureforaccuratelyand preciselydistinguishingdetailedfingermovements.

B. Speech Detection and Conversion to Text

Inthisstudy,thespeech-to-textconversionprocesswas performed using a pre-existing Speech-to-Text (STT) application operating on a mobile phone. This application capturesandanalyzestheuser’sspokencommandsthrough themicrophoneofthemobiledeviceandconvertstheminto writtentextinrealtime.Theconvertedtextistransmitted wirelessly to the Arduino via the HC-06 Bluetooth module and instantly displayed on the TFT LCD screen integrated into the system. In this way, spoken information is transformedintowrittenform,providinghearing-impaired individualswithameansofvisualcommunication.

OneofthemainadvantagesofthemobileSTTapplication usedinthissystemisitshighaccuracyinonlineoperation,as wellasitsabilitytostorepreviouslyconvertedtext onthe phone’sscreen.Thisfeatureallowsuserstoviewnotonlythe currenttextbutalsopreviousconversationcontent,enabling easy access to the communication history. While new text appearsonthescreen,earlierdataremainstoredinmemory, allowinguserstorevisitpastinformationwhenneededand ensuring a clearer and more consistent communication process.TheSTTapplicationinterfaceusedinthesystemis showninFig-4.

Withitsuser-friendlyandportabledesign,thesystemcan be conveniently used in any environment. Thanks to the practicalityofthemobileapplication,userscanquicklyand effectively convert their speech input into text without requiringtechnicalexpertise,anddirectlycommunicatethis texttohearing-impairedindividuals.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 05 | May 2026 www.irjet.net p-ISSN: 2395-0072

2. CONCLUSIONS

Inthisstudy,areal-time,bidirectionalcommunication system is developed to bridge the communication gap betweenhearingandspeech-impairedindividualsandthose unfamiliarwithsignlanguage,usingflexsensorstotranslate hand gestures into speech and speech recognition technology to convert spoken words into text. During the conversion of sign language gestures into speech, the predefinedflexsensorthresholdvaluesfordifferenthand movementswereaccuratelydetected,andthecorresponding audiofileswerecorrectlytriggered.Thetestsdemonstrated highaccuracyratesinrecognizingthedefinedsignlanguage gestures.Inthespeech-to-textphase,thesystemexhibited excellentperformanceunderlowambientnoiseconditions. However,whenenvironmentalnoiseincreased,anoticeable decrease in accuracy was observed. This effect was particularly evident in areas with intense background sounds, where the speech recognition process was negativelyinfluenced.Accordingtouserfeedback,thetext displayed on the screen was generally clear and comprehensible,althoughoccasionaldistortionsinsentence structurewerereported.

The system’s response time was also examined under conditionswherebothcommunicationdirectionsoperated simultaneously.Onaverage,thetimefromasignlanguage gesturetoaudibleoutputthroughthespeakerwasmeasured as 1.2 seconds, while the conversion from speech to text displayontheLCDscreentookapproximately1.5seconds. Thesedurationswerefoundtobeshortenoughnottoaffect user experience negatively, confirming that the system achieveditsgoalofreal-timecommunication.

In conclusion, the developed smart glove prototype enabled effective, fast, and bidirectional communication betweenhearingandspeech-impairedindividualsandnondisabledusers.Thesystem hasthepotentialtocontribute significantlytotheactiveparticipationofhearing-impaired individuals in various fields such as education, public services,andsocialinteractions.

ACKNOWLEDGEMENT

We would like to express our sincere gratitude to the Scientific and Technological Research Council of Türkiye (TÜBİTAK) for their financial support under the BİDEB 2209-A Research Projects Support Program for UndergraduateStudents,andalsototheKaradenizTechnical University Scientific Research Projects Coordination Unit (ProjectCode:FLÖ-2024-16241).

REFERENCES

[1] M. Güngör, A. Bolat, H. Cengiz & N. Aslan, “Özürlülere yönelikteknolojikdüzenlemeler’,SektörelAraştırmave StratejilerDairesiBaşkanlığı,2011.

[2] The Ministry of Home Affairs, Access Date: October 2 2024.https://www.ijcaonline.org/archives/volume118/ number13/20802-3485

[3] Indian Sign Language Portal, Access Date: October 2 2024.https://indiansignlanguage.org/

[4] J. Mathew, A. S. Philip, A. Thomas, A. A. Raju & A. Tennyson, A., “IoT based Alerting Device for Hearing ImpairedPopulation",7thInternationalConferenceon Trends in Electronics and Informatics (ICOEI), 11-13 April,2023,Tirunelveli,India,pp.497-501.

[5] N. Navaitthiporn, P. Rithcharung, P. Hattapath & C. Pintavirooj, “Intelligent glove for sign language communication”, 12th Biomedical Engineering InternationalConference(BMEiCON),19-22November, 2019,UbonRatchathani,Thailand,pp.1-4.

[6] H.Rewari,V.Dixit,D.Batra&N.Hema“Automatedsign languageinterpreter”,11thinternationalconferenceon contemporary computing (IC3), 2-4 August, 2018, Noida,India,pp.1-5.

[7] L.Chin,C.C.Lin,J.W.Wang,W.C.Chin,Y.H.Chen,S.W. Chang,...&S.H.Liu,“AWearableAssistantDevicefor theHearingImpairedtoRecognizeEmergencyVehicle Sirens with Edge Computing”, Sensors, 23(17), 7454, 2023.

[8] CREMA-D Dataset. Access Date: October 2 2024.https://paperswithcode.com/dataset/crema-d.

[9] M.Asif,M.Usaid,M.Rashid,T.Rajab,S.Hussain,S.Wasi, “Large-scaleaudiodatasetforemergencyvehiclesirens androadnoises”,Sci.Data,9,599,2022.

[10] M. Yağanoğlu & C. Köse, “Real-time detection of important sounds with a wearable vibration based deviceforhearing-impairedpeople”,Electronics,7(4), 50,2018.

[11] T. Asakura, “Augmented-reality presentation of householdsoundsfordeafandhard-of-hearingpeople”, Sensors,23(17),7616,2023.

[12] W.H.K.Fernando,P.L.D.N.Perera,M.I.N.Fernando, P. T. E. W. Gunasekara, H. Fernando & R. De Zoysa, “EmpoweringDeafChildrenwithSinhalaSignLanguage, Emotion Detection, and Sound Recognition”, 5th International Conference on Advancements in Computing(ICAC),07-08December,2023,Colombo,Sri Lanka,pp.388-393.

[13] K.Baltacıoğlu,M.T.Başar,V.Akdoğan,M.Karaaslan& E. A. İlçigen, “Baby cry-sensitive armband design for parents with hearing loss”, European Mechanical Science,7(4),253-258,2023.

[14] E.A.Quiapo&K.N.M.Ramos,“Developmentofasign languagetranslatorusingsimplifiedtilt,flexandcontact sensormodules”,IEEEregion10conference(TENCON), 22-25November,2016,Singapore,pp.1759-1763.