LIP READING - AN EFFICIENT CROSS AUDIO-VIDEO RECOGNITION USING 3D CONVOLUTIONAL NEURAL NETWORKS

LIP READING

RECOGNITION USING

AN

1, 2, 3 Student, Dept. of Computer Engineering, Smt. Indira Gandhi College of Engineering, Navi Mumbai 400 701, Maharashtra, India (University of Mumbai)

4 Professor, Dept. of Computer Engineering, Smt. Indira Gandhi College of Engineering, Navi Mumbai 400 701, Maharashtra, India (University of Mumbai)

Abstract When the audio is distorted, Audio Video Recognition (AVR) has been introduced as a solution for speech recognition tasks, as well as a visual recognition approach for speaker verification inmulti speaker situations. The method taken by AVR systems is to use the extracted information from one modality to increase the recognition ability of the other modality by complementing the missing information. By using,arelativelysimplenetworkarchitecture anda considerably smaller data set for training, ourproposed method improves the performance of the existing similar methods for audio visual matching, which use 3D CNNs for feature representation. We also exemplify that an effective pair selectionmethodcansignificantlyincrease performance.

Key Words: Lipreading,DeepLearning,NLP,AVR,ML,CNN

I. INTRODUCTION

Lip reading,thecapacitytoapprehendspeechandtheuseof thesimplestvisibleinformation,isatotallyappealingskill. Forinstanceswhereaudioisn'tavailable,ithasclearvoice transcription programs. The most natural methodology of communicationamongpeople,ingeneral,is“talking”which we call “speech”. But sadly, this natural kind of communicationfortheindividualswhoaredumbandwhose hearing lessened cannot be used. The method of phrase reputation provided to help to listen to lessened or dumb people communicate to the others in the course of an ordinary technique. It’s a visible methodology of talking withinwhichsolelylipmovementsareappliedtoestablished vocalizedwords.Visualspeechrecognitioncanbeawaythat recognizesthephrasesthroughthemotionofthelip.Visual speechrecognitionistheprocessofreadingthelip.Thedeaf personandthehearing impairedpersoncaneasilyrecognize the speech by lip movements. Since earlier times, people havebeenapprehendedthatthemovementoftheliphashad somespeechknowledge.Visualspeechisessentialinvarious contexts, such as speech in a changing environment, scenarios where you should not be allowed to speak, and catastrophicsituationsinvolvingvolcanicactivity. Inday to dayongoinglifeeverythingisgettingdependenton computer based technologies. Computing environment is gettingclosertowardsHumanComputerInteractiondesigns.

As far as outdoor happenings are considered the Audio videorecognition(AVR)hasbeenconsideredasasolution forspeechrecognitiontaskswhentheaudioiscorrupted,as well as a visual recognition method used for speaker verificationinmultispeakerscenarios.TheapproachofAVR systemsistoleveragetheextractedinformationfromone modality to improve the recognition ability of the other modalitybycomplementingthemissinginformationBythe use of a particularly small network structure and plenty smaller statistics set for training, our proposed technique surpasses the overall performance of the present comparable techniques for audio visible matching, which use 3D CNNs for feature representation. We additionally show that a powerful pair choice technique can notably boom the performance.The actorprocessofdetermining theintendedmeaningofthespeakerbyutilizingallvisual cluesaccompanyingspeechattempts,suchaslipmovements, facial expressions, and bodily gestures, used especially by peoplewithimpairedhearing.

Inthispaper,wedevelopasystemforaudio videospeech recognition called ‘Lip Reading An efficient Cross Audio VideoRecognitionusing3DConvolutionalNeuralNetworks’. In this, the system will recognize phrases and sentences being spoken by a talking face, with or without the audio. The system will also support multiple languages using NaturalLanguageProcessing(NLP).Itwillgiveresultsinreal time. In this system we have achievedobjectives such as recognizingphrasesandsentencesbeingspokenbyatalking face, with or without the audio, and supporting multiple languages(English,Hindi)withrealtimeinteractionresults. Whichleadsustocreateauser friendlyinterfacethatcould helptohaveabetterconversationintheabsenceofaudio.

II. LITERATURE SURVEY

The primary concept of lip reading in 1954 turned into proposedwiththeaidofusingSumbyandPollack[1],andit turned into first off proposed that the functions of lip movementwillbeemployedtoidentifythespeaker'sspeech content. In 1984, Petajan [2] created an Audio Visual Automatic Speech Recognition (AV ASR) system by extractingfeaturesfromlipmovementandcombinedthem

withspeechrecognition Theresultsshowthatthesystemis morerobust

ordinaryspeech

Overtheyears,asdeepgainingknowledgeofgenerationhas receivedfantasticachievementsindiversefields,thepointof interestoflip readinghasadditionallychanged.Insteadof trying to design some feature extraction algorithms manuallytoextractfeatures,researchersadoptedthedeep network's powerful representation learning ability to automatically learn good features according to the task objectives. These features often have good generalization ability and can achieve good performance in a variety of scenarios. In 2011, Ngiam et al. [3] proposed an AV ASR system based on depth auto encoder and Restricted Boltzmann Machines (RBMs) [4]. The visual feature extraction method based on the deep learningmethod is introducedintomultimodalspeechrecognitionforthefirst time. In 2014, Waseda University’s Noda et al. [5] implementedCNNasafeatureextractiontoolforlipimage. The experimental results show that the visual functions obtainedusingaConvolutionalNeuralNetwork(CNN)are much higher to those obtained using traditional methods suchaspredominantfactoranalysis.In2016,Wandetal.[6] usedLongShort TermMemory(LSTM)forlip readingand achieved a recognition rate of 79.6% on GRID. In 2016, ChungandZisserman[7]mountedtheprimarylarge scale Englishlip readingdatabaseLRWbeneathherbalsituations inlinewiththeBBCprogram.

Assael et al. [8] proposed LipNet based on the spatial temporal convolution network and recurrent neural networkin2017andusedCTCasanetworklossfunctionin theLipNetnetwork.TheWLASnetworkproposedviawayof means of Chung et al. in 2017, which consists of CNN and Recurrent Neural Networks (RNN), obtains a 46.8% sentence accuracy charge at the LRS database with ten thousandpatternsentences.

At present, lip reading methods are divided into two categoriesaccordingtodifferentfeatureextractionmethods: 1) Lip reading based on traditional manual feature extraction method;2)Lip reading basedon deeplearning feature extraction method. For the traditional manual featureextractionmethod,thelipregionshouldbeextracted firstly;then,thefeatureextractionalgorithmdesignedbythe researchersextractingthebottommovingfeaturesofthelip region; and then through some linear functions such as Principal Component Analysis (PCA) and Discrete Cosine Transform(DCT)isusedtoprocesstheextractedfeatures andencodethemintoequallengthfeaturevectors.Finally, suitable classes such as Artificial Neural Network (ANN), HMM are used for classification. In the deep learning method, it can be employed iterative learning method to automatically extract more features than traditional methodsfromthevideoorimagesequence;Thenobtainthe scoresofeachcategorythroughthedeepmodel,andthen

adjust the network model parameters by way of backpropagationaccordingtothelabelsofthetrainingdata, andfinallyachieveagoodclassificationeffect. Inthispaper, wedevelopedasystemforaudio videospeechrecognition called ‘Lip Reading’. In this, the system is recognizing phrasesandsentencesbeingspokenbyatalkingface,with or without the audio. The system also supports multiple languagesusingNaturalLanguageProcessing(NLP).Itwill giveresultsinrealtime.

3 d CNNs simultaneously extract capabilities from each spatialandtemporaldimensions,sothemovementfactsis capturedandconcatenatedinadjoiningframes.Weuse3 d CNNstogenerateseparatechannelsoffactsfromtheenter frames. The aggregate of all channels of correlated facts createstheverylastcharacteristicrepresentation.

Theattentionofthestudiesattemptdefinedonthispaperis to put into effect non identical 3 d CNNs for audio visual matching.Theintentionistolayoutnonlinearmappingsthat research a non linear embedding area among the corresponding audio video streams the usage of an easy distancemetric.Thisstructuremaybefoundoutwiththeaid ofusingcomparingpairsofaudio videostatisticsandlater used for distinguishing among pairs of matched and non matchedaudio visualstreams.Oneofthemainbenefitofour audio visualmodelisthenoise robustaudiofeatures,which are extracted from speech features with locality characteristics,andthevisualfeatures,whichareextracted from eachspatial and temporal data of lip motions. Both audio visual capabilities are extracted with the use of 3 d CNNs,permittingthetemporalfactstobedealtwithoneata timeforhigherchoicemaking.

III. METHODOLOGY

The goal is to predict the words, phrases, and sentences spoken from a silent video of a talking face by extracting their lip movements. This section proposes an overall architecture for decoding visual speech, as illustratedin Figure I The complete technique is composed of various stages,startingoffwithaDataPreprocessinglevelwherein thelocationofinterestisextractedfromthemoviesusing facial landmark detection to offer the input to the Visual Frontend.

IV. ARCHITECTURE

The architecture is a coupled 3 D convolutional neural communitywhereinexceptionalnetworkswithexceptional unitsofweightsshouldbetrained.Forthevisualnetwork, the lip motions’ spatial and temporal facts are integrated together and might be fused for exploiting the temporal correlation. For the audio network, the extracted energy features are considered as a spatial dimension, and the stackedaudioframescreatethetemporaldimension.Inour proposed3DCNNarchitecture,theconvolutionaloperations areexecutedonsuccessivetemporalframesforeachaudio visualstreams.

Dataset.Eachexampleofthedatasetincludesasynchronized seriesofcolorandintensityimages(each640x480pixels). TheMIRACL VC1datasetcarriesacompleterangeof3000 instances.

of3DCNN

V. DATASET

The datasets that have been used for our project is the MIRACLE VC1 Dataset [9]. MIRACL VC1 is a lip studying datasetwhichincludeeachdepthandcolorimages.Itmaybe used for numerous studies fields like visual speech recognition,facedetection,andbiometrics.Fifteenspeakers (five men and ten women) positioned themselves in the frustumofaMSKinectsensorandutteredtentimesasetof tenwordsandtenphrasesshowninTableI:MIRACLE VC1

Processing

Inthevisualsection,themotionpicturesarepost processed tohaveasameframerateof30f/s.TheDliblibrary[10]is thenusedtodofacetrackingandmouthregionextractionon continuous video frames. Finally, all mouth regions are resized to have the identical length and concatenated to shapetheinputcharacteristiccube.Thedatasetdoesnowno longercompriseanyaudiodocuments.Theaudiodocuments are extracted from motion pictures using FFmpeg framework[11].

The processing pipeline is subdivided into two visual and audio sections. In the visual section, the films are post processed to have a same frame rate of 30 f/s. Then, face tracking and mouth area extraction is performed on the videosusingtheDliblibrary.Finally,allmouthregionsare resizedtohavetheidenticalsize,andconcatenatedtoshape the input feature cube. The dataset does now no longer incorporateanyaudiofiles.Intheaudiosection,theaudio filesareextractedfromvideosusingtheFFmpegframework. Thenthespeechfeatureswillbeextractedfromaudiofiles. Thelibrarythathasbeenusedforspeechfeatureextraction tasks is SpeechPy. The words and phrases spoken by the speakersareshowninTableI.

VI. RESULT

Inthisworkwepresentedbothaprocessingframeworkfor extracting lip data from videos and various 3D CNN architecturesforperformingvisualspeechrecognitionona datasetofmanysimilarwordsrecordedinanuncontrolled environment. Accuracy which is in line with other recent work done on this problem. Most notably our results are similartothosepresentedintheonlycurrentpapertouse 3D CNNsforclassificationoftheMiracledataset.

VII. CONCLUSION

Inthisstudy,weproposedanAVSRsystemprimarilybased totallyondeeplearningarchitecturesforaudioandvisual characteristicextraction.Inthisprojectwedevelopasystem that will allow input voice to see into text format. Communication among human beings is dominated by spokenlanguage,thereforeitisnaturalforpeopletoexpect voiceinterfaceswithcomputers.Thiscanbeaccomplished bydevelopingavoicerecognitionsystem:Audio/Video to textwhichallowscomputerstotranslatevoicerequestsand dictation into text. This project made a clear and simple overviewofworkingofAudio/Videototextsystems.

VIII. FUTURE SCOPE

TheSystemcanbeimplementedinmanydifferentways:

1. It will be useful in applications related to improvedhearingaids.

2. Videoconferencinginsilentenvironments.

3. High quality speech recovery from surrounding noise

4. Individuals who are unable to produce spoken soundscanhavetheirvoicesgenerated(Aphonia)

5. It will be also useful in applications related tobiometricauthentication.

ACKNOWLEDGEMENT

As every project is ever complete with the guidance of experts.Sowewouldliketotakethisopportunitytothank allthoseindividualswhohavecontributedtovisualizingthis project.

WeexpressourgreatestgratitudetoourprojectguidesProf. Sarita Khedikar and Prof. Vaishali Yeole (Computer Department,Smt.IndiraGandhiCollegeofEngineeringand theUniversityofMumbai)fortheirvaluableguidance,moral support,anddevotionbestowedonusthroughoutourwork.

We would also take this opportunity to thank our project coordinatorProf.DeeptiVijayChandranforherguidancein selectingthisprojectandalsoforprovidinguswithallthe details on the proper presentation of this project. We are alsogratefultoourHODDr.KishorT.Patilandforextending hishelpdirectlyandindirectlythroughvariouschannelsin ourproject.

Weextendoursincereappreciationtoourentireprofessors from Smt. Indira Gandhi College of Engineering for their valuableinsideandtipduringthedesigningoftheproject. Theircontributionshavebeenvaluableinmanywaysthat wefinditdifficulttoacknowledgeindividually.

If I can say in words I must at the outset my intimacy for receiptofaffectionatecaretoSmt.IndiraGandhiCollegeof Engineering for providing such a stimulating atmosphere andgreatworkenvironment.

REFERENCES

[1] TheJournaloftheAcousticalSocietyofAmerica26, 212(1954);doi:10.1121/1.1907309

[2] J. S. Chung and A. Zisserman, ``Lip reading in the wild,'' in Proc. Asian Conf. Comput. Vis. Cham, Switzerland:Springer,2016,pp.87103.

[3] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, ``Lip Net: End to end sentence level lipreading,'' 2016, arXiv: 1611.01599. [Online]. Available:http://arxiv.org/abs/1611.01599.

[4] D. E. King, ``Dlib ml: A machine learning toolkit,'' J. Mach.Learn.Res.,vol.10,pp.17551758,Jan.2009.

[5] A.Tor,S.M.Iranmanesh,N.Nasrabadi,andJ.Dawson, ``3Dconvolutionalneuralnetworksforcrossaudio visualmatchingrecognition,''IEEEAccess,vol.5,pp. 2208122091,2017.

[6] J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, ``Lip reading sentences in the wild,'' in Proc. IEEE Conf.Comput.Vis.PatternRecognit.(CVPR),Jul.2017, pp.34443453.

[7] S.Yang,Y.Zhang,D.Feng,M.Yang,C.Wang,J.Xiao,K. Long,S.Shan,andX.Chen,``LRW 1000:Anaturally distributedlarge scalebenchmarkforlipreadingin the wild,''inProc.14thIEEEInt.Conf.Autom.Face GestureRecognit.(FG),May2019,pp.18.

[8] Hao,Mingfeng&Mamut,Mutallip&Yadikar,Nurbiya &Aysa,Alimjan&Ubul,Kurban.(2020).ASurveyof ResearchonLipreadingTechnology.IEEEAccess.8. 204518 204544.10.1109/ACCESS.2020.3036865.

[9] AhmedRekik,AchrafBen Hamadou,WalidMahdi:A NewVisualSpeechRecognitionApproachforRGB D Cameras.ICIAR2014:21 28

@inproceedings{RekikICIAR14, author ={AhmedRekikandAchraf{Ben Hamadou} andWalidMahdi}, title ={ANewVisualSpeechRecognitionApproach for{RGB D}Cameras}, booktitle={ImageAnalysisandRecognition 11th InternationalConference,{ICIAR} 2014,Vilamoura, Portugal,October22 24,2014}

[10] Davis E. King. Dlib ml: A MachineLearning Toolkit. JournalofMachineLearningResearch10,pp.1755 1758,2009

@Article{dlib09, author={DavisE.King}, title={Dlib ml:AMachineLearningToolkit}, journal={JournalofMachineLearningResearch}, year={2009}, volume={10}, pages={1755 1758}, }

[11] Tomar, S., 2006. Converting video formats with FFmpeg. Linux Journal, 2006(146), p.10.@article{tomar2006converting title={ConvertingvideoformatswithFFmpeg}, author={Tomar,Suramya}, journal={LinuxJournal}, volume={2006}, number={146}, pages={10}, year={2006}, publisher={BelltownMedia} }