Generating LaTeX Code for Handwritten Mathematical Equations using Convolutional Neural Network

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Generating LaTeX Code for Handwritten Mathematical Equations using Convolutional Neural Network

* These authors have contributed equally to the work Dept. of Computer Science Engineering, JSS Science & Technology University, Mysore, India 570006 ***

Abstract Handwritten mathematical equation recognition and processing are one of the complicated issues in the area of computer vision. Classification and segmentation of a single character makes it even harder. In this paper, Convolution Neural Network(CNN) is used for recognizing the equations as it provides better accuracy compared to other models like Support Vector Machine (SVM) and Artificial Neural Network (ANN). Furthermore, the obtained result is converted to LaTeX code which can be used for various scientific purposes.

Key Words: Convolutional Neural Network, Computer Vision, Support Vector Machine, Artificial Neural Network, LaTeX code.

1.INTRODUCTION

Due to technological advancement, handwriting which was a natural part of human interaction is now slowly being replaced by a digital pen, stylus, interactive panels andsmartwritingsurface.Thesearealsobeenadoptedin educational institutions and the workplace, which in turn results in the rise in demand of handwriting recognition which also involves the unique handwriting of each individual.

The outbreak of the COVID 19 pandemic increased the need for such applications for users and students. There was a sudden increase in the usage of handwritten interactive applications for the evaluations, which was used majorly in online and distance education modes. There also have been many advances made in sequence recognitionmodelsbasedonCNN.

Mathematical expressions play an important role in engineering, research, finance, education and other domains. Thelargesetofmathematical symbolsare often similar to one another, especially in handwritten expressions which causes few issues in recognition. The input which is the handwritten mathematical expression is often provided by the users through the keyboard or using any other input devices. Given equations will be character segmented and classified using the required techniqueandfurtherwillbereturnedasLaTexcode.The proposed technology will be implemented to recognize a handwritten mathematical expression from an image and

thensolveittoproducetheresult,laterconvertingittoits correspondingLaTexcode.

2.LITERATURESURVEY

2.1 Handwritten Character Recognition from ImagesusingCNN-ECOC

Mayur Bhargab Bora, Dinthisrang Daimary, Khwairakpam Amitab, Debdatta Kandar, proposed a Convolutional Neural Network(CNN) Error Correcting Output Code (ECOC) approach, which is the hybridization of CNN architecture (used for feature extraction) and ECOC classifier(usedforclassification).

2.2 Recognition and Solution for Handwritten EquationUsingConvolutionalNeuralNetwork

MdBipulHossain,FerozaNaznin,Y.A.Joarder,MdZahidul Islam, Md Jashim Uddin, proposed a method to recognize the handwritten quadratics of the form ay2+by+c=0 from the images. They found the correct solution, for each successfuldetectionoftheequations.

2.3 Identification of Handwritten Simple Mathematical Equation Based on SVM and ProjectionHistogram

Sanjay S. Gharde, Pallavi V. Baviskar, K. P. Adhiya, have proposed the following method to identify handwritten simple mathematical equations. Here the unwanted data like dots, loops, curves, and lines were removed from the images using a noise removal algorithm. They also used a projection histogram for feature extraction and for classifying the equations, they applied a Support vector machinealgorithm.

2.4 Handwritten Equation Solver Using ConvolutionalNeuralNetwork

Shweta V. Patil, Apurva S. Patil, Harshada C. Mokal, Asst. Prof. Mr Swapnil Waghmare developed a web application which captures handwritten equations via camera. Initially, the preprocessing of the images is performed. Then, character segmentation is done on the obtained output. Convolution Neural Network is used for the

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

classification of characters which then is converted to a string format. String operations are performed on each recognized equation for the solution. They have also provided the links which explain the solution in detail for eachoftheequations.

3.PROPOSEDWORK

The dataset is collected from an online source. We have considered nearly 30,000 images for training, which consisted of various mathematical symbols. These images are used for training our CNN model. After our model is trained, we input an image containing a handwritten mathematical equation. The inputted image is then grey scaled and binarized so that it will be easy for the model to segment it. Then, the image is segmented and processed. It will be then converted to the corresponding LaTeXstring.

Wehavecreatedasimplewebpageontopofthemodelby using HTML, CSS and javascript. This allows the users to uploadtheimagesandretrievetheresults.

We measured the output of our application with 78 manually picked handwritten equations from the im2LaTeXdataset.Tomeasuretheoverall accuracyofthe model, we used the average minimum edit distance and Charactererrorrate(CER).

3.1SYSTEMARCHITECTURE

Figure 1 Flow Chart

A. Pre processing of Image The objective of the pre processingistoconstructnecessaryinformation.Here,the inputted image is grey scaled and binarised. Then the imageissegmented.

B. Neural network model Theaboveimage isinputted totheConvolutional Neural Network (CNN)model, which was trained previously using the dataset containing handwrittenmathematicalsymbols.

C. Conversion to LaTeX string The output from the above step is converted into its corresponding LaTeX string by parsing the image from left to right. While the

model parses each character, it starts building the expression tree. For every mathematical symbol parsing rules are defined, which changes the parsing direction accordingly. This is a recursive solution which evaluates eachcharacterandtakesthedecisionbasedonthecurrent characterandwhatisontopof thestack,thusproducing therequiredLaTeXstring.

D. Displaying the solution Theoutputsolutionequation is displayed on our web page. We used django to interact withthebackendserverandfetchtheresults.Itsends the Jsonresponsetothewebpage.

3.2METHODOLOGY

3.2.1

Handwritten mathematical equation processing:

Data is selected from an online source i.e., https://www.kaggle.com/xainano/handwrittenmathsymb ols. There were around 100,000 images and we have considered around 30,000 images for training. The resizingandformattingofdataarehandledhere.

Images are converted to Grayscale and represented througha single matrix becausedetectingcharactersona colouredimageismorechallengingthanona

grayscale image. If the grey bitmap is Y and the colour bitmap is R, G and B, then the formula is Y = 0.299R + 0.587G+0.114B.

Binarization is the procedure of choosing a threshold value for the adaptation of pixel values into 0’s and 1’s where 1’s represent the black pixels while 0’s represent the white pixels. The threshold choice of binarization can be approved in two ways: overall threshold and partial threshold.

Then the segmentation of the binarized image is done. The dataset is trained and then the input images will be classified into a particular class. Finally, the model will solvetheequationsandgeneratethecorrespondingLaTeX string.

3.2.2 Webapplication

A simple front end is created so that the user can upload the images containing handwritten mathematical equationstoit.

The equation will be then processed and later converted to its corresponding LaTeX code, which will be displayed onourwebpage.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

4.RESULTS

4.1 Comparison of accuracies of various models :

We took the dataset which consisted of handwritten mathematical symbols from kaggle i.e., https://www.kaggle.com/xainano/handwrittenmathsymb ols.WeappliedafewofthemodelslikeSVM,CNN,ANNto classifyit.Wegottheaccuracyasshownbelow.

Sl No. Model Name

Accuracy (in %)

1 Convolutional Neural Network 99.00

2 Support Vector Machine 98.95 3 Artificial Neural Network 95.85

4.2 Handwritten mathematical equation processing and LaTeX code generation using CNN:

4.2.1 Handwritten mathematical equation processing :

Step 1: Handwritten mathematical equation is the input giventothemodel.

Step 3: The above binarized image is then segmented as shownbelow.

Figure 4 Segmentation of the binarized image

Step 4: The equation is processed and converted to its equivalentLaTeXstring.

Figure 5 Producing the LaTeX string

‘\’ is the symbol that sits according to the LaTeX syntax. The ‘\’ symbol was producing some errors while performing some string operations, as it acted as the escapecharacter.Wehavereplaced‘\’with‘#’.

Inthefinaloutput,thisissuehasbeenresolved.

4.2.2

Web application

Frond end: A simple web page is created on top of the model by using HTML, CSS and javascript. This allows the userstouploadtheimagesandretrievetheresults.

Figure 2 Input: handwritten mathematical equation

Step 2: The input image is grey scaled and binarized so thatitwillbeeasyforthemodeltosegmentit.

Figure 3 Grey scaling and Binarization

Here, we can observe that ‘#’ which was used while producing the LaTeX string has been replaced with ‘\’, thusproducinganacceptableresult.

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

Metrics:

We measured the output of our application with 78 manually picked handwritten equations from the im2LaTeX dataset. We got an average minimum edit distanceof2.33andanaveragecharactererrorrate(CER) of6.67.

5.CONCLUSIONSANDFUTUREENHANCEMENT

Handwritten mathematical equation processing is one of themostinterestinganddifficultfields. Theprocessingof the equations is tedious and it requires a lot of attention because of the variations in handwriting. As mentioned above, CNN model producedhigheraccuracy. Sincethere is a growing popularity for the use of CNN among the practitioners for image recognition related activities, we decidedtogoaheadwiththismodel.

Currently, limited mathematical symbols have been introduced for our project, to reduce the complexity. In the future, we can also include many other mathematical symbols and equations involving integration, differentiation etc. With further time and computational resources,therecanbeanincreaseinexpressionaccuracy obtained.

ACKNOWLEDGEMENT

It gives us immense pleasure to write an acknowledgement to this project, a contribution of all the people who helped to realize it. We extend our deep regards to Dr S.B. Kivade, Honorable Principal of JSS Science and Technology University, for providing an excellent environment for our education and for his encouragement throughout our stay in college. We would liketoconveyourheartfeltthankstoourHOD,DrSrinath, for giving us the opportunity to embark on this topic. We wouldliketothank ourprojectguide,Prof.RakshithPfor their invaluable guidance and enthusiastic assistance and for providing us support and constructive suggestions for the betterment of the project, without which this project would not have been possible. We appreciate the timely help and kind cooperation of our lecturers, other staff members of the department and our seniors, with whom we have come up all the way during our project work withoutwhosesupportthisprojectwouldnothavebeena success. Finally, we would like to thank our friends for providing numerous insightful suggestions. We also convey our sincere thanks to all those who have

contributed to this learning opportunity at every step of thisproject.

REFERENCES

[1] Mayur Bharag Bora, Dinthisrang Daimry, Khwairakpam Amitab,Debdatta Kandar,Handwritten CharacterRecognitionfromImagesusingCNN ECOS.

[2] Md Bipul Hossain, Feroza Naznin, Y.A. Joarder, Md Zahidul Islam, Md Jashim Uddin,Recognition and solutionforhandwrittenequationusingCNN.

[3] Sanjay S. Gharde, Pallavi V. Baviskar, K. P. Adhiya,Identification of Handwritten Simple MathematicalEquationsbasedonSVMandProjection Histogram.

[4] Shweta V. Patil, Apurva S. Patil, Harshada C. Mokal, Asst. Prof. Mr Swapnil Waghmar,Handwritten EquationSolverusingConvolutionalNeuralNetwork.

BIOGRAPHIES

Prof.Rakshith

Professor at JSS Science & TechnologyUniversity, Dept. of Computer Science Engineering

AryanSharma

Student of JSS Science & TechnologyUniversity, Dept. of Computer Science Engineering

AshwithaABhandary Student of JSS Science & TechnologyUniversity, Dept. of Computer Science Engineering

HarshitVKaisare

Student of JSS Science & TechnologyUniversity, Dept. of Computer Science Engineering

International Research Journal of Engineering and Technology (IRJET) e ISSN: 2395 0056

Volume: 09 Issue: 07 | July 2022 www.irjet.net p ISSN: 2395 0072

SaumyaSemwal Student of JSS Science & TechnologyUniversity, Dept. of Computer Science Engineering