A STUDY ON TWITTER SENTIMENT ANALYSIS USING DEEP LEARNING

A STUDY ON TWITTER SENTIMENT ANALYSIS USING DEEP LEARNING

Abstract - Sentiment analysis is a branch of research that examines feelings, attitudes, and reviews from many public spheres. Now-a-days, people share their thoughts and insights on a wide range of issues and topics via social media. Recently social networking sites like Twitter and Face book have become popular because users can able to express their opinions with other internet users through micro blogging. Today Twitter is among the most widely used blogging sites. But the disrespectful, insensitive, or unfair remarks that occasionally appear in online forums drive many people away. The majority of websites are unable to promote productive discourse, thus either heavily restrict or fully disable user comments. Insightful data about what is stated on Twitter is provided when sentiment analysis is combined with Twitter. This study analyzed with various deep-learning techniques for the classification of negative and positive elements. Data set SemEval-2017 from Twitter is used to train the final models and will be useful to identify the model which produces the most accurate results.

Key Words: Sentiment analysis, social media, Twitter, tweets, positive, negative, neutral, deep learning

1. INTRODUCTION

Twitterhaddevelopedtoturnintoawellspringof fluctuatedsortofdatabecauseofthenatureofsmall-scale writes on which individuals post continuous messages about their suppositions on an assortment of themes, examines current issues, whine, and express positive assessmentforitemsusedinday-by-daylife. Theprimary goal of this study is to conduct sentiment analysis on tweets utilizing different deep learning algorithms that classifiesthetweetsintothepositiveornegativecategory. If a tweet contains both positive and negative elements, the final message should be chosen based on which element is more prominent. Emojis, usernames, and hashtags in the tweets must be analyzed and converted intoastandardformat.

However analyzing the sentiment expressed is not an easy task. There are several problems in terms of tonality,polarity,lexicon,andtweetgrammar. Itseems to be highly informal and pseudo-grammatical and it's going to be hard to grasp their background. In contrast, the regular use of slang words, acronyms, and vocabulary

words are very popular when posted online. The categorizationofsuchtermsbypolarityisdifficultforthe naturalprocessorsinvolved.Theidentificationofnegative, neutral, and positive tweets are obtained using Bidirectional- Attention-based LSTMs, CNNs, and finetuning Google's pre-trained BERT architecture, which has generallyperformedasastateofartformostNLPtasks.

2. RELATED WORK

Numerous studies have been conducted on fully automated systems that extract features from datasets devoid of human involvement [2] Utilizing novel features likeDALscoresandn-grams,amongothers,thesentiment analysis for categorization was done at the phrasal level. Syntacticdetails'polarity[4,5]wasemployedasafeature. However, this approach need a precise expression border to capture the intended mood. Due to the difference in how words are produced using DAL, which is not a componentofspeech,italsocannotmanagepolysemy.

VADER is a straightforward rule-based model for broad sentiment analysis[6], and contrasted its performance with well-known state-of-the-art benchmarks like SentiWordNet, LIWC, ANEW, and machine learning methods like Naive Bayes and Support VectorMachine(SVM)algorithms.Expressandemphasize sentiment, VADER blends the lexical elements with generalgrammaticalandsyntacticalnorms.

AnalyzingthetweetsinwrittenEnglishthatcomefrom various KSA telecommunications firms[7] and used supervised machine learning techniques for classification to execute opinion mining on them. Gauging the tweet, howessentialawordisusedinaparticulartweet,andalso employed with TF-IDF (Term Frequency-Inverse DocumentFrequency).

Sentiment analysis approaches embedded in public Arabic tweets and Face book comments[8]. Supervised machine learning algorithms such as Support Vector Machine (SVM) and Naïve Bayes, are Binary Model(BM) and TF-IDF returns the effect of several terms weighting functions on the accuracy of sentiment analysis. Using natural language analysis for Arabic language text[9] sentiment analysis is applied on the Twitter dataset of 4700forSaudidialectsentimentanalysiswith(k=0.807).

Egyptian dialect using a corpus such as tweets, and products review data for sentiment analysis[10]. Natural language processing technique is applied to understand theEgyptiandialectandadditionally,usedalexicon-based classification method to classify the data. Using Support Vector Machine for emotion analysis[11], collected optimistic, negative, and neutral tweets from several sources, including the Sentiment140 database. The features are extracted from each message, such as characters n-grams, number of hash-tags, emoticons, etc., for classification and the F1 score of 69.02 is obtained in three-wayclassification. Similarstudiesintheassessment of document classification techniques in [12], Bi-LSTM model returns the strongest result for tweet three-way classification and F-score value is 68.5 using SemEval dataset.

In the course of a few years, sentimental computing hassetfootintheareaofmachinelearningassocialmedia have been used by different abusers. Also, these are required abilities for many Human-computer interaction applications.Lotofstudieshavebeendoneonsentimental andcontentanalysisinacombinedmannerforidentifying and interpreting human emotional messages[13]. Apart from the traditional parsers that were based on normal searching, a greedy parsing technique named Transition based dependency is used for the classification. This parserwillgivemoreaccuracyoftheclassificationresults

3. SYSTEM ARCHITECTURE

Pre-processing of training data uses natural language processing whichisatechniqueaccustomedtoperceiving computer information and handling human interactions. The text comments that are given to the model are additionally being pre-processed. Both pieces of informationarepassedtothesentimentlibrarywherethe feature extraction of the pre-processed information is beingdone.Fromthis,wetendtogetthetrainedmodelfor theknowledgesets.Thesystemarchitectureisdepictedin Fig.1. The classification is finished using LSTM, which is anotherversionoftheRecurrentNeuralNetwork.

The classifier takes the input and then classifies them as positive and negative emotions. These data are then passed onto another classifier for further classification of positive and negative emotions. The positive features are thenclassifiedasenthusiasm,fun,love,happiness,neutral, relief, and surprise. The negative features are classified intoanger,boredom,hate,emptiness,sadness,andworry. The classifier then predicts the output of the test input, which provides the results of the model. The text comments from the tweets undergo pre-processing since it contains URL id. Since we tend not to think about any

address,weeliminatealltheURLsandavoidallunwanted areas.Theseprocessesaredoneinpre–processingstage.

4. METHODOLOGY

4.1 Dataset

Insocialnetworkingservice,Twitterisarealtime messages that lets its users to post called tweets. Tweets have many unique characteristics. Twitter, with nearly 600millionusersandover 250millionmessagesperday, has rapidly turned into a gold mine for organizations to monitor their reputation and brands by extracting and analyzingthesentimentoftheTweetspostedbythepublic about their remarks, markets, and other contenders. Performing Sentiment Analysis on Twitter is complicated thandoingit forlarge reviews.Thisis becausethetweets areveryshortandmostlycontainslangs,emoticons,hash tagsandothertwitterlanguage[16].Thedataconsistsofa large number of tweets collected from the Kaggle repositoryandTwitter.TheTwitterAPIisusedtocreatea Twitterapplicationandgetauthorizationfromtweets.The collected tweet data is in the form of positive as well as negative. Training and Testing dataset consist of both positiveandnegativetweets.

4.2 Preprocessing

MessagesfromTwitteraretooinformal andhave different styles of using tweets based on the nationality, origin,age,andgender oftheuser.Therefore rawTwitter information must be standardized and prepare a formal dataset that can be effectively learned by different classifiers. Most of the preprocessing techniques are

generic and can be used in various applications except Sentiment Analysis. Several pre-processing methods such as removal of URL, usernames, hashtags, character normalization, special character removal like punctuations, numbers, special character, lower casing steps have been applied to standardize the dataset and decreaseitssize.

4.3 Feature Extraction

Terms Presence and Frequency, Parts of Speech (POS),Opinionwordsandphrases,Negationsaresomethe features used for feature extraction process. Individual wordslike specificunigrams,bigramsandn-gramswords with their frequency counts are either uses the term frequencyweightsorgivesbinaryweightingtothewords. The important indicators of opinions of POS are used for finding descriptive words or adjectives from the content. Opinion and phrases words generally used to express opinions like good or bad or hate. Some phrases are expressopinionswithoutusingopinionwords.Generallya document or sentence expresses one opinion orientation like positive or negative about its subject matter. The presence of negative words might change the opinion orientationlikenotbeautifulisequivalenttougly.

4.4 Classification

Sentiment Classification is the binary classification which deals with a small number of classes. Sentiment classification is one of the simple tasks compared to text auto-categorization. While Opinion mining presents various extra tasks other than sentiment polarity detection. The sentiment training set consists of raw tweets labeled positive, negative and neutral. In an effort of making sentiment classifiers, different methods arecompared.

4.4.1 Decision Tree

Adecisiontreeisatypeofclassifierinwhicheach hubornodereferstoatestonanattributeofadatasetand its off-spring refer to the outcomes[18] Decision tree model is applied on test data for node test information. ThebesttestconditionorchoicehasPtobemadeforeach nodeinthetree.GINIfactorisusedtoselecttheidealsplit Foragivenhubornodet,

GINI(t)=1-∑j[P(j|t)] (1)

where (P(j|t)) is the general recurrence or relativefrequencyofclassjatnodet.

4.4.2 Word Embedding

A machine can only understand numbers that converttext to numbers.Butword embeddingtechniques is for converting text into vectors. The word embedding techniquesareusedtorepresentwordsinmathematically. One Hot Encoding,TF-IDF,Word-to-Vec, Fast Textare frequently used Word Embedding methods. One of these techniques in some case is preferred and used according tothestatus,sizeandpurposeofprocessingthedata.

The word embedding tool uses both Bag-ofWordsmodelandtheskip-grammodelisusedforcreating a vector representations of words [19]. Using Neural Networks and Vectorization of text to numbers, word-tovec is a well-liked approach to Natural Language Processing.

With the help of NLP technology, word optimization, learning, and data correctness can be achieved.TheBOWmodel performsmoreaccuratelythan theskip-grammodelforfindingfrequentwords.Inaprior investigation, it was found that word vectors with semantic links enhanced the NLP process of information retrievalandmachinetranslation[20].

4.4.3 Random Forest

The combination of learning algorithm for classification and regression is called Random Forest. Based on the combined decisions of those trees, Random Forestgeneratesasizablenumberofdecisiontreemodels. For a large number of tweets with the individual assessment marks as sentiment labels such as x1, x2,... xn, packingy1,y2,...yn repeatedlyandselectsarandomsample like Xb, Yb with substitution. Every arrangement tree fb is readytouseadifferentarbitraryexample(Xb,Yb),whereb is a number between 1 and B. Finally, a majority vote is cast on these B-trees' forecasts. Using Scikit-Random ForestClassifierLearn'sfromsklearn.ensemble,arandom forestmethodcanbeimplemented.

4.4.4 Support Vector Machine

Support vector machine is commonly referred to asbinarylinearclassifiersandnon-probabilistic.Whenthe feature space is large, this method is recommended for contentcategorization[21].Themaingoalofthisclassifier is to identify the maximum-margin hyper-plane that separates the points with yi = 1 and yi = -1 for a training set of points (xi, yi) where x is the feature vector and y is theclass.Theequationofthehyper-planeisasfollows:

wherewisavectornormaltothehyper-planeandbisan offset. Support Vector Machine can be run with both unigram as well as the combination of unigram and bigram.

4.4.5 BERT

BERT is also known as the Transformers' Bidirectional Encoder Representations was published at the end of 2018 [22]. Pre-training language representationshave beencreatedusingtheBERTmodel, which NLP experts can access and use without charge. In deep learning model, Transformer that is primarily used by BERT to identify the contextual relationships between words or sub-words in the text. The basic design of Transformer's consists of two distinct mechanisms: an encoderthatreadstextinputandadecoderthatcreatesa task prediction. BERT only needs the encoder mechanism becauseitaimstodevelopalanguagemodel.

Contrary to directional models, which read the inputsequentially,the Transformerencoderreadsthefull input sequence only once (right to left or left to right). Therefore, it is referred to as bidirectional. This feature enablesthemodeltocomprehendaword'smeaningbased on all of its neighbors. In some test cases, Transformers model returns better output than the Google Neural Machine. The encoding portion is made up of a stack of encoders. A stack of identical counted decoders makes up thedecodingportion.

The encoders are all comparable in terms of construction. The inputs of the encoder first travel through a self-attention layer, which enables the encoder to view the other terms in the input expression when a given word is encoded. Each of them is divided into two sub-layers.Theoutputsoftheself-attentionlayerserveas theinputsforthefeed-forwardnetwork.Sincethedecoder contains both levels, the attention layer in between them enables it to focus on the pertinent passages in the input paragraph. Self-attention enables to look at specific locations in the input text for indications that may aid to contribute to an improved encoding of the word as the model processes each word. Finding the Value, Key, and Query matrices comes first. The outputs of the selfattentionlayerarecomputedas:

4.4.6 Bidirectional Network

Hochreiter and Schmidhuber first developed LSTMunitsin1997[23]toaddressthevanishinggradient issue. The fundamental idea is to put in place a reliable gating mechanism that will control how much the LSTM units retain the derived features of the fresh data input and maintain the previous state. The input gate and its subsequentweightmatricesWui,Wvi,Wci,andbi;theforget gate and its subsequent weight matrices Wuf, Wv, Wc, and bf;andtheoutputgateanditssubsequentweightmatrices Wuo, Wvo, Wco, and bo, where pi1 denotes the current state of the cell and vi1 state produced by the preceding stage. Thefollowingequationsrepresenthowtochoosewhether theoutputstatecreatedimmediatelyorlater,takeinputs, andforgetthestoredmemory.

4.4.7. Convolutional Neural Network

The process of convolution neural network involves acquiring input data and selecting a set of characteristics from it. Data must be first preprocessed before being converted to a vector format and added to a convolution layer. Convolution layer output is pooled usingmaxpoolingisthemostpopulartechnique.Dropout isappliedafterpoolingtoimproveaccuracy[24].Createa new feature; a convolution algorithm is applied to filter the input while keeping a window of m words. For producefeaturemaps,filterisexpandedtoanywindowof words

Thegeneratedfeaturemapsarethenputthrough a max pooling process where the maximum value of f = max{f}isused.Foreachfeaturemap,themostpertinent featurewiththehighestimportanceneedstoberetrieved. Pooling method is naturally addresses sentences with different lengths. These features are subsequently passed to a fully connected softmax layer, the penultimate layer, whose output is non-normalized and corresponds to a probability distribution across predicted labels. The pretrained word vectors can either be left static while the model'sotherparameterspickupnewinformation,orcan fine-tunethemafterafewtrainingepochsoncetherestof themodel haspickedupsomevaluableinformation. With arestrictiononthel2-normsoftheweightvector,dropout isutilizedforregularizationonthepenultimatelayer.

5. RESULTS AND EVALUATION

WhereKisthekey matrix, Qisthe querymatrixandd: is the dimension of the key vector. Pre-trained BERT model is used from the hugging face transformer library for PyTorch.

The performance of sentiment classification can be evaluated by using four indexes calculated as the followingequations:

Comparing the three trained classifiers, the performance of the classifiers using 1,600,000 tested tweetsfromthesentiment140dataset.

The Precision, Recall, F1-score, and Accuracy of the BERT, LSTM and CNN model are shown in Table I. In BERT-base-uncased model for training, with bias weight decay set to zero, Layer Norm weights and bias set to 0.001, and all parameters set to 0.001 and fine-tuned for 50epochswithan8-batchbatchsizetooperateflawlessly.

studied for sentiment analysis of tweets. Pre-trained Wikipedia language model and the book corpus, which provide a clearer knowledge of the English language and BERTperformbetterthancomparedtootherclassifiers

REFERENCES

[1] Amit G. Shirbhate, Sachin N and Deshmukh, “ Feature ExtractionforSentimentClassificationonTwitterData”in InternationalJournalofScienceandResearchVol 5,2016

[2] A. Agarwal, F. Biadsy, and K. Mckeown, "Contextual phrase-level polarity analysis using lexical affect scoring and syntactic n-grams," in Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009),pp.24–32,2009.

[3] B. Pang and L. Lee, “A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts,” arXiv preprint cs/0409058, 2004.

[4] S. A. El Rahman, F. A. AlOtaibi, and W. A. AlShehri, "SentimentanalysisofTwitterdata,"in2019International Conference on Computer and Information Sciences (ICCIS),pp.1–4,IEEE,2019.

[5] S.-M. Kim and E. Hovy, “Determining the sentiment of opinions,” in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pp.1367–1373,2004.

Similarly, In the Bidirectional Attention-based LSTM's performance using 50-dimensional glove embeddingsanda50-dimensionalhiddendimension,finetune the embeddings after a few epochs when the other layers start picking up important information. In this model trained50 epochs witha batchsizeof16andused Adam with weight decay and Cross Entropy as the optimizer and loss function to compute gradients and performback-propagation[16].

The performance of the Convolutional Neural Network-based sentiment classifier with Glove embeddingsofdimension50withawindowsizeof[3,4,5], and 128 filters in addition to the BERT and Bi-Attentive LSTM classifiers. Adam with weight decay and Cross Entropy,coupledwithfine-tuningoftheembeddinglayer, serveastheoptimizerandlossfunction,respectively.

6. CONCLUSION

Reductionofdatanoiseandimproveaccuracyofa model, three well-known deep learning models are

[6] Hutto C.J., Gilbert E., VADER: A Parsimonious Rulebased Model for Sentiment Analysis of Social Media Text, AAAI,2014.

[7] Qamar, A., Alsuhibany, S. and Ahmed, S. (2017). Sentiment Classification of Twitter Data Belonging to Saudi Arabian Telecommunication Companies. (IJACSA) International Journal of Advanced Computer Science and Applications,[online]

[8] R. M. Duwairi and I.Qarqaz, "A framework for Arabic sentiment analysis using supervised classification", Int. J. Data Mining, Modelling and Management, Vol. 8, No. 4, pp.369-381,2016.

[9] Hossam S. Ibrahim, Sherif M. Abdou, Mervat Gheith, “Sentiment Analysis For Modern Standard Arabic And Colloquial”, International Journal on Natural Language Computing(IJNLC),Vol.4,No.2,pp.95-109,April2015.

[10] Assiri, A., Emam, A. and Al-Dossari, H. (2016). Saudi TwitterCorpus for Sentiment Analysis. International JournalofComputerandInformationEngineering,[online] Available

at:http://waset.org/publications/10003483/sauditwitter-corpus-for- sentiment-analysis [Accessed 1 Mar. 2018].

[11] S. M. Mohammad, S. Kiritchenko, and X. Zhu, "NrcCanada:Buildingthestate-of-the-artinsentimentanalysis oftweets,"arXivpreprintarXiv:1308.6242,2013.

[12] J. Barnes, R. Klinger, and S. S. i. Walde, “Assessing state-of-the-art sentiment models on state-of-the-art sentiment datasets,” arXiv preprint arXiv:1709.04219, 2017.

[13] S. Niklander and G. Niklander, “Combining sentimental and content analysis for recognizing and interpreting human affects,” in International Conference on Human-Computer Interaction, pp. 465–468, Springer, 2017.

[14] L. M. Rojas-Barahona, “Deep learning for sentiment analysis,” Language and Linguistics Compass, vol. 10, no. 12,pp.701–719,2016.

[15] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Proceedings of the 27th Annual Conference on Neural Information Processing Systems(NIPS’13),pp.3111–3119,December2013.

[16]J.Devlin,M.-W.Chang,K.Lee,andK.Toutanova,“Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805,2018.

[17] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in Advances in neural information processing systems, pp. 649–657,2015.

[18]H.KimandY.-S.Jeong,“Sentimentclassificationusing convolutional neural networks,” Applied Sciences, vol. 9, no.11,p.2347,2019.