Explainability-Driven SMS Spam Detection A Comparative Analysis of Machine Learning Classifiers with

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Explainability-Driven SMS Spam Detection A Comparative Analysis of Machine Learning Classifiers with SHAP Interpretability

N Janani Yadav1, Vinay Vardhan T2, Rawan Fatehi Ahmed Al-Utairi3

1-3

Bachelor of Computer Application [Cybersecurity] Student, Jain (deemed-to-be) University, Bengaluru, India

Abstract - SMS spam remains a pervasive threat to mobile users worldwide, yet most detection systems identify maliciousmessages without providingexplanationsfortheir decisions, limiting the ability of security engineers to audit and refine filters. This paper presents a systematic comparative study of five classifiers Naive Bayes, Support Vector Machine (SVM), Random Forest, XGBoost, and DistilBERT evaluated on the UCI SMS Spam Collection (5,572 messages). The primary contribution is a multimodel SHAP [1] interpretability analysis applied to all four classical classifiers simultaneously, enabling the first crossmodel feature importance comparison on this task. DistilBERT achieves F1 = 0.9660 and AUC = 0.9952; SVM achieves the best cross-validated F1 among classical models (0.9511 ± 0.0049). SHAP analysis identifies “free”, “call”, and “reply” as robust cross-model spam indicators, while also uncovering model-specific patterns that are actionable for filter design. All classical classifiers are implemented in scikit-learn[13]andexperimentsarefullyreproducible.

Key Words: SMS spam detection, SHAP explainability, DistilBERT, machine learning, NLP, TF-IDF, text classification, interpretable AI, mobile security.

1. INTRODUCTION

SMS spam poses a persistent and growing threat to mobile users worldwide. The UCI SMS Spam Collection captures authentic spam patterns from UK reporting servicesandtheSingaporeNUSSMSCorpus,underscoring theglobalscaleofunwantedSMScampaigns.Usersacross regions including India, the US, and the UK are frequent targets, motivating the development of robust automatic detection methods. Unlike email spam, which benefits from rich signal sources such as URL analysis and attachment inspection, SMS spam detection relies primarily on message text, making text classification the principal detection paradigm. Traditional approaches based on keyword blacklists or Naive Bayes classifiers identify messagescontainingknownspamvocabulary but arevulnerableto evasion byadversarieswhorephrase or obfuscatespamcontent.Amorefundamental limitationis the opacity of these models: they do not indicate which words or linguistic features triggered a classification decision, preventing security engineers from diagnosing failures,identifyingemergentattack patterns,orupdating filters in response to new spam tactics. Explainability methods, particularly SHAP [1] and LIME [6], have been appliedtotextclassificationinadjacentdomainsincluding

sentiment analysis and phishing detection [7], but systematic multi-classifier SHAP analysis for SMS spam detection has not been reported in the literature. This paper addresses that gap. We conduct a five-model comparative study and apply SHAP [1] to all classical classifiers simultaneously, enabling the first cross-model feature importance comparison on the SMS spam detection task. All classical classifiers are implemented using scikit-learn [13] with TF-IDF features; DistilBERT [5] is fine-tuned using the HuggingFace Transformers library[14].

Theprimarycontributionsofthispaperare:

• Asystematicfive-modelcomparison:NaiveBayes, SVM, Random Forest, XGBoost, and DistilBERT are evaluated on the UCI SMS Spam Collection [2] using stratified 5-fold cross-validation for classical models and held-out test evaluation for all models, ensuring reproducibleandstatisticallygroundedresults.

• A systematic cross-model SHAP [1] analysis identifying both model-agnostic and model-specific spam indicators, providing actionable insights for interpretable filterdesign.

• A fine-tuned DistilBERT baseline achieving F1 = 0.9660 and AUC = 0.9952, establishing a strong deeplearningreferencepointfortheSMSspamdetectiontask.

• A misclassification analysis identifying three operationally significant failure modes, yielding concrete recommendations for improving detection robustness acrossmodelfamilies.

2. Related Work

Almeida et al. presented the UCI SMS Spam Collection [2] as a carefully chosen benchmark for SMS filter assessment. Their initial Naive Bayes experiments [12] showed that basic probabilistic On this task, classifiers achieve high precision. Later, Hidalgo et al. [3] demonstrated that SVM [11] with TF-IDF features outperforms Naive Bayes in terms of recall, especially for messages that avoid canonical spam. vocabulary as a first sign that model diversity is beneficial. Text classification hasmadeextensiveuseofensembleandgradientboosting techniques, such as Random Forest [10] and XGBoost [9], with competitive outcomes. While Chen and Guestrin's XGBoost [9] provides precise SHAP support via TreeExplainer [1], making it especially appropriate for explainability-centered studies, Breiman's Random Forest

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

[10] offers natural feature importance through mean decrease in impurity. BERT [8], which showed that deep bidirectional pre-training significantly enhances downstream classification, marked the beginning of the transformer era. With 40% fewer parameters, DistilBERT [5] maintains 97% of BERT's language comprehension, makingfine-tuningfeasibleoncommonhardware.Kim[4] demonstrated the usefulness of CNNs for sentence classification, bridging the gap between neural and classicalmethodsforshort-texttasks.

LIME [6] and SHAP [1] have been used to study explainability for text classifiers. Explainability techniques offeroperationallyusefulinsightbeyondaccuracymetrics, as demonstrated byapplications to phishingdetection [7]. The work's natural language processing foundations rely onwell-knownmachinelearningtools[13].Usingthefirst multi-model SHAP comparison across four classifier families implemented in scikit-learn, this paper expands theexplainabilityliteraturetoSMSspamdetection[13].

3. Dataset and Preprocessing

3.1 UCI SMS Spam Collection

TheUCISMSSpamCollection[2],a publiclyaccessible benchmark of 5,572 actual SMS messages classified as spam (747, 13.4%) or ham (4,825, 86.6%), is used in all experiments. Messages were gathered from the Singapore NUS SMS Corpus and UK-based spam reporting services. Stratifiedsamplingisusedtomaintaintheclassimbalance, which represents actual production traffic. The dataset statisticsaresummarisedinTable1.

Table – 1: UCISMSSpamCollection DatasetStatisticsand Train/TestSplit[2]

3.2 Preprocessing

A standard NLP pipeline was used to preprocess each message [13]: (1) lowercasing; (2) URL normalisation (replacing all URLs with the token URL); (3) number normalisation (replacing digit sequences with NUM); (4) non-alphabetic character removal; and (5) whitespace normalisation. TF-IDF feature extraction [13] employed sublinearTFscaling,10,000 maximum features, unigrams and bigrams (ngram_range=(1,2)), and a minimum document frequency of 2. In order to maintain subword

structureforDistilBERT[5],rawlowercasedtextwassent straighttotheDistilBERTtokeniser.

4. Methodology

4.1 Classifiers

• Naïve Bayes

Multinomial Naive Bayes [12] is configured with Laplace smoothing parameter α = 0.1. This value is selected over the default α =1.0toavoidover-smoothingon thelimited SMS vocabulary, while still providing sufficient regularizationforunseentokens.

• Support Vector Machine

PlattprobabilitycalibrationusinglinearSVM[11](C=1.0) (CalibratedClassifierCV [13]). In high-dimensional TF-IDF spaces,linearkernelsworkincrediblywell.AUCcalculation and threshold modification are made possible by probabilitycalibration.

• Random Forest

200decisiontreesinanensemble[10](n_estimators=200, random_state=42). Random Forest offers mean decrease impurity importance as a reference for SHAP comparison and is resilient to irrelevant features through random featuresubsampling.

• XGBoost

Gradient boosted trees [9] are configured with maximum depth 6, learning rate 0.1, and 200 estimators. A key advantage of XGBoost in this study is its native compatibility with SHAP’s TreeExplainer [1], which computes exact SHAP values directly from the tree structure. This provides more precise attribution than the PermutationExplainer approximations required for other classicalmodels.

• DistilBERT

DistilBertForSequenceClassification [5] is fine-tuned for 3 epochs with a batch size of 32, a learning rate of 2e-5, a 10% warmup schedule, and gradient clipping at 1.0. DistilBERTretains97%ofBERT’slanguagecomprehension capacity with 40% fewer parameters, making fine-tuning feasible on standard hardware. Training loss decreased monotonically from 0.1728 (epoch 1) to 0.0309 (epoch 2) to0.0143(epoch3),confirmingstableconvergence.

4.2 Evaluation Protocol

All classical classifiers [13] were evaluated using stratified 5-fold cross-validation on the training set, yielding mean ± standard deviation F1 scores. Final heldoutperformance(Precision, Recall,F1-Score,AUC-ROC)is reported on the 20% test set for all five models. DistilBERT[5]wasevaluatedontheheld-outtestsetonly; 5-fold cross-validation was not applied due to the

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

prohibitive computational cost of repeated fine-tuning runs. This asymmetry is acknowledged as a limitation: direct comparison between DistilBERT’s test-set metrics andtheclassical models’cross-validatedF1scoresshould beinterpretedwiththiscaveatinmind.

Chart - 1: Stratified5-FoldCross-ValidationF1Score(mean± standarddeviation)forclassicalclassifiers[13].

4 3 SHAP Explainability

SHAP [1] computes feature importance grounded in cooperative game theory, ensuring additive consistency and local accuracy. SHAP values were computed on 150 random test samples using: (a) TreeExplainer [1] for XGBoost [9] exact values from tree structure; (b) PermutationExplainer [1] for Naive Bayes [12], SVM [11], and Random Forest [10] applied to the top-500 highestvarianceTF-IDFfeatures.MeanabsoluteSHAPvalueswere aggregated per feature and top spam indicators extracted permodel.

5. Experimental Results

5.1 Classification Performance

The complete evaluation results for each of the five classifiers on the UCI SMS Spam Collection are shown in Table 2 [2]. With the highest F1 (0.9660) and AUC (0.9952), DistilBERT [5] demonstrates that contextual language modelling offers a significant benefit. SVM [11] achieves the best F1 (0.9485) and the most stable crossvalidated performance (0.9511 ± 0.0049) among the classical models implemented with scikit-learn [13]. The performance comparison of various classical classifiers is showninChart-2

Chart - 2: Precision,Recall,andF1Scoreofclassicalclassifiers evaluatedontheheld-outtestset(1,115messages;149spam, 966ham).

Table – 2: Showstheclassificationresultsforeachofthefive models(testset:1,115messages;149spam,966ham).

Naive Bayes [12] achieves perfect precision (1.0000), producing no false positives, but its recall of 0.8859 indicatesthatapproximatelyoneinninespammessagesis misclassified as ham. This behaviour reflects the model’s conservativeposteriorestimates,whichareadvantageous in environments where false alarms are especially costly. Random Forest [10] and XGBoost [9] also achieve high precision,thoughwithlowerrecallthanSVM,suggestinga more conservative decision boundary on this dataset. DistilBERT [5] achieves the highest recall: of 149 spam messagesinthetestset,itcorrectlyidentifies142,missing only 7. This superior sensitivity results from contextual sentence-level encoding, which enables the model to recognise adversarial phrasing that defeats feature-based classifiers. As shown in Chart - 3, training loss decreases monotonically across all three epochs, confirming clean modelconvergence.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Chart - 3: Precision,Recall,andF1Scoreofclassicalclassifiers evaluatedontheheld-outtestset(1,115messages;149spam, 966ham).

5.2 SHAP Feature Importance

The top 5 SHAP [1] spam indicator terms based on table 3 and shown in Chart - 4 via a full set of 12 SHAP shapes for each classical classifier are the focal point of thispaper’scentralnovelcontribution.

Chart - 4. SHAPtop-5spamindicatortermsperclassical classifier(mean|SHAPvalue|,150testsamples)[1].Sub-figures arelabeled(a)NaiveBayes,(b)SVM,(c)RandomForest,(d) XGBoost.

Table - 3: SHAPTop-5SpamIndicatorWordsperClassifier (Mean|SHAPValue|,150testsamples)[1].

Three common words were present in the top 5 list of three or more models: free, call, and reply/txt. These words are a model agnostic spam indicator and have consistentmeaningacrossNaiveBayes[12],SVM[11],and RandomForest[10].Inadditiontothesetopwords,there are patterns of additional interest associated with each model; for example, Naive Bayes [12] considered two separate bigrams associated with video phone as being highly relevant to premium rate video content related spams.Furthermore,ananalysisofthedecisionboundary XG-Boost [12] used for its classification of the shortcodes appearstodemonstratethattheuseof“gt”and“uk”inthe spam text messaging classification app as well as the geographic identifiers that correspond to all three spam types werethe most dominantfeaturesof spam andham. Thesedifferencessuggestthateachalgorithmfamilyused for spam classification exploits distinct features from the samelinguisticboundary.

5.3 Misclassification Analysis

The primary source of classification error across all models is false negatives (missed spam). three principal false-negative categories are identified: (1) spam messages containing no monetary or prize-related vocabulary that evade keyword-based classifiers [12]; (2) spam employing personalised conversational language designedtoresemblelegitimatemessages,bypassingSVM [11] and Random Forest [10]; and (3) spam embedded within ostensibly benign notification formats such as appointmentremindersordeliveryalerts.Fig-1presents confusionmatricesforallfourclassicalclassifiers.

Fig – 1: Confusionmatricesforallfourclassicalclassifiersonthe held-outtestset(1,115messages;149spam,966ham).Subfigures:(a)NaiveBayes,(b)SVM,(c)RandomForest,(d) XGBoost.

DistilBERT [5] uses contextual encoding at the sentence levelofsometypesof"adversarial"messageswhichisthe reason why it has the highest recall rate. Other than on legitimatemessages(3–4falsepositivespermodelon966 ham),thereareveryfew falsepositivesacross all models, mostly occurring for genuine ham messages with words like"free"or"call"intheirtrulynormaluse.

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

6. Discussion

6.1 Practical Model Selection

Thefiveclassifiersexhibitdistinctperformancetradeoffs across precision, recall, and F1, as summarised in Chart – 5. Based on these findings, the following deploymentrecommendationsareproposed:

Chart – 5: All5classifiersperformanceComparison(Precision, Recall,F1).DistilBERTachieveshighestF1=0.9660.

 High-precision environment: Naive Bayes [12]; the onlymodel with1.0000precisionandnofalsealarms but missed 11% of spam; <1 ms/message for inference(fastest).

 Balanced consumers: SVM [11]; best classical F1 (0.9485)andlowestcross-validationvariance(0.9511 ± 0.0049; consistent performance) with low false positiverate.

 Maximum recall environment: DistilBERT [5]; highest overall F1 (0.9660) and AUC (0.9952), with recall of 0.9530 owing to contextual sentence-level encoding thatcapturesadversarialphrasinginvisibletofeaturebased classifiers. Recommended where missed spam carriesthehighestoperationalcost.

 Resource-constrained deployment: Naive Bayes or SVM;bothmodelsoccupylessthan10MBandachieve sub-millisecond per-message inference latency, makingthemsuitableforon-devicereal-timefiltering.

6.2 Actionable Insights for Filter Engineers

SHAP[1]explainabilityvaluesprovidefilterengineers with principled, feature-level decision evidence during filter development and auditing. The per-model feature rankings indicate: (1) which keywords should be prioritised in updates to rule-based filters; (2) which ngramswarrantmonitoringaspotentialadversarialbypass vectors; and (3) which features are suitable for constructing hybrid systems that combine rule-based and machine-learning components. The finding that XGBoost [9] assigns high weight to geographic shortcode tokens (“gt”,“uk”),whileSVM[11]assignshighweighttoopt-out instructions,demonstratesthatdifferentclassifierfamilies exploit distinct properties of the spam-ham decision boundary complementary signal that accuracy metrics alonewouldnotreveal.

6.3 Limitations

Four limitations with this analysis of the SHAP [1] model exist. One, the UCI [2] dataset presented in this report is from a UK and Singapore service from the late 2000s.Therefore,performanceoftheSHAP-modelsshould be verified on contemporary or geographically diverse datasets to ensure continued accuracy of identified spam detectionfeatures.Two,theSHAPdatasetdoesnotcontain adversarial spam that was created to avoid detection throughtheuseoftheSHAP-identifiedfeatures.Three,the computational constraints caused the PermutationExplainer forNaiveBayes[12], SVM[11]and Random Forest [10] to be restricted to only the top-500 variance features for the datasets used and therefore fewer SHAP features for each machine learning type can beverified.Four,theanalysisoftheDistilBERT[5]dataset did not include cross validation and represents a limitation for making direct comparisons with classical models[13]usingcrossvalidationanalysis.

7. CONCLUSIONS

This research provides a comparative analysis of SMS spam filtering using five models and leverages SHAP [1] for explainability purposes. We employed the UCI SMS Spam Collection Dataset [2] and scikit-learn [13] to compare five different spam detection models: Naive Bayes [12], SVM [11], Random Forest [10], XGBoost [9], and DistilBERT [5]. Of these five models, DistilBERT exhibited the best overall performance based on F1 score (0.9660)andAUC(0.9952)metrics,whileSVMperformed best with respect to the precision and recall metrics, as well as to the cross-validated stability of the model (0.9511±0.0049).

In our cross-model analysis using SHAP [1], we identified three robust spam indicators that are model agnostic, namely 'free', 'call', and 'reply', in addition to identifyingmodelspecificpatternsthatprovideactionable insightsnotcapturedbysimplyevaluatingtheaccuracyof eachmodel.Thus,thisresearchprovidesamethodologyto convert the spam detection process into an interpretable andauditabledecision-makingprocess,therebyextending thelong-standingexplainabilitytraditionsofLIME[6]and SHAP[1]intotherealmofSMSspamdetection.

FuturedirectionswillentailtheapplicationofSHAP[1] to fine-tuned transformer models [5], utilizing integrated gradients, evaluating adversarial robustness to the identified SHAP indicators, and extending the methodologytomultilingualSMSdatasetsbeyondtheUCI dataset[2].

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

REFERENCES

[1] S. M. Lundberg and S.-I. Lee, "A Unified Approach to Interpreting Model Predictions," in Proc. NeurIPS, vol. 30,2017,pp.4765–4774.

[2] T. A. Almeida, J. M. G. Hidalgo, and A. Yamakami, "Contributions to the Study of SMS Spam Filtering: New Collection and Results," in Proc. ACM DocEng, 2011,pp.259–262.

[3] J.M.G.Hidalgo,G.C.Bringas,E.P.Sánz,andF.C.García, "Content Based SMS Spam Filtering," in Proc. ACM DocEng,2006,pp.107–114.

[4] Y. Kim, "Convolutional Neural Networks for Sentence Classification,"inProc.EMNLP,2014,pp.1746–1751.

[5] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT,aDistilledVersionofBERT:Smaller,Faster, CheaperandLighter,"arXiv:1910.01108,2019.

[6] M. T. Ribeiro, S. Singh, and C. Guestrin, "Why Should I Trust You?: Explaining the Predictions of Any Classifier,"inProc.ACMSIGKDD,2016,pp.1135–1144.

[7] F.Heiding,B.Schneier,A.Vishwanath,andJ.Bernstein, "Devising and Detecting Phishing: Large Language Modelsvs.SmallerHumanModels,"IEEEAccess,vol.12, pp.1402–1417,2024.

[8] J.Devlin,M.-W.Chang,K.Lee,andK.Toutanova,"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proc. NAACL-HLT, 2019, pp.4171–4186.

[9] T. Chen and C. Guestrin, "XGBoost: A Scalable Tree Boosting System," in Proc. ACM SIGKDD, 2016, pp. 785–794.

[10]L. Breiman, "Random Forests," Machine Learning, vol. 45,no.1,pp.5–32,Oct.2001.

[11]C. Cortes and V. Vapnik, "Support-Vector Networks," Machine Learning, vol. 20, no. 3, pp. 273–297, Sep. 1995.

[12]A. McCallum and K. Nigam, "A Comparison of Event Models for Naive Bayes Text Classification," in Proc. AAAI Workshop on Learning for Text Categorization, 1998,pp.41–48.

[13]F. Pedregosa et al., "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12,pp.2825–2830,Oct.2011.

[14]T. Wolf et al., "Transformers: State-of-the-Art Natural Language Processing," in Proc. EMNLP (System Demonstrations),2020,pp.38–45.