MultiTraitBERT: A BERT-Based Multi-Trait Essay Scoring System with SHAP and Attention Explainability

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

MultiTraitBERT: A BERT-Based Multi-Trait Essay Scoring System with SHAP and Attention Explainability

Sowmya PR1 , Srinidhi Avantikaa R2, Shriya Hosangadi3, Rishitha Rasineni4 , Dr Savitha G5

1,2,3,4Computer Science and Engineering, RV Institute of Technology and Management, Bengaluru, India 5Professor, Computer Science and Engineering, RV Institute of Technology and Management, Bengaluru, India

Abstract - Automated Essay Scoring (AES)hasemergedas a critical application of Natural Language Processing (NLP), offering scalable, consistent, and objective evaluation of student writing. This paper presents a comprehensive multitrait essay scoring system built upon BERT (Bidirectional Encoder Representations from Transformers) applied to the Hewlett Foundation Automated Essay Scoring Dataset, specifically Sets 7 and 8. Our architecture employs a shared BERT encoder coupled withN independentregressionheads one per writing trait enabling simultaneous prediction of traits. To ensure the model’s interpretability and explainability, we incorporate two complementary analysis methods: attention weight visualization and SHAP (SHapley Additive exPlanations). The results are also comparedagainst statistical classical baselines such as Support Vector Regression(SVR)andHuberRegression,showing“significant improvements”inQuadraticWeightedKappa(QWK)andR2. The proposed BERT model obtains (mean) QWK scores above 0.75 for all traits, outperforming baselines by a large margin. Explainability analysis show that SHAP explanations provide actionable feedback for students and instructors.

Key Words: Automated Essay Scoring, BERT, Multi-Trait Scoring,SHAPExplainability,NaturalLanguageProcessing, Deep Learning, Quadratic Weighted Kappa, Transfer Learning, Educational AI, Hewlett Foundation Automated EssayScoringDataset

1.INTRODUCTION

Human essay scoring is time consuming, inconsistent and unreliable. Besides, the human scoring approach has no transparency,andthusitisnoteasyforstudentstogauge the reasoning used in awarding their scores. Automated EssayScoring(AES)isapredictionofessayscoresthrough Machine Learning and Deep Learning models. Most real scenarios require the AES system to perform multi-trait scoringbasedondifferentcriteriasuchasthecoherenceof theessay,grammaticalaccuracy,andcontentamongmany others.

Multi-traitessayscoringismorecomplexascomparedtoa mere prediction of a single score because the model must assessmultipletraitsofwritingsimultaneously.Traditional statistical methods rely on manually crafted features that havelimitedabilitytounderstandthesemanticsandcontext of text. Lastly, explainability is a core aspect of designing modelsinaneducationalcontext.

Althoughrecentdeeplearningapproaches(e.g.,BERT)have demonstrated superior performance in text-based tasks, their relative lack of use in multi-trait scoring and explainability remains. In this work, we propose MutiTraitBERT,aframeworkforsimultaneouslypredicting scoresinmultipletraits.Toenhancetransparency,wealso integrate explainability strategies that enable us to draw insightful observationsabouthowa particularsegment of theessayimpactstheoverallscoreusingSHAPandattention mechanisms.

Finally, we evaluate our approach based on traditional statisticalmodelsusingR2andQuadraticWeightedKappa (QWK), demonstrating an improved performance with interpretablepredictions.

2. RELATED WORK

There have been manyadvancements in AutomatedEssay Scoring (AES), ranging from classical machine learning techniquestostate-of-the-artdeeplearningalgorithmsand transformerarchitectures.Conventionalmodelsfocusedon designingmanualfeaturesalongwithregressionandsupport vector machines for predicting the scores of essays. While thesemethodswereeffectivetosomeextent,theyrequired extensive feature engineering and struggled to capture deepersemanticmeaninginessays.

To address these limitations, neural network-based approaches were introduced. A notable work used Long Short-Term Memory (LSTM) networks to learn essay representationsdirectlyfromrawtext,eliminatingtheneed

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

for manual feature extraction and improving the ability to model sequential dependencies [1]. Building on this, attention-basedmodelswereproposedtoidentifyimportant wordsandsentencesthatcontributemoretothefinalscore, thereby improving performance and interpretability [2]. Furthermore, multi-trait learning approaches were introducedtoevaluate multiple aspectsofwriting,suchas content,organization,andlanguagequalitysimultaneously, makingthescoringprocessmorecomprehensive[3].

Inadditiontodeeplearningmodels,hybridapproacheshave also been explored. These methods combine handcrafted linguistic features with neural representations to capture bothstructuralandsemanticinformation.Suchcombinations havebeenshowntoimproveperformancecomparedtousing eitherapproachalone[4],[5].

Withtheadvancementoftransformer-basedmodels,BERT has become a widelyadopted technique for AES due toits strongcontextualunderstanding.However,mostBERT-based approachesrelyonlyonsupervisedlearning.Toovercome this limitation, a BERT-driven Deep Self-Supervised Contrastive Learning Network (D2SCLN) was proposed, which integrates self-supervised learning and contrastive learningtobetterutilizeunlabelleddataandimprovemodel robustness [6]. Similarly, contrastive learning has been furtherexploredinSimCSE,wheresentenceembeddingsare learned by bringing similar sentences closer and pushing dissimilar ones apart. This method uses dropout-based augmentation in the unsupervised setting and NLI-based supervision in the supervised setting, achieving strong performanceimprovementsonsemanticsimilaritytasks[7].

Recentresearchhasalsoexploredtheuseoflargelanguage models (LLMs) for AES and essay revision tasks. These models use zero-shot and few-shot learning techniques, allowingthemtoperformscoringwithoutheavyrelianceon labeleddatasetswhilemaintainingcompetitiveperformance [8].

Another important direction focuses on improving model generalization across different essay prompts. Domain adaptationtechniqueshavebeenproposedtolearndomaininvariantfeatures,enablingmodelstoperformwellacross different datasets [9]. Similarly, meta-learning approaches havebeenintroducedtohandlecross-promptvariationsand improveadaptabilitytonewessaytopics[10].

Inadditiontoimprovingperformance,recentstudieshave also emphasized the importance of model interpretability. Lundbergand Lee proposedSHAP,a unified explainability framework based ongame theory thatassigns importance

values to individual features, helping to understand how models make predictions [11]. This approach provides consistent and locally accurate explanations, making it suitableforanalysingcomplexmodelsusedinAES.

Despitetheseadvancements,severalchallengesstillremain, includinglimitedutilizationofunlabelleddata,difficultyin capturingfine-graineddifferencesbetweenessayscores,and lack of transparent explanations in deep learning models. These limitations highlight the need for more robust, explainable,andmulti-traitscoringsystems.

3. DATASET AND PREPROCESSING

3.1 Dataset Overview

The dataset used in this study is the Hewlett Foundation Automated Essay Scoring Dataset, a widely recognized benchmarkforAutomatedEssayScoring(AES)[12]tasks. ThisdatasetwasreleasedasapartofaKagglecompetition and was sponsored by the Hewlett Foundation, aiming to developmachinelearningmodelscapableofevaluatingand scoring essays with performance comparable to human raters.

Thedatasetconsistsofapproximately12,976essayswritten by students across eight different essay prompts. Each prompt corresponds to a unique writing task, resulting in variations in essay length, style, and scoring criteria. The diversity of prompts ensures that models trained on this datasetgeneralizewellacrossmultiplewritingdomains.

For this study, specific essay sets (Set 7 and Set 8) were selected.Thesesetsdifferinboththenumberoftraitsand scoring ranges, introducing structural heterogeneity that mustbeaddressedduringpreprocessing.

i) Dataset Structure

Thedatasetisprovidedinastructuredtabular format,whereeachrowcorrespondstoasingle essay.Set7containsfourtraitsperessay,while Set8containssixtraits.

KeyattributesofSet7include:

• Ideas

• Organization

• Style

• Convention

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

KeyattributesofSet8include:

• Ideas

• Organization

• Voice

• WordChoice

• SentenceFluency

• Conventions

Each trait represents a specific dimension of writing quality, such as ideas, organization, style, and conventions. The presence of multiple traits makes the task a multi-output regressionproblem.

ii) Scoring Scheme

Thescoringschemevariesacrossessaysets.

• InSet7,traitscoresrangefrom0to3,and theoveralldomainscorerangesfrom0to 30.

• In Set 7,traitscoresrangefrom0to3,and theoveralldomainscorerangesfrom0to 30.

This variation in scoring scales introduces challengesinmodeltraining,asrawscoresare notdirectlycomparableacrosssets.

iii) Annotation Process

Each essay in the dataset is evaluated independentlybytwohumanraters.Scoresare assignedforeachtraitaswellasfortheoverall domainscore.Theuseofdualratersimproves thereliabilityandconsistencyofannotations.

Toobtainasinglerepresentativescoreforeach trait,thescoresfrombothratersarecombined duringpreprocessing.Thisreducessubjectivity and provides a more stable ground truth for supervisedlearning.

3.2 Preprocessing

i) Dataset Refinement

TheHewlettFoundationAutomatedEssayScoring Dataset was filtered to process individual essay setsseparately.Weremovedirrelevantattributes including rater3_domain1, rater1_domain2, rater2_domain2, domain2_score, and all rater3_trait, retaining only the primary scoring

features.Thisreducesredundancyandensuresthat the dataset contains only meaningful variables relatedtoessayevaluation

ii) Text Cleaning

The essay text was standardized to eliminate inconsistencies and noise. Cleaning involved removing leading and trailing whitespaces, replacingmultipleconsecutivespaceswithasingle space and converting HTML entities into textual representations

iii) Essay Segmentation

We divided each essay into three parts to understanditbetter:

• Introduction

• Body

• Conclusion

Thiswasdonebasedonsentencepositions,where thebeginningpartistheintroduction,themiddle part is the body and the ending part is the conclusion.Thishelpsthemodellearnhowessays are organized. It is especially useful for checking traits like organization and flow. By keeping this structureinsteadoftreatingtheessayasonelong text,themodelcanbetterunderstandthequalityof writing.

iv) Trait Score Construction

Every essay in the dataset contains two scores assignedbytwohumanratersformultipletraits. The average of the two scores was computed to obtainasinglereliablescore.

v) Score Normalization

Due todifferencesinscoringrangesacross essay sets,set-specificnormalizationwasapplied:

• For Set 7,traitscoreswerenormalizedby dividing by 3, and domain scores were normalizedbydividingby30.

• For Set 8, trait scores were normalized using min-max scaling (x−1)/5(x1)/5(x−1)/5, and domain scores were normalizedbydividingby60.

Thisset-awarenormalizationensuresthatallscores are mapped to a common [0,1] range while preserving their relative distributions, enabling consistentandstablemodeltraining.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

vi) Feature Engineering

Toenhancetherepresentationofessays,additional structural features were extracted from the text. These include word, average word length, and sentence count in the detailed preprocessing pipeline.

These features capture structural aspects of writing, such as length, complexity and sentence organization, complementing textual representations.

vii)Data Filtering

Essays with very low word counts (less than 35 words)wereremoved.Suchessaysdonotcontain sufficient information for meaningful evaluation and may negatively impact model performance. This step improves overall data quality and robustness.

viii) Data Splitting Strategy

The dataset was divided into three subsets to ensurea robust model trainingand evaluation.A stratified splitting approach was adopted, where the data was partitioned into 70% training, 15% validation, and 15% testing sets using a fixed randomseedforreproducibility.

ix) Text Representation using TF-IDF

The dataset was divided into three subsets to ensurea robust model trainingand evaluation.A stratified splitting approach was adopted, where the data was partitioned into 70% training, 15% validation, and 15% testing sets using a fixed randomseedforreproducibility.

The feature space was limited to approximately 2000featuresforSet7and4000featuresforSet8, ensuring a balance between representational richnessandcomputationalefficiency.

x) Feature Scaling and Dimensionality Reduction

Numerical features were standardized to ensure equal contribution during model training. Due to the high dimensionality of TF-IDF vectors, dimensionality reduction was performed using TruncatedSingularValueDecomposition(SVD).

AlowernumberofcomponentswasusedforSet7, while higher components were used for Set 8 to capture increased linguistic variability. This step reduces computational complexity while preservingthemostinformativefeatures.

4. METHODOLOGY

4.1 Statistical Models

1) HuberRegression

TheHuberRegressorisalinearregressionmodel thatis designed to be lesssensitive to outliers by combining two typesofloss:MeanSquaredError (MSE) and Mean Absolute Error (MAE). The Regressorbehaveslikealinearregressorformost datapointswhilereducingtheinfluenceofextreme values which results in more stable and reliable predictions. The model also provides a scale parameterthatautomaticallyadjustsforchangein themagnitudeforthetargetvalue.Itisparticularly useful for datasets where most data points are concentratedinacentralregionwithafewoutliers, asiteffectivelycapturesthemaintrendofthedata without being distorted by those extreme values [13].

2) SupportVectorRegression

SupportVectorRegressionmodel(SVR)withradial basis function (RBF) is used to establish a strong baseline for automated essay scoring. The SVR modelwastrainedseparatelyforeachscoringtrait, therebytreatingthemuliti-traitpredictionasaset of separate regression problems. The non-linear relationshipsbetweentheinputfeaturesandessay scoreswerecapturedbytheRBF.SVRwaschosen duetoitseffectivenessinhandlinghighdimensional featurespacesandduetoitsabilitytoportraynonlinearrelationships[14].

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

4.2 Multi-Trait BERT

The Multi-Trait BERT model is a deep learning-based approach used to predict multiple essay scoring traits simultaneouslybyleveragingthecontextualrepresentation capabilitiesofBERT[15].Thepre-trainedbert-base-uncased modelisfine-tunedforamulti-outputregressiontask,where asharedencoderlearnsrichsemanticandsyntacticfeatures fromtheessaytext.

The input essays are tokenized and converted into fixed maximum-length sequences, and the contextual representation is obtained from the [CLS] token. This representation is passed through multiple parallel regressionheads,eachcorrespondingtoaspecifictrait.Each headconsistsoffullyconnectedlayerswithReLUactivation and dropout, followed by a sigmoid function to produce normalized scores in the range [0,1]. This architecture enables simultaneous prediction of multiple traits while capturingsharedandtrait-specificinformation[15].

ThemodelistrainedusingtheHuberlossfunction,which provides robustness to outliers and ensures stable optimization. The AdamW optimizer with learning rate scheduling is used for efficient convergence. Additionally, attentionweightsextractedfromthefinallayerofBERTare usedtoidentifyimportantpartsoftheessay,whileSHAPbased analysis is used to interpret the contribution of individualwordstothepredictedscores[6,15].

Fig. 2:Multi-TraitBERTmodelarchitectureforessay scoring

5. EVALUATION METRICS

5.1 R2 Score:

The R2 score is a metric used to evaluate how well a regressionmodelfitsthedata.Itmeasurestheproportionof variance in the target variable that is explained by the model’s predictions. It indicates how much of the data’s variabilityiscapturedbythemodel.Thescorerangesfrom0 to1wherecloserto1indicatesabetterfitwhileavalueof0 indicates that the model performs no better than simply predictingthemeanofthetargetvalue[16].

5.2 QWK Score:

Quadratic Weighted Kappa (QWK) is used as one of the primary evaluation metrics to assess how closely the predicted essay scores correspond to the actual assigned scores.Unlikesimpleaccuracymeasures,QWKconsidersthe extent of difference between predicted and true scores, assigninggreaterimportancetolargerdeviations.Thismakes it particularly suitable for essay scoring tasks, where the relativedifferencebetweenscoresisimportant.Themetric ranges from −1 to 1, where a value of 1 indicates perfect correspondence,while0representsagreementequivalentto chance.Itiswidelyadoptedinautomatedessayscoringtasks as it better reflects consistency with human evaluation comparedtostandardregressionmetrics[17].

TheCohen’sKappacoefficientiscalculatedasfollows:

κ=(pₒ-pₑ)/(1-pₑ) (eq2) wherepₒistheobservedproportionateagreementandpₑis thehypotheticalprobabilityofchanceagreement.

6. RESULTS

Weevaluatestatistical models(HuberRegressorandSVR) and a multi-trait BERT model across multiple feature sets usingR²andQuadraticWeightedKappa(QWK).

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Table -1: PerformanceofHuberRegressorandSVRonSet7 usingR2 andQWK

Trait Huber R2 Huber QWK

TablesIandIIshowtheperformanceofHuberRegressorand SVRformulti-traitscoringonSet7andSet8usingR²and QWK. For Set 7, both models perform well, however SVR slightlyoutperformsHuberinmosttraits,especiallyinTrait1 wherethecorrelationwithhumanscoringisthehighest.In Trait1SVRachievesaQWKof 0.798 comparedto 0.785 for Huber, with corresponding R² values of 0.682 and 0.676. Traits2and3showmoderateperformance,whileTrait4is comparativelyhardertopredict.InSet8,theperformanceof both models drops across all traits, with QWK scores generallylower,indicatingthatthisdatasetismorecomplex and challenging. Overall, while both models provide good baseline results, they struggle with more complex essays, whichhighlightstheneedforadvancedmodelslikeBERT.

Table -2: PerformanceofHuberRegressorandSVRonSet 8usingR2 andQWK

Traits R² QWK

Trait1 0.4416 0.5701

Trait2

Theresultsoftheproposedmulti-traitBERTmodelonsets7 and8havebeenillustratedbyTableIIIand1Vrespectively. These show that there are evident differences in the predictive abilities of the model on different traits and datasets. In Set 7, the model performs well, and the QWK scoresarebetween0.6904and0.8467andthevaluesofR2 are between 0.5411 and 0.7333. Interestingly, the performance of Trait 1 is the best (R 2 = 0.7333, QWK = 0.8467),whichmeansthatthemodelisabletocapturethe entireunderlyingsemanticandstructuralcharacteristicsthat relatetothistrait.Theothertraitsalsoexhibitsignificantly highlevelsofagreementwithhumanscoringindicatingthat the model can learn meaningful representations of traits whenthereisenoughvariabilityinthedata.Conversely,Set8 scores are relatively poor with a QWK range of 0.51450.6121andR2of0.4283-0.5293.EventhoughTraits5and6 showaratherimprovedperformanceinthisset,theoverall outcomesreveallowerpredictivestrength.Thislossmaybe

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

explainedbythefactthatinSet8thedatasetsizeandrange of scores are smaller, making it difficult to distinguish the different degrees of writing quality that the model can. Nevertheless, the model continues to have moderate consensus with human raters, showing its strength under moredemandingconditions.

OVERALL COMPARISON

Table -5: Multi-TraitBERTvsTraditionalModelsofSet7

Model R²(Approx Range) QWK(Approx Range)

Huber 0.36-0.67 0.57-0.78

SVR 0.39-0.68 0.59-0.79

Bert 0.54-0.73 0.69-0.85

Table -6: Multi-TraitBERTvsTraditionalModelsofSet8

Model R²(Approx Range) QWK(Approx Range)

Huber 0.24-0.34 0.45-0.59

SVR 0.31-0.41 0.42-0.58

Bert 0.42-0.53 0.51-0.61

Table V and VI illustrates the comparative analysis of the experimental results across Set 7 and 8 respectively. It demonstrates the effectiveness of the proposed multi-trait BERTmodelovertraditionalstatisticalapproachessuchas HuberRegressionandSupportVectorRegression(SVR).For Set7,theBERT-basedmodelachievessuperiorperformance, with QWK values reaching up to 0.8467 and consistently outperformingbothHuber(maximumQWK=0.785)andSVR (maximumQWK=0.798)acrossalltraits.Asimilartrendis observedinR²scores,whereBERTattainshighervalues(up to 0.7333), indicating better predictive capability and strongeralignmentwithhumanscoring.Incontrast,forSet8, althoughtheoverallperformanceofallmodelsdecreasesdue to the dataset’s smaller size and compressed score distribution, the BERT model still maintains competitive performance, with QWK values ranging from 0.5145 to 0.6121, generally surpassing or matching the baseline

models. Notably, SVR performs slightly better than Huber Regression among the traditional methods, particularly in capturing non-linear patterns in the data; however, both remainsignificantlyinferiortothedeeplearningapproach. These resultshighlightthatthe proposedmulti-traitBERT model is more effective in capturing both semantic and structural aspects of essays, while also demonstrating robustnessacrossdatasetsofvaryingcomplexity.

7. EXPLAINABILITY AND INTERPRETABILITY

An explainability framework is added to enhance the interpretabilityandtransparencyoftheproposedmulti-trait Automated Essay Scoring (AES) system. As transformerbased architectures like BERT are black-box models, it is necessary to give clues regarding the process of how predictions are made. The explainability in this work is attainedusingafusionofattention-basedvisualizationand SHAP (SHapley Additive exPlanations)-based feature attribution.

7.1 Attention-Based Analysis

Themodelisusedtoobtainattentionweightstodetermineif thetokensaregivenincreasedimportanceduringprediction andthus,wherethemodelpaysattentioninassigningscores withintheessay.

Accordingtoexperimentalobservations,inthecaseofSet7, the model tends to focus on concrete and event-related words(Fig.3) suggestingthatitutilizesthecontext-based andnarrativeinformationtomakeanassessment.

Fig 3: Attentionweightsdistributionforset7essays

Contrastingly,inSet8,thefocusonattentionismorewidely distributed (Fig. 4) to prompt-relatable vocabulary indicatingthe moregeneral themeofpersonal-experience promptalignedwiththisdataset.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Itisalsofoundthatspecialtokenslike[CLS]and[SEP]are alwaysaccordedhighattentionweights,afamiliarproperty of transformer architectures and not directly reflecting a meaningful linguistic property. As a result, attention is a coarse-grainedinterpretationprocessandfailstoprovidea completeunderstandingofthecausalroleofinputfeatures.

7.2 SHAP Based Feature Attribution

SHAP (SHapley Additive Explanations) is an explainability methodusedtounderstandhowamachinelearningmodel makesitspredictions.Itassignsanimportancevaluetoeach inputfeature,showinghowmucheachwordcontributesto increasing or decreasing the predicted score.In this work, SHAPisusedtoprovideclearanddetailedexplanationsatthe wordlevelforeachpredictedtraitscore.

Theanalysisshowsthatthemodelgivesmoreimportanceto meaningful and context-related words, which help in improvingthequalityoftheessay.Wordsthatdescribeclear ideas,actions,ordetailsusuallyincreasethescorebecause theymakethecontentstrongerandmoreunderstandable.On the other hand, common or less informative words sometimesmayhaveanegativeimpactonthescore,asthey donotaddmuchvaluetothewriting.

A slight difference can be observed across the datasets. In somecases,thewordcontributionsaremoreclearlydefined and focused (Fig. 5), while in others they are more evenly spreadoutacrosstheessay(Fig.6).Thisvariationismainly due to differences in dataset size and score distribution, whichaffecthowstronglythemodelcanidentifyimportant features.

Itisalsoobservedthatforsometraits,themodeldoesnot depend only on individual words but also on the overall contextoftheessay.Thismeansthemodelunderstandsthe complete meaning rather than just focusing on specific words. Overall, SHAP helps us understand how the model evaluatesessaysbyshowingwhichwordsareimportantfor scoring.

The SHAP summary heatmaps clearly show different patternsacrossthetwocases.Inonecase,mostwordshave negativecontributions(Fig.5),whichmeanstheyreducethe scores across different traits. In the other case, the words show positive contributions (Fig 6.), meaning they help increasethescores,especiallyfortraitslikeorganizationand conventions. This difference shows that the model reacts differently depending on the input, and some words are morehelpfulthanothersinimprovingthepredictedscores.

7.3 Comparison between Attention and SHAP.

Thiscomparativestudyshowsacleardifferencebetweenthe twointerpretabilityapproaches(Fig.7andFig.8).Attention mechanisms indicate where the model focuses during prediction, while SHAP explains which words actually influencethepredictionoutcome.

AswecanseefromFig.7,attentiondistributesimportance acrossmanywords,includinglessmeaningfultokenssuchas special tokens and common words. Similarly in Fig. 8,

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

attentionshowsa broaderdistributionoffocusacrossthe essay.Incontrast,SHAPassignsimportancetoasmallerset ofkeywordsinboththecases,showingtheirdirectimpact on increasing or decreasing the score. This demonstrates that SHAP provides more precise and meaningful explanations, while attention mainly serves as a general visualizationofmodelfocus.

8. FUTURE WORK

Future work includes focusing on enhancing the representationlearningcapabilityoftheproposedsystemby incorporating advanced contrastive learning techniques, such as SimCSE [7], which has demonstrated a superior performance in generating semantically meaningful embeddings.WhileexistingapproachessuchasBERT-based fine-tuning[6],multi-scalerepresentationlearning[3],and self-supervised contrastive frameworks like D2SCLN [18] havesignificantlyimprovedautomatedessayscoring,there is still scope to further optimize embedding quality and robustness. Integrating contrastive learning methods can help in better capturing semantic similarity and reducing noise in representations. Additionally, combining deep learning features with simple linguistic features like grammar,wordcount,andsentencestructure[19]canhelp improve the quality of the model’s understanding. Also, usinglargelanguagemodelsforscoringandgivingfeedback [8] can make the system more accurate and easier to understand, which can improve overall performance and generalization.

9. CONCLUSION

In this paper, we proposed MultiTraitBERT, a multi-trait AutomatedEssayScoring(AES)systembuiltonBERT,which evaluatesmultipleaspectsofwritingqualitysimultaneously. Unliketraditionalmethodsthatpredictasingleoverallscore, the proposed model uses a shared encoder with multiple regressionheadstocapture bothsharedandtrait-specific features, enabling a more detailed and fine-grained evaluationofessays.

ExperimentalresultsontheHewlettFoundationAutomated EssayScoringdataset(Sets7and8)demonstratethatthe proposed model outperforms traditional statistical approachessuchasHuberRegressionandSupportVector Regression (SVR) in terms of both R² and Quadratic WeightedKappa(QWK).

To improve transparency, an explainability framework combining attention-based visualization and SHAP-based feature attribution is integrated. The analysis shows that attention mechanisms provide a general understanding of modelfocus,whileSHAPoffersmorepreciseandmeaningful insightsintofeaturecontributions.

Overall,theproposedMultiTraitBERTframeworkprovides an effective, interpretable, and scalable solution for automated essay scoring. By offering detailed trait-level feedbackalongwithimprovedpredictiveperformance,the systemshowsstrongpotentialforreal-worldapplicationsin educationalassessment.

REFERENCES

[1] K. Taghipour and H. T. Ng, “A neural approach to automatedessayscoring,”EMNLP,2016.

[2] F. Dong et al., “Attention-based neural network for automatedessayscoring,”CoNLL,2017.

[3] Y. He et al., “Automated essay scoring from multiple traits,”COLING,2022.

[4] M. Uto et al., “Neural automated essay scoring incorporatinghandcraftedfeatures,”COLING,2020.

[5] M. Cozma et al., “Automated essay scoring with string kernelsandwordembeddings,”ACL,2018.

[6]J.WangandL.Luo,“Anintelligentessayscoringsystem basedonaBERT-drivenDeepSelf-SupervisedContrastive LearningNetwork,”AlexandriaEngineeringJournal,2026.

[7]T.Gao,X.Yao,andD.Chen,“SimCSE:Simplecontrastive learningofsentenceembeddings,”arXiv,2022.

[8] Y. Song et al., “Automated essay scoring using large language models,” IEEE Transactions on Learning Technologies,2024.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

[9]Y.Caoetal.,“Domain-adaptiveneuralautomatedessay scoring,”SIGIR,2020.

[10] J. Wang et al., “Meta-learning for cross-prompt automatedessayscoring,”ExpertSystemswithApplications, 2025.

[11] S. M. Lundberg and S.-I. Lee, “A unified approach to interpretingmodelpredictions,”NeurIPS,2017.

[12] Kaggle, “The Hewlett Foundation: Automated Essay Scoring (ASAP-AES),” Kaggle Competition, 2012. [Online]. Available: https://www.kaggle.com/c/asap-aes. Accessed: Apr.17,2026.

[13]scikit-learnDevelopers,“HuberRegressor,”Scikit-learn documentation. [Online]. Available: https://scikitlearn.org/stable/modules/generated/sklearn.linear_model. HuberRegressor.html.Accessed:Apr.14,2026.

[14] A. J. Smola and B. Schölkopf, “A tutorial on support vectorregression,”StatisticsandComputing,vol.14,no.3, pp. 199–222, 2004, doi: 10.1023/B:STCO.0000035301.49549.88.

[15] J. Xue, X. Tang, and L. Zheng, "A Hierarchical BERTBased Transfer Learning Approach for Multi-Dimensional Essay Scoring," School of Foreign Studies, University of ScienceandTechnologyBeijing.

[16]Wikipediacontributors,“Coefficientofdetermination,” Wikipedia, The Free Encyclopedia. [Online]. Available: https://en.wikipedia.org/wiki/Coefficient_of_determination. Accessed:Apr.14,2026.

[17] M. Shermis and J. Burstein, eds., Handbook of AutomatedEssayEvaluation:CurrentApplicationsandNew Directions.NewYork,NY,USA:Routledge,2013.

[18]J.WangandL.Luo,"Anintelligentessayscoringsystem basedonaBERT-drivenDeepSelf-SupervisedContrastive LearningNetwork,"Knowledge-BasedSystems,2023

[19]M.Faseeh,A.Jaleel,N.Iqbal,A.Mehmood,andY.-I.Cho, "HybridApproachtoAutomatedEssayScoring:Integrating Deep Learning Embeddings with Handcrafted Linguistic FeaturesforImprovedAccuracy,"IEEEAccess,2023.