Real-Time Facial Emotion Recognition Using Fine-Tuned ResNet with Adaptive Weight Decay for Emotion-

International Research Journal of Engineering and Technology (IRJET)

Real-Time Facial Emotion Recognition Using Fine-Tuned ResNet with Adaptive Weight Decay for Emotion-Aware Music Recommendation

¹Ms.Noorbasha Zareena , ²Kotha Sree Jyothirmai, ³Mangamuri Vasantha, ⁴Munagala Leeladhar

¹Assistant Professor, Department of CSE, RVR & JC College of Engineering, Chowdavaram, Guntur, A.P, India. ²³´B. Tech Students, Department of CSE, RVR & JC College of Engineering, Chowdavaram, Guntur, A.P, India.

Abstract - Facial emotion recognition (FER) plays a vital rolein affective computing and human–computer interaction by enabling systems to interpret human emotional states from visual cues. Although deep learning models have achieved promising performance in FER tasks, maintaining strong generalization while supporting real-time inference remains a significant challenge. This paper presents a realtime facial emotion recognition framework based on a finetuned Residual Network (ResNet) architecture with adaptive optimization strategies designed to improve convergencestabilityandgeneralization.

The proposed system employs transfer learning using a pretrained ResNet backbone to extract discriminative facial features from grayscale facial images resized to 48×48 pixels. During the fine-tuning phase, selective unfreezing of the final layers enables task-specific feature adaptation while preserving pretrained representations. Batch normalization layers are kept frozen to maintain statistical consistency. Optimization is performed using the AdamW optimizer with decoupled weight decay regularization, providing improved control over overfitting compared to conventionalAdam-basedtraining.

A key contribution of this work is the introduction of an adaptive regularization mechanism that dynamically increases weight decay in response to learning rate reductions This strategy is implemented through a custom callback that monitors validation loss and couples learning rate decay with proportional weight decay adjustment. The approach strengthens regulariza-tion during later training stages, enhancing generalization with-out requiring architectural modifications or complex optimizer redesign. Performance evaluation is conducted using accuracy, macro precision, macro recall, and macro F1-score, ensuring balancedassessmentacrossemotionclasses.Classweighting is applied to address dataset imbalance and improve minorityclassrecognition.

The trained model is deployed in a real-time inference pipeline using a Streamlit based web application. Live video input is captured through a webcam, facial regions are detected using OpenCV-based face detection, and emotion classification is performed frame-by-frame with low latency. The system demonstrates near real-time performanceonstandard consumerhardware,validating its suitability for interactive applications.To illustrate practical applicability, the framework is extended into an emotionaware music recommendation system that maps detected

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

e-ISSN:2395-0056 p-ISSN:2395-0072 Impact Factor value: 8.315

emotionstocuratedSpotify playlists. This use case highlights the integration of affective intelligencewithadaptivemultimediasystems. Overall, the proposed approach bridges the gap between robust deep learning optimization and deployable real-time emotion recognition systems, offering a scalable and extensiblesolutionforhuman-centeredAI applicationssuchas personalized media delivery, mental health support tools, and intelligentuserinterfaces.

Key Words: Facial emotion recognition, transfer learning, ResNet, adaptive weight decay, AdamW optimizer, dynamic regularization, real-time inference, Streamlit deployment, affective computing, emotionawarerecommendationsystem.

I. INTRODUCTION

Humanemotionsplayafundamentalroleincommunication, decision-making, and social interaction. The ability of machines to automatically recognize and interpret emotional states has become a key research focus in affectivecomputingandhuman–computerinteraction(HCI). Facial Emotion Recognition (FER), in particular, aims to classify human emotions based on facial expressions extracted from images or video streams. With the rapid growthofartificialintelligencesystemsintegratedintodaily life ranging from virtual assistants and recommendation enginestomentalhealthmonitoring platformsaccurate and efficient emotion recognition has gained increasing importance

Traditional FER approaches relied heavily on handcrafted features such as Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG), or geometric landmark-based descriptors. While these methods demonstrated moderate success, they were highly sensitive to lighting conditions, pose variations, occlusion, and dataset bias. The advent of deep learning, especially Convolutional Neural Networks (CNNs), significantly improved FER performance by enabling automated feature extraction and hierarchical representationlearning.Deepmodelseliminatetheneedfor manualfeatureengineeringand capture complex spatial patterns within facial images. Among deep architectures, Residual Networks (ResNet) haveemergedasoneofthemosteffectivemodelsforvisual recognition tasks. ResNet introduces identity shortcut

International Research Journal of Engineering and Technology (IRJET)

e-ISSN:2395-0056

connectionsthatmitigatethevanishing gradient problem, enabling the training of significantly deeper networks. These residual connections facilitate stable gradient propagationandim-proveconvergencebehavior.Transfer learningusingpretrainedResNetbackboneshasbecomea common strategy in FER research, as it allows models trained on large-scale image datasets to generalize effectively to emotion classification tasks with limited labeleddata.

Despitethesuccessoftransferlearning,fine-tuningpretrained networks for FER presents several challenges. Emotiondatasetsareoftenrelativelysmallandimbalanced compared to general image datasets. This increases the risk of overfitting during fine-tuning, especially when deeper layers are unfrozen without adequate regularization. Additionally, improper optimization strategies can lead to unstable convergence or de-graded generalization performance. Therefore, beyond model architecture,optimizationtechniquesplayacriticalrolein achievingrobustFERperformance.

Recent studies emphasize the importance of decoupled weight decay regularization, particularly through the AdamW optimizer. Unlike traditional L2 regularization embedded within gradient updates, AdamW separates weight decay from the adaptive gradient step, leading to improved generalization and more consistent optimization dynamics. However, static regularization parameters may not always be optimal through-out the training process. As the learning rate decreases often triggered by validation loss plateau detection the model transitions into a fine-grained convergence phase where strongerregularizationmayhelppreventmemorizationof noiseandoverfitting.

In this work, we propose a real-time facial emotion recognition system based on a fine-tuned ResNet architecture combined with an adaptive optimization strategy. The core contribution lies in dynamically couplinglearningratereductionwithproportionalweight decay adjustment. Specifically, when a learning rate schedulerdetectsstagnationinvalidationlossandreduces the learning rate, a custom callback mechanism increases the optimizer’s weight decay parameter within predefinedbounds.

Thisstrategystrengthensregularizationduringlate-stage training, encouraging improved generalization without requiring architectural changes or complex optimizer redesign.

The training pipeline consists of two phases. In the initial phase, the pretrained backbone remains largely frozen, allowing the classifier head to adapt to the target emotion dataset. In the second phase, a subset of the deeper residual layers is selectively unfrozen to enable task-specific feature refinement. Batch normalization layers are kept frozen to preserve learned statistical

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

p-ISSN:2395-0072 Impact Factor value: 8.315

distributionsandpreventinstabilityduringfine-tuning. The modelistrainedusingSparseCategoricalCrossentropyloss with evaluation metrics including accuracy, macro precision, macro recall, and macro F1-score to en-sure balanced performance assessment across emotion classes. Classweightingisappliedtomitigatedatasetimbalance.

Beyond algorithmic contributions, this work emphasizes real-world deployment feasibility. Many FER studies focus solely on offline accuracy without addressing runtime performanceorpractical integration.Tobridgethisgap,the trainedmodelisdeployedwithinaStreamlitbasedreal-time webapplication.Livevideoinputiscapturedfromawebcam, facial regions are detected using classical computer vision techniques, and emotion classification is performed frameby-frame. The system achieves near real-time inference performance on standard consumer hardware, demonstratingitsapplicabilityforinteractiveenvironments.

This paper we also presents a real-time emotion-based music recommendation system that integrates computer vision and deep learning to enhance user experience throughadaptivecontentdelivery.Theproposedapplication utilizesa webcamtocapturefacialexpressionsandemploys a transformer- based image classification model to detect user emotions with high accuracy. Detected emotions are stabilized using temporal smoothing techniques and mapped to curated music playlists. A user-controlled interface enables seamless interaction, allowing individuals to play or preview recommended music based on their current emotional state. This system demonstrates the potential of affective computing in building intelligent, personalized,andresponsivemultimediaapplications.

To illustrate practical use cases, the framework is extended into an emotion-aware music system Based on detected emotional states such as happiness, sadness, anger, fear, surprise, disgust, or neutrality the application dynamically suggests curated music playlists. This integration highlights thepotential ofaffectiveintelligence systems topersonalize user experiences in multimedia platforms. Emotion-driven recommendation systems can enhance engagement by adaptingcontenttousers’psychologicalstatesinrealtime.

The proposed system contributes to the field in three primaryaspects:

Structured Fine-Tuning Strategy: A two-phase ResNet fine-tuning approach with selective layer unfreezing and frozen batch normalization for stable transfer learning in FER tasks. Adaptive Optimization Mechanism: A novel learning rate–aware weight decay scheduling strategy that dynamically strengthens regularization during convergence phases.

End-to-End Real-Time Deployment: A complete pipeline integrating training, evaluation, and real-time inferencewithinaninteractiveweb-basedsystem.

By combining architectural robustness, adaptive regularization, and deployment practicality, this work bridges the gap between theoretical deep learning

International Research Journal of Engineering and Technology (IRJET)

optimization techniques and real-world affective computing applications. The proposed framework demonstrates that carefully designed fine-tuning and optimization strategies can significantly enhance generalization while preserving real-time performance constraints.

The remainder of this paper is organized as follows. Section II reviews related work in facial emotion recognition and adaptive optimization techniques. Section III describes the proposed methodology, including the fine-tuning strategy and adaptive weight decay mechanism. Section IV presents experimental results and performance evaluation. Section V discusses real-time deploymentandapplicationintegration.Finally,SectionVI concludes the paper and outlines future research directions.

II.LITERATURESURVEY

[1] Facial Emotion Recognition (FER) has evolved significantlywiththeadvancementofcomputervisionand machinelearning techniques. Early FER systems relied on handcrafted feature extraction methods such as Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG), and geometric facial landmark descriptors. These traditional approaches, while computationally efficient, were highly sensitive to lighting conditions, pose variations, and occlusion. As discussed in [?], LBP-based methodsachievedmoderatesuccessbutlackedrobustness underreal-worldvariations.

[2] With the emergence of deep learning, Convolutional Neural Networks (CNNs) replaced handcrafted feature engineering by enabling automatic hierarchical feature extraction.Thestudyin[?]demonstratedtheeffectiveness of deep con-volutional architectures on the FER2013 dataset, significantly outperforming traditional classifiers. However, shallow CNN architectures often struggled with generalizationduetolimitedrepresentationaldepth.

[3]Residual Networks (ResNet) introduced by He et al. in [?] addressed the degradation problem in deep neural networks by incorporating identity shortcut connections. These residual connections allow gradients to flow more effectively through deep architectures, enabling the successful training of networks exceeding 50 layers. ResNet based transfer learning has since become a dominant approach inFERtasks dueto its strong feature extractioncapability.

[4] Transfer learning has been widely adopted in FER to mitigate the limitations of small emotion datasets. Pretrainedmodelsonlarge-scaledatasetssuchasImageNet providerobustlow-levelandmid-levelvisualfeaturesthat canbeadaptedtoemotionclassificationtasks.Asshownin [?], fine- tuning deeper layers of pretrained CNNs improves classification performance compared to training fromscratch.

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

e-ISSN:2395-0056 p-ISSN:2395-0072 Impact Factor value: 8.315

[5] Optimization strategies play a crucial role in fine-tuning pretrained networks. The Adam optimizer introduced in [?] became popular due to its adaptive learning rate mechanism. However, studies revealed that Adam may exhibit poorer generalization compared to stochastic gradient descent (SGD) in certain scenarios. This limitation motivated further research into improved adaptive optimizers.

[6]TheAdamWoptimizer,proposedin[?],decouplesweight decay from gradient-based parameter updates, leading to improved regularization and better generalization performance. Decoupled weight decay ensures consistent penalization of large weights regardless of the adaptive learning rate scaling, making it particularly effective in transferlearningsettings.

[7] Learning rate scheduling techniques such as ReduceLROnPlateau have been widely used to improve convergence stability. By decreasing the learning rate when validation loss stagnates, models can transition from rapid exploration tofine-grainedconvergence.Asdiscussedin[?], adaptivelearningratestrategiessignificantlyimpacttraining dynamicsandfinalperformance.

[8]Recent research emphasizes dynamic regularization strategies that adapt during training rather than remaining static. Dynamic weight decay scheduling has shown potential in improving late-stage generalization by increasingregularizationwhenlearningratesdecrease.Such approachesalignwithlarge-scaletrainingheuristicsusedin moderndeeplearn-ingsystems[?].

[9] Real-time FER systems introduce additional constraints beyond classification accuracy. Deployment-focused research highlights the importance of lightweight inference pipelines and efficient face detection mechanisms. Studies such as [?] demonstrated the feasibility of real-time FER using optimized CNN architectures combined with efficient preprocessingpipelines.

[10] Emotion aware recommendation systems represent a growing application domain of affective computing. Integrating emotion recognition with multimedia personalization enhances user engagement by adapting content to emotional states. As explored in [?], emotiondriven systems can signif-icantly improve interactive user experiences in entertainment and human-centered AI applications.

III. EXISTINGSYSTEM

Facial Emotion Recognition (FER) systems have evolved through multiple technological phases, ranging from traditional handcrafted feature-based machine learning approaches to deep learning architectures and modern transfer learning frameworks. Existing systems primarily focus on improving classification accuracy using static trainingstrategiesandpredefined optimization parameters. While these approaches have achieved considerable

International Research Journal of Engineering and Technology (IRJET)

e-ISSN:2395-0056

progress in controlled benchmark datasets, several limitations persist, particularly in real-world generalizationandreal-timedeployment. Many traditional systems rely on handcrafted descriptors that are sensitive to lighting, pose, and occlusion. Even deep learning-based systems often employ fixed regularization strategies, which may not optimally adapt during training. Furthermore, most FER research emphasizes offline performance evaluation rather than practical integration into interactive systems. This section reviews the dominant existing approaches in FER, highlighting their methodologies, strengths, and inherent limitations.Understandingtheseexistingsystemsprovides the foundation for identifying research gaps and motivating the need for adaptive optimization strategies andreal-timedeploymentframeworks.

A. Traditional Machine Learning Approaches

Traditional FER systems primarily relied on handcrafted feature extraction followed by classical classifiers.Featuressuch as Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG), and ScaleInvariant Feature Trans-form (SIFT) were widely used to capturefacialtextureandedgeinformation.Thesefeatures were then fed into machine learning classifiers including Support Vector Machines (SVM), k-Nearest Neighbors (kNN),DecisionTrees,orNaiveBayesmodels.

The strength of these approaches lies in their computational efficiency and lower hardware requirements. Since handcrafted features reduce dimensionalitybeforeclassification,trainingandinference are relatively fast. Such systems were suitable for early embedded applications and constrained environments. However, traditional approaches suffer from significant limitations. Handcrafted descriptors fail to capture highlevel semantic information and are highly sensitive to variations in lighting, head pose, occlusion, and facial alignment. Moreover, these features require domain expertise for proper design and tuning. As emotion recognition involves subtle facial muscle movements, handcrafted features often struggle to generalize across diverse datasets. Consequently, while traditional machine learning approaches laid the groundwork for FER research, their limited representational power restricts performanceinreal-worldscenarios.

B. Deep Convolutional Neural Network-Based Systems

TheintroductionofConvolutionalNeuralNetworks(CNNs) revolutionizedFERbyenablingautomatic hierarchical feature learning. Unlike handcrafted approaches, CNNs learn spatial filters directly from raw pixel data, capturing both low-leveltexturesandhigh-levelsemantic representations. Early CNN-based FER systems utilized shallow architectures with a limited number of convolutional layers, followed by fully connected classifiers.

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

p-ISSN:2395-0072 Impact Factor value: 8.315

CNN-basedsystemssignificantlyimprovedclassification accuracy compared to traditional methods. Their ability to learn discriminative features directly from facial images eliminated the need for manual feature engineering. Moreover, techniques such as data augmentation and dropout regularization improved robustness against overfitting.

Despite these advantages, early CNN models faced challengesrelatedtovanishinggradientsandlimiteddepth. Shal-low architectures often lacked sufficient representational capacity to capture complex emotional cues. Additionally, training deep CNNs from scratch requiredlargelabeleddatasets,whichareoftenunavailable in FER tasks. These limitations motivated the adoption of deeperarchitecturesandtransferlearningtechniques.

C. Transfer Learning Using Pretrained Deep Networks

Transfer learning has become a dominant strategy in modern FER systems. Pretrained models such as VGG, ResNet, and Inception, trained on large-scale datasets like ImageNet, are fine-tuned for emotion classification. By leveraging pretrained weights, these systems benefit from generalized feature ex-traction while adapting higher-level layerstoemotion-specifictasks.

The primary advantage of transfer learning lies in reducedtrainingtimeandimprovedperformanceonlimited datasets.Selectivelayerfreezingallowspreservationoflowlevel visual features while adapting deeper layers to domain-specific patterns. This strategy significantly enhancesconvergencesta-bilityandoverallaccuracy.

However, many existing transfer learning approaches rely on static fine-tuning strategies. Layers are unfrozen using fixed heuristics without considering adaptive optimization dynamics. Moreover, improper regularization during fine-tuning can lead to overfitting, especially when dataset size is limited. These limitations highlight the need formoreadaptivetrainingstrategies

D. Static optimization and regularization techniques

Optimization strategies play a crucial role in model performance. Most existing FER systems utilize optimizers such as Stochastic Gradient Descent (SGD) or Adam with fixed weight decay or L2 regularization. Learning rate schedulers,includingstepdecayorReduceLROnPlateau,are commonlyusedtoimproveconvergence.

While these techniques enhance training stability, regularization parameters typically remain constant throughout training. Fixed weight decay may not provide optimal regularization across different learning phases. During later stages of convergence, insufficient regularization can lead to memorization of noise and degradedgeneralization.

The lack of dynamic adaptation between learning rate adjustments and weight decay represents a limitation in many existing systems. Modern large-scale training heuristics suggest that coupling optimization parameters mayyieldbettergeneralizationperformance.

E. Real-Time Deployment-Oriented FER Systems

Most FER research emphasizes offline accuracy evaluation rather than real-time deployment feasibility. Real-time systems must balance accuracy with computational efficiency and latency constraints. Some approaches utilize lightweight CNN architectures or mobile-optimizednetworkstoachievefasterinference.

Face detection is typically performed using classical computer vision methods such as Haar Cascades or more advanced detectors like MTCNN. While these pipelines enable real- time inference, many systems compromise model complexity to maintain speed, potentially reducing classificationperformance.

Additionally, integration with real-world applications re-mains limited in existing studies. Few works extend FER into interactive systems such as recommendation enginesoradaptive multimediaplatforms. The absence of optimization-aware deployment frameworks further restrictspracticalscalability.

Overall, existing systems demonstrate substantial progress in FER accuracy; however, they often lack adaptive optimization strategies and comprehensive endto-end deployment pipelines. These gaps motivate the developmentofenhancedfine-tuning methodologies and dynamically regularized real- time FERsystems.

IV.METHODOLOGY

The proposed methodology integrates deep transfer learning,adaptiveoptimization,andreal-timedeployment into a unified facial emotion recognition framework. The systemisdesignedtoaddresstwomajorchallengesinFER research:improvinggeneralizationduringfine-tuningand enabling efficient real-time inference. The methodology follows a structured pipeline consisting of data preprocessing, model architecture selection, two-phase fine-tuning, adaptive optimization with dynamic weight decay coupling, and deployment within a real-time webbased application. A pretrained Residual Net-work (ResNet) backbone is employed for feature extraction, leveraging transfer learning to compensate for limited datasetsize.Toenhanceconvergencestabilityandprevent overfitting, an adaptive regularization mechanism dynamically adjusts weight decay in response to learning rate reductions. The complete framework is modular, allowing reproducibility and scalability. This section describes each component of the proposed methodology indetail.

A. Data Preprocessing and Dataset Preparation

The dataset consists of labeled facial images categorized into seven emotion classes: angry, disgust, fear, happy, sad, surprise, andneutral. Images areresized to a fixed resolution of 48×48 pixels to maintain consistency with FER benchmark standards while ensuring computational efficiency. All images are

e-ISSN:2395-0056

converted to grayscale to reduce dimensionality and eliminate redundant color information, as facial expression cues are primarily texture-based rather than colordependent.

To improve generalization, data augmentation techniques are applied during training. These include random horizontal flipping, small rotations, zoom variations, and brightness adjustments. Augmentation increases data diversity and helps the model become invariant to minor pose and illumination changes. Dataset imbalanceisaddressed usingclassweighting, ensuring that minority emotion classes contribute proportion-ally to the lossfunction.

Fig1 SamplesfromFER2013Dataset

The dataset is split into training and validation sets. During evaluation, augmentation is disabled to ensure unbiased performance measurement. Preprocessing is implementedusingTensorFlowdatapipelineswithbatching and prefetching to optimize training throughput and minimizeI/Obottlenecks.

IRJET

C. Adaptive Optimization Strategy

B. ResNet-Based Transfer Learning Architecture

The backbone architecture is based on a pretrained Residual Network (ResNet), selected for its ability to mitigate vanishing gradient issues through identity shortcut connections. These residual connections allow deepfeatureextractionwhilemain-trainingstablegradient propagation.

Initially, the pretrained backbone is loaded with ImageNetweights.Thefinalclassificationlayerisreplaced with a task-specific dense layer corresponding to the number of emotion classes. Global Average Pooling is applied before the classi-fication head to reduce parametercountandimprovespatialgeneralization.

The training process follows a two-phase fine-tuning strategy. In Phase 1, most of the backbone layers remain frozenwhileonlytheclassifierheadistrained.

Thisallowsthenewlyaddedlayerstoadapttotheemotion dataset without disrupting pretrained representations. In Phase 2, the final N layers of the backbone are selectively unfrozen to enable domain- specific feature refinement. Batch normalization layers remain frozen throughout to preserve learned statistical properties and prevent instabilityduringfine-tuning.

e-ISSN:2395-0056 p-ISSN:2395-0072

Optimization is performed using the AdamW optimizer, which decouples weight decay from gradient-based parameter updates. Unlike traditional L2 regularization, decoupled weight decay ensures consistent penalization of large weights independent of adaptive learning rate scaling.The loss function used is Sparse Categorical Cross entropy,definedas:

where yi represents the true class label and yˆi denotes the predicted probability. Performance evaluation includes accuracy,macroprecision,macrorecall,andmacroF1-score to ensure balanced assessment across classes. A ReduceLROnPlateau scheduler monitors validation loss and reduces the learning rate when improvement stagnates. Earlystoppingisalsoemployedtopreventoverfitting.

D. Dynamic Weight Decay Adjustment Mechanism

A key innovation in the proposed methodology is the dynamic coupling of learning rate reduction with weight decay adjustment. When the learning rate decreases due to plateau detection, the regularization strength is proportionallyincreasedwithinpredefinedbounds.

Letηtrepresentthelearningrateatepochtandλtdenote weightdecay.Whenηt<ηt−1,weightdecayisupdatedas:

whereα>1isthescalingfactorandλmaxistheupperbound. Thismechanismstrengthensregularizationduringlaterconvergence stages, reducing overfitting and improving generalization. The implementation is realized through a custom callback integrated into the TensorFlow training pipeline.

E. Real-Time Inference and Deployment Framework

Aftertraining,themodelisdeployedwithinaStreamlitbased web application for real-time inference. Webcam video is captured using a streaming interface, and face detection is performed using OpenCV-based Haar Cascade classifiers.

Detected facial regions are preprocessed and passed to the trainedResNetmodelforemotionprediction.

Frame-by-frame inference is optimized to maintain low latency on consumer hardware. The system outputs predicted emotion labels in real time and integrates an emotion-aware music recommendation module that maps predictedemotionstocurated playlists.

The modular deployment architecture ensures separation between model inference and user interface components, enabling scalability to cloud environments suchasAWS.

This real-time integration demonstrates the practical

International Research Journal of Engineering and Technology (IRJET)

e-ISSN:2395-0056

applicability of the proposed adaptive FER system in interactivehuman-centeredAIapplications.

V.IMPLEMENTATION

The implementation of the proposed real-time Facial Emotion Recognition (FER) system integrates deep learning model training, adaptive optimization strategies, evaluation protocols, and deployment within an interactivewebapplication.Thesystemisdevelopedusing TensorFlowandKerasformodeltraining,withsupporting librariessuchasNumPy,OpenCV,Matplotlib,andSeaborn for preprocessing and evaluation. Deployment is achieved using Streamlit for web-based inter- action and real-time webcamstreaming.

The implementation is divided into five major stages: dataset pipeline construction, model architecture setup, two-phase fine-tuning with adaptive optimization, evaluation and performance analysis, and real-time inference deployment. Special attention is given to modularity and reproducibility, ensuring that each component of the pipeline can function independently while remaining seamlessly integrated into the complete system. The following subsections describe each implementationstageindetail.

A.

Dataset PipelineandPreprocessing Implementation

The dataset pipeline is constructed using TensorFlow’s image_dataset_from_directory API, which allows structured loading of labeled facial emotion images. Imagesareresizedtoafixedresolutionof48×48pixelsto maintain consistency with FER benchmark standards whileensuringefficientcomputation.Thelabelmodeisset to integer encoding to support Sparse Categorical Cross entropyloss.

Data augmentation is applied during training using Tensor- Flow preprocessing layers. Augmentation operations include random horizontal flipping, small rotations, and brightness adjustments to improve invariance to pose and illumination variations. Augmentationisappliedonlytothetrainingdataset,while validation datasets remain unaltered to ensure unbiased performanceevaluation.

Class imbalance is addressed using class weights computed from label distributions. These weights are passedintothemodeltrainingfunction,ensuringminority emotion classes receive proportional importance during gradient up- dates. Prefetching and parallel mapping are enabled using the AUTOTUNE parameter to optimize data throughputandreducetraininglatency.

Duringevaluation,acleandatasetwithoutaugmentation is generated to compute unbiased performance metrics and con-fusion matrices. This separation between augmented training data and clean evaluation data ensuresreliablegeneralizationmeasurement.

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

p-ISSN:2395-0072 Impact Factor value: 8.315

B.

Model Architecture and Fine-Tuning Strategy

The backbone architecture is based on a pretrained Residual Network (ResNet). The model is initialized with ImageNet weights, leveraging transfer learning to extract high-levelspatialfeatures.

The final fully connected classification layer is replaced with a task-specific dense layer corresponding to the numberofemotionclasses.

Global Average Pooling is applied before the classification layer to reduce the number of trainable parameters and enhance spatial generalization. Dropout regularization may be included in the classifier head to reduceoverfittingrisk.

The fine-tuning process is conducted in two phases. In Phase1,thebackboneremainslargelyfrozenwhileonlythe classification head is trained. This allows the newly added layers to adapt to emotion-specific patterns without disturbingpretrainedfeaturerepresentations.

In Phase 2, the final N layers of the backbone are selectively unfrozen. Batch normalization layers remain frozen to preserve learned mean and variance statistics, preventing instability during gradient updates. This selective unfreezing strategy balances stability and adaptability, enabling domain-specific refinement while maintainingrobustpretrainedfeatures.

C. Adaptive Optimization and Custom Callback Integration

Optimization is implemented using the AdamW optimizer with decoupled weight decay. If AdamW is unavailable in certain TensorFlow versions, the implementation falls back to the standard Adam optimizer to maintain compatibility. The initial learning rate and weight decay parameters are carefully selected to ensure stableconvergence.

A ReduceLROnPlateau scheduler monitors validation loss andreducesthelearningratewhenimprovementstagnates. Earlystoppingisconfiguredwithrestorationofbestweights

topreventoverfitting.

emotionstate.

Acustomcallback,IncreaseWeightDecayOnLRDrop is implemented to dynamically adjust weight decay when learning rate reduction is triggered. The callback monitors changes in optimizer learning rate and proportionally increases weight decay within predefined limits. This dynamic coupling strengthens regularization duringlaterconvergencestages,improvinggeneralization.

The callback carefully handles different optimizer attribute structures to ensure compatibility across TensorFlowversions.Thisrobustimplementationensures adaptive regularization without modifying the core optimizerclass.

D. Model Evaluation and Performance Analysis

Modelevaluationisperformedusingmultiplemetrics to ensure comprehensive performance analysis. Sparse Categorical Accuracy measures overall correctness, while Macro Precision, Macro Recall, Macro F1-score evaluate class-balancedperformance.

After training, the model is evaluated on clean training and validation datasets without augmentation. Confusion matrices are generated using Scikit-learn’s confusion matrix function and visualized using Seaborn heatmaps. This visualization helps identify class-wise misclassificationpatterns.Performancemetricsareprinted instructuredformatfordocumentation.Lossandaccuracy curves are analyzed to ensure stable convergence behavior. Early stopping ensures that the model parameterscorrespondtothebestvalidationperformance ratherthanthefinalepoch.

These evaluation strategies provide detailed insight into classification behavior and validate the effectiveness oftheadaptiveoptimizationstrategy.

E. Real-Time Streamlit Deployment and Integration

The proposed system is implemented as a real-time web application using Streamlit, integrating computer vision,deep learning, and user interface components. The application captures live video input through a webcam using the WebRTC framework, enabling continuous frame acquisition.EachframeisprocessedusingOpenCV,where a Haar Cascade classifier is employed for efficient face detection. Detected facial regions are extracted and preprocessed before being passed to a pre-trained transformer-based image classification model for emotion recognition.

The model, sourced from the Hugging Face library, utilizes an AutoImageProcessor for input transformation and AutoModelForImageClassification for inference. The model predicts emotion probabilities, and the final emotion label is selected based on the highest confidence score. To improve stability and reduce prediction noise, a temporal smoothing mechanism is implemented using a fixed-length deque buffer. Majority voting is applied over recent predictions to determine the most consistent

e-ISSN:2395-0056

p-ISSN:2395-0072

The system maintains state using Streamlit’s session state functionality, ensuring synchronization between the real-timedetectionpipelineandtheuserinterface. Anautorefreshmechanismisincorporatedtoupdatetheinterfaceat regular intervals, creating a near real-time interaction experience

For music recommendation, a predefined mapping betweenemotionsandSpotifyplaylistsisused.Basedonthe detected emotion, a corresponding playlist link is dynamically selected. Instead of automatic playback, the system provides user- controlled interaction through buttons that allow either opening the playlist in Spotify or previewing it within the application using an embedded iframeplayer.

Thismodulardesignensuresscalability,responsiveness, and ease of integration with external APIs or advanced recommendation engines, making the system suitable for real-worlddeploymentinaffectivecomputingapplications

VI.RESULTSANDDISCUSSION

The performance of the proposed Adaptive ResNet-based Facial Emotion Recognition (FER) system is evaluated usingquantitativemetrics,visualanalysis tools,andreal-timedeploymenttesting.Theexperimental results assess classification accuracy, class-wise performance balance, convergence behavior, regularization impact, and practical inference capability. The evaluation framework emphasizes not only overall accuracy but also macro-averaged precision, recall, and F1-score to ensure balanced performance across all emotionclasses.Confusionmatrixanalysisisconductedto identify misclassification patterns and inter-class similarities.Furthermore,theeffectivenessoftheproposed dynamicweightdecayadjustmentmechanismisexamined by comparing training stability and generalization improvements. Finally, the system’s real- time performance is evaluated in a deployment setting using webcam-based inference. The following subsections provide a comprehensive discussion of experimental findingsandpracticalobservations.

A. Overall Classification Performance

The proposed model achieved strong classification accuracy on both training and validation datasets. Sparse categoricalaccuracyindicatesthatthesystemsuccessfully learns discriminative facial representations across seven emotioncategories.

Thevalidationaccuracycloselyfollowstrainingaccuracy, demonstrating effective generalization and controlled overfitting

Macro-averaged precision and recall values confirm balanced performance across classes. Unlike traditional accuracy metrics, macro averaging ensures that minority classes such as “disgust” and “fear” contribute equally to performance evaluation. The macro F1-score further validatestheconsistencybetweenprecisionandrecall.

Loss curves demonstrate stable convergence behavior. During early epochs, rapid loss reduction is observed due tofeatureadaptationintheclassifierhead.Inlaterepochs, gradual improvements occur as fine-tuned backbone layers refine emotion-specific features. The ReduceLROnPlateau scheduler successfully reduces learning rate when validation loss stag- nates, enabling smootherconvergence.

Compared to baseline CNN models trained from scratch, the transfer learning approach significantly improvesaccuracyandconvergencespeed.Thepretrained residual architecture extracts robust low-level and midlevelfeatures,reducingtheneedforextensivedatasetsize.

Overall, the classification results confirm that the adaptive transfer learning strategy effectively enhances recognition performance while maintaining stability and generalization.

e-ISSN:2395-0056

p-ISSN:2395-0072

B. Class-Wise Analysis and Confusion Matrix Interpretation

To further analyze performance, confusion matrices were generated for the validation dataset. The diagonal dominance of the matrix indicates strong correct classification rates across most emotion categories. The “happy” and “surprise” classes show particularly high recognition accuracy due to distinct facial muscle movements and clear visual cues. These expressions are characterizedbystrongmouthcurvatureand eye widening, whichareeasilycapturedbyconvolutionalfilters. Conversely, confusion is observed between “fear” and “surprise,” as well as between “sad” and “neutral.” These misclassificationsariseduetosubtledifferencesineyebrow tension and mouth curvature. The similarity in facial geometry makes these classes inherently challenging, even

International Research Journal of Engineering and Technology (IRJET)

forhumanobservers.

The confusion matrix also reveals improved recognition for minorityclassescomparedtobaselinemodels.Theuse of class weighting during training reduces bias toward dominantcategoriessuchas“happy”and“neutral.”

Fig.9 ClassificationReport

Precisionandrecalltrendsconfirmthatnosingleclass disproportionately influences performance. Balanced macro metrics indicate that the model does not overfit to majorityexpressions.

This analysis highlights both strengths and limitations of the proposed approach, providing insight into future improvements such as attention mechanisms or temporal modeling.

Fig.10.ConfusionMatrix

C. Impact of Adaptive Weight Decay Mechanism

Oneofthekeycontributionsofthisworkisthedynamic coupling of learning rate reduction with weight decay adjustment. Experimental observations indicate that this mechanism improves generalization during later training stages.

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

e-ISSN:2395-0056 p-ISSN:2395-0072 Impact Factor value: 8.315

Whenvalidationlossplateaus,thelearningratescheduler reduces the step size. Simultaneously, the weight decay parameter increases proportionally within predefined limits.Thisadaptiveregularizationstrengthenspenalization of large weights, preventing memorization of training samples

Comparative analysis with static weight decay settings shows that the dynamic strategy reduces the gap between training and validation accuracy. Overfitting is minimized, andvalidationperformanceremainsstableacrossepochs.

Additionally, smoother loss curves are observed during fine-tuning, indicating improved optimization stability. The model avoids abrupt oscillations and converges toward a flatter mini-mum, which is typically associated with better generalization. The results confirm that dynamic regularization acts as an effective complement to adaptive learningratescheduling,providinga systematic mechanism for balancing convergence speed and generalization strength.

D.Training Stability and Convergence Behavior

Training stability is evaluated by analyzing loss and accuracy trends across both fine-tuning phases. During Phase 1, when only the classifier head is trained, rapid convergence occurs due to the limited number of trainable parameters. This phase establishes an initial alignment betweenpretrainedfeaturesandemotionclasses.

During Phase 2, selective unfreezing introduces additional trainable parameters. Despite increased model flexibility, con-vergence remains stable due to frozen batch normalization layers and controlled learning rate adjustments.

No gradient explosion or instability is observed during unfreezing. The two-phase strategy prevents catastrophic for-getting of pretrained representations. Instead, the networkgraduallyadaptstodomain-specificcharacteristics.

Earlystoppingsuccessfullyhaltstrainingwhenvalidation performanceceasestoimprove,ensuringoptimal model selection. The final model corresponds to the epoch with best validation accuracyratherthanthefinalepoch.

Overall, the structured fine-tuning process contributes significantly to training robustness and consistent performanceimprovements.

E. Real-Time Deployment Performance and Practical Observations

The proposed emotion-based music recommendation system was evaluated in a real-time environment to assess its responsiveness, accuracy, and user experience. The system success- fully detected facial emotions from live webcam input and provided corresponding music recommendationswithminimallatency.Theintegrationofa transformer-based model enabled accurate classification across multiple emotional states, including happiness, sadness, anger, and neutrality. The use of a confidence threshold ensured that only reliable predictions were

e-ISSN:2395-0056

considered,therebyreducingincorrectclassifications.

A temporal smoothing mechanism using a sliding window buffer significantly improved the stability of emotion detection. Without smoothing, rapid fluctuations in predictions were observed due to minor facial variations and environmental noise. However, with majority voting applied over recent frames, the system produced consistent and stable emotion outputs, enhancingthereliabilityofrecommendations.

The real-time performance of the system was satisfactory, with near-instantaneous updates facilitated by efficient frame processing and periodic interface refresh. The decision to incorporate user-controlled interaction for music playback im- proved usability by preventing frequent and unwanted changes in the recommended content. Users could choose when to engage with the suggested playlists, resulting in a more intuitiveandlessintrusiveexperience.

The system demonstrated effective mapping between detectedemotionsandcuratedSpotifyplaylists,providing a personalized and context-aware music recommendation experience. However, certain limitations were identified, includingsensitivitytolightingconditionsandtheuseofa basic face detection algorithm, which may affect performanceincomplexenvironments.

Overall,theresultsindicatethattheproposedsystemis capable of delivering a responsive, stable, and userfriendly emotion-aware music recommendation experience, highlighting the potential of combining affective computing with real-time multimedia applications.

Volume:13Issue:04|Apr2026 | www.irjet.net

p-ISSN:2395-0072

VII.CONCLUSION

This research presented a comprehensive and adaptive framework for real-time Facial Emotion Recognition (FER) using a transfer learning-based Residual Network architecture combined with dynamic optimization strategies. The primary objective of this work was to improve emotion classification accuracy while ensuring stable convergence, balanced class performance, and practical deployment feasibility. By integrating deep residual learning, adaptive weight decay mechanisms, and real-time inference capabilities, the proposed system successfully addresses several limitations commonly observed in traditional FER approaches. The results demonstrate that combining structured fine-tuning with adaptive regularization significantly enhances generalization performance without introducing training instability.

One of the major contributions of this work lies in the effective use of transfer learning. Instead of training a deep convolutional neural network from scratch, a pretrained ResNet backbone was leveraged to extract robust low-level and mid-level visual features. This approach reduces computational cost and accelerates convergence while maintaining high representational power. The two-phase fine-tuningstrategyplayed a crucial roleinachievingstable adaptation.Duringtheinitialphase,freezingthemajorityof backbone layers allowed the classifier head to learn emotion-specific map- pings without disturbing pretrained weights. In the second phase, selective unfreezing enabled gradual domain-specific refinement while preserving essential learned representations. This structured approach prevented catastrophic forgetting and ensured smooth convergencebehaviorthroughouttraining.

Anothersignificantinnovationintroducedinthisresearchis the dynamic weight decay adjustment mechanism. Traditional optimization strategies often apply static regularization parameters, which may not remain optimal throughout the training process. In contrast, the proposed method dynamically increases weight decay whenever the learning rateis reduceddue to validation loss plateau. This

International Research Journal of Engineering and Technology (IRJET)

e-ISSN:2395-0056

coupling strengthens regularization during later training stages, when the model is more prone to overfitting. Experimental observations confirm that this adaptive regularization strategy reduces the gap between training and validation accuracy and produces smoother convergencecurves.

The model convergestowardflatterminima, whichare widely associated with improved generalization performance. The evaluation framework used in this study emphasizes balanced performance across emotion classes.Insteadofrelyingsolelyonoverallaccuracy,macro precision, macro recall, and macro F1-score were employed to ensure that minority classes contribute equally to performance assessment. Confusion matrix analysis revealed strong classification performance for highly expressive emotions such as happiness and surprise, while more subtle expressions such as fear and sadness presented moderate confusion. Importantly, class weighting during training reduced bias toward dominant classesandimprovedminorityclassrecognitioncompared to conventional baseline models. This balanced evaluation approach ensures that the system performs reliably acrossdiverseemotionalcategories.

Training stability was another core focus of this research. Deep neural networks, particularly when finetuned, are susceptible to instability caused by abrupt learningratechangesorexcessiveparameterupdates.The use of a ReduceLROn- Plateau scheduler, combined with frozenbatch normalization layers, ensured consistent and controlled optimization. Early stopping further prevented unnecessary training beyond optimal convergence points. The results show that the model maintained stable loss reduction across epochs without experiencing gradient explosionoroscillatorybehavior.Thisstabilityisessential for reproducibility and deployment in real-world applications.

Beyond quantitative evaluation, the system was success-fully deployed in a real-time environment using a Streamlit-based web application. The trained model was integrated with OpenCV-based face detection and optimized preprocessing pipelines to enable frame-byframe emotion prediction through webcam input. Realtimetestingconfirmedthatthesystemoperatesefficiently on standard consumer hardware, maintaining acceptable latency while providing accurate predictions. This practical validation demonstrates that the proposed architecture is not limited to controlled experimental conditions but is capable of functioning in interactive human-centeredAIapplications.

The integration of an emotion-based music recommendation module enhances the practical usability of the system by linking detected facial emotions to curated Spotify playlists. By incorporating real-time emotion detection with user-controlled playback, the application demonstrates how affective computing can

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

p-ISSN:2395-0072 Impact Factor value: 8.315

deliver personalized and interactive user experiences. This approach highlights the potential of facial emotion recognition systems in applications such as adaptive entertainment, mental wellness support, and humancomputer interaction. Furthermore, the modular design of thesystem,combiningStreamlit,computervision,anddeep learning components, ensures flexibility and scalability, allowing future extensions such as cloud deployment, API integration,andmoreadvancedrecommendationstrategies.

Despite the strong performance achieved, certain limitations remain. The system relies on static image-based emotionclassification,whichmaynotfullycapturetemporal dynamics present in real-world emotional expressions. Subtle transitions between emotions or micro-expressions may require temporal modeling approaches such as recurrent neural networks or 3D convolutional architectures. Additionally, extreme lighting conditions, occlusions, or large pose variations can reduce face detection accuracy, indirectly affecting emotion classification performance. Future enhancements may incorporate advanced face detection techniques or attention-based mechanisms to improve robustness under challengingconditions.

Anotherareaforfutureimprovementinvolvesexpanding dataset diversity. Emotion recognition models trained on limited or imbalanced datasets may exhibit reduced generalization when exposed to culturally diverse facial expressions. Incorporating larger, multi-ethnic, and multienvironment datasets could further improve model robustness and fairness. More-over, bias mitigation strategies should be considered to ensure equitable performanceacrossdemographicgroups.

Future research may also explore hybrid architectures that combine convolutional networks with transformerbased attention mechanisms. Vision Transformers and attention modules have demonstrated strong performance in visual recognition tasks and may enhance the model’s ability to focus on salient facial regions. Additionally, lightweight model compression techniques such as pruning orquantizationcouldimproveinferenceefficiencyforedgedevicedeployment.

In conclusion, this research successfully demonstrates thatf adaptive transfer learning combined with dynamic regularization provides an effective and practical solution for real- time Facial Emotion Recognition. The proposed framework achieves strong classification accuracy, balanced class performance, stable convergence, and efficient real-time deployment. The dynamic weight decay mechanism contributes meaning- fully to improved generalization, while the two-phase fine tuning strategy ensures controlled adaptation of pretrained features. The system’s real-time integration and interactive application further validate its practical relevance. Overall, this work contributes a robust, scalable, and adaptable FER frameworkthatbridgesthegap between academic research and

International Research Journal of Engineering and Technology (IRJET)

real- world intelligent applications, laying a strong foundation for future advancements in emotion-aware artificialintelligencesystems.

REFERENCES

[1] I. Goodfellow, J. Pouget-Abadie, M. Mirza, et al., “Generative Adversarial Nets,” in Proc. NIPS, 2014. https://doi.org/10.1145/3422622

[2] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proc. CVPR, 2016. https://doi.org/10.1109/CVPR.2016.90

[3]A.Krizhevsky,I.Sutskever,andG.E.Hinton,“ImageNet ClassificationwithDeepConvolutionalNeuralNetworks,”in https://doi.org/10.1145/3065386Proc.NIPS,2012

[4]M. Abadi, P. Barham, J. Chen, et al., “TensorFlow: MachineASystemforLarge-ScaleLearning,”inProc.OSDI ,https://www.usenix.org/conference/osdi16/technicalsessions/presentation/abadi2016.

[5]D.P.KingmaandJ.Ba,“Adam:AMethodforStochastic Optimization,”inProc.ICLR,2015.https://arxiv.org/abs/141 2.6980

[6] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,”inProc.ICLR,2019. https://arxiv.org/abs/1711.05101

[7]G.B.Huang,M.Ramesh,T.Berg,andE.Learned-Miller, “Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments,” Univ. Massachusetts,Tech.Rep.,2007.www.cs.umass.edu/lfw/

[8] I. J. Goodfellow, D. Erhan, P. Luc Carrier, et al., “Challenges in Representation Learning: A Report on Three MachineLearningContests,”inProc.NeurIPSWorkshop,201 3.https://arxiv.org/abs/1307.0414

[9] P. Viola and M. Jones, “Rapid Object Detection Using a Boosted Cascade of Simple Features,” in Proc.CVPR,2001. https://doi.org/10.1109/CVPR.2001.990517

[10]F.Chollet,“Xception:DeepLearningwithDepth-wise SeparableConvolutions,” inProc.CVPR, 2017. https://doi.org/10.1109/CVPR.2017.195

© 2026, IRJET

Volume:13Issue:04|Apr2026 | www.irjet.net

e-ISSN:2395-0056 p-ISSN:2395-0072 Impact Factor value: 8.315