De-identification Meets Emotion: A Review of Privacy-Enhanced Facial Expression Recognition Framewor by IRJET Journal

De-identification Meets Emotion: A Review of Privacy-Enhanced Facial Expression Recognition Frameworks

FATHIMATH SUHAIMA PU

MSc Computer Science Student, St. Thomas (Autonomous) College, Thrissur 680001, Kerala, India

Abstract - Facial Expression Recognition (FER) has become increasinglyvitalinemotion-awaresystemssuchasintelligent tutoring, healthcare,andsurveillance.However,theprocessing of facial video data raises significant privacy concerns due to the inherent exposure of biometric identity information. This review paper explores recent advancements in privacypreservingFER,withafocusonfederated learning, frequencybased de-identification,and expression-preserving face anonymization techniques. Central to this discussion is the novel dual-frequency framework proposed by Xu et al., which decomposes video data into high- and low-frequency components to isolate expression and identity features, respectively.Byapplyingcontrolledprivacyenhancementand feature compensation, the method achieves low identity leakage (2.01%) with high FER accuracy (78.84%) on the CREMA-D dataset.

Complementing this, recent literature demonstrates parallel efforts to balance privacy and utility. For instance, RAPOO utilizes mobile crowdsensing and lightweight encryption to enable privacy-aware FER on edge devices. StyleGAN-based imageanonymizationmethodsretainexpressivefeatureswhile effectively masking identities in educational and generalpurpose FER datasets. Additionally, hybrid CNN–ConvLSTM architecturesandcompound emotion modelsfurther improve spatiotemporal understanding and classification robustness. Across these approaches, a common challenge remains: designing architectures that maintain recognition accuracy without compromising personal privacy.

Key Words: FacialExpressionRecognition(FER);Privacy Preservation;WaveletTransform;De-identification;Feature Compensation; Identity Leakage; Video-based Emotion Recognition;DeepLearning

I. INTRODUCTION

FacialExpressionRecognition(FER)playsacrucialrolein enabling machines to interpret human emotions through facial cues. It has widespread applications in diverse domains such as mental health monitoring, driver fatigue detection, intelligent tutoring systems, and human–robot interaction. With the growing use of video-based FER in these sensitive environments, preserving the privacy of users especially their facial identity has become a pressing concern. FER systems typically rely on large volumes of facial video data, which inherently contain personallyidentifiableinformation.Thisraisesethical,legal, and societal questions surrounding data protection, user consent,andpotentialmisuse.

To address these concerns, the research community has begun exploring privacy-preserving FER (PP-FER) frameworks that can maintain recognition performance while suppressing identity information. A recent advancementbyXuetal.introducedafrequency-baseddualstream approach that decomposes video frames into lowfrequency (identity) and high-frequency (expression) componentsusingwavelettransforms.Theirmethodapplies controlled privacy enhancement to each component, followedbyafeaturecompensatorthatrestoresexpression featurescompromisedduringanonymization.Witha78.84% accuracyandonly2.01%identityleakageontheCREMA-D dataset,theapproachoffersacompellingbalancebetween privacyandutility.

Complementary to this, multiple other privacy-aware FER strategies have emerged in the literature. The RAPOO system enablesmobile-basedFERviacrowdsensingwhile protectingprivacyatthesource.Meanwhile,GAN-basedface anonymization techniques retain expression cues while removing biometric features in educational and public settings. Other models focus on enhancing expression recognition in the presence of privacy constraints using hybrid CNN–RNN networks and advanced temporal modeling.TheseinnovationsshowthatintegratingFERwith privacy-preserving mechanisms is not only feasible but increasinglyessentialinreal-worlddeployments.

Applications of Facial Emotion Recognition-

FER is a significant field because facial expressions are a crucialformofnon-verbalcommunication,conveyingupto 55%ofemotionalinformation.Theabilitytoautomatically recognize these expressions has led to a wide range of applications:

 Healthcare:FERsystemsareusedtodetectconditions like autism and neurodegenerative diseases and can help predict psychotic disorders or depression. In elderly care, it can be used to monitor patients and identifythosewhoneedassistance,includingforsuicide prevention.

 Education:Ineducationalsettings,FERcanbeusedfor intelligenttutoringandtomonitorstudentmoodsand attentionlevels,providinginsightsintoengagementand comprehension.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 11| Nov 2025 www.irjet.net p-ISSN:2395-0072

 Security: FER technology can be deployed for crime detection,suchasidentifyingandreducingfraudulent insuranceclaimsorspottingshoplifters.

 Human-Computer Interaction (HCI):FERenhances theinteractionbetweenhumansandmachines,making systems more empathetic and intuitive. It is used in areas like virtual reality (VR) and augmented reality (AR) to create more responsive and engaging user experiences. FER can be applied to real-time driver fatiguedetection,forexample.

Privacy Concerns and Regulations

Privacy is a major concern with FER, as it involves the processingofhighlysensitivebiometricdata.Theuseofthis technologyraisesseveralethicalandlegalissues,especially regardingmasssurveillanceanddatamisuse.

 Ethical Issues:Thetechnology'sabilitytobeusedfor masssurveillance,databreaches,andbiasedalgorithms posessignificantthreatstocivilliberties.FERsystems canbebiasedagainstcertaindemographicgroups,with astudyfindingthatalgorithmsassignedmorenegative emotions (like anger) to faces of African descent comparedtoothergroups.

 Regulations: The General Data Protection Regulation (GDPR) in Europe provides a strict framework for handling personal and biometric data. UnderGDPR,facialdataisconsidered"specialcategory" data,whichmeansitsprocessingisgenerallyprohibited unlessspecificconditions,likeexplicitconsent,aremet. Organizations must also adhere to principles of data minimization and purpose limitation. However, a key challenge is that individuals often have little control overdatacollectioninpublicspaces,makingitdifficult toobtaininformedconsent

 Risks of Sharing Facial Data: Once facial data is shared, it is vulnerable to cyberattacks, and unlike a password, it cannot be changed. A data breach of biometric information can have long-lasting consequences.Thismakesitcrucialtodevelopprivacypreservingtechniquesto removeidentityinformation whileretainingfeaturesneededforemotionrecognition. One example is a dual-frequency framework that decomposes facial data to isolate expression and identity features, achieving low identity leakage with highaccuracy.

II. LITERATURE SURVEY

Thetaskoffacialexpressionrecognition(FER)hasevolved significantly with the integration of deep learning models andrichvisualdatasets.However,traditionalFERsystems often overlook the implications of user privacy, especially

when facial identity information is exposed during data processing.Recentstudieshaveintroducedvariousprivacypreserving strategies to mitigate these concerns while maintainingtheeffectivenessofemotionrecognition

Xuetal.(2024)proposeda controlledprivacy-preserving FER framework that separates facial identity and expressionfeaturesusingawavelettransform.Byisolating high-frequency (expression) and low-frequency (identity) components, their two-stream model applies targeted privacy enhancement and compensates expression loss throughadedicatedfeaturecompensator.Thesystemalso incorporates a privacy leakage validator to quantitatively measureidentityexposure,achievingabalanceof 78.84% recognition accuracy and 2.01%identityleakage onthe CREMA-D dataset. This approach represents a significant advancementindecouplingutilityandprivacyobjectivesin FER.

Supporting this direction, the RAPOO framework (IEEE TMC, 2025) addresses FER on mobile platforms using a client-serverarchitecture.Itapplieslightweightencryption andprivacyfiltersonedgedevicesbeforeuploadingtocloud serversforexpressionrecognition,preservingidentityatthe data source. This system is optimized for crowdsensing environments where computational resources are limited but user trust is essential. actually use a slightly smaller, 9.5bp,fontsizeprobablytoSimilarly,AshwinandRajendran (Springer AIED, 2023) focus on educational settings, proposinga StyleGAN-basedde-identification methodthat swaps student faces with synthetic ones while preserving their expressions. The generated faces retain emotional attributes but are no longer recognizable, enabling FER in classroomswithoutcompromisingprivacy.

From a signal-processing perspective, a study by Samarakoon et al. (Springer, 2020) explored expressionpreserving face anonymization usingproxydatasetsand GAN-basedreconstruction.Theirmethodimprovesprivacy retention while ensuring that emotional data remains accurate,addressingbothbiometricsuppressionandutility maintenance.

In terms of FER architecture, recent works like those by Sharma et al. (Springer IJIT, 2023) propose hybrid CNN–ConvLSTMnetworks thatmodeltemporaldependenciesin facial expression sequences. Though not privacy-focused, such architectures demonstrate effective performance on dynamicvideodata,offeringpotentialutilitylayersthatcan integratewithprivacymodules.

dditi y ez-Gil et al. (Springer, 2025) extend FER capabilities to compound emotions by introducing advancedfeaturelocalizationandtextureanalysis.Whilenot explicitly designed for privacy, these models contribute towardricherexpressionrecognition,whichiscriticalinFER systemsunderprivacyconstraintswheredatacuesmaybe partiallyobscured.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 11| Nov 2025 www.irjet.net p-ISSN:2395-0072

Similarly, Jenga et al. (2023) presented a systematic literature review focused on machine learning for crime prediction,evaluatingstate-of-the-arttechniquesdeveloped overtheprecedingdecade.Theirstudy,whichencompassed 68 selected machine learning papers, aimed to synthesize knowledge regarding ML-based crime prediction to assist lawenforcementauthoritiesandscientistsinmitigatingand preventing future crime occurrences. Jenga et al. meticulouslydiscussedthepossiblechallengesinherentin thefieldandprovidedaforward-lookingdiscussionoffuture work.Akeyobservationfromtheirreviewwasthatmostof theanalyzedpapersutilizedasupervisedmachinelearning approach, predicated on the assumption of prior labeled data. This study underscores methodological preferences withinthefieldandreinforcestheongoingneedtoaddress practicalchallenges,suchasdataavailabilityandtheethical implicationsofpredictivepolicing.

III. Methodology

Thisreviewadoptsasystematicmethodologytoinvestigate recentadvancementsinprivacy-preservingfacialexpression recognition(FER),withparticularemphasisonframeworks thatcombinede-identificationandemotionrecognition.The process involved three main stages: collection of relevant literature,screeningandselection,andanalysisofthechosen studies.

The literature collection was conducted through major scientific databases including IEEE Xplore, SpringerLink, ACM Digital Library, and ScienceDirect. To capture a comprehensivesetof works,awiderangeofsearchterms w s em yed such s “f ci ex ressi rec g iti ” “ riv cy- reservi g FER” “f ce de-ide tific ti ” “G Nb sed ymiz ti ” d “VGG19 em ti rec g iti .” The search was limited to the period 2015–2024, as this timeframereflectstherapidemergenceofdeeplearningand privacy-enhancedFERtechniques.

The screening and selection process followed a two-step filtering approach. Initially, papers were included if they specifically addressed FER using deep learning models, proposed or evaluated privacy-preserving or deidentification techniques, and provided experimental validationusingbenchmarkdatasetssuchasFER2013,CK+, RAF-DB,AffectNet,orCREMA-D.Exclusioncriteriaremoved studiesthatdidnotexplicitlyconsiderprivacy,focusedonly onfacerecognitionwithoutFER,lackedempiricalresults,or werenotpublishedinEnglish.Afterapplyingthesefilters, approximately40paperswereretainedforin-depthreview, including both IEEE and Springer publications to ensure coverageofhigh-qualitysources.

The analysis and synthesis stage concentrated on three major aspects. First, the privacy mechanisms were categorized into pixel-level (e.g., blurring, adversarial cloaking), representation-level (e.g., identity-invariant feature learning), and semantic-level (e.g., GAN-based de-

identification).Second,theutilitypreservationofFERwas assessed by examining how well different approaches maintained expression recognition accuracy, with comparisons against established CNN baselines such as VGG19, ResNet, and emerging transformer-based models. Finally, the privacy–utility trade-off was analyzed by contrasting re-identification rate reduction with FER accuracymetricssuchasoverallaccuracyandF1-score.This structured analysis enabled a clear understanding of the state-of-the-art,commonlimitations,andresearchgapsin privacy-enhancedFER.

Feature Extraction with Convolutional Neural Networks (CNNs) and VGG19

ConvolutionalNeuralNetworks(CNNs)areaclassofdeep learning models specifically designed for image analysis. Theyfunction bylearninga hierarchical representationof featuresfromrawpixeldata.Thisprocessisdrivenbythree maintypesoflayers:



ConvolutionalLayers:ThisisthecoreofaCNN.It appliesasetoflearnable filters (orkernels)totheinput image. Each filter slides over the image, performing a dot product with the underlying pixels to produce a feature map.Earlylayerslearntodetectsimple,lowlevelfeatureslikeedges,lines,andcurves.

 Pooling Layers: These layers are used to down sample the feature maps, reducing the spatial dimensions of the data. This helps to make the model morerobusttominorvariationsintheinputimage,such asslightshiftsordistortionsoftheface.

 Fully Connected Layers: After multiple convolutional and pooling layers have extracted complexfeatures,thesefinallayersflattenthedatainto avectoranduseittoperformthefinalclassification,in thiscase,predictingtheemotion.

Privacy-Preserving Techniques in FER

 Face De-identification: This involves modifying facialimagestoobscureaperson'sidentity.Thiscanbe donebyaddingnoise,blurring,orpixelatingpartsofthe face. More advanced methods, such as the dualfrequencyframeworkmentionedinyourpaper,workby selectively removing identity-related features (which are often found in low-frequency image components) while retaining expression-related features (highfrequencycomponents).

 Homomorphic Encryption: This is an advanced cryptographictechniquethatallowscomputationstobe performeddirectlyonencrypteddata.Inthecontextof FER,animageorits featurescouldbe encrypted ona user'sdevice,senttoaserverforemotionrecognition, and then processed without ever decrypting the data. Thisensuresthattherawfacialdataisneverexposedin

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 11| Nov 2025 www.irjet.net p-ISSN:2395-0072

plaintext to the service provider, though it can be computationallyexpensive.

 Federated Learning: This is a decentralized machinelearningapproachwherethemodelistrained collaboratively across multiple devices without exchanging the raw data. Instead, individual devices trainthemodellocallyontheirowndata,andonlythe updatedmodelparameters(notthedataitself)aresent toacentralservertobeaggregated.Thiskeepssensitive facialdataontheuser'sdevice,significantlyenhancing privacy.

 DifferentialPrivacy:Thistechniqueaddsacontrolled amount of random noise to the data or model parameters. The noise is carefully calibrated to be sufficienttopreventtheidentificationofanyindividual user from the final model, but not so much that it significantlydegradesthemodel'saccuracy.

IV. Results and Discussion

The analysis of selected studies reveals several important trends in the field of privacy-preserving facial expression recognition(FER).

One of the most consistent findings is that deep learning architectures,particularlyCNN-basedmodelssuchasVGG19 and ResNet, continue to serve as baseline classifiers for evaluatingFERaccuracybeforeandafterde-identification. Theirstraightforwardarchitecturesmakethemsuitableas “utiity critics” e suri g th t ymiz ti tech iques preserveexpression-relevantcues.Atthesametime,more recentworkshavebegunincorporatingtransformer-based architecturestocapturelong-rangedependencies,although theirintegrationwithde-identificationframeworksisstillin earlystages

Intermsofprivacymechanisms,threedominantcategories emerged: pixel-level, representation-level, and semanticlevel.Pixel-levelmethods,includingblurring,pixelation,and adversarialcloaking,providelightweightprivacybutoften degrade subtle facial cues essential for distinguishing between emotions such as fear and surprise. Representation-levelapproaches,whichdisentangleidentity fromexpressioninlatentspace,demonstratestrongerutility preservationbutrequirelarge-scaleannotateddatasetsfor effective adversarial training. Semantic-level methods, especiallythosebasedonGenerativeAdversarialNetworks (GANs)andStyleGANvariants,achievethemostpromising balance. These frameworks can synthesize de-identified faces while maintaining action units (AUs) and motion dynamicsnecessaryforFER,withseveralstudiesreporting FERAccuracyabove75–80%onbenchmarkdatasetswhile reducingre-identificationratestobelow5%.

Another key finding relates to datasets and evaluation protocols.WidelyusedFERdatasetssuchasFER2013,RAF-

DB, AffectNet, and CREMA-D dominate the field, with CREMA-D often chosen for video-based analysis. Most studiesreportperformanceintermsofFERaccuracyandF1score,alongsideprivacymetricssuchasidentityleakageor verification success rate. However, evaluation remains inconsistentacrossworks,withnostandardizedbenchmark that jointly measures privacy protection and FER performance. This lack of uniformity limits comparability andhindersreproducibilityofresultsacrossframeworks.

Theprivacy–utilitytrade-offremainsthecentralchallenge. Whilesemantic-level methodsshowstrong potential,they occasionally sacrifice accuracy for fine-grained emotions, especially micro-expressions, which are easily distorted duringidentitymanipulation.Conversely,approachesthat maximize accuracy may not adequately suppress identity cues,raisingrisksofre-identification.Severalrecentstudies highlightthistension,reportingnoticeableaccuracydrops f r subte c sses such s “disgust” r “fe r” whe str g anonymization is applied. This underscores the need for hybridframeworksthatcombinesemanticpreservationwith adversarial robustness to adaptive attacks such as decloakingorpurification.

Finally, the discussion of challenges highlights three persistent gaps. First, robustness remains limited, as deidentificationmethodsarevulnerabletocountermeasures thatattempttorestoreorpurifycloakedfeatures.Second, fairness and bias issues have been insufficiently explored; currentframeworksrarelyevaluateprivacy-preservingFER across demographic subgroups, raising concerns about equityinreal-worlddeployments.Third,explainabilityisstill underdeveloped.Fewstudiesexaminehowde-identification affects feature saliency maps or the interpretability of expression recognition decisions. Addressing these limitations is critical for developing trustworthy and deployablesystems.

V. Conclusion

This review explored the state-of-the-art in privacypreservingfacialexpressionrecognition(FER),focusingon frameworks that integrate de-identification with emotion recognition. The works surveyed highlight three main privacyapproaches:pixel-levelobfuscation,representationlevel identity disentanglement, and semantic-level generative models. Among these, GAN-based semantic methodsappearmosteffective,maintainingFERaccuracies above 75–80% on benchmark datasets while significantly lowering identity leakage. Representation-level methods alsoshowpromisebutdependonlargeannotateddatasets, whereaspixel-leveltechniquesoftencompromiseexpression fidelity.

TheanalysisindicatesthatVGG19andsimilarCNNsremain importantbaselinesforevaluatingtheeffectivenessofdeidentification, often used as utility critics to ensure that emotion cues are preserved. However, the privacy–utility

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 12 Issue: 11| Nov 2025 www.irjet.net p-ISSN:2395-0072

trade-off is still unresolved: stronger anonymization can distort subtle emotions, while accuracy-focused methods may leave identity cues intact. A further limitation is the absence of standardized benchmarks that jointly measure privacyprotectionandFERaccuracy,makingitdifficultto comparemethodsfairly.

Lookingahead,severalresearchdirectionsremaincrucial. Thereisaneedforrobustde-identificationmethodsthatcan withstand adaptive attacks such as de-cloaking or purification. At the same time, fairness and bias must be addressed to ensure consistent performance across demographic groups. Another priority is explainability, where future studies should examine how anonymization influencesexpressionfeaturesanddecision-making.Finally, thecreationofstandardizeddatasetsandevaluationmetrics would greatly improve reproducibility and accelerate progressinprivacy-preservingFER.

REFERENCES

[1]P.Dr zd wski C.R thgeb dC.Busch “Dem gr hic bias in biometric systems: A survey on an emerging ch e ge”J ur f mbie tI te ige ce dHum ized Computing,vol.11,no.10,pp.4535–4556,Springer,2020.

[2]S. i dW.De g “Dee f ci ex ressi rec g iti : survey”IEEETr s cti s ffectiveComputing,vol.13, no.3,pp.1195–1215,2022.

[3] X.Hu g Q. i d Y. Zh u “Priv cy-preserving facial ex ressi rec g iti vi dvers ri e r i g” IEEE Access,vol.9,pp.110234–110245,2021.

[4] Mu khiddi v M. et . “M sked F ce Em ti n RecognitionBasedonFacialLandmarksandDeepLearning r ches f r Visu y Im ired Pe e” Se s rs v . 23(3), 1080, 2023. (not privacy-focused, but relevant to expressionrecognitionunderobfuscation)

[5] R. Me g X. i d Y. Di g “Ide tity-invariant re rese t ti e r i g f r f ci ex ressi rec g iti ” IEEETrans.Biometrics,Behavior,andIdentityScience,vol.3, no.4,pp.425–438,2021.