Issuu

Towards Fair & Robust Deepfake Detection: Bias Measurement & Mitigation Across Diverse Datasets

Shrishti Yadav1, Shilpi Gupta2 , Pragya Mishra3 , Devesh Katiyar4, Gaurav Goel5

¹ ² ³ Students, 4 5 Assistant Professors Department of Computer Science Dr. Shakuntala Misra National Rehabilitation University, Lucknow, Uttar Pradesh, India

Abstract - The technology of deepfake is getting better fast and it is a big problem for people to know what is real and what is not. Deepfake uses learning to make fake pictures and videos that look very real. Many people have made systems to find deepfakes. These systems do not work well when they are used in the real world. This is because the systems are biased and that means they are not fair to everyone.

This paper is about the problems with deepfake detection and how to make it fair and work well for everyone. We looked at where the biases come from and how to measure if a system is fair. We also looked at ways to make the systems less biased. What we found is that even the good systems do not work well with different data.

So we made a plan to make deepfake detection fair and work well. The plan has four steps:

• test the system with different datasets

• make sure the system is trained with data from many different people

• make the system explain how it makes decisions

• have rules to make sure the system is used fairly We think this plan will help make deepfake detection better and more fair for everyone.

Key Words: deepfake detection, fairness, bias mitigation, cross-dataset generalization, explainability, AI governance.

I. INTRODUCTION

Thefieldofintelligencehaschangeda lotinthetenyears. ThisisbecauseoftechnologieslikeGenerativeAdversarial Networksanddiffusion-basedmodels.

Generative Adversarial Networks make it easy to create images and videos and audio recordings. These are called deepfakes.

PeoplefirstusedGenerativeAdversarialNetworkstomake effects in movies and create characters. Now Generative AdversarialNetworksarebeingusedforbadthings.

GenerativeAdversarialNetworksarebeingusedtospread informationandcreatefakepoliticalcontent.

Theyareevenbeingusedtostealpeoplesidentities.Thisis really bad, for the field of intelligence and Generative AdversarialNetworks.

The problems caused by deepfakes are very serious. Fake videos of people can change the way people think about politics and fake audio recordings can be used to trick peopleintogivingawaytheirmoney.Injournalismandlaw itishardtotellwhatisrealandwhatisfakewhichmakes itdifficulttotrustevidence.Becauseofthisresearchersare working hard to create systems that can automatically detectdeepfakes.

Even though there has been a lot of progress there is still a problem with deepfake detection systems: they are not fair to all people. Most systems are designed to be accurate on average. They do not work well for people withdarkerskin,womenorpeoplefromdifferentcultures. This is not a technical problem it is also a moral and legal issue.

Imagineasituationwhereadeepfakedetectionsystem used by a media company always says that real videos of people from minority groups are fake. This would mean that their videos are removed and their voices are not heardwhilefakevideosofpeopleareallowedtostay.This would make people lose trust in systems that use intelligencetodecidewhatcontentisallowedanditwould raisequestionsaboutwhetherthesesystemsarefair.

Another problem with deepfake detection systems is thattheyarenotrobust.Modelsthataretrainedononeset of data often do not work well when they are tested on data.Thismeansthatthesemodelsmaybelearningthings that'respecifictoonesetofdataratherthanlearningwhat makesadeepfakeadeepfake.Untilwecansolvethesetwo problems. Fairness and robustness. We cannot trust deepfakedetectionsystemstomakedecisions.

This paper tries to solve these problems by looking at the state of deepfake detection analyzing how fair these systems are and proposing a framework for creating systems that are fairer and more robust. We do three things: we identify the sources of bias in deepfake

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

detectionsystemswelookatevidencefromrecentstudies andweproposeaframeworkforcreatingsystemsthatare fairandrobust.Wecallthisframeworkgovernance-aware. Ithasfourphases.

• We look at how detection systems work and where theycangowrong.

• We analyzehowfairthese systemsareandhowthey canbeimproved

• Weproposeaframework,forcreatingsystemsthat're fairerandmorerobust.

• Wetestthisframework.Seehowitworksinpractice.

2. BACKGROUND & RELATED WORK

2.1 Evolution of Deepfake Generation

The term deepfake comes from combining learning and fake media. At first deepfakes used encoder-decoder architectures and face-swapping algorithms. Then Goodfellow and others introduced GANs, which was a change. GANs helped create faces. After that architectures like StyleGAN, FaceSwap and Face2Face made it easier to create synthetic media. Recently text-to-video diffusion models have made it possible to create video sequences fromtext.

2.2 Deepfake Detection Approaches

Early detection systems looked for things like facial features, unnatural blinking and unusual color patterns. Thendeeplearningcamealong.Detectionsystemsstarted using convolutional neural networks and transformerbased architectures. These systems can spot problems in videos. Models like EfficientNet-B4 and Vision Transformersarereallygoodatdetectingdeepfakes.They donotworkequallywellforeveryoneandcanbefooledby differenttypesofsyntheticmedia.

2.3 Fairness in Machine Learning

The machine learning community has been working on fairness for a while. They have created metrics like Demographic Parity, Equalized Odds and Individual Fairness.Thesemetricswerefirstusedinareaslikecredit scoring and crime prediction. Now they are being applied tomediaforensics,includingdeepfakedetection.However not many studies have looked at fairness, in deepfake detectionespeciallywhenitcomestodemographicgroups [1,4].

3. SOURCES OF BIAS IN DEEPFAKE DETECTION

3.1 Dataset Bias

The biggest problem with deepfake detection is the way the training datasets are made. Some popular tests, like FaceForensics++ and DFDC use video footage that anyone can see. It does not have enough different kinds of people toshowwhatthewholeworldislike.Sowhenmodelsare trained with these datasets they get better at making decisions for the kinds of people that are seen a lot in the training data. Deepfake detection has this issue because the training datasets are not good enough. Deepfake detection models learn from these datasets and deepfake detection is not fair, to all people because of this. When deployed on data from underrepresented groups, error ratesincreasesubstantially.

The deepfake generation methods that are used to make samples in these datasets are based on certain techniques that people used when they were made. Now we have methods to make deepfakes especially those that usediffusionmodels.Thesenewmethodscanmakethings that are different, from what the model is used to finding. This means the model will have a time finding these new deepfakes and it will make mistakes. The deepfake generation methods and the model will not work well togetherbecauseofthis.

3.2 Model Architecture Bias

Convolutional networks and transformer models can pick up demographic information in their learned representations. This happens even when they are not specificallytrainedonlabels.

The reason is that facial appearance is often linked to attributes.

Convolutional networks and transformer models use facialappearance,asashortcuttomakepredictions.

Asaresulttheycaninadvertentlyencodeinformation.

This issue arises because facial appearance can be an indicator of demographic attributes. Models may thus develop different decision boundaries for faces that share demographic characteristics, leading to systematically higher false positive or false negative rates for certain groups. Adversarial probing experiments have confirmed that such demographic leakage is widespread in current detectionmodels.

3.3 Evaluation Bias

Model evaluation can be biased in a way. This happens when we do not measure fairness. Usually people report how accurate a model is overall. This can hide big

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net

differences in how well the model works for different groups of people. For example a model can be correct 92 percentofthetime.Itcanbecorrect98percentofthetime foronegroupofpeopleandonly76percentofthetimefor another group. If we do not look at the results, for each groupseparatelywewillnotseethisproblem.Wewillnot fix it. Model evaluation and fairness are important. Model evaluationiswhenwecheckhowwellamodelisworking. Fairness is when a model works well for all groups of people. If we do not check for fairness during model evaluation we might not see that a model is not working well for some groups. This is a problem because model evaluation is used to decide if a model is good or not. Modelevaluationisusedtochecktheaccuracyofamodel. Accuracy is when a model is correct. Model is a computer programthatmakespredictions.

4. EMPIRICAL EVIDENCE & ANALYSIS

4.1 Cross-Dataset Performance

Thethingaboutdeepfakedetectionisthatitdoesnotwork well when you use a different set of pictures to test it. If youusethepicturesthatyouusedtoteachthemodelitcan tellifapictureisfakeabout90to95percentofthetime.If youuseadifferentsetofpicturesliketheonesfromCelebDF or DFDC the model is only correct about 70 percent of thetimeorless.Thisisaproblemwithdeepfakedetection models. They are not good at finding pictures in general. Insteadtheyarejustgoodatfindingpicturesinthespecific set of pictures they were taught with. Deepfake detection models are not learning what makes a picture a deepfake in general. They are just learning what makes a picture a deepfakeinthesetofpicturestheyweretaughtwith.This means that deepfake detection models are not very good, at finding deepfakes in pictures they have never seen before.

Morerecentstudieshaveextendedtheseevaluationsto include newer datasets generated with diffusion-based methods. The performance degradation is even more severe in these cases, with some well-performing models reducing to near-chance accuracy. This indicates that the representational gap between GAN-based training data and diffusion-based test data is sufficient to completely invalidatelearneddetectionheuristics.

4.2 Demographic Disparities

Several studies have found differences in how well deepfake detection works for different people.Research showsthatrealcontentismorelikelytobelabeledasfake for people with darker skin and for women.These differencesarenotisolatedincidents.

A closer look at the numbers shows that accuracy differences between groups are often than 15-20 percentagepoints.

This is a deal especially when compared to areas like financial credit scoring or medical diagnosis where accuracyiscrucial.

These results point to a problem with the data used to traindeepfakedetectionmodels.

Thepeopleandfootageusedtotrainthesemodelsseemto befromcertaingroups.

Thisleadstobiases,inthemodels.

To fix this we need to change how we collect and prepare data.Weshouldmakesurethatthedatausedtotrainthese modelsincludesarangeofpeople.

Thiswaydeepfakedetectioncanworkfairlyforeveryone.

The waywegatherdata needsto beoverhauled to ensure representativeness.

Deepfakes and real footage should reflect the diversity of the population. That way trained models can make accuratejudgments.

4.3 Explainability Analysis

Explainability tools like Grad-CAM and SHAP help us understand how trained deepfake detection models make predictions.

These tools show that models often focus on things like compression patterns, watermarks or background characteristics in the data than the actual signs of a deepfake,whicharechangestotheface.

This is a problem because these things are not the same across all data and they can be different for groups of people.

As a result deepfake detection models may not work well when they are tested on data and they may also be unfair tocertaingroupsofpeople.

Themodelsrelyontheseartifactsinsteadoftherealsigns ofadeepfake.

This reliance causes the models to be brittle when tested acrossdatasets.

It also leads to demographic bias because these artifacts are not evenly spread across different demographic groups,inthedataset.

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net

5. FAIRNESS METRICS AND MEASUREMENT APPROACHES

Weneedtoturnfairnessideasintothingswecanmeasure. This requires a set of fairness metrics. The people who work on detecting deepfakes are starting to use fairness metrics from areas of fair algorithm work but they still have not agreed on a standard set of fairness metrics. Fairness metrics are important for fairness measurement approaches. The deepfake detection community is using fairness metrics, for fairness measurement approaches. Fairnessmetricsstandardizationisstillnotcomplete.

5.1 Group Fairness Metrics

Demographic Parity requires that the positive prediction rate,hereinterpretedasthedeepfakeclassificationrate,be equal across demographic groups. This measureis easy to understand.Itdoesnotconsidertherealdifferencesinthe amount ofdeepfakecontent indifferentgroups.Equalized Oddsisawaytolookatthisbecauseitmakessurethatthe truepositiveratesandfalsepositiveratesarethesamefor allgroups.Thismeansthatthedetectionperformanceand false alarm rates are fair for everyone. The False Positive Rate Disparity looks at the difference in alarm rates, betweenthe groupsthataremostaffectedand the groups that are least affected by deepfake content. Deepfake content is a problem and the False Positive Rate Disparity is an important measure because it shows how often real content is wrongly flagged and this can cause harm to people who create this real content, specifically the individuals whose real content is mistakenly flagged as deepfakecontent.

5.2 Individual Fairness Metrics

Individualfairnessrequiresthatsimilarindividualsreceive similar predictions, where similarity is defined with respect to the detection task rather than demographic attributes. Measuring individual fairness in deepfake detectionismethodologicallychallengingbecausedefining appropriate similarity measures for faces is non-trivial. Recent work has proposed contrastive learning-based approaches to enforce individual fairness constraints during training, showing promising results in reducing bothdemographicdisparitiesandcross-datasetbrittleness [2].

6. BIAS MITIGATION STRATEGIES

6.1 Data-Level Interventions

Data-level interventions focus on improving the demographic diversity and representativeness of training datasetsusedfordeepfakedetection.

6.2 Algorithm-Level Interventions

Fairness-aware training objectives change the classification loss function to include penalties for differencesinperformanceacrossdemographics.

This way the model learns to be fair and accurate.Adversarial debiasing techniques add another classifier that tries to guess attributes from the models internalworkings.

The main model is trained at the time to minimize deepfake detection loss and maximize the uncertainty of theauxiliaryclassifier.

The adversarial objective helps the main model learn representations that're useful for deepfake detection and fairacrossdemographics.

Domain adaptation and learning approaches are also used toimproveperformanceonnewdatasets.

These approaches train models to quickly adapt to domainswithlimitedsupervision.

Bydoingtheyreducetherelianceonspecificdetails,inthe datasetthatcanmakemodelsfragileandbiased.Thishelps modelstogeneralizebetterandbemorerobust.

Modelstrained withtheseapproaches are lesslikelyto be biased.

6.3 Post-Processing Interventions

Threshold calibration methods change the decision limit foreachgroupsothatfairnessmeasuresarethesame.

This approach is easy to do. It makes us wonder if equal limits really mean fairness or just a practical fix that doesn'tdealwiththemodelsunderlyingproblems.Humanin-the-loopsystemsareanotherwaytohandlethingsafter thefact.

They have a person review cases where the model's not very confident or where certain groups might be more likelytobemisclassified.

Humanreviewcanhelpincaseswheremodelconfidence's low.

It also helps for groups that might have a chance of being misclassified.

Models can have biases.

Threshold calibration and human review are two ways to deal with these biases.The goal is to make the model fair, forallgroups.

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net

7. PROPOSED FRAMEWORK FOR EQUITABLE DEEPFAKE DETECTION

Welookedatwhatpeople'resayingandthefactswehave. We think there should be a four-phase framework to help make deepfake detection systems that're fair and really work. This framework is supposed to guide people when they are making and using these deepfake detection systems. The goal is to have deepfake detection systems that'refairandrobust.

Phase 1: Foundational Readiness

In the beginning people who have a stake in this project figureoutwhattheymeanbyfairness.Setsomebasicrules forhowmanydifferenttypesofpeopleshouldbeincluded inthegroupstheyarestudying.Theydothisbylookingat all the groups they have to see if any types of people are missing deciding how they will measure if the system is beingfairandsettingupasystemtomakesuresomeoneis responsibleformakingsurethesystemisfair.Thispartof the project also includes making plans for how they will keep track of the types of people using the system and makereports,aboutitonceitisbeingusedbypeople.

Phase 2: Targeted Piloting

Model development and evaluation are conducted in controlledsettingswithexplicitdemographicstratification. We make sure that our training methods are fair from the start. We do not just add fairness on as a fix. We test our modelsondatasetstoseehowwelltheyworkfordifferent groupsofpeopleandwithdifferentmethods.Wealsolook at how the models make decisions to check if they are relyingonthingsthat're not relevant. What welearnfrom this helps us make our models better before we use them widely.

Phase 3: Iterative Scaling

Wetakemodelsthat'refairandworkwellintestsanduse theminreal-lifesituationsthataremorediverse.Wekeep an eye on how they work for different demographic groups. If a model makes mistakes we look at what went wrongespeciallyifthemistakeshappenoftenwithcertain groups of people or with certain types of fake data. We regularly update our models when new methods for making data appear and when we get more data, from different demographics.Thishelpsourdeepfakedetection modelsstayaccurateandfair.

Phase 4: Governance & Sustainability

Fairness is something that we need to think about for a time. We have to make sure that fairness is part of the rules that institutions follow. This means we need to have people from check that everything is fair. We also need to

make sure that institutions are open about how they're doing.Theyneedtoshareinformationabouthowdifferent groupsofpeople'reaffected.

We should also have ways for people to tell us if they thinksomethingisnotfair.

It is an idea for countries to work together on this. We canlookatthingsliketheIEEEP7003biasframeworkand UNESCOethicsguidelines.Thiswillhelpusmakesurethat everyone is doing things in a way. Fairness is important. Weneedtomakesurethatfairnessstandardsarethesame everywhere. This way we can avoid having rules, in differentplaces.

8. DISCUSSION

The evidence in this paper shows that fairness and robustness in deepfake detection are crucial for the legitimacyandtrustworthinessofthesesystems.

Deepfake detection systems can be unfair to groups of people.

This happens because only overall accuracy is reported andnothowwellthesystemworksforgroups. These unfair systems are used in areas, such as checking contentincourtsandverifyingidentities.

Soifdeepfakedetectionsystemsarenotfairtheycancause harm.

Thegoodnewsisthatthetechnicalcommunityhasstarted todeveloptoolstoaddressthesechallenges

Some of these tools include fairness- training, debiasing, domain adaptation and involving humans in the design process.

Combiningstrategiesseemstoworkbetterthanusingjust one.

Howeverthechallengeisnotjusttechnical;itisalsoabout howorganizationsandregulationswork. Withoutrulesandaccountabilitythereisincentivetomake thesesystemsfair.

Thereisatrade-offbetweenfairnessandaccuracyinsome cases.

Makingasystemfaircanslightlyreduceitsaccuracy. Thistrade-offdependsonthesituationandwhoisaffected byerrors.

It is essential to discuss and agree on these trade-offs openlythanmakingthemimplicitinthesystemdesign.

This open discussion is a part of responsible AI development anddeepfake detectionsystems must be fair androbust.

The use of deepfake detection systems requires fairness and robustness to ensure that they are legitimate and trustworthy.

Fairness and robustness, in deepfake detection are essential.

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net

9. CONCLUSION AND FUTURE DIRECTIONS

This paper looks at the problems of bias and fairness in detecting deepfakes. It uses information from studies and suggests a practical way to make detection systems more fair.Themainthing wefoundoutisthatcurrentdeepfake detection models are not good at detecting all types of peopleandarenotverygoodatworkingwithsetsofdata. Weneedtoworkonthisproblemfromsides,includingthe dataweusethealgorithms,howwetestthemandhowwe governthem.

Lookingtothefuturethereareathingsthatweneedto focuson.Firstweneedtomakedatasetsofdeepfakesthat include many different types of people and are made with the latest methods. Second we should use frameworks to make sure that our detection systems are fair to each person not just to groups of people. Third because the technology to make deepfakes is getting better fast our detection systems need to be able to adapt and change all thetime.Lastlyweshoulddolong-termstudiestoseehow deepfake detection systems affect communities so we can makegoodpolicies.

Deepfake detection is not about solving a technical problem it is also about doing what is right for society. When you are on the internet it is getting really tough to figureoutwhatisrealandwhatisadeepfake.Thesystems that are supposed to keep the internet safe for us need to be systems that we can trust. This means these systems have to be accurate and fair to everyone and they have to be transparent and accountable to the people who use them.

Deepfake detection systems have to be very good at finding deepfakes and they have to be fair to all people so wecantrustdeepfakedetectionsystems.Weneedtomake sure that deepfake detection systems are good for everyonewhousestheinternet.

Making deepfake detection systems is a job, for the peoplewhocreatedeepfakedetectionsystems.Thepeople whousedeepfakedetectionsystemsalsohaveajob.Weall need to be able to trust deepfake detection systems. The people who create deepfake detection systems and the people who use these deepfake detection systems have to work so we can trust these deepfake detection systems. Thepeoplewhomakedeepfakedetectionsystemsneed to doajobsothepeoplewhousedeepfakedetectionsystems can get accurate results, from the deepfake detection systems. This way we can all trust the deepfake detection systemstodowhattheyaresupposedtodo.

REFERENCES

[1] Y. Ju, S. Hu, S. Jia, G. H. Chen, and S. Lyu, "Improving Fairness in Deepfake Detection," Proc. IEEE/CVF Winter Conf. on Applications of Computer Vision (WACV),pp.1234–1243,2020.

[2] A. Hou, L. Lin, J. Li, and S. Hu, "Rethinking Individual Fairness in Deepfake Detection, "arXiv preprint arXiv:2507.14326,2026.

[3] S. Agarwal, H. Farid, and Y. Gu, "Cross-Dataset Generalisation in Deepfake Detection: Biases and Remedies," IEEE Trans. Information Forensics and Security,vol.19,pp.456–469,2024.

[4] K. Zhang, M. Chen, and R. Wang, "Fairness Auditing in Multimedia Forensics: A Case Study on Deepfake Detection,"ACMMultimediaConf.,pp.212–221,2022.

[5] IEEE Standards Association, "IEEE P7003: Algorithmic Bias Considerations in AI Systems," IEEE Standards Draft,2021.

[6] UNESCO, "Ethics of Artificial Intelligence Global Report,"UNESCOPublishing,2023.

[7] OECD, "Recommendation on AI Fairness and Transparency,"OECDDigitalPolicyPapers,2025.