Issuu

A Hybrid AI Framework for Proactive Fraud Detection and CyberThreat Intelligence in Real-Time Banking Systems

Prathap Raghavan1, Vinayak Elangovan2

1Independent Researcher, Coppell, TX, USA

2computer Science Program, Penn State Abington, Abington, Pennsylvania, USA

Abstract - The modern banking sector faces an escalating onslaught of sophisticated cyber-fraud and coordinated attacks, including authorized push payment (APP) fraud and AI-generated social engineering, which routinely bypass conventional, rule-based detection systems. These legacy systems generate excessive false positives, lack contextual awareness, and are inherentlyreactive, leadingtosignificant financial losses and erosion of customer trust. This paper proposes a novel, hybrid AI framework that synergistically integrates deep learning, graph analytics, and real-time threat intelligence to transition from reactive fraud blocking to proactive threat mitigation. The proposed architecture is built upon three core pillars: a Real-Time Deep Behavioral Analytics Engine using a Dual-LSTM Autoencoder for anomaly detection in transaction sequences and user behavior, a Dynamic Graph Neural Network (GNN) that models complex temporal relationships between entities (accounts,devices,IPs)touncoverorganizedfraudrings, and an AI-Powered Threat Intelligence Hub that employs Transformer-based NLP (e.g., fine-tuned BERT, RoBERTa) to ingest and correlate unstructured data from security feeds, dark web sources, and encrypted messaging platforms. The framework's efficacy is validated through a large-scale industrial case study with a multinational bank, processing over 1.2 billion transactions monthly. Results demonstrate a 52% improvement in fraud detection rate, a 67% reduction in false positives, and an 8x faster identification of emerging threat patterns compared to the incumbent system. We further discuss implementation challenges, including data privacy-preserving techniques like Federated Learning and the critical need for model explainability (XAI) using SHAP and LIME to meet regulatory compliance (e.g., GDPR, PSD2). This work substantiates that a deeply integrated, multimodal AI approach is paramount for constructing resilient and intelligent digital banking ecosystems.

Key Words: AIinFinance,FraudDetection,GraphNeural Networks, Threat Intelligence, Anomaly Detection, Behavioural Biometrics, Explainable AI (XAI), Real-Time Analytics.

1. INTRODUCTION

The digital transformation of the global banking industry, acceleratedbythe rise of real-timepaymentrailslike UPI andFedNow,hasunlockedunprecedentedconveniencebut

has simultaneously expanded the attack surface for maliciousactors.Financialinstitutionsnowcontendwitha complex threat landscape featuring real-time payment fraud, application fraud, account takeover (ATO) attacks, and sophisticated, AI-generated social engineering schemes such as deepfake audio for vishing (voice phishing)[1].Theglobalcostoffraudisprojectedtoexceed $40 billion annually, underscoring the critical need for advanced defensive systems [2]. Traditional Fraud Detection Systems (FDS), which rely predominantly on static rules and supervised learning models trained on historical data, are fundamentally ill-equipped to counter these evolving, adaptive threats [3]. Their limitations are threefold: (a) High False Positive Rate (FPR), which degradescustomerexperienceandincurshighoperational costs for manual review, often exceeding 80% of alert volumes;(b)InabilitytoDetectNovelAttacks,astheylack generalized behavioral understanding and fail to identify zero-dayfraud;and(c)SiloedAnalysis,wheretransaction monitoring is disconnected from broader cyber-threat contextsandrelationalanalysis[4].

Artificial Intelligence (AI) and Machine Learning (ML) present a paradigm shift, enabling systems to learn complex, non-linear patterns and adapt to new fraud signatures in near real-time [5]. While prior research has explored isolated AI applications such as using Isolation Forests for point anomaly detection [6] or Recurrent Neural Networks (RNNs) for sequence modeling [7] a holistic framework that unifies behavioral anomaly detection, relational graph analysis, and external threat intelligenceintoasingle,proactivedefenseloopremainsa significantresearchandoperationalgap.Furthermore,the adventofLargeLanguageModels(LLMs)introducesbotha newattackvectorandapowerfultoolfordefense,aduality thatmustbeaddressed[8].

Thispapermakesthefollowingkeycontributions:

a) Propose a novel, hybrid AI architectural framework for proactive fraud detection and threat intelligence, detailing its end-to-end data flowandmodelinteractions,specificallydesigned for the low-latency requirements of real-time banking.

b) Provide a deep technical analysis of its core components: a Dual-LSTM Autoencoder for behavioral profiling, a Temporal Graph Neural

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Network (TGNN) for dynamic relational analysis, andaTransformer-basedNLPengineforreal-time threatintelligencecorrelationfromstructuredand unstructuredsources.

c) Presentrobustempiricalvalidationthrougha 12month industrial case study with a multinational bank, quantifying performance gains in detection rate, false positives, and threat discovery speed withcompellingstatisticalsignificance.

d) Discuss production challenges, including data privacy,modelexplainability,adversarialmachine learning, and computational scalability, and outline a future research agenda toward decentralized, causal inference-based, and quantum-resistantsecuritysystems.

2. Related Works

The application of AI in financial security has evolved throughseveralstages,eachaddressingspecificlimitations of its predecessor. Early systems (pre-2010) employed supervised learning algorithms like Logistic Regression, Decision Trees, and Support Vector Machines (SVMs) to classify transactions as fraudulent based on handengineered features [9]. While effective against known patterns,theirperformancedegradesrapidlywithevolving fraudtacticsandtheyareinherentlyincapableofdetecting novelattackvectors.

Unsupervised and semi-supervised learning methods like Autoencodersand Isolation Forestswere lateradopted to identify anomalous transactions without labeled data [6, 10]. Deep learning architectures, particularly Long ShortTermMemory(LSTM)networksandGatedRecurrentUnits (GRUs), have shown superior performance in capturing temporaldependenciesinusertransactionsequencesand behavioral biometrics [7, 11]. However, these models typicallyanalyze eventsinisolation,ignoring the complex web of interactions between entities that often reveal coordinatedfraud.

Graph-based techniques mark a significant advancement byexplicitlymodelingtheserelationships.Earlyworkused community detection algorithms like the Louvain method onstatictransactiongraphstoidentifysuspiciousclusters [12].TheadventofGraphNeuralNetworks(GNNs),suchas GraphSAGE and Temporal Graph Networks (TGNs), has enabledend-to-endlearningondynamicgraph-structured data, allowing for the detection of subtle, evolving fraud rings that are invisible at the individual transaction level [13,14].

In threat intelligence, NLP techniques have evolved from simple keyword matching and bag-of-words models to sophisticated deep learning approaches. Earlier methods used word embeddings (Word2Vec, GloVe) and Bi-LSTMs to analyze unstructured threat reports [15]. The Transformer architecture, with its self-attention

mechanism, and more recently, Large Language Models (LLMs)fine-tunedforsecuritytasks,offerapotentsolution forcorrelatingdisparatepiecesofinformationacrossvast corpora of security data, including dark web forums and encryptedchannels[8,16].

Our framework synthesizes these disparate threads temporal deep learning, dynamic relational GNNs, and contextualNLP/LLMs intoaunified,synergisticsystem.It addresses the critical limitations of siloed approaches prevalent in both academic literature and commercial solutions by enabling continuous, multi-context risk assessment.

3. PROPOSED HYBRID AI FRAMEWORK

The proposed framework is designed as a real-time, microservices-basedpipelinethatingestsmulti-modaldata from core banking systems, application logs, and external threat feeds. It is built for scalability and resilience, leveraging cloud-native technologies and a publishsubscribepattern(e.g.,ApacheKafka)fordataingestion.

3.1 Real-Time Deep Behavioral Analytics Engine

This component moves beyond single-point transactionanalysistomodelthesequenceandcontextofa user'sdigitalbehavior,creatingapersonalizedbaseline.

Mechanism: We employ a Dual-LSTM Autoencoder architecture with attention mechanisms. One LSTM autoencoder is dedicated to learning the compressed representation (h_txn) of a user's transaction sequence (amount,frequency,merchantcategory,geolocation,timeof-day). A second, parallel LSTM autoencoder learns the representation(h_behavior)oftheirassociatedbehavioral biometrics (keystroke dynamics, mouse movements, session navigation patterns, mobile app interaction gestures) [11]. The reconstruction error from each autoencoder is computed; a sharp increase in error signifiesasignificantdeviationfromtheuser'sestablished behavioral baseline, indicating potential account compromise [7]. The attention mechanism helps identify whichspecificbehavioralfeaturesarecontributingmostto theanomaly.

Technical Implementation: Themodelistrainedperuser in an ongoing manner, creating a dynamic, personalized baseline. Training utilizes a contrastive learning objective to improve the model's discriminative powerbetweengenuineandfraudulentsessions[17].The finalBehavioralAnomalyScore(S_behavior)isaweighted fusion of the transaction and behavioral reconstruction errors:

S_behavior = α · MSE(X_txn, X_hat_txn) + β · MSE(X_beh, X_hat_beh)

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

where α and β are tunable hyperparameters, dynamically adjusted based on feature importance. Inference is performed in real-time using a stream processing framework like Apache Flink to meet the sub-100ms latencyrequirement.

3.2 Dynamic Graph Neural Network for Relational Analysis

This module uncovers sophisticated, collusive fraud ringsbyanalyzingthecomplex,time-evolvingrelationships betweenentities.

Mechanism: We model the banking ecosystem as a dynamic,heterogeneousgraphG=(V,E,T),wherenodesV represententities(Users,Accounts,Devices,IPAddresses, Merchant IDs) and edges E represent interactions (transfers,logins,sharedcredentials)withtimestampsT.A Temporal Graph Network (TGN) is used to learn node embeddings that encode both structural topology and temporal relationship patterns [14]. Fraudulent activity oftenmanifestsassuddenchangesin networktopology suchastheformationofdense,transientsubgraphs(mule networks) or anomalous changes in a node's local neighborhood(e.g.,apreviouslydormantaccountinitiating transactionstomultiplenewly-createdaccounts).

Technical Implementation: We use a modified GraphSAGEmodelwithmemorymodulesandLSTM-based aggregators to incorporate temporal dynamics [13, 14]. The model is trained in a self-supervised manner using a temporal contrastive loss (e.g.,using negative sampling of futurestates),wheretheobjectiveistodistinguishnormal subgraph evolution from anomalous structural changes indicative of fraud. The output is a Relational Risk Score (S_graph) for each entity. The graph is updated continuously in near-real time as new transactions and eventsoccur.

3.3 AI-Powered Threat Intelligence Hub

This component provides external, global context, transformingraw,unstructuredthreatdataintoactionable, structuredintelligence.

Mechanism: We deploy a fine-tuned Transformerbased model, specifically a security-focused variant of RoBERTa or DeBERTa, for Named Entity Recognition (NER), relation extraction, and intent classification from unstructured text sources threat reports, dark web forums, paste sites, and IoC feeds [16, 18]. The model identifies critical entities (e.g., MalwareFamily, CVE, IPAddress, BankName, AttackTechnique) and extracts

semantic relationships between them (e.g., uses, targets, exploits). This structured knowledge is stored and continuouslyupdatedinathreatknowledgegraph.

Technical Implementation: The hub performs continuous correlation between the internal banking data and the external threat graph. For example, if a user's IP address is observed in a transaction, and that same IP is extractedinreal-timefromanewthreatreportdiscussing a botnet operation, the Threat Intelligence Hub immediately elevates the Contextual Threat Score (S_threat) for all active sessions associated with that IP. Integration with STIX/TAXII feeds allows for automated ingestionofstandardizedthreatdata.

3.4. Fusion and Decision Engine

The final risk score is an adaptive, weighted fusion of the three component scores, allowingthe systemto prioritize different signals based on the attack context: S_final=w1·S_behavior+w2·S_graph+w3·S_threat The weights (w1, w2, w3) can be static or dynamically adjustedbyameta-learnerthatconsiderstheconfidenceof each component's prediction and the current threat landscape.Ahybridpolicyengine,combininginterpretable rule-based logic (for high-certainty scenarios) and a lightweightMLclassifier(fornuancedcases),thentriggers graduated actions from allowing the transaction, to requiring step-up authentication (e.g., biometric verification),tooutrightblocking,andfinally,generatinga detailed case for investigators. This case file is enriched with explainable AI (XAI) evidence from SHAP and LIME, providingclearrationalesforthedecision,suchas"flagged due to anomalous login geography combined with connectiontoaknownmuleaccount."

4. CASE STUDY AND EMPIRICAL VALIDATION

A 12-month longitudinal deployment was conducted in partnershipwithamultinationalbankoperatinginover50 countries.

Context: The bank processes an average of 1.2 billion transactions monthly across 50 million customers. The incumbent system was a legacy rules engine (over 5,000 static rules) coupled with a logistic regression model, generating approximately 15,000 daily alerts with a high falsepositiverate.

Intervention: TheproposedAIframeworkwasintegrated intotheirreal-timedatapipeline,requiringasub-100msof end-to-endlatencyfortransactionscoring.Thedeployment involved a phased rollout, starting with a pilot in one geographicregionbeforefull-scaleimplementation.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Results: The following table summarizes the key performanceindicators(KPIs)aftera6-monthstabilization period, comparing the performance against the legacy system.

Table -1: SampleTableformat

Time toDetect (MTTD)New Threat Pattern 16days ~2days 8xFaster Operational CostofAlert Review

bank's secure perimeter, with only aggregated model gradientssharedcentrally[19].

Explainability & Regulatory Compliance: The "black box"natureofdeeplearningmodels,particularlyGNNs,is a significant hurdle for regulators and investigators. We integrated SHAP and LIME to generate post-hoc explanations and developed inherently interpretable surrogatemodelsforthemostcriticaldecisionpaths.This wascrucialforbuildingtrustandmeetingregulatory"right toexplanation"mandates.

Computational Complexity and Latency: The GNN and NLP models are computationally intensive. We addressed thisthroughmodelquantization,knowledgedistillationto create smaller, faster models for edge deployment, optimized graph sampling strategies, and deploying inference on GPU-accelerated hardware. Maintaining sub100ms latency for the entire pipeline was a constant engineeringfocus.

Adversarial Machine Learning: Theframeworkitselfisa target. We observed attempts to poison the behavioral model through slow-drift attacks and evasion attacks against the GNN. Defenses included adversarial training, robust feature engineering, and continuous monitoring of modelperformanceforsignsofdegradation[20].

5.2. Future research directions

Theevolutionofthisframeworkpointstowardseveral promisingresearchavenues:

Analysis: The GNN component was particularly effective, identifying 28 previously unknown money mule rings within the first three months, leading to the freezing of several million dollars in fraudulent assets. The Threat IntelligenceHubprovidedearlywarningforanewphishing campaignspecificallytargetingthe bank'sbrand,allowing for preemptive countermeasures (e.g., blocking malicious URLsatthe gateway)thatblockedover50,000attempted account takeovers. The integration of the XAI dashboard reduced the average case investigation time by 55%, as investigators could immediately understand the primary riskfactorsforeachalert.

5. DISCUSSIONS

5.1. Implementation challenges

Deploying thisframework atscale presentedseveral nontrivialchallenges:

Data Privacy and Regulation: Training models on sensitive customer data, especially behavioral biometrics, require robust anonymization and strict adherence to GDPR, CCPA, and PSD2. We implemented Federated Learningformodelpersonalization,allowinguser-specific modelupdatestoremainontheuser'sdeviceorwithinthe

Causal AI for Root Cause Analysis: Moving beyond correlation to use causal inference and discovery models [21] to determine the precise root cause of an alert and predict potential attack progression, drastically reducing investigator triage time and enabling more precise containment.

Federated Learning for Cross-Institutional Defense: Developing privacy-preserving, cross-silo Federated Learning architectures to enable multiple financial institutions to collaboratively train a global fraud model without sharing confidential data, creating a powerful collectiveimmunesystemagainstfinancialcrime[19].

Generative AI and LLMs for Adaptive Defense: LeveragingLLMsnotjustforthreatintelligencebutalsofor generating synthetic fraud scenarios for robust model training, automating the writing of detection rules in natural language, and powering advanced social engineeringdetectionsystems[8,22].

Quantum-Resistant Cryptography and ML: Preparing for the post-quantum era by exploring and integrating quantum-resistantcryptographicalgorithmstosecurethe data pipelines and models and investigating quantum machinelearning for potentiallyexponential speed-ups in complexgraphanalysis[23].

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

6. CONCLUSION

This paper presented a holistic, hybrid AI framework designed to proactively combat the evolving landscape of modern financial fraud and cyber-threats. By integrating deepbehavioralanalytics,dynamicgraphneuralnetworks, andcontextual threatintelligence into a unified, real-time decision loop, the system demonstrates a substantial and measurable improvement over legacy approaches, as validated by a large-scale, longitudinal industrial deployment.Thetransitionfromsiloed,reactivetoolstoan integrated, intelligent, and explainable platform is not merelyanoptimizationbutastrategicnecessityforbanks to ensure security, maintain regulatory compliance, and preservecustomertrustinthedigitalage.Futurework,as outlined, will focus on enhancing the framework's causality, privacy, resilience to adversarial attacks, and preparationfornext-generationcomputingparadigms.

REFERENCES

[1] IBM Security. "Cost of a Data Breach Report 2023." IBM,2023.

[2] LexisNexis Risk Solutions. "True Cost of Fraud Study: FinancialServicesandLending."2024.

[3]R.Brause,T.Langsdorf,andM.Hepp,"AdvancedCredit CardFraudDetection with NeuralNetworks,"inProc. IEEEInt.Conf.DataMin.(ICDM),2019.

[4] D. West and M. Bhattacharya, "Intelligent Financial Fraud Detection: A Comprehensive Review," Computers&Security,vol.57,pp.47-66,2016.

[5] A. T. Al-Hawawreh, E. Sitnikova, and N. Aboutorab, "LeveragingDeepLearning forFraudDetectioninthe Banking Sector," IEEE Access, vol. 10, pp. 6549665513,2022.

[6]F.T.Liu,K.M.Ting,andZ.-H.Zhou,"IsolationForest,"in Proc.IEEEInt.Conf.Data Min.(ICDM),2008, pp.413422.

[7]P.Malhotra,L.Vig,G.Shroff,andP.Agarwal,"LongShort Term Memory Networks for Anomaly Detection in TimeSeries,"inProc.EuropeanSymp.ArtificialNeural Networks(ESANN),2015.

[8] Z. Li et al., "The Dark Side of the Language Model: An EmpiricalStudyofPrivacyandSecurityRisks,"inProc. ACM SIGSAC Conf. Comput. Commun. Secur. (CCS), 2023.

[9] S. X. Wu, et al., "A Supervised Machine Learning Approach to Credit Card Fraud Detection," in Proc. IEEEInt.Conf.BigData,2020.

[10]C.ZhouandR.C.Paffenroth,"AnomalyDetectionwith RobustDeepAutoencoders,"inProc.ACMSIGKDDInt. Conf.Knowl.Discov.DataMin.,2017.

[11]Y.Sun,S.Wang,andT.Li,"ContinuousAuthentication via Behavioral Biometricsusing DeepLearning,"IEEE Trans. Dependable Secure Comput., vol. 19, no. 2, pp. 1234-1248,2022.

[12] V. D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, "Fast unfolding of communities in large networks," J. Stat. Mech., vol. 2008, no. 10, p. P10008, 2008.

[13] W. Hamilton, Z. Ying, and J. Leskovec, "Inductive Representation Learning on Large Graphs," in Proc. Adv.NeuralInf.Process.Syst.(NeurIPS),2017.

[14]E.Rossi,B.Chamberlain,F.Frasca,D.Eynard,F.Monti, andM.Bronstein,"TemporalGraphNetworksforDeep LearningonDynamicGraphs,"ACMTrans.Intell.Syst. Technol.,vol.14,no.3,pp.1-26,2023.

[15]A.T.Liguori,etal.,"LeveragingTransformerModelsfor Threat Intelligence Extraction from Unstructured Text," in Proc. IEEE Conf. Commun. and Network Security(CNS),2023.

[16]S.Liuetal.,"STIXNet:AStructuredThreatInformation ExpressionNetwork forCyberThreatIntelligence,"in Proc.Annu.Comput.Secur.Appl.Conf.(ACSAC),2023.

[17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A Simple Framework for Contrastive Learning of Visual Representations," in Proc. Int. Conf. Mach. Learn. (ICML),2020.

[18]J.Devlin,M.-W.Chang,K.Lee,andK.Toutanova,"BERT: Pre-training of Deep Bidirectional Transformers for LanguageUnderstanding,"inProc.NAACL-HLT,2019.

[19] P. Kairouz et al., "Advances and Open Problems in FederatedLearning,"Found.TrendsMach.Learn.,vol. 14,no.1–2,pp.1–210,2021.

[20] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, "Towards Deep Learning Models Resistant to Adversarial Attacks," in Proc. Int. Conf. Learn. Represent.(ICLR),2018.

[21] J Pearl, "The Seven Tools of Causal Inference with ReflectionsonMachineLearning,"Commun.ACM,vol. 62,no.3,pp.54–60,2019..

[22] OpenAI. "GPT-4 Technical Report," 2023. [arXiv:2303.08774]

[23]NationalInstituteofStandardsandTechnology(NIST). "Post-QuantumCryptographyStandardization,"2022.