Issuu

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

AUTOMATED ANOMALY DETECTION IN AWS PRIVATELINK AND VPC

ENDPOINT TRAFFIC

USING

MACHINE LEARNING FOR CLOUD NETWORK SECURITY

1Kiran Dashrath Sahani

Intelligent Cloud Connectivity Monitoring and Threat Mitigation Framework

1Vidyavardhini's College of Engineering and Technology, Mumbai, India

Abstract: This paper presents an intelligent anomaly detection framework for AWS PrivateLink and VPC Endpoint traffic using machine learning algorithms. As enterprises increasingly adopt private connectivity models to isolate workloads from the public internet, the attack surface within private network paths remains largely unmonitored. This study proposes a supervised and unsupervised hybrid ML pipeline that ingests VPC Flow Logs, AWS CloudTrail events, and DNS query logs to detect anomalous patterns in PrivateLink endpoint traffic. The model achieves a detection accuracy of 94.3% with a false positive rate of 2.1%, validated on a synthetic enterprise-grade dataset simulating real-world PrivateLink deployments across AWS, Azure Private Link, and GCP Private Service Connect environments.

Index Terms AWS PrivateLink, VPC Endpoint, Anomaly Detection, Machine Learning, Cloud Network Security, VPC Flow Logs, CloudTrail, Zero Trust, Isolation Forest, LSTM.

I. INTRODUCTION

Cloud-native architectures increasingly rely on private connectivity mechanisms to enable secure, low-latency communication between services without traversing the public internet. AWS PrivateLink and VPC Endpoints represent foundational constructs for this paradigm, allowing consumers to access services hosted in different AWS accounts or regionsthroughprivateIPaddresseswithintheirownVPC.

Traditionalnetworksecuritytoolsareprimarilydesignedforperimeter-basedthreatdetectionandareill-equippedto analyze traffic patterns within private endpoint channels. Anomalous behaviors such as unusual cross-account assumerole activity, DNS resolution failures within Private Hosted Zones (PHZs), abnormal connection pool behavior, and AZspecificendpointcoveragegapsposesignificantrisksthatevadeconventionalmonitoringsolutions.

This paper addresses these challenges by proposing a Machine Learning-based Anomaly Detection and Automation (ML-ADA) framework for PrivateLink and VPC Endpoint environments. The framework combines Isolation Forest for unsupervised outlier detection with an LSTM-based sequence model for time-series behavioral profiling, integrated with AWS-nativetelemetrysourcesandautomatedremediationworkflows.

II. RELATED WORK

Prior research on cloud network security has focused predominantly on public-facing traffic analysis. Chen et al. [1] demonstratedflow-basedanomalydetectioninpubliccloudenvironmentsbutlackedcoverageforprivateendpointtraffic. Chandola etal.[3]provided a comprehensivetaxonomy ofanomalydetectiontechniques.IsolationForest,introducedby Liuetal.[4],hasdemonstratedeffectivenessforhigh-dimensionalnetworktelemetry.LSTM-basedmodelsfortime-series anomalydetectionwereexploredbyBontempsetal.[5],showingsuperiorperformanceoverARIMAmodelsfordetecting low-and-slow attack patterns. To the best of our knowledge, no prior work has proposed an end-to-end ML pipeline specificallytargetingVPCEndpointandPrivateLinktrafficanomalydetectionwithautomatedremediation.

III. SYSTEM ARCHITECTURE AND DATA PIPELINE

3.1 Architecture Overview

The proposed ML-ADA framework follows a three-tier architecture: (1) Telemetry Ingestion Layer, (2) Feature Engineering and ML Inference Layer, and (3) Automated Response Layer. Figure 1 illustrates the end-to-end system architecture.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

1. ML-ADA Three-Tier System Architecture for PrivateLink Anomaly Detection

3.2 Telemetry Sources and Feature Extraction

VPC Flow Logs provide the primary network-layer signal, capturing source/destination IP, port, protocol, bytes transferred, and connection acceptance status. Flow records are aggregated into five-minute windows with statistical featuresincludingmeanbytesperflow,connectionrate,rejectedflowratio,andAZdistributionentropy.CloudTrailevents are joined on a temporal key to capture IAM principal, source IP, and API action for endpoint-related activity. Route 53 DNSquerylogscapturePHZresolutionpatterns,enablingdetectionofNXDOMAINspikesandresolvermismatchevents.

3.3 Feature Vector Composition

The final feature vector per observation window contains 42 features: network flow statistics (14), IAM/identity signals (12), DNS resolution metrics (8), and endpoint configuration state deltas (8). Categorical features such as AWS regionandservicenameareencodedusingtargetencoding.

IV. MACHINE LEARNING MODEL DESIGN

4.1 Hybrid Detection Pipeline

Theframeworkemploysatwo-stagedetectionpipeline.Stage1appliesIsolationForestoverthefull42-featurevector. Stage 2 applies a stacked LSTM network to sequences of 20 consecutive observation windows, capturing temporal patterns.Finalclassificationcombinesbothstageoutputsthroughalogisticmeta-learner.Figure2showstheMLpipeline flow.

2. Hybrid ML Detection Pipeline: Two-Stage Anomaly Scoring with Logistic Meta-Learner Fusion

4.2 Anomaly Taxonomy

Themodeldetectsfivecategories:(A1)DNSResolutionFailureSpike NXDOMAINresponseincreaseindicatingPHZ misconfiguration;(A2)Cross-AccountPrincipal Injection unexpectedIAMprincipal accessinga VPCendpoint;(A3)AZ CoverageImbalance disproportionatetrafficonsubsetofendpointENIs;(A4)IdleConnectionPoolAbuse persistent

Fig.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

low-byte, high-frequency flows; and (A5) Security Group Egress Bypass accepted traffic on unexpected destination ports.Figure3illustratestheanomalytaxonomyandseveritylevels.

Fig. 3. Anomaly Taxonomy: Five PrivateLink-Specific Anomaly Categories with Severity Classification

4.3 Automated Remediation

DetectedanomaliestriggerLambda-basedremediationviaAmazonEventBridge.LowseveritygeneratesSecurityHub findings. Medium severity triggers AWS Config rule validation and SNS notification. High severity invokes Lambda to quarantinetheaffectedendpointbymodifyingitssecuritygroup,pendingSlack-basedanalystapproval.

V. EXPERIMENTAL EVALUATION

5.1 Dataset

A synthetic dataset was generated across a multi-account AWS environment of 12 accounts, 8 regions, and 340 VPC endpoints. The dataset contains 180,000 five-minute observation windows over 30 days, with 9,200 labeled anomaly instances(~5.1%prevalence)acrossfiveanomalycategories.

5.2 Performance Results

Table 5.1: Model Performance Comparison Across Anomaly Detection Approaches

Table 5.2: Per-Category Detection Performance of the Proposed ML-ADA Framework

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 03 | Mar 2026 www.irjet.net p-ISSN: 2395-0072

VI. RESULTS AND DISCUSSION

TheproposedML-ADAframeworkdemonstratesconsistentlysuperiorperformanceacrossallevaluationmetrics.The hybrid architecture's advantage lies in the complementary nature of its two stages: Isolation Forest surfaces volumetric multivariateoutliersatper-windowlevel,whileLSTMcapturestemporaldeviationsacrosssustainedsequences.TheFalse Positive Rate of 2.1% is a meaningful improvement over static threshold approaches (18.0%). Automated remediation reduced mean time to containment from 47 minutes to 3.2 minutes a 93.2% reduction. Cross-cloud generalization showedAUC-ROCof0.94forAzureand0.92forGCPdeployments.

VII. CONCLUSION

This paper presented ML-ADA, a hybrid machine learning framework for automated anomaly detection in AWS PrivateLink and VPC Endpoint environments. By combining Isolation Forest with LSTM-based temporal modeling over enrichedtelemetry,theframeworkachieves94.3%F1-scorewitha2.1%falsepositiverateacrossfiveanomalycategories. The93.2%reductioninmeantimetocontainmentanddemonstratedcross-cloudgeneralizabilitypositionthisworkasa viable production-grade security solution for enterprise multi-cloud architectures. Future work will explore federated learning,application-layermetadataintegration,andreinforcementlearning-basedadaptiveremediation.

ACKNOWLEDGMENT

The authors acknowledge the support of the cloud infrastructure and security teams who provided operational context andvalidatedanomalytaxonomiesbasedonreal-worldenterprisePrivateLinkdeploymentexperience.

REFERENCES

[1] Chen, Y., Paxson, V., and Katz, R. H., "What's New About Cloud Computing Security," UC Berkeley Technical Report, EECS-2010-5,2010.

[2] Varghese, B. and Buyya, R., "Next Generation Cloud Computing: New Trends and Research Challenges," Future GenerationComputerSystems,vol.79,pp.849–861,2018.

[3]Chandola,V.,Banerjee,A.,andKumar,V.,"AnomalyDetection:ASurvey,"ACMComputingSurveys,vol.41,no.3,pp.1–58,2009.

[4]Liu,F.T.,Ting,K.M.,andZhou,Z.H.,"IsolationForest,"inProc.IEEEICDM,pp.413–422,2008.

[5]Bontemps,L.,McDermott,J.,Le-Khac,N.A.,andBhargava,N.,"CollectiveAnomalyDetectionBasedonLSTMRecurrent NeuralNetworks,"Proc.FDSE,Springer,pp.141–152,2016.

[6] Patel, M., Rathod, J., and Shah, D., "Cloud Intrusion Detection System Using ML on AWS CloudTrail Logs," Proc. IEEE ICCCS,pp.1–6,2021.

Fig. 4. Comparative Performance of Anomaly Detection Models Precision, Recall, and F1-Score

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

[7]AmazonWebServices,"AWSPrivateLinkConceptsandUseCases,"AWSDocumentation,2024.

[8]Sommer,R.andPaxson,V.,"OutsidetheClosedWorld:OnUsingMLforNetworkIntrusionDetection,"Proc.IEEES&P, pp.305–316,2010.

[9] Mirsky, Y., Doitshman, T., Elovici, Y., and Shabtai, A., "Kitsune: An Ensemble of Autoencoders for Online Network IntrusionDetection,"Proc.NDSS,2018.

[10]GoogleCloud,"PrivateServiceConnectOverview,"GCPDocumentation,2024.