Issuu

Intelligent Cloud Based Log Analyzer for Security Monitoring Using AWS

B. Harish Goud

1, Ambeer

Shravan Kumar2, Bandari Lakshmana Prasad3

1Department of Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India

2Department of Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India

3Department of Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India

Abstract Log data is a critical component of understanding system behavior and identifying security threats in modern cloud computing environments [10]. However, log analysis systems are often expensive, complicated, and difficult to scale. This paper proposes the design of an intelligent log analysis system using Amazon Web Services. The proposed log analysis system is designed using a unified data model that combines rule-based and machine learning-based techniques for identifying suspicious patterns. The proposed system is designed to scale using a serverless architecture. The design of the log analysis system using a serverless architecture is consideredto be beneficial for simplifying log analysis.

Key Words: Cloud Computing, Log Analysis, Cybersecurity, AWS, SIEM, Anomaly Detection

1. INTRODUCTION

Intoday’scloudcomputinganddistributedapplications, which are rapidly evolving, systems are producing enormousamountsoflogdatainreal-time.Logdatais highly valuable, as it can provide valuable information onthebehaviorofthesystem,howusersinteractwithit, andpotentialsecuritythreats[10].Effectiveanalysisof logdataiscriticalinkeepingthesystemingoodhealth andfendingoffpotentialcyberthreats. Traditional log monitoring tools were originally designed for centralized systems and do not map well ontotoday’scloud-nativeenvironments[2].Thesetools oftenrequireheavyhardware,complexsetup,andtheir rule-based approach makes it difficult to identify unknownthreats.

Inthiswork,weproposeaintelligentcloud-basedlog analyzertoolthatcanaddressthechallenges

2. RELATED WORK

Log analysis has improved greatly with with the emergence of cloud computing and machine learning. Nowadays, SIEM systems gather logs from various and detectanomaliesindicativeofpotentialrisks[1].Studies onlogsystemspointouttheneedthatcategorizinglogs intodistinctclassesisimportant.TheWhen,What,Who andWheremodelcanbeusedtoorganizelogdata[2]. This makes it easier and clearer to analyze efficient. Machine learning methods, particularly anomaly algorithms such as Isolation Forest, have been highly effectiveness in detecting unusual behavior [6,7]. In comparison to server-based architectures, processing eventsviacloudservicesisacost-efficientandscalable wayofestablishingorder[4].Itoffersbenefitsinterms ofcostandscalability.

3. LIMITATIONS OF EXISTING SYSTEMS

Withalltechnologicalimprovementsinlogmonitoring, therechallengesstillremain.

The logs are dispersed across different systems and are in different formats. This leads to atomized data whichishardtogetaholisticpicture.

Rule-based systems are not very effective in identifying new patterns in attacks. These systems can becircumventedbyadheringtocertainlimits[9].

Theinfrastructureiscomplex. Intraditional systems, re ource provisioning is an issue because it is costly.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Furthermore, they may not be user-friendly. nontechnicalusers.

4. PROPOSED SYSTEM

The system offers a simplest and scalable system to analyze logs, supporting massive data of data. Our systemdiscoversrisksthatmaydamageoursystems.

Thesystemtakeslogsfromtheapplicationsandsafely storestheminthecloudonS3AWS.Thisconfiguration enables the system to run without continual interventionwhileensuringdataavailability.Thesystem isalsoabletoprocesslogsassoonaslogsassoonasthey areuploaded.[4].

The analysis of real-time log data can be done using AWS Lambda Functions. Logs will be manipulated by extracting helpful information from them and then convertingthemtoadesiredformat.Duetonothaving tohaveaserverrunning,costswillbereduced.

Thesystemsplitslogfilesintofourmajorcomponents to ease the analysis: the date of the event (when the eventoccurred),thetypeofaction(whatwasdone),the user (who did something), andthe locationto perform theevent(wheretheeventwasdone)[2]. Thisformatallowsustoeasilyidentifysuspiciousevents andtofindpatternsofthreats.

The proposed system is very effective in detecting threats. It does this in two ways: by identifying known patterns of attacks and machine learning to detect unusual patterns. For example, if a user logs in many times the system will trigger the alert. The machine learningsystemlooksatpatternsofhumanbehaviorand findsanomalies[6,9].

Once it has checked all the logs it keeps the informationithasfound.Ifitfindssomethingitwillalert us immediately. It also offers a simple way to visualize thedatatoseewhatisgoingonwithoursystems. Our new system is different from other log analysis systems. It’s easy to use and processes a lot of data. Is good at finding threats. So it’s a good way to improve security[5].

4.1 Architecture Overview

The solution is built from scratch with a cloud eventdriven and cloud native system [3,4] to facilitate management of large amounts of log data. Logs from various such as operatingsystems, applications and so on,aregatheredandstoredcentrallywithAWSS3.When thereisaflowofnewloginformation,thesystemuses AWS Lambda functions to process the new real-time. You don’t have to keep servers running to process the information in real-time. The second step is to use an engine to detect information using a detection engine. The detection engine uses a combination of rules and machinelearningtodetectknownthreatsandunknown threats.

The proposed system contains three primary components: cloud storage, serverless processing, and detectionengines.

Fig. 1 shows the overall workflow of the proposed system, illustrating how cloud storage, serverless processing, and detection mechanisms works together aspartofaunifiedpipeline.

4.2 Database Schema

As shown in tables I and II, the system maintains separate schema for processed log data and security alerts. This separation helps in efficient data managementandallowsfasterqueryingforbothregular loganalysisandanomalydetectiontasks.

Table1:ProcessedLogsTableSchema

Attribute Type

Description user_id String(PK) Uniqueuseridentifier timestamp String (Sort Key) Timeofloginevent ip String IPaddressofuser status String Login status (success/failure)

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

Table2:SecurityAlertsTableSchema

Attribute Type Description

user_id String(PK) Identifier of affected user

timestamp String (Sort Key) Time of detected anomaly

ip String SourceIPaddress threat_flag String Type of detected anomaly

1:ArchitectureoftheProposedCloudBased LogAnalyzer

4.3 Unified Data Model

Tomaintainconsistency,thesystemstructureslogdata using the previously defined four-dimensional model, ensuringuniformrepresentationacrossalllogsources.

• When:Timestampoftheevent

• What:Typeofactivityperformed

• Who:Userorsystemresponsible

• Where:SourcelocationorIPaddress

Toformallyrepresentthestructureoflogdata,eachlog entrycanbemodeledas:

where L represents a structured log event, Wn denotes When, Wt denotes What, Wu denotes Who,and Wl denotes Where.Thisstructuremakesthedataeasiertoanalyze andhelpsinidentifyingrelationshipsbetweendifferent eventsmoreeffectively[2].

4.4 Hybrid Detection Mechanism

Thesystemisbasedonacombinationofrule-basedand machinelearningtodetectthreats.Rule-basedmethods aregoodatdetectingcommonpatterns,suchasmultiple failed login attempts [1,9]. However, threats are not always predictable. To handle this, machine learning is applied to user behavior and identifies anomalous behavior [9], therefore, providing the ability to detect potential threats that would not be identified by traditionalrule-baseddetection.Therule-basedmethod ofdetectingthreatsmaybedescribedasfollows:

where Sr representstherule-basedsuspicionflag, Nf is the number of failed login attempts, and T is the predefinedthreshold.

To capture deviations from normal behavior, an anomalyscoreiscomputedas:

Figure

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

where x is the observed feature value, µ is the mean behavior, and σ isthestandarddeviation.

Thefinalhybridsuspicionscoreiscomputedby combiningbothapproaches:

where Sh is the final hybrid suspicion score, Sr is the rule-based score, Sa is the anomaly score, and α is the weighting factor between the two detection mechanisms.

5. METHODOLOGY

Thesystemfollowsastructuredmulti-stageprocessfor log analysis, where each stage handles a specific task fromdatacollectiontoalertgeneration.

5.1 DataCollection

Logsarecollectedfromthewebapplication.Theselogs are continuously sent to cloud storage for further processing.

5.2 Preprocessing

Inthisstage,thecollectedlogsareprocessedtoextract useful information. The data is then cleaned and standardized to ensure consistency. This step is important because properly structured data improves the accuracy of further analysis. Once processed, the dataisreadyforthenextstage.

5.3 Feature Extraction

The processed logs used to extract key attributes, including frequency of logins, pattern of IP addresses, andtimeofactivity[12],areanalyzedusingbothrulesbasedandmachine-baseddetectionmethods.

5.4 Detection Process

Detection is done using two phases: Firstly, using predeterminedregulations,anypotentiallylargethreats willbeassessedanddefinedaseitherpresentorabsent; secondly through Existing Data Analysis Processing/Files." This is where object anomaly detection systems take place creating various combinations of unusual behavior displaying what wouldn’tordinarilyfallwithinnormallimits.[6,7].

5.5 Alert Generation

When suspicious activity is detected, alerts are generatedandstored.Thesealertshelpadministrators in further investigation and enable timely response to potentialthreats.

6. EVALUATION AND DISCUSSION

The system was assessed for its scalability, cost efficiency, and its threat detection capabilities. The serverless design of the system enables it to scale workloads automatically. In addition, the use of cloud services help reduce overall operational costs [4]. The combination of both rule-based and machine learning enhances the system’s capability to identify and unknown threats. But the effectiveness of anomaly detectiononthequalityofthedataandfeaturesselected foranalysis[8,9].Theserverlessarchitectureallowsthe systemtoautomaticallyscaletoaccommodatedifferent loadsintervention.

Table3:ComparisonofLogMonitoringSolutions

Feature Traditional Systems SIEMs Proposed

Architecture Server / Containers Hybrid Serverless

Scalability Manual / ASG Enterprise Automatic

Detection Rule-Based Advanced Rules HybridML

Overhead High Medium Low

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

7. USE CASE SCENARIO

As shown in Fig. 2, the proposed serverless system significantly reduces operational costs compared to traditional SIEM solutions, making it more suitable for scalabledeployments.

As shown in Fig. 3 shows that the hybrid detection approach achieves higher accuracy compared to individual rule-based and machine learning methods, demonstrating the effectiveness of combining both techniques.

To illustrate how the proposed new system operates, let’s assume an attacker is trying to gain unauthorized access by attempting to log in with several different shortperiodoftime.

Mostlogmonitoringtoolsdetectthesetypesofattacks attacks by monitoring failed attempts. However, attackerscanavoiddetectionbythesesystemsbyusing severalusingmultipleusernamesandIPaddresses.

The new system does this by examining the login activitymoreeffectively.Recognizesthingslikemultiple from a single source and classifies them as suspicious. Thisenablesthesystemtoidentifyattacksandthenand generatereal-timealerts.

Meanwhile, the system monitors user behavior in varyingcriteriasuchastimeofday,locationandaccess system, Even if the hacker is trying to be stealthy, the systemcanidentifyunusualbehavior.

For instance, if a user typically logs in from a single thentriestoaccessthesystemfromseverallocationsat once, this is detected as suspicious and an alert is generated.Time,whenthisoccurs,realtimenotifications aresenttoasithappens,soadministratorscanrespond accordingly,suchasblockingtheIPaddressorlaunching aninvestigation.Thismethodenhancestheaccuracyof detection and minimizes false positives by combining knownpatternsandbehavioralanomalies.

Figure2:CostComparisonBetweenTraditionalSIEM andProposedAWSServerlessSystem

Figure3:DetectionAccuracyComparisonofRuleBased,MLBased,andHybridApproaches

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

8. PROJECT OUTCOMES

Table4:ProjectOutcomes

Category Outcome Description

System Architecture Scalable Serverless Pipeline Utilizes AWS S3 and Lambda for event-driven log processing, eliminating dedicated servers and reducing operational overhead.

Data Standardization Unified 4W LogModel Organizeslogs into When, What, Who, and Where categories, ensuring consistency and efficient analysis.

Security Detection High-Fidelity Anomaly Alerts Combines rule-based detectionwith machine learning to identify both knownthreats and anomalous behavior.

Operational Insight Administrative Visibility Enables monitoring of user activity, systemhealth, and access patterns through structured logs.

Incident Response Automated Threat Mitigation Generates real-time alerts for suspicious activities, enabling timely response to security incidents.

Data Persistence Optimized Log Retrieval Stores processed logs in DynamoDB for efficient querying and fastretrieval.

The system has a number of benefits related to scalability,consistencyandsecurity.

It provides us with an economical approach to log analysisandcanbescaledupifweneedtodothat,which isgreatusefulforlookingatlogs.Thesystemalsomakes useofatypeofofdatathatmeanswecangetconsistent resultsandthathelpsiteasiertoworkwiththedata.

Thesystemusesdifferentmethodstodetectproblems, makesthesystemmoresecureandthisisbecauseitcan findthreatsweknowaboutandthreatswedon’tknow aboutandthesystemcandothisveryeffectively. Moreover, the hybrid detection strategy enhances system security by allowing for both known and unknownthreats.Italsoenhancesoperationalvisibility byclassifyinglogdataintocategories

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

9. FUTURE WORK

Thesystemproposedinthispaperlaysthegroundwork for enhanced security monitoring in the cloud. The following paths are outlined for further developing its functionality into a more proactive and autonomous platform.

1. Learning Behavior with Autoencoders and ContinualLearning: Theanomalydetectionengine is currently based on thresholds and rule-based scoring.Futureworkwillincorporatedeeplearning models like Autoencoders to develop a behavioral modelofeachuserandsystem[11].Withtheuseof lifelonglearningtechniques,thesystemwilltomore examplesofnormalactivity[9],andimproveitsto more data over time, evolving from reactive to behaviorintelligence.

2. Proactive Defense through SOAR: The existing system alerts of threats but does not act on those threats. Integrating Security Orchestration, Automation, and Response (SOAR) capabilities [1] will allow the system to perform actions automatically - like quarantining a suspicious IP address, authentication - as soon as suspicious conduct is confirmed. authentication - as soon as suspiciousactivityisdetected,cuttingtheresponse timedownfromminutestomilliseconds.

3. Explainable AI (XAI) for Actionable Alerting: Securityanalystscanbestbenefitfromknowing why it was raised, not just if it was raised. Using explanation systems such as SHAP or LIME will enablethesystemtoprovideexplanationswitheach alert[9,11]-forinstance,listingthatanattemptwas madebecausetheattemptwasunknownlocationat an odd time and attempting trying to access a critical system. This builds analyst confidence and falsepositiveinvestigationtimes.

4. Team Forensics with a Joint Investigation Workspace: Workspace: In security emergency situations, it is not uncommon may need to collaborateonasinglealert.Acommon“WarRoom” area,whereanalystscancommentlogs,investigate incidents,andmakedecisionsinvestigationsinreal time would help speed up times and collaboration [10].

5. Integration with Threat Intelligence Feeds: Linkingthesystemtoworld-widethreatdatabases such as AlienVault OTX or VirusTotal will allow known malicious IP addresses, domains and file hashestobeidentifiedblockedassoonastheycome into contact with the environment [5, 12]. This enables the system to take advantage of security knowledge of the entire community, rather experiences, rather than just those from its own logs.

6. Visualization Dashboards: Incorporating realtimevisualdashboardswillenableadministratorsa real-timepictureofsystemevents,threattimelines. These dashboards, along with the improvements above, will turn the envisioned system into a powerful, "smart" into an holistic, smart security operations system - not only detecting threats but understanding, and automatically reacts to them [1,9].

10. CONCLUSION

Inthispaper,acloud-basedloganalysissystemhasbeen presented that focuses on scalability, efficiency and intelligent threat detection. The proposed approach addresses key limitations of tradi tional log analysis systems by leveraging AWS cloud services and a structureddatamodel.Theproposedsystemcombines a hybrid detection approach with reliability and flexibility. The proposed system demonstrates the potential benefits of cloud technology in developing efficientsecuritymonitoringtools

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

ACKNOWLEDGEMENT

The authors wish to thank the Department of InformationTechnologyatChaitanyaBharathiInstitute of Technology, Hyderabad, for providing the infrastructure and academic environment that made this work possible. Finally, we thank our peers and reviewers whose feedback helped sharpen the ideas presentedinthispaper.

REFERENCES

[1] M. Khayat, E. Barka, M. A. Serhani, F. Sallabi, K. Shuaib,andH.M.Khater,"AdvancedTechniquesfor Alert Management in Security Information and EventManagementSystemsWithEnsembledDeep Learning,HybridOptimization,andMulti-Feature Extraction," IEEE Open Journal of the Communications Society, vol. 6, pp. 7349–7368, 2025, doi: 10.1109/OJCOMS.2025.3603000, https://ieeexplore. ieee.org/document/11142305.

[2] A. Oliner, A. Ganapathi, and W. Xu, "Designing a Unified Cloud Log Analytics Platform," in Proc. IEEE/USENIX HotCloud,2012.[Online].Available: https://ieeexplore.ieee.org/document/7870995

[3] J.RobertsandC.Chapin,"Event-DrivenServerless Architectures on Cloud Platforms," in Proc. IEEE Int. Conf. Cloud Eng. (IC2E),2018,pp.1–8.

[4] R. Poorvadevi, Surendar H and SriRamakrishnan S,"Serverless Data Processing Using AWS," in 2025 8th International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, 2025, doi:10.1109/ICOEI65986.2025.11013090.

[5] A.B.Nassif,M.A.Talib,Q.Nasir,H.Albadani,andF. M.Dakalbab,"Machinelearningforcloudsecurity: A systematic review," IEEE Access, vol. 9, pp. 20717–20735, 2021, doi: 10.1109/ACCESS.2021.3054129.

[6] F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation Forest,” in Proc. IEEE Int. Conf. Data Mining (ICDM),2008,pp.413–422.

[7] F. T. Liu, K. M. Ting, and Z.-H. Zhou, “IsolationBased Anomaly Detection,” ACM Trans. Knowl. DiscoveryData,vol.6,no.1,2012.

[8] C. C. Aggarwal, Outlier Analysis, 2nd ed., Springer, 2017.

[9] V. Chandola, A. Banerjee, and V. Kumar, “Anomaly Detection:ASurvey,”ACMComputingSurveys,vol. 41,no.3,2009.

[10] K. Kent and M. Souppaya, Guide to Computer SecurityLogManagement,NISTSpecialPublication 800-92, National Institute of Standards and Technology,Gaithersburg,MD,USA,2006.

[11] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning Cambridge,MA,USA:MITPress,2016.

[12] W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan,"Detecting Large-Scale System Problems by MiningConsoleLogs,"in Proc. ACM Symp. Oper. Syst. Princ. (SOSP),2009,pp.117–132.