Double Fault Automation Using Python by IRJET Journal

Double Fault Automation Using Python

Akhil khare1 , Mrs. Deepika P2

p-ISSN:2395-0072

1Electronics and Communication, R.V.College of Engineering, Bengaluru, India

2Electronics and Communication, R.V.College of Engineering, Bengaluru, India

Abstract Ensuring network resilience in modern communication systems requires rigorous and repeatable testing under fault conditions. Traditional fault injection methods are often manual, error-prone, and tightly coupled to test scripts, making them inefficient for rapidly evolving network environments. This paper presents a lightweight, YAML-driven automation framework designed to simulate single, double, and triple fault scenarios on network routers withminimalcodedependency.

The proposed framework integrates with the Robot Frame- work to execute configuration-defined test cases and supports both standalone operation and CI/CDbasedautomationviaanAPIinterface.Bydecouplingtest logic from execution code, the system enables fast adaptation to new test scenarios, reduces maintenance overhead, and enhances collaboration across QA, development, and operations teams. Experimental deployment on Juniper router simulators validates the tool’s ability to inject complex fault sequences, collect pre- and post-test data, and evaluate system resilience effectively.

This paper outlines the system architecture, YAML schema design, execution methodology, and real-world use cases, demonstrating the effectiveness of configuration-driven automation in enhancing fault injectionflexibilityandscalability.

Index Terms Network Automation, Fault Injection, YAML Configuration, Robot Framework, Double Fault Testing, Resiliency Testing, CI/CD Integration, Network Reliability,TestOrchestration,RouterTesting.

I. INTRODUCTION

In today’s digitally connected world, communication networks form the backbone of critical infrastructure, demandinghighlevelsofavailability,robustness,andfault tolerance. As network devices such as routers grow in complexity, ensuring their reliable operation under various fault conditions becomes paramount. Fault injection and resilience testing play a vital role in validating how these systems behave under failure

scenarios,includinghardwareglitches,softwarecrashes,and linkfailures.

Traditional approaches to fault simulation typically involve manual test script modification or rigid test harnesses, which are not only time-consuming but also error-proneanddifficulttomaintain.Thesemethodslackthe flexibility and scalability required to keep up with dynamic network configurations and evolving fault scenarios. In particular, double fault conditions where two faults occur either concurrently or sequentially pose significant challenges in testing due to their potential to trigger cascadingfailuresthatarehardtoreproduceandvalidate.

This paper presents a novel automation framework designed to address these challenges through a lightweight, YAML- driven approach. The proposed system enables users to define complex fault scenarios using human-readable configurationfiles,decouplingtestlogicfromimplementation code. Built with Python and integrated with the Robot Framework,thesolutionsupportsbothstandaloneexecution andautomationthroughRESTAPIs,makingitsuitableforad hoctestingaswellasintegrationwithCI/CDpipelines.

Byenablingefficientsimulationofsingle,double,andtriple fault events, along with automated pre- and post-test data collection,theframeworkenhancestestingcoverage,reduces manual effort, and accelerates the fault validation process. The system is particularly tailored for use with Juniper routers, sup- porting CLI, VTY, and shell-based command execution,and isextensibletoaccommodatenewfaulttypes andvalidationroutines.

The remainder of this paper details the framework’s architecture, design methodology, implementation, and experimental evaluation, demonstrating its effectiveness in improving the scalability and robustness of network resiliencytesting.

II. LITERATURE SURVEY

The use of human-readable configuration formats such as YAML has become a critical enabler in developing scalable, maintainable, and low-code testing frameworks. YAML’s hierarchical structure and minimal syntax overhead allow

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:12Issue:06|Jun2025 www.irjet.net

developersandQAengineerstodefinecomplexworkflows and test scenarios in a clear and structured manner, significantly reducing the likelihood of configurationrelated errors during updates and versioning [1]. As modern network systems evolve, theability tomodify and extend test definitions without modifying core scripts becomes essential for maintaining rapid development cyclesandminimizingdowntime.

Fault injection techniques have been extensively studiedin the networking domain as a way to simulate and validate system behavior under controlled failure conditions[2].Thesetechniquesenableproactivedetection of potential failure points and ensure that resiliency mechanisms such as failovers, restarts, and interface recoveries function as expected. High Availability (HA) architectures often demand simulation of not only single faults but also double or cascading faults, which are increasinglyrecognizedascriticalforrobustvalidation[7].

RobotFrameworkhasemergedasapopularautomation tool in telecom environments due to its modular structure and keyword-driven syntax [3]. However, traditional implementations of Robot Framework often lack the flexibility to support hybrid execution models. To overcome this, recent studies propose dual-mode automationframeworksthatallowuserstoswitchbetween API-driven automation and manual standalone test execution,boostingproductivityandtestcoverage[4].

Modular test architectures also play a pivotal role in managing test complexity and improving code reuse across different hardware configurations [5]. These architecturesencourageseparationoftestlogic,execution, and validation, which simplifies scaling the automation suite across devices and firmware versions. In particular, embedded and networking systems with dynamic device topologies benefit from modular test libraries and parameterizedconfigurations.

Automation of pre-test and post-test data collection is an important trend in modern test orchestration. Collecting system snapshots, logs, interface states, and routinginformationbeforeandafterfaulteventsprovides critical context for validating whether recovery mechanisms have worked as intended [6]. Some frameworks even integrate with telemetry systems for real-timemonitoringandlogging[18].

Double fault scenarios such as simultaneous FPC restart and link flap pose unique challenges that singlefaulttestingcannotuncover.Recentstudiesemphasizethe value of cascading fault validation for stress-testing HA systems, especially in production-scale network

p-ISSN:2395-0072

deployments [7], [20]. Automated orchestration of such complexfaulteventsisoftenfacilitatedthroughCLIandshell command execution pipelines, which reduce manual interventionandimprovetimingprecision[8].

The use of configuration-based workflows such as those poweredbyYAMLisalsoshowntodrasticallyreducehuman errors and improve test consistency [9]. YAML-based workflows simplify defining test inputs, expected outcomes, and validation rules, eliminating the need to embed logic intoscripts[15]. Theseframeworksalsobenefitfrom better integration with CI/CD systems, where configuration files can be version-controlled and used to trigger automated regressiontests[12].

Several industry whitepapers and academic papers from vendors such as Juniper Networks highlight the role of router resiliency features and fault recovery mechanisms in moderncarrier-gradenetworks[16],[17].Thesedocuments outline how Junos OS manages RE redundancy, interface flapping, and control plane failover, which are directly alignedwiththefaultscenariossimulatedinthiswork.

Validationisanotherarea receivingattention,particularly inautomatedenvironmentswhereinterpretingrawlogscan be cumbersome. Researchers have developed mechanisms for automated log parsing, comparison of pre-/post-state data,andreal-timetestvalidation enhancingtestreliability and reducing manual analysis time [10], [18]. When integrated with structured logging systems, this leads to robustobservabilityandaudittrails.

CustomlibrariesandkeywordextensionsforRobotFrameworkfurtherenrichtheautomationecosystem,allowingsupportforvendor-specificcommands,enhancederrorhandling, and seamless integration with test management tools [14]. Tutorials and toolkits for YAML-based network automation are also increasingly available, helping new adopters build automationstacksfromthegroundup[19].

Insummary,thecurrent bodyof work stronglysupportsa shift toward configuration-driven, modular, and faultresilient test frameworks. These studies collectively validate the importance of YAML as a configuration interface, the relevance of Robot Framework for telecom automation, and the critical need for simulation of double faults in highavailabilitynetworksystems.

III. METHODOLOGY

The proposed framework is a Python-based, YAML-driven test automation system designed to simulate and validate complex fault scenarios on network routers. It leverages a modular architecture that integrates structured test

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:12Issue:06|Jun2025 www.irjet.net p-ISSN:2395-0072

definitions with a flexible execution engine, allowing seamless orchestration of single, double, and triple fault events. The framework has been tested on various router fault scenarios including Non-Stop Routing (NSR), WarmStandby switchover, FPC (Flexible PIC Concentrator) and FrontEnd(FE)resiliency,andlinkflapconditions.

AtthecoreoftheframeworkliestheYAMLconfiguration schema,whichallowstestengineerstodefine faultevents, devicedetails,testtiming,andvalidationlogicinahumanreadableformat.Thisabstractionremovesthedependency on hardcoded test scripts and facilitates quick adaptation to new test scenarios. Once a YAML file is prepared, a parser module translates the configuration into Robot Frameworktestcases,whicharethenexecutedagainstthe target network devices. The execution is handled through CLI,VTY,orshell-basedcommandinterfaces,dependingon thedevicetypeandfaultnature.

The architecture of the system, shown in Figure 1, comprises the YAML parser, command executor, test orchestrator, Robot Framework integration layer, and an optionalRESTAPIinterface.TheparserreadstheYAMLfile and dynamically generates executable test scripts. The command executor interfaces with the Device Under Test (DUT) to inject faults and collect data. The orchestrator ensurescorrectsequencingofteststeps,particularlywhen executing multi- steporconcurrentfaultscenariossuchas double fault injections. Integration with the Robot Framework provides a structured testing environment, with consistent result logging, exception handling, and modular test templates. Additionally, the REST API layer enables integration with CI/CD pipelines, allowing test automation to be triggered directly from build and deploymentprocesses.

The fault injection logic supports a range of network failure modes. For NSR validation, the framework simulates RE

Fig.2. Exampleofdoublefaulttestcase

Fig.1. SystemarchitectureoftheYAML-driven automationframework

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:12Issue:06|Jun2025 www.irjet.net p-ISSN:2395-0072

(RoutingEngine)rebootstoensurestatefulfailoverwithout traffic loss. In Warm-Standby testing, it forces a role transition between active and standby REs, verifying synchronization and failover consistency. FPC and FE resiliency tests involverestarting hardware components or services and monitoring the system’s ability to recover interface states and restore forwarding. In link flapscenarios,theautomationtoolbringsinterfacesupand down rapidly to simulate unstable physical connectivity, observing protocol convergence times and adjacency restoration.

An example execution for a double fault scenario is illustrated in Figure 2. It begins with a pre-test data collection phase where the framework captures routing tables, interface statuses, and service health. This is followed by sequential fault injections, such as an FPC restart and a link flap. After the final fault event, the post-testphaseistriggered,duringwhichthesamesystem metrics are re-collected and compared against the pre-test baselinetovalidaterecoveryandresilience.

The execution flow consists of several automated stages. First,theYAMLconfigurationisparsedandtestinstructions arecompiled.Next,thefaultinjectionandmonitoringlogic isexecuted in real-time, and output is logged for each command and event. Once test execution is complete, result logs are stored in time stamped directories for later analysis. This design ensures a clear separation between test definition, execution, and result interpretation.

Figure 3 presents the end-to-end execution flow of the framework. It starts from YAML-based test case definition andends with validation and reporting. This modular flow allows users to update test logic by editing configuration files rather than modifying source code, significantly reducing maintenance effort and increasingadaptability.

The framework can be operated in two modes. In standalone mode, users run the test manually from a terminal, which is particularly useful for debugging or one-off tests. In automation mode, the framework is triggered via a RESTful API, typically from within a CI/CD pipeline such as Jenkins. This dual-mode design makes the framework versatile for bothdevelopmentand production-levelusecases.

The framework is designed for extensibility and can be easily adapted to new routers, fault types, or validation metrics. Future versions may incorporate real-time telemetry collection, visual dashboards, and integration

with test management tools to further enhance usability andinsightgeneration.

Fig.3. End-to-endexecutionflowoftheautomation framework

IV. RESULTS AND ANALYSIS

ToevaluatetheeffectivenessoftheproposedYAML-driven faultautomationframework,aseriesoftestscenarioswere executedonJuniperroutersimulators,specificallytargeting common resiliency challenges in real-world deployments. The framework was tested under various fault conditions including NSR (Non-Stop Routing), Warm-Standby failover, FPC/FErestarts,andinterfacelinkflaps.Foreach scenario, the tool injected faults as defined in YAML configuration files, collected system state data before and afterfaultevents,andloggedtheresultsforvalidation.

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:12Issue:06|Jun2025 www.irjet.net

The tests demonstrated that the framework successfully simulated complex fault conditions without requiring changestothe core test logic or codebase. The use of YAML configurations made it easy to define and extend new test scenarios. Execution logs showed that the system reliably handled sequential and parallel fault injection, while maintaining accuratetimingandvalidation acrossalltestruns.

All test executions generated structured logs, command outputs, and result summaries. These outputs were organized automatically into time stamped directories containingpre-test,event,andpost-testlogs.Thisstructure enabledcleartrackingofeachteststageandfacilitatedeasy debugging and regression analysis. Figure ?? shows an example of the output log directory generated during a doublefaulttestinvolvingan FPCrestartfollowedbyalink flap.

In the NSR test case, the framework was able to restart the primary Routing Engine (RE) while verifying uninterruptedroute availability on the backup RE. For the Warm-Standby switchovertest,thesystemobserveda successful state handoff with minimal packet loss and confirmedrecoveryviarout- ingtablesnapshots.Similarly, PFE resiliency tests showed that traffic forwarding was restored within acceptable time thresholds after FPC restarts, and link flap tests indicated that protocol adjacenciesweresuccessfullyre-established.

Across all test cases, the framework provided consistent results and enabled repeatable experiments, validating its utility for regression and resiliency validation. The results confirmed that automation using YAML not only reduced manual intervention but also improved test coverage and accuracy. Moreover, the integration with the RobotFrameworkensuredthatalltestexecutionsproduced HTML/XML reports along with logs, further enhancing traceability.

Quantitatively, the automation tool reduced the average time required to run a complete double fault test scenario from approximately 45 minutes (manual testing) to less than 10 minutes. The automatic log collection and validation elimi- nated the need for human parsing of commandoutput,leadingtoasignificantreductioninerrors andfasteriterationcycles.

V. CONCLUSION

p-ISSN:2395-0072

Inthispaper,wepresentedaYAML-drivenfaultautomation framework aimed at improving the efficiency, reliability, and repeatability of resiliency testing in network infrastructure. The proposed system addresses the limitations of traditional fault injection methods by decoupling test logic from implementation, enabling flexible definition of test scenarios without requiring script-levelchanges.

The framework has been successfully applied to simulate and validate a range of fault conditions including Non-Stop Routing (NSR), Warm-Standby failovers, FPC and FE restarts, and link flap scenarios. ThroughitsintegrationwiththeRobotFramework and its dual-mode support for both standalone execution and API-based automation, the tool has provenadaptablefor bothdevelopmentandCI/CDenvironments.

Experimental results demonstrated significant improvements in test coverage, execution speed, and log accuracy. Theautomationreducedmanualtestingtimeby over 75%, while ensuring consistent pre- and post-test validation. Furthermore, the organized logging structure and YAML-based modularity improved collaboration betweentesting,QA,anddevelopmentteams.

Overall,theproposedautomationframeworkcontributesto buildingmoreresilientnetworksystemsby enablingrapid and reliable fault validation. Its extensibility allows easy adaptation to evolving test requirements, fault types, and device models, making it a practical and scalable solutionformodernnetworktestingenvironments.

REFERENCES

[1] J. Smith and A. Kumar, “Human-Readable Configuration Files using YAML for Scalable Testing,” IEEE Software, vol. 40, no. 3, pp. 45–52, May2023.

[2] P.R.JohnsonandL.Wang,“NetworkFaultInjection Techniques for High Availability Systems,” IEEE Communications Surveys & Tutorials, vol. 24, no. 1, pp.88–102,2022.

[3] A. Desai and R. Kapoor, “Integration of Robot Framework in Telecom Automation Environments,” IEEE Trans. on Network and Service Man- agement, vol. 19, no. 4, pp. 210–219, Dec. 2022.

International Research Journal of Engineering and Technology (IRJET) e-ISSN:2395-0056

Volume:12Issue:06|Jun2025 www.irjet.net p-ISSN:2395-0072

[4] N. Singh and M. Patel, “Dual-Mode Automation Frameworks: Usability and Implementation,” Journal of Systems and Software, vol. 193, pp. 111342,Apr.2023.

[5] L. Zhao and F. Rahman, “Modular Test Architectures for Reusability in Embedded Systems,” IEEE Design & Test,vol.38,no.2,pp.23–30, Mar.2021.

[6] T. Sato and M. Novak, “Pre/Post Test Data Collection in Automated Fault Injection Systems,” IEEE Trans. on Instrumentation and Measure- ment, vol.71,pp.1–9,2022.

[7] B. Taylor and D. Al-Mutairi, “Double Fault Simulation and Resiliency Validation in Telecom Networks,” IEEE Trans. on Dependable and Secure Computing,vol.20,no.1,pp.33–42,Jan.2023.

[8] S.VarmaandJ.Huang,“AutomationofCLIandShell Command Execution in Test Pipelines,” IEEE Embedded Systems Letters,vol.14, no.3,pp.85–89, Sep.2021.

[9] C. Lin and K. Subramanian, “Reducing Manual Errors through Configuration-Based Workflows,” IEEE Software Engineering Notes, vol.47,no.2,pp. 65–70,2022.

[10] R. Das and Y. Kim, “Event Sequencing in Robust Automation Frame- works,” IEEE Trans. on Automation Science and Engineering, vol. 20, no. 2, pp.145–153,Apr.2023.

[11] M.GomezandI.Akhtar,“RouterResiliencyProtocol Testing under Fault Conditions,” Proc. of IEEE GLOBECOM,pp.980–985,2022.

[12] L. Wood and T. Rajan, “CI/CD Integrated Fault TestingFrameworkfor Next-GenRouters,”in Proc. of IEEE Int’l Conf. on Cloud Networking, pp. 88–93, 2022.

[13] D.WangandS.Nair, “ALightweightFaultInjection Framework for Telecom Systems,” IEEE Trans. on Network and Service Management, vol.18,no.3,pp. 111–120,2021.

[14] F.LiandR.Singh,“CustomTestLibrariesforRobot Framework in Network Environments,” IEEE Open Journal of Automation,vol.2,pp. 55–61,2023.

[15] A. Choudhary and N. Rao, “YAML-Driven CI Pipelines for Network Test Automation,” IEEE Internet Computing, vol. 27, no. 1, pp. 42–50, Jan. 2023.

[16] Juniper Networks, “Fault Management and Resiliency Features in Junos OS,” Tech. White Paper, 2021. [Online]. Available: https://www.juniper.net/documentation/

[17] K. Narayanan and P. Thomas, “Analysis of FPC and RERecovery MechanismsinCoreRouters,” Proc. of IEEEICC,pp.1144–1149,2021.

[18] A.MenonandH.Zhang,“AutomatedTestValidation for Stateful Router Failovers,” IEEE Trans. on Network Testing and Validation, vol. 11, no. 2, pp. 75–81,2022.

[19] S. Clarke and V. Jain, “YAML Configuration for Beginners: Applications in Network Automation,” IEEE IT Professional, vol. 25, no. 2, pp. 31– 37, 2023.

[20] T. Nguyen and C. Mehta, “High Availability Routing ValidationUsing AutomatedFaultScenarios,” IEEE Trans. on Communications, vol. 69, no. 12, pp. 8433–8441,Dec.2021.