Agentic Multi-Stage Retrieval-Augmented Generation System with Hybrid Retrieval and Self-Evaluation

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

e-ISSN:2395-0056

p-ISSN:2395-0072

Agentic Multi-Stage Retrieval-Augmented Generation System with Hybrid Retrieval and Self-Evaluation

Mayank Patoliya

Student,DeptofArtificialIntelligenceEngineering,ShriShankaracharyaInstituteOfProfessionalManagement& Technology,Chhattisgarh,India

Abstract- Retrival-Augmented Generation(RAG) improves the factual accuracy of Large Language Models by incorporating external knowledge during inference. However, traditional RAG systems suffer from limitations such as poor retrieval quality, redundancy, and hallucinated responses[14].ThispaperproposesaHybridRAGframework that combines sparse retrieval (BM25) and dense retrieval (FAISS embeddings) using Reciprocal Rank Fusion (RRF)[2], followed by Cross-Encoder reranking for enhanced semantic relevance. Maximal Marginal Relevance (MMR) is applied to ensure diversity among retrieved documents. Additionally, a confidence scoring mechanism[22] and failure detection module are introduced to improve reliability and identify insufficient information scenarios. Experimental evaluation across multiple query types[2] demonstrates that the proposed system achieves improved retrieval performance (MRR = 0.8056) while maintaining balanced generation quality. The results highlight trade-offs between accuracy, latency, and reasoning complexity, contributing toward the development of reliable and scalable RAG systems for real-worldapplications[1].

Key Words: Retrieval-Augmented Generation (RAG), Agentic AI, Multi-Hop Reasoning, Hybrid Retrieval, Large Language Modes (LLMs), Cross-Encoder Re-ranking

1.INTRODUCTION

Large Language Models (LLMs) have significantly advancednaturallanguageprocessingtasks[17]such as question answering, summarization, and conversationalAI[10].Despitetheircapabilities,these models rely on static training data[8] and often generate hallucinatedresponses[10]whenconfronted with unfamiliar queries[2]. Retrieval-Augmented Generation (RAG) addresses this limitation[1] by integrating external knowledge retrieval mechanisms[3] into the generation pipeline[4], allowingmodelstogroundtheirresponsesinretrieved evidence.

However, conventional RAG systems face several challenges.SparseretrievalmethodssuchasBM25rely on lexical matching and fail to capture semantic similarity[15], while dense retrieval methods rely on embeddingsandmayoverlookexactkeywordmatches. Additionally, traditional pipelines lack reranking[15], diversity optimization[6], and mechanisms for evaluating response[1].

To address these issues, this work proposes a Hybrid RAG system[1] that integrates sparse and dense retrieval using Reciprocal Rank Fusion (RRF). Cross-Encoder reranking is used to refine relevance[17], while Maximum Marginal Relevance (MMR) ensures diversity. A confidence scoring mechanism and failure detectionmodule[23]are introduced toenhancereliability[7].

2. Related Work

The idea of using information to help Large Language Models is not new[1]. For a time, people have been working on ways to combine the knowledge that a model haswithinformationfromoutsidesources[4].One wayto do this is to use a method called retrieval which finds information based on keywords. Another way is to use a methodcalledretrieval,whichfindsinformation basedon themeaningofthetext.

Recently, people have been working on ways to combinethesetwomethods[1][2].Onewaytodothisisto use a method called Reciprocal Rank Fusion, which combines the results of searches. Another way is to use a method called Cross-Encoder, which checks if the results are relevant to the question. We are going to use these methodsinoursystem.

3. Mathematical Formulation

Tomake our systemwork weneedto use someformulas. These formulas help us combine the results of searches, checkiftheresultsarerelevantanddetectwhenwedonot haveenoughinformation.

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

3.1.ReciprocalRankFusion(RRF)

First, we use a formula called Reciprocal Rank Fusion to combine the results of searches. This formula takesthe resultsofeachsearch.Combinethemintoonelist.Thelist

isranked,sotherelevantresultsareatthetop.

3.2. Final Hybrid Score

The final scoring function combines the retrieval score with the Cross-Encoder score. The Cross-Encoder checks howwell a queryanddocumentmatchbyencoding them. This gives an idea of how relevant they are. The parameters alpha and beta decide how much each part, retrieval,andsemanticre-rankingcontributetothe score. This scoring system makes sure that both how well a document is retrieved and how semantically relevant it is areconsideredwhenselectingdocuments.

3.3. Confidence Score

The confidence score shows how reliable the generated response is. It does this by averaging the scores of the documents that were used to create the response. The system also uses a function to make sure the confidence score is lower when the system is not being truthful or whenithastouseadefaultresponse.Thiswaythesystem can tell users how much they can trust what it says. The confidence score is very important because it helps users know if they can believe the generated response. The system uses the confidence score to give users an idea of howreliabletheresponses.

e-ISSN:2395-0056

p-ISSN:2395-0072

parameter λ work together to make sure the documents that are picked are both relevant to the query and different,fromeachother.

3.5. Failure Rate

The failure detection metric[14] looks at how the system canfind queriesthatitcannotanswer because itdoes not have enough information. If the system has a high failure detection rate that means it is good at knowing what it cannotdo.Itgivesagoodresponsewhenitcannotanswer something. The failure detection metric is important because it helps the system avoid giving answers. The system uses the failure detection to know when it should saythatitdoesnotknowsomething.Thefailuredetection metricisapartofthesystemandithelpsthesystemtobe better,atthefailuredetection.

4. Dataset and Preprocessing

To test our system we need a lot of text data. We use a dataset that has different types of text, including articles and books. We preprocess the data by removing any characters and splitting it into smaller chunks[6]. This makes it easier for our system to find the information it needs[11].

We hope that our system will be able to give accurate answers, than other systems. We also hope that it will be abletodetectwhenitdoesnothaveinformationtogivea good answer. In the section we will talk about how we testedoursystemandwhattheresultswere.

Each document is broken down into pieces and then processed in twoseparatewaysatthesametime.Forthe 3.4.Maximal

Marginal Relevance (MMR)

The MMR formulation is a way to balance the relevance and differentness of the documents. It picks documents that're very relevant to the query and not similar to the documents that were picked before. The parameter λ is what controls this balance. The system can decide how important it is to have different documents compared to how relevanttheyare to the query. This helpsto stop the systemfrompickingthekindofdocumentsoverandover whichmakestheresponsesthataregeneratedalotbetter. The MMR formulation does this by making sure the documents are not too similar to each other. The parameterλ isimportant because ithelpsthe system find the balance between relevance[16] and diversity. This balance is what makes the MMR formulation so useful for generating responses. The MMR formulation and the

way the system uses something called BM25 indexing to figure out how relevant the document is based on how oftencertainwordsappearandhowraretheyareinother documents. For the way the system uses a special kind of model to create embeddings,which are like maps of what the document is about and stores them in a special index calledFAISSthat makesit easytofind similar documents. The system also keeps track of things like where the document came from and what it's about so that it can provide more accurate and useful results. This two-part approach lets the system use both the words in the documentandwhattheymeantofindtheresults.

5. Methodology

The method we are using has a lot of steps to make sure we get the information[2]. First we take the user's

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

question. Get it ready for the system to look at. We then sendthisquestiontotwosystemsthatfindinformation[9]: onesystemlooksforkeywords,andtheothersystemlooks forthingsthatmeanthesamething[2].

The system that looks for keywords finds documents that have the words as the question. The other system finds documents that're similar in meaning even if they do not have the same words. We then combine the results from both systems[2], which makes sure that the documents thatbothsystemsthinkareimportantgetapriority.

We then use another system to look at the question and thedocumentstogether[15]whichhelpsusfigureouthow relevanteachdocumentistothe question[3].Thissystem is better at understanding what the question means than systems.

After that we pick documents that are not only relevant butalsodifferentfromeachother.Thismakessurethatwe donotgettheinformationoverandoveragain[5].Wethen usethesedocumentstocreatetheanswer[20].

At the time we are also checking how sure we are about the answer. If we are not sure the system will say so instead of making something up[3]. This makes sure that the information we give is not relevant but also trustworthy.Themethodisdesignedtomakesurethatthe information retrieval system gets the information and is alsoreliable[1].

We use the information retrieval system to get the documents and the language model to create the final answer. The information retrieval system has a lot of parts[7]includingthekeywordsystemandthesystemthat looks for meanings. The language model uses the documents to create the answer. This is how the method workstogettheinformation.

6. Result and Analysis

6.1. Retrieval

Performance

TheHybridRAGsystemdoesreallywellwhenitcomes to finding the information[22]. It gets the Mean Reciprocal Rank with a score of 0.8056. This is better than the Basic RAG system which has a score of 0.7931 and the Agentic RAGsystemwhichhasa score of0.7533. The HybridRAG systemisgoodbecauseitcombinestwowaysofsearching forinformation.Itlooksatthewordsandwhattheymean.

The Basic RAG system is a little better at picking the documents. It gets a Precision@5 score of 0.2867.. The Hybrid RAG system is still better at ranking all documents[11].The Hybrid RAGsystemisgoodatfinding the information because it uses both the Hybrid RAG system method and the Hybrid RAG system way of

e-ISSN:2395-0056

p-ISSN:2395-0072

searching. The Hybrid RAG system is just better than the BasicRAGsystemandtheAgenticRAGsystem.

6.2. Generation Quality

The results of how things were made show that Agentic RAGmakesthemostcompleteandrelevantanswers[13].It got scores of 0.5655 for being complete and 0.565 for being relevant.. Basic RAG did the best when it came to beingfaithfultowhatitfound.Itgotascoreof0.455which means its answers were more based on what it found. Hybrid RAG did a job with all the measures. This shows that using ways to find information helps make better answerswithoutlosingtoomuchfaithfulnesstowhatwas found. Rag and Basic RAG and Hybrid RAG all have their good points. Agentic RAG makes answers and relevant answers. Basic RAG is good at being faithful to what it found.HybridRAGdoesabitofeverything.

Chart-2:GenerationQualityComparison

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

6.3. Failure Detection

The Basic RAG system is really good at finding failures; it gets it most of the time about 0.9167. The Hybrid RAG systemisnextitisgoodtoobutnotasgoodasBasicRAGit getsitright0.85ofthetime.TheAgenticRAGsystemisnot asgoodatfindingfailures;itonlygetsitrightabout0.6833 ofthetime.

This shows that the simpler systems, like Basic RAG and Hybrid RAG are better at finding out when something is missing.

But when it comes to failures the Agentic RAG system is actually really good. It can find all of them which is great when you have to think about really hard problems, like complex reasoning scenarios and the Agentic RAG system canhandlethem.

Chart-3:FailureDetectionPerformance

6.4. Latency Comparison

The Basic RAG system is really fast. It takes 9.7356 secondstorespond.TheHybridRAGsystemisabitslower. Ittakesaround12.8824seconds[18]torespondbecauseit hastodosomeworklikefusionandreranking.

The Agentic RAG system is really slow. It takes 65.5583 seconds to respond. This is because the Agentic RAG system has to do a lot of thinking and it has to do it in steps.TheBasicRAGsystemisthefastest.TheHybridRAG system is, in the middle. The Agentic RAG system is the slowest.

e-ISSN:2395-0056

p-ISSN:2395-0072

:LatencyComparison

6.4. Overall Comparison

The results really show the bad points of each system. Hybrid RAG is the best at finding the information. Basic RAGisgreatwhenitcomestobeingaccurate andfast. On the hand Agentic RAG is very good, at reasoning but it takeslongertogetthe results.HybridRAGandBasicRAG andAgenticRAGallhavetheirstrengthsandweaknesses.

7. Challenges

Table:OverallComparison

Thisworkhadaproblem.Wehadtomakesurethatthe systemcouldfindwhatwewerelookingforanddoit quickly.Whenwecombinedsystemstofindthingsand

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

thenrearrangedtheresultstogetthebestonesandmake themdifferentittooklongerandgotmorecomplicated[8].

Thesystemalsohadtobeabletotellwhenitwasgiving answers.Wehadtotrythingstogetitjustrightlike figuringoutthebestnumberstouseforthings,likeRRF constantsandMMRlambdavaluesandrerankingweights.

8. Applications

The new system can be used in different areas where we needtotrusttheartificialintelligenceandunderstandhow itworks[4].Wecanusethissysteminschoolstohelpwith research in companies to keep track of what they know and in computer programs that talk to customers[7]. It is also useful in systems that help lawyers and doctors find theinformationbecauseintheseareasitisveryimportant tobeaccurateandtrustworthy.Thesystemisreallyuseful when we need to be sure that the artificial intelligence is working correctly like in the systems that lawyers and doctorsuse[9]tofindinformation.

9. Limitations

The system has some limitations. It really needs quality data to work properly. The Cross-Encoder reranking method uses a lot of computer power, which means it is not very good, for applications that need to work in time. Thesystemalsousesaconfidencescoringmechanismthat is based on guesses[12]. This may not be able to find all the subtle cases of hallucination detection[16]. The hallucination detection is something that the system is supposedto do. Itis not always able todoit well because of these limitations. The system and the hallucination detection are closely. The limitations of the system can affectthehallucinationdetection[19].

10. Future Work

The future work will be, about making the system work faster. This can be done by using model distillation and othermethodstogettheinformationweneedquickly[11]. We also want to use reinforcement learning to help the system[7] decide what information is important[6]. This will help the system get better at finding what we are looking for.We also need to work on stopping the system from giving us information. This can be done by using verificationmodelstochecktheinformation.

Another thing we can do is make the system work with types of data like pictures and videos. This is something wecanlookintointhefuture.

11. Conclusion

Thispaper isabout a HybridRAG framework. The Hybrid RAG framework doesalotofthings.Itcombinesretrieval

e-ISSN:2395-0056

p-ISSN:2395-0072

fusion and semantic reranking. It also does diversity optimizationandconfidence-basedreliabilityestimation. The system is really good at finding the information and creating text that is balanced and makes sense. It also helpstostophallucinations[13].

Theresultsshowthatusinganapproachisveryimportant for building strong and trustworthy RAG systems. This is especiallytrueforreal-worldapplicationswhereyouneed tobeaccurateandreliable[1].TheHybridRAGframework is a help in these situations because it uses a hybrid approach. The Hybrid RAG framework is very useful for buildingRAGsystemsthat'rerobustandtrustworthy.

REFERENCES

[1] Mishra,Saroj,etal."Sok:Agenticretrieval-augmented generation(rag):Taxonomy,architectures,evaluation, and research directions." arXiv preprint arXiv:2603.07379(2026).

[2] Cento, Ramiro López. "Agentic Retrieval Augmented GenerationforEstimationofDistributionAlgorithms." Algorithms(2025).

[3] Wei, Tianxin, et al. "Agentic reasoning for large language models." arXiv preprint arXiv:2601.12538 (2026).

[4] Cheng, Mingyue, et al. "A survey on knowledge-oriented retrieval-augmented generation." arXivpreprintarXiv:2503.10677(2025)..

[5] Aggarwal, Vedanth. "Empowering Large Language Model Reasoning: Hybridizing Layered Retrieval Augmented Generation and Knowledge Graph Synthesis."Int.J.HighSch.Res6(2024):80-92.

[6] e Aquino, Gustavo de Aquino, et al. "From rag to multi-agent systems: A survey of modern approaches inllmdevelopment."(2025).

[7] Gong, Ming. Toward A Self-Evolving Agent In Multi-Turn Dialogue Question-Answering Systems. Diss.UniversityofDayton,2025.

[8] Gao,Huan-ang,etal."ASurveyofSelf-EvolvingAgents: What,When,How,andWheretoEvolveonthePathto Artificial Super Intelligence." arXiv preprint arXiv:2507.21046(2025).

[9] Gautam, Aditya. "Multi-agent systems for misinformation lifecycle: Detection, correction and source identification." arXiv preprint arXiv:2505.17511(2025).

[10] Daull,Xavier,etal."ComplexQAandlanguagemodels hybrid architectures, Survey." arXiv preprint arXiv:2302.09051(2023).

[11] Lensu, Urho. "Impact of Chunking Granularity on Accuracy and Token Consumption in Retrieval-Augmented Generation for Question-Answering."(2025).

International Research Journal of Engineering and Technology (IRJET)

Volume:13Issue:04|Apr 2026 www.irjet.net

[12] Raza, Shaina, et al. "Responsible Agentic Reasoning and AI Agents: A Critical Survey." Authorea Preprints (2025).

[13] Baibakova, Viktoriia, and Alexey Serov. "Agentic framework for programmatic crystal structure generationusingafine-tunedworker–supervisorlarge languagemodel."EnergyandAI(2026):100710.

[14] Wegener, Gregor. "SORT-AI: A Structural Safety and Reliability Framework for Advanced AI Systems with Retrieval-Augmented Generation as a Diagnostic Testbed."(2025).

[15] Singh,Divyansh."BridgingKnowledgeandGeneration: ASurveyonRetrieval-AugmentedGeneration."

[16] Li,Xiaopeng,etal."Asurveyofpersonalization: From rag to agent." ACM Transactions on Information Systems(2025).

[17] Gao, Yunfan, et al. "Synergizing rag and reasoning: A systematic review." arXiv preprint arXiv:2504.15909 (2025).

[18] Wang, Zhichao, Cheng Wan, and Dong Nie. "Review of Inference-Time Scaling Strategies: Reasoning, Search andRAG."arXivpreprintarXiv:2510.10787(2025).

[19] Wang,Ziqi,etal."Asurveyonparallelreasoning."arXiv preprintarXiv:2510.12164(2025).

[20] Aggarwal, Vedanth. "Empowering Large Language Model Reasoning: Hybridizing Layered Retrieval Augmented Generation and Knowledge Graph Synthesis."Int.J.HighSch.Res6(2024):80-92.

[21] Tang,Fei,etal."Asurveyon(m)llm-basedguiagents." arXivpreprintarXiv:2504.13865(2025).

[22] Ioannis, et al. "From illusion to insight: A taxonomic survey of hallucination mitigation techniques in LLMs." AI 6.10 (2025): 260.Cento, Ramiro López. "Agentic Retrieval Augmented Generation for Estimation of Distribution Algorithms." Algorithms (2025).

[23] Wei, Tianxin, et al. "Agentic reasoning for large language models." arXiv preprint arXiv:2601.12538 (2026).

[24] Baig, Afnan, and D. Sc Mikko Raatikainen. "Autonomous LLM Monitoring and Remediation Framework."(2025)

e-ISSN:2395-0056

p-ISSN:2395-0072