Improving Retrieval-Augmented Generation Performance Using the MAF-RAG Architecture, EVR–VOR Vector Retrieval, and Multi-Agent Fallback Reasoning

Abstract

Retrieval-Augmented Generation (RAG) AI chatbots have gained popularity for their effectiveness in producing accurate, fast, and reliable responses; however, they have faced critical challenges stemming from limited datasets, outdated documents, and noisy, unfiltered data. This study proposes a Multi-Agent Fallback in Retrieval Augmented Generation (MAF-RAG). This robust RAG system testing pipeline integrates three-phase retrieval, filtering, and re-ranking data, along with a multi-agent debating process to address these challenges. This study demonstrates MAF-RAG's ability to perform under a constrained dataset, using a near-deployment dataset of 1,100 real-world documents. The pipeline utilizes 150 testing queries, carefully selected to reflect real-world RAG-based chatbot scenarios. A sentence-transformers/all-MiniLM-L6-v encoder encodes various chunks of documents into a 384-dimensional query vector embedding, ensuring an accurate relationship between testing queries and vectorized documents. The results show that the proposed MAF-RAG significantly outperforms the baseline system, achieving a mean F1-score of 0.556, an improvement of 18.8% over the Enhanced Baseline (mean F1-score = 0.469) and a 70.0% improvement over the Legacy Baseline (mean F1-score = 0.327). MAF-RAG also achieves the highest success rate, with 78% of the queries, while other baseline systems manage only 34% and 62%, respectively. MAF-RAG also reduces the failure rate by 42.1%, significantly increasing system reliability. Although MAF-RAG exhibits an increase in latency of 4.9%, these trade-offs are outweighed by the significant improvements in system reliability and performance. These findings highlight the contribution of this study: by implementing a robust retrieval testing pipeline, system accuracy can be improved, reducing the presence of noisy and unfiltered documents, and increasing system performance even when faced with challenging and varied datasets, making it a suitable solution for a RAG-based chatbot system that faces dataset challenges.