Implementation of RAG in Jambi Cultural Information Question Answering System Using Qwen2.5-7B

Abstract

Large Language Models (LLMs) are inherently closed-book, making them prone to hallucination when answering domain-specific queries with limited training data, such as local culture. Information on Jambi's cultural heritage remains scattered across various sources, causing inefficient integrated retrieval. This study implements a Retrieval-Augmented Generation (RAG) architecture using the open-source LLM Qwen2.5-7B, strengthened by Hybrid Search (BM25 and dense retrieval fused via Reciprocal Rank Fusion), BGE-Reranker-Large reranking, and a Confidence Scorer. Three retrieval configurations Dense RAG, BM25 RAG, and Hybrid RAG were evaluated end-to-end using 130 In-Domain (ID) and 50 Out-of-Context (OOC) test queries, measured through Precision@10, Recall@10, Final Context Precision/Recall, BLEU-1, BLEU-4, ROUGE-L, Groundedness, and Rejection metrics (FRR/TRR). Results show that Dense RAG performed most dominantly end-to-end, achieving the highest Pure Recall@10 (0.992), BLEU-1 (0.499), ROUGE-L (0.506), Groundedness (0.611), and lowest False Rejection Rate (11.5%). BM25 RAG showed a specific advantage in Final Context Precision (0.746) after reranking. Hybrid RAG offered a competitive performance compromise, with Final Recall (0.938) and BLEU-4 (0.288) on par with Dense, alongside absolute hallucination protection (100% True Rejection Rate) on OOC queries. These findings indicate that while Hybrid fusion and the Confidence Scorer effectively mitigate hallucination, pure Dense retrieval consistently provides the most robust context support on the tested Jambi Culture dataset. These results demonstrate that systematic Hybrid Search and Confidence Scorer integration, combined with a dual In-Domain/Out-of-Context evaluation framework, provides a practical and empirically grounded configuration guideline for building hallucination-resistant RAG systems in low-resource local-culture domains.