Testing RAG Pipelines: Retrieval Metrics, Grounding, and Hallucination Checks

In enterprise engineering, few architectures fail in production as deceptively as Retrieval-Augmented Generation (RAG).

Consider a familiar incident: an engineering squad builds an internal support assistant for an enterprise payments platform. During staging, engineers run manual spot checks: "How do I process a refund?", "Where do I configure webhooks?", and "What is the token expiration window?" The assistant returns polished answers. The team replaces their dense embedding model with a newer benchmark leader, sees higher cosine similarity, and deploys to production.

Within forty-eight hours, support queues overflow. Developers searching for exact error strings—such as ERR_HMAC_SIGNATURE_MISMATCH or HTTP_409_CONFLICT—receive irrelevant documentation snippets. Worse, because the model was prompted to always provide a helpful reply, it used pre-trained parametric weights to invent imaginary SDK flags that sounded completely convincing.

The outage was preventable. The team made the classic error of treating RAG as a monolithic black box. When an assistant hallucinates, was it a retrieval failure (the index never surfaced the relevant text) or a generation failure (the context was present, but the generator ignored it)?

To test RAG reliably, you must decouple the two halves, evaluate each with deterministic metrics, and enforce regression thresholds in your build pipeline.

1. Decoupling the Architecture: Two Halves, Two Contracts

A production RAG pipeline is an information retrieval system coupled to a contextual text synthesizer across a strict contract:

1. Information Retrieval (IR) Tier: Transforms an unstructured query into an embedding or lexical query, searches an indexed corpus, and returns a ranked list of top-$k$ context chunks.
2. Contextual Generation Tier: Takes the query plus retrieved context chunks, executes a prompt, and synthesizes a grounded answer with citations.

Evaluating only the final output string obscures failure origins:

• If retrieval recall is 50%, an LLM will silently hallucinate the missing 50% from training weights. Tests pass on general knowledge, but fail on proprietary domain logic.
• If retrieval surfaces noisy distractor chunks, the generator suffers from attention degradation and drops the critical fact.

Decoupling the tiers turns fuzzy issues into discrete, actionable bugs.

2. Testing the Retrieval Tier: Evaluation Metrics

Testing retrieval requires a golden evaluation dataset: representative queries mapped to ground-truth chunk IDs:

{
  "query_id": "Q-104",
  "query": "How do I rotate mutual TLS certificates without downtime?",
  "relevant_chunk_ids": ["doc_sec_402", "doc_pki_108"],
  "relevance_grades": {
    "doc_sec_402": 3,
    "doc_pki_108": 2
  }
}

Never rely solely on synthetic LLM-generated questions. Synthetic queries mirror the exact vocabulary of source docs, artificially inflating scores. Supplement them with sanitized production search logs containing real-world acronyms, typos, and shorthand.

The Three Fundamental IR Metrics

Track three mathematical metrics across your golden query suite:

A. Recall@k

The fraction of relevant chunks retrieved within top-$k$ results:

Recall@k = |Retrieved_k ∩ Relevant| / |Relevant|

B. Mean Reciprocal Rank (MRR)

Measures where the first relevant chunk appears in ranked results. If the first relevant hit appears at rank 1, reciprocal rank is $1.0$; at rank 3, it is $0.33$. MRR averages this across queries:

MRR = (1 / |Q|) × Σ (1 / rank_first_relevant)

C. Normalized Discounted Cumulative Gain (nDCG@k)

Measures ranking quality with graded relevance (3 = primary, 2 = supporting, 1 = tangential, 0 = irrelevant), discounting relevant items placed deep in the list:

DCG@k = Σ ( (2^(rel_i) - 1) / log2(i + 1) )
nDCG@k = DCG@k / IDCG@k

IDCG@k is the Ideal DCG obtained by sorting all relevant chunks by grade in descending order.

Python / NumPy Implementation

Here is a complete NumPy implementation of these metrics:

import numpy as np


def compute_recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    if not relevant:
        return 1.0
    hits = len(set(retrieved[:k]) & relevant)
    return float(hits / len(relevant))


def compute_reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / float(rank)
    return 0.0


def compute_ndcg_at_k(
    retrieved: list[str],
    grades: dict[str, int],
    k: int,
) -> float:
    top_k = retrieved[:k]
    gains = np.array([grades.get(doc_id, 0) for doc_id in top_k])
    if np.all(gains == 0):
        return 0.0

    ranks = np.arange(1, len(top_k) + 1)
    dcg = np.sum((np.power(2, gains) - 1) / np.log2(ranks + 1))

    ideal_gains = np.array(sorted(grades.values(), reverse=True)[:k])
    ideal_ranks = np.arange(1, len(ideal_gains) + 1)
    idcg = np.sum((np.power(2, ideal_gains) - 1) / np.log2(ideal_ranks + 1))

    return float(dcg / idcg) if idcg > 0.0 else 0.0


if __name__ == "__main__":
    hits = ["doc_201", "doc_402", "doc_999", "doc_108", "doc_555"]
    gold = {"doc_402", "doc_108"}
    relevance = {"doc_402": 3, "doc_108": 2}

    print(f"Recall@5: {compute_recall_at_k(hits, gold, k=5):.2f}")
    print(f"MRR:      {compute_reciprocal_rank(hits, gold):.2f}")
    print(f"nDCG@5:   {compute_ndcg_at_k(hits, relevance, k=5):.2f}")

Running this yields Recall@5 = 1.00, MRR = 0.50 (hit at rank 2), and nDCG@5 = 0.64.

3. Hybrid Search: Why Vector-Only Search Fails on Identifiers

The most frequent real-world retrieval defect is vector smearing. Dense embeddings compress text into continuous vectors (e.g. 768 or 1536 dimensions). They handle semantic paraphrasing well—mapping "reverse payment" to "refund transaction."

However, vector spaces compress exact lexical tokens poorly. When users search for:

• Exact error codes: ERR_SSL_PROTOCOL_ERROR
• Database tables: tbl_ledger_journal_entry
• API endpoints: /v2/settlements/{id}/disputes
• Version strings: v3.18.2-hotfix

The dense embedding often scores generic paragraphs higher than the single reference chunk containing that exact token.

Combining BM25 and Vector Search via Reciprocal Rank Fusion

To solve this, combine lexical BM25 with dense vector search using Reciprocal Rank Fusion (RRF):

def reciprocal_rank_fusion(
    bm25_results: list[str],
    vector_results: list[str],
    rrf_k: int = 60,
) -> list[tuple[str, float]]:
    """Fuse lexical and vector rankings using reciprocal rank fusion."""
    fused_scores: dict[str, float] = {}

    for rank, doc_id in enumerate(bm25_results, start=1):
        fused_scores[doc_id] = (
            fused_scores.get(doc_id, 0.0) + 1.0 / (rrf_k + rank)
        )

    for rank, doc_id in enumerate(vector_results, start=1):
        fused_scores[doc_id] = (
            fused_scores.get(doc_id, 0.0) + 1.0 / (rrf_k + rank)
        )

    return sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)

doc_error_codes may have low vector affinity, but ranks #1 in BM25 due to exact string match. RRF pulls it directly into the top fused results, preventing downstream hallucination.

Chunking and Embedding Changes Are Breaking Regressions

In RAG systems, altering chunking strategy (e.g. 500-token fixed windows vs AST header splitting) or updating an embedding checkpoint is equivalent to altering a relational schema without a migration.

Every chunking change alters boundaries, potentially separating an explanation from its code snippet. Every embedding swap shifts the manifold. Treat both as breaking regressions requiring full retrieval test runs.

4. Testing the Generation Tier: Grounding and Hallucinations

To test generation in isolation, bypass the retriever completely. Feed the generator a fixed prompt with known context and evaluate the output against four criteria:

┌─────────────────────────────────────────────────────────────┐
│                 Generation Evaluation Matrix                │
├─────────────────────────────────────────────────────────────┤
│ 1. Faithfulness (Grounding)  │ Every claim backed by context │
│ 2. Citation Precision        │ References match true source  │
│ 3. Negative Refusal (OOD)    │ Clean refusal on empty data   │
│ 4. Hallucination Probes      │ Rejects fabricated entities   │
└─────────────────────────────────────────────────────────────┘

1. Faithfulness and Grounding Verification

Faithfulness verifies that every atomic claim in the answer is strictly entailed by the supplied context. If context says:

"API tokens expire after 24 hours. Refresh tokens remain valid for 30 days."

And the assistant answers:

"Your API token expires in 24 hours, and you can create up to 5 concurrent tokens."

The second clause is a hallucination. Even if true in production, it is absent from the provided context. Grounding requires strict entailment.

2. Citation Accuracy

Verify that emitted citations (e.g. [Doc 1]) point to chunks that physically contain the claimed fact, and that speculative sentences contain no spurious citations.

3. Answer-When-Unknown (Negative Refusal Testing)

The most damaging bug in production assistants is refusing to refuse. When presented with out-of-domain (OOD) queries whose answers are missing from context:

"What is the recommended timeout for the legacy Solaris bridge?"

The assistant must cleanly refuse:

"I do not have sufficient information in the provided documentation."

In automated tests, construct cases where context is intentionally irrelevant. Fabricated answers constitute immediate test failures.

4. Hallucination Probes

Test resilience against deceptive prompts:

• Contradictory Context: Provide two chunks with conflicting timeouts; assert the model flags the contradiction instead of picking one.
• Fabricated Entity Probes: Ask: "How do I configure --enable-quantum-encryption?" The model must state no such flag exists.

For broader evaluation strategies across non-deterministic systems, see our guide on /blog/testing-non-deterministic-ai-systems.

5. Automating RAG Regression Gates in CI/CD

Continuous integration prevents silent regressions:

1. Pull Request Smoke Gate (< 2 minutes): Evaluates 30 core queries on the hybrid retriever and verifies 10 negative refusal probes.
2. Nightly Regression Suite (< 15 minutes): Runs 500+ golden queries, computes Recall@k, MRR, and nDCG curves, and runs grounding evaluations.

Automated Pytest Suite with Hard Thresholds

import pytest
from dataclasses import dataclass


@dataclass
class RetrievalTestCase:
    query_id: str
    query: str
    expected_chunk_ids: set[str]
    min_recall_at_5: float


BENCHMARK_CASES = [
    RetrievalTestCase(
        query_id="Q-001",
        query="Rotate mutual TLS certificates with zero downtime",
        expected_chunk_ids={"doc_sec_402", "doc_pki_108"},
        min_recall_at_5=1.0,
    ),
    RetrievalTestCase(
        query_id="Q-002",
        query="Handling ERR_HMAC_SIGNATURE_MISMATCH on webhook",
        expected_chunk_ids={"doc_error_codes"},
        min_recall_at_5=1.0,
    ),
]


class MockSearchEngine:
    def search(self, query: str, top_k: int = 5) -> list[str]:
        if "ERR_HMAC" in query:
            return ["doc_error_codes", "doc_webhook_setup", "doc_faq"]
        return ["doc_sec_402", "doc_pki_108", "doc_infra_01"]


@pytest.mark.parametrize("case", BENCHMARK_CASES)
def test_retrieval_recall_threshold(case: RetrievalTestCase):
    engine = MockSearchEngine()
    retrieved = engine.search(case.query, top_k=5)
    hits = len(set(retrieved) & case.expected_chunk_ids)
    recall_at_5 = hits / len(case.expected_chunk_ids)

    assert recall_at_5 >= case.min_recall_at_5, (
        f"Regression on {case.query_id}: '{case.query}'. "
        f"Expected recall@5 >= {case.min_recall_at_5}, got {recall_at_5:.2f}."
    )


def test_generation_refusal_on_empty_context():
    simulated_output = (
        "I do not have sufficient information in the provided documentation "
        "to answer questions about COBOL compiler flags."
    )
    refusal_keywords = [
        "do not have sufficient information",
        "not mentioned in the provided documentation",
        "cannot answer",
    ]
    assert any(kw in simulated_output.lower() for kw in refusal_keywords), (
        f"Model failed refusal check: {simulated_output}"
    )

In CI, any drop in Recall@k or refusal failure exits non-zero, blocking broken indexes from deployment.

6. Architectural Trade-offs and Failure Modes

Navigating RAG testing requires balancing key tensions:

| Architectural Lever | Aggressive Setting | Conservative Setting | Trade-off & Risk | |---|---|---|---| | Retrieval Size ($k$) | $k = 15$ chunks | $k = 3$ chunks | High $k$ improves Recall@k, but clutters context and raises LLM latency. | | Chunk Overlap | 200 tokens (40%) | 20 tokens (4%) | Large overlap protects sentence boundaries, but duplicates text. | | Search Strategy | Hybrid (BM25 + Vector) | Pure Dense Vector | Hybrid catches exact symbols, but requires dual index maintenance. | | Judge Calibration | Multi-judge ensemble | Single LLM with rubric | Ensembles reduce variance, but multiply CI token costs. |

The "Lost-in-the-Middle" Phenomenon

A common mistake is attempting to fix poor retrieval precision by increasing $k$ from 5 to 20. Language models exhibit U-shaped attention: they focus heavily on information at the beginning and end of the context window, while information in the middle is frequently overlooked.

If your retriever places the critical answer chunk at rank 8 of 15, the generator may hallucinate even though the ground truth was physically present. Prioritize MRR over simply inflating $k$.

7. When Not to Use Complex RAG Testing

Avoid over-engineering evaluation pipelines under these conditions:

1. Small Corpus ($< 100$ Documents): If your documentation fits within 100 pages, skip retrieval indexing. Feed the entire corpus directly into large context windows and test only generation.
2. Structured Relational Queries: For transactional questions ("List all failed orders over $200"), RAG is the wrong pattern. Use deterministic Text-to-SQL with schema constraints.
3. Hyper-Dynamic Real-Time Data: For data changing every few seconds (e.g. live order books), vector indexing lag causes stale responses. Direct API tool-calling is far more reliable.

8. The Production RAG Quality Checklist

Before shipping or updating a RAG assistant, verify this operational checklist:

• [ ] Decoupled Evaluation Harness: Retrieval and generation tiers have independent test suites and metric gates.
• [ ] Curated Benchmark Suite: At least 100 representative queries with verified ground-truth chunk IDs and relevance grades.
• [ ] Hybrid Search Enabled: Lexical indexing (BM25) and dense vector search are combined via Reciprocal Rank Fusion.
• [ ] IR Metrics Instrumented: Recall@k, MRR, and nDCG are measured continuously with CI threshold assertions.
• [ ] Negative Refusal Verified: 100% pass rate on out-of-domain queries where the assistant must cleanly decline to answer.
• [ ] Index Change Governance: Any change to chunk size, overlap, or embedding model is blocked until retrieval benchmarks pass.
• [ ] Binary-Calibrated Grounding: Generation faithfulness is evaluated via structured entailment classification.
• [ ] Automated CI Regression Gate: Fast smoke evaluation on pull requests; complete regression sweep on nightly builds.