How to Evaluate a RAG System Before It Ships

Most RAG demos look flawless and most RAG products quietly lie to users. The gap between those two states is evaluation, and it is the step teams skip because a chatbot that answers three questions in a meeting feels like proof. It is not. Knowing how to evaluate a RAG system is what separates a demo you cheer at from a system you can put in front of a client and defend.

I have shipped retrieval-augmented systems over legal documents, product catalogues and internal knowledge bases, and every one of them passed the eyeball test long before it was safe. This post is the eval process I now run before any RAG build reaches a sign-off. If you are still assembling the pipeline itself, start with how to build a RAG chatbot on your knowledge base and come back here to prove it works.

Why does RAG need its own evaluation, separate from the LLM?

Because a RAG system can fail in two completely different places, and a single quality score hides which one broke. A generic LLM eval asks whether the model wrote a good answer. A RAG eval has to ask two questions in sequence: did the retriever fetch the right context, and then did the generator stay faithful to it? Those are independent failure modes with independent fixes.

Consider what a wrong answer actually tells you. If the retriever never surfaced the relevant chunk, the model was doomed regardless of how good it is, and no prompt tuning saves you. If the retriever did surface the right chunk and the model ignored it or invented a detail, your retriever is fine and your generation step is hallucinating. Collapse both into one number and you will spend a week tuning the wrong half.

"A single accuracy score on a RAG system tells you the patient is sick. It never tells you which organ is failing."

This is also why swapping frameworks rarely fixes a bad RAG system. The choice between LangChain and LlamaIndex changes ergonomics, not retrieval quality. Evaluation, not framework selection, is what surfaces the real defect.

The two halves: retrieval quality versus generation quality

Split every RAG evaluation into a retrieval stage and a generation stage, and measure each against its own ground truth. Retrieval is an information-retrieval problem with decades of established metrics. Generation is a grounding problem that needs newer, LLM-scored metrics. Treat them as one and you lose all diagnostic power.

The retrieval stage takes a query and returns a ranked list of chunks. You judge it on whether the chunks that should have been retrieved actually were, and how highly they ranked. The generation stage takes those chunks plus the query and produces an answer. You judge it on whether the answer is supported by the retrieved chunks and whether it actually addresses the question. Keep the two ledgers separate and every regression points at a specific component.

What retrieval metrics should you actually track?

Track recall@k first, precision second, and a rank-aware metric like MRR or nDCG third. Recall@k answers the only question that can outright kill an answer: was the correct document anywhere in the top k results the model got to see? If recall@k is low, nothing downstream can recover.

The practical move is to tune k with these numbers rather than guessing. Raise k and recall climbs but precision usually falls and cost rises. The right k is the smallest one where recall@k plateaus, because past that point you are paying tokens to feed the model distractions.

What generation metrics catch hallucinations?

The generation metric that matters most is faithfulness, also called groundedness: every factual claim in the answer must be supported by the retrieved context. Faithfulness is your direct hallucination detector. An answer can be fluent, confident and completely fabricated, and faithfulness is the score that flags it.

The mechanics are simple and worth knowing. A judge decomposes the answer into individual claims, then checks each claim against the retrieved chunks. Faithfulness is the proportion of claims that are entailed by the context. An answer with three claims where only two are supported scores 0.67, and that is a number you can gate a release on.

Faithfulness and answer relevance together catch the two most damaging failure modes: confidently making things up, and confidently answering a question nobody asked. This is exactly the class of failure that turns into an incident once real users arrive, which is why I treat it with the same seriousness as deploying agents to production safely.

How do you build a golden test set for RAG?

Before any metric means anything, you need a golden test set: a fixed collection of questions, each paired with the source chunks that should answer it and, ideally, a reference answer. Without ground truth, every metric above is unanchored. The golden set is the single most valuable and most skipped artefact in RAG evaluation.

The unanswerable questions are the ones people forget, and they are where hallucination hides. A RAG system that never says 'I don't know' is not confident, it is dangerous, and only a test set that includes questions with no valid answer will ever expose it.

LLM-as-judge and where it quietly lies to you

You cannot hand-grade faithfulness across hundreds of questions on every commit, so you use a strong model as an automated judge. LLM-as-judge is how metrics like faithfulness and answer relevance get computed at scale: the judge model reads the question, context and answer, and returns a structured verdict. It works, and it also introduces its own biases you have to manage.

The discipline that keeps LLM-as-judge honest is calibration. Take a sample of a hundred items, grade them by hand, and measure how well the judge agrees with your human labels. If agreement is high, trust the judge on the full set. If it is low, fix the judge prompt before you trust a single automated number. A judge you have never calibrated is a random number generator with good manners.

Ask the judge for structured output with a per-claim breakdown and a short rationale, not a bare score. The rationale is what lets you audit a disagreement, and structured verdicts are what let you compute faithfulness as a real proportion instead of guessing at a five-point scale.

from typing import Literal
from pydantic import BaseModel, Field

# The schema a faithfulness judge must return per answer.
class ClaimVerdict(BaseModel):
    claim: str
    supported: bool
    evidence: str = Field(description="Quote from context, or empty if unsupported.")

class FaithfulnessResult(BaseModel):
    claims: list[ClaimVerdict]
    verdict_rationale: str

    @property
    def faithfulness(self) -> float:
        if not self.claims:
            return 0.0
        supported = sum(c.supported for c in self.claims)
        return supported / len(self.claims)


def evaluate_case(case, rag_pipeline, judge) -> dict:
    result = rag_pipeline(case.question)
    retrieved_ids = {c.id for c in result.contexts}
    gold_ids = set(case.gold_chunk_ids)

    recall_at_k = len(retrieved_ids & gold_ids) / len(gold_ids)
    precision = len(retrieved_ids & gold_ids) / max(len(retrieved_ids), 1)
    faith = judge.score_faithfulness(
        question=case.question,
        answer=result.answer,
        contexts=result.contexts,
    )  # returns a FaithfulnessResult

    return {
        "id": case.id,
        "recall_at_k": recall_at_k,
        "context_precision": precision,
        "faithfulness": faith.faithfulness,
    }


# CI gate: fail the build if the suite regresses on the metric that matters.
MIN_FAITHFULNESS = 0.90
MIN_RECALL_AT_K = 0.85

def assert_release_ready(scores: list[dict]) -> None:
    n = len(scores)
    mean_faith = sum(s["faithfulness"] for s in scores) / n
    mean_recall = sum(s["recall_at_k"] for s in scores) / n
    assert mean_faith >= MIN_FAITHFULNESS, f"Faithfulness {mean_faith:.3f} below gate"
    assert mean_recall >= MIN_RECALL_AT_K, f"Recall@k {mean_recall:.3f} below gate"

Regression testing RAG in CI

Run the golden set as an automated test suite on every change that can move quality: prompt edits, chunk-size changes, embedding-model swaps, retriever settings and reranker tweaks. RAG systems have an enormous surface of silent regressions, because a change that improves one query class often degrades another, and only a fixed suite catches the trade.

The pattern is the same one you already use for code. Compute mean faithfulness, recall@k and answer relevance across the golden set, compare against a stored baseline, and fail the build if any headline metric drops beyond a tolerance. This turns 'the chatbot feels worse since Tuesday' into a red pipeline on the exact commit that caused it. The same instinct for guardrails and rollback that I described in nine lessons from running agents in production applies directly here.

How this maps to a POC sign-off

When a client asks whether the RAG proof-of-concept is ready to ship, 'it seemed good in the demo' is not an answer you can stand behind. The eval suite is. A sign-off becomes a table of numbers against thresholds you agreed up front, and the conversation shifts from opinion to evidence.

That checklist is deliberately boring, and that is the point. Boring, numeric and repeatable is what lets you charge for a system rather than a demo, and it is how I run every retrieval build inside my AI engineering services. For B2B SaaS teams shipping a knowledge assistant to their own customers, this eval discipline is the difference between a feature and a liability.

Knowing how to evaluate a RAG system is not a phase you finish, it is the harness you keep. The corpus grows, the models change under you, and the only thing that tells you whether last week's edit helped or hurt is a golden set, two ledgers of metrics, and a CI gate that is willing to say no.