How to Build a RAG Chatbot on Your Own Knowledge Base

Almost every company that asks me for an AI chatbot actually wants the same thing: an assistant that can answer questions from their own documents without making things up. That is what retrieval augmented generation delivers, and knowing how to build a RAG chatbot properly is the difference between a demo that impresses a boardroom and a system your support team actually trusts on a Monday morning.

I have shipped this pattern across support docs, clinical guidance and internal wikis. This is the full pipeline I use, the trade-offs that matter, and the failure modes that quietly wreck a chatbot built for a website when nobody instruments retrieval.

What is retrieval augmented generation, in plain terms?

Retrieval augmented generation is a pattern where the model does not answer from memory. Instead, at query time you search a knowledge base for the passages most relevant to the question, paste those passages into the prompt, and ask the model to answer using only that supplied context. The LLM supplies fluency and reasoning; your documents supply the facts.

This separation is the whole point. The model's weights never see your private data, so you can update the knowledge base by editing a document rather than retraining anything. A policy change that would take a fine-tuning run to absorb becomes a one-line edit that is live on the next query.

Should you use RAG or fine-tuning for a knowledge base chatbot?

For a knowledge base chatbot, RAG wins almost every time. Fine-tuning teaches a model a style, a format or a narrow skill; it is a poor way to teach it facts, because facts change and retraining is slow and expensive. RAG injects fresh facts at inference time, which is exactly what a document-grounded assistant needs.

The two are not mutually exclusive. Fine-tune when you need the model to reliably speak in a house style or emit a strict format, and layer RAG on top for the facts. But if your only problem is "answer questions about our documents", start with RAG and do not touch fine-tuning until retrieval is measurably solid.

The full RAG pipeline, stage by stage

A production RAG chatbot is a pipeline of seven stages, and each one is a place quality can leak. Treat them as separately testable components, because when the bot answers badly you need to know whether retrieval missed or generation drifted.

Chunking: the decision that quietly determines your quality ceiling

Chunking is the most underrated stage. Chunk too large and you dilute the relevant sentence in a wall of irrelevant text, which wastes tokens and confuses the reranker. Chunk too small and you sever the context a passage needs to make sense, so a retrieved fragment answers half the question. Most of the bad RAG systems I am asked to fix have a chunking problem, not a model problem.

Embeddings and why hybrid search beats pure vectors

An embedding model maps text to a vector so that similar meaning lands close together. Pure vector search is excellent at semantics, it understands that "refund window" and "return period" are related, but it is surprisingly weak at exact matches: product codes, error strings, names and acronyms. Those are precisely the tokens users paste into a support chatbot.

The fix is hybrid search: run a keyword index such as BM25 alongside the vector search and fuse the two result sets, usually with reciprocal rank fusion. You get the semantic recall of embeddings and the exact-match precision of lexical search. Every serious knowledge base chatbot I have built uses hybrid retrieval, because pure vector search misses the SKU and pure keyword search misses the paraphrase.

Reranking: the cheapest quality win in the stack

Retrieval gives you a fast, approximate top 20. A reranker is a cross-encoder that reads the question and each candidate together and scores real relevance, then you keep the top 3 to 5. This second pass is slower per document, which is why you only run it on the shortlist, but it consistently lifts answer quality more than swapping to a bigger generation model. If you measure one thing after chunking, measure the effect of adding a reranker.

A minimal chunk, embed and retrieve example

Here is the core loop in Python: chunk a document, embed the chunks into Qdrant, then retrieve for a query. It uses a LangChain splitter for chunking and the Qdrant client directly so the moving parts stay visible.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from openai import OpenAI
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, PointStruct, VectorParams

client = OpenAI()
qdrant = QdrantClient(url="http://localhost:6333")
EMBED_MODEL = "text-embedding-3-small"  # 1536 dims


def embed(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(model=EMBED_MODEL, input=texts)
    return [d.embedding for d in resp.data]


# 1. Chunk: split on structure, keep a small overlap.
splitter = RecursiveCharacterTextSplitter(chunk_size=1800, chunk_overlap=200)
chunks = splitter.split_text(open("handbook.md").read())

# 2. Embed + store, carrying the source text as payload for citation.
qdrant.recreate_collection(
    collection_name="kb",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
)
vectors = embed(chunks)
qdrant.upsert(
    collection_name="kb",
    points=[
        PointStruct(id=i, vector=v, payload={"text": c, "source": "handbook.md"})
        for i, (c, v) in enumerate(zip(chunks, vectors))
    ],
)

# 3. Retrieve: embed the question, pull the closest chunks.
query = "How many days do I have to expense a client dinner?"
hits = qdrant.search(
    collection_name="kb", query_vector=embed([query])[0], limit=5
)
for h in hits:
    print(round(h.score, 3), h.payload["source"], h.payload["text"][:80])

In production you would add a BM25 index for hybrid search, a reranking pass over these hits, and batching for the embedding calls. But this is the honest skeleton, and it runs. The choice of LangChain versus LlamaIndex for the orchestration layer matters less than getting chunking and retrieval right underneath it.

Which vector database should you use?

The vector database stores your embeddings and serves nearest-neighbour queries. The honest answer for most teams is: if you are already on Postgres, start with pgvector and only move to a dedicated engine when scale or latency forces it.

I lean towards pgvector on Supabase for early builds because it keeps chunks, metadata and application data in one database you can back up and reason about, and towards Qdrant when metadata filtering and hybrid search at volume become the priority. Do not agonise over this choice up front; the abstraction is thin and migrating an embedding store is a day of work, not a rewrite.

How do you stop a RAG chatbot from hallucinating?

You control hallucination in a RAG chatbot with grounding and refusal, not by hoping the model behaves. Grounding means the answer must be traceable to retrieved chunks, and refusal means the bot says "I could not find that" when retrieval comes back thin rather than inventing a plausible paragraph.

A schema-constrained answer makes grounding enforceable. Here is the shape I use, where an answer is invalid unless it carries the chunk IDs it was built from.

from pydantic import BaseModel, Field


class Citation(BaseModel):
    chunk_id: int
    source: str
    quote: str = Field(description="The exact sentence supporting the claim.")


class GroundedAnswer(BaseModel):
    answer: str
    citations: list[Citation] = Field(
        description="Every factual claim must map to at least one citation."
    )
    answered: bool = Field(
        description="False when retrieved context does not support an answer."
    )


# If answered is False, show the refusal message, not the model's guess.
# If citations is empty but answered is True, reject and retry: no source, no answer.

"A RAG answer with no citation is not an answer, it is a confident guess wearing your logo."

How do you evaluate a RAG chatbot before trusting it?

You evaluate retrieval and generation separately, because a bad answer can come from either. Retrieval evaluation asks whether the right chunk was in the results at all; generation evaluation asks whether the model used the retrieved context faithfully. Conflating the two is how teams spend a week tuning prompts when the real problem was chunking.

Build a golden set of 50 to 100 real questions with known good answers before launch, and rerun it on every change to chunking, retrieval or prompts. I go deeper on the tooling and metrics in how to evaluate RAG systems; the short version is that without an eval set you are tuning blind, and every improvement is a guess.

What does a RAG chatbot cost to run?

Running costs split into three lines: a one-time embedding pass over the corpus, ongoing embedding of queries, and generation tokens per answer. For most internal knowledge bases the generation tokens dominate, and the embedding of the entire corpus is a rounding error you pay once.

The lever that actually controls the bill is how much context you stuff into each generation call. Retrieving and pasting 20 chunks when 4 would do triples your token cost per answer and, counterintuitively, often lowers quality by burying the relevant passage. This is exactly why the reranker earns its place: it lets you send fewer, better chunks. Model prices move constantly, so size a proof of concept against current published rates rather than a number from a blog post, mine included.

A realistic build timeline

A focused internal RAG chatbot is a two-to-four-week build, not a research project, provided the documents are accessible and reasonably clean. The slow part is almost never the model; it is ingestion and evaluation.

In a regulated setting the timeline stretches, mostly for access control and audit, not for the RAG core. When I built the assistant layer for SynthiCare NHS, grounding and citation were non-negotiable from day one, because a clinical answer a user cannot trace to a source is worse than no answer at all. That constraint is a good default even when nobody is forcing it on you.

If you want this built and evaluated properly rather than demoed and abandoned, that is the scope I run as an AI services engagement: ingestion, hybrid retrieval, reranking, grounded generation with citations, and an eval harness so you can prove it works before it faces real users.