Most website chatbots fail the same way. Someone wires a general model to a chat bubble, it answers confidently about a pricing tier that was retired last year, and support ends up cleaning up after it.
The fix is not a bigger model. It is retrieval. I have built grounded assistants over clinical guidance, financial data and product catalogues, and the architecture barely changes between them: ingest, chunk, embed, retrieve, ground, refuse, evaluate, ship. This is the build I actually run when a client asks me to build an AI chatbot for their website.
What does it take to build an AI chatbot for a website?
It takes an ingestion pipeline, a vector store, a retrieval step and a grounded prompt. The model is the least interesting part. Answer quality is dominated by what lands in the context window rather than by which model reads it, so ingestion and retrieval deserve most of your build time.
The eight stages of a RAG chatbot, and what goes wrong at each one:
| Stage | What you build | Most common failure |
|---|---|---|
| Ingestion | Crawler or CMS export into clean text plus URL | Nav, footer and cookie banners embedded as content |
| Chunking | 300 to 700 token chunks split on headings | One chunk per page, so retrieval returns noise |
| Embedding | Vector per chunk from an embedding model | Re-embedding everything on every deploy |
| Storage | pgvector column plus an HNSW index | No metadata, so you cannot filter or cite |
| Retrieval | Hybrid vector plus keyword search, then rerank | Top-k too large, diluting the context |
| Grounding | Prompt that permits only retrieved facts | Model free-associates when retrieval is thin |
| Refusal | Explicit escalation path to a human | Bot invents an answer rather than saying it does not know |
| Evaluation | Fixed question set scored on every change | Vibes-based testing, regressions ship silently |
Why RAG instead of fine-tuning or pasting your whole site into the prompt?
RAG wins because your content changes and a fine-tuned model does not. Retrieval lets you update a pricing page at 10am and have the bot correct at 10:05, with a link back to the source.
Fine-tuning teaches style and format, not facts. It also bakes knowledge in with no way to cite where an answer came from, which kills trust the moment a customer asks how you know.
Stuffing the entire site into a long context window is tempting now that context windows are large, and it does work for tiny corpora. It stops working on cost and latency: you pay for every token on every turn, and accuracy drops as irrelevant material crowds the answer. Retrieval keeps the context small and specific.
"Fine-tuning changes how the model talks. Retrieval changes what it knows. Almost every website chatbot problem is a knowledge problem."
How do you ingest and chunk your content?
Strip the page down to its main content, then split on heading boundaries into chunks of roughly 300 to 700 tokens with a small overlap. Every chunk carries its source URL and heading path so the answer can cite it.
Chunking is where most builds quietly go wrong. Fixed 1000-character splits cut sentences in half and separate a heading from the paragraph that explains it. Structure-aware splitting on H2 and H3 boundaries preserves the semantic unit a user is actually asking about.
Rules I apply to every ingestion pipeline:
- Drop navigation, footers, cookie banners and repeated CTAs before chunking. They pollute every embedding.
- Prepend the page title and heading path to each chunk body so a standalone chunk still has context.
- Store a content hash per chunk and only re-embed when the hash changes. Embedding is cheap, but not free at scale.
- Keep tables and FAQ pairs intact. Splitting a table row from its header makes it unretrievable.
- Record last-crawled timestamps so you can expire stale chunks instead of accumulating contradictions.
Which embedding model and vector store should you pick?
For most website corpora, an OpenAI text-embedding-3 model into pgvector on Postgres is the right default. You avoid running a second database, and you can filter by metadata with plain SQL in the same query.
OpenAI documents text-embedding-3-small at 1536 dimensions and text-embedding-3-large at 3072, and supports a dimensions parameter that shortens vectors to trade a little accuracy for storage and speed. That matters because the pgvector README notes HNSW and IVFFlat indexes cover up to 2,000 dimensions on the vector type, so a 3072-dimension vector needs shortening or a halfvec column to be indexable.
If you are already on Supabase, pgvector is one extension away. If you are calling models through OpenAI or Anthropic Claude, keep the embedding provider and the generation provider decoupled so you can swap either one without a re-index.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE doc_chunk (
id bigserial PRIMARY KEY,
url text NOT NULL,
heading text,
content text NOT NULL,
content_hash text NOT NULL,
embedding vector(1536) NOT NULL,
tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
crawled_at timestamptz NOT NULL DEFAULT now()
);
-- approximate nearest neighbour for semantic search
CREATE INDEX ON doc_chunk USING hnsw (embedding vector_cosine_ops);
-- keyword half of hybrid retrieval
CREATE INDEX ON doc_chunk USING gin (tsv);How do you retrieve the right chunks?
Run vector search and keyword search in parallel, merge the results, then rerank and keep only the top four to eight chunks. Pure semantic search misses exact terms like SKUs, error codes and plan names, which is exactly what customers type.
Hybrid beats vector-only
The cosine operator in pgvector finds conceptually similar text. Postgres full-text search finds literal matches. Fusing both rank lists catches the query where someone types an exact product code that the embedding treats as noise.
Smaller top-k, better answers
Retrieving twenty chunks feels safer and reads worse. Every marginal chunk adds tokens, latency and a chance the model latches onto something irrelevant. I start at top-k of five after reranking and only raise it when the eval set says recall is the bottleneck.
How do you ground answers and make the bot refuse?
Ground the model by giving it retrieved chunks with IDs and instructing it to answer only from those chunks, cite the ID it used, and return a refusal token when the context does not contain the answer. Refusal has to be a first-class output, not an accident.
On Synthicare, an NHS-compliant clinical decision support build, the refusal path mattered more than the answer path. The same principle holds on a commercial site: an honest "I do not have that, here is a human" costs you nothing, while a confident wrong answer about refunds costs you a chargeback.
You answer questions about ACME using ONLY the numbered context blocks below.
Rules:
1. If the answer is not fully contained in the context, reply exactly:
NEED_HUMAN
2. Never infer prices, dates, availability or policy from general knowledge.
3. Cite the block numbers you used, e.g. [2][5].
4. Quote figures verbatim. Do not round, convert or recalculate.
5. Keep answers under 120 words unless asked for detail.
CONTEXT:
[1] ...
[2] ...Then handle NEED_HUMAN in application code: swap it for a friendly message, offer a handoff, and log the question. That log becomes your content roadmap, because every refusal is a page you have not written yet. If handoff volume is high, the honest comparison is in AI chatbot versus live chat.
How do you build an eval set before you ship?
Write 50 to 150 real questions with known-correct answers, then score every prompt or retrieval change against them. Without this you are guessing, and every fix silently breaks something else.
The evaluation loop I run on every RAG chatbot:
- 1Pull real questions from support tickets, site search logs and sales call notes. Do not invent them; invented questions are always too easy.
- 2For each question record the expected answer and the chunk that should be retrieved. That gives you retrieval accuracy separately from answer accuracy.
- 3Add adversarial cases on purpose: questions your site genuinely cannot answer, competitor comparisons, and prompt-injection attempts.
- 4Score three metrics per run: retrieval hit rate, groundedness (is every claim in a retrieved chunk), and correct refusal rate on the unanswerable set.
- 5Re-run the whole set on every prompt edit, chunking change or model swap, and store results so regressions are visible in a diff.
A bot that refuses honestly whenever the context is thin is worth more than one that answers every question at some unmeasured accuracy. Refusals are visible, countable and fixable. Confident errors are none of those things, and you usually learn about them from an annoyed customer.
How do you deploy the widget without slowing the site?
Serve the chat as a small async script that lazy-loads the actual UI on first interaction, and keep all model calls server-side behind your own API. Never put a provider API key in the browser.
Deployment details that decide whether the thing survives contact with traffic:
- Stream tokens over SSE. Perceived latency, not total latency, is what users judge.
- Rate-limit per session and per IP, and cap tokens per conversation. Public chat endpoints attract abuse within days.
- Cache embeddings for repeated queries and cache full answers for your top questions.
- Log every turn with the retrieved chunk IDs so you can reconstruct why a bad answer happened.
- Ship a kill switch that falls back to a contact form if the model provider degrades.
For an ecommerce catalogue, wire retrieval to live inventory rather than crawled pages, since stock and price change faster than a crawler runs. I cover that pattern in AI for ecommerce teams.
How much does it cost to build an AI chatbot for your website?
Running cost scales with retrieved tokens multiplied by conversations, not with how big your corpus is. Storage and embedding are near-trivial for a normal site; generation dominates the bill.
Where the money actually goes in a RAG chatbot:
| Cost line | Scales with | Practical lever |
|---|---|---|
| Embedding | Chunks ingested, one time plus deltas | Hash-based re-embedding only on change |
| Vector storage | Chunk count and dimensions | Shorten dimensions, use halfvec |
| Retrieval | Queries per second | HNSW index, cached hot queries |
| Generation | Context tokens times turns times conversations | Smaller top-k, shorter system prompt, cheaper model for routing |
| Human handoff | Refusal rate | Write the missing content the refusal log reveals |
Provider prices change often, so check the current OpenAI and Anthropic pricing pages rather than trusting any number in a blog post, including mine. The structural point holds regardless: halving your context window roughly halves your input cost. Build effort is the larger line item on a first build, and it lands mostly in ingestion and evals, not in prompt writing.
If you are weighing a custom build against an off-the-shelf tool, start with choosing an AI chatbot for a small business. Build custom when your content is the product, your answers must be citable, or the bot has to touch your own systems. That is the work I do under AI engineering services, and the same retrieval spine sits under DaulatAI, where every answer has to trace back to a real data source.
Key takeaways
- Retrieval-augmented generation, not fine-tuning, is the correct architecture for a website chatbot that must stay current and citable.
- Chunk on heading boundaries at roughly 300 to 700 tokens and attach the source URL to every chunk so answers can cite them.
- pgvector inside your existing Postgres removes the need for a separate vector database for most site-sized corpora.
- Hybrid retrieval plus reranking with a small top-k beats vector-only search with a large top-k.
- Refusal must be an explicit, designed output with a human handoff, and refusal logs are your content roadmap.
- Write the eval set before the prompt, and score retrieval accuracy separately from answer accuracy.
Frequently asked questions
- How long does it take to build an AI chatbot for a website?
- A working prototype over a clean set of pages takes a day or two. A production chatbot with a real ingestion pipeline, hybrid retrieval, refusal handling, an eval set and a deployed widget typically takes two to six weeks, depending on how messy the source content is and how many systems it must reach.
- What is the difference between RAG and fine-tuning for chatbots?
- RAG retrieves relevant content at question time and feeds it to the model, so updates are instant and answers can cite sources. Fine-tuning adjusts model weights on examples, which is good for tone and output format but poor for facts, since knowledge is frozen and uncitable until you retrain.
- Can I use pgvector instead of a dedicated vector database?
- Yes, for most website corpora. pgvector adds vector columns and HNSW or IVFFlat indexes to Postgres, so you keep transactions, metadata filtering and joins in one database. Consider a dedicated vector store when you exceed tens of millions of chunks or need specialised sharding and filtering behaviour.
- How big should my chunks be for a RAG chatbot?
- Roughly 300 to 700 tokens, split at heading boundaries, with a small overlap. Smaller chunks retrieve precisely but lose context; larger chunks carry context but dilute the embedding. Prepend the page title and heading path to each chunk so it still makes sense in isolation.
- How do I stop my chatbot from hallucinating?
- Constrain it to retrieved context, require citations to chunk IDs, and give it an explicit refusal token to emit when the context is insufficient. Then measure groundedness on an eval set. Hallucination is mostly a retrieval failure: if the right chunk never arrives, no prompt will save the answer.
- Is it worth building a custom chatbot instead of buying one?
- Buy when your needs are generic FAQ deflection on a small site. Build when your content is the product, answers must cite sources, the bot needs to read your own systems, or you have compliance requirements around what it may say. Custom costs more upfront and gives you control over retrieval and refusal.
- Which embedding model should I use for a website chatbot?
- Start with OpenAI text-embedding-3-small at 1536 dimensions, which indexes cleanly in pgvector and is inexpensive. Move to text-embedding-3-large only if evals show retrieval is the bottleneck, and shorten its dimensions with the API dimensions parameter so the vector stays within index limits.
- How do I measure whether my website chatbot is working?
- Track four numbers: retrieval hit rate against your eval set, groundedness of generated answers, correct refusal rate on unanswerable questions, and human handoff rate in production. Business-side, measure deflected tickets and assisted conversions rather than raw conversation counts, which say nothing about quality.
Sources
Written by Syed Husnain Haider Bukhari
AI engineer, data scientist, and founder of Revolutionary Technologies LLC. Ships production AI agents, automations, and data platforms for teams in the US, UK, and UAE — including AgentFlow, AI Walay, and ProLeads.
Get in touch →Related pages
Want this built instead of researched?
Book a 30-minute scoping call. You get a one-page plan and a fixed-scope quote within 48 hours.
Start a conversation