How to Build a RAG Chatbot with Dify

Upload your documents to Dify, configure retrieval-augmented generation, and deploy a chatbot that answers questions from your own knowledge base.

18 minIntermediate

Visit Dify

A RAG chatbot answers questions using your documents instead of relying on what the model memorized during training. Dify handles the chunking, embedding, and retrieval pipeline — you just upload files and flip a toggle. This guide builds a support chatbot that answers from a product documentation folder.

What you need

Dify running locally or on Dify Cloud
A set of documents to index (PDF, Markdown, or plain text)
An embedding model connected (OpenAI text-embedding-3-small or similar)
About 20 minutes

Step 1: Create a knowledge base

Go to Knowledge in the left sidebar and click Create Knowledge. Give it a name, then upload your files. Dify accepts PDF, TXT, Markdown, HTML, and CSV. For a product docs folder, drag the whole folder in — Dify processes each file automatically.

Step 2: Configure chunking

In the indexing settings, choose Automatic chunking for a first pass. Dify will split documents at natural paragraph breaks and keep chunks between 200 and 500 tokens. If your docs have long tables or code blocks, switch to Fixed chunk size and set 400 tokens with 50-token overlap.

Chunk size rule of thumb

Smaller chunks (200-300 tokens) improve retrieval precision but miss context that spans multiple paragraphs. Larger chunks (600-800 tokens) capture more context but dilute relevance. Start at 400 and tune from there.

Step 3: Enable reranking

In the retrieval settings, turn on Reranking and pick a reranker model. Reranking re-scores the top 20 retrieved chunks against your actual query, keeping only the most relevant 5 before sending them to the model. This single step cuts hallucinations more than any other setting change.

Step 4: Link the knowledge base to a chatbot

Create a new Chatbot app and open the Context panel on the right. Click Add and select the knowledge base you just created. Set Citation and Attribution to on — the chatbot will cite the exact document and page number for every answer.

Step 5: Write a retrieval-aware system prompt

Tell the model what to do when retrieval finds nothing relevant. Without this, it will make up an answer using its training data.

Dify - RAG system prompt

You

You are a support assistant. Answer only from the provided documentation. If the documentation does not contain the answer, say: 'I do not have that information in the docs. Please contact support.'

The fallback instruction prevents hallucination when retrieval misses.

Step 6: Test and publish

Debug the chatbot with questions you know are in the docs and a few that are not. Check that citations appear and that out-of-scope questions get the fallback reply. When both work, click Publish and copy the embed code or share link.