How to Build a RAG Knowledge Base in Dify

Create a Knowledge base in Dify, upload and chunk your documents, embed them, and attach retrieval to a Chatflow so the model answers from your own content.

16 minIntermediate

Visit Dify

RAG lets a model answer from your own documents instead of only its training data. In Dify you build this with a Knowledge base: you upload files, Dify splits them into chunks and embeds them, and then a Chatflow retrieves the most relevant chunks at question time and feeds them to the LLM. This guide takes you from an empty Knowledge base to a working chatbot grounded in your content.

What you need

Dify on Cloud or self-hosted
An embedding model available in Model Provider (for example an OpenAI or a local embedding model)
A few documents to index: PDFs, Markdown, text, or a website to crawl
About 16 minutes

Step 1: Create a Knowledge base and add documents

Open the Knowledge tab in Dify and click Create Knowledge. Choose your source: upload files, sync from Notion, or import a web page. Upload the documents you want the chatbot to answer from. Dify accepts common formats including PDF, Markdown, plain text, HTML, and CSV.

Dify - Knowledge base sources

Create Knowledge -------------------------

(*) Import from file

( ) Sync from Notion

( ) Sync from website

-------------------------------------------

handbook.pdf pricing.md faq.txt

Pick a source, then upload the files you want the model to answer from.

Step 2: Set chunking and embedding

Dify now asks how to split and index the documents. You can keep Automatic chunking to start, or switch to Custom to control the chunk size and overlap. Smaller chunks give more precise retrieval; larger chunks keep more context together. Pick your embedding model here too, since that is what turns each chunk into a vector for search.

Setting	What it controls	Guidance
Chunk size	How much text goes in each indexed piece	Smaller for FAQs, larger for prose and manuals
Overlap	Text shared between neighbouring chunks	A small overlap keeps sentences from being cut mid-idea
Embedding model	How chunks are turned into vectors	Use the same model for indexing and retrieval
Index mode	High quality (vector) or economical (keyword)	High quality retrieves by meaning, not just exact words

Retrieval setting matters as much as chunking

In the retrieval settings you choose vector search, full-text search, or hybrid. Hybrid combines semantic and keyword matching and is a safe default. You can also set the number of chunks to retrieve (top K) and a score threshold to drop weak matches.

Click Save and Process. Dify chunks every document, embeds the chunks, and stores them. Larger uploads take a moment; you can watch each document move to an indexed state.

Step 3: Test retrieval in the Knowledge base

Before wiring it into an app, use the Retrieval Testing tab inside the Knowledge base. Type a question and see which chunks come back and with what score. If the right passage is not near the top, adjust chunk size or switch to hybrid retrieval and reprocess. Getting this right here saves debugging later.

Step 4: Attach it to a Chatflow

Create a Chatflow app. On the canvas, add a Knowledge Retrieval node between the Start node and the LLM node, and select your Knowledge base in it. Wire the retrieved context into the LLM node's prompt so the model reads it. A prompt like Answer the question using only the context below. Context: {{#context#}} keeps the model grounded and reduces made-up answers.

Dify - Chatflow with retrieval

[ Start ] -> [ Knowledge Retrieval ] -> [ LLM ] -> [ Answer ]

----------------------------------------------------------------

Knowledge Retrieval: Company Handbook KB

LLM prompt: Answer only from context: {{#context#}}

The Knowledge Retrieval node feeds matched chunks into the LLM node.

Step 5: Test, then publish

Open Debug and Preview and ask a question you know the answer to from your documents. Check that the reply matches the source and, if you enabled citations, that it points at the right chunk. Tune the prompt and retrieval settings until answers are accurate, then click Publish. Your grounded chatbot is now reachable as a web app or over the API.

Keep the knowledge base fresh

When your source documents change, re-upload or re-sync them so the index reflects the latest content. A stale knowledge base is the most common reason a RAG bot gives outdated answers.

Result: a chatbot that answers from your own documents, built from a Knowledge base you can update any time, with retrieval you tested and tuned before shipping.