Conversational RAG: Teaching Your System to Remember
March 3, 2025
Standard RAG treats every question in isolation — and stumbles the moment a user asks a follow-up. Conversational RAG adds a memory layer that pulls the running dialogue into every retrieval call.
What It Is
When a user first asks about "Kubernetes" and then types "How does it scale?", the system has to know that "it" still refers to Kubernetes. Conversational RAG resolves these references by feeding the dialogue history into every retrieval step — a memory component keeps state across turns.
How It Works
The system maintains a rolling buffer of recent interactions — typically the last five to ten exchanges. When a new query arrives, a dedicated rewriting step combines the conversation history with the fresh question to produce a fully self-contained query. For instance, if the prior exchange was about an Enterprise subscription and the user now asks "Can you reset it?", the rewriter produces something like "Can you reset the Enterprise subscription API key?" This enriched query then feeds into the standard retrieval and generation pipeline.
Where It Delivers Value
Any application involving multi-turn dialogue benefits from this pattern. Customer support bots, SaaS help desks, and advisory systems all rely on users building context over multiple messages. Without memory, these interactions feel robotic and frustrating.
Strengths
The user experience improves dramatically. People can communicate naturally without repeating themselves. The conversation flows more like a human interaction, which increases satisfaction and resolution rates.
Limitations
Memory introduces its own challenges. Outdated context from earlier in the conversation can interfere with current queries — a phenomenon known as memory drift. The query rewriting step also adds token costs to every interaction, which compounds at scale.
The Bottom Line
If your users regularly ask follow-up questions, Conversational RAG is a natural next step after Standard RAG. If your workload consists mainly of one-shot queries, skip it — the overhead is not justified.
For a test you can try this chat.
Sources
Liu et al. — ChatQA: Surpassing GPT-4 on Conversational QA and RAG (2024)
Qu et al. — Open-Retrieval Conversational Question Answering (2020)