Building RAG Pipelines with LangChain: A Developer's Guide

Most RAG tutorials stop at "embed your docs, retrieve the top chunks, stuff them into a prompt." That gets you a demo. It does not get you something a non-technical team will trust when the answer is wrong and there is no way to tell. At Cybelle Data I built French-language AI assistants on LangChain and OpenAI that let operational teams query internal databases and documentation in plain French — and for French-market clients I shipped pipelines that made years of internal docs and support history answerable in natural language, with source citations and configurable confidence thresholds. This is the guide I wish I had before that work.

I will walk through the pipeline the way I actually build it: loading and chunking, embeddings and retrieval, assembling the chain, augmenting the prompt, returning citations, and the guardrails and production concerns that decide whether anyone keeps using the thing after week one.

Loading and chunking: where answer quality is won or lost

Retrieval quality is capped by how you split your documents, not by how clever your prompt is. This is the single lever I spent the most time tuning on the French client work — the internal docs were a mess of PDFs, exported support tickets, and wiki pages, and naive splitting produced chunks that either cut a procedure in half or bundled three unrelated topics together.

Start with a recursive splitter that respects structure (paragraphs, then sentences) rather than blindly cutting at N characters:

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,
)

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_documents(docs)

On chunk size, the tradeoff is real and it goes both ways. Small chunks (300–500 tokens) retrieve precisely but the model loses context and answers thinly. Large chunks (1000+) carry context but dilute the embedding, so retrieval gets fuzzy and you pull in noise. For French support history I landed around 700–800 tokens with ~15% overlap. There is no universal number — I picked it by evaluating, not by guessing.

One thing that mattered more than chunk size: attaching metadata to every chunk — source file, page, last-modified date, document type. You need this later for citations and for filtering stale data. Do it at load time; retrofitting it is painful.

Embeddings and the vector store

Once chunks are clean, you embed them and store the vectors. LangChain keeps this boring, which is what you want:

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

emb = OpenAIEmbeddings(model="text-embedding-3-small")
store = Chroma.from_documents(
    chunks, emb, persist_directory="./index"
)
retriever = store.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 20},
)

Two practical notes from production. First, embedding model choice matters for non-English content — I validated retrieval quality specifically on French queries rather than trusting English benchmarks, because a model that looks great on English can underperform on French phrasing and accents. Second, I use MMR (maximal marginal relevance) rather than plain similarity so the five chunks I retrieve are not five near-duplicates of the same paragraph. Diversity in retrieval beats redundancy when a question touches more than one document.

Assembling the RAG chain

The chain is where retrieval, prompt, and model meet. With LangChain Expression Language it stays readable — retrieved context flows in on one side, the user question on the other:

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

prompt = ChatPromptTemplate.from_template(
    "Reponds uniquement avec le contexte.\n"
    "Contexte:\n{context}\n\n"
    "Question: {question}"
)
llm = ChatOpenAI(model="gpt-4o", temperature=0)
chain = prompt | llm

Temperature zero is deliberate. For an assistant that non-technical teams query about operational facts, creativity is a bug — I want the same question to return the same grounded answer every time.

Prompt augmentation and returning source citations

Stuffing raw text into {context} works, but it throws away the one thing that earns trust: where the answer came from. On the French-market pipelines, citations were not a nice-to-have — they were the reason people believed the output. So I format each retrieved chunk with its source metadata, and I ask the model to cite by tag.

def format_context(chunks):
    lines = []
    for i, c in enumerate(chunks):
        src = c.metadata.get("source", "?")
        lines.append(
            "[" + str(i + 1) + "] (" + src + ") "
            + c.page_content
        )
    return "\n\n".join(lines)

Then the answer comes back with [1], [2] markers, and I map those back to real filenames and pages in the UI. The effect is that when the assistant says something, the user can click through and verify it in the actual document. That single feature moved these tools from "interesting" to "used daily."

Guardrails: refusing to answer when retrieval is weak

The most important line of code in a production RAG system is the one that says "I do not know." A model asked a question with no relevant context will happily hallucinate a confident, wrong answer in fluent French. That is worse than silence, because it looks authoritative.

So I made confidence thresholds configurable and enforced them before ever calling the LLM. Check the retrieval score; if the best chunk is below threshold, refuse and say so:

hits = store.similarity_search_with_score(q, k=5)
best = min(score for _, score in hits)
if best > THRESHOLD:  # lower distance = closer
    return "Je n'ai pas trouve d'information fiable."
answer = chain.invoke({
    "context": format_context([h for h, _ in hits]),
    "question": q,
})

Making the threshold configurable per client mattered — a legal-docs assistant should refuse far more readily than an internal-FAQ bot. The refusal path, plus mandatory citations, is what let non-technical teams rely on these assistants without a developer in the loop.

Evaluation, and the production concerns nobody warns you about

You cannot tune chunk size, k, or thresholds by vibes. I built a small evaluation set of real questions with known-good source documents, then measured whether the right chunk was retrieved and whether the answer cited it. Every change to chunking or retrieval got re-run against that set. Without it, "improvements" are just guesses.

Beyond retrieval quality, three operational realities hit every pipeline I shipped:

• Stale data. Internal docs change. If you embed once and forget, the assistant confidently quotes last year's procedure. I keyed chunks by last-modified metadata and re-indexed changed documents on a schedule — freshness is a pipeline concern, not a one-time load.
• Latency. Embedding the query, retrieving, and a GPT-4o call add up. I kept the retrieval step tight (smaller k, MMR to avoid re-ranking noise) and reserved the expensive model for the generation step only.
• Cost. Every query is an embedding call plus a generation call. Caching frequent queries and using a smaller embedding model where quality allowed kept the bill sane at scale.

These same instincts carried into a related build: a multi-agent SQL audience-selection system orchestrated in n8n, combining GPT-4o, Gemini, and Perplexity to turn natural-language briefs into SQL. It compressed roughly three days of analyst work into under three minutes. RAG-style grounding and hard guardrails were the difference between a toy and something the business ran on.

Takeaways

• Chunking decides your ceiling — tune chunk_size and overlap against a real evaluation set, not intuition.
• Attach source metadata at load time; you need it for citations and for killing stale data.
• Citations are what earn trust from non-technical users — make the model cite, and link back to the real document.
• The refusal path is a feature. Enforce a configurable confidence threshold before calling the LLM.
• Treat freshness, latency, and cost as first-class pipeline concerns, not afterthoughts.

RAG is not hard to demo and genuinely hard to make trustworthy. The gap between the two is exactly the citations, the guardrails, and the evaluation loop — the unglamorous parts that decide whether anyone still uses it next month.