Skip to content
dhii.ai

RAG

The Production RAG Checklist: 25 Things to Get Right Before Launch

A practical checklist for building retrieval-augmented generation systems that work in production — ingestion, chunking, retrieval, grounding, permissions, evaluation and operations.

By Amit Valecha · · 6 min read

Retrieval-augmented generation is the most common pattern for putting language models to work on an organisation’s own knowledge. It is also the pattern where the gap between a demo and a production system is widest. A prototype that answers ten hand-picked questions well can still fail a third of real users’ questions.

This checklist collects what we verify before a RAG system goes live. It’s grouped by stage, because problems at each stage show up differently.

Ingestion

1. Inventory sources and owners. List every source — drives, wikis, ticketing systems, databases — with an owner who can answer “is this current?” Content without an owner tends to rot.

2. Exclude what shouldn’t be there. Drafts, duplicates, superseded versions and personal files degrade answers. Deciding what not to index is as important as connecting sources.

3. Parse with structure. PDFs, slides and spreadsheets lose meaning when flattened to plain text. Preserve headings, tables and reading order. Use OCR for scanned pages and check its quality on your worst documents.

4. Capture metadata. Title, source, URL, author, last modified date, document type, product, region and — critically — access permissions. Metadata powers filtering, citations and freshness.

5. Handle updates and deletions. Sync incrementally. When a document changes, re-index it; when it’s deleted or access is revoked, remove it. Stale chunks are a common source of wrong answers.

Chunking

6. Chunk by document type. Policies split naturally by section; FAQs by question; tickets by conversation; code by function. One fixed chunk size for everything is rarely optimal.

7. Keep context with each chunk. A chunk saying “this does not apply to contractors” is useless without knowing which policy it belongs to. Prepend document title and section headings, or generate a short contextual summary per chunk.

8. Size for your queries. Too small and chunks lack context; too large and they dilute relevance and waste tokens. Test a few sizes against your evaluation questions rather than guessing.

Retrieval

9. Use hybrid search. Vector search captures meaning; keyword search catches exact terms like product codes, error numbers and names. Combine them — most production systems benefit.

10. Re-rank. Retrieve broadly (say, the top 30–50 candidates), then use a re-ranking model to select the best few. This often improves answer quality more than changing the main language model.

11. Filter before searching. Use metadata to restrict search to the right product, region, language or time period when the query or user context makes it clear.

12. Choose embeddings deliberately. Test a few embedding models on your own data, including multilingual needs. Domain vocabulary matters; general benchmarks don’t tell the whole story.

13. Handle conversational queries. Follow-up questions like “what about for part-timers?” must be rewritten into standalone queries using conversation history before retrieval.

14. Measure retrieval separately. For each evaluation question, check whether the passage containing the answer was retrieved. If it wasn’t, no prompt will fix the answer.

Grounding and generation

15. Instruct the model to answer from context. Make it explicit that answers must be based on the provided passages and that missing information should be stated, not invented.

16. Cite sources. Show which passages support each answer, with links. Citations let users verify and build trust; they also make errors easy to diagnose.

17. Design the refusal. When evidence is missing or contradictory, the assistant should say so helpfully: “I couldn’t find this in the HR policies. The closest document is X, or contact Y.” A good refusal is better than a confident guess.

18. Handle conflicting sources. When two documents disagree, prefer the most recent or the authoritative source, and consider showing both with dates.

19. Control length and format. Users generally want a direct answer first, then details. Define the format, and keep answers concise.

Security and permissions

20. Enforce access control at retrieval. Filter results by the user’s permissions before anything reaches the model. Never rely on the model to withhold information it has already seen.

21. Protect against injection. Indexed documents may contain text that looks like instructions. Delimit retrieved content clearly and don’t give the assistant high-risk tools in the same context without additional controls.

22. Decide on logging and retention. Questions and answers can contain sensitive information. Decide what’s logged, who can see it and how long it’s kept.

Evaluation and operations

23. Build an evaluation set from real questions. Collect them from support tickets, chat logs, search logs and interviews. Include hard ones, ambiguous ones and questions the system should refuse. Aim for hundreds, not tens.

24. Score multiple dimensions. Retrieval recall, answer correctness, faithfulness to sources and appropriate refusal. Use automated scoring calibrated against human judgement, and re-run on every change.

25. Close the feedback loop. Capture thumbs up/down and unanswered questions. Review weekly. Many fixes will be content fixes — a missing page, an outdated policy — rather than technical ones. Assign someone to own that review.

Operations after launch

A RAG system is a living service. After launch, plan for a few ongoing responsibilities:

  • Content ownership. Each source has a named owner who fixes or retires content that produces bad answers.
  • Index health. Monitor sync jobs, document counts per source and parsing failures, with alerts when a connector stops updating.
  • Quality reviews. A short weekly review of negative feedback and unanswered questions, with each issue tagged as a content, retrieval or generation problem.
  • Cost and latency. Track tokens per answer and response times, especially after adding sources or changing retrieval depth.
  • Model updates. Re-run the evaluation set before switching embedding or generation models; changing the embedding model also means re-indexing.

Teams that assign these responsibilities explicitly see quality improve month over month. Teams that don’t usually see it decay as content drifts.

A note on scope

It is tempting to index everything on day one. Resist it. A focused assistant over one or two well-maintained sources, answering the questions a specific team actually asks, will outperform a sprawling index of every drive in the company — and it will earn the trust you need to expand. Add sources one at a time, re-running the evaluation set after each, so you can see whether the new content helped or simply added noise.

The most common failure modes

If you remember nothing else, remember these patterns, which account for most RAG disappointments we’re asked to fix:

  • The answer wasn’t retrieved. Usually a parsing, chunking or search-strategy problem. Measure retrieval first.
  • Stale or duplicate content. The system faithfully quotes a superseded document.
  • Missing context in chunks. The right paragraph was retrieved but without its heading, so the model misapplied it.
  • Permission leaks. Content from restricted sources reachable by everyone because access control wasn’t carried into the index.
  • No evaluation. Nobody can say whether last week’s change helped or hurt.

Final thought

RAG looks simple: search, then ask the model. In production, it’s a search engineering problem, a content management problem and an evaluation problem wrapped around a language model. Teams that treat it that way build assistants people rely on. Use this checklist as a pre-launch review: go through each item with the engineer, the content owner and someone who will actually use the assistant, and write down which items are done, which are deliberately skipped and why.

If you’re planning a knowledge assistant, our RAG service applies this checklist from day one, and our RAG cost guide explains typical effort.

Services related to this article

Keep reading

Start a project

Have a workflow that should run itself?

Tell us what you want automated or built. You get a written assessment of feasibility, risks and a fixed quote — usually within two business days.