Agentic RAG in production: architecture, retrieval evaluation, guardrails and cost control

Retrieval-augmented generation (RAG) is now the default way to make an LLM answer from your data. The prototype takes an afternoon. Getting it to production quality, safely, at a predictable cost, takes engineering discipline. This guide walks through an agentic RAG architecture we'd be comfortable running for a real organisation, with code snippets you can adapt.

Simple RAG vs agentic RAG

Simple RAG is a pipeline: embed the question, fetch the top-k chunks, put them in the prompt, generate. It works well for direct factual questions over a clean document set.

Agentic RAG turns that pipeline into a bounded loop. The model (or a small controller) decides what to do next: answer directly, search, rewrite the query, call a tool, or ask for clarification, and it checks whether the evidence is good enough before answering.

Reference architecture

The components:

1. Input guardrail: screen for prompt injection, personal data and out-of-scope requests.
2. Planner: a small, cheap model call (or rules) that decides the route.
3. Hybrid retrieval with access control: keyword (BM25) plus vector search, filtered by the user's permissions in the query itself.
4. Re-ranker: reorders the top 20–50 candidates and keeps the best few (example: Vertex AI ranking API).
5. Evidence grader: decides whether the context is sufficient; if not, rewrite the query and retry, up to a small cap.
6. Generator: answers only from the context, with citations, and is allowed to say "I don't know".
7. Output guardrail: checks faithfulness, citations, PII and policy before the answer is shown.
8. Tracing: every step is logged with tokens, latency, cost and sources.

Ingestion and chunking

Retrieval quality is decided at ingestion time. Practical rules:

• Parse structure, not just text. Keep headings, table structure and page numbers. A chunk that knows it came from "Leave policy › Section 4.2" retrieves and cites far better.
• Attach metadata to each chunk: source, section path, updated date, and access-control groups.
• Try more than one strategy and let your evaluation decide.

from dataclasses import dataclass, field

@dataclass
class Chunk:
    text: str
    source: str
    section: str
    acl: list[str] = field(default_factory=list)   # groups allowed to read this chunk

def chunk_by_headings(doc_text: str, source: str, acl: list[str], max_chars=1800, overlap=200):
    """Split on markdown-style headings, then window long sections with overlap."""
    chunks, section, buf = [], "Intro", []
    for line in doc_text.splitlines():
        if line.startswith("#"):
            if buf:
                chunks += _window("\n".join(buf), source, section, acl, max_chars, overlap)
            section, buf = line.lstrip("# ").strip(), []
        else:
            buf.append(line)
    if buf:
        chunks += _window("\n".join(buf), source, section, acl, max_chars, overlap)
    return chunks

def _window(text, source, section, acl, max_chars, overlap):
    out, start = [], 0
    while start < len(text):
        piece = text[start:start + max_chars]
        out.append(Chunk(text=f"[{section}]\n{piece}", source=source, section=section, acl=acl))
        start += max_chars - overlap
    return out

Prefixing the section title into the chunk text is a cheap form of context enrichment. Anthropic describes a stronger version, adding a short model-generated context to each chunk, as contextual retrieval; measure whether it helps on your data before paying for it.

Retrieval: hybrid search and re-ranking

Pure vector search misses exact terms (product codes, section numbers, names). Pure keyword search misses paraphrases. Hybrid search combines both. A simple, robust way to merge the two result lists is reciprocal rank fusion (RRF):

def rrf_merge(keyword_hits, vector_hits, k=60, top_n=30):
    """Each *_hits is a list of chunk ids in rank order."""
    scores = {}
    for hits in (keyword_hits, vector_hits):
        for rank, cid in enumerate(hits):
            scores[cid] = scores.get(cid, 0.0) + 1.0 / (k + rank + 1)
    return [cid for cid, _ in sorted(scores.items(), key=lambda x: -x[1])[:top_n]]

Then re-rank the merged candidates and keep the best 4–8 for the prompt. Fewer, better chunks improve answers and cut cost.

Access control belongs in the retrieval query, never only in the prompt:

def retrieve(query, user_groups, search):
    # search() must apply the filter server-side; the model never sees forbidden chunks.
    return search(query, filter={"acl": {"any_of": user_groups}}, top_k=50)

Evaluating retrieval and generation

Build an evaluation set of 50–200 real questions with the passage(s) that answer each and a reference answer. Then measure two layers separately.

Retrieval metrics

def recall_at_k(results: list[list[str]], gold: list[set[str]], k=5):
    hits = sum(1 for r, g in zip(results, gold) if g & set(r[:k]))
    return hits / len(gold)

def mrr(results, gold):
    total = 0.0
    for r, g in zip(results, gold):
        for i, cid in enumerate(r, start=1):
            if cid in g:
                total += 1 / i
                break
    return total / len(gold)

If recall@5 is low, no prompt will save you. Fix parsing, chunking, hybrid weights or the re-ranker first.

Generation metrics

• Faithfulness / groundedness: is every claim supported by the retrieved context?
• Answer relevance: does it answer the question asked?
• Correctness: does it match the reference answer?
• Refusal quality: when the answer isn't in the documents, does it say so?

Libraries such as Ragas and managed services such as Vertex AI's Gen AI evaluation service provide model-graded versions of these metrics. Calibrate any model-graded metric against a sample of human judgements before you trust it.

The agent loop, with hard limits

Here is a framework-agnostic sketch of the bounded loop. llm, search and rerank are your own wrappers.

MAX_STEPS, MAX_TOKENS = 3, 12_000

def answer(question, user):
    trace, context, query = [], [], question
    for step in range(MAX_STEPS):
        hits = rerank(query, retrieve(query, user.groups, search))[:6]
        context += [h for h in hits if h not in context]
        verdict = llm.json(
            "Given the question and passages, reply as JSON: "
            '{"sufficient": bool, "rewrite": str}',
            question=question, passages=[h.text for h in context],
        )
        trace.append({"step": step, "query": query, "hits": [h.id for h in hits], "verdict": verdict})
        if verdict["sufficient"] or tokens_used(trace) > MAX_TOKENS:
            break
        query = verdict["rewrite"] or question
    draft = llm.generate(SYSTEM_ANSWER_ONLY_FROM_CONTEXT, question=question, passages=context)
    checked = output_guardrail(draft, context)
    log_trace(user, question, trace, checked)
    return checked if checked.ok else fallback_message()

Notice what is not here: no unlimited loop, no tool that can write or delete without approval, and no answer that skips the output check.

Guardrails that actually hold

Prompt instructions are helpful but not a security boundary. Put controls in code:

The OWASP Top 10 for LLM Applications is a good checklist for threat modelling. On Google Cloud, Model Armor can screen prompts and responses, and Gemini has configurable safety filters.

Cost control

1. Measure cost per request from day one (input tokens, output tokens, model, number of loop steps).
2. Route by difficulty: a small model for the planner and grader; a stronger model only for the final answer when needed.
3. Send fewer, better chunks: re-ranking to 4–8 chunks often beats stuffing 30.
4. Cache: reuse long, repeated context with context caching where supported, and cache answers to identical questions within the same permission group.
5. Cap the loop: most questions should finish in one step; alert if the average climbs.

Production checklist

• Evaluation set versioned; retrieval and answer metrics in CI with a minimum threshold
• ACL filtering enforced server-side and tested with "forbidden document" cases
• Input/output guardrails, plus red-team prompts in the test suite
• Step, token, time and rate limits
• Tracing with sources, latency and cost per request
• Feedback button wired into a review queue that feeds the evaluation set
• Runbook: re-indexing, key rotation, model version changes, rollback

Learn to build it, step by step

Our Applied AI Engineer Bootcamp covers this stack hands-on: Python LLM APIs, production RAG with ChromaDB, LangGraph multi-agent orchestration, evaluation, and FastAPI/Docker deployment on Cloud Run, taught live on Sat & Sun, 10:00–11:30 AM IST.