The RAG Audit Checklist: What to Evaluate Before Declaring a Retrieval Pipeline Production-Ready

Most RAG pipelines are declared production-ready based on demo performance. A clean corpus, curated questions, and a satisfying answer rate in a notebook create the impression that the system is ready. Then real users arrive with documents that change weekly, access permissions that vary by team, and questions the demo set never covered.

The audit checklist below covers what to validate before a retrieval pipeline serves actual users at actual scale. It is organized by domain: ingestion quality, retrieval accuracy, generation faithfulness, observability, security, and cost. Each domain has concrete pass/fail criteria. If a system cannot pass a domain, it is not ready for that domain.

This checklist is not a replacement for evaluation discipline built during development. It is the gate before the system exits that development phase.

Why Demo Performance Is a Poor Predictor

A RAG demo works for three reasons that do not hold in production.

First, the demo corpus is curated. Documents were selected because they answer the test questions well. Real enterprise corpora include contradictory versions, deprecated policies, and documents that have been superseded but never deleted.

Second, the demo questions are known in advance. The team asks the questions the system was built to answer. Real users ask adjacent questions, partial questions, and questions that require synthesizing across multiple documents the demo never combined.

Third, the demo has no operational history. There are no staleness issues, no access control edge cases, no cost spikes from unexpectedly long retrieved chunks, and no log to look at when something goes wrong.

Domain 1: Ingestion Quality

Ingestion quality problems are a common cause of retrieval failure, and they are hard to diagnose after the fact because they are invisible at query time.

Source freshness. Every source in the corpus needs a defined refresh cadence. SharePoint pages that change quarterly need a quarterly re-index. Internal policy documents that update monthly need a monthly crawl with delta detection. The ingestion pipeline should track document modification timestamps and flag sources that have exceeded their SLA without a re-index.

Chunk quality. Fixed-size chunking by token count is the default in most frameworks. It is also the most likely to produce chunks that cut across logical unit boundaries. A chunk that ends mid-sentence or splits a table header from its rows will retrieve correctly by keyword but fail to provide coherent context to the generation step.

The evaluation method for chunk quality is manual: sample chunks at random, read them, and assess whether each one is a self-contained unit of information. If a meaningful share require context from an adjacent chunk to be interpretable, the chunking strategy needs revision. Fixed-size chunking is a common source of retrieval confusion in otherwise well-built pipelines: teams invest time tuning embedding models while the actual problem is that their chunks cut across paragraph and table boundaries. See chunk strategy failures in production RAG for the specific failure patterns.

Delete handling. When a source document is deleted or access is revoked, the corresponding chunks need to be removed from the index. Pipelines that lack explicit delete handling will serve stale or unauthorized content indefinitely. The index and the source corpus should be compared on a regular schedule to identify orphaned embeddings.

Semantic coherence testing with Pydantic.

from pydantic import BaseModel, Field, model_validator
from typing import List
from enum import Enum

class ChunkQuality(str, Enum):
    PASS = "pass"
    FLAG_TRUNCATED = "flag_truncated"
    FLAG_NO_CONTEXT = "flag_no_context"
    FAIL = "fail"

class ChunkAuditResult(BaseModel):
    chunk_id: str
    source_doc_id: str
    token_count: int = Field(ge=1)
    starts_mid_sentence: bool
    ends_mid_sentence: bool
    contains_table: bool
    table_header_present: bool
    quality: ChunkQuality = ChunkQuality.PASS

    @model_validator(mode="after")
    def assess_quality(self) -> "ChunkAuditResult":
        if self.starts_mid_sentence or self.ends_mid_sentence:
            self.quality = ChunkQuality.FLAG_TRUNCATED
        if self.contains_table and not self.table_header_present:
            self.quality = ChunkQuality.FLAG_NO_CONTEXT
        return self

class RAGAuditReport(BaseModel):
    pipeline_id: str
    audit_date: str
    corpus_size_docs: int
    corpus_size_chunks: int
    ingestion_score: float = Field(ge=0.0, le=1.0)
    retrieval_recall_at_5: float = Field(ge=0.0, le=1.0)
    retrieval_mrr: float = Field(ge=0.0, le=1.0)
    generation_faithfulness: float = Field(ge=0.0, le=1.0)
    citation_accuracy: float = Field(ge=0.0, le=1.0)
    observability_coverage: float = Field(ge=0.0, le=1.0)
    acl_enforcement_verified: bool
    pii_detection_active: bool
    cost_per_query_p95_usd: float
    cost_ceiling_enforced: bool
    chunk_audit_results: List[ChunkAuditResult] = []
    approved_thresholds: dict[str, float] = Field(default_factory=dict)

    @property
    def production_ready(self) -> bool:
        thresholds = {
            "ingestion_score": self.approved_thresholds.get("ingestion_score", 0.0),
            "retrieval_recall_at_5": self.approved_thresholds.get("retrieval_recall_at_5", 0.0),
            "retrieval_mrr": self.approved_thresholds.get("retrieval_mrr", 0.0),
            "generation_faithfulness": self.approved_thresholds.get("generation_faithfulness", 0.0),
            "citation_accuracy": self.approved_thresholds.get("citation_accuracy", 0.0),
            "observability_coverage": self.approved_thresholds.get("observability_coverage", 0.0),
        }
        return (
            self.ingestion_score >= thresholds["ingestion_score"]
            and self.retrieval_recall_at_5 >= thresholds["retrieval_recall_at_5"]
            and self.retrieval_mrr >= thresholds["retrieval_mrr"]
            and self.generation_faithfulness >= thresholds["generation_faithfulness"]
            and self.citation_accuracy >= thresholds["citation_accuracy"]
            and self.observability_coverage >= thresholds["observability_coverage"]
            and self.acl_enforcement_verified
            and self.cost_ceiling_enforced
        )

    @property
    def blocking_failures(self) -> List[str]:
        failures = []
        thresholds = self.approved_thresholds
        for metric_name in (
            "ingestion_score",
            "retrieval_recall_at_5",
            "retrieval_mrr",
            "generation_faithfulness",
            "citation_accuracy",
            "observability_coverage",
        ):
            threshold = thresholds.get(metric_name)
            if threshold is not None and getattr(self, metric_name) < threshold:
                failures.append(
                    f"{metric_name} {getattr(self, metric_name):.2f} below approved threshold"
                )
        if not self.acl_enforcement_verified:
            failures.append("ACL enforcement not verified — security gate blocked")
        if not self.cost_ceiling_enforced:
            failures.append("No cost ceiling configured — cost control gate blocked")
        return failures

This model makes the audit report machine-readable and catches threshold violations before manual review.

Domain 2: Retrieval Accuracy

Retrieval accuracy is the ratio of queries where the relevant document appears in the retrieved set. It is the most important metric to have before production and the most commonly absent one.

Building the evaluation set. An evaluation set for retrieval consists of (question, relevant_document_id) pairs. The minimum viable set should cover the major topic areas, query types, and known failure modes in the corpus. RAGAS and LlamaIndex both provide tools to generate synthetic evaluation sets from an existing corpus, which reduces the need to write every question manually.

Recall@K. Recall@K measures whether the relevant document appears in the retrieved results. The threshold should be set from the domain’s risk tolerance and validated baseline. Below that approved floor, users will encounter retrieval misses frequently enough to erode trust in the system.

MRR (Mean Reciprocal Rank). MRR measures where in the retrieved list the relevant document appears. A system can meet its Recall@K target while still burying the relevant document late in the list, which reduces the quality of context passed to the generation step.

Hybrid retrieval. Dense vector retrieval alone performs poorly on exact-match queries (product codes, policy numbers, named entities). Hybrid retrieval — combining dense embeddings with BM25 keyword search — consistently outperforms pure dense retrieval on enterprise corpora. If the pipeline uses only dense retrieval, run the evaluation set with and without BM25 to measure the gap before deciding whether to add it.

The field’s fixation on embedding model selection misses the point. In many enterprise RAG failures, the embedding model is not the bottleneck — the retrieval architecture is. Evaluate retrieval architecture before optimizing model choice.

Domain 3: Generation Faithfulness

Generation faithfulness measures whether the output is grounded in the retrieved context. It is distinct from factual accuracy: a response can be faithful to the retrieved context and still be wrong if the retrieved context itself is incorrect.

Faithfulness scoring. RAGAS provides a faithfulness metric that decomposes the generated answer into individual claims and checks whether each claim can be attributed to a retrieved passage. Set the production threshold from the risk profile of the domain. Below that approved floor, the system is generating content that cannot be traced to what was retrieved, which is a hallucination signal.

Citation accuracy. If the system surfaces citations (document names, page numbers, URLs), verify that the cited source actually contains the claim attributed to it. Citation fabrication — where the model generates a plausible-looking citation to a real document that does not contain the claim — is a failure mode that is invisible without explicit verification. Sample cited responses and verify them manually or with a second LLM call that reads the cited passage.

Answer relevance. A faithful answer may still fail to address the question asked. Answer relevance measures whether the response addresses the query rather than summarizing retrieved content that is topically adjacent but not directly responsive. RAGAS includes an answer relevance metric alongside faithfulness.

Context utilization. If retrieval returns five chunks and the generation step references only one, the retrieval budget is being wasted and the answer may be missing important context. Log the percentage of retrieved chunks that are referenced in the generated response. Consistent low utilization may indicate that retrieval is returning too many marginally relevant chunks.

Domain 4: Observability

A pipeline with no observability is a black box. When something breaks, the team has no structured way to determine whether the failure originated in ingestion, retrieval, generation, or infrastructure.

Per-query tracing. Every query should produce a trace that links: the original query string, the retrieved document IDs and their similarity scores, the prompt sent to the LLM, the generated response, and any user feedback signals. This trace is the minimum required to investigate a quality complaint.

Metrics to instrument from day one:

• Retrieval hit rate — percentage of queries where at least one retrieved chunk has similarity score above a defined threshold. A declining hit rate often signals corpus staleness.
• Fallback rate — percentage of queries where the system returns a “I could not find relevant information” response rather than an answer. Sustained high fallback rates indicate corpus gaps.
• P95 latency — tail query latency. Retrieval and generation latency should be tracked separately to identify which component is the bottleneck.
• Hallucination rate — percentage of responses flagged by the faithfulness checker. This requires running the faithfulness check inline or in a post-processing sampling loop.
• User feedback signals — thumbs up/down, explicit corrections, or repeat query patterns (a user who asks the same question three times in a session is likely signaling that the first two responses were inadequate).

Alert thresholds. Define alert conditions before launch, not after the first incident. A material retrieval hit rate drop, tail-latency increase above a defined ceiling, or hallucination rate spike should all trigger automated alerts. The specific thresholds depend on the application, but the pattern of monitoring these signals is broadly useful.

For a related discussion on broader AI system audit discipline, see 5 Signs Your AI System Needs a Production Audit.

Domain 5: Security

Security in a RAG pipeline has three distinct concerns: access control, PII handling, and prompt injection.

Access control. In multi-user enterprise deployments, retrieval must respect the access permissions of the querying user. If user A is not authorized to view document X, document X should not appear in user A’s retrieved context — regardless of how semantically relevant it is to the query.

ACL enforcement must operate at retrieval time, not as a post-processing filter. Filtering after semantic search wastes retrieval budget on unauthorized results and, in edge cases, leaks information through system behavior even if the content is withheld. The correct pattern is to include user-specific permission metadata in the vector index and filter on that metadata before ranking by similarity score. See the detailed treatment of ACL propagation in enterprise RAG contexts for implementation specifics.

PII detection. If the indexed corpus contains customer records, employee data, medical information, or other PII categories, the pipeline needs a PII detection step at ingestion time. This step identifies PII-bearing chunks, applies appropriate handling (redaction, restricted-access tagging, or exclusion from the index), and maintains an audit log. Running PII detection at query time alone is insufficient because retrieved content reaches the LLM context window before any post-hoc filter can act.

Prompt injection. A RAG pipeline is a prompt injection surface because the retrieved content is concatenated directly into the LLM prompt. A malicious document in the corpus — or a malicious query — can attempt to override system instructions. The mitigation is a prompt injection test suite that checks whether adversarial content in retrieved chunks or queries can alter system behavior. Rebuff and Garak both provide injection testing utilities for this purpose.

Domain 6: Cost

Per-query cost is a production constraint, not an afterthought. A pipeline that looks inexpensive at pilot volume can become expensive at production volume if context windows, reranking, or faithfulness checks expand without a ceiling.

Model the cost components separately. A RAG query has four distinct cost components: embedding the query (typically fractions of a cent), vector search (infrastructure cost, not token-based), generating the LLM response (token cost based on retrieved context length plus prompt overhead), and any post-processing LLM calls (faithfulness checking, reranking with a cross-encoder). Most cost models lump these together. Separating them identifies which component offers the most optimization potential.

Context window economics. Retrieving more chunks means passing more tokens to the generation step. Use the current pricing sheet for the deployed model to calculate context cost per query. Retrieval strategies that return fewer, higher-precision chunks (via reranking or hybrid retrieval with tighter K values) reduce generation cost while maintaining or improving answer quality.

Cost ceiling enforcement. Define a maximum acceptable cost per query before launch. Enforce it at the infrastructure layer by capping context length, limiting K, or routing to a lower-cost model when the query appears simple. A cost ceiling ensures that no single query pattern can spike the infrastructure bill unexpectedly.

Embedding cost at scale. For large corpora, the upfront embedding cost is significant. Re-embedding the full corpus on every model change or schema update is a budget item that needs to be modeled in advance using current provider pricing. Delta-embedding — re-embedding only changed or new documents — reduces ongoing embedding cost but requires accurate change detection in the ingestion pipeline.

For the engineering mechanics of keeping embeddings synchronized with document changes over time, see Building Durable RAG Pipelines with Temporal Ingestion, Embedding, and Index Management.

The Full Audit Checklist

• Ingestion freshness verified: Every source document has a defined refresh cadence; the ingestion pipeline tracks modification timestamps and flags sources that have exceeded their SLA without a re-index.
• Chunk quality sampled: A random sample of chunks has been reviewed for semantic coherence; the share requiring adjacent chunk context stays below the approved threshold; table headers are co-located with table content.
• Delete handling confirmed: The pipeline has an explicit process for removing chunks from the index when source documents are deleted or access is revoked; orphaned embeddings are identified and purged on a regular schedule.
• Retrieval evaluation set built and versioned: Representative question-document pairs cover the major topic areas in the corpus; Recall@K and MRR meet approved thresholds; the evaluation set is versioned alongside the index schema.
• Generation faithfulness measured: Faithfulness meets the approved threshold on a representative sample; citation accuracy is verified on sampled responses; hallucination rate is tracked as a named metric with an alert threshold.
• Observability layer complete: Per-query traces log retrieved document IDs, similarity scores, prompt, response, and user feedback; alerts are defined for retrieval hit rate drops, P95 latency increases, and hallucination rate spikes.
• ACL enforcement verified at retrieval time: User-specific permission metadata is enforced before similarity ranking, not as a post-retrieval filter; ACL test cases covering cross-user access are passing.
• PII handling policy implemented: PII detection runs at ingestion; PII-bearing chunks are redacted, restricted, or excluded from the index; an audit log of PII handling decisions is maintained.
• Prompt injection test suite passing: Adversarial content in retrieved chunks and queries cannot override system instructions; the test suite is part of the CI/CD pipeline and runs on every ingestion schema change.
• Per-query cost modeled by component: Embedding, vector search, generation, and post-processing costs are tracked separately; per-query cost is within budget at P95 query volume; a cost ceiling is enforced at the infrastructure layer.

What Happens Without the Audit

The failure mode for pipelines that skip this checklist is not immediate and dramatic. It is gradual and hard to diagnose.

Users notice that answers are sometimes wrong but cannot tell when. Trust in the system erodes faster than the team can identify the cause. The team adds prompt engineering patches that fix one failure class while creating another. Costs rise because the context window grows to compensate for retrieval imprecision. And when a security incident occurs — a user receiving content they should not have seen — there are no traces to reconstruct how it happened.

The audit checklist exists because production readiness cannot be inferred from demo quality. It must be established through measurement across each domain before the system serves users who depend on it.

For teams whose pipeline has already reached production without this evaluation layer in place, the same checklist applies as a retrospective audit. The only difference is that gaps need to be addressed in a running system rather than before launch — which is harder, but not unusual.

The Decision Rule

Do not declare a RAG pipeline production-ready because the demo answers look good. Declare it ready only when ingestion freshness, retrieval quality, generation faithfulness, observability, access control, and cost ceilings are all measured against approved gates.