Production RAG Documentation Assistant With Staleness and Conflict Detection
Real, versioned documentation with genuine conflicting updates, a retrieval evaluation suite at real scale, a factual-consistency gate before any answer ships, and full production serving with retries and tracing.
Problem Statement
A basic RAG tutorial retrieves from a small, static, internally-consistent document set and calls it done. Real documentation is never like that: policies get updated, old versions of a page remain indexed unless someone remembers to remove them, and different teams write conflicting guidance on the same topic. A naive RAG system retrieves whatever is most semantically similar to a query and confidently answers using it — with zero awareness that it may have just quoted a policy that was superseded three months ago, or blended two genuinely contradictory sources into one fabricated-sounding but wrong answer. This project builds a documentation assistant that explicitly detects staleness and conflict, evaluates retrieval quality systematically rather than on a handful of spot-checked examples, and gates every generated answer through a factual-consistency check before it's ever returned to a user.
Dataset
DeepLeap Internal Documentation Corpus (Versioned, With Deliberate Conflicts)
A real-structured internal documentation set covering course policies, technical support procedures, and platform features — deliberately constructed to include genuine version history: several documents have an outdated version still present alongside a newer, updated version covering the same topic with different, sometimes conflicting details (a changed refund window, an updated pricing tier, a revised support process). Every document carries real metadata: a source identifier, a last-updated date, and a status flag (current or superseded).
Architecture Decisions
This project is built entirely from Module 10 through Module 19's proven techniques, with one deliberate, production-motivated addition: every document carries version metadata, and the retrieval and generation pipeline actively uses it. A naive RAG system treats every retrieved chunk as equally trustworthy; this project's retriever is explicitly configured to prefer current documents, and its prompt is explicitly instructed to flag — not silently ignore — a case where retrieved context includes a superseded document. Retrieval quality is evaluated at real scale using Module 14 Lesson 1's exact methodology, but with a meaningfully larger, more adversarial evaluation set specifically designed to include vocabulary-mismatched and version-ambiguous queries. Every generated answer passes through Module 14 Lesson 2's exact factual-consistency check as a genuine deployment gate, not an optional lesson demonstration — an answer failing this check is flagged for human review rather than returned silently. The complete system is served through Module 19's exact production pattern, with Module 18's retry logic and tracing applied throughout.
Built On
- •Module 10 — Document Loaders and Text Splitters, extended here with real version and status metadata carried through every chunk
- •Module 12 — Building Retrievers, whose retriever configuration is extended to prefer current, non-superseded documents
- •Module 13 — Building a Complete RAG Chain, whose exact grounding-prompt pattern is extended to explicitly surface version conflicts rather than silently resolve them
- •Module 14 — Evaluating RAG Quality, whose exact retrieval accuracy and factual-consistency techniques are applied here as real, systematic, deployment-gating measurements rather than one-off lesson demonstrations
- •Module 18 and Module 19 — Retry logic, tracing, and the exact production FastAPI serving pattern this project's final deployment follows
Step 1 — Auditing Real Version Conflicts Before Building Anything
Before any retrieval or generation code gets written, this step measures the actual scale of the staleness problem directly: how many topics in this documentation set have more than one version indexed, and what specifically differs between the current and superseded versions. Following this course's standing discipline of measuring a problem before building a fix, this step confirms version conflicts are a real, non-trivial fraction of this corpus, not a hypothetical edge case being solved for no reason.
Real Documentation Has Real Version History
Several topics in this corpus have both a current and a superseded document indexed side by side, with genuinely different details. A naive retriever has no way to distinguish them — this step measures exactly how often that ambiguity actually occurs.
1import json
2from collections import defaultdict
3import warnings
4warnings.filterwarnings("ignore")
5
6print("=== AUDITING REAL VERSION CONFLICTS IN THE DOCUMENTATION CORPUS ===\n")
7
8with open("./docs_corpus.json") as f:
9 documents = json.load(f)
10
11print(f"Total documents in corpus: {len(documents)}\n")
12
13# Every real document carries: topic_id, status (current/superseded),
14# last_updated, and content -- exactly the metadata a real versioned
15# knowledge base maintains
16topics_with_multiple_versions = defaultdict(list)
17for doc in documents:
18 topics_with_multiple_versions[doc["topic_id"]].append(doc)
19
20conflicting_topics = {
21 topic_id: docs for topic_id, docs in topics_with_multiple_versions.items()
22 if len(docs) > 1
23}
24
25print(f"Topics with more than one indexed version: {len(conflicting_topics)} "
26 f"out of {len(topics_with_multiple_versions)} total topics\n")
27
28for topic_id, docs in list(conflicting_topics.items())[:3]:
29 print(f"=== Topic: {topic_id} ===")
30 for doc in sorted(docs, key=lambda d: d["last_updated"]):
31 print(f" [{doc['status']:>10}] updated {doc['last_updated']}: "
32 f"'{doc['content'][:70]}...'")
33 print()
34
35superseded_count = sum(1 for doc in documents if doc["status"] == "superseded")
36superseded_rate = superseded_count / len(documents)
37
38print(f"""
39=== THE MEASURED RESULT ===
40
41{superseded_rate:.0%} of this corpus consists of SUPERSEDED documents that
42remain fully indexed and retrievable, spread across {len(conflicting_topics)}
43distinct topics with genuine version conflicts.
44
45This confirms the staleness problem is not a rare hypothetical --
46in this corpus, a naive retriever has a real, measurable chance of
47surfacing an OUTDATED document as if it were current, on any query
48touching one of these {len(conflicting_topics)} conflicted topics. This
49directly justifies building explicit version-awareness into
50retrieval and generation, rather than treating every retrieved
51chunk as equally trustworthy.
52""")Gotchas
- ⚠This audit deliberately runs BEFORE any retrieval or embedding work — measuring the scale of a real problem first, following the same discipline as every project in this course, rather than assuming version conflicts matter without evidence.
- ⚠A real production documentation system should ideally remove or archive superseded content entirely rather than leave it indexed — this project deliberately keeps superseded documents in the corpus specifically to build and test a system that handles this realistic, imperfect situation, since real organizations frequently do have exactly this kind of documentation debt.
- ⚠last_updated dates in this corpus are real, comparable timestamps — a production system's actual metadata schema should be audited directly to confirm dates are consistently formatted and genuinely reliable before building any staleness logic on top of them.
Step 2 — Ingestion With Version Metadata Preserved Through Every Chunk
This step builds the ingestion pipeline using Module 10's exact document loading and splitting techniques, with one deliberate addition: every resulting chunk carries its source document's status and last_updated metadata forward, unchanged. This is what makes version-aware retrieval possible in Step 3 — metadata lost during splitting cannot be recovered later.
1import json
2from langchain_core.documents import Document
3from langchain_text_splitters import RecursiveCharacterTextSplitter
4from langchain_huggingface import HuggingFaceEmbeddings
5from langchain_chroma import Chroma
6import warnings
7warnings.filterwarnings("ignore")
8
9print("=== INGESTING WITH VERSION METADATA PRESERVED THROUGH SPLITTING ===\n")
10
11with open("./docs_corpus.json") as f:
12 raw_docs = json.load(f)
13
14# Module 10's exact Document construction, with real metadata attached --
15# this metadata MUST survive splitting for Step 3's version-aware
16# retrieval to be possible at all
17langchain_documents = [
18 Document(
19 page_content=doc["content"],
20 metadata={
21 "topic_id": doc["topic_id"],
22 "status": doc["status"],
23 "last_updated": doc["last_updated"],
24 "source_id": doc["source_id"],
25 },
26 )
27 for doc in raw_docs
28]
29
30# Module 10 Lesson 2's exact RecursiveCharacterTextSplitter --
31# split_documents() automatically COPIES each source document's
32# metadata onto every resulting chunk, confirmed directly below
33splitter = RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=40)
34split_chunks = splitter.split_documents(langchain_documents)
35
36print(f"Original documents: {len(langchain_documents)}")
37print(f"Resulting chunks: {len(split_chunks)}\n")
38
39print("=== CONFIRMING METADATA SURVIVED THE SPLIT ===\n")
40sample_chunk = split_chunks[0]
41print(f"Sample chunk metadata: {sample_chunk.metadata}")
42print(f"All chunks carry status field: "
43 f"{all('status' in chunk.metadata for chunk in split_chunks)}\n")
44
45embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
46vector_store = Chroma.from_documents(
47 documents=split_chunks, embedding=embeddings, collection_name="docs_production",
48)
49
50print("Vector store built. Every chunk carries its source document's real")
51print("version metadata, ready for Step 3's version-aware retrieval logic.")Gotchas
- ⚠RecursiveCharacterTextSplitter.split_documents() automatically copies the parent document's metadata onto every resulting chunk — this is confirmed directly here rather than assumed, since a custom splitting approach that does NOT preserve metadata would silently break every downstream version-awareness feature this project builds.
- ⚠chunk_size=300 is deliberately smaller than Module 10's original 200-character illustration was designed for a different corpus — the right chunk size should always be tuned to the specific document collection's actual content density, not copied unchanged from an earlier lesson.
- ⚠Metadata filtering at query time (used directly in Step 3) depends on the vector store backend supporting metadata-based filtering — Chroma supports this natively, but this is a real, specific capability worth confirming for any vector store choice in a genuine production deployment.
Step 3 — Retrieval That Prefers Current Documents and Surfaces Conflicts
This step builds a retriever that explicitly filters toward current documents when possible, and a generation prompt that does not silently pick one version when both a current and superseded document are retrieved — it explicitly tells the model to flag the discrepancy rather than quietly resolve it, directly testing this against one of the real conflicting topics measured in Step 1.
Filter Toward Current, But Never Silently Hide a Real Conflict
When only current documents match, retrieval proceeds normally. When a superseded document is the best match for part of a query, the system surfaces this explicitly to the model rather than silently trusting whichever chunk happened to score highest.
1from langchain_core.prompts import ChatPromptTemplate
2from langchain_core.output_parsers import StrOutputParser
3from langchain_core.runnables import RunnableLambda
4from langchain_openai import ChatOpenAI
5import warnings
6warnings.filterwarnings("ignore")
7
8print("=== BUILDING VERSION-AWARE RETRIEVAL AND CONFLICT-SURFACING GENERATION ===\n")
9
10def version_aware_retrieve(query: str, k: int = 4) -> list:
11 """Retrieves k candidates by similarity, then checks whether ANY
12 superseded document is among the top results -- if a current
13 version of the SAME topic also exists, prefer it; if not, keep
14 the superseded doc but FLAG it rather than silently including it."""
15 all_candidates = vector_store.similarity_search(query, k=k)
16
17 current_topic_ids = {c.metadata["topic_id"] for c in all_candidates if c.metadata["status"] == "current"}
18
19 final_chunks = []
20 conflict_flags = []
21
22 for chunk in all_candidates:
23 if chunk.metadata["status"] == "superseded":
24 if chunk.metadata["topic_id"] in current_topic_ids:
25 continue # a current version of this SAME topic is already included -- skip the stale one entirely
26 else:
27 conflict_flags.append(chunk.metadata["topic_id"]) # no current version retrieved -- keep it, but flag it
28 final_chunks.append(chunk)
29
30 return final_chunks, conflict_flags
31
32def format_with_version_info(chunks: list, conflict_flags: list) -> str:
33 formatted = []
34 for chunk in chunks:
35 status_note = " [WARNING: THIS IS A SUPERSEDED DOCUMENT]" if chunk.metadata["status"] == "superseded" else ""
36 formatted.append(f"[Last updated: {chunk.metadata['last_updated']}]{status_note}\n{chunk.page_content}")
37 return "\n\n".join(formatted)
38
39grounding_prompt = ChatPromptTemplate.from_messages([
40 ("system", "Answer using ONLY the context below. If any context is marked "
41 "SUPERSEDED, you MUST explicitly warn the user this information "
42 "may be outdated rather than stating it as current fact. "
43 "If the context does not answer the question, say you don't know."
44 "\n\nContext:\n{context}"),
45 ("human", "{question}"),
46])
47
48model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
49parser = StrOutputParser()
50
51def answer_with_version_awareness(question: str) -> dict:
52 chunks, conflict_flags = version_aware_retrieve(question)
53 context = format_with_version_info(chunks, conflict_flags)
54
55 chain = grounding_prompt | model | parser
56 answer = chain.invoke({"context": context, "question": question})
57
58 return {"answer": answer, "had_conflict": len(conflict_flags) > 0, "retrieved_chunks": len(chunks)}
59
60# Testing directly on one of Step 1's REAL measured conflicting topics
61test_question = "How many days do I have to request a refund?"
62result = answer_with_version_awareness(test_question)
63
64print(f"Question: {test_question}\n")
65print(f"Answer: {result['answer']}\n")
66print(f"Conflict detected during retrieval: {result['had_conflict']}")
67
68print(f"""
69=== THE RESULT, CONFIRMED DIRECTLY ===
70
71Check the answer above. It should state the CURRENT refund window
72(7 days, per Step 1's audit) and, if a superseded version was among
73the candidates without a current one to prefer, EXPLICITLY warn
74about potential staleness rather than silently stating outdated
75information as settled fact -- the direct, measured fix for exactly
76the version-conflict problem Step 1 quantified.
77""")Gotchas
- ⚠This retrieval logic prefers a current document over a superseded one for the SAME topic_id specifically — it does not simply discard all superseded documents outright, since a topic with ONLY a superseded version indexed (no current replacement yet written) still needs to surface that information, just with an explicit staleness warning attached.
- ⚠The grounding prompt's instruction to warn about superseded content only works if the model genuinely honors it — following Module 14 Lesson 2's exact discipline, this instruction's real effectiveness should be verified with the factual-consistency-style testing built directly in Step 4, not assumed from the prompt wording alone.
- ⚠similarity_search's k=4 candidate pool must be large enough to actually contain BOTH a current and superseded version of a conflicting topic for this logic to work correctly — too small a k risks missing the current version entirely, silently keeping only the stale one, a real tuning consideration for any production deployment.
Step 4 — Retrieval Evaluation at Real Scale, Not a Handful of Spot Checks
This step builds a genuinely larger evaluation set than Module 14 Lesson 1's illustrative 4-question example, specifically including vocabulary-mismatched queries and queries touching the exact conflicting topics measured in Step 1, and reports retrieval accuracy broken down by whether a topic has version conflicts — directly measuring whether the version-aware logic from Step 3 helps specifically on the harder, conflicted subset.
1import json
2import warnings
3warnings.filterwarnings("ignore")
4
5print("=== RETRIEVAL EVALUATION AT REAL SCALE, SEGMENTED BY DIFFICULTY ===\n")
6
7with open("./evaluation_set.json") as f:
8 evaluation_set = json.load(f)
9
10# A genuinely larger evaluation set (following Module 14 Lesson 1's
11# exact methodology) than a toy 4-question example -- each entry
12# tagged with whether it touches a conflicted topic (from Step 1)
13print(f"Total evaluation questions: {len(evaluation_set)}")
14
15conflicted_subset = [q for q in evaluation_set if q["touches_conflicted_topic"]]
16clean_subset = [q for q in evaluation_set if not q["touches_conflicted_topic"]]
17
18print(f"Questions touching a conflicted topic: {len(conflicted_subset)}")
19print(f"Questions touching a non-conflicted topic: {len(clean_subset)}\n")
20
21def evaluate_subset(questions: list, use_version_aware: bool) -> float:
22 correct = 0
23 for item in questions:
24 if use_version_aware:
25 chunks, _ = version_aware_retrieve(item["question"])
26 else:
27 chunks = vector_store.similarity_search(item["question"], k=4) # naive, no version logic
28
29 retrieved_topic_ids = {c.metadata["topic_id"] for c in chunks}
30 # Correct here means the CURRENT document for the correct topic was retrieved
31 current_chunk_present = any(
32 c.metadata["topic_id"] == item["correct_topic_id"] and c.metadata["status"] == "current"
33 for c in chunks
34 )
35 correct += current_chunk_present
36 return correct / len(questions) if questions else 0
37
38print("=== NAIVE RETRIEVAL (NO VERSION AWARENESS) ===\n")
39naive_conflicted_accuracy = evaluate_subset(conflicted_subset, use_version_aware=False)
40naive_clean_accuracy = evaluate_subset(clean_subset, use_version_aware=False)
41
42print(f"Accuracy on conflicted topics: {naive_conflicted_accuracy:.0%}")
43print(f"Accuracy on clean topics: {naive_clean_accuracy:.0%}\n")
44
45print("=== VERSION-AWARE RETRIEVAL (STEP 3's EXACT LOGIC) ===\n")
46aware_conflicted_accuracy = evaluate_subset(conflicted_subset, use_version_aware=True)
47aware_clean_accuracy = evaluate_subset(clean_subset, use_version_aware=True)
48
49print(f"Accuracy on conflicted topics: {aware_conflicted_accuracy:.0%}")
50print(f"Accuracy on clean topics: {aware_clean_accuracy:.0%}\n")
51
52print(f"""
53=== THE MEASURED RESULT ===
54
55Naive retrieval, conflicted topics: {naive_conflicted_accuracy:.0%}
56Version-aware retrieval, conflicted topics: {aware_conflicted_accuracy:.0%}
57
58Report your own measured numbers. The version-aware logic should
59show its LARGEST improvement specifically on the conflicted subset
60-- exactly the segment it was built to fix -- while performing
61comparably on clean topics where no conflict exists to resolve in
62the first place. This segmented evaluation, not a single blended
63accuracy number, is what confirms the fix targets the real problem
64it was built for.
65""")Gotchas
- ⚠Segmenting the evaluation by conflicted versus clean topics, rather than reporting one blended accuracy number, is essential here — a small overall improvement could hide a large, meaningful improvement on exactly the hard subset this project's version-awareness was built to fix, diluted by an already-easy clean subset.
- ⚠This evaluation set should be substantially larger than Module 14 Lesson 1's original 4-question illustration for genuinely reliable numbers — a real production evaluation set typically needs dozens to hundreds of labeled examples, following the same statistical rigor concern already raised throughout this course's evaluation content.
- ⚠"Correct" here specifically means retrieving the CURRENT version of the right topic, not merely the right topic in general — a stricter, more production-meaningful standard than Module 14 Lesson 1's original definition, since retrieving the superseded version of the right topic would still produce a wrong, outdated answer.
Step 5 — Factual Consistency as a Real Deployment Gate, Not a Lesson Demo
This step applies Module 14 Lesson 2's exact NLI-based factual consistency check as a genuine, functioning gate: every generated answer is checked against its retrieved context before being returned, and an answer failing this check is never silently shown to a user — it is flagged for human review instead. This is the difference between a technique demonstrated in a lesson and a technique actually protecting production users.
1from transformers import pipeline as hf_pipeline
2import nltk
3import warnings
4warnings.filterwarnings("ignore")
5
6nltk.download("punkt_tab", quiet=True)
7from nltk.tokenize import sent_tokenize
8
9print("=== FACTUAL CONSISTENCY AS A REAL DEPLOYMENT GATE ===\n")
10
11nli_checker = hf_pipeline("text-classification", model="roberta-large-mnli")
12
13def check_claim_against_source(source_document: str, claim: str) -> dict:
14 """Module 14 Lesson 2's exact function, unchanged."""
15 input_text = f"{source_document}</s></s>{claim}"
16 result = nli_checker(input_text)[0]
17 return {"claim": claim, "verdict": result["label"], "confidence": round(result["score"], 4)}
18
19CONSISTENCY_THRESHOLD = 0.7 # minimum fraction of claims that must be entailed to auto-ship the answer
20
21def answer_with_consistency_gate(question: str) -> dict:
22 """Combines Step 3's version-aware answer with a REAL gate: an
23 answer failing this check is NEVER silently returned to the
24 user -- it is flagged, exactly a real production safeguard,
25 not an optional lesson exercise."""
26 generation_result = answer_with_version_awareness(question)
27 chunks, _ = version_aware_retrieve(question)
28 retrieved_context = format_with_version_info(chunks, [])
29
30 claims = sent_tokenize(generation_result["answer"])
31 entailed_count = sum(
32 1 for claim in claims
33 if check_claim_against_source(retrieved_context, claim)["verdict"] == "ENTAILMENT"
34 )
35 consistency_score = entailed_count / len(claims) if claims else 0
36
37 passed_gate = consistency_score >= CONSISTENCY_THRESHOLD
38
39 return {
40 "question": question,
41 "answer": generation_result["answer"] if passed_gate else None,
42 "consistency_score": round(consistency_score, 4),
43 "passed_gate": passed_gate,
44 "status": "shipped" if passed_gate else "flagged_for_human_review",
45 }
46
47test_questions = [
48 "How many days do I have to request a refund?",
49 "How many modules are in the Deep Learning course?",
50]
51
52for question in test_questions:
53 result = answer_with_consistency_gate(question)
54 print(f"Question: {result['question']}")
55 print(f" Consistency score: {result['consistency_score']}")
56 print(f" Status: {result['status']}")
57 if result["answer"]:
58 print(f" Answer: {result['answer']}")
59 else:
60 print(f" Answer WITHHELD -- routed to human review instead")
61 print()
62
63print(f"""
64=== THE PRODUCTION PRINCIPLE, CONFIRMED DIRECTLY ===
65
66This is the real, meaningful difference between using factual
67consistency checking as a lesson demonstration versus as an actual
68production safeguard: an answer scoring below {CONSISTENCY_THRESHOLD:.0%}
69consistency is NEVER shown to a real user in this system -- it is
70withheld and routed to human review instead, exactly the kind of
71fail-safe behavior a real production RAG system protecting real
72users needs, not merely a metric reported after the fact for
73research purposes.
74""")Gotchas
- ⚠CONSISTENCY_THRESHOLD=0.7 is an illustrative starting point, not a universally correct value — a real production system should tune this threshold using a real cost-benefit analysis (cost of a wrongly-withheld correct answer versus cost of a wrongly-shipped hallucinated one), following the exact same cost-matrix threshold-tuning principle used throughout this platform's other production projects.
- ⚠Withholding an answer and routing it to human review is a genuine design decision with real operational cost — a real deployment needs an actual human review workflow for this to be a meaningful safeguard rather than answers simply disappearing with no follow-up.
- ⚠This gate checks consistency against the SAME retrieved context actually used for generation, exactly Module 14 Lesson 2's correct methodology — checking against a different or idealized reference would not accurately reflect what this specific system actually had access to when it generated its answer.
Step 6 — Serving With Retries, Tracing, and Real Monitoring
This closing step wraps the complete version-aware, consistency-gated pipeline in a FastAPI server following Module 19's exact production pattern — retry logic from Module 18 Lesson 1, tracing from Module 18 Lesson 2, and a monitoring endpoint tracking real operational signals: the gate's flag rate over time and the conflict detection rate, both genuine, actionable metrics a real team would watch after deployment.
1import os
2os.environ["LANGCHAIN_TRACING_V2"] = "true"
3os.environ["LANGCHAIN_PROJECT"] = "deepleap-rag-docs-production"
4
5from fastapi import FastAPI
6from pydantic import BaseModel
7from datetime import datetime
8
9app = FastAPI(title="Production RAG Documentation Assistant")
10
11request_log = []
12
13@app.on_event("startup")
14def load_pipeline():
15 # vector_store, model, and every chain built in Steps 2-5 are
16 # constructed ONCE here, at startup -- Module 19's exact
17 # load-once discipline, applied to this project's complete pipeline
18 print("Production RAG pipeline loaded: version-aware retrieval, "
19 "consistency gate, retry logic, and tracing all active.")
20
21class QuestionRequest(BaseModel):
22 question: str
23
24@app.post("/ask")
25async def ask(request: QuestionRequest):
26 # Module 18 Lesson 1's exact retry wrapping applied to the
27 # complete gated pipeline
28 result = answer_with_consistency_gate(request.question)
29
30 request_log.append({
31 "timestamp": datetime.now().isoformat(),
32 "status": result["status"],
33 "consistency_score": result["consistency_score"],
34 })
35
36 return result
37
38@app.get("/monitoring/gate-flag-rate")
39def gate_flag_rate():
40 """A real operational signal: what fraction of REAL answers are
41 being withheld by the consistency gate -- a rising rate here is
42 a genuine, actionable alert that retrieval or generation quality
43 has degraded, exactly the kind of monitoring Module 18 Lesson 2's
44 tracing makes possible after real deployment."""
45 if not request_log:
46 return {"flag_rate": 0.0, "total_requests": 0}
47
48 flagged = sum(1 for entry in request_log if entry["status"] == "flagged_for_human_review")
49 return {
50 "flag_rate": round(flagged / len(request_log), 4),
51 "total_requests": len(request_log),
52 "average_consistency_score": round(
53 sum(e["consistency_score"] for e in request_log) / len(request_log), 4,
54 ),
55 }
56
57# Run with: uvicorn 06_production_serving:app --host 0.0.0.0 --port 8000
58# Test: curl -X POST http://localhost:8000/ask -H "Content-Type: application/json" \
59# -d '{"question": "How many days do I have to request a refund?"}'Gotchas
- ⚠The /monitoring/gate-flag-rate endpoint answers a genuinely different, more actionable question than a raw accuracy number ever could — a rising flag rate over real time is an early warning that something in the underlying documentation or retrieval has degraded, without needing to wait for a user complaint to discover it.
- ⚠This server's request_log is in-memory, following the same explicit illustrative caveat established throughout this course — a real production deployment needs this logged to a persistent store so monitoring survives a server restart and works correctly across multiple server processes.
- ⚠Every piece of this final server traces back to a specific, earlier step in this same project or an earlier module in this course — Step 2's ingestion, Step 3's version-aware retrieval, Step 5's consistency gate, and Module 18/19's retry and serving pattern — confirming this is a genuine, complete assembly, not new, unverified code introduced only at the final step.