Resilient Tool-Using Research Assistant With Source Conflict Reconciliation
A genuine multi-tool research agent that survives real tool failures, reconciles conflicting sources instead of picking one silently, and reports its own confidence honestly.
Problem Statement
A basic tool-using agent tutorial assumes every tool call succeeds and every source it retrieves is trustworthy and current. Neither assumption survives real usage: a real search or lookup tool times out or returns an error, and when an agent successfully gathers information from multiple genuinely different sources on the same question, those sources sometimes disagree — an outdated statistic from one, a more current figure from another. A naive agent either crashes on the first tool failure, or silently picks whichever source it happened to process first and presents it with unwarranted confidence. This project builds a genuinely resilient research agent: one that gracefully handles individual tool failures without failing the entire request, explicitly reconciles disagreement between multiple real sources rather than silently choosing one, and reports calibrated confidence rather than uniform certainty regardless of how solid its underlying evidence actually is.
Dataset
Simulated Multi-Source Research Environment (Deliberately Unreliable)
A set of fictional but realistic research tools — a web search simulator, a statistics database lookup, and a news archive lookup — deliberately engineered to fail intermittently (simulating real timeouts and rate limits) and to occasionally return genuinely conflicting information on the same query across different tools, exactly the kind of unreliable, disagreeing multi-source environment a real research agent operates in.
Architecture Decisions
This project builds directly on Module 15 and Module 16's proven tool-calling and agent mechanics, with two deliberate, production-motivated additions neither module covered. First, every tool call is wrapped with Module 18 Lesson 1's exact retry logic AND a genuine circuit-breaker fallback: if a specific tool fails even after retries, the agent is explicitly informed the tool is unavailable and instructed to proceed using only its remaining working tools rather than the entire request failing. Second, when the agent gathers information from multiple tools touching the same question and those tools disagree, the agent is explicitly prompted to surface the disagreement and state which source it is favoring and why, rather than silently synthesizing one answer that erases the conflict. A final confidence-scoring step, built from the same NLI entailment mechanism used throughout this course, checks whether the agent's final answer is actually well-supported by its gathered tool results, producing a genuine, calibrated confidence signal rather than uniform false certainty.
Built On
- •Module 15 — Tool Calling and Function Binding, whose exact tool definition and execution mechanism this project's three research tools are built on
- •Module 16 — Building a Simple Tool-Using Agent, whose exact create_agent pattern this project's repeating loop is built from
- •Module 18 — Error Handling, Retries, and Basic Observability, whose exact retry logic is extended here with a genuine graceful-degradation fallback for a tool that fails even after retrying
- •Module 14 Lesson 2 — The exact NLI-based factual consistency mechanism, reused here to produce a genuine, calibrated confidence score for the agent's final synthesized answer
Step 1 — Measuring Real Tool Failure Rates Before Building Resilience
Before building any fallback logic, this step measures the actual, simulated failure rate of each research tool directly, confirming tool unreliability is a real, quantified problem this project needs to solve, not an assumed concern. This mirrors the exact measure-before-fix discipline applied to every real production failure mode across this platform's projects.
Not Every Tool Call Succeeds — Measured, Not Assumed
Each of the three research tools has a real, measured chance of failing on any given call, simulating genuine rate limits and timeouts. This step quantifies exactly how often, directly motivating the resilience built in later steps.
1import random
2import warnings
3warnings.filterwarnings("ignore")
4
5print("=== MEASURING SIMULATED TOOL FAILURE RATES DIRECTLY ===\n")
6
7random.seed(42)
8
9# Each tool has a REAL, deliberately configured failure probability,
10# simulating genuine rate limits and timeouts a production research
11# tool actually experiences
12TOOL_FAILURE_RATES = {
13 "web_search_tool": 0.15,
14 "stats_database_tool": 0.08,
15 "news_archive_tool": 0.20,
16}
17
18def simulate_tool_call(tool_name: str) -> bool:
19 """Returns True if the call succeeds, False if it fails --
20 following the tool's real, configured failure rate."""
21 return random.random() > TOOL_FAILURE_RATES[tool_name]
22
23N_TRIALS = 500
24
25print(f"Running {N_TRIALS} simulated calls per tool to measure real failure rates:\n")
26
27for tool_name, configured_rate in TOOL_FAILURE_RATES.items():
28 failures = sum(1 for _ in range(N_TRIALS) if not simulate_tool_call(tool_name))
29 measured_rate = failures / N_TRIALS
30 print(f" {tool_name:>20}: configured failure rate = {configured_rate:.0%}, "
31 f"measured over {N_TRIALS} trials = {measured_rate:.1%}")
32
33print(f"""
34=== THE MEASURED RESULT ===
35
36Every tool has a REAL, non-trivial chance of failing on any given
37call. If a research agent needs all THREE tools to answer a single
38complex question, and each has an independent failure chance, the
39probability that AT LEAST ONE tool fails during that single request
40is meaningfully higher than any single tool's own failure rate --
41confirmed directly:
42""")
43
44combined_success_rate = 1
45for rate in TOOL_FAILURE_RATES.values():
46 combined_success_rate *= (1 - rate)
47combined_failure_rate = 1 - combined_success_rate
48
49print(f"Probability ALL THREE tools succeed on a single request: {combined_success_rate:.1%}")
50print(f"Probability AT LEAST ONE tool fails: {combined_failure_rate:.1%}")
51
52print(f"""
53This confirms directly: a research agent using multiple tools per
54request will hit AT LEAST one tool failure on roughly {combined_failure_rate:.0%}
55of real requests requiring all three tools. An agent with no
56graceful degradation would fail this same fraction of ALL requests
57completely -- a real, measured, unacceptable production failure
58rate this project's remaining steps directly address.
59""")Gotchas
- ⚠This lesson's failure rates are deliberately configured and simulated for reproducible measurement — a real production tool's actual failure rate should be measured directly from real logged call history, following the exact same measure-don't-assume principle applied here to a simulated environment.
- ⚠The combined failure probability calculation assumes independent tool failures — in a real system, correlated failures (a shared network issue affecting multiple tools simultaneously) could make the real combined failure rate higher than this independent-probability calculation suggests, a genuine, honest limitation of this simplified model.
- ⚠random.seed(42) makes this specific measurement reproducible across runs for this lesson's illustration — a real production measurement would aggregate real failure data over real time, not a single seeded simulation run.
Step 2 — Tools With Retry Logic and Genuine Graceful Degradation
This step wraps each research tool with Module 18 Lesson 1's exact retry logic, then adds a genuinely new capability beyond that module: when a tool fails even after retrying, the agent is explicitly told this tool is currently unavailable, rather than the whole request crashing — directly tested by forcing one tool to fail completely and confirming the agent still produces a partial, honest answer using its remaining tools.
1from langchain_core.tools import tool
2from langchain_core.runnables import RunnableLambda
3import random
4import warnings
5warnings.filterwarnings("ignore")
6
7print("=== BUILDING TOOLS WITH RETRY LOGIC AND GENUINE GRACEFUL DEGRADATION ===\n")
8
9# A fictional but internally consistent knowledge base for these
10# simulated tools, INCLUDING deliberate cross-tool disagreement on
11# one specific fact (Step 3 tests this directly)
12FICTIONAL_FACTS = {
13 "global_ai_market_size": {
14 "web_search_tool": "$450 billion as of the most recent industry report",
15 "stats_database_tool": "$390 billion, per the latest verified quarterly database update",
16 },
17}
18
19def make_resilient_tool(tool_name: str, failure_rate: float, lookup_fn):
20 """Wraps a tool function with Module 18 Lesson 1's exact retry
21 logic, PLUS a genuine fallback: if it fails even after retries,
22 return an explicit UNAVAILABLE marker rather than raising an
23 exception that would crash the entire agent request."""
24
25 attempt_tracker = {"count": 0}
26
27 def flaky_lookup(query: str) -> str:
28 attempt_tracker["count"] += 1
29 if random.random() < failure_rate:
30 raise ConnectionError(f"{tool_name} temporarily unavailable")
31 return lookup_fn(query)
32
33 retried = RunnableLambda(flaky_lookup).with_retry(
34 stop_after_attempt=3, wait_exponential_jitter=True,
35 )
36
37 def safe_call(query: str) -> str:
38 try:
39 return retried.invoke(query)
40 except ConnectionError:
41 # THE genuinely new piece beyond Module 18: explicit,
42 # honest degradation instead of a crashed request
43 return f"[{tool_name} is currently UNAVAILABLE after retries -- proceed using other available sources]"
44
45 return safe_call
46
47def web_search_lookup(query: str) -> str:
48 if "ai market" in query.lower():
49 return f"Web search result: {FICTIONAL_FACTS['global_ai_market_size']['web_search_tool']}"
50 return "No relevant web results found."
51
52def stats_database_lookup(query: str) -> str:
53 if "ai market" in query.lower():
54 return f"Stats database result: {FICTIONAL_FACTS['global_ai_market_size']['stats_database_tool']}"
55 return "No matching database entry found."
56
57resilient_web_search = make_resilient_tool("web_search_tool", 0.15, web_search_lookup)
58resilient_stats_lookup = make_resilient_tool("stats_database_tool", 0.08, stats_database_lookup)
59
60print("=== TESTING GRACEFUL DEGRADATION DIRECTLY: FORCING A TOOL TO ALWAYS FAIL ===\n")
61
62# Forcing web_search_tool to a 100% failure rate to directly confirm
63# the fallback path, rather than relying on random chance to hit it
64always_failing_search = make_resilient_tool("web_search_tool", 1.0, web_search_lookup)
65
66result = always_failing_search("What is the global AI market size?")
67print(f"Result from a tool forced to fail every time: {result}\n")
68
69print(f"""
70=== THE RESULT, CONFIRMED DIRECTLY ===
71
72Even with a tool GUARANTEED to fail every single attempt (forced to
73a 100% failure rate specifically to test this path directly), the
74result is a clear, explicit "unavailable" message, NOT a raised
75exception that would crash the calling agent. This is the genuine,
76new capability this project adds beyond Module 18's retry-only
77pattern: real, honest degradation, letting an agent using this tool
78proceed with whatever OTHER sources remain available, rather than
79the entire request failing because of one struggling tool.
80""")Gotchas
- ⚠This step's always_failing_search test deliberately forces a 100% failure rate rather than relying on the random chance from Step 1's normal failure rates — this is a genuine, deliberate controlled test of the fallback path, following this course's standing discipline of directly testing a specific failure case rather than only observing it occasionally in random runs.
- ⚠The explicit '[tool unavailable]' string returned on fallback is deliberately worded to be genuinely informative to the AGENT reading it, not just a generic error — the agent's own reasoning in Step 3 depends on understanding which specific source is missing to make an honest decision about how to proceed.
- ⚠stop_after_attempt=3 still applies before the fallback triggers — a tool experiencing a brief, genuinely transient failure still gets Module 18 Lesson 1's real retry benefit; the fallback is specifically for a tool that fails even after those retries are exhausted.
Step 3 — An Agent That Surfaces Disagreement Instead of Hiding It
This step builds the complete agent using Module 16's exact create_agent pattern with both resilient tools bound, and tests it directly on the deliberately conflicting fact from Step 2 — confirming the agent's system prompt genuinely produces an answer that surfaces the disagreement between sources, rather than silently picking one number and presenting it with unwarranted, false confidence.
Two Real Sources, One Real Disagreement, Surfaced Not Hidden
When the web search and stats database tools return genuinely different figures for the same fact, a naive agent picks one silently. This agent's prompt explicitly requires it to report the disagreement and state its reasoning for favoring one source.
1from langchain.agents import create_agent
2from langchain_core.tools import tool
3from langchain_core.messages import HumanMessage, SystemMessage
4from langchain_openai import ChatOpenAI
5import warnings
6warnings.filterwarnings("ignore")
7
8print("=== BUILDING THE AGENT WITH EXPLICIT CONFLICT-SURFACING INSTRUCTIONS ===\n")
9
10@tool
11def web_search_tool(query: str) -> str:
12 """Searches the web for current information on a topic."""
13 return resilient_web_search(query)
14
15@tool
16def stats_database_tool(query: str) -> str:
17 """Looks up verified statistics from a curated database."""
18 return resilient_stats_lookup(query)
19
20model = ChatOpenAI(model="gpt-4o-mini", temperature=0)
21
22# The system prompt EXPLICITLY forbids silently resolving a conflict --
23# this is the genuinely new instruction beyond Module 16's basic agent
24system_instruction = SystemMessage(content=(
25 "You are a careful research assistant. When you use multiple tools "
26 "and they return DIFFERENT information on the same fact, you MUST "
27 "explicitly report BOTH values and the discrepancy between them in "
28 "your final answer, along with which source you consider more "
29 "reliable and why. NEVER silently pick one value and present it as "
30 "the sole, uncontested fact. If a tool reports itself as "
31 "UNAVAILABLE, explicitly note in your answer that you were unable "
32 "to check that source."
33))
34
35agent = create_agent(model, tools=[web_search_tool, stats_database_tool])
36
37conflicting_question = "What is the current global AI market size?"
38
39print(f"Question: {conflicting_question}\n")
40
41result = agent.invoke({
42 "messages": [system_instruction, HumanMessage(content=conflicting_question)],
43})
44
45final_answer = result["messages"][-1].content
46print(f"Agent's final answer:\n{final_answer}\n")
47
48both_figures_mentioned = "450" in final_answer and "390" in final_answer
49
50print(f"""
51=== THE RESULT, CONFIRMED DIRECTLY ===
52
53Both conflicting figures mentioned in the final answer: {both_figures_mentioned}
54
55Check the answer above directly. A correctly-behaving agent under
56this explicit instruction should mention BOTH the $450 billion and
57$390 billion figures, note they come from different sources, and
58state a reasoned preference -- NOT silently report only one number
59as if no disagreement existed. This is a genuine, measurable test
60of whether the conflict-surfacing instruction actually changes
61behavior, not merely an assumption that adding the instruction works.
62""")Gotchas
- ⚠This system instruction's real effectiveness must be verified directly, exactly as this lesson's both_figures_mentioned check does — following the same honest standard Module 13 Lesson 2 applied to grounding instructions, an instruction added to a prompt is not automatically obeyed just because it was written, and should always be tested against a real case designed to trigger it.
- ⚠The agent's tool-calling loop, built from Module 16's exact create_agent pattern, must actually call BOTH tools for this conflict to even be detectable — if the model judged only one tool necessary and never called the second, no conflict would ever surface, regardless of the system instruction's wording; a real system should log which tools were actually invoked to distinguish these two different failure modes.
- ⚠This lesson's single conflicting fact is deliberately constructed for clear, direct testability — a real production research agent would need this same conflict-surfacing behavior verified across many genuinely different kinds of factual disagreement, not just one hand-built example.
Step 4 — Calibrated Confidence Instead of Uniform False Certainty
This step reuses Module 14 Lesson 2's exact NLI entailment mechanism to build a genuine confidence score for the agent's final answer, checking each claim in the answer against the actual tool results gathered — directly comparing this calibrated score across a well-supported answer and a partially-degraded one (where one tool failed) to confirm the score genuinely reflects the real evidence quality, not a fixed, uniform confidence regardless of circumstances.
1from transformers import pipeline as hf_pipeline
2import nltk
3import warnings
4warnings.filterwarnings("ignore")
5
6nltk.download("punkt_tab", quiet=True)
7from nltk.tokenize import sent_tokenize
8
9print("=== BUILDING A GENUINE, CALIBRATED CONFIDENCE SCORE ===\n")
10
11nli_checker = hf_pipeline("text-classification", model="roberta-large-mnli")
12
13def check_claim_against_source(source_document: str, claim: str) -> dict:
14 """Module 14 Lesson 2's exact function, reused unchanged here to
15 score an AGENT's answer instead of a RAG chain's answer."""
16 input_text = f"{source_document}</s></s>{claim}"
17 result = nli_checker(input_text)[0]
18 return {"claim": claim, "verdict": result["label"], "confidence": round(result["score"], 4)}
19
20def score_agent_answer(answer: str, gathered_tool_results: str) -> float:
21 """Checks every claim in the agent's final answer against what
22 its tools ACTUALLY returned -- a genuine, calibrated confidence
23 signal, not a fixed value regardless of how solid the underlying
24 evidence really was."""
25 claims = sent_tokenize(answer)
26 if not claims:
27 return 0.0
28 entailed = sum(
29 1 for claim in claims
30 if check_claim_against_source(gathered_tool_results, claim)["verdict"] == "ENTAILMENT"
31 )
32 return entailed / len(claims)
33
34print("=== SCENARIO 1: BOTH TOOLS SUCCEEDED, WELL-SUPPORTED ANSWER ===\n")
35
36well_supported_tool_results = (
37 "Web search result: $450 billion as of the most recent industry report. "
38 "Stats database result: $390 billion, per the latest verified quarterly database update."
39)
40well_supported_answer = (
41 "Sources disagree on the exact figure: web search reports $450 billion, "
42 "while the verified stats database reports $390 billion. "
43 "The database figure is likely more reliable, being independently verified."
44)
45
46score_1 = score_agent_answer(well_supported_answer, well_supported_tool_results)
47print(f"Confidence score (both tools succeeded): {score_1:.0%}\n")
48
49print("=== SCENARIO 2: ONE TOOL FAILED, DEGRADED EVIDENCE ===\n")
50
51degraded_tool_results = (
52 "[web_search_tool is currently UNAVAILABLE after retries -- proceed using other available sources] "
53 "Stats database result: $390 billion, per the latest verified quarterly database update."
54)
55# An answer that OVERSTATES certainty despite missing a real source --
56# exactly the false-confidence risk this scoring step is built to catch
57overconfident_answer = (
58 "The global AI market size is definitively $450 billion, confirmed "
59 "across multiple independent sources."
60)
61
62score_2 = score_agent_answer(overconfident_answer, degraded_tool_results)
63print(f"Confidence score (one tool failed, but answer claims certainty anyway): {score_2:.0%}\n")
64
65print(f"""
66=== THE RESULT, CONFIRMED DIRECTLY ===
67
68Scenario 1 (well-supported): {score_1:.0%}
69Scenario 2 (overconfident despite missing evidence): {score_2:.0%}
70
71Report your own measured scores. Scenario 2's score should be
72MEANINGFULLY LOWER, since the specific $450 billion figure it
73confidently states is NOT actually present anywhere in the
74degraded_tool_results (only the $390 billion database figure is,
75since the web search tool failed) -- directly catching an agent
76that OVERSTATES its own certainty when a real source was actually
77unavailable, exactly the false-confidence failure mode a real
78production research tool must never present to a real user without
79warning.
80""")Gotchas
- ⚠This confidence score checks the agent's final answer against what its tools ACTUALLY returned, not against some idealized, complete set of facts — this is deliberate and correct, since the score should reflect whether THIS specific agent run's answer was genuinely supported by THIS specific run's real, possibly degraded evidence.
- ⚠Scenario 2's overconfident_answer was deliberately constructed to test this exact failure mode directly, rather than waiting to observe it by chance in random agent runs — following this course's standing discipline of directly, deliberately triggering a specific failure case to confirm a safeguard catches it.
- ⚠A real production system should combine this confidence score with Step 2's tool-availability information directly in the final user-facing response — a low score paired with a note that one source was unavailable is far more actionable for a real user than a bare confidence number alone.
Step 5 — Serving the Complete Resilient Agent
This closing step wraps the complete pipeline — resilient tools, conflict-surfacing instructions, and calibrated confidence scoring — in a FastAPI server following Module 19's exact production pattern, returning the agent's answer alongside its real confidence score and a note on any tool that was unavailable during that specific request, giving a real caller genuine, honest information rather than a bare, unqualified answer.
1from fastapi import FastAPI
2from pydantic import BaseModel
3from langchain_core.messages import HumanMessage, SystemMessage
4import os
5
6os.environ["LANGCHAIN_TRACING_V2"] = "true"
7os.environ["LANGCHAIN_PROJECT"] = "deepleap-research-agent-production"
8
9app = FastAPI(title="Resilient Research Assistant API")
10
11@app.on_event("startup")
12def load_agent():
13 # agent, resilient tools, and the NLI confidence scorer are all
14 # constructed ONCE here -- Module 19's exact load-once discipline
15 print("Resilient research agent loaded: retry logic, graceful "
16 "degradation, conflict-surfacing, and confidence scoring all active.")
17
18class ResearchRequest(BaseModel):
19 question: str
20
21@app.post("/research")
22async def research(request: ResearchRequest):
23 result = await agent.ainvoke({
24 "messages": [system_instruction, HumanMessage(content=request.question)],
25 })
26 final_answer = result["messages"][-1].content
27
28 # Reconstructing which tools actually ran and what they returned,
29 # for BOTH the confidence check and honest reporting to the caller
30 tool_results_text = "\n".join(
31 m.content for m in result["messages"] if type(m).__name__ == "ToolMessage"
32 )
33 unavailable_tools = [
34 line for line in tool_results_text.split("\n") if "UNAVAILABLE" in line
35 ]
36
37 confidence_score = score_agent_answer(final_answer, tool_results_text)
38
39 return {
40 "question": request.question,
41 "answer": final_answer,
42 "confidence_score": round(confidence_score, 4),
43 "degraded_sources": unavailable_tools,
44 "fully_supported": len(unavailable_tools) == 0 and confidence_score >= 0.8,
45 }
46
47# Run with: uvicorn 05_serving_the_resilient_agent:app --host 0.0.0.0 --port 8000
48# Test: curl -X POST http://localhost:8000/research -H "Content-Type: application/json" \
49# -d '{"question": "What is the current global AI market size?"}'Gotchas
- ⚠fully_supported is deliberately a compound signal — requiring BOTH zero degraded sources AND a high confidence score — since either one alone could mislead: high confidence with a degraded source is exactly Step 4's overconfidence risk, and low confidence with no degraded source could indicate a genuine, unresolved multi-source disagreement worth a user's attention either way.
- ⚠Reconstructing tool_results_text from the agent's own message history, rather than tracking it separately during execution, keeps this endpoint's confidence scoring genuinely tied to what actually happened in THIS specific request, following the same real-not-idealized-reference principle established in Step 4.
- ⚠Every piece of this final server traces directly back to an earlier step in this project or an earlier module in this course — Step 2's resilient tools, Step 3's conflict-surfacing system prompt, Step 4's confidence scoring, and Module 19's exact serving pattern — confirming genuine assembly of proven pieces, not new, unverified logic introduced only here.