RAG + Web Search Still Gets It Wrong: The Three-Stage Fact Chain for AI Agents

Many teams building Agents experience a honeymoon period: plug in search, and the model finally stops making things up. Add a browser, and it feels like you’ve got a research assistant.

Then production starts serving a harder-to-catch kind of error: it provides links, writes with absolute confidence, but the conclusion is wrong.

This type of error isn’t necessarily because the model can’t reason. The more common scenario is: the user’s question contains a wrong name, search retrieves an article that “looks close enough,” and the model extracts a sentence from it that’s correct but irrelevant. None of the three stages completely collapsed — but combined, they steer you straight into a ditch.

On July 24, DeepLearning.AI published an analysis of a news retrieval benchmark. Researchers had models with web search tools answer current-day news questions, deliberately testing scenarios with false premises, multilingual content, and no-correct-option cases. The most striking finding wasn’t “model X won” — it was: errors typically get stuck at the retrieval stage, not the generation stage.

Research Analysis: Web Retrieval Flusters LLMs External link

0. Don’t Mistake “Can Search” for “Can Verify”

Think of an Agent as someone sent to an archive to look up an incident record.

The user hands over a slip: “Company A announced B in Shanghai yesterday, right?” If the company name is wrong and the city is wrong, no matter how diligent the archivist is, they might come back with a stack of “close enough” materials. Then, they read one true sentence from the materials, and package it into a plausible-sounding answer.

It’s the same with models and search. Factual tasks have at least three gates:

1. Is the question retrievable: Are the entities, time, location, and qualifiers correct?
2. Can the materials serve as evidence: Is the retrieved page a primary source? Does it contain the specific fact needed to answer?
3. Did the model answer from the materials: Did it correctly interpret language, timeline, and negation — or did it just spot a few keywords and start autocompleting?

Collapse these three gates into a single “web toggle” and debugging becomes excruciating. All you get is “the model is occasionally inaccurate,” with no clue whether to fix the prompt, the retrieval, or the citation mechanism.

Lesson 1: Retrieval augmentation isn’t a single capability — it’s a fact supply chain.

Figure: Every external fact should trace back to source evidence; when evidence is lacking, the system must be allowed to output “insufficient information.”

1. Stage One: Process the Question First, Then Send It to the Search Engine

Many products dump the user’s raw query straight into the search box. This is convenient — and about as safe as handing a new intern a vague requirement and telling them to go modify the production database.

A more stable approach is to first decompose the question into checkable constraints:

{
  "subject": "who or what",
  "claim": "the action or attribute to verify",
  "time_range": "which time window the event falls in",
  "location": "whether geography affects the answer",
  "language": "preferred retrieval language",
  "unknowns": ["which fields are just user guesses"]
}

This isn’t about making the experience complicated — it’s about reserving space for “what the user might have gotten wrong.”

For example, if a user asks “Did model X go open-source last night?”, don’t assume “last night” and “open-source” are both true. The system can construct two parallel retrieval directions: one searching for release announcements, one searching for repository licenses. If neither direction finds evidence, it should be allowed to return “cannot confirm,” rather than rewriting a launch preview into open-source news.

Treat False Premises as First-Class Citizens

Ordinary Q&A likes to hide uncertainty. A factual Agent should do the opposite: display it explicitly.

A qualified intermediate result should at minimum include:

• Which entities have been confirmed;
• Which conditions have no reliable evidence;
• What queries the system rewrote for this;
• Whether the answer is “confirmed,” “refuted,” or “insufficient information.”

If your product doesn’t have an “insufficient information” state, the model will eventually fill the period with whatever answer “sounds most like it.”

2. Stage Two: The Retrieval Target Isn’t Similarity — It’s Citability

Vector similarity, keyword matching, re-ranking — these all matter. But they only solve “does it look similar,” not “can it serve as proof.”

This trap is especially common with news and real-time information: two articles talk about the same company, same week, same product — but only one contains the specific number, timestamp, or direct quote you need. If the retrieval system only gives the model a bunch of topically similar pages, the model will pick whichever detail looks most convenient.

So the retrieval stage should classify candidate pages into three types:

In practice, you can add a simple field to every evidence fragment: supports_claim. It doesn’t ask the model “is this article valuable” — it asks a narrower question: can this text directly support this specific statement?

Question: Has this API been opened to all developers?
Evidence: The announcement says "limited beta."
Conclusion: Cannot support "full availability"; it actually constitutes a counterexample.

This step will noticeably reduce the most frustrating kind of incident — where the link is real, but the conclusion is false.

Lesson 2: The retrieval KPI shouldn’t just be relevance — it should also include evidence coverage.

3. Stage Three: Do a “Sentence-by-Sentence Reconciliation” Before Answering

The generation stage is the most underestimated. The model can read the correct page and still merge updates from different dates, or read an English conditional clause as an established fact.

This is especially true with cross-language content: in the study, English performance was generally better, while retrieval and comprehension in other languages were more likely to break. For Chinese products, this isn’t something you fix by “just translating English search results.” Between a Chinese query, English source text, and a Chinese answer, there are at least three transformation steps — each one can drop qualifiers.

A practical answer generation protocol can be simple:

1. Every external fact must be bound to an evidence fragment.
2. Sentences lacking sufficient evidence should be rewritten as conditionals or removed entirely.
3. Times, version numbers, prices, and entity relationships must be individually double-checked.
4. For conflicting sources, don’t “average” them; explicitly state the differences between sources.

The final response doesn’t need to dump the entire reasoning chain on the user, but it should let the user see where the facts land: source, date, and what exactly that source supports.

4. Add Four Tests to Your Agent — Don’t Just Measure “Answer Accuracy”

If you’re building a search-enabled Agent, the following four test categories are more useful than “ask ten encyclopedia questions”:

4.1 One-Wrong-Character Entity

Deliberately misspell a company name, person name, or product name by one character. Check whether the system corrects it first, or solemnly answers around the wrong entity.

4.2 Expired Timeline

Use words like “today,” “yesterday,” “latest,” then provide pages on the same topic but from different dates. Check whether it treats old news as breaking.

4.3 Topic Match, No Evidence

Retrieve ten articles all discussing a model, but none can prove “it has reached GA.” Check whether the system can say “not enough evidence found.”

4.4 Bilingual Materials

Ask in Chinese, provide evidence in English, and request a Chinese answer. Check whether qualifiers, numeric units, and negations were lost in translation.

These four tests share one trait: they’re not testing whether the model can recite knowledge — they’re testing whether your fact chain has any breaks.

Figure: These four case types should enter the continuous regression set for retrieval-type Agents — not just run once before launch.

5. A Minimal Closed Loop You Can Actually Ship

If you don’t want to jump into a complex multi-agent architecture right away, nail these four steps first:

User question
  -> Structured constraints and uncertain fields
  -> Multi-path queries with candidate source classification
  -> Evidence fragment support check against the current claim
  -> Answer with source, timestamp, and confidence state

Don’t rush to add more search sources. An Agent that can say “I don’t know, because I’m missing this piece of evidence” is usually more trustworthy than one that can stitch a beautiful paragraph from twenty webpages.

Using This Approach in China

In domestic deployment, the most easily overlooked aspects are data authorization and retrieval scope. Don’t treat all scraped webpages as a knowledge base that can be stored long-term and redistributed.

1. Prioritize authorized search or content services: Keep retrieval results with source, crawl time, and accessibility status.
2. Layer enterprise data permissions first: Internal documents, customer files, and public webpages must not enter the same default retrieval pool.
3. Decouple model and retrieval for independent replacement: Whether the backend is Tongyi Qianwen, Zhipu GLM-5, Wenxin Yiyan, or a local Ollama model, the fact chain’s “question → evidence → answer” protocol should remain unchanged.

You can swap the model. You can’t swap the responsibility for evidence.

References

1. DeepLearning.AI, Web Retrieval Flusters LLMs External link , 2026-07-24. The factual basis for this article regarding multilingual news retrieval, false premises, and retrieval failure patterns comes from this research analysis.
2. The “three-stage fact chain,” test cases, and implementation recommendations in this article are engineering analysis by 梦兽编程 and do not represent product conclusions from the source.

Follow 「全栈之巅-梦兽编程」 official account for weekly updates on Rust, AI programming, and Agent engineering practices.

Also welcome to explore 梦兽编程 AI Programming Assistant Service to integrate verifiable AI workflows into your daily development.