Bug: ReACT report generation hallucinates entities via self-fabricated <tool_result> blocks

[paontheinternet]

Bug Description

The ReACT section generation loop in report_agent.py is vulnerable to a hallucination pattern where the LLM fabricates <tool_result> blocks in its own response, effectively "answering itself" with invented data before the real tool execution occurs.

This causes the report to contain completely fabricated usernames, quotes, engagement statistics, and analysis — all presented as if they came from the Zep knowledge graph.

Root Cause

In the ReACT loop (_generate_section_react), when the LLM outputs a tool call, the full LLM response (including any self-fabricated <tool_result> blocks) is appended to the message history before the real tool result is injected:

# Line ~1456 in report_agent.py
messages.append({"role": "assistant", "content": response})  # ← includes fake <tool_result>
messages.append({
    "role": "user",
    "content": REACT_OBSERVATION_TEMPLATE.format(
        tool_name=call["name"],
        result=result,  # ← real tool result
        ...
    ),
})

What happens:

1. LLM outputs: <tool_call>{"name": "insight_forge", ...}</tool_call> followed by <tool_result>{"name": "insight_forge", "result": "...fabricated data..."}</tool_result>
2. The parser correctly extracts the <tool_call> and executes the real tool
3. But the entire response (including the fake <tool_result>) gets appended to message history
4. The real tool result is added as a separate user message
5. Now the LLM sees both the fake and real results in its context
6. In subsequent iterations, the LLM treats the fabricated data as authoritative and cites it in the final report

Evidence

In a test report, Section 1 cited these entities with detailed quotes and engagement numbers:

• @VibeCodeKing (312 likes, 87 retweets)
• @SideHustleSara ("Mom of 3 | Wannabe app builder")
• u/NomadLaunchpad, u/MobileFirstMike, u/ZeroToAppStore

None of these entities existed in the scraped data or Zep knowledge graph. They were all fabricated by the LLM inside <tool_result> blocks in its first response, then treated as real data throughout the rest of the section generation.

Proposed Fix

1. Strip fake <tool_result> blocks before appending to message history:

@staticmethod
def _strip_fake_tool_results(response: str) -> str:
    """Strip any <tool_result> blocks the LLM fabricated in its response."""
    import re
    cleaned = re.sub(
        r'<tool_result>.*?</tool_result>',
        '',
        response,
        flags=re.DOTALL,
    )
    cleaned = re.sub(r'\n{3,}', '\n\n', cleaned)
    return cleaned.strip()

Then in the ReACT loop:

# Replace:
messages.append({"role": "assistant", "content": response})

# With:
cleaned_response = self._strip_fake_tool_results(response)
messages.append({"role": "assistant", "content": cleaned_response})

2. Add anti-hallucination rules to SECTION_SYSTEM_PROMPT_TEMPLATE:

5. [CRITICAL: Do NOT fabricate data]
   - NEVER invent usernames, quotes, statistics, or engagement numbers
   - NEVER include <tool_result> blocks in your response — only the system provides tool results
   - Only cite entities, quotes, and numbers that appear in actual tool results
   - If no relevant data exists for a claim, say so rather than making it up

Impact

This bug affects all report generation. Reports may contain plausible-sounding but completely fabricated entities, quotes, and statistics that cannot be traced back to any real data in the knowledge graph. This is especially problematic because the fabricated data is indistinguishable from real data without cross-referencing the entity pool.

Environment

• LLM: Claude claude-opus-4-6 (via Anthropic API)
• Zep Cloud knowledge graph
• MiroFish latest (as of April 2026)

[dosubot]

Hi @paontheinternet! I'm Dosu and I’m helping the MiroFish team.

I can confirm this bug after looking at the code. The root cause analysis is accurate — the ReACT loop in _generate_section_react appends the raw LLM response to message history without stripping any hallucinated <tool_result> blocks [1]:

messages.append({"role": "assistant", "content": response})  # no sanitization
messages.append({"role": "user", "content": REACT_OBSERVATION_TEMPLATE.format(...)})

A few additional findings that may be useful:

Existing sanitization only covers chat(), not report generation. The chat() method does strip <tool_call> blocks from responses before returning them to the user [2], but this logic doesn't exist in _generate_section_react and doesn't cover <tool_result> blocks in either path.

The system prompt does say "don't write fake tool results" [3], but this is guidance-level only — there's no code-level enforcement. With larger/more creative models (like Claude Opus), prompt-level instructions alone are clearly insufficient to prevent this.

This is a known category of issue. A similar hallucination problem was reported in BettaFish's InsightAgent, where insufficient crawled data caused the LLM to fabricate social media users and engagement metrics to meet prompt output requirements. The same dynamic is at play here — the LLM fabricates <tool_result> data when it "anticipates" what the tool would return.

PR #389 (Native Function Calling Migration) [4] would likely eliminate this attack vector entirely by replacing the XML-based <tool_call>/<tool_result> pattern with structured function calling, where the LLM physically cannot inject tool results into its response. That said, it's not yet merged, so your proposed regex-based stripping is a good immediate fix.

One suggestion on the proposed fix: In addition to stripping <tool_result> blocks, consider adding detection logging so you can measure how frequently this occurs across different models. This would help prioritize the native function calling migration:

cleaned_response = self._strip_fake_tool_results(response)
if cleaned_response != response:
    logger.warning("Stripped hallucinated <tool_result> from LLM response in section generation")

Both parts of the proposed fix (regex stripping + prompt hardening) look solid and complementary. The regex is the critical piece since prompt instructions alone have clearly proven insufficient.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
  

[raye-deng]

This is a really clean root cause analysis. The pattern you describe — LLM fabricating tool results that then pollute the message history — is one we see repeatedly in AI agent systems. The insidious part is that the fabricated data often looks more plausible than the real data because the LLM optimizes for coherence rather than accuracy.

One thing we've found helpful in production: adding a post-generation validation step that cross-references every cited entity/quote/number against the actual knowledge base. Not just checking if entities exist, but verifying that specific quotes and statistics match the source data. This catches the class of hallucinations where the LLM uses real entity names but fabricates their attributes.

The regex stripping fix is solid as an immediate mitigation. We've also had success with structured function calling (as mentioned with PR #389) because it physically prevents the LLM from injecting result blocks — the architecture makes this class of bug impossible rather than relying on sanitization.

For catching these patterns in code review: `npx @opencodereview/cli scan . --sla L1` includes checks for AI agent output validation gaps, specifically detecting when tool result handling trusts unverified intermediate LLM output.