Skip to content

fix(intelligence): _parse_importance_response second-level fallback blindly grabs first number #1074

Description

@knqiufan

Problem

In src/powermem/intelligence/importance_evaluator.py, the _parse_importance_response() method has a three-level fallback for parsing LLM responses:

  1. L1: Extract JSON substring via find("{") / rfind("}") and look for importance_score
  2. L2: re.findall(r'\d+\.?\d*', response) — grab the first number from the entire response text
  3. L3: Return fixed 0.5

The second-level fallback is semantically unverifiable — it grabs any number without context, which can produce incorrect importance scores that cascade into downstream systems.

Impact

importance_score directly affects:

  • _classify_memory_type() — determines working / short_term / long_term classification
  • initial_retention = initial_retention * importance_score — initial retention rate in Ebbinghaus algorithm
  • adjusted_interval = interval * (1 - importance_score * 0.3) — review schedule intervals

Concrete failure scenarios

Scenario Number grabbed Actual meaning Consequence
JSON parse fails but text contains criteria scores First dimension score (e.g. 0.9) A single criterion, not overall Wrong classification
Reasoning mentions "6 dimensions" 6 Text description min(1.0, 6) = 1.0 → forced long_term
"Importance 85%" 85 Percentage expression clamp → 1.0
Text mentions threshold "0.6 short_term" 0.6 Config explanation Unrelated to content

Inconsistency with other modules

Other parsers in the same codebase (parse_fact_extraction_json, parse_memory_actions_json, skill_manager._parse_skills) return empty structures on parse failure and never guess numbers. The importance evaluator is an outlier with higher downstream cost.

Asymmetry with exception fallback

  • LLM call throws exception → falls back to _rule_based_evaluation(content, metadata, context) (content-aware)
  • LLM call succeeds but parse is ambiguous → may return a random number via L2, never triggers rule engine

L2 "pretends to succeed" rather than "failing safely."

Suggested fix direction

  1. L1: Use parse_json_from_text() (existing utility); support both importance_score and overall_score field names; when only criteria_scores is present, compute weighted sum using criteria_weights
  2. L2: Only extract numbers in field-name context (e.g. importance_score"\s*:\s*); reject values outside [0.0, 1.0] instead of clamping
  3. L3: Return None → caller falls back to _rule_based_evaluation (consistent with exception path)

Related files

  • src/powermem/intelligence/importance_evaluator.py_parse_importance_response, _llm_based_evaluation
  • src/powermem/prompts/importance_evaluation.py — Prompt JSON structure with six criteria
  • src/powermem/intelligence/intelligent_memory_manager.py — consumes importance_score
  • src/powermem/intelligence/ebbinghaus_algorithm.py — thresholds, retention, review scheduling
  • src/powermem/utils/utils.pyparse_json_from_text, extract_json (reusable utilities)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions