Problem
In src/powermem/intelligence/importance_evaluator.py, the _parse_importance_response() method has a three-level fallback for parsing LLM responses:
- L1: Extract JSON substring via
find("{") / rfind("}") and look for importance_score
- L2:
re.findall(r'\d+\.?\d*', response) — grab the first number from the entire response text
- L3: Return fixed
0.5
The second-level fallback is semantically unverifiable — it grabs any number without context, which can produce incorrect importance scores that cascade into downstream systems.
Impact
importance_score directly affects:
_classify_memory_type() — determines working / short_term / long_term classification
initial_retention = initial_retention * importance_score — initial retention rate in Ebbinghaus algorithm
adjusted_interval = interval * (1 - importance_score * 0.3) — review schedule intervals
Concrete failure scenarios
| Scenario |
Number grabbed |
Actual meaning |
Consequence |
| JSON parse fails but text contains criteria scores |
First dimension score (e.g. 0.9) |
A single criterion, not overall |
Wrong classification |
| Reasoning mentions "6 dimensions" |
6 |
Text description |
min(1.0, 6) = 1.0 → forced long_term |
| "Importance 85%" |
85 |
Percentage expression |
clamp → 1.0 |
| Text mentions threshold "0.6 short_term" |
0.6 |
Config explanation |
Unrelated to content |
Inconsistency with other modules
Other parsers in the same codebase (parse_fact_extraction_json, parse_memory_actions_json, skill_manager._parse_skills) return empty structures on parse failure and never guess numbers. The importance evaluator is an outlier with higher downstream cost.
Asymmetry with exception fallback
- LLM call throws exception → falls back to
_rule_based_evaluation(content, metadata, context) (content-aware)
- LLM call succeeds but parse is ambiguous → may return a random number via L2, never triggers rule engine
L2 "pretends to succeed" rather than "failing safely."
Suggested fix direction
- L1: Use
parse_json_from_text() (existing utility); support both importance_score and overall_score field names; when only criteria_scores is present, compute weighted sum using criteria_weights
- L2: Only extract numbers in field-name context (e.g.
importance_score"\s*:\s*); reject values outside [0.0, 1.0] instead of clamping
- L3: Return
None → caller falls back to _rule_based_evaluation (consistent with exception path)
Related files
src/powermem/intelligence/importance_evaluator.py — _parse_importance_response, _llm_based_evaluation
src/powermem/prompts/importance_evaluation.py — Prompt JSON structure with six criteria
src/powermem/intelligence/intelligent_memory_manager.py — consumes importance_score
src/powermem/intelligence/ebbinghaus_algorithm.py — thresholds, retention, review scheduling
src/powermem/utils/utils.py — parse_json_from_text, extract_json (reusable utilities)
Problem
In
src/powermem/intelligence/importance_evaluator.py, the_parse_importance_response()method has a three-level fallback for parsing LLM responses:find("{")/rfind("}")and look forimportance_scorere.findall(r'\d+\.?\d*', response)— grab the first number from the entire response text0.5The second-level fallback is semantically unverifiable — it grabs any number without context, which can produce incorrect importance scores that cascade into downstream systems.
Impact
importance_scoredirectly affects:_classify_memory_type()— determinesworking/short_term/long_termclassificationinitial_retention = initial_retention * importance_score— initial retention rate in Ebbinghaus algorithmadjusted_interval = interval * (1 - importance_score * 0.3)— review schedule intervalsConcrete failure scenarios
0.9)6min(1.0, 6)= 1.0 → forcedlong_term850.6Inconsistency with other modules
Other parsers in the same codebase (
parse_fact_extraction_json,parse_memory_actions_json,skill_manager._parse_skills) return empty structures on parse failure and never guess numbers. The importance evaluator is an outlier with higher downstream cost.Asymmetry with exception fallback
_rule_based_evaluation(content, metadata, context)(content-aware)L2 "pretends to succeed" rather than "failing safely."
Suggested fix direction
parse_json_from_text()(existing utility); support bothimportance_scoreandoverall_scorefield names; when onlycriteria_scoresis present, compute weighted sum usingcriteria_weightsimportance_score"\s*:\s*); reject values outside[0.0, 1.0]instead of clampingNone→ caller falls back to_rule_based_evaluation(consistent with exception path)Related files
src/powermem/intelligence/importance_evaluator.py—_parse_importance_response,_llm_based_evaluationsrc/powermem/prompts/importance_evaluation.py— Prompt JSON structure with six criteriasrc/powermem/intelligence/intelligent_memory_manager.py— consumesimportance_scoresrc/powermem/intelligence/ebbinghaus_algorithm.py— thresholds, retention, review schedulingsrc/powermem/utils/utils.py—parse_json_from_text,extract_json(reusable utilities)