Skip to content

fix(governor): prune mode_transitions, which grew without bound - #905

Open
minion1227 wants to merge 1 commit into
GeniePod:mainfrom
minion1227:minion_gov_prune_transitions
Open

fix(governor): prune mode_transitions, which grew without bound#905
minion1227 wants to merge 1 commit into
GeniePod:mainfrom
minion1227:minion_gov_prune_transitions

Conversation

@minion1227

@minion1227 minion1227 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Store's doc comment says "Older rows pruned on startup and every hour", but prune only ever deleted from tegrastats. mode_transitions is created, indexed and inserted into — and nothing in the crate deletes from it — so a long-running device accumulates one row per mode transition for the life of the install.

Changes

  • Prune mode_transitions as well as tegrastats.
  • Name the retention windows (TEGRASTATS_RETENTION_MS, MODE_TRANSITION_RETENTION_MS) instead of inlining 24 * 3600 * 1000.
  • prune returns per-table counts (PruneCounts) rather than one number, so the hourly debug line says which table shed rows. Both existing callers discard the return value, so nothing else changes.
  • Correct the type doc to describe both windows.

Why 30 days for transitions rather than 24 hours

Transitions arrive orders of magnitude more slowly than samples — a couple of day/night flips plus media start/stop, versus ~17,280 samples/day — and the history is exactly what makes mode flapping diagnosable after the fact. A longer bounded window costs very little. This is a judgement call, so it is a named constant: say the word and I'll align it with the 24-hour sample window instead.

The growth is not inherently bounded

transition is called from the 5-second tick whenever the target mode differs from the current one. Memory-pressure flapping around stop_optins_mb can therefore record a row per tick — the same worst-case rate as the sample table, which is pruned. So this isn't only a slow-accumulation concern.

Real Behavior Proof

  • I have built and run the affected code locally (or noted why I could not).
  • I have verified the change end-to-end on Jetson hardware.
  • I have NOT verified on Jetson hardware, and I explain the equivalent verification path or validation gap below.

Tested profile / hardware (check all that apply):

  • jetson
  • raspberry_pi
  • portable_sbc
  • laptop
  • mac
  • CI-only / docs-only
  • Not run locally

What I ran

Linux x86_64 laptop. This is SQLite retention logic with no device dependency, so the laptop exercises it fully. I drove it through a temporary test against a real on-disk SQLite file — first on unmodified main to confirm the leak, then on this branch to confirm the fix — then removed the temporary test (see the note at the bottom).

cargo test -p genie-governor
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy --no-default-features -p genie-core -p genie-ctl --all-targets -- -D warnings
cargo fmt --all -- --check

What I observed

Against unmodified main — insert a transition dated one year back, call prune, count what survives:

MAIN: prune() deleted=0 rows; 1-YEAR-OLD mode_transitions still present = 1

prune reports zero deletions and the row is still there. That is the leak.

On this branch — one 25-hour-old sample, one 31-day-old transition, one recent row of each:

BEFORE prune: mode_transitions rows = 2
AFTER  prune: counts=PruneCounts { tegrastats: 1, mode_transitions: 1 } mode_transitions=1 tegrastats=1

The stale transition is removed, the recent one survives, and the sample table behaves as before.

Gates: cargo clippy clean under -D warnings on both the workspace and --no-default-features axes; cargo fmt --all -- --check clean; cargo test -p genie-governor 25 passed; cargo test --workspace green.

One pre-existing flake worth reporting (unrelated to this change — it lives in genie-common, which this PR does not touch): tegrastats::tests::mem_available_mb_async_matches_sync_version reads /proc/meminfo twice and asserts the two readings are equal, so it fails whenever MemAvailable shifts between them. I measured it failing 1 run in 5 in isolation on both this branch and clean main. Happy to fix it separately — comparing two independent reads of live kernel state can't be made reliable, so it probably wants to assert both are Ok rather than equal.

Test plan

cargo test -p genie-governor

To see the leak on current main: insert a row into mode_transitions with an old ts_ms, call Store::prune(), and observe the row survives while prune returns 0.

Notes for reviewers

  • No permanent test, deliberately, and I want to be straight about why. genie-governor is bin-only — no [lib] target and no tests/ directory — so a permanent test would have to be an inline #[cfg(test)] module in store.rs. I verified the behaviour with exactly such a module and then removed it, pasting its output above, rather than leave the crate's layout half-changed. If you'd prefer the regression pinned, I'm happy to either add the inline module or add a [lib] target plus tests/store_test.rs — just say which.
  • PruneCounts is pub because Store is; it derives Copy/Default/Eq so callers can compare or accumulate without ceremony.
  • Kept the two DELETEs as separate literal statements rather than a table-name-parameterised helper — dynamic SQL here would buy nothing and reads worse.
  • No filed issue; found while reading the governor's retention path after fix(governor): stop substituting a fake memory level when /proc/meminfo fails #900. Happy to open one first if you prefer that order.

Summary by CodeRabbit

  • New Features

    • Added configurable retention windows for system metrics and mode transition history.
    • Pruning now reports deleted records separately by data type, along with a combined total.
  • Bug Fixes

    • Improved data cleanup so both metrics and mode transition records are pruned according to their respective retention periods.
    • Enhanced cleanup logging with detailed deletion counts.

`Store` documents "Older rows pruned on startup and every hour", but `prune`
only ever deleted from `tegrastats`. `mode_transitions` is created, indexed and
inserted into and nothing in the crate deletes from it, so a long-running device
accumulated one row per mode transition for the life of the install.

Demonstrated against the current code: insert a transition dated a year back,
call `prune`, and it is still there —

    MAIN: prune() deleted=0 rows; 1-YEAR-OLD mode_transitions still present = 1

Typical volume is low (a couple of day/night flips plus media start/stop), but it
is not inherently bounded: `transition` is called from the 5-second tick whenever
the target mode differs, so memory-pressure flapping around `stop_optins_mb` can
record a row per tick — the same worst-case rate as the sample table.

Prune both tables, with the retention windows named rather than inlined.
Transitions keep 30 days instead of 24 hours: they arrive orders of magnitude
more slowly than samples and the history is what makes flapping diagnosable
after the fact, so a longer bounded window costs little.

`prune` now returns per-table counts (`PruneCounts`) instead of a single number,
so the hourly log line says which table shed rows. Both existing callers discard
the value, so nothing else changes.
@github-actions github-actions Bot added the bug Something isn't working label Jul 30, 2026
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bffda40c-5f48-4edd-8498-f94e4fd088a8

📥 Commits

Reviewing files that changed from the base of the PR and between 02a577d and 09091ce.

📒 Files selected for processing (1)
  • crates/genie-governor/src/store.rs

📝 Walkthrough

Walkthrough

Changes

Store retention pruning

Layer / File(s) Summary
Retention policy and pruning results
crates/genie-governor/src/store.rs
Separate retention constants and documentation are added for tegrastats and mode_transitions; Store::prune deletes from both windows and returns PruneCounts with a total() helper and per-table debug logging.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly states the main fix: pruning the unbounded mode_transitions table in the governor store.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant