Skip to content

refactor(history): move schema and index helpers into store_schema - #1333

Draft
gcs8 wants to merge 3 commits into
mainfrom
refactor/store-schema
Draft

gcs8 wants to merge 3 commits into
mainfrom
refactor/store-schema

Conversation

@gcs8

@gcs8 gcs8 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

What moved

The connection-bound schema and index helpers of HistoryStore move from history_service/store.py into the new module history_service/store_schema.py. There are 22 of them, in their original order:

  • Static helpers: _synchronize_table_counts, _rowid_span, _has_table, _recount_implausible_table_counts, _recount_table, _recount_restored_table_counts, _backfill_disk_identity_keys_once, _ensure_slot_state_columns, _ensure_slot_event_columns, _ensure_metric_sample_columns, _backfill_disk_identity_batch, _ensure_identity_indexes, _missing_chronological_indexes, _ensure_chronological_indexes, _apply_bulk_maintenance_profile, _filesystem_key, _index_key_columns, _index_build_disk_needs and _ensure_columns.
  • Instance helpers: _describe_free_space, _require_index_build_space and _describe_startup_space_failure.

HistoryStore keeps the connection, the migration lock and every commit and rollback. These stay in store.py unchanged: _initialize_schema, _backfill_disk_identity_keys (it commits each batch), _index_build_lower_bound_bytes (it commits a corrected counter), build_chronological_index, _release_failed_index_build (it rolls back), pending_chronological_indexes, index_maintenance, exact_count_scan, _wait_for_exact_count_scans and sqlite_temporary_directory. The schema text and every constant also stay in store.py.

Behaviour is unchanged

  • HistoryStore keeps a method under every moved name. Each one keeps the original signature (including the HISTORY_COUNTED_TABLES default) and the original staticmethod or instance kind, so inspect.getattr_static still finds a staticmethod for each kill phase. Each one calls the moved function and passes the store module (and the store, for instance helpers).
  • The moved code reads every name it used to take from store.py globals through that module at call time: HistoryStore, the constants, logger, host_repair_path, HistoryIndexBuildSpaceError, os and shutil. A replacement on the store module or class therefore still takes effect. Examples are store_module.CHRONOLOGICAL_INDEX_INLINE_MAX_ROWS, store_module.DISK_IDENTITY_BACKFILL_BATCH_SIZE and setattr(HistoryStore, "_backfill_disk_identity_batch", ...). Log records still come from the history_service.store logger.
  • store_schema imports nothing first-party and does not import store, so there is no cycle and no new cross-package import. store.py keeps host_repair_path imported (with noqa: F401) because the moved code reads it there.
  • The moved bodies are copied unchanged apart from code indentation and the receiver. Multi-line SQL literals keep their original interior indentation, so every statement (and the index SQL a new database stores in sqlite_master) is byte-for-byte what it was; a script compared all 59 string literals of the 22 moved functions with main and found no difference. A bare store global became store_module.NAME, self became store, and each signature gained the store_module parameter (plus store for instance helpers). No statement was reordered, reworded or fixed. Migration policy and the v1 and v2 contracts are unchanged.

Tests

Seven new tests are in tests/test_history_deferred_indexes.py (StoreSchemaModuleTests):

  • The new module imports on its own, in a child process, without loading history_service.store, and it has no first-party import.
  • Every moved name is still on HistoryStore, with the same kind, parameters and return annotation as the moved function.
  • A fresh database stores the two identity index statements byte-for-byte as before. This test failed on the first pushed version, which had dedented the SQL text with the code; the third commit fixed it.
  • The new module never commits, rolls back, closes, connects or takes a lock.
  • One test per lookup path shows that a replacement on the old owner reaches the moved code:
    • a HistoryStore method (_ensure_columns);
    • store-module names (SLOT_EVENT_OPTIONAL_COLUMNS, CHRONOLOGICAL_INDEX_INLINE_MAX_ROWS, logger);
    • an instance method plus a module function (_filesystem_key, sqlite_temporary_directory, host_repair_path).

Before the move, the first six failed because history_service.store_schema did not exist (Ran 6 tests ... FAILED (failures=1, errors=5)). No existing test line changed. The one other test-file edit is the history_service/store.py ceiling in tests/test_source_shape.py, lowered from 6000 to 5500 as that check asks. store.py went from 5507 to 5231 lines, and store_schema.py is 415 lines.

Local runs used python -m unittest (SQLite 3.50.4; this change relies on nothing that differs from 3.45.1). All passed with 0 skipped:

  • tests.test_history_service: 340 tests.
  • tests.test_history_deferred_indexes, tests.test_history_released_schema_upgrades, tests.test_image_upgrade_contract, tests.test_dev_check and tests.test_ci_contract: 279 tests.
  • tests.test_source_shape, tests.test_unittest_shards, tests.test_unittest_identities and tests.test_changelog_fragments with StoreSchemaModuleTests: 139 tests.

ruff check --select E4,E7,E9,F and git diff --check are clean.

Left out

The plan expected 500 to 750 lines. This move is 363 method lines. The helpers that commit or roll back (_backfill_disk_identity_keys, _index_build_lower_bound_bytes, _release_failed_index_build) stay on HistoryStore, so that it keeps every transaction.

Refs #1328

gcs8 added 2 commits November 16, 2026 23:38
Move the connection-bound schema and index helpers of HistoryStore into
the new module history_service/store_schema.py. HistoryStore keeps a
method under every moved name, with the same signature and the same
staticmethod or instance kind, that passes the store module (and the
store, for instance helpers) to the moved code. The moved code looks
every store-module name up on that module at call time, so a
replacement made on the store module or class still takes effect. The
new module imports nothing first-party and does not import the store
module.

HistoryStore keeps the connection, the migration lock and every commit
and rollback: _initialize_schema, _backfill_disk_identity_keys,
_index_build_lower_bound_bytes, build_chronological_index and
_release_failed_index_build stay where they are. No behaviour, error
text, log line or migration policy changes.

Refs #1328
@github-actions github-actions Bot added the internal Refactor, chore, test or CI change with no operator-visible behavior label Nov 17, 2026
The move dedented the interior lines of the multi-line SQL literals along
with the code, so a fresh database stored its two identity index
statements with four fewer spaces than before. Restore the original
indentation inside every non-docstring literal of the moved helpers,
and pin the stored index SQL in a test.

Refs #1328
@gcs8

gcs8 commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-17T05:40:49.411733Z 33434c1 Manual request
🔒 Security Review ✅ Completed 2026-11-17T05:43:46.721131Z 33434c1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 33434c18e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 33434c18e4

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal Refactor, chore, test or CI change with no operator-visible behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant