Skip to content

perf(audit): use WAL journal to avoid fsync per commit on event loop - #392

Open
hi-neason wants to merge 1 commit into
andrewyng:mainfrom
hi-neason:fix/audit-store-wal-pragma
Open

perf(audit): use WAL journal to avoid fsync per commit on event loop#392
hi-neason wants to merge 1 commit into
andrewyng:mainfrom
hi-neason:fix/audit-store-wal-pragma

Conversation

@hi-neason

Copy link
Copy Markdown

Summary

While reading through the engine I kept seeing self._audit(...) called directly inside the async loop — 17 of them — and each one ends up at AuditStore.append, which does an INSERT ... commit() against SQLite.

AuditStore never sets a journal mode, so SQLite runs with the defaults: DELETE journal + synchronous=FULL. That means every audit commit is an fsync. With a couple of tool calls per turn that's a handful of syscalls, but on a busy turn (multiple tools, retries, plan/approval stages) it adds up, and because the sink is invoked synchronously from inside TurnEngine._audit, each fsync parks the whole event loop — roughly 1–10ms per audited tool stage on a typical disk.

The fix is two lines at connect time:

  • PRAGMA journal_mode=WAL
  • PRAGMA synchronous=NORMAL

WAL means writers don't block readers and commit doesn't fsync; synchronous=NORMAL is the recommended pairing with WAL and is still durable across crashes (the WAL checkpoints to the main DB on close/rotation). Transaction atomicity is unchanged — a crash mid-commit still rolls back. No API or behavior change beyond the pragma.

Change

  • coworker/audit.py — set both pragmas right after connecting, before the CREATE TABLE IF NOT EXISTS runs. Added a short comment explaining why (since the reason for the pragma is non-obvious without knowing how hot _audit is).

Tests

I noticed coworker/audit.py had no test file at all, even though it's responsible for redacting secrets before they're written to coworker.db. So along with the regression test for the pragma I added tests/test_audit.py covering the parts that are silent if they regress:

  • test_audit_store_uses_wal_journal — asserts journal_mode == "wal" and synchronous == 1 (NORMAL). This is the one that actually catches the perf bug; on main it fails with assert 'delete' == 'wal'.
  • test_append_and_list_roundtrip, test_list_filters — basic storage shape and the session_id/tool filters.
  • test_sanitize_args_redacts_secrets — every key in _SECRET_KEYS / _BODY_KEYS plus the *_token suffix path.
  • test_sanitize_args_redacts_browser_type_textbrowser_type's text field gets [redacted input] (form input, not a body blob).
  • test_resource_extracts_well_known_keys — url / owner-repo / zendesk subdomain / result-URL fallback.
  • test_close_is_idempotent and test_summarize_truncates_long_strings.

I confirmed the new pragma test fails without the fix (stashed audit.py, saw assert 'delete' == 'wal', then popped) and passes with it.

Full suite: 1006 → 1014 passed, 1 skipped. The 9 failures present locally (test_fake_slack ×8 needs the messaging extra / slack-bolt, plus one test_ui_refresh_e2e) are pre-existing on a clean main and unrelated to this change.

No screenshots — backend-only (SQLite pragma + tests). Happy to add anything else that'd help review.

Checklist

  • pytest tests -q passes
  • ruff check / ruff format --check clean on the added test file
  • No hardcoded API keys, tokens, or internal addresses
  • No new dependencies
  • Regression test added, verified to fail without the fix
  • Conventional commit message

AuditStore.append runs synchronously from TurnEngine._audit at every tool
stage (17 call sites inside the async engine loop). With SQLite's default
DELETE journal + synchronous=FULL, each append's commit() fsyncs, blocking
the event loop for ~1-10ms per tool call.

Set journal_mode=WAL and synchronous=NORMAL at connect time so a commit
returns without an fsync while preserving transaction atomicity; WAL
checkpoints flush to disk periodically. Add tests/test_audit.py covering
the pragma, round-trip/filters, secret/body redaction, browser_type input
redaction, resource extraction, close idempotency, and truncation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant