Skip to content

Library sync on folder change - #58

Merged
gosku merged 27 commits into
mainfrom
library-sync-on-folder-change
Jul 6, 2026
Merged

Library sync on folder change#58
gosku merged 27 commits into
mainfrom
library-sync-on-folder-change

Conversation

@gosku

@gosku gosku commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Sync the Library when a folder is added or updated

What

The Library page now imports images the moment you add a folder or change a folder's path, instead of only at make start. Each folder shows its sync progress live, and it works in both install modes (lite: SQLite, no broker; full: PostgreSQL + Celery).

Removing a folder is unchanged: it only stops monitoring and never deletes images (add-only, per ADR 010).

Why

Until now, adding or repointing a folder on the Library page only updated the monitored list. Nothing was imported until the next restart, so a user could register a folder full of photos and see nothing happen, with no feedback and no obvious reason. This closes that gap: you register or repoint a folder and it syncs right away, with visible progress, while the page stays responsive even for a first import of tens of thousands of files.

How

  • Trigger. The add and path-update views call a trigger_folder_sync use case after the folder change; removal does nothing.
  • Execution strategy lives in the application layer, not the view.
    • Full mode: enqueue one Celery task per new image and return immediately.
    • Lite mode: run the sync in a background daemon thread (via a small services/background.py runner) so the request never blocks. The thread lives in the web process and outlives the request, so navigating away does not stop it.
  • Progress. A new SyncRun model records state (scanning, processing, completed, failed, interrupted), counters, and timestamps, persisted so any request can read it. The folder row polls a status endpoint over HTMX and shows Scanning..., a progress bar, then a summary like Imported 36, skipped 3. A conditional unique constraint allows at most one active run per folder (which doubles as the "already syncing" guard), and counters use atomic increments with a race-safe finalize so concurrent Celery workers stay consistent.
  • Concern separation. The pure process_image operation and the generic image-processing task (shared with the manual import command) are untouched. A process_synced_image use case composes process_image with the progress bookkeeping; both the new Celery task and the lite thread delegate to it.
  • Single-folder scope with recovery. The per-folder sync logic is factored out; the startup sync_library loops over it and first marks any dangling run interrupted, so a run killed mid-flight self-heals on the next start.
  • SQLite tuning (lite). WAL and a busy timeout are enabled from the engine, so the background writer never blocks foreground reads and a colliding write waits rather than failing with "database is locked".

Notes

  • Full mode: restart the Celery worker after deploying so it registers the new task. Until then those messages are discarded and the next make start re-syncs.
  • Documentation updated (docs/library_sync.md) and a new decision record added (ADR 011).
  • Implemented layer by layer (data, domain, application, interface) with tests at every layer.

gosku and others added 27 commits July 2, 2026 17:39
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gosku
gosku merged commit e2932b6 into main Jul 6, 2026
4 checks passed
@gosku
gosku deleted the library-sync-on-folder-change branch July 6, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant