From 82ab695a991c3770ef4e3f9d3a56ea523582c952 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 15:32:16 +0200 Subject: [PATCH 01/27] Add missing ADRs 009, 010 and recipe graphs to index Co-Authored-By: Claude Opus 4.8 --- docs/index.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/index.md b/docs/index.md index 8fb1d5f..0c90e75 100644 --- a/docs/index.md +++ b/docs/index.md @@ -10,6 +10,7 @@ - [Library Sync](library_sync.md) — how make start scans library folders, deduplicates against the catalog, and uses timestamps to skip unchanged directories - [EXIF Mapping](exif_mapping.md) — how Fujifilm EXIF fields map to database model fields - [Recipe Naming](recipe_naming.md) — how recipes are named and the constraints inherited from the camera +- [Recipe Graphs](recipe_graphs.md) — the film simulation graph and version-line graph views, and how to read node distance - [Image Matching](favorite_image_matching.md) — how images are matched to the catalogue when rating in bulk - [PTP Encodings](ptp_encodings.md) — PTP/USB encoding reference for camera communication @@ -31,4 +32,6 @@ - [ADR 006 — QR Decode Library and Minimum QR Code Size](ADRs/006-qr-decode-library-and-size.md) — QR decode library choice and minimum QR code size - [ADR 007 — Normalize Recipe Data Before Storage](ADRs/007-normalize-recipe-data.md) — normalizing recipe data before storage - [ADR 008 — Recipe Versioning via Generalised Grouping](ADRs/008-recipe-versioning.md) — version lines and recipe families via a shared grouping abstraction +- [ADR 009 — Moving a Recipe Between Version Lines](ADRs/009-move-recipe-between-version-lines.md) — reassigning an existing recipe to a different VERSION_LINE group while keeping positions contiguous +- [ADR 010 — Image Library: Folder Monitoring and Catalog Sync](ADRs/010-image-library.md) — persisting monitored folders and detecting and importing new images automatically at startup From fc78787236dffb457b0cd120bf4c49975a525deb Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:40:49 +0200 Subject: [PATCH 02/27] Add ADR 011 for library sync on folder change Co-Authored-By: Claude Opus 4.8 --- .../ADRs/011-library-sync-on-folder-change.md | 214 ++++++++++++++++++ docs/index.md | 1 + 2 files changed, 215 insertions(+) create mode 100644 docs/ADRs/011-library-sync-on-folder-change.md diff --git a/docs/ADRs/011-library-sync-on-folder-change.md b/docs/ADRs/011-library-sync-on-folder-change.md new file mode 100644 index 0000000..287ef5a --- /dev/null +++ b/docs/ADRs/011-library-sync-on-folder-change.md @@ -0,0 +1,214 @@ +# ADR 011 — Library sync on folder add/update + +**Status**: Accepted +**Date**: 2026-07-02 + +--- + +## Context + +ADR 010 introduced the Library: a list of `LibraryFolder` rows the app monitors, and a **startup** sync (`make start` runs `manage.py sync_library`) that walks every folder and imports new images. That covered the "detect new photos over time" need, but it deliberately did nothing while the app is running. + +The Library page (added alongside ADR 010) already lets the user add a folder, update a folder's path, and remove a folder, each through a use case. But those actions only change the monitored **list**. Nothing is imported until the next restart. A user who registers a folder full of photos sees it appear in the table and then... nothing happens, with no feedback and no obvious reason. The catalog silently lags behind the library until `make start` is run again. + +The app ships in two modes (ADR 003): **lite** (SQLite, no broker, `process_image` runs inline) and **full** (PostgreSQL + Celery, `process_image` runs in a worker). Any solution has to behave sensibly in both. + +--- + +## Problem + +Triggering the import from a web request is not as simple as calling the sync in the view — several forces make a naive implementation fail: + +- A folder's first import can be tens of thousands of files, so running it inside the request would hang the page or time out. +- The two install modes process images differently — inline in lite, enqueued to a worker in full — so a single trigger has to serve both. +- Once the work runs outside the request, its progress must live somewhere durable, so the UI can show it and it survives the user leaving the page. +- The image-processing code is shared with the manual `import` command, so sync-specific state must not leak into it. +- The process doing the work can die mid-run, leaving partial progress that must be recoverable. + +So: how do we run a potentially long import off the request, in both modes, with visible and recoverable progress, without contaminating the shared processing code? + +Removal is out of scope for importing: ADR 010 is add-only (the sync never deletes images), so removing a folder just deregisters it. + +--- + +## Options considered + +The central question is **how the triggered sync runs** relative to the HTTP request. + +### Option A — Run the sync synchronously inside the request + +The add/update view calls the sync directly and returns when it finishes. + +**Why we did not choose this option:** + +In full mode this is tolerable (the request only enqueues Celery tasks and returns), but in lite mode the request itself runs `process_image` for every new file. A first import of a large folder would hang the page for minutes and risk a proxy/browser timeout. It also fails problem (2): the work is tied to the request, so navigating away or a dropped connection could interrupt it. + +### Option C — Run the lite-mode sync in a detached subprocess + +Spawn `manage.py sync_library --folder ` as an independent OS process. + +**Why we did not choose this option:** + +It solves (1) and (2), and it even survives a web-process reload. But it is heavier: process spawning, argument plumbing, and status coordination that can only happen through the database anyway. The durability advantage over a thread — surviving a web reload — is already provided by the startup sync, which idempotently catches up on the next `make start`. The extra machinery is not justified for a single-user local app. + +### Option B — Decide the execution strategy in a use case; run lite in a background thread (chosen) + +A `trigger_folder_sync` **use case** decides how to run the sync based on the mode. In full mode it runs the single-folder sync inline (which only enqueues tasks and returns fast). In lite mode it hands the sync to a background daemon thread via a small `services/background.py` runner, so the request returns immediately while the thread processes images server-side. Progress is persisted in the database and polled over HTMX. + +**Why this was chosen:** + +- The thread lives in the web process, independent of the request, so it satisfies (2) directly: leaving the page only stops the polling, not the work. +- Putting the celery-vs-thread choice in a use case (not the view) keeps the decision in the application layer; the view just calls the use case, and raw threading stays behind a service, mirroring how `workertasks.enqueue_task` hides Celery. +- The startup sync remains the catch-up net, so the thread's one weakness — dying on a web-process reload — is already covered without a subprocess. + +--- + +## Decision + +- **Trigger only on add and path-update.** Remove stays pure deregistration. A path-update additionally resets `last_checked_at`, because the repointed tree is a different directory and mtime gating would otherwise skip its older subdirectories. + +- **Single-folder scope.** The per-folder walk-and-dispatch logic is factored into a `sync_folder` use case. `sync_library` (startup) becomes a loop over `sync_folder`, so an add never re-walks every registered folder. + +- **Execution strategy lives in `trigger_folder_sync`.** Full mode calls `sync_folder` directly (fast: it enqueues Celery tasks). Lite mode runs `sync_folder` in a daemon thread via `services/background.py`. The view only calls the use case. + +- **Progress lives in a `SyncRun` model, polled over HTMX.** Because the sync runs server-side, its state must be durable and readable from any request. Each run records `state` (`SCANNING` → `PROCESSING` → `COMPLETED`/`FAILED`/`INTERRUPTED`), `total`, `processed`, `skipped`, `errors`, and timestamps. History is kept; the latest run per folder is shown. A conditional `UniqueConstraint` allows **at most one active run per folder**, which doubles as the "already syncing" guard. + +- **Per-image work is composed, not coupled.** The domain `process_image` operation and the generic `process_image_task` (also used by the manual import command) stay sync-agnostic. A `process_synced_image` use case composes `process_image` with progress bookkeeping: it counts skips (`NoFilmSimulationError`) and errors, and finalises the run when every image is accounted for. Both interface adapters — a thin `sync_process_image_task` (full) and the thread loop (lite) — delegate to it. + +- **SQLite tuning for lite concurrency.** WAL and a busy timeout are enabled via `DATABASES["OPTIONS"]`, derived from `DB_ENGINE`. WAL lets the foreground request threads read while the background thread writes; the busy timeout makes a colliding foreground write wait rather than raise "database is locked". Per-image transactions plus JPEG-only fast hashing keep write-lock holds tiny, so rating/recipe writes are never starved. + +- **Crash recovery.** A run left `SCANNING`/`PROCESSING` when its process dies is marked `INTERRUPTED` at the start of the next `make start` sync, which then idempotently re-imports. + +- **Full-mode worker-down.** `trigger_folder_sync` pings for a worker up front; if none responds it surfaces a Library-page error and creates no run (no stuck badge). + +--- + +## Diagrams + +### Data model + +```mermaid +erDiagram + LibraryFolder { + int id PK + string path "normalized absolute, unique" + datetime last_processed_at + datetime last_checked_at "reset on path update" + } + SyncRun { + int id PK + int folder_id FK + string state "SCANNING | PROCESSING | COMPLETED | FAILED | INTERRUPTED" + int total "null while scanning" + int processed + int skipped + int errors + datetime started_at + datetime finished_at + } + Image { + int id PK + string filepath + } + + LibraryFolder ||--o{ SyncRun : "has runs (≤1 active)" + LibraryFolder ||..o{ Image : "monitors (no FK)" +``` + +### Full mode — add folder triggers an enqueue-and-return sync + +```mermaid +sequenceDiagram + actor User + participant View as LibraryFolderAdd view + participant Trigger as trigger_folder_sync uc + participant Sync as sync_folder uc + participant Worker as Celery worker(s) + participant PSI as process_synced_image uc + participant DB + + User->>View: POST /library/new/ + View->>Trigger: trigger_folder_sync(folder_id) + Trigger->>Trigger: worker reachable? (else error, no run) + Trigger->>Sync: sync_folder(folder_id) + Sync->>DB: start_sync_run (SCANNING) + Sync->>Sync: walk folder, diff vs known paths + Sync->>DB: begin_processing(total=N) + Sync-->>Worker: enqueue N sync_process_image tasks + Sync-->>View: return + View-->>User: redirect to /library/ + loop each task (concurrent) + Worker->>PSI: process_synced_image(path, run_id) + PSI->>DB: process_image + record_processed/skipped/error (F() atomic) + PSI->>DB: complete_sync_run if all accounted (conditional, one winner) + end + User->>View: folder row polls sync-status every 2s (HTMX) +``` + +### Lite mode — add folder triggers a background thread + +```mermaid +sequenceDiagram + actor User + participant View as LibraryFolderAdd view + participant Trigger as trigger_folder_sync uc + participant BG as background.run_in_background + participant Sync as sync_folder uc + participant PSI as process_synced_image uc + participant DB + + User->>View: POST /library/new/ + View->>Trigger: trigger_folder_sync(folder_id) + Trigger->>BG: run_in_background(sync_folder, folder_id) + Trigger-->>View: return + View-->>User: redirect to /library/ + Note over BG: daemon thread, outlives the request + BG->>Sync: sync_folder(folder_id) + Sync->>DB: start_sync_run (SCANNING) → begin_processing(total=N) + loop each new path (sequential) + Sync->>PSI: process_synced_image(path, run_id) + PSI->>DB: process_image + record progress + end + PSI->>DB: complete_sync_run (COMPLETED) + Note over User,DB: User navigates away and back. The thread keeps running
and the row re-reads DB state on the next poll. +``` + +--- + +## Progress tracking + +### Options considered + +**Option 1 — Introspect the Celery queue / result backend.** Derive progress in full mode from broker queue depth or `inspect()`, or from a `GroupResult.completed_count()`. + +*Rejected.* Broker/inspect counts are cluster-wide, not per-folder, and imperfect (queue depth vs reserved vs active); polling `inspect()` is a broadcast RPC. The clean `group`/`GroupResult` route needs a real result backend, but the app is configured with `rpc://`. None of it helps lite mode, which has no broker at all. + +**Option 2 — A dedicated `SyncRun` table (chosen).** Each task/thread reports against a per-folder run row. + +*Chosen.* It is per-folder-accurate, backend-agnostic, and identical across modes: lite's thread and full's tasks increment the **same** model, read by the **same** HTMX status endpoint. Under concurrent Celery workers, counters use atomic `F()` increments and completion is a conditional `UPDATE ... WHERE state = 'PROCESSING'` so exactly one finisher transitions the run and emits the completion event. + +--- + +## Consequences + +- New `SyncRun` model and migration; a `services/background.py` thread runner; `sync_folder`, `process_synced_image`, and `trigger_folder_sync` use cases; a `sync_process_image_task`; a sync-status view with HTMX polling in the folder row. +- `sync_library` becomes an all-folders loop over `sync_folder` that first interrupts dangling runs. The `manage.py sync_library` entry point and its result contract are unchanged. +- `process_image` and the generic `process_image_task` are untouched, so the manual `import` path is unaffected. +- Lite installs run with SQLite WAL enabled (a persistent, idempotent property of the database file). +- A new Celery task means the worker must be restarted after deploying this change before full-mode syncs will be processed; until then those messages are discarded and the next `make start` re-syncs. + +--- + +## Interface layer + +| Artifact | Location | +|---|---| +| Trigger use case | `src/application/usecases/library/trigger_folder_sync.py` | +| Single-folder sync use case | `src/application/usecases/library/sync_folder.py` | +| Per-image use case | `src/application/usecases/library/process_synced_image.py` | +| Background runner | `src/services/background.py` | +| Celery task | `sync_process_image_task` in `src/interfaces/tasks.py` | +| Status view | `LibraryFolderSyncStatus` → `library//sync-status/` | +| Status partial | `src/interfaces/templates/library/partials/sync_status.html` | + +The add and path-update views call `trigger_folder_sync` after the folder mutation and map `CeleryWorkerUnavailable` to a Library-page error. The folder row lazy-loads its status partial on load and, while a run is active, polls the status URL every 2 seconds, swapping to a terminal summary (which drops the poll trigger) when the run finishes. diff --git a/docs/index.md b/docs/index.md index 0c90e75..ce39280 100644 --- a/docs/index.md +++ b/docs/index.md @@ -34,4 +34,5 @@ - [ADR 008 — Recipe Versioning via Generalised Grouping](ADRs/008-recipe-versioning.md) — version lines and recipe families via a shared grouping abstraction - [ADR 009 — Moving a Recipe Between Version Lines](ADRs/009-move-recipe-between-version-lines.md) — reassigning an existing recipe to a different VERSION_LINE group while keeping positions contiguous - [ADR 010 — Image Library: Folder Monitoring and Catalog Sync](ADRs/010-image-library.md) — persisting monitored folders and detecting and importing new images automatically at startup +- [ADR 011 — Library Sync on Folder Add/Update](ADRs/011-library-sync-on-folder-change.md) — triggering a single-folder sync from the Library page with server-side, DB-backed progress in both install modes From da64f78658dbba72889f03b158a723731881520f Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:41:15 +0200 Subject: [PATCH 03/27] Configure SQLite WAL and busy_timeout Co-Authored-By: Claude Opus 4.8 --- src/config/settings.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/config/settings.py b/src/config/settings.py index d502524..03919bc 100644 --- a/src/config/settings.py +++ b/src/config/settings.py @@ -42,6 +42,16 @@ } } +if DB_ENGINE.endswith("sqlite3"): + # Lite mode runs SQLite with a background sync thread writing while the web + # request threads read. WAL lets readers proceed alongside the single writer; + # the busy timeout makes a colliding writer wait rather than raise "database + # is locked". journal_mode is persistent on the file (idempotent to re-set). + DATABASES["default"].setdefault("OPTIONS", {}).update({ + "timeout": 5, # SQLite busy_timeout, applied per connection + "init_command": "PRAGMA journal_mode=WAL;", + }) + DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField" PTP_DEVICE: str = env.str("PTP_DEVICE", default="src.domain.camera.ptp_usb_device.PTPUSBDevice") # dotted import path to the PTP device implementation; swap for a stub/mock in tests From 580b47e9819cea49d0fa7b5ceb2d5a3a1c1fc306 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:47:35 +0200 Subject: [PATCH 04/27] Add SyncRun model Co-Authored-By: Claude Opus 4.8 --- src/data/migrations/0034_syncrun.py | 36 +++++++++ src/data/models/__init__.py | 2 + src/data/models/_sync_run.py | 114 ++++++++++++++++++++++++++++ tests/factories.py | 10 +++ 4 files changed, 162 insertions(+) create mode 100644 src/data/migrations/0034_syncrun.py create mode 100644 src/data/models/_sync_run.py diff --git a/src/data/migrations/0034_syncrun.py b/src/data/migrations/0034_syncrun.py new file mode 100644 index 0000000..d1cc149 --- /dev/null +++ b/src/data/migrations/0034_syncrun.py @@ -0,0 +1,36 @@ +# Generated by Django 6.0.3 on 2026-07-02 15:45 + +import django.db.models.deletion +import django.utils.timezone +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('data', '0033_libraryfolder'), + ] + + operations = [ + migrations.CreateModel( + name='SyncRun', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('state', models.CharField(max_length=16)), + ('total', models.IntegerField(null=True)), + ('processed', models.IntegerField(default=0)), + ('skipped', models.IntegerField(default=0)), + ('errors', models.IntegerField(default=0)), + ('error_message', models.TextField(null=True)), + ('started_at', models.DateTimeField(default=django.utils.timezone.now)), + ('finished_at', models.DateTimeField(null=True)), + ('created_at', models.DateTimeField(default=django.utils.timezone.now)), + ('updated_at', models.DateTimeField(auto_now=True)), + ('folder', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='sync_runs', to='data.libraryfolder')), + ], + options={ + 'indexes': [models.Index(fields=['folder', '-started_at'], name='idx_sync_run_folder_started')], + 'constraints': [models.UniqueConstraint(condition=models.Q(('state__in', ('SCANNING', 'PROCESSING'))), fields=('folder',), name='unique_active_sync_run_per_folder')], + }, + ), + ] diff --git a/src/data/models/__init__.py b/src/data/models/__init__.py index be09f0d..1a240b6 100644 --- a/src/data/models/__init__.py +++ b/src/data/models/__init__.py @@ -8,6 +8,7 @@ RecipeGroupMember, Sensor, ) +from ._sync_run import SyncRun __all__ = [ "RECIPE_FIELDS", @@ -20,4 +21,5 @@ "RecipeGroup", "RecipeGroupMember", "Sensor", + "SyncRun", ] diff --git a/src/data/models/_sync_run.py b/src/data/models/_sync_run.py new file mode 100644 index 0000000..ad18021 --- /dev/null +++ b/src/data/models/_sync_run.py @@ -0,0 +1,114 @@ +from django.db import models +from django.utils import timezone + +from ._library import LibraryFolder + +_STATE_SCANNING = "SCANNING" +_STATE_PROCESSING = "PROCESSING" +_STATE_COMPLETED = "COMPLETED" +_STATE_FAILED = "FAILED" +_STATE_INTERRUPTED = "INTERRUPTED" + +# A run is "active" while it is scanning or processing; at most one active run is +# allowed per folder (enforced by a conditional UniqueConstraint below). +_ACTIVE_STATES = (_STATE_SCANNING, _STATE_PROCESSING) + +_STATE_MAX_LEN = 16 + + +class SyncRun(models.Model): + STATE_SCANNING = _STATE_SCANNING + STATE_PROCESSING = _STATE_PROCESSING + STATE_COMPLETED = _STATE_COMPLETED + STATE_FAILED = _STATE_FAILED + STATE_INTERRUPTED = _STATE_INTERRUPTED + ACTIVE_STATES = _ACTIVE_STATES + + folder = models.ForeignKey( + LibraryFolder, + on_delete=models.CASCADE, + related_name="sync_runs", + ) + state = models.CharField(max_length=_STATE_MAX_LEN) + total = models.IntegerField(null=True) # unknown during the scanning phase + processed = models.IntegerField(default=0) + skipped = models.IntegerField(default=0) + errors = models.IntegerField(default=0) + error_message = models.TextField(null=True) + started_at = models.DateTimeField(default=timezone.now) + finished_at = models.DateTimeField(null=True) + created_at = models.DateTimeField(default=timezone.now) + updated_at = models.DateTimeField(auto_now=True) + + class Meta: + indexes = [ + models.Index(fields=["folder", "-started_at"], name="idx_sync_run_folder_started"), + ] + constraints = [ + models.UniqueConstraint( + fields=["folder"], + condition=models.Q(state__in=_ACTIVE_STATES), + name="unique_active_sync_run_per_folder", + ), + ] + + # Factories + + @classmethod + def create(cls, *, folder: LibraryFolder) -> "SyncRun": + return cls.objects.create(folder=folder, state=_STATE_SCANNING) + + # Mutators + + def begin_processing(self, *, total: int) -> None: + self.state = _STATE_PROCESSING + self.total = total + self.save(update_fields=["state", "total", "updated_at"]) + + def record_processed(self) -> None: + self._increment("processed") + + def record_skipped(self) -> None: + self._increment("skipped") + + def record_error(self) -> None: + self._increment("errors") + + def _increment(self, field: str) -> None: + # Atomic increment so concurrent Celery tasks reporting against the same + # run never lose a count. The caller must refresh_from_db() to read the + # updated value. + type(self).objects.filter(pk=self.pk).update( + updated_at=timezone.now(), + **{field: models.F(field) + 1}, + ) + + def mark_completed(self) -> bool: + # Conditional so exactly one caller wins the finalize under concurrency. + # Returns True if this call transitioned the run to COMPLETED. + now = timezone.now() + rows = type(self).objects.filter(pk=self.pk, state=_STATE_PROCESSING).update( + state=_STATE_COMPLETED, + finished_at=now, + updated_at=now, + ) + return rows > 0 + + def mark_failed(self, *, message: str) -> None: + self.state = _STATE_FAILED + self.error_message = message + self.finished_at = timezone.now() + self.save(update_fields=["state", "error_message", "finished_at", "updated_at"]) + + # Queries + + def all_images_accounted_for(self) -> bool: + """ + Return True once every image in the run has a terminal outcome. + """ + if self.total is None: + return False + return self.processed + self.skipped + self.errors >= self.total + + def __str__(self) -> str: + return f"#{self.id} {self.state} folder #{self.folder_id}" diff --git a/tests/factories.py b/tests/factories.py index 27ece83..ad8b79f 100644 --- a/tests/factories.py +++ b/tests/factories.py @@ -100,3 +100,13 @@ class Meta: # path has a unique constraint, so use a sequence to avoid collisions. path = factory.Sequence(lambda n: f"/photos/library_{n:04d}") + + +class SyncRunFactory(factory.django.DjangoModelFactory): + class Meta: + model = models.SyncRun + + folder = factory.SubFactory(LibraryFolderFactory) + state = models.SyncRun.STATE_SCANNING + # total defaults to None (scanning phase); counters default to 0. Override + # per-test, e.g. SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=3). From 364235f8473caeb86cf49ef8ccd29696eac8067c Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:48:03 +0200 Subject: [PATCH 05/27] Add clear_last_checked_at mutator to LibraryFolder Co-Authored-By: Claude Opus 4.8 --- src/data/models/_library.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/data/models/_library.py b/src/data/models/_library.py index 9004285..64ce591 100644 --- a/src/data/models/_library.py +++ b/src/data/models/_library.py @@ -41,5 +41,9 @@ def set_last_checked_at(self, *, value: datetime) -> None: self.last_checked_at = value self.save(update_fields=["last_checked_at", "updated_at"]) + def clear_last_checked_at(self) -> None: + self.last_checked_at = None + self.save(update_fields=["last_checked_at", "updated_at"]) + def __str__(self) -> str: return f"#{self.id} {self.path}" From a609857e3b9fcf1f58e21a03c8ef11444c736e9e Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:49:30 +0200 Subject: [PATCH 06/27] Add get_latest_sync_run domain query Co-Authored-By: Claude Opus 4.8 --- src/domain/library/queries.py | 12 +++++++ .../domain/library/test_queries.py | 32 ++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletion(-) diff --git a/src/domain/library/queries.py b/src/domain/library/queries.py index 9283c72..0a36986 100644 --- a/src/domain/library/queries.py +++ b/src/domain/library/queries.py @@ -59,3 +59,15 @@ def list_subdirectories(*, path: str) -> tuple[str, ...]: if entry.is_dir() and not entry.name.startswith(".") ) return tuple(entries) + + +def get_latest_sync_run(*, folder_id: int) -> models.SyncRun | None: + """ + Return the most recently started sync run for *folder_id*, or None if the + folder has never been synced. + """ + return ( + models.SyncRun.objects.filter(folder_id=folder_id) + .order_by("-started_at", "-id") + .first() + ) diff --git a/tests/integration/domain/library/test_queries.py b/tests/integration/domain/library/test_queries.py index f0872b2..992f161 100644 --- a/tests/integration/domain/library/test_queries.py +++ b/tests/integration/domain/library/test_queries.py @@ -1,13 +1,16 @@ import pytest +import time_machine +from src.data import models from src.domain.library.queries import ( FolderNotFound, LibraryFolderNotFound, get_all_library_folders, + get_latest_sync_run, get_library_folder, list_subdirectories, ) -from tests.factories import LibraryFolderFactory +from tests.factories import LibraryFolderFactory, SyncRunFactory @pytest.mark.django_db @@ -95,3 +98,30 @@ def test_raises_folder_not_found_for_file_path(self, tmp_path): with pytest.raises(FolderNotFound) as exc_info: list_subdirectories(path=str(file_path)) assert exc_info.value.path == str(file_path) + + +@pytest.mark.django_db +class TestGetLatestSyncRun: + def test_returns_none_when_folder_never_synced(self): + folder = LibraryFolderFactory() + assert get_latest_sync_run(folder_id=folder.pk) is None + + def test_returns_the_most_recently_started_run(self): + folder = LibraryFolderFactory() + # Only one run may be active per folder, so the earlier one is terminal. + with time_machine.travel("2026-07-01", tick=False): + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_COMPLETED) + with time_machine.travel("2026-07-02", tick=False): + latest = SyncRunFactory(folder=folder) + + result = get_latest_sync_run(folder_id=folder.pk) + + assert result is not None + assert result.pk == latest.pk + + def test_ignores_runs_for_other_folders(self): + folder = LibraryFolderFactory() + other = LibraryFolderFactory() + SyncRunFactory(folder=other) + + assert get_latest_sync_run(folder_id=folder.pk) is None From c30db9a01ea2f1edcd3b800576caa11cd2cf7c37 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:50:11 +0200 Subject: [PATCH 07/27] Add get_active_sync_run domain query Co-Authored-By: Claude Opus 4.8 --- src/domain/library/queries.py | 16 +++++++++ .../domain/library/test_queries.py | 35 +++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/src/domain/library/queries.py b/src/domain/library/queries.py index 0a36986..5a5dd2b 100644 --- a/src/domain/library/queries.py +++ b/src/domain/library/queries.py @@ -71,3 +71,19 @@ def get_latest_sync_run(*, folder_id: int) -> models.SyncRun | None: .order_by("-started_at", "-id") .first() ) + + +def get_active_sync_run(*, folder_id: int) -> models.SyncRun | None: + """ + Return the in-progress (scanning or processing) sync run for *folder_id*, or + None if no run is currently active. At most one active run can exist per + folder (enforced by a database constraint). + """ + return ( + models.SyncRun.objects.filter( + folder_id=folder_id, + state__in=models.SyncRun.ACTIVE_STATES, + ) + .order_by("-started_at", "-id") + .first() + ) diff --git a/tests/integration/domain/library/test_queries.py b/tests/integration/domain/library/test_queries.py index 992f161..cd70861 100644 --- a/tests/integration/domain/library/test_queries.py +++ b/tests/integration/domain/library/test_queries.py @@ -5,6 +5,7 @@ from src.domain.library.queries import ( FolderNotFound, LibraryFolderNotFound, + get_active_sync_run, get_all_library_folders, get_latest_sync_run, get_library_folder, @@ -125,3 +126,37 @@ def test_ignores_runs_for_other_folders(self): SyncRunFactory(folder=other) assert get_latest_sync_run(folder_id=folder.pk) is None + + +@pytest.mark.django_db +class TestGetActiveSyncRun: + def test_returns_none_when_no_run_active(self): + folder = LibraryFolderFactory() + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_COMPLETED) + + assert get_active_sync_run(folder_id=folder.pk) is None + + def test_returns_scanning_run(self): + folder = LibraryFolderFactory() + run = SyncRunFactory(folder=folder, state=models.SyncRun.STATE_SCANNING) + + result = get_active_sync_run(folder_id=folder.pk) + + assert result is not None + assert result.pk == run.pk + + def test_returns_processing_run(self): + folder = LibraryFolderFactory() + run = SyncRunFactory(folder=folder, state=models.SyncRun.STATE_PROCESSING, total=3) + + result = get_active_sync_run(folder_id=folder.pk) + + assert result is not None + assert result.pk == run.pk + + def test_ignores_active_runs_for_other_folders(self): + folder = LibraryFolderFactory() + other = LibraryFolderFactory() + SyncRunFactory(folder=other, state=models.SyncRun.STATE_PROCESSING, total=1) + + assert get_active_sync_run(folder_id=folder.pk) is None From ff3674f83ef428582ea655c5ec02f9be10c7a33e Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:51:59 +0200 Subject: [PATCH 08/27] Add start_sync_run domain operation Co-Authored-By: Claude Opus 4.8 --- src/domain/library/events.py | 1 + src/domain/library/operations.py | 30 +++++++++++++ .../domain/library/test_operations.py | 42 ++++++++++++++++++- 3 files changed, 72 insertions(+), 1 deletion(-) diff --git a/src/domain/library/events.py b/src/domain/library/events.py index 5dac612..53e9767 100644 --- a/src/domain/library/events.py +++ b/src/domain/library/events.py @@ -7,6 +7,7 @@ LIBRARY_FOLDER_ADDED = "library.folder.added" LIBRARY_FOLDER_REMOVED = "library.folder.removed" LIBRARY_FOLDER_PATH_UPDATED = "library.folder.path.updated" +LIBRARY_SYNC_RUN_STARTED = "library.sync.run.started" def publish_event(*, event_type: str, **kwargs: object) -> None: diff --git a/src/domain/library/operations.py b/src/domain/library/operations.py index 75252e6..3b26582 100644 --- a/src/domain/library/operations.py +++ b/src/domain/library/operations.py @@ -17,6 +17,15 @@ class FolderAlreadyInLibrary(Exception): path: str +@attrs.frozen +class SyncAlreadyInProgress(Exception): + """ + Raised when a sync run is started for a folder that already has an active run. + """ + + folder_id: int + + def _normalize_path(path: str) -> str: return str(Path(path).expanduser().resolve()) @@ -97,3 +106,24 @@ def update_library_folder_path(*, folder_id: int, path: str) -> models.LibraryFo path=folder.path, ) return folder + + +def start_sync_run(*, folder: models.LibraryFolder) -> models.SyncRun: + """ + Create a new sync run for *folder* in the scanning state. + + :raises SyncAlreadyInProgress: If *folder* already has an active (scanning or + processing) run. + """ + try: + with transaction.atomic(): + run = models.SyncRun.create(folder=folder) + except IntegrityError: + raise SyncAlreadyInProgress(folder_id=folder.pk) + + events.publish_event( + event_type=events.LIBRARY_SYNC_RUN_STARTED, + run_id=run.pk, + folder_id=folder.pk, + ) + return run diff --git a/tests/integration/domain/library/test_operations.py b/tests/integration/domain/library/test_operations.py index 68f5668..c664682 100644 --- a/tests/integration/domain/library/test_operations.py +++ b/tests/integration/domain/library/test_operations.py @@ -4,12 +4,14 @@ from src.domain.library import events from src.domain.library.operations import ( FolderAlreadyInLibrary, + SyncAlreadyInProgress, add_library_folder, remove_library_folder, + start_sync_run, update_library_folder_path, ) from src.domain.library.queries import FolderNotFound, LibraryFolderNotFound -from tests.factories import LibraryFolderFactory +from tests.factories import LibraryFolderFactory, SyncRunFactory @pytest.mark.django_db @@ -127,3 +129,41 @@ def test_raises_folder_already_in_library_when_path_taken(self, tmp_path): with pytest.raises(FolderAlreadyInLibrary) as exc_info: update_library_folder_path(folder_id=folder_b.pk, path=str(dir_a)) assert exc_info.value.path == str(dir_a) + + +@pytest.mark.django_db +class TestStartSyncRun: + def test_creates_scanning_run(self): + folder = LibraryFolderFactory() + + run = start_sync_run(folder=folder) + + assert models.SyncRun.objects.filter(pk=run.pk).exists() + assert run.state == models.SyncRun.STATE_SCANNING + assert run.folder_id == folder.pk + + def test_publishes_sync_run_started_event(self, captured_logs): + folder = LibraryFolderFactory() + + run = start_sync_run(folder=folder) + + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_STARTED] + assert len(matching) == 1 + assert matching[0]["run_id"] == run.pk + assert matching[0]["folder_id"] == folder.pk + + def test_raises_when_folder_already_has_active_run(self): + folder = LibraryFolderFactory() + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_PROCESSING, total=1) + + with pytest.raises(SyncAlreadyInProgress) as exc_info: + start_sync_run(folder=folder) + assert exc_info.value.folder_id == folder.pk + + def test_allows_new_run_after_previous_completed(self): + folder = LibraryFolderFactory() + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_COMPLETED) + + run = start_sync_run(folder=folder) + + assert run.state == models.SyncRun.STATE_SCANNING From d0b7818606e6045fa591c76950aafd0992868754 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:52:30 +0200 Subject: [PATCH 09/27] Add complete_sync_run domain operation Co-Authored-By: Claude Opus 4.8 --- src/domain/library/events.py | 1 + src/domain/library/operations.py | 18 ++++++++++ .../domain/library/test_operations.py | 33 +++++++++++++++++++ 3 files changed, 52 insertions(+) diff --git a/src/domain/library/events.py b/src/domain/library/events.py index 53e9767..747363d 100644 --- a/src/domain/library/events.py +++ b/src/domain/library/events.py @@ -8,6 +8,7 @@ LIBRARY_FOLDER_REMOVED = "library.folder.removed" LIBRARY_FOLDER_PATH_UPDATED = "library.folder.path.updated" LIBRARY_SYNC_RUN_STARTED = "library.sync.run.started" +LIBRARY_SYNC_RUN_COMPLETED = "library.sync.run.completed" def publish_event(*, event_type: str, **kwargs: object) -> None: diff --git a/src/domain/library/operations.py b/src/domain/library/operations.py index 3b26582..e9cb98b 100644 --- a/src/domain/library/operations.py +++ b/src/domain/library/operations.py @@ -127,3 +127,21 @@ def start_sync_run(*, folder: models.LibraryFolder) -> models.SyncRun: folder_id=folder.pk, ) return run + + +def complete_sync_run(*, run: models.SyncRun) -> bool: + """ + Mark *run* as completed if it is still processing. + + Uses a conditional update so that, under concurrent workers, exactly one + caller transitions the run and publishes the completion event. Returns True + if this call completed the run. + """ + completed = run.mark_completed() + if completed: + events.publish_event( + event_type=events.LIBRARY_SYNC_RUN_COMPLETED, + run_id=run.pk, + folder_id=run.folder_id, + ) + return completed diff --git a/tests/integration/domain/library/test_operations.py b/tests/integration/domain/library/test_operations.py index c664682..3fb38c1 100644 --- a/tests/integration/domain/library/test_operations.py +++ b/tests/integration/domain/library/test_operations.py @@ -6,6 +6,7 @@ FolderAlreadyInLibrary, SyncAlreadyInProgress, add_library_folder, + complete_sync_run, remove_library_folder, start_sync_run, update_library_folder_path, @@ -167,3 +168,35 @@ def test_allows_new_run_after_previous_completed(self): run = start_sync_run(folder=folder) assert run.state == models.SyncRun.STATE_SCANNING + + +@pytest.mark.django_db +class TestCompleteSyncRun: + def test_transitions_processing_run_to_completed(self): + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + result = complete_sync_run(run=run) + + assert result is True + run.refresh_from_db() + assert run.state == models.SyncRun.STATE_COMPLETED + assert run.finished_at is not None + + def test_publishes_sync_run_completed_event(self, captured_logs): + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + complete_sync_run(run=run) + + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_COMPLETED] + assert len(matching) == 1 + assert matching[0]["run_id"] == run.pk + assert matching[0]["folder_id"] == run.folder_id + + def test_returns_false_and_publishes_nothing_when_already_completed(self, captured_logs): + run = SyncRunFactory(state=models.SyncRun.STATE_COMPLETED, total=1) + + result = complete_sync_run(run=run) + + assert result is False + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_COMPLETED] + assert matching == [] From d5f3c680c62fd6bd85c7234f953e7df0077138f0 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:53:14 +0200 Subject: [PATCH 10/27] Add fail_sync_run domain operation Co-Authored-By: Claude Opus 4.8 --- src/domain/library/events.py | 1 + src/domain/library/operations.py | 13 ++++++++++ .../domain/library/test_operations.py | 25 +++++++++++++++++++ 3 files changed, 39 insertions(+) diff --git a/src/domain/library/events.py b/src/domain/library/events.py index 747363d..5d40007 100644 --- a/src/domain/library/events.py +++ b/src/domain/library/events.py @@ -9,6 +9,7 @@ LIBRARY_FOLDER_PATH_UPDATED = "library.folder.path.updated" LIBRARY_SYNC_RUN_STARTED = "library.sync.run.started" LIBRARY_SYNC_RUN_COMPLETED = "library.sync.run.completed" +LIBRARY_SYNC_RUN_FAILED = "library.sync.run.failed" def publish_event(*, event_type: str, **kwargs: object) -> None: diff --git a/src/domain/library/operations.py b/src/domain/library/operations.py index e9cb98b..bd71efa 100644 --- a/src/domain/library/operations.py +++ b/src/domain/library/operations.py @@ -145,3 +145,16 @@ def complete_sync_run(*, run: models.SyncRun) -> bool: folder_id=run.folder_id, ) return completed + + +def fail_sync_run(*, run: models.SyncRun, message: str) -> None: + """ + Mark *run* as failed, recording *message* as the failure reason. + """ + run.mark_failed(message=message) + events.publish_event( + event_type=events.LIBRARY_SYNC_RUN_FAILED, + run_id=run.pk, + folder_id=run.folder_id, + reason=message, + ) diff --git a/tests/integration/domain/library/test_operations.py b/tests/integration/domain/library/test_operations.py index 3fb38c1..b8d0eef 100644 --- a/tests/integration/domain/library/test_operations.py +++ b/tests/integration/domain/library/test_operations.py @@ -7,6 +7,7 @@ SyncAlreadyInProgress, add_library_folder, complete_sync_run, + fail_sync_run, remove_library_folder, start_sync_run, update_library_folder_path, @@ -200,3 +201,27 @@ def test_returns_false_and_publishes_nothing_when_already_completed(self, captur assert result is False matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_COMPLETED] assert matching == [] + + +@pytest.mark.django_db +class TestFailSyncRun: + def test_marks_run_failed_with_message(self): + run = SyncRunFactory(state=models.SyncRun.STATE_SCANNING) + + fail_sync_run(run=run, message="folder no longer exists") + + run.refresh_from_db() + assert run.state == models.SyncRun.STATE_FAILED + assert run.error_message == "folder no longer exists" + assert run.finished_at is not None + + def test_publishes_sync_run_failed_event(self, captured_logs): + run = SyncRunFactory(state=models.SyncRun.STATE_SCANNING) + + fail_sync_run(run=run, message="boom") + + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_FAILED] + assert len(matching) == 1 + assert matching[0]["run_id"] == run.pk + assert matching[0]["folder_id"] == run.folder_id + assert matching[0]["reason"] == "boom" From e2eb689ca9e12f78b4d20ac6d756d7c797f5beca Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:53:52 +0200 Subject: [PATCH 11/27] Add interrupt_active_sync_runs domain operation Co-Authored-By: Claude Opus 4.8 --- src/domain/library/events.py | 1 + src/domain/library/operations.py | 24 +++++++++++ .../domain/library/test_operations.py | 40 +++++++++++++++++++ 3 files changed, 65 insertions(+) diff --git a/src/domain/library/events.py b/src/domain/library/events.py index 5d40007..7888150 100644 --- a/src/domain/library/events.py +++ b/src/domain/library/events.py @@ -10,6 +10,7 @@ LIBRARY_SYNC_RUN_STARTED = "library.sync.run.started" LIBRARY_SYNC_RUN_COMPLETED = "library.sync.run.completed" LIBRARY_SYNC_RUN_FAILED = "library.sync.run.failed" +LIBRARY_SYNC_RUN_INTERRUPTED = "library.sync.run.interrupted" def publish_event(*, event_type: str, **kwargs: object) -> None: diff --git a/src/domain/library/operations.py b/src/domain/library/operations.py index bd71efa..d24a4ab 100644 --- a/src/domain/library/operations.py +++ b/src/domain/library/operations.py @@ -2,6 +2,7 @@ from pathlib import Path from django.db import IntegrityError, transaction +from django.utils import timezone from src.data import models from src.domain.library import events @@ -158,3 +159,26 @@ def fail_sync_run(*, run: models.SyncRun, message: str) -> None: folder_id=run.folder_id, reason=message, ) + + +def interrupt_active_sync_runs() -> int: + """ + Mark every active (scanning or processing) sync run as interrupted. + + Called at startup to recover runs abandoned by a killed process, so no run + is left permanently active. Returns the number of runs interrupted. + """ + now = timezone.now() + count = models.SyncRun.objects.filter( + state__in=models.SyncRun.ACTIVE_STATES, + ).update( + state=models.SyncRun.STATE_INTERRUPTED, + finished_at=now, + updated_at=now, + ) + if count: + events.publish_event( + event_type=events.LIBRARY_SYNC_RUN_INTERRUPTED, + count=count, + ) + return count diff --git a/tests/integration/domain/library/test_operations.py b/tests/integration/domain/library/test_operations.py index b8d0eef..7f0c536 100644 --- a/tests/integration/domain/library/test_operations.py +++ b/tests/integration/domain/library/test_operations.py @@ -8,6 +8,7 @@ add_library_folder, complete_sync_run, fail_sync_run, + interrupt_active_sync_runs, remove_library_folder, start_sync_run, update_library_folder_path, @@ -225,3 +226,42 @@ def test_publishes_sync_run_failed_event(self, captured_logs): assert matching[0]["run_id"] == run.pk assert matching[0]["folder_id"] == run.folder_id assert matching[0]["reason"] == "boom" + + +@pytest.mark.django_db +class TestInterruptActiveSyncRuns: + def test_marks_scanning_and_processing_runs_interrupted(self): + scanning = SyncRunFactory(state=models.SyncRun.STATE_SCANNING) + processing = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=2) + + count = interrupt_active_sync_runs() + + assert count == 2 + for run in (scanning, processing): + run.refresh_from_db() + assert run.state == models.SyncRun.STATE_INTERRUPTED + assert run.finished_at is not None + + def test_leaves_terminal_runs_untouched(self): + completed = SyncRunFactory(state=models.SyncRun.STATE_COMPLETED) + + count = interrupt_active_sync_runs() + + assert count == 0 + completed.refresh_from_db() + assert completed.state == models.SyncRun.STATE_COMPLETED + + def test_publishes_event_with_count_when_runs_interrupted(self, captured_logs): + SyncRunFactory(state=models.SyncRun.STATE_SCANNING) + + interrupt_active_sync_runs() + + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_INTERRUPTED] + assert len(matching) == 1 + assert matching[0]["count"] == 1 + + def test_publishes_nothing_when_no_active_runs(self, captured_logs): + interrupt_active_sync_runs() + + matching = [e for e in captured_logs if e.get("event_type") == events.LIBRARY_SYNC_RUN_INTERRUPTED] + assert matching == [] From 684697123ba5e194a70a4a5a3df321f374665c39 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:54:16 +0200 Subject: [PATCH 12/27] Reset last_checked_at when library folder path changes Co-Authored-By: Claude Opus 4.8 --- src/domain/library/operations.py | 5 +++++ tests/integration/domain/library/test_operations.py | 13 +++++++++++++ 2 files changed, 18 insertions(+) diff --git a/src/domain/library/operations.py b/src/domain/library/operations.py index d24a4ab..80dd424 100644 --- a/src/domain/library/operations.py +++ b/src/domain/library/operations.py @@ -101,6 +101,11 @@ def update_library_folder_path(*, folder_id: int, path: str) -> models.LibraryFo except IntegrityError: raise FolderAlreadyInLibrary(path=normalized) + # The new path is a different tree, so the previous scan timestamp no longer + # applies. Clearing it forces a full rescan (mtime gating would otherwise skip + # directories older than the old last_checked_at). + folder.clear_last_checked_at() + events.publish_event( event_type=events.LIBRARY_FOLDER_PATH_UPDATED, folder_id=folder.pk, diff --git a/tests/integration/domain/library/test_operations.py b/tests/integration/domain/library/test_operations.py index 7f0c536..9f9a6e1 100644 --- a/tests/integration/domain/library/test_operations.py +++ b/tests/integration/domain/library/test_operations.py @@ -1,4 +1,5 @@ import pytest +from django.utils import timezone from src.data import models from src.domain.library import events @@ -94,6 +95,18 @@ def test_updates_path_on_folder(self, tmp_path): folder.refresh_from_db() assert folder.path == str(new_dir) + def test_resets_last_checked_at_so_new_tree_is_fully_rescanned(self, tmp_path): + old_dir = tmp_path / "old" + new_dir = tmp_path / "new" + old_dir.mkdir() + new_dir.mkdir() + + folder = LibraryFolderFactory(path=str(old_dir), last_checked_at=timezone.now()) + update_library_folder_path(folder_id=folder.pk, path=str(new_dir)) + + folder.refresh_from_db() + assert folder.last_checked_at is None + def test_publishes_folder_path_updated_event(self, tmp_path, captured_logs): old_dir = tmp_path / "old" new_dir = tmp_path / "new" From 8f82fb691f9ce7264b897393578a33df5e9d1256 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:56:26 +0200 Subject: [PATCH 13/27] Add background runner service Co-Authored-By: Claude Opus 4.8 --- src/services/background.py | 30 ++++++++++++++++ tests/unit/services/test_background.py | 48 ++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 src/services/background.py create mode 100644 tests/unit/services/test_background.py diff --git a/src/services/background.py b/src/services/background.py new file mode 100644 index 0000000..4d62f4b --- /dev/null +++ b/src/services/background.py @@ -0,0 +1,30 @@ +import threading +from collections.abc import Callable + +import structlog +from django import db + +logger = structlog.get_logger("services.background") + + +def run_in_background(func: Callable[..., object], /, **kwargs: object) -> None: + """ + Run ``func(**kwargs)`` in a daemon thread, detached from the calling request. + + The thread outlives the request that started it, so the caller returns + immediately. Used in lite mode (no Celery) to keep a long sync off the request + thread while still running server-side. Any unexpected error is logged to + Sentry; the thread always closes its own database connection on exit, since + Django only auto-closes connections at request boundaries. + """ + + def target() -> None: + try: + func(**kwargs) + except Exception: + logger.exception("Background task failed") + finally: + db.connection.close() + + thread = threading.Thread(target=target, daemon=True) + thread.start() diff --git a/tests/unit/services/test_background.py b/tests/unit/services/test_background.py new file mode 100644 index 0000000..69b4913 --- /dev/null +++ b/tests/unit/services/test_background.py @@ -0,0 +1,48 @@ +import threading +from unittest.mock import MagicMock, patch + +from src.services.background import run_in_background + + +class TestRunInBackground: + def test_runs_func_with_kwargs(self): + done = threading.Event() + captured: dict[str, object] = {} + + def work(*, a: int, b: int) -> None: + captured["a"] = a + captured["b"] = b + done.set() + + with patch("src.services.background.db", MagicMock()): + run_in_background(work, a=1, b=2) + + assert done.wait(timeout=2) + assert captured == {"a": 1, "b": 2} + + def test_swallows_and_logs_unexpected_exception(self): + logged = threading.Event() + + def boom() -> None: + raise ValueError("nope") + + with ( + patch("src.services.background.logger") as mock_logger, + patch("src.services.background.db", MagicMock()), + ): + mock_logger.exception.side_effect = lambda *a, **k: logged.set() + run_in_background(boom) + assert logged.wait(timeout=2) + + mock_logger.exception.assert_called_once() + + def test_closes_db_connection_on_exit(self): + closed = threading.Event() + fake_db = MagicMock() + fake_db.connection.close.side_effect = lambda: closed.set() + + with patch("src.services.background.db", fake_db): + run_in_background(lambda: None) + assert closed.wait(timeout=2) + + fake_db.connection.close.assert_called_once() From 770bc65728978db957832645006c2b942ac03cea Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 17:57:36 +0200 Subject: [PATCH 14/27] Add get_sync_run domain query Co-Authored-By: Claude Opus 4.8 --- src/domain/library/queries.py | 22 +++++++++++++++++++ .../domain/library/test_queries.py | 15 +++++++++++++ 2 files changed, 37 insertions(+) diff --git a/src/domain/library/queries.py b/src/domain/library/queries.py index 5a5dd2b..7d81f84 100644 --- a/src/domain/library/queries.py +++ b/src/domain/library/queries.py @@ -23,6 +23,15 @@ class FolderNotFound(Exception): path: str +@attrs.frozen +class SyncRunNotFound(Exception): + """ + Raised when no SyncRun row matches the given run_id. + """ + + run_id: int + + def get_all_library_folders() -> list[models.LibraryFolder]: """ Return all registered library folders ordered by path. @@ -73,6 +82,19 @@ def get_latest_sync_run(*, folder_id: int) -> models.SyncRun | None: ) +def get_sync_run(*, run_id: int) -> models.SyncRun: + """ + Return the SyncRun with the given id. + + :raises SyncRunNotFound: If no run with *run_id* exists (e.g. the folder was + removed while a task for this run was still queued). + """ + try: + return models.SyncRun.objects.get(pk=run_id) + except models.SyncRun.DoesNotExist: + raise SyncRunNotFound(run_id=run_id) + + def get_active_sync_run(*, folder_id: int) -> models.SyncRun | None: """ Return the in-progress (scanning or processing) sync run for *folder_id*, or diff --git a/tests/integration/domain/library/test_queries.py b/tests/integration/domain/library/test_queries.py index cd70861..36e52e7 100644 --- a/tests/integration/domain/library/test_queries.py +++ b/tests/integration/domain/library/test_queries.py @@ -5,10 +5,12 @@ from src.domain.library.queries import ( FolderNotFound, LibraryFolderNotFound, + SyncRunNotFound, get_active_sync_run, get_all_library_folders, get_latest_sync_run, get_library_folder, + get_sync_run, list_subdirectories, ) from tests.factories import LibraryFolderFactory, SyncRunFactory @@ -128,6 +130,19 @@ def test_ignores_runs_for_other_folders(self): assert get_latest_sync_run(folder_id=folder.pk) is None +@pytest.mark.django_db +class TestGetSyncRun: + def test_returns_run_by_id(self): + run = SyncRunFactory() + result = get_sync_run(run_id=run.pk) + assert result.pk == run.pk + + def test_raises_sync_run_not_found_for_unknown_id(self): + with pytest.raises(SyncRunNotFound) as exc_info: + get_sync_run(run_id=99999) + assert exc_info.value.run_id == 99999 + + @pytest.mark.django_db class TestGetActiveSyncRun: def test_returns_none_when_no_run_active(self): From 656be95d183ccd98d07791cca34d9372c5965190 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 21:22:36 +0200 Subject: [PATCH 15/27] Add process_synced_image use case Co-Authored-By: Claude Opus 4.8 --- .../usecases/library/process_synced_image.py | 41 ++++++++++ .../library/test_process_synced_image.py | 78 +++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 src/application/usecases/library/process_synced_image.py create mode 100644 tests/integration/application/library/test_process_synced_image.py diff --git a/src/application/usecases/library/process_synced_image.py b/src/application/usecases/library/process_synced_image.py new file mode 100644 index 0000000..0531e8a --- /dev/null +++ b/src/application/usecases/library/process_synced_image.py @@ -0,0 +1,41 @@ +import structlog + +from src.domain.images import operations as image_operations +from src.domain.images.queries import NoFilmSimulationError +from src.domain.library import operations as library_operations +from src.domain.library import queries as library_queries + +logger = structlog.get_logger("application.library.process_synced_image") + + +def process_synced_image(*, image_path: str, sync_run_id: int) -> None: + """ + Process a single image as part of sync run *sync_run_id* and record progress. + + Composes the pure image-processing operation with sync-run bookkeeping so the + processing operation stays unaware of sync: non-Fujifilm files count as + skipped, unexpected failures are logged and counted as errors (the run + continues), and successful imports count as processed. When every image in the + run has a terminal outcome, the run is completed. + + If the run no longer exists (its folder was removed while this work was + queued), the call returns without processing. + """ + try: + run = library_queries.get_sync_run(run_id=sync_run_id) + except library_queries.SyncRunNotFound: + return + + try: + image_operations.process_image(image_path=image_path) + except NoFilmSimulationError: + run.record_skipped() + except Exception: + logger.exception("Failed to process image during sync") + run.record_error() + else: + run.record_processed() + + run.refresh_from_db() + if run.all_images_accounted_for(): + library_operations.complete_sync_run(run=run) diff --git a/tests/integration/application/library/test_process_synced_image.py b/tests/integration/application/library/test_process_synced_image.py new file mode 100644 index 0000000..638eefe --- /dev/null +++ b/tests/integration/application/library/test_process_synced_image.py @@ -0,0 +1,78 @@ +import shutil +from pathlib import Path +from unittest.mock import patch + +import pytest + +from src.application.usecases.library.process_synced_image import process_synced_image +from src.data import models +from tests.factories import SyncRunFactory + +FIXTURES_DIR = Path(__file__).resolve().parent.parent.parent.parent / "fixtures" / "images" +FUJIFILM_FIXTURE = FIXTURES_DIR / "XS107114.JPG" +NON_FUJIFILM_FIXTURE = FIXTURES_DIR / "sub-folder" / "img_4968_dng_embedded.jpg" + + +@pytest.mark.django_db +class TestProcessSyncedImage: + def test_imports_image_and_records_processed(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + process_synced_image(image_path=str(image_path), sync_run_id=run.pk) + + run.refresh_from_db() + assert run.processed == 1 + assert run.skipped == 0 + assert run.errors == 0 + assert models.Image.objects.filter(filepath=str(image_path)).exists() + + def test_records_skipped_for_non_fujifilm_image(self, tmp_path): + image_path = tmp_path / NON_FUJIFILM_FIXTURE.name + shutil.copy(NON_FUJIFILM_FIXTURE, image_path) + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + process_synced_image(image_path=str(image_path), sync_run_id=run.pk) + + run.refresh_from_db() + assert run.skipped == 1 + assert run.processed == 0 + + def test_records_error_and_continues_on_unexpected_failure(self): + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + with patch( + "src.application.usecases.library.process_synced_image.image_operations.process_image", + side_effect=ValueError("boom"), + ): + process_synced_image(image_path="/whatever.jpg", sync_run_id=run.pk) + + run.refresh_from_db() + assert run.errors == 1 + assert run.processed == 0 + + def test_completes_run_when_all_images_accounted_for(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + process_synced_image(image_path=str(image_path), sync_run_id=run.pk) + + run.refresh_from_db() + assert run.state == models.SyncRun.STATE_COMPLETED + assert run.finished_at is not None + + def test_leaves_run_processing_while_images_remain(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=2) + + process_synced_image(image_path=str(image_path), sync_run_id=run.pk) + + run.refresh_from_db() + assert run.state == models.SyncRun.STATE_PROCESSING + + def test_returns_silently_when_run_missing(self): + # No exception should escape when the run (and its folder) is already gone. + process_synced_image(image_path="/whatever.jpg", sync_run_id=99999) From 4753b032057ff980f8a9512c58da62ac032fce53 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 21:50:57 +0200 Subject: [PATCH 16/27] Add sync_folder use case Co-Authored-By: Claude Opus 4.8 --- .../usecases/library/sync_folder.py | 71 +++++++++++ .../application/library/test_sync_folder.py | 115 ++++++++++++++++++ 2 files changed, 186 insertions(+) create mode 100644 src/application/usecases/library/sync_folder.py create mode 100644 tests/integration/application/library/test_sync_folder.py diff --git a/src/application/usecases/library/sync_folder.py b/src/application/usecases/library/sync_folder.py new file mode 100644 index 0000000..2a48277 --- /dev/null +++ b/src/application/usecases/library/sync_folder.py @@ -0,0 +1,71 @@ +from datetime import datetime, timezone + +from django.conf import settings + +from src.application.usecases.library.process_synced_image import process_synced_image +from src.domain.images import queries as image_queries +from src.domain.library import operations as library_operations +from src.domain.library import queries as library_queries +from src.services import workertasks + +_SYNC_PROCESS_IMAGE_TASK = "src.interfaces.tasks.sync_process_image_task" + + +def sync_folder(*, folder_id: int) -> None: + """ + Scan a single library folder and import new images, tracking progress in a + SyncRun. + + Creates a run, walks the folder (mtime-gated by its last_checked_at), and + dispatches each new image: in async mode by enqueuing a Celery task, in sync + mode by processing inline. The run is completed here only when nothing new is + found; otherwise the last processed image completes it. + + Returns without doing anything if the folder no longer exists or already has + an active run (the concurrency guard). + """ + try: + folder = library_queries.get_library_folder(folder_id=folder_id) + except library_queries.LibraryFolderNotFound: + return + + try: + run = library_operations.start_sync_run(folder=folder) + except library_operations.SyncAlreadyInProgress: + return + + # last_checked_at records when the scan started, so files added during or + # after this scan are still caught on the next run. + now = datetime.now(tz=timezone.utc) + + try: + found_paths = image_queries.get_image_paths_in_folder( + folder_path=folder.path, + last_checked_at=folder.last_checked_at, + ) + except FileNotFoundError: + folder.set_last_checked_at(value=now) + library_operations.fail_sync_run(run=run, message="Folder does not exist") + return + + known_paths = image_queries.get_all_known_image_paths() + new_paths = sorted(set(found_paths) - known_paths) + + run.begin_processing(total=len(new_paths)) + folder.set_last_checked_at(value=now) + if new_paths: + folder.set_last_processed_at(value=now) + + if not new_paths: + library_operations.complete_sync_run(run=run) + return + + for path in new_paths: + if settings.USE_ASYNC_TASKS: + workertasks.enqueue_task( + task_name=_SYNC_PROCESS_IMAGE_TASK, + kwargs={"image_path": path, "sync_run_id": run.pk}, + queue=settings.PROCESS_IMAGE_QUEUE, + ) + else: + process_synced_image(image_path=path, sync_run_id=run.pk) diff --git a/tests/integration/application/library/test_sync_folder.py b/tests/integration/application/library/test_sync_folder.py new file mode 100644 index 0000000..f0ff101 --- /dev/null +++ b/tests/integration/application/library/test_sync_folder.py @@ -0,0 +1,115 @@ +import shutil +from pathlib import Path +from unittest.mock import patch + +import pytest +from django.test import override_settings + +from src.application.usecases.library.sync_folder import sync_folder +from src.data import models +from tests.factories import ImageFactory, LibraryFolderFactory, SyncRunFactory + +FIXTURES_DIR = Path(__file__).resolve().parent.parent.parent.parent / "fixtures" / "images" +FUJIFILM_FIXTURE = FIXTURES_DIR / "XS107114.JPG" + + +@pytest.mark.django_db +class TestSyncFolderLiteMode: + @override_settings(USE_ASYNC_TASKS=False) + def test_imports_new_image_and_completes_run(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + folder = LibraryFolderFactory(path=str(tmp_path)) + + sync_folder(folder_id=folder.pk) + + run = models.SyncRun.objects.get(folder=folder) + assert run.state == models.SyncRun.STATE_COMPLETED + assert run.total == 1 + assert run.processed == 1 + assert models.Image.objects.filter(filepath=str(image_path)).exists() + + @override_settings(USE_ASYNC_TASKS=False) + def test_completes_run_with_zero_total_when_no_new_files(self, tmp_path): + folder = LibraryFolderFactory(path=str(tmp_path)) + + sync_folder(folder_id=folder.pk) + + run = models.SyncRun.objects.get(folder=folder) + assert run.state == models.SyncRun.STATE_COMPLETED + assert run.total == 0 + + @override_settings(USE_ASYNC_TASKS=False) + def test_ignores_already_known_images(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + ImageFactory(filepath=str(image_path)) + folder = LibraryFolderFactory(path=str(tmp_path)) + + sync_folder(folder_id=folder.pk) + + run = models.SyncRun.objects.get(folder=folder) + assert run.total == 0 + assert run.state == models.SyncRun.STATE_COMPLETED + + @override_settings(USE_ASYNC_TASKS=False) + def test_updates_folder_timestamps(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + folder = LibraryFolderFactory(path=str(tmp_path)) + + sync_folder(folder_id=folder.pk) + + folder.refresh_from_db() + assert folder.last_checked_at is not None + assert folder.last_processed_at is not None + + @override_settings(USE_ASYNC_TASKS=False) + def test_fails_run_and_stamps_check_time_when_folder_missing(self, tmp_path): + missing = tmp_path / "gone" + folder = LibraryFolderFactory(path=str(missing)) + + sync_folder(folder_id=folder.pk) + + run = models.SyncRun.objects.get(folder=folder) + assert run.state == models.SyncRun.STATE_FAILED + folder.refresh_from_db() + assert folder.last_checked_at is not None + + +@pytest.mark.django_db +class TestSyncFolderAsyncMode: + @override_settings(USE_ASYNC_TASKS=True) + def test_enqueues_a_task_per_new_image_and_leaves_run_processing(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + folder = LibraryFolderFactory(path=str(tmp_path)) + + with patch("src.application.usecases.library.sync_folder.workertasks.enqueue_task") as mock_enqueue: + sync_folder(folder_id=folder.pk) + + run = models.SyncRun.objects.get(folder=folder) + assert run.state == models.SyncRun.STATE_PROCESSING + assert run.total == 1 + mock_enqueue.assert_called_once() + kwargs = mock_enqueue.call_args.kwargs + assert kwargs["task_name"] == "src.interfaces.tasks.sync_process_image_task" + assert kwargs["kwargs"] == {"image_path": str(image_path), "sync_run_id": run.pk} + + +@pytest.mark.django_db +class TestSyncFolderGuards: + @override_settings(USE_ASYNC_TASKS=False) + def test_does_nothing_when_folder_already_has_active_run(self, tmp_path): + folder = LibraryFolderFactory(path=str(tmp_path)) + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_PROCESSING, total=5) + + sync_folder(folder_id=folder.pk) + + assert models.SyncRun.objects.filter(folder=folder).count() == 1 + + @override_settings(USE_ASYNC_TASKS=False) + def test_returns_without_creating_a_run_when_folder_missing(self): + sync_folder(folder_id=99999) + + assert models.SyncRun.objects.count() == 0 From 4b76b0712d74e1e3fd2618ccd83f03e28e933744 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 21:55:02 +0200 Subject: [PATCH 17/27] Refactor sync_library use case to per-folder sync with run recovery Co-Authored-By: Claude Opus 4.8 --- .../usecases/library/sync_library.py | 67 ++++++------------- .../application/library/test_sync_library.py | 19 +++++- 2 files changed, 40 insertions(+), 46 deletions(-) diff --git a/src/application/usecases/library/sync_library.py b/src/application/usecases/library/sync_library.py index e131ecc..956a29d 100644 --- a/src/application/usecases/library/sync_library.py +++ b/src/application/usecases/library/sync_library.py @@ -1,11 +1,9 @@ import attrs -from datetime import datetime, timezone from django.conf import settings -from src.domain.images import operations as image_operations -from src.domain.images import queries as image_queries -from src.domain.images.queries import NoFilmSimulationError +from src.application.usecases.library.sync_folder import sync_folder +from src.domain.library import operations as library_operations from src.domain.library import queries as library_queries from src.services import workertasks @@ -27,13 +25,16 @@ class SyncLibraryResult: def sync_library() -> SyncLibraryResult: """ - Scan all registered library folders and import new images into the catalog. + Scan every registered library folder and import new images into the catalog. - Loads all known image paths in a single DB query, then walks each folder - and processes only paths not yet in the catalog. Paths that appear in - multiple overlapping folders are deduplicated across the entire sync run. + Recovers any runs abandoned by a previous process (marking them interrupted), + then syncs each folder in turn via the single-folder use case. In async mode, + checks for a reachable Celery worker before doing any work. - In async mode, checks for a reachable Celery worker before doing any work. + The result aggregates per-folder outcomes from each folder's sync run: in async + mode ``new_files_found`` counts images enqueued for processing, in sync mode it + counts images actually imported. Folders that no longer exist on disk are + reported in ``missing_folders``. :raises CeleryWorkerUnavailable: If USE_ASYNC_TASKS is True and no Celery worker responds within the ping timeout. @@ -41,49 +42,25 @@ def sync_library() -> SyncLibraryResult: if settings.USE_ASYNC_TASKS and not workertasks.is_celery_worker_available(): raise CeleryWorkerUnavailable() - known_paths = image_queries.get_all_known_image_paths() - folders = library_queries.get_all_library_folders() + library_operations.interrupt_active_sync_runs() - all_found_paths: set[str] = set() + folders = library_queries.get_all_library_folders() new_files_found = 0 skipped_non_fujifilm = 0 missing_folders: list[str] = [] - now = datetime.now(tz=timezone.utc) for folder in folders: - try: - found_paths = image_queries.get_image_paths_in_folder( - folder_path=folder.path, - last_checked_at=folder.last_checked_at, - ) - except FileNotFoundError: - missing_folders.append(folder.path) - folder.set_last_checked_at(value=now) + sync_folder(folder_id=folder.pk) + run = library_queries.get_latest_sync_run(folder_id=folder.pk) + if run is None: continue - - new_in_folder = set(found_paths) - known_paths - all_found_paths - all_found_paths |= set(found_paths) - - processed_in_folder = 0 - for path in new_in_folder: - if settings.USE_ASYNC_TASKS: - workertasks.enqueue_task( - task_name="src.interfaces.tasks.process_image_task", - kwargs={"image_path": path}, - queue=settings.PROCESS_IMAGE_QUEUE, - ) - processed_in_folder += 1 - else: - try: - image_operations.process_image(image_path=path) - processed_in_folder += 1 - except NoFilmSimulationError: - skipped_non_fujifilm += 1 - - new_files_found += processed_in_folder - folder.set_last_checked_at(value=now) - if processed_in_folder > 0: - folder.set_last_processed_at(value=now) + if run.state == run.STATE_FAILED: + missing_folders.append(folder.path) + elif settings.USE_ASYNC_TASKS: + new_files_found += run.total or 0 + else: + new_files_found += run.processed + skipped_non_fujifilm += run.skipped return SyncLibraryResult( folders_scanned=len(folders), diff --git a/tests/integration/application/library/test_sync_library.py b/tests/integration/application/library/test_sync_library.py index 9912490..feccbbd 100644 --- a/tests/integration/application/library/test_sync_library.py +++ b/tests/integration/application/library/test_sync_library.py @@ -13,7 +13,7 @@ sync_library, ) from src.data import models -from tests.factories import ImageFactory, LibraryFolderFactory +from tests.factories import ImageFactory, LibraryFolderFactory, SyncRunFactory FIXTURES_DIR = Path(__file__).resolve().parent.parent.parent.parent / "fixtures" / "images" FUJIFILM_FIXTURE = FIXTURES_DIR / "XS107114.JPG" @@ -227,3 +227,20 @@ def test_does_not_check_worker_when_async_is_disabled(self, tmp_path): with patch("src.services.workertasks.is_celery_worker_available") as mock_check: sync_library() mock_check.assert_not_called() + + +@pytest.mark.django_db +class TestSyncLibraryRecovery: + @override_settings(USE_ASYNC_TASKS=False) + def test_marks_dangling_active_runs_interrupted(self, tmp_path): + folder = LibraryFolderFactory(path=str(tmp_path)) + dangling = SyncRunFactory( + folder=folder, + state=models.SyncRun.STATE_PROCESSING, + total=5, + ) + + sync_library() + + dangling.refresh_from_db() + assert dangling.state == models.SyncRun.STATE_INTERRUPTED From 0594a649e0870b82718c2b570d0f4425e3e9e436 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 21:55:56 +0200 Subject: [PATCH 18/27] Add trigger_folder_sync use case Co-Authored-By: Claude Opus 4.8 --- .../usecases/library/trigger_folder_sync.py | 25 ++++++++++ .../library/test_trigger_folder_sync.py | 48 +++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 src/application/usecases/library/trigger_folder_sync.py create mode 100644 tests/unit/application/library/test_trigger_folder_sync.py diff --git a/src/application/usecases/library/trigger_folder_sync.py b/src/application/usecases/library/trigger_folder_sync.py new file mode 100644 index 0000000..4cfa47d --- /dev/null +++ b/src/application/usecases/library/trigger_folder_sync.py @@ -0,0 +1,25 @@ +from django.conf import settings + +from src.application.usecases.library.sync_folder import sync_folder +from src.application.usecases.library.sync_library import CeleryWorkerUnavailable as CeleryWorkerUnavailable +from src.services import background, workertasks + + +def trigger_folder_sync(*, folder_id: int) -> None: + """ + Trigger a sync of a single folder from a web request. + + Decides how to run the sync so the request returns promptly. In async mode it + verifies a Celery worker is reachable and then runs the sync inline (which only + enqueues per-image tasks, so it is fast). In sync mode it runs the sync in a + background thread, so image processing does not block the request. + + :raises CeleryWorkerUnavailable: If USE_ASYNC_TASKS is True and no Celery + worker responds; no run is created in that case. + """ + if settings.USE_ASYNC_TASKS: + if not workertasks.is_celery_worker_available(): + raise CeleryWorkerUnavailable() + sync_folder(folder_id=folder_id) + else: + background.run_in_background(sync_folder, folder_id=folder_id) diff --git a/tests/unit/application/library/test_trigger_folder_sync.py b/tests/unit/application/library/test_trigger_folder_sync.py new file mode 100644 index 0000000..fc32bfb --- /dev/null +++ b/tests/unit/application/library/test_trigger_folder_sync.py @@ -0,0 +1,48 @@ +from unittest.mock import patch + +import pytest +from django.test import override_settings + +from src.application.usecases.library.trigger_folder_sync import ( + CeleryWorkerUnavailable, + trigger_folder_sync, +) + +MODULE = "src.application.usecases.library.trigger_folder_sync" + + +class TestTriggerFolderSync: + @override_settings(USE_ASYNC_TASKS=True) + def test_runs_sync_inline_when_worker_available(self): + with ( + patch(f"{MODULE}.workertasks.is_celery_worker_available", return_value=True), + patch(f"{MODULE}.sync_folder") as mock_sync, + patch(f"{MODULE}.background.run_in_background") as mock_bg, + ): + trigger_folder_sync(folder_id=7) + + mock_sync.assert_called_once_with(folder_id=7) + mock_bg.assert_not_called() + + @override_settings(USE_ASYNC_TASKS=True) + def test_raises_and_does_not_sync_when_worker_unavailable(self): + with ( + patch(f"{MODULE}.workertasks.is_celery_worker_available", return_value=False), + patch(f"{MODULE}.sync_folder") as mock_sync, + ): + with pytest.raises(CeleryWorkerUnavailable): + trigger_folder_sync(folder_id=7) + + mock_sync.assert_not_called() + + @override_settings(USE_ASYNC_TASKS=False) + def test_runs_sync_in_background_in_lite_mode(self): + with ( + patch(f"{MODULE}.background.run_in_background") as mock_bg, + patch(f"{MODULE}.sync_folder") as mock_sync, + patch(f"{MODULE}.workertasks.is_celery_worker_available") as mock_worker, + ): + trigger_folder_sync(folder_id=7) + + mock_bg.assert_called_once_with(mock_sync, folder_id=7) + mock_worker.assert_not_called() From 6c8620941cd943f9d8316d51cb0857b3989ad1d8 Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 21:57:05 +0200 Subject: [PATCH 19/27] Add sync_process_image Celery task Co-Authored-By: Claude Opus 4.8 --- src/interfaces/tasks.py | 11 ++++++++ .../test_sync_process_image_task.py | 28 +++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 tests/functional/test_sync_process_image_task.py diff --git a/src/interfaces/tasks.py b/src/interfaces/tasks.py index 516ccac..b85db14 100644 --- a/src/interfaces/tasks.py +++ b/src/interfaces/tasks.py @@ -4,6 +4,7 @@ from celery import shared_task from django.conf import settings +from src.application.usecases.library.process_synced_image import process_synced_image from src.domain.images import events, operations from src.domain.images.thumbnails import operations as thumbnail_operations @@ -31,6 +32,16 @@ def process_image_task(self: Any, /, *, image_path: str, **kwargs: object) -> st return f"Processed {recipe.filename}" +@shared_task(name="library.sync_process_image", bind=True, queue=settings.PROCESS_IMAGE_QUEUE) +def sync_process_image_task(self: Any, /, *, image_path: str, sync_run_id: int, **kwargs: object) -> str: + """ + Celery task that processes a single image for a library sync run and reports + progress against the run. + """ + process_synced_image(image_path=image_path, sync_run_id=sync_run_id) + return f"Processed {image_path} for sync run {sync_run_id}" + + @shared_task(name="domain.generate_thumbnail", bind=True, queue=settings.PROCESS_IMAGE_QUEUE) def generate_thumbnail_task(self: Any, /, *, filepath: str, width: int, **kwargs: object) -> str: """ diff --git a/tests/functional/test_sync_process_image_task.py b/tests/functional/test_sync_process_image_task.py new file mode 100644 index 0000000..4c8d5ee --- /dev/null +++ b/tests/functional/test_sync_process_image_task.py @@ -0,0 +1,28 @@ +import shutil +from pathlib import Path + +import pytest + +from src.data import models +from src.interfaces.tasks import sync_process_image_task +from tests.factories import SyncRunFactory + +FIXTURES_DIR = Path(__file__).resolve().parent.parent / "fixtures" / "images" +FUJIFILM_FIXTURE = FIXTURES_DIR / "XS107114.JPG" + + +@pytest.mark.django_db +class TestSyncProcessImageTask: + def test_processes_image_and_records_progress_against_run(self, tmp_path): + image_path = tmp_path / FUJIFILM_FIXTURE.name + shutil.copy(FUJIFILM_FIXTURE, image_path) + run = SyncRunFactory(state=models.SyncRun.STATE_PROCESSING, total=1) + + sync_process_image_task.apply( + kwargs={"image_path": str(image_path), "sync_run_id": run.pk} + ).get() + + run.refresh_from_db() + assert run.processed == 1 + assert run.state == models.SyncRun.STATE_COMPLETED + assert models.Image.objects.filter(filepath=str(image_path)).exists() From e56cb6e532606abd41e6816e55b0f7111a2c416f Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 22:00:23 +0200 Subject: [PATCH 20/27] Trigger folder sync on library folder add and path update Co-Authored-By: Claude Opus 4.8 --- src/interfaces/library/views.py | 19 ++++++++++- tests/functional/test_library_views.py | 46 ++++++++++++++++++++++++-- 2 files changed, 62 insertions(+), 3 deletions(-) diff --git a/src/interfaces/library/views.py b/src/interfaces/library/views.py index f873ce9..1be55c8 100644 --- a/src/interfaces/library/views.py +++ b/src/interfaces/library/views.py @@ -5,6 +5,7 @@ from src.application.usecases.library import browse_filesystem as browse_filesystem_uc from src.application.usecases.library import dataclasses as library_dataclasses from src.application.usecases.library import remove_library_folder as remove_library_folder_uc +from src.application.usecases.library import trigger_folder_sync as trigger_folder_sync_uc from src.application.usecases.library import update_library_folder_path as update_library_folder_path_uc from src.data import models from src.domain.library import queries as domain_queries @@ -39,7 +40,7 @@ def post(self, request: http.HttpRequest) -> http.HttpResponse: if not path: return http.HttpResponseBadRequest("path is required") try: - add_library_folder_uc.add_library_folder(path=path) + folder = add_library_folder_uc.add_library_folder(path=path) except add_library_folder_uc.FolderNotFound as exc: return shortcuts.render(request, "library/library.html", { "folders": _list_all_folders(), @@ -50,6 +51,14 @@ def post(self, request: http.HttpRequest) -> http.HttpResponse: "folders": _list_all_folders(), "error": f"Folder is already in the library: {exc.path}", }) + + try: + trigger_folder_sync_uc.trigger_folder_sync(folder_id=folder.folder_id) + except trigger_folder_sync_uc.CeleryWorkerUnavailable: + return shortcuts.render(request, "library/library.html", { + "folders": _list_all_folders(), + "error": "Folder added, but no image worker is running to sync it. Start one with 'make worker'.", + }) return shortcuts.redirect(urls.reverse("library-list")) @@ -91,6 +100,14 @@ def post(self, request: http.HttpRequest, folder_id: int) -> http.HttpResponse: "folders": _list_all_folders(), "error": f"Folder is already in the library: {exc.path}", }) + + try: + trigger_folder_sync_uc.trigger_folder_sync(folder_id=folder_id) + except trigger_folder_sync_uc.CeleryWorkerUnavailable: + return shortcuts.render(request, "library/library.html", { + "folders": _list_all_folders(), + "error": "Path updated, but no image worker is running to sync it. Start one with 'make worker'.", + }) return shortcuts.redirect(urls.reverse("library-list")) diff --git a/tests/functional/test_library_views.py b/tests/functional/test_library_views.py index f352565..dbb9928 100644 --- a/tests/functional/test_library_views.py +++ b/tests/functional/test_library_views.py @@ -1,9 +1,14 @@ +from unittest.mock import patch + import pytest from bs4 import BeautifulSoup +from src.application.usecases.library.trigger_folder_sync import CeleryWorkerUnavailable from src.data import models from tests.factories import LibraryFolderFactory +TRIGGER = "src.interfaces.library.views.trigger_folder_sync_uc.trigger_folder_sync" + @pytest.mark.django_db class TestLibraryFolderList: @@ -44,12 +49,36 @@ def test_adds_folder_and_redirects_to_list(self, client, tmp_path): new_dir = tmp_path / "photos" new_dir.mkdir() - response = client.post("/library/new/",{"path": str(new_dir)}) + with patch(TRIGGER): + response = client.post("/library/new/",{"path": str(new_dir)}) assert response.status_code == 302 assert response["Location"] == "/library/" assert models.LibraryFolder.objects.filter(path=str(new_dir)).exists() + def test_triggers_sync_for_the_new_folder(self, client, tmp_path): + new_dir = tmp_path / "photos" + new_dir.mkdir() + + with patch(TRIGGER) as mock_trigger: + client.post("/library/new/",{"path": str(new_dir)}) + + folder = models.LibraryFolder.objects.get(path=str(new_dir)) + mock_trigger.assert_called_once_with(folder_id=folder.pk) + + def test_shows_error_when_worker_unavailable_on_add(self, client, tmp_path): + new_dir = tmp_path / "photos" + new_dir.mkdir() + + with patch(TRIGGER, side_effect=CeleryWorkerUnavailable()): + response = client.post("/library/new/",{"path": str(new_dir)}) + + assert response.status_code == 200 + soup = BeautifulSoup(response.content, "html.parser") + assert soup.find(class_="error-banner") is not None + # The folder is still registered even though the sync could not start. + assert models.LibraryFolder.objects.filter(path=str(new_dir)).exists() + def test_returns_error_for_nonexistent_path(self, client, tmp_path): missing = str(tmp_path / "does_not_exist") @@ -100,13 +129,26 @@ def test_updates_path_and_redirects_to_list(self, client, tmp_path): new_dir.mkdir() folder = LibraryFolderFactory(path=str(old_dir)) - response = client.post(f"/library/{folder.pk}/edit/",{"path": str(new_dir)}) + with patch(TRIGGER): + response = client.post(f"/library/{folder.pk}/edit/",{"path": str(new_dir)}) assert response.status_code == 302 assert response["Location"] == "/library/" folder.refresh_from_db() assert folder.path == str(new_dir) + def test_triggers_sync_after_path_update(self, client, tmp_path): + old_dir = tmp_path / "old" + new_dir = tmp_path / "new" + old_dir.mkdir() + new_dir.mkdir() + folder = LibraryFolderFactory(path=str(old_dir)) + + with patch(TRIGGER) as mock_trigger: + client.post(f"/library/{folder.pk}/edit/",{"path": str(new_dir)}) + + mock_trigger.assert_called_once_with(folder_id=folder.pk) + def test_returns_404_for_unknown_folder_id(self, client, tmp_path): new_dir = tmp_path / "new" new_dir.mkdir() From 61add96cb1b8da9c0d4ad3125c8dca82ea399aac Mon Sep 17 00:00:00 2001 From: Gosku Date: Thu, 2 Jul 2026 22:50:16 +0200 Subject: [PATCH 21/27] Add library folder sync-status view Co-Authored-By: Claude Opus 4.8 --- .../usecases/library/dataclasses.py | 16 +++++ src/interfaces/library/urls.py | 1 + src/interfaces/library/views.py | 32 ++++++++++ .../library/partials/sync_status.html | 27 ++++++++ .../test_library_sync_status_view.py | 64 +++++++++++++++++++ 5 files changed, 140 insertions(+) create mode 100644 src/interfaces/templates/library/partials/sync_status.html create mode 100644 tests/functional/test_library_sync_status_view.py diff --git a/src/application/usecases/library/dataclasses.py b/src/application/usecases/library/dataclasses.py index 4e1c8ca..afc8371 100644 --- a/src/application/usecases/library/dataclasses.py +++ b/src/application/usecases/library/dataclasses.py @@ -12,6 +12,22 @@ class LibraryFolderData: last_checked_at: datetime | None +@attrs.frozen +class SyncRunData: + folder_id: int + total: int | None + processed: int + skipped: int + errors: int + percent: int + is_active: bool + is_scanning: bool + is_processing: bool + is_completed: bool + is_failed: bool + is_interrupted: bool + + @attrs.frozen class FilesystemEntry: name: str diff --git a/src/interfaces/library/urls.py b/src/interfaces/library/urls.py index 5767d3e..dbd4ee7 100644 --- a/src/interfaces/library/urls.py +++ b/src/interfaces/library/urls.py @@ -8,4 +8,5 @@ path("library/browse/partial/", views.FilesystemBrowser.as_view(), name="library-browse"), path("library//delete/", views.LibraryFolderRemove.as_view(), name="library-folder-delete"), path("library//edit/", views.LibraryFolderPathUpdate.as_view(), name="library-folder-edit"), + path("library//sync-status/", views.LibraryFolderSyncStatus.as_view(), name="library-folder-sync-status"), ] diff --git a/src/interfaces/library/views.py b/src/interfaces/library/views.py index 1be55c8..826ffae 100644 --- a/src/interfaces/library/views.py +++ b/src/interfaces/library/views.py @@ -25,6 +25,26 @@ def _list_all_folders() -> list[library_dataclasses.LibraryFolderData]: return [_folder_data(f) for f in domain_queries.get_all_library_folders()] +def _sync_status(run: models.SyncRun) -> library_dataclasses.SyncRunData: + total = run.total + handled = run.processed + run.skipped + run.errors + percent = int(handled / total * 100) if total else 0 + return library_dataclasses.SyncRunData( + folder_id=run.folder_id, + total=total, + processed=run.processed, + skipped=run.skipped, + errors=run.errors, + percent=percent, + is_active=run.state in models.SyncRun.ACTIVE_STATES, + is_scanning=run.state == models.SyncRun.STATE_SCANNING, + is_processing=run.state == models.SyncRun.STATE_PROCESSING, + is_completed=run.state == models.SyncRun.STATE_COMPLETED, + is_failed=run.state == models.SyncRun.STATE_FAILED, + is_interrupted=run.state == models.SyncRun.STATE_INTERRUPTED, + ) + + class LibraryFolderList(generic.View): """Display the list of monitored library folders.""" @@ -111,6 +131,18 @@ def post(self, request: http.HttpRequest, folder_id: int) -> http.HttpResponse: return shortcuts.redirect(urls.reverse("library-list")) +class LibraryFolderSyncStatus(generic.View): + """Return an HTMX partial with the latest sync-run status for a folder.""" + + def get(self, request: http.HttpRequest, folder_id: int) -> http.HttpResponse: + run = domain_queries.get_latest_sync_run(folder_id=folder_id) + status = _sync_status(run) if run is not None else None + return shortcuts.render(request, "library/partials/sync_status.html", { + "status": status, + "folder_id": folder_id, + }) + + class FilesystemBrowser(generic.View): """Return an HTMX partial for the filesystem browser. diff --git a/src/interfaces/templates/library/partials/sync_status.html b/src/interfaces/templates/library/partials/sync_status.html new file mode 100644 index 0000000..f41297b --- /dev/null +++ b/src/interfaces/templates/library/partials/sync_status.html @@ -0,0 +1,27 @@ +{% if status %} +
+ {% if status.is_scanning %} + Scanning… + {% elif status.is_processing %} + Processing {{ status.processed }}/{{ status.total }} + + {% elif status.is_completed %} + Imported {{ status.processed }}{% if status.skipped %}, skipped {{ status.skipped }}{% endif %}{% if status.errors %}, {{ status.errors }} error{{ status.errors|pluralize }}{% endif %} + {% elif status.is_failed %} + Sync failed + {% elif status.is_interrupted %} + Sync interrupted + {% endif %} +
+{% else %} +
+ Not synced +
+{% endif %} diff --git a/tests/functional/test_library_sync_status_view.py b/tests/functional/test_library_sync_status_view.py new file mode 100644 index 0000000..cfd5700 --- /dev/null +++ b/tests/functional/test_library_sync_status_view.py @@ -0,0 +1,64 @@ +import pytest + +from src.data import models +from tests.factories import LibraryFolderFactory, SyncRunFactory + + +@pytest.mark.django_db +class TestLibraryFolderSyncStatus: + def test_shows_not_synced_when_no_run_exists(self, client): + folder = LibraryFolderFactory() + + response = client.get(f"/library/{folder.pk}/sync-status/") + + assert response.status_code == 200 + content = response.content.decode() + assert "Not synced" in content + assert "hx-trigger" not in content + + def test_shows_scanning_and_polls_while_active(self, client): + folder = LibraryFolderFactory() + SyncRunFactory(folder=folder, state=models.SyncRun.STATE_SCANNING) + + response = client.get(f"/library/{folder.pk}/sync-status/") + + content = response.content.decode() + assert "Scanning" in content + assert 'hx-trigger="every 2s"' in content + + def test_shows_progress_while_processing(self, client): + folder = LibraryFolderFactory() + run = SyncRunFactory(folder=folder, state=models.SyncRun.STATE_PROCESSING, total=4) + run.processed = 1 + run.save(update_fields=["processed"]) + + response = client.get(f"/library/{folder.pk}/sync-status/") + + content = response.content.decode() + assert "Processing 1/4" in content + assert 'hx-trigger="every 2s"' in content + assert " Date: Thu, 2 Jul 2026 22:50:53 +0200 Subject: [PATCH 22/27] Show library folder sync progress in the folder row Co-Authored-By: Claude Opus 4.8 --- .../templates/library/includes/folder_row.html | 7 +++++++ src/interfaces/templates/library/library.html | 1 + tests/functional/test_library_views.py | 9 +++++++++ 3 files changed, 17 insertions(+) diff --git a/src/interfaces/templates/library/includes/folder_row.html b/src/interfaces/templates/library/includes/folder_row.html index c9c3005..d71417a 100644 --- a/src/interfaces/templates/library/includes/folder_row.html +++ b/src/interfaces/templates/library/includes/folder_row.html @@ -14,6 +14,13 @@ Never {% endif %} + +
+