Library sync removes missing images

docs/ADRs/014-remembering-images-that-cannot-be-imported.md

@@ -0,0 +1,115 @@
+# ADR 014 — Remembering images that cannot be imported
+
+**Status**: Accepted
+**Date**: 2026-08-09
+**Supersedes**: nothing. It repairs a consequence of [ADR 013](013-library-sync-removes-missing-images.md)'s decision to retire mtime gating, and revises that ADR's risk 7.
+
+---
+
+## Context
+
+ADR 013 retired mtime-based directory gating, correctly: renaming a directory updates its parent's mtime and never its own, so a gated walk never revisited a renamed subtree, and once removal existed that would have silently dropped every image under a renamed folder.
+
+What that ADR did not notice is what the gate had also been doing. A file the sync cannot import leaves **no trace of any kind**:
+
+- `process_image` raises `NoFilmSimulationError` one line after the `exiftool` subprocess and before any database write;
+- `InvalidFujifilmRecipeData` is raised inside `@transaction.atomic()`, so the `FujifilmExif` row created moments earlier is rolled back;
+- an unexpected error persists nothing either.
+
+The outcome survived only as a counter on `SyncRun` and a log line. Since the sync decides what is new by diffing found paths against `Image.filepath`, a file with no `Image` row is new **every single time**.
+
+Under gating that cost was hidden: the file's directory was usually unchanged, so it was skipped. Without gating it is paid on every sync, forever.
+
+Measured on a folder holding one non-Fujifilm JPEG, before this change:
+
+```
+run1: total=1 skipped=1
+run2: total=1 skipped=1
+run3: total=1 skipped=1
+```
+
+On a real 40,000-image library with ~14,500 non-Fujifilm JPEGs and 93 errored files, that is 14,589 `exiftool` processes per `make start`, every one reaching a conclusion already reached.
+
+The same numbers caused a second, visible symptom. `sync_folder` published one Celery message per new file, synchronously, and `make start` runs `sync_library` to completion before starting the server. 14,589 messages, each resolving the same dotted path, running the structlog processor chain, acquiring from the producer pool and publishing a retry-wrapped AMQP frame, took roughly **40 seconds before the server was reachable**.
+
+## Problem
+
+Two questions, with one root cause between them:
+
+- How does the sync stop reconsidering files it has already judged, without permanently condemning a file that might later become importable?
+- How does dispatch stop being proportional to the number of files, when it sits on the critical path of startup?
+
+Constraints that shape the answer:
+
+- **"Cannot import" is not always permanent.** A non-Fujifilm JPEG never becomes importable; an error might be a locked file, a dead disk or a bug since fixed. Treating them identically is wrong in one direction or the other.
+- **Re-deciding is expensive; re-checking need not be.** The verdict costs a process spawn and a full metadata parse. Whether the file still *is* what it was costs one `stat`.
+- **Nothing may imply the file was touched.** Filmcase does not delete or modify photos (ADR 013). A list of thousands of "ignored" files invites exactly that fear.
+
+## Decisions
+
+### Remember the outcome, keyed to the file's state
+
+A new `IgnoredImage` row records the path, why it was rejected, any detail, and the file's **size and modification time at the moment it was examined**.
+
+A file whose fingerprint still matches cannot have become importable, so the sync passes over it for the cost of one `stat`. A file the user edits or replaces changes its fingerprint and is examined again **on its own**, with nothing to click. That is what makes remembering safe: the record is a statement about a particular version of a file, not a permanent verdict on a path.
+
+Re-recording replaces the fingerprint. Without that, a file that changed, was examined again and failed again would keep its stale fingerprint and be re-examined on every sync from then on.
+
+Success deletes the record, so one that no longer describes reality cannot show a photo that is in the gallery as though it had been rejected.
+
+**Skips and errors are treated the same way**, rather than remembering only the deterministic rejections. Errors are far fewer but not free (93 files is 93 process spawns per startup), and the recorded message is what turns "93 errors" into 93 files a user can actually look at. The escape hatches below cover the transient case.
+
+### Check only the candidates
+
+The fingerprint check runs against paths that survived the known-paths diff, so the extra `stat` calls are bounded by how many files are ignored, not by the size of the tree. A folder with nothing ignored does no extra work at all.
+
+### Batch the dispatch
+
+One message per `SYNC_IMAGE_BATCH_SIZE` files instead of one per file, so 14,589 files become ~146 messages. Each image is still handled and accounted for individually inside the batch, so progress, ignore records and run completion are untouched; only the number of broker round trips falls.
+
+`enqueue_tasks` also resolves the task once and logs once per call rather than per message. **Sharing a broker producer and skipping the result-backend call was considered and rejected**: batching removes two orders of magnitude of messages, after which per-message overhead is not worth the machinery.
+
+### Give the records a page
+
+A per-folder page lists them, paginated and filterable by reason. Without the filter the 93 errors are buried under 14,496 skips and unfindable, and the errors are the ones worth reading.
+
+Three ways back: per-row retry, "retry all errors" (the common case: an environmental failure worth reconsidering without dragging thousands of non-Fujifilm files along), and "retry everything" behind a confirmation. `sync_library --retry-failed` does the same from the command line.
+
+**Retry on an unchanged non-Fujifilm file is a genuine no-op** — it will be re-read and rejected again. The button is still offered, with wording that says so, rather than hidden (which removes control) or shown bare (which looks broken when nothing happens).
+
+### Say plainly that nothing was touched
+
+The page leads with it, the Library column carries it in its tooltip, and the retry confirmation repeats it. An "ignored" file was never imported; forgetting its record only means the next sync looks at it again.
+
+---
+
+## Consequences
+
+1. **One more slow start after deploying.** The 14,589 files must be examined once to be recorded. Batching means the *command* returns quickly even while the worker is still busy.
+2. **A file that changes without becoming importable costs one extra examination**, then settles again with a fresh fingerprint.
+3. **mtime is the weak half of the fingerprint.** A tool that preserves modification time while changing content the same size would go unnoticed. Content hashing would close that, at the cost of reading every ignored file on every sync, which is the thing being avoided. Size plus mtime is the standard trade and the right one here.
+4. **The batch task is a new task name, so the worker must be restarted once.** Messages queued under the old name are rejected as unknown; their run is recovered as interrupted on the next start and re-imported.
+5. **A batch that dies loses the images it had not reached.** The run stays incomplete and is recovered as interrupted, exactly as a single dropped message was before.
+6. **Nested folders**: a file is recorded against whichever registered folder synced it first, so it appears on that folder's page only. Uniqueness is on the path, so it is never recorded twice.
+7. **`get_all_known_image_paths()` is still called once per folder**, so a 40k catalog is scanned once per registered folder per sync. Untouched here, and now the largest remaining fixed cost.
+8. **`read_image_exif` still spawns one `exiftool` per image** with `-a -G1` and string-parses everything. That is the cost of a genuine first import, and the next real win.
+
+---
+
+## Diagram
+
+```mermaid
+flowchart TD
+    A[Path found by the walk] --> B{Already in the catalog?}
+    B -->|yes| Z[Nothing to do]
+    B -->|no| C{Ignored before?}
+    C -->|no| E[Examine it]
+    C -->|yes| D{Size and mtime unchanged?}
+    D -->|yes| Z2[Pass over it: one stat, no process]
+    D -->|no| E
+    E --> F{Outcome}
+    F -->|imported| G[Create the image, forget any record]
+    F -->|not Fujifilm| H[Record it, with today's fingerprint]
+    F -->|recipe unreadable| H
+    F -->|error| I[Record it, with the message]
+```

docs/index.md

@@ -36,4 +36,5 @@
 - [ADR 010 — Image Library: Folder Monitoring and Catalog Sync](ADRs/010-image-library.md) — persisting monitored folders and detecting and importing new images automatically at startup
 - [ADR 011 — Library Sync on Folder Add/Update](ADRs/011-library-sync-on-folder-change.md) — triggering a single-folder sync from the Library page with server-side, DB-backed progress in both install modes
 - [ADR 012 — Pluggable Card Designs](ADRs/012-pluggable-card-designs.md) — a CardDesign abstraction replacing the flat CardTemplate, enabling fundamentally different card layouts (supersedes ADR 005's composition model)
-
+- [ADR 013 — Library Sync Removes Missing Images](ADRs/013-library-sync-removes-missing-images.md) — removing catalog entries whose files are gone, telling a move apart from a deletion, and the guard against a mass wipe (supersedes ADR 010's add-only sync and mtime gating, and ADR 011's removal and rescan decisions)
+- [ADR 014 — Remembering Images That Cannot Be Imported](ADRs/014-remembering-images-that-cannot-be-imported.md) — recording files the sync cannot import so they are not re-examined on every run, and batching worker dispatch so a large import does not block startup

docs/library_sync.md

@@ -31,7 +31,9 @@ run and excludes it from subsequent folders.
 
 **Missing folders.** If a registered folder is no longer present on disk, the command records
 it as missing, updates its last-checked timestamp, and moves on to the next folder. The
-missing path is reported in the command output and does not abort the sync.
+missing path is reported in the command output and does not abort the sync. **Nothing is
+removed from the gallery in this case**, because an unplugged drive looks exactly like a folder
+whose photos were all deleted, and the safe reading is that the drive will come back.
 
 **Processing new files.** New images are handled according to your install mode:
 
@@ -47,8 +49,8 @@ missing path is reported in the command output and does not abort the sync.
 
 You no longer have to restart the app to pick up a newly registered folder. Adding a folder,
 or changing an existing folder's path, triggers a sync of that one folder straight away.
-Removing a folder does not trigger anything: it only stops the folder being monitored and
-never deletes images that were already imported.
+Removing a folder asks what you want to happen to its images: you can keep them in the gallery
+or take them out along with the folder. Either way the photo files themselves stay on disk.
 
 The triggered sync reuses the same per-folder scan described above and behaves according to
 your install mode:
@@ -60,25 +62,9 @@ your install mode:
   once. If no worker is reachable, the folder is still added but a message explains that it
   could not be synced (start a worker with `make worker`, then re-add or re-save the folder).
 
-Changing a folder's path also clears its last-checked timestamp, so the whole new location is
-rescanned from scratch. Progress appears live in the folder's **Sync** column (see below).
-
-## Timestamp-based directory gating
-
-For large collections, walking every subdirectory on every startup would be slow. To avoid
-that, the sync uses filesystem modification times to skip directories that cannot have changed.
-
-Each library folder in the database records a `last_checked_at` timestamp, set at the end of
-every sync pass. Before listing the files inside a directory during a walk, the sync compares
-that directory's modification time against `last_checked_at`. If the directory's modification
-time is at or before the last check time, the directory is skipped entirely. Adding a file to
-a directory updates that directory's modification time, so any directory that has received new
-files since the last check is always included.
-
-This gating applies independently to each directory in the tree. If a parent directory has
-not changed but one of its subdirectories has, the subdirectory is still scanned. The result
-is that the second `make start` after an initial import typically does very little work, even
-if the library spans thousands of files across many folders.
+Changing a folder's path rescans the whole new location. If you moved the folder rather than
+pointing it somewhere new, the photos inside it are recognised and simply follow the move; you
+do not lose ratings or favourites. Progress appears live in the folder's **Sync** column.
 
 ## Timestamps shown in the Library page
 
@@ -94,3 +80,119 @@ while the folder is being walked, a progress bar while images are imported, and
 summary such as `Imported 36, skipped 3` when it finishes. While a sync is active, the column
 refreshes on its own every couple of seconds, so you can watch it progress without reloading
 the page.
+
+## Removing images that disappeared
+
+The sync keeps the gallery in step with your folders in both directions. When a photo is no
+longer where the catalog expects it, its entry is taken out of the gallery.
+
+**Filmcase never deletes your photo files.** "Removing" an image only removes Filmcase's record
+of it. Every file stays on disk exactly where it is. The only case where a photo leaves the
+gallery is the case where you already deleted or moved the file yourself.
+
+**Moves and renames are not removals.** If you rename a photo, move it into another subfolder,
+rename a whole subfolder, or move a photo from one library folder to another, Filmcase
+recognises the file by its contents and simply updates where it is. The photo keeps its
+rating, its favourite mark and its album membership. Nothing is lost and nothing is
+re-imported.
+
+The distinction is made by looking at the old location: if the file is no longer there, the
+photo moved. If it is still there, you made a copy, and the copy does not become a second
+entry.
+
+**Ordering.** Everything new is imported before anything is removed, so a photo that moved has
+already been re-linked by the time removal is considered.
+
+### The safety guard
+
+Removal is permanent, and "the file is missing" is ambiguous: an external drive that is not
+plugged in, a network share that has not mounted, or a folder that has become unreadable all
+look identical to "every photo in here was deleted".
+
+So if a single sync would remove more than half of a folder's images, **and** more than twenty
+of them, it removes nothing and tells you instead:
+
+```
+Skipped removing 340 of 512 image(s) in /Volumes/Photos (safety guard). That usually means a
+drive is not mounted rather than that the photos were deleted. Re-run with --force-prune to
+remove them anyway.
+```
+
+The same warning appears on the Library page against that folder. Both thresholds have to be
+crossed, so ordinary cleanups (emptying a folder of a handful of photos) are applied without
+any fuss. You can change where the line sits with `LIBRARY_PRUNE_GUARD_FRACTION` and
+`LIBRARY_PRUNE_GUARD_MIN_IMAGES`.
+
+### Controlling removal from the command line
+
+```sh
+python manage.py sync_library --dry-run-prune   # list what would go, remove nothing
+python manage.py sync_library --force-prune     # remove even if the guard would stop it
+python manage.py sync_library --no-prune        # import only, never remove
+```
+
+In full install mode the removal happens in the Celery worker after the command has exited, so
+the command's own count is always zero; watch the Library page for the result.
+
+## Removing a folder from the Library
+
+Pressing **Remove** on a folder asks what should happen to its images. It tells you how many
+images in the gallery come only from that folder, and offers two choices:
+
+- **Remove folder only** stops monitoring the folder and leaves its images in the gallery.
+- **Remove folder and its images** also takes those images out of the gallery.
+
+Again, no photo file is deleted either way.
+
+If folders are nested (say both `/Photos` and `/Photos/2024` are registered), removing the
+inner one never takes images the outer one still monitors. Only images that come *exclusively*
+from the folder you are removing are counted, and only those can go.
+
+## Files the sync cannot import
+
+Not every JPEG in a library folder can be imported. Photos from another camera carry no Fujifilm
+recipe, and occasionally a file fails outright. Filmcase remembers those files instead of
+re-reading them on every sync.
+
+**Nothing is deleted or changed.** An ignored file was never in the gallery to begin with; the
+record only means the sync stops looking at it. Your files stay on disk exactly where they are.
+
+Why it matters: reading a photo's metadata costs a separate `exiftool` process. On a library with
+15,000 non-Fujifilm JPEGs, re-examining them on every startup is 15,000 processes to reach the same
+conclusion as last time.
+
+**A file you fix comes back on its own.** Each record stores the file's size and modification time
+as they were when it was examined. If either changes, the file is examined again automatically, so
+re-exporting a photo with proper EXIF at the same path is enough. Nothing needs clicking.
+
+### Seeing and undoing it
+
+Each folder row in the Library page shows how many of its files are ignored, linking to a page that
+lists them with the reason and, for failures, the error. Filter by reason to find the handful of
+real errors among the many "not a Fujifilm photo" entries.
+
+From there you can retry a single file, retry every error at once, or retry everything. Retrying an
+unchanged non-Fujifilm file does nothing, since it will be read and rejected again; the page says so
+on each such row.
+
+From the command line:
+
+```sh
+python manage.py sync_library --retry-failed   # examine every ignored file again
+```
+
+Expect that run to be slow: examining them again is exactly the cost the records avoid.
+
+## Why a large import no longer blocks startup
+
+In full install mode the sync hands images to the Celery worker, and it cannot return until every
+message has been published. One message per file meant a large import held up `make start` for as
+long as publishing took: on tens of thousands of files, tens of seconds before the server was even
+reachable.
+
+Images are now sent in batches, so the message count falls by the batch size (`SYNC_IMAGE_BATCH_SIZE`,
+100 by default). Each image is still processed and counted individually, so progress in the Sync
+column is unchanged.
+
+Note that changing this required a new worker task, so **restart your Celery worker once** after
+upgrading (`make worker`).