feat: add durable SQLite-backed job queue for desktop mode

[Abhash-Chakraborty]

Problem

The desktop runtime plan removes Redis as a required desktop dependency, but Find currently uses Redis/RQ for background indexing and clustering jobs.

Goal

Add a durable SQLite-backed job queue path for desktop mode while preserving Redis/RQ for Docker and server-style deployments.

Scope

• Define a SQLite job table for queued/running/completed/failed jobs.
• Add enqueue/dequeue helpers compatible with the existing worker flow.
• Preserve retry/reprocess semantics.
• Keep Redis/RQ as the default for Docker mode.
• Do not rewrite ML processing logic in this issue.

Acceptance criteria

• Desktop queue mode can enqueue analysis and clustering jobs.
• Jobs survive process restart.
• Failed jobs keep inspectable error state.
• Existing Docker/RQ mode still works unchanged.
• Tests cover enqueue, dequeue, completion, failure, and restart persistence.

Suggested files

• backend/src/find_api/core/queue.py
• backend/src/find_api/workers/jobs.py
• backend/src/find_api/workers/processors.py
• backend/tests/
• docs/plans/not-started/desktop-runtime-adr.md

Related

• Follows the desktop runtime design from closed issue feat: design local desktop runtime without requiring Docker #43.

[rajesh-puripanda]

Hi @Abhash-Chakraborty,

I’d like to take on issue #257. I plan to implement a durable SQLite-backed job queue that mirrors the existing RQ interface, allowing us to drop the Redis dependency for desktop mode while keeping the workflow consistent.

Could you please assign this to me so I can get started on a draft implementation?

[rajesh-puripanda]

/assign gssoc