Repository navigation
fix(ingest): keep ProcessLock exclusive when a contender fails - #1659
Open
SrkianFardin25 wants to merge 1 commit into
Open
SrkianFardin25 wants to merge 1 commit into
SrkianFardin25 wants to merge 1 commit into
Conversation
ProcessLock.acquire opened the lock file with "w+", which truncated the owner's PID before flock was even tried. The contender then read an empty PID, treated the lock as stale and unlinked the path while the owner still held its flock. A third caller created a new inode and acquired it, so two ingest processes could run at once. release() also unlinked the path, which could remove a lock file another process had just created. The lock now relies on flock alone: the kernel drops a flock when the holder's descriptors close (including when the process dies), so a lock left by a dead process is never held and needs no cleanup. The file is opened without truncation, the PID is written only after flock succeeds, contenders only wait and retry, and release() clears the PID but leaves the file in place. This also covers the restore in cps/admin.py, which holds this lock with "restore_calibre_db" written in it instead of a PID; a contender no longer treats that as stale and unlinks it. Fixes crocodilestick#1657
SrkianFardin25
force-pushed
the
fix-1657-process-lock
branch
from
November 6, 2026 10:43
4d4723f to
6e85b7a
Compare
SrkianFardin25
marked this pull request as ready for review
November 6, 2026 10:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1657
Problem
ProcessLock.acquire()inscripts/ingest_processor.pyopened the lock file withopen(path, 'w+'), which truncates it beforeflockis tried. So a second caller:flock,_cleanup_stale_lock(), which unlinked the path while the owner still held its kernel lock.A third caller then created a new inode at the same path and got the lock, so two ingest processes could run at the same time.
release()also unlinked the path, which can delete a lock file that another process has just created.The same "non-PID content means stale" check also fired while
cps/admin.pyholds this lock during a library restore, because it writesrestore_calibre_dbinto the file instead of a PID.Fix
flockis now the only source of truth, which matches how the kernel behaves: aflockis dropped when the holder's descriptors close, including when the process dies. A lock left by a dead process is never held, so no stale-lock cleanup is needed.os.open(..., O_RDWR | O_CREAT)(no truncation) and write the PID only afterflocksucceeds.flockjust waits and retries until the timeout. It never truncates or unlinks the file.release()clears the PID while it still holds the lock, unlocks, and leaves the file in place._check_stale_lock()and_cleanup_stale_lock(). Nothing else referenced them._get_holding_pid()is unchanged and now reads a PID that a contender can no longer erase.The lock path, name, timeout and return values are unchanged, so
initialize_runtime()and the restore code incps/admin.pywork as before. Thecwa-initservice still deletes leftover lock files at container start.Tests
Added
tests/unit/test_ingest_process_lock.py(markedunit):test_failed_contender_keeps_owner_lock_file: owner acquires; a contender times out; the path still exists with the owner's inode and PID; a third caller also fails; after the owner releases, the third caller gets the same inode.test_contender_does_not_treat_non_pid_content_as_stale: a holder that writesrestore_calibre_db(likecps/admin.py) keeps its file.test_unlocked_file_left_by_dead_process_is_reused: an unlocked file with a dead PID is taken over, and the PID is cleared on release.All three fail on
main(the path is unlinked) and pass with this change.What I ran locally (Linux, Python 3.13,
requirements.txt+requirements-dev.txt;python-ldap,Flask-SimpleLDAP,py7zrandfaust-cchardetwere left out because they need a C compiler my environment does not have):I did not run the Docker integration tests (
tests/docker,tests/integration). Those are left to CI.