feat: event listeners on Worker, Queue, and Job

[jotarios]

Summary

Close the BullMQ-parity gap for event-driven consumption. The engine already emits a cross-process events stream covering waiting / active / completed / failed / retry-scheduled / delayed / dlq / drained; this slice finishes the high-level shim surfaces on top so application code mirrors BullMQ exactly. Zero engine changes (empty git diff main -- chasquimq/src/).

Three new user-visible surfaces, mirrored across both shims:

• Worker-level listeners. worker.on('drained' | 'paused' | 'resumed' | ...) on both shims. Node adds the three new events to the existing EventEmitter; Python gains a full .on() / .off() / .once() listener API alongside the existing native handler. 'drained' lazily spawns an embedded QueueEvents subscriber on the first listener attach (zero-cost when unused); 'paused' / 'resumed' fire from the local pause() / resume() methods. 'progress' / 'stalled' documented as accepted-but-no-op for parity (engine doesn't emit those transitions yet).
• Queue-level listeners (QueueEvents). Per-id channels (active:<jobId> / completed:<jobId> / failed:<jobId>) emitted alongside the existing broadcast channels so targeted subscribers can listen on a single job without paying the O(N-listeners) broadcast dispatch cost. Python gains the .on() / .off() / .once() listener API alongside the existing async-iterator; sync and async callbacks both supported.
• Job-level promise. Job.waitUntilFinished(queueEvents, ttl?) (Node) and job.wait_until_finished(queue_events, *, timeout=None) (Python). Event-driven (no polling), resolves with the stored handler result via Queue.getJobResult when storeResults=true, rejects with new Error(failedReason) on failed, throws new WaitUntilFinishedTimeoutError on ttl elapse.

Design notes

• Why completed events don't carry the return value. BullMQ's Job.waitUntilFinished exposes the value because BullMQ JSON-encodes it onto the events stream. Re-implementing that on ChasquiMQ would either regress the msgpack wire-format invariant or force every subscriber to depend on msgpack. Composing event + Queue.getJobResult keeps the events stream small (1 MB result × 50 dashboards = 50 MB/job otherwise), keeps subscribers msgpack-free, and works without the result backend for "just detect completion" cases. Documented as a deliberate cross-shim contract.
• Lost-event race. Events emitted before a subscriber's first XREAD BLOCK lands are missed. Mitigations: Node Worker.run() awaits a ready promise on the embedded drained subscriber before kicking the engine; Python QueueEvents exposes await events.wait_until_ready() so callers can gate on subscriber-is-listening.
• Events-emit-before-result-write race. The engine emits completed events before the per-entry JOB_OK_SCRIPT writes the result key (events emit lives off the ack hot path). waitUntilFinished handles this with a short retry loop (10× 50ms) on the getJobResult fetch; falls through to undefined for workers running without storeResults=true.
• Lazy subscriber lifecycle. Workers that never subscribe to 'drained' open zero extra Redis connections. The embedded subscriber uses blockingTimeout: 1000 (vs the QueueEvents 10s default) for snappy shutdown.
• Cancellation is not failure. Python Worker does not fire 'failed' or 'error' on asyncio.CancelledError — cancellation is a control-flow signal (test teardown, shutdown), not a handler failure. Matches BullMQ.

Test plan

• Engine workspace tests: cargo test --workspace -- --include-ignored against live Redis 8.6.2 → 356 passed (38 suites).
• Node shim: npm test (vitest) → 194 passed | 1 skipped | 1 todo (185 before + 9 new in __test__/event-listeners.test.ts).
• Python shim: pytest → 160 passed (149 before + 11 new in tests/test_event_listeners.py).
• cargo fmt --all -- --check clean; cargo clippy --all-targets --workspace -- -D warnings clean (only pre-existing non-root profile warnings).
• Engine source diff vs main is empty — pure shim work, no hot-path change.
• Same-host bench (contended ~3-4 load): queue-add-bulk 185.7k → 186.9k (+0.7%, flat); worker-concurrent retried run shows branch 116.8k vs main 109.8k (+6.3%, contention-driven). Per CLAUDE.md host-load gate, an empty engine diff defends any small worker-concurrent movement against host noise.
• Bug-hunt fix-pass: queue-mismatch guard on waitUntilFinished; cancellation-vs-failure distinction; sync .on() outside coroutine surfaces clear RuntimeError; lazy subscriber error-forwarding skips close-time races.

Docs updated symmetrically on both shim READMEs (Subscribing to events + Awaiting a single job's completion), Node + Python API reference, new concepts/events-and-listeners.md registered in site/astro.config.mjs, root README feature table, and docs/history.md.