Skip to content

docs: fix drift in Background Processing #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jack-arturo merged 1 commit into from
Jun 22, 2026
+16 −16
Merged

docs: fix drift in Background Processing #181

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions src/content/docs/docs/architecture/background-processing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,14 @@ sidebar:

:::note[Source files]
Key GitHub sources:
- [automem/enrichment/runtime_queue_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/enrichment/runtime_queue_bindings.py) — Enrichment queue setup
- [automem/enrichment/runtime_worker.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/enrichment/runtime_worker.py) — Enrichment worker thread
- [automem/embedding/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/embedding/runtime_bindings.py) — Embedding queue setup
- [automem/embedding/runtime_pipeline.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/embedding/runtime_pipeline.py) — Embedding worker and batch processing
- [automem/consolidation/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/consolidation/runtime_bindings.py) — Consolidation scheduler
- [automem/sync/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/sync/runtime_bindings.py) — Sync worker
- [automem/runtime_wiring.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/runtime_wiring.py) — Startup sequence and worker initialization
- [.env.example](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/.env.example) — Background processing configuration variables
- [automem/enrichment/runtime_queue_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/enrichment/runtime_queue_bindings.py) — Enrichment queue setup
- [automem/enrichment/runtime_worker.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/enrichment/runtime_worker.py) — Enrichment worker thread
- [automem/embedding/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/embedding/runtime_bindings.py) — Embedding queue setup
- [automem/embedding/runtime_pipeline.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/embedding/runtime_pipeline.py) — Embedding worker and batch processing
- [automem/consolidation/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/consolidation/runtime_bindings.py) — Consolidation scheduler
- [automem/sync/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/sync/runtime_bindings.py) — Sync worker
- [automem/runtime_wiring.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/runtime_wiring.py) — Startup sequence and worker initialization
- [.env.example](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/.env.example) — Background processing configuration variables
:::

AutoMem implements four background processing systems that operate independently of the main Flask API request/response cycle. These systems handle computationally expensive operations without blocking client requests.
Expand Down Expand Up @@ -152,9 +152,9 @@ All background workers run in daemon threads started during Flask application in

| Component | Started At | Daemon | Lifecycle |
|---|---|---|---|
| `enrichment_worker` | [automem/enrichment/runtime_queue_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/enrichment/runtime_queue_bindings.py) | Yes | Runs until app shutdown |
| `embedding_worker` | [automem/embedding/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/embedding/runtime_bindings.py) | Yes | Runs until app shutdown |
| Consolidation scheduler | [automem/consolidation/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/consolidation/runtime_bindings.py) | Yes | Custom thread-based scheduler |
| `enrichment_worker` | [automem/enrichment/runtime_queue_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/enrichment/runtime_queue_bindings.py) | Yes | Runs until app shutdown |
| `embedding_worker` | [automem/embedding/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/embedding/runtime_bindings.py) | Yes | Runs until app shutdown |
| Consolidation scheduler | [automem/consolidation/runtime_bindings.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/consolidation/runtime_bindings.py) | Yes | Custom thread-based scheduler |

**Thread Safety:**
- `enrichment_queue` and `embedding_queue` use Python's thread-safe `Queue` class
Expand All @@ -167,7 +167,7 @@ All background workers run in daemon threads started during Flask application in

### Application Startup Sequence

Startup is orchestrated by [automem/runtime_wiring.py](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/automem/runtime_wiring.py):
Startup is orchestrated by [automem/runtime_wiring.py](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/automem/runtime_wiring.py):

1. `init_falkordb()` — Establish FalkorDB connection
2. `init_qdrant()` — Establish optional Qdrant connection
Expand Down Expand Up @@ -306,7 +306,7 @@ Consolidation tasks catch exceptions and continue:
**Relationship Count Caching (80% consolidation speedup):**
- LRU cache with 10,000 entry capacity
- Hourly cache invalidation via timestamp key
- Dramatically reduces graph queries during decay cycles ([consolidation.py:85-100](https://github.com/verygoodplugins/automem/blob/ed36b98e3e1569dde71aa430417b6549520f7068/consolidation.py#L85-L100))
- Dramatically reduces graph queries during decay cycles ([consolidation.py:152](https://github.com/verygoodplugins/automem/blob/ebcf5f16d8a0eecc9400957be1503efaf97fa530/consolidation.py#L152))

---

Expand All @@ -318,10 +318,10 @@ The `POST /memory` endpoint response includes indicators of queued background wo

```json
{
"status": "success",
"memory_id": "uuid",
"message": "Memory stored successfully",
"enrichment_queued": true,
"embedding_queued": true
"enrichment": "queued",
"embedding_status": "queued"
}
```

Expand Down