[arnaucoma24]

Problem

isaura push failed for wide models, and the failure was reported as success.

1. Too many concurrent uploads. Push relays 4 files at a time (hard-coded ThreadPoolExecutor(max_workers=4)), each split into parallel multipart parts (MINIO_MAX_CONCURRENCY). For wide models, which have dozens of ~250 MB chunks (eos8aa5: 51 objects, 12 GB), the cloud MinIO rejected almost every object with:

RequestTimeout ... A timeout occurred while trying to lock a resource, please reduce your request rate

The same 241 MB chunk uploads in 39 s when it is the only upload. Narrow models (~5 small objects) were not affected.

2. Failed uploads were swallowed. Failed objects were only logged as warnings. Push then merged the bloom filter and wrote catalog.json, printed ✓ … pushed to cloud and exited 0. So:

• callers (and their retries) could not detect the failure;
• the cloud bloom listed every molecule of the model while its chunks were missing. For eos8aa5 the cloud held 2 of 51 objects, but the bloom claimed all 1,355,109 molecules.

Changes

• New env var PUSH_WORKERS (default 4, so default behaviour is unchanged) sets how many files push uploads in parallel. Documented in docs/CONFIGURATION.md.
• If any object fails to upload, push logs N/M objects failed to upload … Aborting push. and exits with code 1, before merging the bloom filter and catalog. Re-running push uploads every object again.

Testing

• pytest tests/test_wide_io.py: 49 passed.
• ruff check on the changed files: no new findings (17 before and after).
• Pushed three wide models with PUSH_WORKERS=1 MINIO_MAX_CONCURRENCY=1 MINIO_MULTIPART_CHUNKSIZE=33554432. Each push exited 0 with no warnings, and every object is on the cloud with the same size as in the local store:

Model	Objects	Size	Push time
eos9q2i v1	15/15	2.91 GB	4.5 min
eos8gop v1	20/20	3.93 GB	9 min
eos8aa5 v2	51/51	11.88 GB	29 min

The bloom filters hold 1,355,109 entries both locally and on the cloud for each model.

Not addressed

• The server-side limit itself. Large pushes still need a lower PUSH_WORKERS.
• A re-push re-uploads objects that are already on the cloud.

🤖 Generated with Claude Code