Skip to content

feat: improve bulk import efficiency#25

Merged
emilylm merged 2 commits into
mainfrom
improve-bulk-import
May 18, 2026
Merged

feat: improve bulk import efficiency#25
emilylm merged 2 commits into
mainfrom
improve-bulk-import

Conversation

@emilylm

@emilylm emilylm commented May 18, 2026

Copy link
Copy Markdown
Collaborator

📌 Summary

The /taxonomy-info/bulk-insert endpoint was timing out on AWS for large imports (~200 organisms took 8.5 minutes locally). The bottleneck was sequential NCBI Dataset API calls — taxonomy batches, parent lineage lookups, and organelle lookups were all processed one at a time.


🔄 Type of Change

  • 🧹 Refactor (code change that neither fixes a bug nor adds a feature)

🧩 Key Changes

  • Parallel taxonomy batches: all NCBI taxonomy batch requests now fire concurrently via ThreadPoolExecutor instead of sequentially
  • Parallel per-organism field extraction: extract_taxonomy_fields calls (which each make parent + organelle NCBI requests) now run concurrently across all organisms in a batch
  • Parallel parent + organelle lookups per organism: process_parents and organelle_ref_lookup were sequential within each organism — they are now run concurrently since they are independent
  • Concurrency cap: a threading.Semaphore(10) limits concurrent in-flight NCBI HTTP requests to avoid rate-limiting

✅ Checklist

  • Tests are passing locally / in CI
  • New tests have been added for new or changed functionality
  • Self-review completed
  • Documentation updated (if needed)
  • No sensitive data or secrets included

🔍 Review Notes

  • The module-level related_mitos and no_mitos caches are accessed from multiple threads. CPython's GIL makes simple dict/set reads and writes safe; the only race is two threads both missing the cache for the same taxid and making redundant NCBI calls — this is harmless since the result is idempotent.
  • The semaphore value of 10 is conservative. NCBI allows up to 10 req/s with an API key; tuning this value up may yield further gains if rate limiting is not a concern.

📎 Related Issues / Tickets

  • Closes #
  • Related to #

emilylm and others added 2 commits May 19, 2026 00:34
Use ThreadPoolExecutor to run taxonomy batches concurrently and
extract_taxonomy_fields calls (parent + organelle lookups) concurrently
within each batch. Add a semaphore to cap concurrent NCBI connections at 10.

Reduces 200-organism import from 8.5 min to ~3m50s locally.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
process_parents and organelle_ref_lookup were called sequentially inside
extract_taxonomy_fields. Since they're independent, run them concurrently
so each organism's processing time is max(parent_lookup, organelle_walk)
instead of the sum.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@emilylm
emilylm marked this pull request as ready for review May 18, 2026 14:58
@emilylm
emilylm merged commit 3f82136 into main May 18, 2026
2 checks passed
@emilylm
emilylm deleted the improve-bulk-import branch May 18, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant