feat: improve bulk import efficiency#25
Merged
Merged
Conversation
Use ThreadPoolExecutor to run taxonomy batches concurrently and extract_taxonomy_fields calls (parent + organelle lookups) concurrently within each batch. Add a semaphore to cap concurrent NCBI connections at 10. Reduces 200-organism import from 8.5 min to ~3m50s locally. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
process_parents and organelle_ref_lookup were called sequentially inside extract_taxonomy_fields. Since they're independent, run them concurrently so each organism's processing time is max(parent_lookup, organelle_walk) instead of the sum. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
emilylm
marked this pull request as ready for review
May 18, 2026 14:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 Summary
The
/taxonomy-info/bulk-insertendpoint was timing out on AWS for large imports (~200 organisms took 8.5 minutes locally). The bottleneck was sequential NCBI Dataset API calls — taxonomy batches, parent lineage lookups, and organelle lookups were all processed one at a time.🔄 Type of Change
🧩 Key Changes
ThreadPoolExecutorinstead of sequentiallyextract_taxonomy_fieldscalls (which each make parent + organelle NCBI requests) now run concurrently across all organisms in a batchprocess_parentsandorganelle_ref_lookupwere sequential within each organism — they are now run concurrently since they are independentthreading.Semaphore(10)limits concurrent in-flight NCBI HTTP requests to avoid rate-limiting✅ Checklist
🔍 Review Notes
related_mitosandno_mitoscaches are accessed from multiple threads. CPython's GIL makes simple dict/set reads and writes safe; the only race is two threads both missing the cache for the same taxid and making redundant NCBI calls — this is harmless since the result is idempotent.📎 Related Issues / Tickets