Skip to content

Replace the C NEM with PyNem - #414

Closed
rsiminel wants to merge 7 commits into
devfrom
pynem-only
Closed

rsiminel wants to merge 7 commits into
devfrom
pynem-only

Conversation

@rsiminel

@rsiminel rsiminel commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

PPanGGOLiN drops its vendored C NEM (−10,392 lines) and now builds as one pure-Python wheel. NEM runs in pynem, installed from PyPI. Partition and rarefaction build NEM's input in memory and no longer write files, so --tmpdir and --keep_tmp_files are removed from ppanggolin partition, and --tmpdir from ppanggolin rarefaction.

  • Same partitions as the C NEM: on PanGBank data, the two disagree on a median 0.20% of gene families. None of the disagreements are between persistent and cloud.
  • No underflow: pynem computes in float64 instead of float32 so it no longer returns a log-likelihood of -inf in big pangenomes.
  • Faster: from 500 genomes up, NEM runs about 4.0× faster than with the C NEM. With --cpu 8, the whole ppanggolin partition command is about 1.4× faster.
  • Threads: pynem's numba and BLAS threads are limited to one per worker, so they don't oversubscribe the cores when workers run in parallel. threadpoolctl is a new dependency.

@rsiminel
rsiminel changed the base branch from master to dev November 5, 2026 15:00
@rsiminel rsiminel closed this Nov 5, 2026
@rsiminel
rsiminel deleted the pynem-only branch November 10, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant