Repository navigation
Conversation
95905a3 to
0a7d01d
Compare
b781a21 to
92fefac
Compare
b49418b to
8f6c77d
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Thread limits are set only in worker initializers, leaving parent-process K scoring and whole-pangenome fitting unconstrained so even --cpu 1 can use all available cores.
1 open issue
What changed in this PR
Replaces PPanGGOLiN’s vendored C NEM implementation with in-memory pynem calls, simplifying packaging and reorganizing partitioning and rarefaction.
Changes:
- Introduces reusable matrix inputs, a pynem wrapper, and dataclass-based options.
- Revises rarefaction sampling, chunk processing, and plotting.
- Removes native-extension build machinery and switches publishing to one pure-Python wheel.
| File | Description |
|---|---|
tests/conftest.py |
Removes obsolete partition global resets. |
setup.py |
Deletes native-extension build configuration. |
pyproject.toml |
Adds pynem and threadpoolctl; removes Cython. |
ppanggolin/workflow/all.py |
Adopts partition and rarefaction option objects. |
ppanggolin/nem/rarefaction.py |
Reworks sampling, in-memory partitioning, and plotting. |
ppanggolin/nem/partition.py |
Reworks K estimation and chunk orchestration. |
ppanggolin/nem/NEM/nem_ver.h |
Removes legacy version declarations. |
ppanggolin/nem/NEM/nem_ver.c |
Removes legacy version reporting. |
ppanggolin/nem/NEM/nem_typ.h |
Removes legacy C types and constants. |
ppanggolin/nem/NEM/nem_stats.pyx |
Removes the Cython binding. |
ppanggolin/nem/NEM/nem_rnd.h |
Removes legacy random-function declarations. |
ppanggolin/nem/NEM/nem_rnd.c |
Removes legacy random routines. |
ppanggolin/nem/NEM/nem_nei.h |
Removes legacy neighborhood declarations. |
ppanggolin/nem/NEM/nem_nei.c |
Removes legacy neighborhood routines. |
ppanggolin/nem/NEM/nem_mod.h |
Removes legacy model declarations. |
ppanggolin/nem/NEM/nem_hlp.h |
Removes legacy help declarations. |
ppanggolin/nem/NEM/nem_hlp.c |
Removes legacy help output. |
ppanggolin/nem/NEM/nem_exe.h |
Removes the legacy C entry-point declaration. |
ppanggolin/nem/NEM/nem_alg.h |
Removes legacy algorithm declarations. |
ppanggolin/nem/NEM/lib_io.h |
Removes legacy I/O declarations. |
ppanggolin/nem/NEM/lib_io.c |
Removes legacy I/O routines. |
ppanggolin/nem/NEM/genmemo.h |
Removes legacy allocation declarations. |
ppanggolin/nem/NEM/genmemo.c |
Removes legacy allocation routines. |
ppanggolin/nem/nem_input.py |
Adds matrix indexing, slicing, and smoothing graphs. |
ppanggolin/nem/model.py |
Adds pynem fitting, labeling, and thread controls. |
ppanggolin_env.yaml |
Adds threadpoolctl to the environment. |
docs/api/ppanggolin.nem.md |
Documents the new model and input modules. |
.github/workflows/pypi.yml |
Consolidates wheel and source-distribution builds. |
.github/workflows/main.yml |
Removes deleted setup.py from workflow triggers. |
Review effort: Balanced
| nem_input = build_nem_input(index, max_degree_smoothing) | ||
| if cpu <= 1: | ||
| return [score(nem_input, 0, free_dispersion, k) for k in ks] |
| with get_context("fork").Pool( | ||
| cpu, initializer=keep_index, initargs=(index,) | ||
| ) as pool: | ||
| with tqdm( | ||
| total=len(chunks), | ||
| unit=" samples partitioned", | ||
| disable=disable_bar, | ||
| ) as bar: | ||
| for result in pool.imap_unordered(_partition_chunk_worker, args): | ||
| votes.add(result.partitions) | ||
| bar.update() |
There was a problem hiding this comment.
you can combine the two with statements
| with get_context("fork").Pool( | ||
| cpu, initializer=_keep_input, initargs=(nem_input,) | ||
| ) as pool: | ||
| with tqdm( | ||
| total=len(ks), initial=1, unit="Number of partitions", disable=disable_bar | ||
| ) as bar: |
There was a problem hiding this comment.
same here. combine with statement so it is easier to read
| # pynem 0.6.1's parallel kernels run on OpenMP, which kills workers forked after them | ||
| if "NUMBA_THREADING_LAYER" not in os.environ: | ||
| numba.config.THREADING_LAYER = "forksafe" |
There was a problem hiding this comment.
This change globally the numba settings. not sure if it's a problem ? if ppanggolin is used as a library, importing it will change user numba setting which may be a problem?
| from dataclasses import dataclass | ||
|
|
||
| # installed libraries | ||
| import numba |
There was a problem hiding this comment.
as numba is directly imported here it should be listed in the pyproject.toml explicitly
| - bokeh>=3.0.0,<3.4.0 | ||
| - graph-tool>=2 | ||
| - gb-io>=0.4.0 | ||
| - threadpoolctl>=3 | ||
| # Tools that are not in Python | ||
| - infernal=1 | ||
| - mmseqs2=15 |
There was a problem hiding this comment.
pynem is not listed here but it should and maybe also add numba as it is imported by ppanggolin directly.

Replaces PPanGGOLiN's vendored C NEM with a pynem dependancy. Implemented with help from Claude Code.
Why
The C NEM had to be compiled for every platform, read its input from files written to disk for every chunk, and computed in float32. On large pangenomes it returned a log-likelihood of −inf. This PR replaces it with pynem, installed from either PyPI or bioconda: a NumPy/numba implementation of the NEM algorithm. The C sources are deleted (−10,392 lines) and PPanGGOLiN now builds as one pure-Python wheel in a single CI job.
How
Bugs
Performance stats (measured on the PanGBank's refseq pangenomes)