Skip to content

Give NaN for a pair of samples with no shared called site - #343

Merged
nspope merged 3 commits into
mainfrom
fix/322-undefined-pair-distances
Oct 27, 2026
Merged

nspope merged 3 commits into
mainfrom
fix/322-undefined-pair-distances

Conversation

@andrewkern

Copy link
Copy Markdown
Member

Fixes #322.

A pair of samples with no site called in both has no distance. Before this PR, these functions gave such a pair 0.0, which reads as "identical" (or, for IBS, as "share nothing"):

  • pairwise_diffs_haploid and pairwise_diffs_diploid;
  • decomposition.pairwise_distance;
  • relatedness.ibs.

The distance moments and pcoa then used the false value. This PR gives these pairs NaN, the convention of #321. Each consumer skips the NaN pairs or raises an error.

Behavior changes

  • Distances: a pair with no jointly called site is NaN in the four functions above. For ibs, this applies to the eager, streaming and empty-source paths. The diagonal stays 1.
  • dist_var, dist_skew, dist_kurt (scalar and windowed dist_* columns) skip the NaN pairs. A moment needs 2, 3 or 4 defined pairs. With fewer, the moment is NaN when missing data caused the shortage, and 0.0 when the sample itself is too small (the rule before this PR).
  • pcoa raises ValueError on a NaN distance. The message counts the NaN pairs and rows and names the two sources of NaN (pairwise_distance, pc_dist).
  • lostruct leaves out of the MDS each window that pc_dist cannot compare (too few variants, or no variation when normalize is set), and gives that window a NaN mds row. Before, such a window made pcoa fail with LinAlgError. lostruct raises ValueError when fewer than two windows can be compared, or when corners are requested and the MDS has fewer than two dimensions. corners now gets the first two MDS axes, so n_components other than 2 works.
  • pairwise_diffs_diploid (and so dist_moments on a GenotypeMatrix) accepts a population given as a list of rows. Before, this raised TypeError.

Refactors with no change in value

  • _pairwise_diffs_matrix_gpu holds the NaN rule in one place. Its 'normalize' mode serves both pairwise_diffs functions. A new 'scale' mode serves pairwise_distance and divergence.pairwise_distance_matrix, which had their own copies of the scaling. The return_joint_valid parameter is gone, because no caller needs it now.
  • For pairs with at least one jointly called site, all values are the same as before.

Left as they are, on purpose

Docs

The changelog, missing_data.rst, features.rst and the local PCA tutorial describe the NaN rule. The tutorial now filters the NaN mds rows before it plots, and its corner indices count all windows, as lostruct's do.

Tests

tests/test_undefined_pair_distances.py covers:

  • haploid and diploid distances;
  • the moments, against a SciPy reference;
  • pairwise_distance for each metric, and the pcoa errors;
  • the fused and chunked windowed paths;
  • eager and streaming IBS, including missing_data='exclude';
  • lostruct on three engine paths, including the jackknife, and its guards.

Each new test fails on main. Test references that used 0.0 for an undefined pair now use NaN.

Full suite: 2034 passed, 81 skipped, 15 xfailed. ruff check pg_gpu/ tests/ examples/ passes.

A pair of samples that is never called at the same site has no
distance, but pairwise_diffs (haploid and diploid),
decomposition.pairwise_distance and relatedness.ibs gave it 0.0. That
marked the pair as identical, or for IBS as sharing nothing, and the
distance moments and pcoa used the false value.

These functions now give NaN for such a pair:
- _pairwise_diffs_matrix_gpu holds the rule once. Its 'normalize' mode
  serves both pairwise_diffs functions, and a new 'scale' mode serves
  pairwise_distance and divergence.pairwise_distance_matrix.
- relatedness.ibs gives NaN in the eager, streaming and empty-source
  paths. The diagonal stays 1.
- dist_moments skips the undefined pairs, so the windowed dist_*
  columns do too, and gives NaN when there are pairs but none is
  defined.
- pcoa raises ValueError on a NaN distance and counts the undefined
  pairs.
- lostruct leaves a window with no local PCA out of the MDS and gives
  it a NaN mds row. It raises ValueError when fewer than two windows
  have a local PCA. Before, such a window made pcoa fail with
  LinAlgError.

Fixes #322.
pairwise_diffs_diploid looked up a population by name only, so a
population given as a list of rows raised TypeError, and so did
dist_moments on a GenotypeMatrix. It now selects rows with
population_rows, as the haploid path and ibs do.
dist_moments still gave 0.0 when missing data left too few defined
pairs for a moment, for example one defined pair among six, which looked
like a window with no spread. A moment with too few defined pairs is now
NaN when some pair has no distance, and keeps the 0.0 small-sample rule
otherwise.

lostruct failed inside corners with an unclear error when the valid
windows gave a one-dimensional MDS, and when n_components was not 2.
corners now gets the first two MDS axes, and lostruct names the cause
when there are fewer than two. The pcoa error counts NaN rows as well as
pairs, so a NaN only on the diagonal is not reported as 0 pairs.

The docs now give every reason a window can be invalid, the tutorial's
corners call returns window indices as lostruct does, and the missing
data page covers missing_data='exclude' with no complete site. Tests
cover these cases, and the test references that still used 0.0 for an
undefined pair use NaN.

@nspope nspope left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me; pulled it down and don't see any issues in practice.

@nspope
nspope merged commit 42cddbd into main Oct 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pairwise distances report 0.0 for a pair with no shared called site

2 participants