Skip to content

A file may lack a rep its streams never select from - #23

Closed
selmanozleyen wants to merge 1 commit into
mainfrom
feat/rep-only-where-selected
Closed

selmanozleyen wants to merge 1 commit into
mainfrom
feat/rep-only-where-selected

Conversation

@selmanozleyen

Copy link
Copy Markdown
Collaborator

A dataset split across files can keep each side's rep only where it lives, e.g. obsm/X_src only in the controls' file and obsm/X_tgt only in the perturbed one (a quadratic coupling). Before, every file had to carry every rep a stream asks for, even files the stream never selects rows from.

  • Streamed reps: Source.rep(loc, needed) gives a file that lacks the rep and has no selected rows a never-read placeholder (an unwritten in-memory zarr array or empty CSR group when backed, a broadcast zero view or empty CSR matrix in memory), so annbatch still gets one backing per file. A missing rep in a selected file raises.
  • In-memory streams and eval: _read_rows reads the rep only from files that hold selected rows.
  • load_backed_adata / open_source skip a key a file does not have, leaving the decision to Source.rep.

@selmanozleyen

Copy link
Copy Markdown
Collaborator Author

Superseded: scfit already supports this through separate sources (a link stream on its own source_key binds to the primary via match_on). sckitflow will route each file to the target and/or control source by role, so no placeholder is needed here.

@selmanozleyen
selmanozleyen deleted the feat/rep-only-where-selected branch October 20, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant