Basic regions support

[tomwhite]

We should add the equivalent of the bcftools -r/--regions option to filter by regions.

The work in sgkit-dev/sgkit#658 (which was never merged) could form the basis for the implementation.

Also, it would be simpler to implement -t/--targets first, since unlike -r/--regions it only needs to check position, and not the variant length. To support the latter we probably need to store the variant length as a separate Zarr array in order to do efficient queries.

[jeromekelleher]

Agreed. See sgkit-dev/vcf-zarr-spec#21 and sgkit-dev/vcf-zarr-spec#22 for previous thoughts on how we support this efficiently.

I hadn't thought of storing a length array, that's an excellent idea. Would that really suffice to answer overlap queries efficiently?

[tomwhite]

I hadn't thought of storing a length array, that's an excellent idea. Would that really suffice to answer overlap queries efficiently?

Yes. I would actually store an array of variant end positions, and then use pyranges to efficiently compute the overlap.

For -t/--targets, pyranges is not needed since regular NumPy slicing is sufficient.

[tomwhite]

Bioframe is another option for overlap queries.

[jeromekelleher]

Thanks @tomwhite. I think the first step is to do -t as you say, as that can just read in the variant_position array in full as a first pass. I think we do need to consider the latency issues when we have, e.g., a whole genome stored in one Zarr and much smaller variant chunk sizes (which we may need: currently latency and memory usage are terrible in vcztools view for large datasets).

I think we do need to consider what would be an efficient index that we could store in a single/small number of chunks that would allow us to implement range queries (sgkit-dev/vcf-zarr-spec#21, sgkit-dev/vcf-zarr-spec#23)