Lazily generate batches

[tjvandal]

Generating batches in __init__ is slow and memory intensive, related to #111 and #109. Initialization is changed to load indices into memory rather than corresponding datasets. The change enabled the initialization a 1.7 TB dataset with 1m+ samples, 30+ spatial features, in about 10 seconds which was previously overloading memory.

Rather than filling _batches with DataArrays and Datasets, it is filled with indices from selector = {key: slice for key, slice in zip(dims, slices)}. This required an update to the concat_input_dims option where the operation is done in __getitem__(). It is possible that this change decreases performance when this concat_input_dims=True.

[codecov-commenter]

Codecov Report

Merging #112 (f6535f5) into main (4d8e2c8) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main      #112   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            5         5           
  Lines          190       192    +2     
  Branches        35        35           
=========================================
+ Hits           190       192    +2     

Impacted Files	Coverage Δ
xbatcher/generators.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[maxrjones]

LGTM after Joe's last comment. Thanks @tjvandal for the contribution and @jhamman for the code review!

[jhamman]

I made 3 small one-liner comments/suggestions but this looks rad. Its right along the lines of some ideas that @maxrjones and I discussed a while back about schematizing the generation of batch selectors rather than eagerly generating batches. This is a great step in that direction. Thanks @tjvandal!