HDF5 data with uint32_t type unreadable and > 1000 frames fails with NXmx

[graeme-winter]

ethics-gradient:EP graeme$ dials.image_viewer imported.expt 
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
The following parameters have been modified:

input {
  experiments = imported.expt
}

Traceback (most recent call last):
  File "/Users/graeme/git/dials/conda_base/bin/dials.image_viewer", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/command_line/image_viewer.py", line 209, in run
    show_image_viewer(params=params, reflections=reflections, experiments=experiments)
  File "/Users/graeme/git/dials/modules/dials/src/dials/command_line/image_viewer.py", line 174, in show_image_viewer
    wrapper.display(experiments=experiments, reflections=reflections)
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/spotfinder_wrap.py", line 86, in display
    self.frame.load_image(chooser_wrapper(imagesets[0], 0))
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/spotfinder_frame.py", line 657, in load_image
    super().load_image(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/slip_viewer/frame.py", line 438, in load_image
    self.pyslip.tiles.set_image(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/slip_viewer/tile_generation.py", line 70, in set_image
    self.raw_image.set_image_data(get_image_data(self.raw_image))
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/spotfinder_frame.py", line 851, in get_image_data
    image_data = image.get_image_data()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/image_viewer/slip_viewer/frame.py", line 57, in get_image_data
    return self.image_set.get_corrected_data(self.index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatMultiImage.py", line 47, in read
    return format_instance.get_raw_data(index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatNXmx.py", line 164, in get_raw_data
    raw_data = dxtbx.nexus.get_raw_data(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/nexus/__init__.py", line 587, in get_raw_data
    data_as_flex = _dataset_as_flex(
                   ^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/nexus/__init__.py", line 548, in _dataset_as_flex
    raise TypeError(f"Unsupported integer dtype {data_np.dtype}")
TypeError: Please report this error at https://github.com/dials/dials/issues or to dials-user-group@jiscmail.ac.uk: Unsupported integer dtype uint32

from

/entry/data/data         Dataset {1000/Inf, 1630/1630, 1553/1553}
    Attribute: image_nr_high scalar
        Type:      native int
    Attribute: image_nr_low scalar
        Type:      native int
    Location:  1:3024
    Links:     1
    Chunks:    {1, 1630, 1553} 10125560 bytes
    Storage:   10125560000 logical bytes, 1078062909 allocated bytes, 939.24% utilization
    Filter-0:  bitshuffle; see https://github.com/kiyo-masui/bitshuffle-32008  {0, 4, 4, 4096, 2}
    Type:      native unsigned int

This feels like something which really should work

The issue could be that this is Pilatus 4 data and for Eiger we may have a magic work around?

ethics-gradient:EP graeme$ cat dials.import.log 
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1345-g26ce668e5
The following parameters have been modified:

input {
  experiments = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatNXmx.FormatNXmx'>
  template: /Users/graeme/Downloads/EP_cs_01-7_003_master.h5:1:1800
  num images: 1800
  sequences:
    still:    0
    sweep:    1
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt

I will explore that

[graeme-winter]

OK, this is something which should be worked around except for the fact that bit_depth_readout in the header is 16 not 32: if it was 32 (as the data file is) then we would be OK.

[graeme-winter]

ethics-gradient:EP graeme$ h5ls -rvd ../EP_cs_01-7_003_master.h5/entry/instrument/detector/bit_depth_image
Opened "../EP_cs_01-7_003_master.h5" with sec2 driver.
entry/instrument/detector/bit_depth_image Dataset {SCALAR}
    Location:  1:14464
    Links:     1
    Storage:   8 logical bytes, 8 allocated bytes, 100.00% utilization
    Type:      native unsigned long
    Data:
         32
ethics-gradient:EP graeme$ h5ls -rvd ../EP_cs_01-7_003_master.h5/entry/instrument/detector/bit_depth_readout
Opened "../EP_cs_01-7_003_master.h5" with sec2 driver.
entry/instrument/detector/bit_depth_readout Dataset {SCALAR}
    Location:  1:14736
    Links:     1
    Storage:   8 logical bytes, 8 allocated bytes, 100.00% utilization
    Type:      native unsigned long
    Data:
         16

I feel a nasty hack coming on

[graeme-winter]

From the department of "it's nasty but it works"

ethics-gradient:dxtbx graeme$ git diff
diff --git a/src/dxtbx/format/FormatNXmx.py b/src/dxtbx/format/FormatNXmx.py
index 6963d5f5..45fe530d 100644
--- a/src/dxtbx/format/FormatNXmx.py
+++ b/src/dxtbx/format/FormatNXmx.py
@@ -115,7 +115,7 @@ class FormatNXmx(FormatNexus):
         self._static_mask = mask_cache.store_unique_and_get(
             dxtbx.nexus.get_static_mask(nxdetector)
         )
-        self._bit_depth_readout = nxdetector.bit_depth_readout
+        self._bit_depth_readout = max(nxdetector.bit_depth_readout, nxdetector.bit_depth_image)
 
         if self._scan_model:
             self._num_images = len(self._scan_model)

However other issues now follow which also ring bells:

Found 1358 strong pixels on image 885
Found 1377 strong pixels on image 886
Found 1338 strong pixels on image 887
Found 1278 strong pixels on image 888
Found 1303 strong pixels on image 889
Found 1450 strong pixels on image 890
Found 1518 strong pixels on image 891
Found 1489 strong pixels on image 892
Found 1497 strong pixels on image 893
Found 1547 strong pixels on image 894
Found 1478 strong pixels on image 895
Found 1626 strong pixels on image 896


easy_mp crash detected; subprocess trace: ----
Traceback (most recent call last):
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/job_scheduler.py", line 74, in job_cycle
    value = target( *args, **kwargs )
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/mp.py", line 56, in __call__
    return [self.__function(item) for item in iterable]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 257, in __call__
    result = self.function(task)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 80, in __call__
    image = self.imageset.get_corrected_data(index)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatMultiImage.py", line 47, in read
    return format_instance.get_raw_data(index)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/format/FormatNXmx.py", line 164, in get_raw_data
    raw_data = dxtbx.nexus.get_raw_data(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dxtbx/src/dxtbx/nexus/__init__.py", line 585, in get_raw_data
    sliced_outer = data[index]
                   ~~~~^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/h5py/_hl/dataset.py", line 802, in __getitem__
    return self._fast_reader.read(args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_selector.pyx", line 361, in h5py._selector.Reader.read
  File "h5py/_selector.pyx", line 151, in h5py._selector.Selector.apply_args
Index (1000) out of range for (0-999)
----------------------------------------------


Traceback (most recent call last):
  File "/Users/graeme/git/dials/conda_base/bin/dials.find_spots", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/command_line/find_spots.py", line 234, in run
    results = do_spotfinding(experiments, params)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/command_line/find_spots.py", line 126, in do_spotfinding
    reflections = flex.reflection_table.from_observations(experiments, params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/array_family/flex_ext.py", line 190, in from_observations
    return spotfinder.find_spots(experiments)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 694, in find_spots
    table, hot_mask = self._find_spots_in_imageset(imageset)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 808, in _find_spots_in_imageset
    r, h = extract_spots(imageset[j0:j1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 414, in __call__
    return self._find_spots(imageset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/algorithms/spot_finding/finder.py", line 502, in _find_spots
    batch_multi_node_parallel_map(
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/mp.py", line 136, in batch_multi_node_parallel_map
    return multi_node_parallel_map(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/mp.py", line 107, in multi_node_parallel_map
    result = libtbx.easy_mp.parallel_map(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/easy_mp.py", line 637, in parallel_map
    result = res()
             ^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/result.py", line 119, in __call__
    self.traceback( exception = self.exception() )
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/stacktrace.py", line 117, in __call__
    self.raise_handler( exception = exception )
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/stacktrace.py", line 134, in raise_with_traceback
    raise_(type(exception), exception, self.traceback)
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/six.py", line 724, in reraise
    raise value
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/mainthread.py", line 100, in poll
    value = target( *args, **kwargs )
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/modules/dials/src/dials/util/mp.py", line 30, in __call__
    return libtbx.easy_mp.parallel_map(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/easy_mp.py", line 637, in parallel_map
    result = res()
             ^^^^^
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/result.py", line 119, in __call__
    self.traceback( exception = self.exception() )
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/stacktrace.py", line 117, in __call__
    self.raise_handler( exception = exception )
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/stacktrace.py", line 147, in raise_with_stacktrace
    self.stacktrace( exception = exception )
  File "/Users/graeme/git/dials/conda_base/lib/python3.12/site-packages/libtbx/scheduling/stacktrace.py", line 88, in __call__
    raise exception
IndexError: Please report this error at https://github.com/dials/dials/issues or to dials-user-group@jiscmail.ac.uk: Index (1000) out of range for (0-999)

[graeme-winter]

OK, prior art on this:

#700

I note that this did not contain anything like a fix

[graeme-winter]

Though it contained a hint / workaround which could be useful