Skip to content

Decode HDF5 text that its writer declared ASCII - #16

Merged
cboulay merged 1 commit into
devfrom
fix/decode-hdf5-text
Aug 3, 2026
Merged

Decode HDF5 text that its writer declared ASCII#16
cboulay merged 1 commit into
devfrom
fix/decode-hdf5-text

Conversation

@cboulay

@cboulay cboulay commented Aug 3, 2026

Copy link
Copy Markdown
Member

Why

hdmf keys its string-decoding decision directly off the HDF5 character set. A writer that declares its strings H5T_CSET_ASCII rather than UTF-8 — aqnwb, and therefore every recording Orion produces — hands every string back as bytes where a pynwb-written file gives str. Filed upstream as NeurodataWithoutBorders/aqnwb#316, and being looked at.

That fix will not retire this one. Files already on disk keep the character set they were written with, so a reader has to survive both indefinitely.

Nothing raises. str(value) on bytes yields the repr — "b'cond_0'" — which looks like text, compares equal to nothing, and parses as no JSON.

What

Three read boundaries in NWBSlicer forwarded those bytes into places that only ever compare or parse them:

  • Interval tables (slicer.py:239) were stringified with map(str, ...). A marker payload arrived as "b'{\"cause\": ...}'" — text json.loads rejects, so a consumer such as INTENT's cursor marker parser dropped every marker and reported nothing wrong.
  • The electrodes label column (slicer.py:344) becomes the ch-axis coordinates that every downstream name-based channel selection matches against.
  • Device.manufacturer (slicer.py:116) is parsed as a prefix when a caller names a stream by bare device (stream_keys=["NPLAY"] matching CereLink_NPLAY); a bytes value there would discard the stream in silence.

The third is a guard, not a live bug: hdmf happens to decode ASCII attributes on the way out even though it leaves datasets alone. I only found that out because the test asserting otherwise failed, so the comment and a test now pin the asymmetry rather than claim a fix.

A text series (markers, annotations) is decoded once at load rather than at each slice, so the three places that index dset — both slicer read paths and the iterator's — can't each forget to. Safe to materialize: text series are markers, short and few, on the order of the interval tables already read whole beside them.

as_text / as_text_array are exported from the package, since a downstream reader of these files needs the same primitive.

Test

The suite could not have caught this. Written by pynwb throughout, its text reads back as str and none of these paths is exercised — which is how the bug reached hardware.

The new ascii_nwb_path fixture restates the strings as fixed-length ASCII the way aqnwb writes them, carrying one of each kind of text a reader has to survive: spec-defined and custom electrodes columns, a custom interval-table column, a marker series, and the Device manufacturer attribute. It asserts first that they really do read back as bytes; without that guard the tests would pass against the very bug they exist to catch.

Reverting slicer.py alone fails exactly the four data-path tests.

Verification

  • Full suite: 170 passed.
  • ruff check / ruff format clean.

hdmf keys its string-decoding decision directly off the character set. A
writer that declares H5T_CSET_ASCII rather than UTF-8 -- aqnwb, and so
every recording Orion produces -- therefore hands every string back as
bytes where a pynwb-written file gives str
(NeurodataWithoutBorders/aqnwb#316).

Nothing raises. Three read boundaries in the slicer forwarded those bytes
into places that only ever compare or parse them:

- Interval tables were stringified with `map(str, ...)`, which on bytes
  yields the repr. A marker payload arrived as "b'{\"cause\": ...}'" --
  text that json.loads rejects, so a consumer like intent's cursor marker
  parser dropped every marker and reported nothing wrong.
- The electrodes `label` column becomes the ch-axis coordinates that every
  downstream name-based channel selection matches against.
- `Device.manufacturer` is parsed as a prefix when a caller names a stream
  by bare device; a bytes value there would discard the stream silently.
  hdmf happens to decode ASCII attributes even though it leaves datasets
  alone, so that last one is a guard rather than a live bug.

A text series (markers, annotations) is decoded once at load rather than
at each slice, so the three places that index `dset` -- both slicer read
paths and the iterator's -- can't each forget to.

The suite could not have caught this: written by pynwb throughout, its
text reads back as str and none of these paths is exercised. The new
fixture restates the strings as fixed-length ASCII and asserts first that
they really do read back as bytes; without that guard it would pass
against the very bug it exists to catch. Fixing aqnwb upstream won't
retire it either -- files already on disk keep their character set.
@cboulay
cboulay merged commit 6bdf794 into dev Aug 3, 2026
19 of 26 checks passed
@cboulay
cboulay deleted the fix/decode-hdf5-text branch August 3, 2026 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant