Summary
NWBSinkConsumer writes annotation rows into a pynwb.misc.AnnotationSeries per table_name. As of pynwb 4.0.0 / NWB schema 2.10.0 (NWBEP001), AnnotationSeries is deprecated in favour of EventsTable in the new top-level /events group. Constructing an AnnotationSeries under pynwb >= 4.0 emits a UserWarning; reading existing files still works. We should migrate the writer.
From the pynwb docs:
DEPRECATED. Use an EventsTable instead, placed in the top-level /events group of the NWBFile.
timestamps → timestamp column, data → annotation column, and source_description should record the origin of the events.
Current state
src/ezmsg/nwb/writer.py
_prep_annotation_series() (~L792) — creates an empty pynwb.misc.AnnotationSeries with resizable H5DataIO data/timestamps datasets, nwbfile.add_acquisition(series), then flush + reopen to grab the live h5py datasets.write_annotation() (~L834) — resizes and appends one (timestamp, data) pair directly on the h5py datasets.NWBSink.on_annotation() (~L999) — duck-typed INPUT_ANNOTATION subscriber, routes flatten_for_table()["data"] to write_annotation.- The reopen path (~L760) re-resolves
annotation_data / annotation_ts after each flush.
src/ezmsg/nwb/pipeline_settings.py — the typed-column sink is described as an alternative to "the JSON-AnnotationSeries path"; docstrings reference AnnotationSeries and need updating.
Proposed change
- Replace the per-
table_name AnnotationSeries in /acquisition with a pynwb.event.EventsTable per table_name added via nwbfile.add_events_table(...) (lands in /events).
- Columns: built-in
timestamp (TimestampVectorData) plus an annotation text column carrying today's data string. Set source_description to something meaningful (e.g. the producer/table origin) instead of the current generic description.
- Append with
table.add_event(timestamp=..., annotation=...) on the live re-opened table, mirroring the existing TimeIntervals pattern in pipeline_settings.py (table.add_interval(...) after _flush_io(reopen=True)) rather than hand-managing resizable h5py datasets. That should let us drop annotation_data / annotation_ts from NWBSinkState and the reopen re-resolution logic.
- Keep the existing timestamp convention (
timestamp - start_timestamp once a data message has anchored the file; wall-clock passthrough before that) — the docstring on write_annotation still applies.
Open questions
- Dependency floor.
pyproject.toml pins only pynwb (unversioned); the current lockfile resolves to 3.1.3, which has no pynwb.event module. Migrating requires pynwb>=4.0.0, which also raises the minimum NWB schema to 2.10.0. Do we bump unconditionally, or keep an AnnotationSeries fallback for older pynwb? Recommendation: bump the floor — dual-path writing doubles the test surface for a sink that is not yet widely deployed.
- Backwards compatibility for readers. Files written previously carry
/acquisition/<name> AnnotationSeries; new files carry /events/<name>. Nothing in reader.py / iterator.py / slicer.py currently reads annotations back, so this is only a concern for external consumers — worth a note in the changelog/README.
- Per-row flush cost.
add_event on a re-opened DynamicTable may be more expensive per row than the current single-element h5py resize. Worth a quick benchmark if annotation rates are high.
- API surface. Should
write_annotation gain a way to pass extra typed columns (the EventsTable supports arbitrary add_column), or stay string-only to keep pipeline_settings.py as the typed path?
Acceptance criteria
References
Summary
NWBSinkConsumerwrites annotation rows into apynwb.misc.AnnotationSeriespertable_name. As of pynwb 4.0.0 / NWB schema 2.10.0 (NWBEP001),AnnotationSeriesis deprecated in favour ofEventsTablein the new top-level/eventsgroup. Constructing anAnnotationSeriesunder pynwb >= 4.0 emits aUserWarning; reading existing files still works. We should migrate the writer.From the pynwb docs:
Current state
src/ezmsg/nwb/writer.py_prep_annotation_series()(~L792) — creates an emptypynwb.misc.AnnotationSerieswith resizableH5DataIOdata/timestamps datasets,nwbfile.add_acquisition(series), then flush + reopen to grab the live h5py datasets.write_annotation()(~L834) — resizes and appends one(timestamp, data)pair directly on the h5py datasets.NWBSink.on_annotation()(~L999) — duck-typedINPUT_ANNOTATIONsubscriber, routesflatten_for_table()["data"]towrite_annotation.annotation_data/annotation_tsafter each flush.src/ezmsg/nwb/pipeline_settings.py— the typed-column sink is described as an alternative to "the JSON-AnnotationSeriespath"; docstrings referenceAnnotationSeriesand need updating.Proposed change
table_nameAnnotationSeriesin/acquisitionwith apynwb.event.EventsTablepertable_nameadded vianwbfile.add_events_table(...)(lands in/events).timestamp(TimestampVectorData) plus anannotationtext column carrying today'sdatastring. Setsource_descriptionto something meaningful (e.g. the producer/table origin) instead of the current generic description.table.add_event(timestamp=..., annotation=...)on the live re-opened table, mirroring the existingTimeIntervalspattern inpipeline_settings.py(table.add_interval(...)after_flush_io(reopen=True)) rather than hand-managing resizable h5py datasets. That should let us dropannotation_data/annotation_tsfromNWBSinkStateand the reopen re-resolution logic.timestamp - start_timestamponce a data message has anchored the file; wall-clock passthrough before that) — the docstring onwrite_annotationstill applies.Open questions
pyproject.tomlpins onlypynwb(unversioned); the current lockfile resolves to 3.1.3, which has nopynwb.eventmodule. Migrating requirespynwb>=4.0.0, which also raises the minimum NWB schema to 2.10.0. Do we bump unconditionally, or keep anAnnotationSeriesfallback for older pynwb? Recommendation: bump the floor — dual-path writing doubles the test surface for a sink that is not yet widely deployed./acquisition/<name>AnnotationSeries; new files carry/events/<name>. Nothing inreader.py/iterator.py/slicer.pycurrently reads annotations back, so this is only a concern for external consumers — worth a note in the changelog/README.add_eventon a re-openedDynamicTablemay be more expensive per row than the current single-element h5py resize. Worth a quick benchmark if annotation rates are high.write_annotationgain a way to pass extra typed columns (theEventsTablesupports arbitraryadd_column), or stay string-only to keeppipeline_settings.pyas the typed path?Acceptance criteria
/events/<table_name>as anEventsTable, one row per message.UserWarningabout deprecated types when writing with pynwb >= 4.0.table_names still produce distinct tables (tests/test_writer.py::...around L651).tests/test_writer.py(~L580, ~L651),tests/test_integration.py(~L436).writer.pyandpipeline_settings.pyno longer referenceAnnotationSeriesas the current mechanism.pyproject.tomldependency floor updated anduv.lockrefreshed.References
pynwb.misc.AnnotationSeries(deprecation note)pynwb.event.EventsTable