Include pipeline settings in the NWB epochs table

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.8.3
+    rev: v0.15.12
     hooks:
       # Run the linter.
       - id: ruff

README.md

@@ -11,6 +11,7 @@ Key features:
 * **NWB Reader** - Stream data from NWB files (local or remote) as AxisArray messages
 * **NWB Writer** - Write incoming AxisArray streams to NWB files with automatic container management
 * **Flexible clock handling** - Support for system, monotonic, and unknown reference clocks
+* **Pipeline settings logging** - Automatically record every component's settings into a `pipeline_settings` intervals table inside the NWB file
 
 ## Installation
 
@@ -48,6 +49,38 @@ from ezmsg.nwb import NWBIteratorUnit, NWBSink
 
 For general ezmsg tutorials and guides, visit [ezmsg.org](https://www.ezmsg.org).
 
+### Pipeline settings table
+
+When `NWBSink` is used inside an `ez.run` pipeline (with `ezmsg>=3.9.0`), it
+opens a `GraphContext` against the running graph server, snapshots the
+settings of every component in its session, and subscribes to subsequent
+settings change events. Each snapshot is flattened into dotted column names
+(e.g. `MY.UNIT.MyUnitSettings.endpoint.host`) and appended as a row to a
+`pipeline_settings` `TimeIntervals` table inside the NWB file, alongside an
+`updated_component` column identifying which component triggered the
+transition. Reading back is straightforward:
+
+```python
+from pynwb import NWBHDF5IO
+
+with NWBHDF5IO(path, "r") as io:
+    nwbfile = io.read()
+    df = nwbfile.intervals["pipeline_settings"].to_dataframe()
+```
+
+Notes:
+
+* Settings logging is best-effort. If the writer cannot connect to the graph
+  server (e.g. when running the consumer outside of `ez.run`), it logs a
+  warning and continues writing data without the table.
+* Settings values are sanitized for NWB storage: primitives, NumPy scalars,
+  enums, paths, and fixed-shape sequences/arrays are stored natively;
+  mappings and irregular structures are JSON-encoded; `None` becomes the
+  string `"None"`.
+* If a settings update changes a column's shape (scalar↔array or rank
+  change), the writer rotates into a new file segment (`<name>_01.nwb`,
+  `<name>_02.nwb`, …) so each segment's table stays internally consistent.
+
 ## Development
 
 We use [`uv`](https://docs.astral.sh/uv/getting-started/installation/) for development.

pyproject.toml

@@ -10,7 +10,7 @@ requires-python = ">=3.10"
 dynamic = ["version"]
 dependencies = [
   "ezmsg>=3.9.0",
-  "ezmsg-baseproc>=1.9.0",
+  "ezmsg-baseproc>=1.10.2",
   "numpy>=1.26.0",
   "pynwb",
   "h5py",
@@ -34,6 +34,8 @@ lint = [
 test = [
   "pytest>=8.0.0",
   "pytest-asyncio>=0.24.0",
+  "pytest-xdist>=3.5.0",
+  "filelock>=3.0",
 ]
 docs = [
   "sphinx>=8.0",
@@ -59,6 +61,7 @@ packages = ["src/ezmsg"]
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
 asyncio_default_fixture_loop_scope = "function"
+addopts = "-n auto --dist=loadfile"
 
 [tool.ruff]
 line-length = 120

src/ezmsg/nwb/__init__.py

@@ -5,6 +5,21 @@
 from .iterator import NWBAxisArrayIterator as NWBAxisArrayIterator
 from .iterator import NWBIteratorSettings as NWBIteratorSettings
 from .iterator import NWBIteratorState as NWBIteratorState
+from .pipeline_settings import (
+    NWBPipelineSettingsSink as NWBPipelineSettingsSink,
+)
+from .pipeline_settings import (
+    NWBPipelineSettingsSinkConsumer as NWBPipelineSettingsSinkConsumer,
+)
+from .pipeline_settings import (
+    NWBPipelineSettingsSinkSettings as NWBPipelineSettingsSinkSettings,
+)
+from .pipeline_settings import (
+    PipelineSettingsTableCollection as PipelineSettingsTableCollection,
+)
+from .pipeline_settings import (
+    PipelineSettingsTableCollectionSettings as PipelineSettingsTableCollectionSettings,
+)
 from .reader import NWBIteratorUnit as NWBIteratorUnit
 from .slicer import NWBSlicer as NWBSlicer
 from .util import ReferenceClockType as ReferenceClockType