Per-channel grids: one value, or a stack of waveforms - #6
Merged
Conversation
Both come from intent-tools, where they were written for an evoked-potential grid and have no business being application code. decimate.py is the stateless form of what SweepBuffer already does on a ring: split the sample axis into buckets and keep each bucket's min and max, so a waveform with far more samples than the screen has pixels draws its peaks rather than whatever stride decimation happened to land on. The sweep reduces samples as they stream past; a grid of retained waveforms has the whole array in hand and reduces on demand. Same arithmetic, no ring or lock. grid_layout.py is where a value heatmap and a trace grid agree: they draw completely different things into their cells but place those cells identically, one per channel at its own position and size. Most of it is graceful degradation, because real geometry is routinely missing or partial. Two deliberate omissions from the copy: - extract_layout does not come along. It reads x/y/size/label/headstage off a structured channel axis, which is decoding one data model's convention -- the business of whatever owns that convention, not of a renderer. phosphor takes plain arrays. - tile_by_headstage becomes tile_by_group. The mechanism is 'these groups reused each other's coordinate range, fan them out'; nothing about it is headstages. The geometry had no tests in intent-tools. It does now: 14 of them, mostly on the fallbacks, since bad geometry does not raise -- it draws every channel on top of every other.
One square per channel at its own position and size, coloured by value. From intent-tools, where it draws electrode impedance. It arrived knowing what a good impedance is: a mode called 'threshold', a good/open pair of cut-offs, three named colours, a classify() returning 'good'/'questionable'/'open', kOhm as the default unit, and a tooltip that called a channel open. None of that is rendering. It is one domain's reading of one measurement, and it stopped anything else from using the widget without inheriting an opinion about electrodes. So the caller classifies and the grid renders. set_colors takes the colours it decided on; set_annotations takes the words. Values still colour themselves through a colormap when nobody says otherwise, which is what a continuously varying quantity wants. classify() stays behind in the impedance app, which is the only thing that knows what those thresholds mean. Explicit colours are used verbatim, including for a channel with no value: a caller computing its own colours is the one that knows what missing should look like. The tooltip text moves into its own method so it can be tested without a canvas, which is also where 21 new tests live -- colour selection, clamping, and what hovering says.
Split out from the widget for the reason SweepBuffer is: it is pure numpy, so
the behaviour that is easy to get wrong can be tested without a GPU.
Two changes from what intent-tools does today.
A waveform now overwrites one slot instead of rolling the whole buffer so the
newest lands at index 0. Rolling copies every retained sample on every arrival,
which is affordable at one evoked potential a second and not at spike rates.
Per push, 128 channels:
evoked 10 x 15000 roll 4268 us ring 106 us 41x
spikes 100 x 48 roll 124 us ring 0.5 us 188x
spikes 200 x 48 roll 240 us ring 1.2 us 383x
The roll grows with history and the ring does not, so the deeper the stack the
worse it was. Nothing is reordered now, so callers read "ages" to find the
newest -- and an age at or above n_retained marks a slot never written, which
is the empty-slot test without a separate mask.
The mean and standard deviation now come from every waveform pushed, not from
the retained ones. An evoked response is the average of hundreds of sweeps
while only a handful are worth overlaying; tying the two together meant
retaining hundreds to average hundreds. "history" is now how many to draw, and
nothing else.
Statistics are optional, because a stack of action potentials from
possibly-different units has no meaningful average, and accumulating one costs
~4 ms per push at evoked sizes.
NaN means "this channel contributed nothing" throughout: kept as a gap in the
ring, left out of the accumulator rather than counted as a zero, and returned
as NaN rather than a flat line, which would invent a signal that was never
recorded. 22 tests.
One mini waveform plot per channel at its own position, overlaying a stack of recent waveforms and their mean. From intent-tools, where it draws evoked potentials. It recreated every graphic whenever anything changed, on the reasoning that epochs are infrequent. That holds for an evoked potential arriving once a second. It does not hold for action potentials, where the same widget would tear down and re-upload every retained waveform -- 128 channels by 100 deep -- to draw one new spike. So a change of numbers now writes into the graphics that exist, and only a change of shape recreates them: the history depth, the decimated length, or a toggle that adds or removes lines. Retention moves to TraceGridBuffer, which brings the ring and the running statistics with it. Two things follow. The mean is now over every waveform pushed rather than the retained ones, so an evoked average of hundreds of sweeps no longer requires overlaying hundreds of sweeps. history means how many to draw and nothing else. Brightness follows a slot's age rather than its position, because nothing is reordered on write. Keying on position would light up a stale waveform as soon as the ring wrapped. New, for the two cases this did not previously serve: - show_error draws +/- one standard deviation around the mean, computed at full resolution and decimated only for display, like the mean already was. - track_statistics=False for a stack with no meaningful average. - age_fade=False for spike rates, where fading costs a rewrite of the whole colour buffer per arrival: fastplotlib holds one colour per vertex, so a per-line alpha is a run of entries, and every retained waveform's age changes when one arrives. 17 tests on what can be exercised without a canvas: which slot draws brightest, what forces a recreate, and the band's geometry.
The grid drew nothing: correct cell layout, epochs arriving, empty cells for as long as the app ran. The widget is built before its first waveform, so the first animation frame rebuilt against an empty buffer, recorded the shape it had built for, and returned without creating any graphics. The shape key described sizes and toggles but not whether there was anything to draw, so it did not change when data turned up -- every later arrival took the in-place path, found all three graphics still None, and wrote nowhere. The key now says whether each graphic *should exist*, which the empty buffer answers no to and the first waveform answers yes to. The error band gets its own entry because a spread needs a second waveform, so it appears a frame later than the mean. Also fixes a bug the app never reached, because it never created a graphic at all: colours were being passed per-vertex at build time, where fastplotlib takes one per line. It holds one per vertex afterwards, so an in-place recolour does have to expand -- with the stride read off the buffer, since it also covers whatever separator vertices the renderer inserted. Same asymmetry SweepWidget already handles. Five tests drive the real sequence against a recording stub -- build, first arrival, later arrivals, clear, refill -- rather than testing the shape key in isolation, which is what let this through. Three of them fail against the previous code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four new modules, no changes to anything existing. They come from intent-tools,
where they were written for an evoked-potential viewer and had no business being
application code.
Between them they serve three plots:
ChannelGridWidgetset_values+ a colormap, orset_colorsTraceGridWidgetshow_error=True, smallhistoryTraceGridWidgettrack_statistics=False,age_fade=False, deephistorydecimate.pyMin/max envelope decimation. The stateless form of what
SweepBufferalreadydoes on a ring: split the sample axis into buckets, keep each bucket's min and
max, so a waveform with far more samples than the screen has pixels draws its
peaks rather than whatever stride decimation happened to land on. The sweep
reduces samples as they stream past; a grid of retained waveforms has the whole
array in hand and reduces on demand.
grid_layout.pyWhere a value heatmap and a trace grid agree: they draw completely different
things into their cells but place those cells identically. Most of it is
graceful degradation, because real geometry is routinely missing or partial.
Two things deliberately did not come across:
extract_layout, which readsx/y/size/label/headstageoff astructured channel axis. That is decoding one data model's convention, so it
belongs with whatever owns that convention — it has gone to
ezmsg-toolsbeside
chmeta.channel_names, which already translates the same axis intonames. phosphor takes plain arrays.
tile_by_headstage, which is nowtile_by_group. The mechanism is "thesegroups reused each other's coordinate range, fan them out"; nothing about it
is headstages.
channel_grid.pyOne square per channel at its own position and size, coloured by value.
It arrived knowing what a good electrode impedance is: a mode called
"threshold", a good/open pair of cut-offs, three named colours, aclassify()returning
"good"/"questionable"/"open",kOhmas the default unit, and atooltip that called a channel open. None of that is rendering, and it stopped
anything else — SNR, spike rate, noise floor — from using the widget without
inheriting an opinion about electrodes.
So the caller classifies and the grid renders:
set_colorstakes the colours itdecided on,
set_annotationsthe words. Values still self-colour through acolormap when nobody says otherwise.
trace_grid.py+trace_grid_buffer.pyOne mini waveform plot per channel, overlaying a stack of recent waveforms and
their mean. Split into buffer and widget for the reason
SweepBufferis: thebuffer is pure numpy, so the parts that are easy to get wrong run headlessly.
A waveform overwrites one slot instead of rolling the whole buffer so the
newest lands at index 0. Per push, 128 channels:
The roll grows with history and the ring does not, so the deeper the stack the
worse it was. Nothing is reordered, so callers read
agesto find the newest —and an age at or above
n_retainedmarks a slot never written, which is theempty-slot test without a separate mask.
Statistics span every waveform pushed, not the retained ones. An evoked
response is the average of hundreds of sweeps while only a handful are worth
overlaying; tying the two together meant retaining hundreds to average hundreds.
historynow means how many to draw and nothing else.A change of numbers writes into the graphics that exist; only a change of
shape recreates them. The original recreated everything on any change, which
holds for one epoch a second and does not for spikes, where it would tear down
and re-upload every retained waveform to draw one new one.
NaN means "this channel contributed nothing" throughout: kept as a gap, left out
of the accumulator rather than counted as a zero, and returned as NaN rather
than a flat line, which would invent a signal that was never recorded.
Testing
126 tests, up from 46. The geometry had none at all in intent-tools and now has
14, weighted toward the fallbacks, because bad geometry does not raise — it
draws every channel on top of every other.
Driven against live hardware (128-channel array, tones into the analog inputs),
which caught one bug the tests had missed: the widget is built before its first
waveform, so the first frame rebuilt against an empty buffer and recorded the
shape it had built for. The shape key described sizes and toggles but not
whether there was anything to draw, so it never changed when data arrived —
every push took the in-place path, found no graphics, and drew nothing. The
grid rendered empty cells for as long as the app ran.
The tests missed it because they checked the shape key in isolation, where "a
new waveform leaves the shape unchanged" is true and is precisely the bug. They
now drive the real sequence — build, first arrival, later arrivals, clear,
refill — against a recording stub.
🤖 Generated with Claude Code