Repository navigation
Commit 50b7e01
authored
Fix/memorystore buffer aliasing (#4157)
* fix: byte-order handling for structured dtypes in the bytes codec (#220)
* fix: byte-order handling for structured dtypes in the bytes codec
The bytes codec neither byte-swapped structured-dtype fields to its
configured endian on encode (numpy reports byteorder '|' for void
dtypes, so the top-level byteorder comparison never detected a
mismatch) nor honored its endian when decoding, silently corrupting
any structured data whose field byte order differed from the stored
one (e.g. virtual references to external big-endian data).
Encode now detects byte-order mismatches by comparing full dtypes via
newbyteorder, and decode reinterprets raw bytes in the stored byte
order before converting to the data type's declared byte order, so the
stored layout (codec state) and the in-memory layout (array data type)
are independent.
Closes #4141
Assisted-by: ClaudeCode:claude-fable-5
* test: fold structured byte-order cases into existing bytes codec tests
Extend test_endian's parametrization with structured dtypes and
test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus
stored-layout and decoded-dtype assertions, instead of adding parallel
test functions for the same properties.
Assisted-by: ClaudeCode:claude-fable-5
* refactor: rename stored_dtype to view_dtype in BytesCodec decode
The variable is the dtype used to view the raw chunk bytes (byte order
from the codec's endian configuration), not a property of the stored
data or of the returned buffer, which always carries the array's
declared dtype.
Assisted-by: ClaudeCode:claude-fable-5
* docs: note that the decode-side byte-order conversion copies the chunk
Assisted-by: ClaudeCode:claude-fable-5
* fix(store): copy buffers on write in MemoryStore
Encoding an uncompressed chunk hands the store a zero-copy view of the
caller's array, and MemoryStore keeps whatever it is given alive in a
dict rather than serializing it. Mutating the source array after a write
therefore rewrote chunks already committed to the store, silently.
Stores that serialize on write (LocalStore, ZipStore, remote stores) are
unaffected, so they keep the full benefit of #3885. Only MemoryStore pays
the copy, and only where it was aliasing to begin with: an uncompressed
34 MB write goes from ~23 ms to ~39 ms, while compressed writes are
unchanged.
Copying at the store boundary rather than narrowing the fast path in
_merge_chunk_array also fixes the single-chunk case, which aliased in
v3.2.1 too.
Assisted-by: ClaudeCode:claude-opus-4.8
* docs: add changelog entry for MemoryStore buffer copy
Assisted-by: ClaudeCode:claude-opus-4.8
* docs: correct changelog1 parent 3265452 commit 50b7e01
4 files changed
Lines changed: 84 additions & 4 deletions
File tree
- changes
- src/zarr
- core/buffer
- storage
- tests/test_store
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
48 | 50 | | |
49 | 51 | | |
50 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
29 | 41 | | |
30 | 42 | | |
31 | 43 | | |
| |||
42 | 54 | | |
43 | 55 | | |
44 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
45 | 63 | | |
46 | 64 | | |
47 | 65 | | |
| |||
117 | 135 | | |
118 | 136 | | |
119 | 137 | | |
120 | | - | |
| 138 | + | |
121 | 139 | | |
122 | 140 | | |
123 | 141 | | |
| |||
178 | 196 | | |
179 | 197 | | |
180 | 198 | | |
181 | | - | |
| 199 | + | |
182 | 200 | | |
183 | 201 | | |
184 | 202 | | |
185 | 203 | | |
186 | 204 | | |
187 | | - | |
| 205 | + | |
188 | 206 | | |
189 | 207 | | |
190 | 208 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
79 | 128 | | |
80 | 129 | | |
81 | 130 | | |
| |||
0 commit comments