Repository navigation
Commit fcfc152
committed
Let a write move file-backed data into the output instead of copying it
`FileBackedData` keeps a large blob, such as a delegate's compiled engine, in a file until the
program is written. Writing it copies the file into the `.pte`. So until the caller deletes the
file afterwards, the blob is on disk twice. For an engine of tens of gigabytes, that copy alone
can make an export run out of disk.
This adds `consume=True` to `FileBackedData.write_to_file`, `Cord.write_to_file` (whose new
`keep` names buffers to copy anyway), and `ExecutorchProgramManager.write_to_file`,
`write_tensor_data_to_file` and `save`. With it, data that owns its file (`cleanup=True`, as
from `move_from`) is moved into the output:
- Chunks are copied from the end of the data back to the start, each to its final offset. The
source is truncated after each chunk, once the write returned the full length and was flushed.
- The part of the output not yet written stays a hole (a sparse file), so the disk holds the data
once, plus at most one chunk.
- Afterwards the program can't be written or read again. Any `FileBackedData` on an emptied file
now raises instead of reading a short file, because the check is on the file's size, not a flag.
- A buffer that deduplication put in both the `.pte` and a `.ptd` (or in two `.ptd` files) is
copied, because no single write knows it is the last one to need it.
Everything else gets a copy, as before: data that does not own its file, a file with another
hard link or that is not writable, an output that is not a regular file opened through
`io.FileIO` or a buffered writer, an output opened with `O_APPEND`, and Windows, where files are
not sparse by default.
A consuming write that fails before its first truncation leaves the data intact, so it can be
written again. One that fails later adds a note to the error that says which bytes are still in
the source and where the rest is in the output, and leaves that source file in place.
`ExecutorchProgramManager.save` now re-raises a failed write instead of only logging it. Before,
a failure returned normally and left a broken `.pte` behind.
The default is unchanged: without `consume`, every write copies.
Test plan: `exir/_serialize/test/test_cord.py` and `exir/program/test/test_program.py`. New tests
check:
- the source still holds each range when it reaches the output, and allocated blocks never
exceed the data plus one chunk;
- only a consuming write consumes;
- the copy fallbacks: unowned data, a second hard link, Windows, `BytesIO`, `/dev/null`,
`O_APPEND`, gzip, a write-only object, and a pipe that reports it can seek, as pipes do on
Windows;
- a short write, and a failure on a later chunk;
- a second buffer on an emptied file, a read-only file, and buffers listed in `keep`;
- a buffer shared by the `.pte` and a `.ptd` ends up whole in both;
- a save without `consume` can be repeated, and `buffer` still works after it;
- `save` re-raises, and `save` and `write_tensor_data_to_file` pass `consume` on.
`test_named_data_store.py` and `exir/tests/test_serde.py` also pass.1 parent a532d10 commit fcfc152
4 files changed
Lines changed: 423 additions & 19 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | | - | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
18 | 33 | | |
19 | 34 | | |
20 | 35 | | |
| |||
56 | 71 | | |
57 | 72 | | |
58 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
59 | 86 | | |
60 | | - | |
| 87 | + | |
61 | 88 | | |
62 | 89 | | |
63 | 90 | | |
64 | 91 | | |
65 | 92 | | |
66 | | - | |
| 93 | + | |
67 | 94 | | |
68 | 95 | | |
69 | 96 | | |
70 | 97 | | |
71 | 98 | | |
72 | 99 | | |
73 | | - | |
| 100 | + | |
74 | 101 | | |
75 | 102 | | |
76 | | - | |
77 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
78 | 110 | | |
79 | 111 | | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
80 | 164 | | |
81 | 165 | | |
82 | 166 | | |
| |||
131 | 215 | | |
132 | 216 | | |
133 | 217 | | |
134 | | - | |
135 | | - | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
136 | 234 | | |
137 | 235 | | |
138 | 236 | | |
139 | 237 | | |
140 | | - | |
| 238 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| 12 | + | |
11 | 13 | | |
| 14 | + | |
12 | 15 | | |
13 | 16 | | |
14 | 17 | | |
| |||
82 | 85 | | |
83 | 86 | | |
84 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
0 commit comments