Repository navigation
Commit 9fab878
committed
Move file-backed data into the output file instead of copying it
`FileBackedData` keeps a large blob, such as a delegate's compiled engine, in a file until the program
is written. Writing it copies the file into the `.pte`. So until the caller deletes the file
afterwards, the blob is on disk twice. For an engine of tens of gigabytes, that copy alone can make
an export run out of disk.
This adds `consume_on_write` for a caller that deletes the file after the write anyway. The first
write into a seekable file then moves the data instead of copying it:
- It copies chunks from the end of the data back to the start, each to its final offset in the
output, and truncates the source after each chunk.
- The part of the output not yet written stays a hole (a sparse file), so the disk holds the data
once, plus at most one chunk.
- Afterwards the source is empty, and a later read of the data raises an error instead of returning
a short file.
- A stream that cannot seek, such as a pipe, or a file opened to append, still gets a copy.
The default is unchanged.
Test plan: in `exir/_serialize/test/test_cord.py`, a new test counts the bytes on disk after every
write. It fails if the data is copied (twice its size on disk at the end) or if the source is only
truncated after the whole copy. Other new tests cover a pipe, a file opened to append, and a
second buffer on the same file, which now raises instead of writing a hole. `test_cord.py`,
`test_named_data_store.py` and `exir/tests/test_serde.py` pass.1 parent a532d10 commit 9fab878
2 files changed
Lines changed: 127 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
13 | | - | |
| 13 | + | |
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
18 | 25 | | |
19 | 26 | | |
20 | 27 | | |
| |||
23 | 30 | | |
24 | 31 | | |
25 | 32 | | |
| 33 | + | |
26 | 34 | | |
27 | 35 | | |
28 | 36 | | |
29 | 37 | | |
| 38 | + | |
| 39 | + | |
30 | 40 | | |
31 | 41 | | |
32 | 42 | | |
| |||
56 | 66 | | |
57 | 67 | | |
58 | 68 | | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
59 | 76 | | |
60 | | - | |
| 77 | + | |
61 | 78 | | |
62 | 79 | | |
63 | 80 | | |
64 | 81 | | |
65 | 82 | | |
66 | | - | |
| 83 | + | |
67 | 84 | | |
68 | 85 | | |
69 | 86 | | |
70 | 87 | | |
71 | 88 | | |
72 | 89 | | |
73 | | - | |
| 90 | + | |
74 | 91 | | |
75 | 92 | | |
76 | 93 | | |
77 | | - | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
78 | 103 | | |
79 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
80 | 125 | | |
81 | 126 | | |
82 | 127 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
82 | 83 | | |
83 | 84 | | |
84 | 85 | | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
0 commit comments