Skip to content

fix(zstd): read entries whose input the decoder takes over several calls - #175

Open
vdavid wants to merge 1 commit into
bearcove:mainfrom
vdavid:fix-zstd-partial-input
Open

vdavid wants to merge 1 commit into
bearcove:mainfrom
vdavid:fix-zstd-partial-input

Conversation

@vdavid

@vdavid vdavid commented Oct 28, 2026

Copy link
Copy Markdown

I've found an issue: A zstd (method 93) entry fails to read when the decoder can't take all of the entry's input in one call, with this:

Decompression { method: Zstd, msg: "expected ZSTD trailer or no ZSTD trailer, but not a 35-byte trailer" }

I ran into this in Cmdr (a file manager that browses zips with rc-zip), where I noticed that a 200 KB zstd entry wouldn't open.

The cause is that the decoder stops taking input once its 32 KiB output buffer is full (or a frame ends). The check that should loop in that case compares n with in_buf.len() after in_buf was re-sliced, so it compares the bytes taken with the bytes left. When it doesn't loop on the entry's last call (HasMoreInput::No), the rest of the stream lands in the trailer check and fails.

In this PR, the decoder keeps going while it has no output yet, and otherwise hands out what it has and lets the entry FSM feed it the rest. The one-byte trailer of found-me-zstd.zip still reads (it may now be consumed on a later call, once the output before it is handed out).

  • Test: found-me-zstd-large.zip is the same "Oh no, you found me" text, 400 KB this time (made with Python's zipfile, ZIP_ZSTANDARD). On main it fails the sync, tokio, and CLI corpus tests. With the fix they pass, also with ONE_BYTE_READ=1.
  • Independent of build: update zstd for RUSTSEC advisory reasons #173: it fails the same way on zstd 0.13 and 0.14, and the fix works on both.

The leftover check compared the bytes the decoder took against the bytes left (`in_buf` was already re-sliced), so when the decoder stopped early on an entry's last call, the rest of the stream failed as "expected ZSTD trailer or no ZSTD trailer, but not a 35-byte trailer". Now leftover input either loops or hands out the output so far and lets the entry fsm feed the rest.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant