Repository navigation
Conversation
The leftover check compared the bytes the decoder took against the bytes left (`in_buf` was already re-sliced), so when the decoder stopped early on an entry's last call, the rest of the stream failed as "expected ZSTD trailer or no ZSTD trailer, but not a 35-byte trailer". Now leftover input either loops or hands out the output so far and lets the entry fsm feed the rest.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've found an issue: A
zstd(method 93) entry fails to read when the decoder can't take all of the entry's input in one call, with this:I ran into this in Cmdr (a file manager that browses zips with
rc-zip), where I noticed that a 200 KBzstdentry wouldn't open.The cause is that the decoder stops taking input once its 32 KiB output buffer is full (or a frame ends). The check that should loop in that case compares
nwithin_buf.len()afterin_bufwas re-sliced, so it compares the bytes taken with the bytes left. When it doesn't loop on the entry's last call (HasMoreInput::No), the rest of the stream lands in the trailer check and fails.In this PR, the decoder keeps going while it has no output yet, and otherwise hands out what it has and lets the entry FSM feed it the rest. The one-byte trailer of
found-me-zstd.zipstill reads (it may now be consumed on a later call, once the output before it is handed out).found-me-zstd-large.zipis the same "Oh no, you found me" text, 400 KB this time (made with Python'szipfile,ZIP_ZSTANDARD). Onmainit fails the sync, tokio, and CLI corpus tests. With the fix they pass, also withONE_BYTE_READ=1.zstd0.13 and 0.14, and the fix works on both.