fix(parquet): validate DataPageV2 decoded length#911
Conversation
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for adding the decoded-length validation — the goal is good, and the encrypted path plus the caller-level got != lenUncompressed check are correct. But readV2Unencrypted has an accumulation bug that regresses the happy path for compressed V2 pages (details inline). Requesting changes on that one check, plus a happy-path test to lock it in. Everything else looks fine.
| } | ||
| return nil | ||
| decodedLen += len(values) | ||
| if len(values) != lenUncompressed-decodedLen { |
There was a problem hiding this comment.
This is the blocking issue. decodedLen += len(values) on the line above already makes decodedLen == levelsBytelen + len(values), so lenUncompressed - decodedLen here reduces to lenUncompressed - levelsBytelen - len(values) — which is 0 for any valid page. That turns this into len(values) != 0, i.e. it rejects every non-empty compressed DataPageV2. (The encrypted sibling gets it right: len(decoded) != lenUncompressed-levelsBytelen.)
Suggested fix:
decodedLen += len(values)
if decodedLen != lenUncompressed {
return decodedLen, fmt.Errorf("parquet: metadata said %d bytes uncompressed data page, got %d bytes", lenUncompressed, decodedLen)
}I verified on this branch with a valid compressed V2 page (Snappy, no levels): Next() returns false with the self-contradictory parquet: metadata said 128 bytes uncompressed data page, got 128 bytes. The new tests here don't catch it because they only exercise the mismatched and encrypted paths — please also add a happy-path test for a valid compressed DataPageV2, since the existing TestDataPageV2 is uncompressed and TestCompression is V1-only.
zeroshade
left a comment
There was a problem hiding this comment.
The compressed-V2 regression is fixed — readV2Unencrypted now checks decodedLen != lenUncompressed, and the new TestDataPageV2Compressed guards the happy path. Verified locally: that test passes (valid compressed V2 now reads, Data() round-trips) while TestDataPageV2CorruptDecodedLength and the encrypted-mismatch test still error correctly, and the full parquet/file package is green. Thanks for the quick turnaround — LGTM.
What changed
Validation