Enable more HTTP decompression methods for remote cache #27271
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
team-Remote-Exec
fmeum
reviewed
Oct 15, 2025
src/main/java/com/google/devtools/build/lib/remote/http/HttpDownloadHandler.java
Outdated
Show resolved
Hide resolved
fmeum
approved these changes
Oct 17, 2025
tjgq
approved these changes
Oct 17, 2025
tjgq
added
awaiting-PR-merge
Contributor
|
FYI - we need to do some internal work before we can add support for Brotli, so we will submit this PR minus the Brotli changes for now. We will follow up with the Brotli changes at a later time. |
github-actions
bot
removed
the
awaiting-PR-merge
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bazel has support for compressed transfers from/to gRPC remote caches (#14041). However, this is not fully supported for HTTP protocol. This feature would allow to significantly decrease the network traffic for remote cache. The related issue is #23463.
In general, the
HTTPCacheClientclass can decompress the incoming HTTP content because it uses internally theHttpContentDecompressorfrom Netty library. But current implementation allows only "gzip" as acceptable encoding.The version of Netty library used in Bazel has been updated recently to 4.1.110. This version supports much more compression methods than before. The suggested change allows using the new compression options with HTTP remote cache. "Zstd" compression is of particular interest because it is used in popular
bazel-remotesoftware.Note: The suggested change forces the
bazel-remoteto send HTTP content with "zstd" encoding. For compatibility we may consider applying the--remote_cache_compressionoption to enable this functionality. I leaved it as is for now because "gzip" was used unconditionally.