Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to restore etcd snapshot from Netapp StorageGrid S3 endpoint #11603

Open
brandond opened this issue Jan 14, 2025 · 0 comments
Open

Unable to restore etcd snapshot from Netapp StorageGrid S3 endpoint #11603

brandond opened this issue Jan 14, 2025 · 0 comments
Assignees
Labels
area/etcd kind/upstream-issue This issue appears to be caused by an upstream bug

Comments

@brandond
Copy link
Member

brandond commented Jan 14, 2025

From rancher/rke2#6285 (reply in thread)

NetApp replied that they were able to reproduce the error in a lab environment. They inspected the network traffic and determined that the issue is caused by "the minio client library in RKE2 having a bug where gzip streamed data is terminated prematurely". They say that RKE2 sets Accept-Encoding: gzip but that the header is not honored by AWS or MinIO and thus they send non-compressed data back to RKE2. NetApp say that this is why it works on AWS but not in StorageGRID, and that StorageGRID always honors the Accept-Encording header.

It looks like minio has their own default transport which disables compression to work around this issue. We don't observe that same behavior when setting up the transports that was pass in to minio - which we do so that we can configure TLS options.
https://github.com/minio/minio-go/blob/v7.0.83/transport.go#L41-L64

We should disable compression in the http.Transport that we are passing to minio, as they do the same internally when constructing their default transports.

@brandond brandond moved this from New to Peer Review in K3s Development Jan 14, 2025
@brandond brandond self-assigned this Jan 14, 2025
@brandond brandond added area/etcd kind/upstream-issue This issue appears to be caused by an upstream bug labels Jan 14, 2025
@brandond brandond added this to the 2025-02 Release Cycle milestone Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/etcd kind/upstream-issue This issue appears to be caused by an upstream bug
Projects
Status: Peer Review
Development

No branches or pull requests

1 participant