Risingwave cannot work on EKS with S3

[pkit]

Describe the bug

Risingwave uses old asf aws-sdk-s3 library that does not support Web Identity tokens.
Which means it will fail to connect to S3 or gain any roles/permissions from IRSA on EKS.

Error message/log

called `Result::unwrap()` on an `Err` value: ObjectStore(PermissionDenied (permanent) at stat, context: { uri: https://my-bucket.s3.us-east-2.amazonaws.com/hummock/cluster_id/0, response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "04M7R7MYW6358C0K", "x-amz-id-2": "Kmf0mJj6gIpj2oTb8YoBNNRMVzEIszLxDTksPm8sG/7JeOi/WE5gE9fUyozSbXem8+EFTH5SERAWRFjI9ayQ1O0LsG+Q45qsY7QgeJ469/g=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Sat, 25 Jan 2025 16:18:09 GMT", "server": "AmazonS3"} }, service: s3, path: hummock/cluster_id/0 })

To Reproduce

Use service account with IRSA:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: risingwave-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/risingwave-pod

Expected behavior

No 403 and used role.

How did you deploy RisingWave?

Helm on EKS

The version of RisingWave

image:
  registry: docker.risingwave.com
  repository: risingwavelabs/risingwave
  tag: "v2.1.2"

Additional context

No response

[pkit]

It works on Karpenter nodes.
So it will fail only if there are two types of permissions: node-level and web-level. I suppose it's still aws_config.rs problem though. But not in RisingWave.