S3Client.deleteObjects doesn't work in 2.30.0

[wendigo]

Describe the bug

deleteObjects now fails with Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Missing required header for this request: Content-Md5. (Service: S3, Status Code: 400, Request ID: 181B4664E1D4B169, Extended Request ID: 39697cb5cac07bf2a227e0cd44c7b2e9184e7598e2abe41c55eb5028b44676ca) on Minio and other S3-compatible storages, while deleteObject with a single location works just fine.

This fails regardless of the:

.responseChecksumValidation()
.requestChecksumCalculation()

set to WHEN_REQUIRED. WHEN_REQUIRED seems to work fine for other API calls but not for deleteObjects.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I'd expect deleteObjects to work the same as in the 2.29.x.

Current Behavior

For deleteObjects:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Missing required header for this request: Content-Md5. (Service: S3, Status Code: 400, Request ID: 181B46929A850D93, Extended Request ID: 39697cb5cac07bf2a227e0cd44c7b2e9184e7598e2abe41c55eb5028b44676ca)

Since I'm passing list of locations, I'm not sure why the Content-Md5 would be required.

For deleteObject: passes.

Reproduction Steps

Batch delete files stored on Minio. Single delete works fine.

Possible Solution

Downgrade to 2.29.x

Additional Information/Context

No response

AWS Java SDK version used

2.30.0

JDK version used

23

Operating System and version

Linux

[zoewangg]

Taking a look

[debora-ito]

Hi @wendigo, we tested deleteObjects with the aws s3 endpoint and it succeeds with the WHEN_REQUIRED setting.

My guess is that the issue is on the Minio/third-party side, they have to accommodate the changes released in 2.30.0. If the issue is indeed on the third-party side, I can't really explain why deleteObject works fine.

One workaround is to pin the Java SDK version to 2.29.52 until Minio adds support for the change.

[wendigo]

@debora-ito why the deleteObject works while deleteObjects doesn't? This doesn't make much sense

[debora-ito]

The exception is being returned by Minio, the SDK is only unmarshalling the response. There's probably a difference in how Minio implemented the s3 compatibility internally.

[lucix-aws]

DeleteObjects has always required a checksum, before today SDKs would compute an MD5 one for these requests. After today's object integrity change, SDKs now send a CRC32 checksum instead. Minio's implementation still appears to expect MD5.

DeleteObject singular optionally used checksums before, all the SDKs changed in that regard is sending one by default now.