Skip to content

Conversation

TingDaoK
Copy link
Contributor

Issue #, if available:

  • The only step for preparation needs to be serial is reading from the original body for now.
  • So, invoke the callback after reading from separate threads to make the following steps to happen in parallel
  • For s3 express, which we can hit 100Gb/s upload from a 30 GiB streaming data, the rest of the steps happens in the critical path make the performance drops to around 60 - 70 gbps
  • With this change, we can hit 100gbps upload again with parts level checksum enabled for that workload.

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@TingDaoK TingDaoK changed the title prepare in parallel prepare in parallel after reading Apr 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant