Add note to use s5cmd for large transfers to LTS#948
Conversation
There was a problem hiding this comment.
This is true, but should be temporary. The issue should be resolved in a couple of weeks or so when Ceph backports a bugfix to our version. Please note the temporary nature and we'll get this merged.
Additionally, this affects all data going in or out of LTS right now, to any other endpoint.
Thank you for this.
|
I'm not sure this is true. I know there's a slowdown due to that bug, but Globus has always been slower to transfer data to LTS than using the CLI. For instance, on Friday I tested a transfer of 390 GB of data using both s5cmd and Globus. s5cmd with a modest worker setup (12 cores split into 4 workers with 3 concurrent transfers) transferred that amount of data in around 30-40 minutes while Globus took 3 hours. Maybe the amount of compute Globus had access to for copying those files was much lower than the s5cmd task, but I'm not confident that once the bug is patched that Globus will immediately become as performant as s5cmd. |
|
I'm going to change the wording for this in the PR to something like:
With appropriate URLs. We'll need to get the I'll go ahead and move #956 forward, or something like it, in the meantime. |
|
Closing in favor of #965 |
Added note at the top of the Globus section on LTS's interfaces explaining that large data should be transferred using s5cmd instead of Globus. We've run into a number of tickets asking about slow transfers when using Globus. Many people probably use Globus since it's the first option listed on the page and is generally the easiest to use. I added a note at the top of that section directing people to s5cmd for larger data transfers