Infinite loop in url retry

[chris-smith-zocdoc]

Describe the bug
I'm seeing bad urls (non-existant or return a bad status code) trigger an infinite loop, never times out or completes.

pants run :bad_file

file(
    name="bad_file",
    source=http_source(
        url="http://localhost:8080/foo.txt",
        len=123,
        sha256="sha",
    )
)

run_shell_command(
    name="list_files",
    command="ls -lah {chroot}",
    execution_dependencies=[":bad_file"],
)

Pants version
2.24.1

OS
Mac

Additional info
Possibly related to #21308

[sureshjoshi]

Can you check what your download attempts/delays are set to? And can you try removing retries? It might also be that the last retry just keeps going?

https://www.pantsbuild.org/stable/reference/global-options#file_downloads_max_attempts

I wonder if http_source bypasses it somehow.

[chris-smith-zocdoc]

Can you check what your download attempts/delays are set to?

I'm using the default of 4 and 0.2

It might also be that the last retry just keeps going?

It appears to be related to the retry delay. If I only set file-downloads-max-attempts higher, it stalls out on the 4th request (which happens to be the default)

Setting the delay lower however, it can progress past the 4th request and eventually complete.

[chris-smith-zocdoc]

Added some logging to figure out what is going wrong, the exponential backoff is going a little haywire.

replaced the jitter with this

pub fn jitter(duration: Duration) -> Duration {
    let r = rand::random::<f64>();
    let m = duration.mul_f64(r);
    log::debug!("Jitter random: {} old duration: {} new duration: {}", r, duration.as_millis(), m.as_millis());
    return m;
}

Using retry delay of 200 ms
Downloading http://localhost:8080/foo.txt (attempt #1)
Error while downloading http://localhost:8080/foo.txt: Error downloading file: error sending request for url (http://localhost:8080/foo.txt) (retryable)
Jitter random: 0.1434208799854798 old duration: 200 new duration: 28
Downloading http://localhost:8080/foo.txt (attempt #2)
Error while downloading http://localhost:8080/foo.txt: Error downloading file: error sending request for url (http://localhost:8080/foo.txt) (retryable)
Jitter random: 0.6670330227038747 old duration: 40000 new duration: 26681
Downloading http://localhost:8080/foo.txt (attempt #3)
Error while downloading http://localhost:8080/foo.txt: Error downloading file: error sending request for url (http://localhost:8080/foo.txt) (retryable)
Jitter random: 0.9847793764108195 old duration: 8000000 new duration: 7878235

[chris-smith-zocdoc]

The typical way I'm used to seeing exponential backoff work is described by aws here

# exponential 
sleep = min(cap, base * 2 ** attempt)

# exponential w/ jitter
sleep = random(0, min(cap, base * 2 ** attempt))

So for a base of 200ms (ignoring the cap and jitter)

The four attempts would be 200ms, 400ms, 800ms, 1600ms

Instead, what we're seeing here is 200ms, 40000ms, 8000000ms, (presumably 1600000000ms)

[chris-smith-zocdoc]

srijs/rust-tokio-retry#22

[sureshjoshi]

Good catch, that’s a ridiculous set of back off timings…

[chris-smith-zocdoc]

I'm very new to rust, but I got it running #21959 Welcome your feedback 🙏