Fix flaky test_no_connection_refused_on_timeout #612

Job	Run time
Test wheels building / Prepare matrix to run for 3.13 on `linux,macos-x86,macos-arm,windows,linux-aarch64`	3s
Test wheels building / Build source distribution	12s
Test wheels building / Build wheels for windows on windows-2022	46m 7s
Test wheels building / Build wheels for linux-aarch64 on ubuntu-24.04-arm	14m 26s
Test wheels building / Build wheels for linux on ubuntu-24.04	14m 31s
Test wheels building / Build wheels for macos-arm on macos-14	12m 21s
Test wheels building / Build wheels for macos-x86 on macos-15-intel	20m 30s
Test wheels building / upload_pypi	0s
	1h 48m 10s

Provide feedback