HttpClient may leak UnobservedTaskException in CheckUsabilityOnScavenge

[salvois]

Description

We have a .NET 9 console application (namely, a Windows Service running on Microsoft.Extensions.Hosting.BackgroundService, hosted by Windows Server 2019 on x86-64) making multiple HTTP requests (GETs or POSTs) using a singleton instance of HttpClient created during service startup using new HttpClient { Timeout = TimeSpan.FromMinutes(10) };

All otherwise unhandled application exceptions are handled by a top level catch that does any logging and deadlettering required for our processing. We also register a handler to TaskScheduler.UnobservedTaskException to log any exception that may leak, usually from our code forgetting an await.

In several cases, under heavy load, our handler for UnobservedTaskException's received exceptions from what appear to be the internals of HttpClient itself, such as in the following stack trace, apparently related to #61256:

System.IO.IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
---> System.Net.Sockets.SocketException (995): The I/O operation has been aborted because of either a thread exit or an application request.
— End of inner exception stack trace —
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
at System.Net.Http.HttpConnection.<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|39_0()

We also experienced #104324 when the application ran on .NET 8, and upgraded to .NET 9 hoping to solve the issue, but we are still receiving the exception about CheckUsabilityOnScavenge nonetheless.

Reproduction Steps

Unfortunately we have not been able to reproduce the problem deterministically. It seems related to our system being under heavy load.

Expected behavior

We expect no unobserved exceptions, even less exceptions from internal implementation.

Actual behavior

An UnobservedTaskException is caught by the handler registered to TaskScheduler.UnobservedTaskException, which is very scary to us because in our mind "unobserved task excetion == possible data loss" 😄 (due to the deadletter mechanism being bypassed).
If I understand the implementation correctly (which I don't fully, I must confess), that should not be the case in our scenario, though, because the exception seems to be thrown when the HTTP connection is about to be dropped anyway, and our application should proceed normally. Please advise whether this is the case.

Regression?

We don't have memory of this issue nor #104324 prior to switching to .NET 8 (the application used .NET 6, .NET 5, .NET Core 2 and .NET Framework 4.6.2 before).

Known Workarounds

No response

Configuration

No response

Other information

Based on a quick review of the source of CheckUsabilityOnScavenge in HttpConnection, I think a solution similar to PR #104335 may do the job. I can try to create a pull request based on it.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.