OSError: [Errno 9] Bad file descriptor with uvloop #10506

tkukushkin · 2025-03-02T20:25:49Z

Describe the bug

Hello! After updating aiohttp to 3.11.13 we see new errors.

To Reproduce

I don't know how to reproduce these errors, but it seems they are somehow connected with cancellation.

Expected behavior

No errors

Logs/tracebacks

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/connector.py", line 1123, in _wrap_create_connection
    connection = await self._loop.create_connection(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 2076, in create_connection
  File "uvloop/loop.pyx", line 2066, in uvloop.loop.Loop.create_connection
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/cian_http/client.py", line 159, in request
    async with self._get_session().request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
                           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/client.py", line 703, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/connector.py", line 548, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/connector.py", line 1056, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/connector.py", line 1380, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/connector.py", line 1141, in _wrap_create_connection
    sock.close()
  File "/usr/local/lib/python3.12/socket.py", line 505, in close
    self._real_close()
  File "/usr/local/lib/python3.12/socket.py", line 499, in _real_close
    _ss.close(self)
OSError: [Errno 9] Bad file descriptor

Python Version

3.13.1

aiohttp Version

3.11.13

multidict Version

6.1.0

propcache Version

0.3.0

yarl Version

1.18.3

OS

Debian 12.9

Related component

Client

Additional context

No response

Code of Conduct

I agree to follow the aio-libs Code of Conduct

mklokocka · 2025-03-03T17:36:10Z

Also having trouble with file descriptors on the latest versions of aiohttp and aiohappyeyeballs. Not using uvloop.

BlockingIOError: [Errno 115] Operation now in progress
  File "asyncio/selector_events.py", line 509, in _sock_connect
    sock.connect(address)

RuntimeError: File descriptor 265 is used by transport <_SelectorSocketTransport fd=265 read=polling write=<idle, bufsize=0>>
  File "usr/local/lib/python3.10/site-packages/<internal library>.py", line 53, in mpub_asyncio
    async with self.session_asyncio.post(f"{self.url}/mpub", timeout=timeout, **kwargs) as r:
  File "aiohttp/client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
  File "ddtrace/contrib/trace_utils_async.py", line 35, in wrapper
    return await func(mod, pin, wrapped, instance, args, kwargs)
  File "ddtrace/contrib/aiohttp/patch.py", line 108, in _traced_clientsession_request
    resp = await func(*args, **kwargs)  # type: aiohttp.ClientResponse
  File "aiohttp/client.py", line 703, in _request
    conn = await self._connector.connect(
  File "ddtrace/contrib/aiohttp/patch.py", line 62, in connect
    result = await self.__wrapped__.connect(req, *args, **kwargs)
  File "aiohttp/connector.py", line 548, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "aiohttp/connector.py", line 1056, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "aiohttp/connector.py", line 1380, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "aiohttp/connector.py", line 1116, in _wrap_create_connection
    sock = await aiohappyeyeballs.start_connection(
  File "aiohappyeyeballs/impl.py", line 93, in start_connection
    raise first_exception
  File "aiohappyeyeballs/impl.py", line 71, in start_connection
    sock = await _connect_sock(
  File "aiohappyeyeballs/impl.py", line 163, in _connect_sock
    await loop.sock_connect(sock, address)
  File "asyncio/selector_events.py", line 499, in sock_connect
    self._sock_connect(fut, sock, address)
  File "asyncio/selector_events.py", line 515, in _sock_connect
    self._ensure_fd_no_transport(fd)
  File "asyncio/selector_events.py", line 248, in _ensure_fd_no_transport
    raise RuntimeError(

bdraco · 2025-03-03T20:24:35Z

@mklokocka Are you using uvloop as well?

tkukushkin · 2025-03-03T21:57:44Z

Not using uvloop.

bdraco · 2025-03-03T22:02:46Z

Not using uvloop.

The trace back you posted has uvloop in it.

tkukushkin · 2025-03-03T22:04:20Z

I'm talking about @mklokocka comment

bdraco · 2025-03-03T22:38:18Z

@mklokocka Does the problem still happen if you disable the patching of the internals in https://github.com/DataDog/dd-trace-py/blob/main/ddtrace/contrib/internal/aiohttp/patch.py ?

The previous issues thatw ere reported focused on the aiohappyeyeballs staggered race, however the trace you have posted looks like its using the non-staggered race path https://github.com/aio-libs/aiohappyeyeballs/blob/035d976dee1f5e731852649f3fffd4e1aca21825/src/aiohappyeyeballs/impl.py#L68

Which version of aiohappyeyeballs do you have installed?

related aiohttp issue aio-libs/aiohttp#10506 In #101 we replaced the staggered race implementation since the cpython version had races that were not fixed at the time. cpython has since updated the implementation to fix additional races. Our current implementation still has problems with cancellation and cpython has fixed that in python/cpython#128475 and python/cpython#124847 This PR ports the latest cpython implementation

bdraco · 2025-03-04T03:23:47Z

Please update to aiohappyeyeballs 2.4.8 and check if the problem goes away

mklokocka · 2025-03-04T21:26:57Z

@bdraco Thanks, the project ran for a few hours with the latest aiohappyeyeballs without any issues.

bdraco · 2025-03-04T21:29:55Z

Thanks. If the problem reoccurs, let me know and we can reopen

tkukushkin · 2025-03-12T12:11:49Z

@bdraco Hello! We've updated aiohappyeyeballs to 2.6.1 and it haven't helped with errors from my original message.

bdraco · 2025-03-12T18:06:21Z

I've dug though the aiohappyeyeballs code again, and can find no place where we can work around this. so for uvloop, we are waiting for MagicStack/uvloop#646 to get fixed upstream

In the mean time you might be able to get away with disabling happyeyeballs on the connector if you don't have any use cases where you have non working IPv6

https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.TCPConnector

The amount of time in seconds to wait for a connection attempt to complete, before starting the next attempt in parallel. This is the “Connection Attempt Delay” as defined in RFC 8305. To disable Happy Eyeballs, set this to None. The default value recommended by the RFC is 0.25 (250 milliseconds).

tkukushkin · 2025-03-14T10:26:59Z

Hi @bdraco! We've tried setting happy_eyeballs_delay=None, but we're still encountering the same errors.

bdraco · 2025-03-14T17:52:54Z

Disabling happy eyeballs will only reduce the chance the problem can happen with uvloop. There isn't anything we can do further but wait for uvloop to fix the issue.

tkukushkin · 2025-03-14T18:10:18Z

In our case, all the domains we make requests to with aiohttp resolve to a single IPv4. Perhaps disabling Happy Eyeballs couldn't help us because Happy Eyeballs isn't used in the first place?

One thought keeps really bothering me – why don't we see any similar errors on aiohttp 3.9.5, but on 3.10-3.11.13, with Happy Eyeballs disabled, we do? As if something else has changed.

bdraco · 2025-03-14T23:45:59Z

#10464 was probably the most relevant change in 3.11.13

bdraco · 2025-03-14T23:49:47Z

One difference between how aiohappyeyeballs and that pr handles the close is that aiohappyeyeballs expects the socket.close() to be able to raise and traps it in https://github.com/aio-libs/aiohappyeyeballs/blob/e3bd5bdf44f5d187802de6dcb08d27e1ca6da048/src/aiohappyeyeballs/impl.py#L227

It probably makes sense to re-raise the OSError from the socket.close() as client_error(req.connection_key, exc) from exc

…ose connector socket This is a followup to #10464 to handle the case where socket.close() can also raise. This matches the logic we have in aiohappyeyeballs: https://github.com/aio-libs/aiohappyeyeballs/blob/e3bd5bdf44f5d187802de6dcb08d27e1ca6da048/src/aiohappyeyeballs/impl.py#L227 fixes #10506

bdraco · 2025-03-14T23:59:55Z

@tkukushkin Can you give #10551 a try?

…close connector socket (#10551)  ## What do these changes do? This is a followup to #10464 to handle the case where `socket.close()` can also raise. This matches the logic we have in aiohappyeyeballs: https://github.com/aio-libs/aiohappyeyeballs/blob/e3bd5bdf44f5d187802de6dcb08d27e1ca6da048/src/aiohappyeyeballs/impl.py#L227 We shouldn't raising `OSError` externally from this method as callers expect a `ClientError` ## Are there changes in behavior for the user? bugfix ## Is it a substantial burden for the maintainers to support this? no ## Related issue number fixes #10506 ## Checklist - [x] I think the code is well written - [x] Unit tests for the changes exist - [x] Documentation reflects the changes - [x] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES/` folder * name it `<issue_or_pr_num>.<type>.rst` (e.g. `588.bugfix.rst`) * if you don't have an issue number, change it to the pull request number after creating the PR * `.bugfix`: A bug fix for something the maintainers deemed an improper undesired behavior that got corrected to match pre-agreed expectations. * `.feature`: A new behavior, public APIs. That sort of stuff. * `.deprecation`: A declaration of future API removals and breaking changes in behavior. * `.breaking`: When something public is removed in a breaking way. Could be deprecated in an earlier release. * `.doc`: Notable updates to the documentation structure or build process. * `.packaging`: Notes for downstreams about unobvious side effects and tooling. Changes in the test invocation considerations and runtime assumptions. * `.contrib`: Stuff that affects the contributor experience. e.g. Running tests, building the docs, setting up the development environment. * `.misc`: Changes that are hard to assign to any of the above categories. * Make sure to use full sentences with correct case and punctuation, for example: ```rst Fixed issue with non-ascii contents in doctest text files -- by :user:`contributor-gh-handle`. ``` Use the past tense or the present tense a non-imperative mood, referring to what's changed compared to the last released version of this project. (cherry picked from commit d067260)

…ionError when failing to explicitly close connector socket (#10561) **This is a backport of PR #10551 as merged into master (d067260).**  ## What do these changes do? This is a followup to #10464 to handle the case where `socket.close()` can also raise. This matches the logic we have in aiohappyeyeballs: https://github.com/aio-libs/aiohappyeyeballs/blob/e3bd5bdf44f5d187802de6dcb08d27e1ca6da048/src/aiohappyeyeballs/impl.py#L227 We shouldn't raising `OSError` externally from this method as callers expect a `ClientError` ## Are there changes in behavior for the user? bugfix ## Is it a substantial burden for the maintainers to support this? no ## Related issue number fixes #10506 ## Checklist - [x] I think the code is well written - [x] Unit tests for the changes exist - [x] Documentation reflects the changes - [x] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES/` folder * name it `<issue_or_pr_num>.<type>.rst` (e.g. `588.bugfix.rst`) * if you don't have an issue number, change it to the pull request number after creating the PR * `.bugfix`: A bug fix for something the maintainers deemed an improper undesired behavior that got corrected to match pre-agreed expectations. * `.feature`: A new behavior, public APIs. That sort of stuff. * `.deprecation`: A declaration of future API removals and breaking changes in behavior. * `.breaking`: When something public is removed in a breaking way. Could be deprecated in an earlier release. * `.doc`: Notable updates to the documentation structure or build process. * `.packaging`: Notes for downstreams about unobvious side effects and tooling. Changes in the test invocation considerations and runtime assumptions. * `.contrib`: Stuff that affects the contributor experience. e.g. Running tests, building the docs, setting up the development environment. * `.misc`: Changes that are hard to assign to any of the above categories. * Make sure to use full sentences with correct case and punctuation, for example: ```rst Fixed issue with non-ascii contents in doctest text files -- by :user:`contributor-gh-handle`. ``` Use the past tense or the present tense a non-imperative mood, referring to what's changed compared to the last released version of this project. Co-authored-by: J. Nick Koston <[email protected]>

…ionError when failing to explicitly close connector socket (#10562) **This is a backport of PR #10551 as merged into master (d067260).**  ## What do these changes do? This is a followup to #10464 to handle the case where `socket.close()` can also raise. This matches the logic we have in aiohappyeyeballs: https://github.com/aio-libs/aiohappyeyeballs/blob/e3bd5bdf44f5d187802de6dcb08d27e1ca6da048/src/aiohappyeyeballs/impl.py#L227 We shouldn't raising `OSError` externally from this method as callers expect a `ClientError` ## Are there changes in behavior for the user? bugfix ## Is it a substantial burden for the maintainers to support this? no ## Related issue number fixes #10506 ## Checklist - [x] I think the code is well written - [x] Unit tests for the changes exist - [x] Documentation reflects the changes - [x] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` * The format is <Name> <Surname>. * Please keep alphabetical order, the file is sorted by names. - [x] Add a new news fragment into the `CHANGES/` folder * name it `<issue_or_pr_num>.<type>.rst` (e.g. `588.bugfix.rst`) * if you don't have an issue number, change it to the pull request number after creating the PR * `.bugfix`: A bug fix for something the maintainers deemed an improper undesired behavior that got corrected to match pre-agreed expectations. * `.feature`: A new behavior, public APIs. That sort of stuff. * `.deprecation`: A declaration of future API removals and breaking changes in behavior. * `.breaking`: When something public is removed in a breaking way. Could be deprecated in an earlier release. * `.doc`: Notable updates to the documentation structure or build process. * `.packaging`: Notes for downstreams about unobvious side effects and tooling. Changes in the test invocation considerations and runtime assumptions. * `.contrib`: Stuff that affects the contributor experience. e.g. Running tests, building the docs, setting up the development environment. * `.misc`: Changes that are hard to assign to any of the above categories. * Make sure to use full sentences with correct case and punctuation, for example: ```rst Fixed issue with non-ascii contents in doctest text files -- by :user:`contributor-gh-handle`. ``` Use the past tense or the present tense a non-imperative mood, referring to what's changed compared to the last released version of this project. Co-authored-by: J. Nick Koston <[email protected]>

tkukushkin added the bug label Mar 2, 2025

This was referenced Mar 3, 2025

fix: update staggered race implementation to use latest cpython version aio-libs/aiohappyeyeballs#142

Closed

fix: close runner up sockets in the event there are multiple winners aio-libs/aiohappyeyeballs#143

Merged

bdraco closed this as completed Mar 4, 2025

bdraco reopened this Mar 12, 2025

bdraco changed the title ~~OSError: [Errno 9] Bad file descriptor~~ OSError: [Errno 9] Bad file descriptor with uvloop Mar 12, 2025

bdraco added the waiting-for-upstream We are waiting for an upstream change label Mar 12, 2025

bdraco mentioned this issue Mar 14, 2025

Re-raise OSError as ClientConnectionError when failing to explicitly close connector socket #10551

Merged

5 tasks

bdraco closed this as completed in #10551 Mar 16, 2025

bdraco closed this as completed in d067260 Mar 16, 2025

patchback bot mentioned this issue Mar 16, 2025

[PR #10551/d067260d backport][3.11] Re-raise OSError as ClientConnectionError when failing to explicitly close connector socket #10561

Merged

5 tasks

patchback bot mentioned this issue Mar 16, 2025

[PR #10551/d067260d backport][3.12] Re-raise OSError as ClientConnectionError when failing to explicitly close connector socket #10562

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSError: [Errno 9] Bad file descriptor with uvloop #10506

OSError: [Errno 9] Bad file descriptor with uvloop #10506

tkukushkin commented Mar 2, 2025

mklokocka commented Mar 3, 2025

bdraco commented Mar 3, 2025

tkukushkin commented Mar 3, 2025

bdraco commented Mar 3, 2025

tkukushkin commented Mar 3, 2025

bdraco commented Mar 3, 2025 •

edited

Loading

bdraco commented Mar 4, 2025

mklokocka commented Mar 4, 2025

bdraco commented Mar 4, 2025

tkukushkin commented Mar 12, 2025

bdraco commented Mar 12, 2025 •

edited

Loading

tkukushkin commented Mar 14, 2025

bdraco commented Mar 14, 2025

tkukushkin commented Mar 14, 2025

bdraco commented Mar 14, 2025

bdraco commented Mar 14, 2025 •

edited

Loading

bdraco commented Mar 14, 2025

OSError: [Errno 9] Bad file descriptor with uvloop #10506

OSError: [Errno 9] Bad file descriptor with uvloop #10506

Comments

tkukushkin commented Mar 2, 2025

Describe the bug

To Reproduce

Expected behavior

Logs/tracebacks

Python Version

aiohttp Version

multidict Version

propcache Version

yarl Version

OS

Related component

Additional context

Code of Conduct

mklokocka commented Mar 3, 2025

bdraco commented Mar 3, 2025

tkukushkin commented Mar 3, 2025

bdraco commented Mar 3, 2025

tkukushkin commented Mar 3, 2025

bdraco commented Mar 3, 2025 • edited Loading

bdraco commented Mar 4, 2025

mklokocka commented Mar 4, 2025

bdraco commented Mar 4, 2025

tkukushkin commented Mar 12, 2025

bdraco commented Mar 12, 2025 • edited Loading

tkukushkin commented Mar 14, 2025

bdraco commented Mar 14, 2025

tkukushkin commented Mar 14, 2025

bdraco commented Mar 14, 2025

bdraco commented Mar 14, 2025 • edited Loading

bdraco commented Mar 14, 2025

bdraco commented Mar 3, 2025 •

edited

Loading

bdraco commented Mar 12, 2025 •

edited

Loading

bdraco commented Mar 14, 2025 •

edited

Loading