Uvicorn app shuts down automatically - deadlock?

[NYEvernrth23]

We have a unvicorn app running via the below :

uvicorn api:app --limit-max-requests=1000 --proxy-headers --access-log --log-level=trace --host 0.0.0.0 --port 5001

Our uvicorn app hosts a websocket-based FastAPI app. Part of the logic for the app involves looping on an open user connection and continuously polling the backend for new messages to send the user.

With reliable frequency, via the logs we see the uvicorn app shuts itself down:

...
INFO:     - "GET / HTTP/1.1" 200 OK
TRACE:    - ASGI [1002] Send {'type': 'http.response.body', 'body': '<40 bytes>'}
TRACE:     - ASGI [1002] Completed
TRACE:     - HTTP connection lost
INFO:     Shutting down
TRACE:     - WebSocket connection lost
DEBUG:    = connection is CLOSED
DEBUG:    ! failing connection with code 1006
DEBUG:    x half-closing TCP connection
INFO:     Waiting for background tasks to complete. (CTRL+C to force quit)
...

but we see processes within the uvicorn app continue, and no server shutdown is reached. The processes running are within an async function.

It seems that we are deadlocked - Gunicorn is trying to shut down the worker process (see
relevant Uvicorn FastAPI issue ), but it must wait for the async process to complete before shutting down.

The net result is our Uvicorn server goes into a 502 state and cannot accept new connections, caught between trying to shutdown and waiting for our websocket-connected continuously polling to complete.

Is this relationship between Uvicorn, Gunicorn workers, and closing child processes known (where Gunicorn tries to shut itself down due to default graceful timeout but the process continues, beginning but not completing the uvicorn server shutdown)? Is there a recommendation to manage child processes? How do we prevent uvicorn from shutting itself down?

[sasha-vtz]

I have the same issue
I serve Flask with Gunicorn and FastAPI with uvicorn. I have async functions. The issue is that uvicorn processes “killing itself”. The processes on 8000 port do not exist.
I do not know how to solve that issue yet

[sasha-vtz]

We have a unvicorn app running via the below :

uvicorn api:app --limit-max-requests=1000 --proxy-headers --access-log --log-level=trace --host 0.0.0.0 --port 5001

Our uvicorn app hosts a websocket-based FastAPI app. Part of the logic for the app involves looping on an open user connection and continuously polling the backend for new messages to send the user.

With reliable frequency, via the logs we see the uvicorn app shuts itself down:

...
INFO:     - "GET / HTTP/1.1" 200 OK
TRACE:    - ASGI [1002] Send {'type': 'http.response.body', 'body': '<40 bytes>'}
TRACE:     - ASGI [1002] Completed
TRACE:     - HTTP connection lost
INFO:     Shutting down
TRACE:     - WebSocket connection lost
DEBUG:    = connection is CLOSED
DEBUG:    ! failing connection with code 1006
DEBUG:    x half-closing TCP connection
INFO:     Waiting for background tasks to complete. (CTRL+C to force quit)
...

but we see processes within the uvicorn app continue, and no server shutdown is reached. The processes running are within an async function.

It seems that we are deadlocked - Gunicorn is trying to shut down the worker process (see relevant Uvicorn FastAPI issue ), but it must wait for the async process to complete before shutting down.

The net result is our Uvicorn server goes into a 502 state and cannot accept new connections, caught between trying to shutdown and waiting for our websocket-connected continuously polling to complete.

Is this relationship between Uvicorn, Gunicorn workers, and closing child processes known (where Gunicorn tries to shut itself down due to default graceful timeout but the process continues, beginning but not completing the uvicorn server shutdown)? Is there a recommendation to manage child processes? How do we prevent uvicorn from shutting itself down?

Have you solved it?

[NYEvernrth23]

We've solved the shut down issue, but not the deadlock issue. For our app success at least, removing "limit max requests" worked:

uvicorn api:app --proxy-headers --access-log --log-level=trace --host 0.0.0.0 --port 5001

as this was responsible for the shutdown itself. If the unvicorn app never shuts down, it won't deadlock on shut down.

The other portion that may be to blame for our success - continuously poll the websockets:

 while websocket.client_state == WebSocketState.CONNECTED:
    try:
        await asyncio.wait_for(websocket.receive(), 0.0001)
    except Exception as e:
        # to keep ws session alive
        pass

As it'd turn out, our websocket object forever had the original state from the first while loop execution and would not have its state updated until we websocket.recieve(). That meant the server still considered processes as running. My guess is this is what you'd have to add in - periodic websocket.recieve().