Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Child process died #2506

Open
2 tasks done
currenttime opened this issue Nov 7, 2024 · 3 comments
Open
2 tasks done

Child process died #2506

currenttime opened this issue Nov 7, 2024 · 3 comments

Comments

@currenttime
Copy link

Initial Checks

  • I confirm this was discussed, and the maintainers suggest I open an issue.
  • I'm aware that if I created this issue without a discussion, it may be closed without a response.

Discussion Link

Command:
uvicorn.run(app='api_server:app', host="0.0.0.0", port=8002, workers=2)

Log:
/ai/miniconda3/bin/conda run -n py311_catvton --no-capture-output python /ai/brx/CatVTON-edited/api_server.py 
/ai/miniconda3/envs/py311_catvton/lib/python3.11/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
An error occurred while trying to fetch booksforcharlie/stable-diffusion-inpainting: booksforcharlie/stable-diffusion-inpainting does not appear to have a file named diffusion_pytorch_model.safetensors.
Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.
/ai/miniconda3/envs/py311_catvton/lib/python3.11/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
INFO:     Uvicorn running on http://0.0.0.0:8002 (Press CTRL+C to quit)
INFO:     Started parent process [15707]
INFO:     Waiting for child process [15885]
INFO:     Child process [15885] died
Fetching 10 files: 100%|█████████████████████| 10/10 [00:00<00:00, 46345.90it/s]
Fetching 10 files: 100%|█████████████████████| 10/10 [00:00<00:00, 91779.08it/s]
Fetching 10 files: 100%|█████████████████████| 10/10 [00:00<00:00, 90394.48it/s]
INFO:     Waiting for child process [15886]
INFO:     Child process [15886] died
/ai/miniconda3/envs/py311_catvton/lib/python3.11/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
/ai/miniconda3/envs/py311_catvton/lib/python3.11/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
`text_config_dict` is provided which will be used to initialize `CLIPTextConfig`. The value `text_config["id2label"]` will be overriden.
Fetching 10 files: 100%|█████████████████████| 10/10 [00:00<00:00, 22465.47it/s]
INFO:     Waiting for child process [15887]
/ai/miniconda3/envs/py311_catvton/lib/python3.11/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
INFO:     Child process [15887] died

Description

Child process died

Example Code

No response

Python, Uvicorn & OS Version

Running uvicorn 0.30.0 with CPython 3.11.10 on Linux
@Yilimmilk
Copy link

Yilimmilk commented Dec 16, 2024

same here.

I encountered issues running my FastAPI project using different versions of Uvicorn within a Docker container. Details are as follows:

  1. Environment

    • FastAPI version: 0.115.6
    • Python version: 3.12-alpine
    • Running in Docker
  2. Issue with uvicorn <= 0.29.0
    When I use Uvicorn versions <= 0.29.0, the process gets permanently stuck at the following log message with no more output:

2024/12/16 11:44:36.354     INFO: [uvicorn.error] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)  
2024/12/16 11:44:36.354     INFO: [uvicorn.error] Started parent process [1]  
  1. Issue with uvicorn >= 0.30.0 (including the latest version 0.34.0)
    With newer versions of Uvicorn, the process starts, but the child processes keep dying repeatedly:
2024/12/16 12:20:06.593     INFO: [uvicorn.error] Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024/12/16 12:20:06.594     INFO: [uvicorn.error] Started parent process [1]
2024/12/16 12:20:07.622     INFO: [uvicorn.error] Waiting for child process [18]
2024/12/16 12:20:07.623     INFO: [uvicorn.error] Child process [18] died
2024/12/16 12:20:07.623     INFO: [uvicorn.error] Waiting for child process [19]
2024/12/16 12:20:07.624     INFO: [uvicorn.error] Child process [19] died
2024/12/16 12:20:08.127     INFO: [uvicorn.error] Waiting for child process [19]
2024/12/16 12:20:08.129     INFO: [uvicorn.error] Child process [19] died
2024/12/16 12:20:13.136     INFO: [uvicorn.error] Waiting for child process [23]
2024/12/16 12:20:13.137     INFO: [uvicorn.error] Child process [23] died
... (repeats indefinitely) ...  
  1. Docker Setup
    It's my Dockerfile:
FROM python:3.12-alpine  

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories \  
    && apk add --update caddy gcc musl-dev libffi-dev  

WORKDIR /app  
COPY Backend/requirements.txt /tmp/requirements.txt  
RUN pip install --no-cache-dir -r /tmp/requirements.txt -i https://mirrors.aliyun.com/pypi/simple  

COPY Backend ./backend  

EXPOSE 8000  

COPY startup.sh /app/startup.sh  
RUN chmod +x /app/startup.sh  
CMD ["/app/startup.sh"]  

My startup.sh script:

#!/bin/sh  
set -e  

cd /app/backend  

exec uvicorn main:get_app \  
    --host 0.0.0.0 \  
    --port 8000 \  
    --workers 2 \  
    --proxy-headers \  
    --forwarded-allow-ips '*' \  
    --factory \  
    --log-config logging_config.yaml  
  1. Local Machine Behavior
    When I run the same command directly on my Windows11 PC, The program can run normally:
uvicorn main:get_app --host 0.0.0.0 --port 8000 --workers 2 --proxy-headers --factory --log-config logging_config.yaml  

@hunter2009pf

This comment has been minimized.

@sriramr98
Copy link

I've also experienced a similar behavior, but in my case, the child process doesn't die on startup. When a request is received, it fetches a lot of data from a database to process it. As soon as the machine RAM exceeds 4-5GB, the child process dies. Now my machine's RAM is 64GB and when I was testing, there was no other request to cause RAM congestion.

However, I noticed that if I run it in a single worker mode, it does not happen.

Here's the issue link I posted
https://stackoverflow.com/questions/79311202/fastapi-child-process-gets-killed-even-with-enough-unused-ram-and-cpu-left-in-th?noredirect=1#comment139857620_79311202

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants