Skip to content

question: running ProcessPoolExecutor inside web-app served by hypercorn #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
konstantin-baidin-y42 opened this issue Feb 15, 2024 · 5 comments · May be fixed by #283
Open

question: running ProcessPoolExecutor inside web-app served by hypercorn #191

konstantin-baidin-y42 opened this issue Feb 15, 2024 · 5 comments · May be fixed by #283

Comments

@konstantin-baidin-y42
Copy link

konstantin-baidin-y42 commented Feb 15, 2024 •
edited
Loading

Hello, attempt to use ProcessPoolExecutor leads to

...
  File "/Users/baidinkn/my-service/my_service/app.py", line 133, in my_endpoint
    loop.run_in_executor(pool, long_fun)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 826, in run_in_executor
    executor.submit(func, *args), loop=self)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 782, in submit
    self._adjust_process_count()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 741, in _adjust_process_count
    self._spawn_process()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/process.py", line 759, in _spawn_process
    p.start()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
AssertionError: daemonic processes are not allowed to have children

ThreadPoolExecutor could work here, but it doesn't suit me, because I have a CPU-bound heavy function, so it must be a separate process instead of a thread.

Example of application:

pool = ProcessPoolExecutor()

def create_app() -> Quart:
    app = Quart(__name__)
    app.config.from_object(settings.quart)
    
    @security_scheme([])
    @app.route("/my_endpoint")
    async def my_endpoint() -> ResponseReturnValue:
        loop = asyncio.get_running_loop()
        loop.run_in_executor(pool, long_fun)
        return "Done"
        
def long_fun():
    time.sleep(4)
hypercorn dbt_service/asgi:app

Is there a way to use ProcessPoolExecutor in the web-apps served by hypercorn?
@konstantin-baidin-y42
Copy link
Author

A little update:
I found that this code works with Uvicorn and Daphne. I don't know what is exact difference. I suppose Uvicorn and Daphne don't spawn workers and work as a single process. That is why there is no daemonic process error.

But I would like to keep hypercorn as I use quart framework. Is there a way to run hypercorn as a single process, without spawning a worker? I don't need multiple workers.

@umairanis03
Copy link

umairanis03 commented May 15, 2024 •
edited
Loading

@pgjones Looks like Hypercorn worker processes are by default run with daemon set (ref). This prevents APIs like multiprocessing.Process / concurrent.futures.ProcessPoolExecutor from creating children processes from inside the application.

Is there any reason why worker processes need to be daemonized or if this flag could be configured? Note that this is not the case with application servers like uWSGI or uvicorn.

@umairanis03
Copy link

@konstantin-baidin-y42 Were you able to workaround this issue?

@konstantin-baidin-y42
Copy link
Author

@konstantin-baidin-y42 Were you able to workaround this issue?

Finally, I just switched my service to uvicorn, it works well for me 🤷‍♂️

@rafalkrupinski rafalkrupinski mentioned this issue Jul 10, 2024
Open
@vkehfdl1 vkehfdl1 mentioned this issue Nov 20, 2024
Closed
@SerGeRybakov SerGeRybakov linked a pull request Mar 13, 2025 that will close this issue
Open
@AnshSinghal
Copy link

A little update: I found that this code works with Uvicorn and Daphne. I don't know what is exact difference. I suppose Uvicorn and Daphne don't spawn workers and work as a single process. That is why there is no daemonic process error.

But I would like to keep hypercorn as I use quart framework. Is there a way to run hypercorn as a single process, without spawning a worker? I don't need multiple workers.

Can you provide the flask app code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Daemon mode switch SerGeRybakov/hypercorn
3 participants
@umairanis03 @AnshSinghal @konstantin-baidin-y42