Add and test Gunicorn workers (#116)

This project supports the Gunicorn web server. Gunicorn's server design
includes a primary "arbiter" process that spawns "worker" processes.
Workers are child processes, each with their own running server. Workers
are implemented as Python classes. Custom workers can be supplied.

This project also supports the Uvicorn web server. In the past, Uvicorn
supplied workers for use with Gunicorn. The Uvicorn workers were not
tested. The `uvicorn.workers` module was completely omitted from
coverage measurement due to use of the coverage.py `include` setting
to specify source files.

Efforts were made to test the Uvicorn workers
(encode/uvicorn#1834,
encode/uvicorn#1995),
but the workers were arbitrarily deprecated and moved to
someone's personal project (encode/uvicorn#2302),
instead of an Encode-managed project as would have been expected
(encode/uvicorn#517 (comment))

Rather than introducing a production dependency on a separate Uvicorn
workers package that is not managed by Encode, this commit will add the
Gunicorn workers directly to this project.

This commit will add the code from `uvicorn.workers` to a new module
`inboard/gunicorn_workers.py`. The code will be preserved as it was
prior to deprecation, with a copy of the Uvicorn license and necessary
updates for compliance with the code quality settings in this project:

- Ruff
  [UP008](https://docs.astral.sh/ruff/rules/super-call-with-parameters/)
- Ruff
  [UP035](https://docs.astral.sh/ruff/rules/deprecated-import/)
- mypy `[attr-defined]` - Module "uvicorn.main" does not explicitly
  export attribute "Server"
- mypy `[import-untyped]` - `gunicorn.arbiter`
- mypy `[import-untyped]` - `gunicorn.workers.base`
- mypy `[misc]` - Class cannot subclass "Worker" (has type "Any")
- mypy `[type-arg]` - Missing type parameters for generic type "dict"
  (on `config_kwargs`)

This commit will also add tests of 100% of the Gunicorn worker code to a
new module `tests/test_gunicorn_workers.py`.

A test fixture starts a subprocess running Gunicorn with a Uvicorn worker
and an ASGI app. The subprocess includes an instance of `httpx.Client`
for HTTP requests to the Uvicorn worker's ASGI app, and saves its output
to a temporary file for assertions on `stdout`/`stderr`. Tests can send
operating system signals to the process.

The coverage.py configuration will be updated for subprocess test
coverage measurement. Changes to coverage measurement include:

- Enable the required parallel mode (note that it is important to ensure
  the `.gitignore` ignores files named `.coverage.*` because many coverage
  files are generated when subprocesses are measured in parallel mode)
- Set the required `COVERAGE_PROCESS_START` environment variable
- Add the `coverage_enable_subprocess` package to invoke
  `coverage.process_startup`
- Combine coverage reports before reporting coverage
- Add instructions to `contributing.md` about how to omit subprocess
  tests

Related:

https://github.com/encode/uvicorn/blob/4fd507718eb0313e2de66123e6737b054088f722/LICENSE.md
https://github.com/encode/uvicorn/blob/4fd507718eb0313e2de66123e6737b054088f722/uvicorn/workers.py
encode/uvicorn#517 (comment)
encode/uvicorn#1834
encode/uvicorn#1995
encode/uvicorn#2302

https://coverage.readthedocs.io/en/latest/subprocess.html
https://docs.gunicorn.org/en/latest/design.html
https://docs.gunicorn.org/en/latest/signals.html
https://www.uvicorn.org/deployment/#gunicorn

.github/workflows/ci.yml

@@ -110,9 +110,14 @@ jobs:
       - name: Run Hatch script for code quality checks
         run: hatch run ${{ env.HATCH_ENV }}:check
       - name: Run tests
-        run: hatch run ${{ env.HATCH_ENV }}:coverage run
+        run: |
+          export COVERAGE_PROCESS_START="$PWD/pyproject.toml"
+          hatch run ${{ env.HATCH_ENV }}:coverage run
+        timeout-minutes: 5
       - name: Enforce test coverage
-        run: hatch run ${{ env.HATCH_ENV }}:coverage report
+        run: |
+          hatch run ${{ env.HATCH_ENV }}:coverage combine -q
+          hatch run ${{ env.HATCH_ENV }}:coverage report
       - name: Build Python package
         run: hatch build
       - name: Upload Python package artifacts

docs/contributing.md

@@ -72,6 +72,7 @@ As explained in the [VSCode docs](https://code.visualstudio.com/docs/containers/
 - [pytest configuration](https://docs.pytest.org/en/latest/reference/customize.html) is in _[pyproject.toml](https://github.com/br3ndonland/inboard/blob/develop/pyproject.toml)_.
 - [FastAPI testing](https://fastapi.tiangolo.com/tutorial/testing/) and [Starlette testing](https://www.starlette.io/testclient/) rely on the [Starlette `TestClient`](https://www.starlette.io/testclient/).
 - Test coverage reports are generated by [coverage.py](https://github.com/nedbat/coveragepy). To generate test coverage reports, first run tests with `coverage run`, then generate a report with `coverage report`. To see interactive HTML coverage reports, run `coverage html` instead of `coverage report`.
+- Some of the tests start separate subprocesses. These tests are more complex in some ways, and can take longer, than the standard single-process tests. A [pytest mark](https://docs.pytest.org/en/stable/example/markers.html) is included to help control the behavior of subprocess tests. To run the test suite without subprocess tests, select tests with `coverage run -m pytest -m "not subprocess"`. Note that test coverage will be lower without the subprocess tests.
 
 ## Docker
 

docs/environment.md

@@ -94,8 +94,8 @@ ENV APP_MODULE="package.custom.module:api" WORKERS_PER_CORE="2"
 [`WORKER_CLASS`](https://docs.gunicorn.org/en/latest/settings.html#worker-processes)
 
 - Uvicorn worker class for Gunicorn to use.
-- Default: `uvicorn.workers.UvicornWorker`
-- Custom: For the [alternate Uvicorn worker](https://www.uvicorn.org/deployment/), `WORKER_CLASS="uvicorn.workers.UvicornH11Worker"` _(the H11 worker is provided for [PyPy](https://www.pypy.org/) and hasn't been tested)_
+- Default: `inboard.gunicorn_workers.UvicornWorker`
+- Custom: For the [alternate Uvicorn worker](https://www.uvicorn.org/deployment/), `WORKER_CLASS="inboard.gunicorn_workers.UvicornH11Worker"` _(the H11 worker is provided for [PyPy](https://www.pypy.org/))_
 
 ### Worker process calculation
 

inboard/gunicorn_workers.py

@@ -0,0 +1,134 @@
+"""
+Copyright © 2017-present, [Encode OSS Ltd](https://www.encode.io/).
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+* Redistributions of source code must retain the above copyright notice, this
+  list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright notice,
+  this list of conditions and the following disclaimer in the documentation
+  and/or other materials provided with the distribution.
+
+* Neither the name of the copyright holder nor the names of its
+  contributors may be used to endorse or promote products derived from
+  this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+"""
+
+import asyncio
+import logging
+import signal
+import sys
+from typing import Any
+
+from gunicorn.arbiter import Arbiter  # type: ignore[import-untyped]
+from gunicorn.workers.base import Worker  # type: ignore[import-untyped]
+from uvicorn.config import Config
+from uvicorn.server import Server
+
+
+class UvicornWorker(Worker):  # type: ignore[misc]
+    """
+    A worker class for Gunicorn that interfaces with an ASGI consumer callable,
+    rather than a WSGI callable.
+    """
+
+    CONFIG_KWARGS: dict[str, Any] = {"loop": "auto", "http": "auto"}
+
+    def __init__(self, *args: Any, **kwargs: Any) -> None:
+        super().__init__(*args, **kwargs)
+
+        logger = logging.getLogger("uvicorn.error")
+        logger.handlers = self.log.error_log.handlers
+        logger.setLevel(self.log.error_log.level)
+        logger.propagate = False
+
+        logger = logging.getLogger("uvicorn.access")
+        logger.handlers = self.log.access_log.handlers
+        logger.setLevel(self.log.access_log.level)
+        logger.propagate = False
+
+        config_kwargs: dict[str, Any] = {
+            "app": None,
+            "log_config": None,
+            "timeout_keep_alive": self.cfg.keepalive,
+            "timeout_notify": self.timeout,
+            "callback_notify": self.callback_notify,
+            "limit_max_requests": self.max_requests,
+            "forwarded_allow_ips": self.cfg.forwarded_allow_ips,
+        }
+
+        if self.cfg.is_ssl:
+            ssl_kwargs = {
+                "ssl_keyfile": self.cfg.ssl_options.get("keyfile"),
+                "ssl_certfile": self.cfg.ssl_options.get("certfile"),
+                "ssl_keyfile_password": self.cfg.ssl_options.get("password"),
+                "ssl_version": self.cfg.ssl_options.get("ssl_version"),
+                "ssl_cert_reqs": self.cfg.ssl_options.get("cert_reqs"),
+                "ssl_ca_certs": self.cfg.ssl_options.get("ca_certs"),
+                "ssl_ciphers": self.cfg.ssl_options.get("ciphers"),
+            }
+            config_kwargs.update(ssl_kwargs)
+
+        if self.cfg.settings["backlog"].value:
+            config_kwargs["backlog"] = self.cfg.settings["backlog"].value
+
+        config_kwargs.update(self.CONFIG_KWARGS)
+
+        self.config = Config(**config_kwargs)
+
+    def init_process(self) -> None:
+        self.config.setup_event_loop()
+        super().init_process()
+
+    def init_signals(self) -> None:
+        # Reset signals so Gunicorn doesn't swallow subprocess return codes
+        # other signals are set up by Server.install_signal_handlers()
+        # See: https://github.com/encode/uvicorn/issues/894
+        for s in self.SIGNALS:
+            signal.signal(s, signal.SIG_DFL)
+
+        signal.signal(signal.SIGUSR1, self.handle_usr1)
+        # Don't let SIGUSR1 disturb active requests by interrupting system calls
+        signal.siginterrupt(signal.SIGUSR1, False)
+
+    def _install_sigquit_handler(self) -> None:
+        """Install a SIGQUIT handler on workers.
+
+        - https://github.com/encode/uvicorn/issues/1116
+        - https://github.com/benoitc/gunicorn/issues/2604
+        """
+
+        loop = asyncio.get_running_loop()
+        loop.add_signal_handler(signal.SIGQUIT, self.handle_exit, signal.SIGQUIT, None)
+
+    async def _serve(self) -> None:
+        self.config.app = self.wsgi
+        server = Server(config=self.config)
+        self._install_sigquit_handler()
+        await server.serve(sockets=self.sockets)
+        if not server.started:
+            sys.exit(Arbiter.WORKER_BOOT_ERROR)
+
+    def run(self) -> None:
+        return asyncio.run(self._serve())
+
+    async def callback_notify(self) -> None:
+        self.notify()
+
+
+class UvicornH11Worker(UvicornWorker):
+    CONFIG_KWARGS = {"loop": "asyncio", "http": "h11"}

inboard/start.py

@@ -53,7 +53,7 @@ def set_app_module(logger: logging.Logger = logging.getLogger()) -> str:
 def set_gunicorn_options(app_module: str) -> list[str]:
     """Set options for running the Gunicorn server."""
     gunicorn_conf_path = os.getenv("GUNICORN_CONF", "python:inboard.gunicorn_conf")
-    worker_class = os.getenv("WORKER_CLASS", "uvicorn.workers.UvicornWorker")
+    worker_class = os.getenv("WORKER_CLASS", "inboard.gunicorn_workers.UvicornWorker")
     if "python:" not in gunicorn_conf_path and not Path(gunicorn_conf_path).is_file():
         raise FileNotFoundError(f"Unable to find {gunicorn_conf_path}")
     return ["gunicorn", "-k", worker_class, "-c", gunicorn_conf_path, app_module]

pyproject.toml

@@ -48,10 +48,12 @@ starlette = [
 ]
 tests = [
   "coverage[toml]>=7,<8",
+  "coverage_enable_subprocess==1.0",
   "httpx>=0.23,<1",
   "pytest>=8.1.1,<9",
   "pytest-mock>=3,<4",
   "pytest-timeout>=2,<3",
+  "trustme>=1.2,<2",
 ]
 uvicorn-fast = [
   "httptools>=0.5.0",
@@ -75,6 +77,7 @@ show_missing = true
 
 [tool.coverage.run]
 command_line = "-m pytest"
+parallel = true
 source = ["inboard", "tests"]
 
 [tool.hatch.build.targets.sdist]
@@ -154,6 +157,9 @@ strict = true
 
 [tool.pytest.ini_options]
 addopts = "-q"
+markers = [
+  "subprocess: test requires a subprocess (deselect with '-m \"not subprocess\"')",
+]
 minversion = "6.0"
 testpaths = ["tests"]
 

tests/test_gunicorn_conf.py

@@ -94,7 +94,7 @@ def test_gunicorn_config(
             "-c",
             gunicorn_conf_path,
             "-k",
-            "uvicorn.workers.UvicornWorker",
+            "inboard.gunicorn_workers.UvicornWorker",
             app_module,
         ]
         subprocess.run(gunicorn_options)
@@ -138,7 +138,7 @@ def test_gunicorn_config_with_custom_options(
             "-c",
             gunicorn_conf_path,
             "-k",
-            "uvicorn.workers.UvicornWorker",
+            "inboard.gunicorn_workers.UvicornWorker",
             app_module,
         ]
         subprocess.run(gunicorn_options)