Skip to content

try claude - add a basic config and play around with some quality of life changes i was offputting #584

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
5a292b3
Add CLAUDE.md with development guidance
RonnyPfannschmidt Jun 19, 2025
d203f70
Add HookimplConfiguration class to replace HookimplOpts dict
RonnyPfannschmidt Jun 20, 2025
82aee10
Add get_hookconfig method to HookimplMarker for type-safe config extr…
RonnyPfannschmidt Jun 20, 2025
a85f465
Remove deprecated HookimplOpts usage and minimize internal dependencies
RonnyPfannschmidt Jun 20, 2025
d9c4e2d
Replace internal hook implementation with HookimplConfiguration
RonnyPfannschmidt Jun 20, 2025
6a505ec
Add ProjectSpec unified project management API
RonnyPfannschmidt Jun 21, 2025
a6c8a9c
Remove legacy HookimplOpts normalization and from_opts method
RonnyPfannschmidt Jun 21, 2025
3d717ed
Replace internal hook implementation with HookspecConfiguration
RonnyPfannschmidt Jun 21, 2025
ac8e4dd
Centralize hook configuration extraction in ProjectSpec
RonnyPfannschmidt Jun 21, 2025
9533c42
Fix benchmark.py to use ProjectSpec for hook config extraction
RonnyPfannschmidt Jun 21, 2025
ccd761d
Split historic hook functionality into separate HistoricHookCaller class
RonnyPfannschmidt Jun 21, 2025
f46834b
Refactor HookCaller class tree with protocol-based architecture
RonnyPfannschmidt Jun 21, 2025
b69db1f
Refactor _hooks.py into organized sibling modules by category
RonnyPfannschmidt Jun 21, 2025
9096292
Move Final annotations to class level and add type annotations to __s…
RonnyPfannschmidt Jun 21, 2025
3721d8d
Split hook lists and add insertion utility in NormalHookCaller
RonnyPfannschmidt Jun 21, 2025
6c75844
Separate normal and wrapper hook implementations with distinct types
RonnyPfannschmidt Jun 22, 2025
ad71541
Refactor argument verification logic from _multicall to HookImpl
RonnyPfannschmidt Jun 22, 2025
fc3270e
Replace complex _multicall wrapper logic with streamlined completion …
RonnyPfannschmidt Jun 22, 2025
e754127
Refactor _hooks.py into organized sibling modules by category
RonnyPfannschmidt Jun 21, 2025
0c956fd
Move Final annotations to class level and add type annotations to __s…
RonnyPfannschmidt Jun 21, 2025
44f68a0
Split hook lists and add insertion utility in NormalHookCaller
RonnyPfannschmidt Jun 21, 2025
cbe879e
Separate normal and wrapper hook implementations with distinct types
RonnyPfannschmidt Jun 22, 2025
59b5453
Replace complex _multicall wrapper logic with streamlined completion …
RonnyPfannschmidt Jun 22, 2025
a4772bb
add utilities to deal with async in pluggy
RonnyPfannschmidt Jun 23, 2025
cb748f6
FIXUP: rebase artifacts
RonnyPfannschmidt Jun 25, 2025
6570d43
fixup: older python support
RonnyPfannschmidt Jun 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"permissions": {
"allow": [
"Bash(uv run pytest:*)",
"Bash(git add:*)",
"Bash(uv run mypy:*)",
"Bash(uv run pre-commit:*)"

],
"deny": []
}
}
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -67,3 +67,6 @@ pip-wheel-metadata/

# pytest-benchmark
.benchmarks/

# Claude Code local settings
.claude/settings.local.json
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,4 @@ repos:
- id: mypy
files: ^(src/|testing/)
args: []
additional_dependencies: [pytest]
additional_dependencies: [pytest, types-greenlet]
50 changes: 50 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Pluggy is a minimalist production-ready plugin system that serves as the core framework for pytest, datasette and devpi.
It provides hook specification and implementation mechanisms through a plugin manager system.

## Development Commands

### Testing
- `uv run pytest` - Run all tests, prefer runnign all tests to quickly get feedback
- `uv run pytest testing/benchmark.py` runs the benchmark tests

### Code Quality
- `uv run pre-commit run -a` - Run all pre-commit hooks - gives linting and typing errors + corrects files
- reread files that get fixed by pre-commit


## Development process

- always read `src/pluggy/*.py` to get a full picture
- consider backward compatibility
- always run all tests
- always run pre-commit before try to commit
- prefer running full pre-commit over ruff/mypy alone



## Core Architecture

### Main Components

- **PluginManager** (`src/pluggy/_manager.py`): Central registry that manages plugins and coordinates hook calls
- **HookCaller** (`src/pluggy/_hooks.py`): Executes hook implementations with proper argument binding
- **HookImpl/HookSpec** (`src/pluggy/_hooks.py`): Represent hook implementations and specifications
- **Result** (`src/pluggy/_result.py`): Handles hook call results and exception propagation
- **Multicall** (`src/pluggy/_callers.py`): Core execution engine for calling multiple hook implementations

### Package Structure
- `src/pluggy/` - Main package source
- `testing/` - Test suite using pytest
- `docs/` - Sphinx documentation and examples
- `changelog/` - Towncrier fragments for changelog generation

## Configuration Files
- `pyproject.toml` - Project metadata, build system, tool configuration (ruff, mypy, setuptools-scm)
- `tox.ini` - Multi-environment testing configuration
- `.pre-commit-config.yaml` - Code quality automation (ruff, mypy, flake8, etc.)
15 changes: 13 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,23 @@ readme = {file = "README.rst", content-type = "text/x-rst"}
requires-python = ">=3.9"

dynamic = ["version"]

[project.optional-dependencies]
dev = ["pre-commit", "tox"]
testing = ["pytest", "pytest-benchmark", "coverage"]
async = ["greenlet"]

[dependency-groups]
dev = ["pre-commit", "tox", "mypy", "ruff"]
testing = ["pytest", "pytest-benchmark", "coverage", "greenlet", "types-greenlet"]



[tool.setuptools]
packages = ["pluggy"]
package-dir = {""="src"}
package-data = {"pluggy" = ["py.typed"]}



[tool.ruff.lint]
extend-select = [
"I", # isort
Expand All @@ -67,6 +74,9 @@ lines-after-imports = 2

[tool.setuptools_scm]

[tool.uv]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update docs/index.rst, Development section, about using uv.

default-groups = ["dev", "testing"]

[tool.towncrier]
package = "pluggy"
package_dir = "src/pluggy"
Expand Down Expand Up @@ -107,6 +117,7 @@ template = "changelog/_template.rst"

[tool.mypy]
mypy_path = "src"
python="3.9"
check_untyped_defs = true
# Hopefully we can set this someday!
# disallow_any_expr = true
Expand Down
25 changes: 18 additions & 7 deletions src/pluggy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,32 +3,43 @@
"PluginManager",
"PluginValidationError",
"HookCaller",
"HistoricHookCaller",
"HookCallError",
"HookspecOpts",
"HookimplOpts",
"HookspecConfiguration",
"HookimplConfiguration",
"HookImpl",
"HookRelay",
"HookspecMarker",
"HookimplMarker",
"ProjectSpec",
"Result",
"PluggyWarning",
"PluggyTeardownRaisedWarning",
]
from ._hooks import HookCaller
from ._hooks import HookImpl
from ._hooks import HookimplMarker
from ._hooks import HookimplOpts
from ._hooks import HookRelay
from ._hooks import HookspecMarker
from ._hooks import HookspecOpts
from ._hook_callers import HistoricHookCaller
from ._hook_callers import HookCaller
from ._hook_callers import HookImpl
from ._hook_callers import HookRelay
from ._hook_config import HookimplConfiguration
from ._hook_config import HookimplOpts
from ._hook_config import HookspecConfiguration
from ._hook_config import HookspecOpts
from ._hook_markers import HookimplMarker
from ._hook_markers import HookspecMarker
from ._manager import PluginManager
from ._manager import PluginValidationError
from ._project import ProjectSpec
from ._result import HookCallError
from ._result import Result
from ._warnings import PluggyTeardownRaisedWarning
from ._warnings import PluggyWarning


__version__: str


def __getattr__(name: str) -> str:
if name == "__version__":
from importlib.metadata import version
Expand Down
198 changes: 198 additions & 0 deletions src/pluggy/_async.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
"""
Async support for pluggy using greenlets.

This module provides async functionality for pluggy, allowing hook implementations
to return awaitable objects that are automatically awaited when running in an
async context.
"""

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonw i'd appreciate early feedback on this wrt utility for datasette

its not the deep integration like I wanted/hoped to implement at first

however its a directly working integration that doesn’t break on color mixing - but it does have the caveat of breaking never officieally supported threading patterns some people use

i currently intend to clean up history and then have this land and release

from __future__ import annotations

from collections.abc import AsyncGenerator
from collections.abc import Awaitable
from collections.abc import Callable
from collections.abc import Generator
from typing import Any
from typing import TYPE_CHECKING
from typing import TypeVar


_T = TypeVar("_T")
_Y = TypeVar("_Y")
_S = TypeVar("_S")
_R = TypeVar("_R")

if TYPE_CHECKING:
import greenlet


def make_greenlet(func: Callable[..., Any]) -> greenlet.greenlet:
"""indirection to defer import"""
import greenlet

return greenlet.greenlet(func)


class Submitter:
# practice we expect te root greenlet to be the key submitter
_active_submitter: greenlet.greenlet | None

def __init__(self) -> None:
self._active_submitter = None

def __repr__(self) -> str:
return f"<Submitter active={self._active_submitter is not None}>"

def maybe_submit(self, coro: Awaitable[_T]) -> _T | Awaitable[_T]:
"""await an awaitable if active, else return it

this enables backward compatibility for datasette
and https://simonwillison.net/2020/Sep/2/await-me-maybe/
"""
active = self._active_submitter
if active is not None:
# We're in a greenlet context, switch with the awaitable
# The parent will await it and switch back with the result
res: _T = active.switch(coro)
return res
else:
return coro

def require_await(self, coro: Awaitable[_T]) -> _T:
"""await an awaitable, raising an error if not in async context

this is for cases where async context is required
"""
active = self._active_submitter
if active is not None:
# Switch to the active submitter greenlet with the awaitable
# The active submitter will await it and switch back with the result
res: _T = active.switch(coro)
return res
else:
raise RuntimeError("require_await called outside of async context")

async def run(self, sync_func: Callable[[], _T]) -> _T:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if this question is obvious—I don’t have much experience with greenlets.

Isn’t the purpose of using greenlet to have multiple worker greenlets and switch between them as they execute? In other words, if one worker is busy and can’t proceed, we can switch to another worker in the meantime.

If that’s correct, what does greenlet actually bring to the table here? As I understand it, we only have two greenlets: the main one and a worker. The main greenlet runs a while loop, switching to the worker greenlet to obtain a new awaitable.


From my (admittedly limited) understanding, to support async hooks in PluggyManager, it seems we only need to explicitly handle the awaitable objects returned by the hooks. For example:

    async def run_async2(self, func: Callable[[], _T]) -> _T:
        results = func()
        for index, result in enumerate(results):
            if inspect.isawaitable(result):
                result = await result
                results[index] = result
        return results

If we replace pm.run_async with pm.run_async2 in test_async.py, all the tests still pass. What am I missing here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greenlet allows to switch stacks

Which allows to switch from 10 sync calls in back to a async function that can await a awaitable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The run function will pass into asyncio trio or anyio. Then it will switch back into the sync function

And now any sync call below can send a sync call to the outer async function

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @RonnyPfannschmidt, but it’s still not clear to me how greenlet fits into this.

In other words, what does run_async do that run_async2 does not? From my understanding, they seem functionally equivalent—is that correct?

If not, we should create a test case that only works with run_async but fails with run_async2. That would help clarify why greenlet is necessary here.

"""Run a synchronous function with async support."""
try:
import greenlet
except ImportError:
raise RuntimeError("greenlet is required for async support")

if self._active_submitter is not None:
raise RuntimeError("Submitter is already active")

# Set the current greenlet as the main async context
main_greenlet = greenlet.getcurrent()
result: _T | None = None
exception: BaseException | None = None

def greenlet_func() -> None:
nonlocal result, exception
try:
result = sync_func()
except BaseException as e:
exception = e

# Create the worker greenlet
worker_greenlet = greenlet.greenlet(greenlet_func)
# Set the active submitter to the main greenlet so maybe_submit can switch back
self._active_submitter = main_greenlet

try:
# Switch to the worker greenlet and handle any awaitables it passes back
awaitable = worker_greenlet.switch()
while awaitable is not None:
# Await the awaitable and send the result back to the greenlet
awaited_result = await awaitable
awaitable = worker_greenlet.switch(awaited_result)
except Exception as e:
# If something goes wrong, try to send the exception to the greenlet
try:
worker_greenlet.throw(e)
except BaseException as inner_e:
exception = inner_e
finally:
self._active_submitter = None

if exception is not None:
raise exception
if result is None:
raise RuntimeError("Function completed without setting result")
return result


def async_generator_to_sync(
async_gen: AsyncGenerator[_Y, _S], submitter: Submitter
) -> Generator[_Y, _S, None]:
"""Convert an async generator to a sync generator using a Submitter.

This helper allows wrapper implementations to use async generators while
maintaining compatibility with the sync generator interface expected by
the hook system.

Args:
async_gen: The async generator to convert
submitter: The Submitter to use for awaiting async operations

Yields:
Values from the async generator

Returns:
None (async generators don't return values)

Example:
async def my_async_wrapper():
yield # Setup phase
result = await some_async_operation()

# In a wrapper hook implementation:
def my_wrapper_hook():
async_gen = my_async_wrapper()
gen = async_generator_to_sync(async_gen, submitter)
try:
while True:
value = next(gen)
yield value
except StopIteration:
return
"""
try:
# Start the async generator
value = submitter.require_await(async_gen.__anext__())

while True:
try:
# Yield the value and get the sent value
sent_value = yield value

# Send the value to the async generator and get the next value
try:
value = submitter.require_await(async_gen.asend(sent_value))
except StopAsyncIteration:
# Async generator completed
return

except GeneratorExit:
# Generator is being closed, close the async generator
try:
submitter.require_await(async_gen.aclose())
except StopAsyncIteration:
pass
raise

except BaseException as exc:
# Exception was thrown into the generator,
# throw it into the async generator
try:
value = submitter.require_await(async_gen.athrow(exc))
except StopAsyncIteration:
# Async generator completed
return
except StopIteration as sync_stop_exc:
# Re-raise StopIteration as it was passed through
raise sync_stop_exc

except StopAsyncIteration:
# Async generator completed normally
return
Loading
Loading