Add static typing #1065

MarcSkovMadsen · 2025-07-11T08:04:10Z

Closing #376

This PR adds static typing to param in a way that will not fundamentally change the current implementation and keep the api.

A more radical approach would be to leave descriptor pattern and engage with static typing stakeholders (including mypy) and get param supported in same way as dataclasses, attrs and pydantic.

Recommendation (from analysis.md)

Adopt PEP 681’s @dataclass_transform for Param’s metaclass or base class.
Provide and maintain high-quality stubs for Param descriptors.
Recommend using class-specific stubs for full static typing, including init signatures.
Avoid custom plugins unless absolutely necessary.
Do not recommend wrapper functions with @overload as a general solution for Param static typing.

Update - Plugin

Ahh. I tried experimenting with a plugin to generate the __init__ signature - but this was too complicated for me and AI. See https://docs.pydantic.dev/latest/integrations/mypy/#enabling-the-plugin and https://github.com/pydantic/pydantic/blob/main/pydantic/mypy.py.

Todo

Define problem (see problem.md)
Analyze and experiment (see experiment.md and experiment.py. Run with mypy static-typing/experiment.py --no-incremental.)
Propose solution (see analysis.md)
Collect feedback
Get more contributors onboard. Technically I might be able to succeed. But there will be so many opinions about this that more people need to get deeply involved. We will have to take decissions.
Implement, test and review.

philippjfr · 2025-07-11T08:55:16Z

Woah, this looks extremely promising.

maximlt · 2025-07-11T13:02:05Z

param/parameterized.pyi

+from typing import Any, Optional, overload
+
+class String:
+    default: Optional[str]


How do you handle allow_None=False?

I don't so far. This just explores the principles via real code. If allow_None can be changed dynamically it would be the right type annotation anyways.

Ok. I'd be interested in learning how we might need to adapt Param to make it work better with typing. For instance, yes I think you can modify allow_None dynamically? (you can do pretty much you want in Python dynamically anyway). In practice though, I don't think users change it dynamically. But allow_None is used quite a lot so it's important to support it, otherwise the hints will be wrong in a lot of times.

What I understand is that the problem is that we are using descriptors (classes) and they cannot be overloaded - thus Optional[str] is the best type annotation we can provide.

In experiment.py I also shown how you can make overloaded param_string functions instead - might solve some issues. But I don't think we would like to go down that route?

But allow_None is used quite a lot so it's important to support it, otherwise the hints will be wrong in a lot of times.

Though not ideal I think it's fine if the type is more permissive than the actual validation, so I'd expect we have to make all types Optional to account for allow_None.

As far as I can see there is no way around it unless we want to make breaking changes to param and leave the "descriptor" pattern behind.

We could split String into String and OptionalString parameters to follow Pythons type system.

If made more permissive than in reality I expect a type checker to complain at some point or another, see the code below. How would we let people know that the library provides inexact type hints? I haven't worked so much with type hints so far but know that when type checking is enabled in a code base dealing with wrong types gets quickly pretty frustrating.

def algo(input: str): pass class P(param.Parameterized): value = param.String(allow_None=False) def process(self): # Wouldn't a type checker be complaining there # if the type of `self.value` is `str | None`? Given # that `algo` only accepts `str`, I'd expect it to complain. return algo(self.value)

We could split String into String and OptionalString parameters to follow Pythons type system.

That's one approach. You can integrate it in your proposal if you want.

One solution to make a type checker happy here is to add assert isinstance(self.value, str) before calling algo. Not sure that's a very satisfying solution but that is the sort of thing I'd expect us to have to document if we release a version of Param that doesn't always provide correct type hints.

MarcSkovMadsen · 2025-07-11T13:27:06Z

static-typing/experiment.py

@@ -0,0 +1,66 @@
+from typing import Any


This is where the value is :-) Showing how dataclass_transform can be applied etc.

maximlt · 2025-07-12T09:23:34Z

@MarcSkovMadsen I'm not really sure what I'm supposed to review here and what the actual proposal is.

MarcSkovMadsen · 2025-07-12T14:54:17Z

Review the recommendation/ proposal

Would you accept a PR like that? If yes then I will start implementing.

The .md documents provides the details and .py experiment file can enable you to try this hands on.

maximlt

Ok so I spent some time reviewing the Markdown files and exploring experiment.py. Some points I wanted to raise in addition to the comments:

I found some inaccuracies in the problem statement in problem.md
I'm not sure what's the difference between analysis.md and experiment.md, they contain somewhat similar content
The dataclass_transform is used but I haven't found it to work? From the documents, I don't see a description of why it doesn't work, and what it would take for it to work.
Only param.String and param.Number are used as examples. I would have expected an analysis of this kind to experiment with a wider variety of Parameters (selectors, mappings, sequences, callable, class selector, etc.), to find potential limitations and suggest approaches how to deal with them.
Have you looked into how SQLAlchemy implemented their typing support in v2? Some of their requirements look similar to ours https://docs.sqlalchemy.org/en/20/changelog/whatsnew_20.html#orm-declarative-models.

My intuition with making Param work better with type hints is that:

it would need quite some work
to make it a good experience some changes in Param would be required (and personally I would be in favor of whatever changes needed, even if it means losing some of the runtime capabilities of Param).

This analysis and the changes made both look too shallow. I also don't see what changes we would need to make to Param so it works well with type hints. Overall, I struggle to see from this analysis what are our options? Surely, there are many ways to improve the current situation.

Maybe it is because the goal is to get some more type hints in an IDE? If so, is that a goal we can reach without bringing some frustrations because the type hints we'd provide are not correct?

Would you accept a PR like that? If yes then I will start implementing.

I don't think I would accept a PR like that as this analysis didn't give me the confidence this is a valid approach. Which shouldn't prevent you from exploring further that approach or another one! Also, I see myself as a release manager of Param; @jbednar, @philippjfr and @jlstevens are its official maintainers and can have a different opinion than mine; Philipp seems to be pretty interested in how this work started.

maximlt · 2025-07-12T17:04:02Z

static-typing/problem.md

@@ -0,0 +1,52 @@
+# Param - Static Typing - Problem
+
+[HoloViz Param](https://param.holoviz.org/) is a powerful library for defining parameters on `param.Parameterized` classes:


This is a strange definition of Param?

But does it matter?

You wrote on Discord that you used AI to work on this PR. I wouldn't have pointed this out otherwise but maybe in that case it matters.

maximlt · 2025-07-12T17:04:33Z

static-typing/problem.md

+**Param does not play well with Python's static typing ecosystem.**
+
+Modern Python development expects type annotations to:
+- Enable code completion and inline documentation in editors (like VS Code, PyCharm)


Type annotations enable inline documentation?

Yes. Tooltips with init signatures and in general more information.

maximlt · 2025-07-12T17:07:14Z

static-typing/problem.md

+
+Modern Python development expects type annotations to:
+- Enable code completion and inline documentation in editors (like VS Code, PyCharm)
+- Allow static analysis and error checking with tools like mypy and pylint


Do you mean pylance? I haven't used pylint in a while but don't really expect it to use type annotations much, at least not as much as mypy/pylance/ty and others.

mypy and pylance use it more. But for example pylint/ ruff have rules for working with type annotations - for example enforcing them.

maximlt · 2025-07-12T17:11:39Z

static-typing/problem.md

+But with Param, these benefits are lost:
+
+- **No Automatic Type Inference:** When you declare a parameter (e.g. `my_parameter = param.String(...)`), neither the class nor its instances have a type annotation for `my_parameter`. Editors and type checkers cannot know its type.
+- **Poor Editor Experience:** You don't get tab completion, hover help, or docstrings for parameter attributes on instances. For example, `instance.my_parameter` is invisible to your IDE.


I get tab completion in VS Code on the class or on an already created instance.

I don't get tab completion when creating an instance, i.e. P( doesn't show test.

maximlt · 2025-07-12T17:14:13Z

static-typing/problem.md

+    data: pd.DataFrame = param.DataFrame(precedence=0.1)
+```
+
+Even with a type annotation, editors and type checkers do not recognize `data` as a `pd.DataFrame` on the instance. This leads to poor auto-completion and false warnings.


That's not what I see.

Its when data = param.DataFrame(precedence=0.1). Then its not recognized.

Then this example and the text need to be updated as it clearly states "even with a type annotation, ...":

Even with a type annotation, editors and type checkers do not recognize data as a pd.DataFrame on the instance. This leads to poor auto-completion and false warnings.

maximlt · 2025-07-12T18:01:50Z

static-typing/experiment.py

+
+    obj = MyClass(my_parameter="hello", my_number=3.14)
+
+    reveal_type(obj.my_parameter)  # Should be "builtins.str"


VS Code shows str | None.

Yes. It cannot infer more. But ideally it should be str. Cannot be achieved without

type annotation or

overloaded function or

plugin

Same comment as #1065 (comment)

maximlt · 2025-07-12T18:13:09Z

param/parameterized.pyi

+from typing import Any, Optional, overload
+
+class String:
+    default: Optional[str]


If made more permissive than in reality I expect a type checker to complain at some point or another, see the code below. How would we let people know that the library provides inexact type hints? I haven't worked so much with type hints so far but know that when type checking is enabled in a code base dealing with wrong types gets quickly pretty frustrating.

def algo(input: str): pass class P(param.Parameterized): value = param.String(allow_None=False) def process(self): # Wouldn't a type checker be complaining there # if the type of `self.value` is `str | None`? Given # that `algo` only accepts `str`, I'd expect it to complain. return algo(self.value)

We could split String into String and OptionalString parameters to follow Pythons type system.

That's one approach. You can integrate it in your proposal if you want.

maximlt · 2025-07-12T18:13:39Z

param/parameters.pyi

+    default: float
+    def __get__(self, instance: Any, owner: Any) -> float: ...
+    def __set__(self, instance: Any, value: float) -> None: ...
+    def __init__(self, default: Optional[float] = ..., doc: Optional[str] = ...) -> None: ...


Is the __init__ required? I got more info before in VS Code:

Before:

Now:

Regarding first question. Yes static type checkers would complain that you would have to check for None. But now you only have two possibilities str | None. To some that is a big help. The ones that do no appreciate types can just let them be.

I agree that having type annotations would be very useful. But I also think that if we have a library that provides type annotations that are not always correct, then we should provide a solution for people to deal with that. It might be it would just be a documentation thing, stating that these annotations should only be used in an IDE and in a special mode. I don't really know to be honest but I don't find that answer to be complete and clear enough:

The ones that do no appreciate types can just let them be.

maximlt · 2025-07-12T18:17:15Z

param/parameters.pyi

+from typing import Any, Optional
+
+class Number:
+    default: float


Why is default required here?

Good question. I thought it defined the return type of some_number = param.Number(...). But probably its just a type annotation of the default attribute should be float | None to be correct.

maximlt · 2025-07-12T18:43:44Z

static-typing/analysis.md

+### Param Descriptors and Typing
+- Param descriptors (e.g., param.String, param.Number) can provide good attribute type inference in mypy if their stubs define correct __get__ return types (e.g., str, Optional[str], float).
+- Type annotations on descriptor assignments cause mypy errors unless suppressed with # type: ignore[assignment].
+- Wrapper functions using @overload (e.g., param_string) can help mypy infer the type of the value returned by the function, but the attribute is still a descriptor, so __init__ signatures are not synthesized.


Why is the __init__ signature not correct when the attribute is a selector?

Sorry. Don't understand the question?

Sorry I used the wrong word and meant descriptor instead of selector.

Why is the __init__ signature not correct when the attribute is a descriptor?

philippjfr · 2025-07-15T09:43:41Z

Thanks again for starting this @MarcSkovMadsen. After reviewing what's here I'm wavering a little bit. Effectively based on my reading, and until Python adopts type-aware descriptors we don't have many good options. With the list effectively being:

This approach which is fairly imprecise in the types, i.e. yes we can correctly type param.Integer but can't handle allow_None correctly. It's still an improvement but it's quite limited.
Combine approach 1. with manual type annotations, i.e. the descriptor is typed loosely but the user is asked to be more specific:

from typing import Optional

class Integer:
    def __get__(self, instance: object, owner: type) -> Optional[int]:
        ...
    def __set__(self, instance: object, value: Optional[int]) -> None:
        ...

class Model(param.Parameterized):
    a: int = Integer()
    b: Optional[int] = Integer(allow_None=True)

Using generics as in @hoxbro's PR type: Infer parameters types for __init__(default...), __get__ and __set__ #985, would move the definition into the parameter declaration:

class Model(param.Parameterized):
    a = Integer[int]()
    b = Integer[Optional[int]](allow_None=True)

with some extra effort in defining TypeVar (which Simon already started playing with) with bound types you only have to define more restrictive types, i.e. it would simplify to:

class Model(param.Parameterized):
    a = Integer[int]()
    b = Integer(allow_None=True)

Create distinct Descriptor Types, i.e. have an OptionalInteger and an Integer parameter type. I think this is out by default because it fundamentally changes param's design.

On balance I think the approach using generics is most promising, even if it is somewhat verbose. So my feeling is that @hoxbro's PR https://github.com/holoviz/param/pull/985/files is the correct place to start from.

maximlt · 2025-07-15T12:36:16Z

Traitlets seems to have figured out something interesting, you can see that with allow_none=True the type of y is int | None. Certainly worth looking into their typing PRs https://github.com/ipython/traitlets/pulls?q=is%3Apr+typing+is%3Aclosed and implementation. I'm focused on finishing hvPlot's doc overhaul this summer and would very much like to dive into Param's typing story in September.

# t.py
from traitlets import HasTraits, Int

class P(HasTraits):
    x = Int(default=2)
    y = Int(allow_none=True)

p = P(x=3, y=None)

reveal_type(p.x)
reveal_type(p.y)

❯ mypy t.py
t.py:9: note: Revealed type is "builtins.int"
t.py:10: note: Revealed type is "Union[builtins.int, None]"

static typing analysis

d24ae0a

MarcSkovMadsen requested a review from philippjfr July 11, 2025 08:04

MarcSkovMadsen marked this pull request as draft July 11, 2025 08:04

maximlt reviewed Jul 11, 2025

View reviewed changes

MarcSkovMadsen commented Jul 11, 2025

View reviewed changes

MarcSkovMadsen requested a review from maximlt July 11, 2025 17:36

maximlt reviewed Jul 12, 2025

View reviewed changes

philippjfr mentioned this pull request Jul 15, 2025

Implement descriptor factory typing system #1066

Draft

		@@ -0,0 +1,52 @@
		# Param - Static Typing - Problem

		[HoloViz Param](https://param.holoviz.org/) is a powerful library for defining parameters on `param.Parameterized` classes:


		obj = MyClass(my_parameter="hello", my_number=3.14)

		reveal_type(obj.my_parameter) # Should be "builtins.str"

Uh oh!

Add static typing #1065

Are you sure you want to change the base?

Add static typing #1065

Uh oh!

Conversation

MarcSkovMadsen commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Recommendation (from analysis.md)

Update - Plugin

Todo

Uh oh!

philippjfr commented Jul 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcSkovMadsen Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcSkovMadsen Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcSkovMadsen Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maximlt commented Jul 12, 2025

Uh oh!

MarcSkovMadsen commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maximlt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcSkovMadsen Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcSkovMadsen commented Jul 11, 2025 •

edited

Loading

MarcSkovMadsen Jul 11, 2025 •

edited

Loading

MarcSkovMadsen Jul 11, 2025 •

edited

Loading

MarcSkovMadsen Jul 12, 2025 •

edited

Loading

MarcSkovMadsen commented Jul 12, 2025 •

edited

Loading

MarcSkovMadsen Jul 13, 2025 •

edited

Loading

philippjfr commented Jul 15, 2025 •

edited

Loading