A new home for pyarrow-stubs?

[yangdanny97]

The maintainer of pyarrow-stubs (a set of Python type stubs for arrow) has expressed interest in donating the stubs since they no longer have time to maintain them: zen-xu/pyarrow-stubs#186

I'm opening a discussion here to see if there's any interest in merging the stubs into the main arrow project. They may remain as stubs, or if there is interest, they can also be merged into the Python bindings as inline types.

I'm happy to help set up the appropriate CI to support future maintenance and make sure the types are accurate/up-to-date going forward.

I see an existing issue for this topic (#30426), but not a lot of active discussion, so I'm hoping to hear what the maintainers think.

[AlenkaF]

Thank you for opening up this discussion!

There is a general consensus on the need for type annotations in PyArrow. In my opinion, the main challenge to progress is the lack of agreement on the approach to take.

To add some context, here are a few key comments from the ongoing discussion in #32609:

• The latest summary from @jorisvandenbossche: [Python] Type checking support #32609 (comment)
• Insights from @paleolimbot regarding the nanoarrow experience: [Python] Type checking support #32609 (comment)
• Advice from the pyarrow-stubs author: [Python] Type checking support #32609 (comment)

[raulcd]

Thanks for opening the discussion @yangdanny97 and thanks for the links @AlenkaF , really useful to get the history around it!
I've closed:

• [Python] Type stubs #30426

As a duplicate of:

• [Python] Type checking support #32609

And added some links to try and keep things organized on a single place.

[yangdanny97]

Thanks for the context!

If there isn't a strong consensus on inline types, I'd probably recommend the types be kept as stubs (with some tools like stubtest to make sure they're complete and in-sync). The reason is that if you change your mind, it's easier to move from stubs to inline types, whereas removing inline types will cause a lot of churn and merge conflicts.

Hand-written stubs will be much better quality than completely auto-generated stubs, and to the best of my knowledge all existing stub-generation tools do not produce stubs that are good enough to be used out-of-the-box without manual tweaks. Almost all the stubs in typeshed are maintained by hand, with a few exceptions like protobuf which has its own stub generation script.

I can see generated stubs being used as a temporary supplement to fill any gaps that pyarrow-stubs has (according to the author it's relatively complete), but after being generated the stubs will need to be fixed and maintained by hand.

[AlenkaF]

Just wanted to keep this discussion active—following up with my thoughts on a possible path forward. I believe a good approach would be to coordinate efforts with the pyarrow-stubs project and build on the work that has already been done there. That said, I’m a bit uncertain, since @paleolimbot and @jorisvandenbossche have both expressed support for a simpler, auto-generated solution—and they’re usually right. 😊

That said, PyArrow developers are currently managing several priorities, so this likely isn’t something we can kick off immediately.

Still, it would be valuable to start planning how we want to approach this. Based on the ongoing discussions, I’d like to highlight a couple of key points:

• As Joris suggested, we could start with minimal, lightweight stubs—something simpler than what currently exists in pyarrow-stubs—and gradually extend them as needed (comment).

  • If transferring ownership of the pyarrow-stubs project would still be relevant, bringing it under the Apache Arrow umbrella would involve the official Apache donation process, starting with a vote on the mailing list.

• It might also be worth creating a proof of concept for auto-generated stubs to compare approaches.

Lastly, I was recently pointed to a new type checker tool: https://github.com/astral-sh/ty. Does anyone have experience with it?

[yangdanny97]

Do you have an estimate for how much development resources/funding an upstreaming effort like this would require?

[AlenkaF]

Not really, no.

@paleolimbot — since you’ve worked on type checking in nanoarrow, could you give us a hand here?

We're wondering how much development effort might be needed for something similar in PyArrow, building on the existing pyarrow-stubs work — including both implementation and ongoing maintenance.

[paleolimbot]

My experience in nanoarrow is just with stub generation, mostly for the purposes of halfway decent autocomplete (we never quite got to setting up a type checker). That's still my main beef with writing pyarrow code (IDEs have no idea what most of the modules contain), although full on type checked everything would be great!

[rustyconover]

I'd like to help out with some pyarrow stub writing. I'd really like to be able to type check my PyArrow code with mypy.

[AlenkaF]

Just noting that I’d also like to be involved in this work.
I’d naively estimate that, with 2-3 people working part-time, this could be completed within one release cycle—or two at most, if additional review iterations are needed.

[rok]

As a sprint at EuroPython @paddyroddy and I created a draft for including @zen-xu's pyarrow-stubs with pyarrow (as proposal nr 2 in #32609 (comment)). We used ty and included a CI job that checks for annotation issues (following numpy's example).

If we decide to proceed with this we would of course need @zen-xu donating pyarrow-stubs first.

Things that come to mind after working on this:

• It would be nice to inline as much annotations as possible for maintainability.
• PR currently copies pyarrow-stubs verbatim, but we would probably want to remove docs from pyi files if possible.

[zen-xu]

I’m happy to see that pyarrow-stubs can be merged

[rok]

Great to hear @zen-xu ! We'll probably need some time for discussion and it's great to know this at the start.

[dangotbanned]

As a recent contributor and user of pyarrow-stubs (narwhals-dev/narwhals#2124), this looks exciting!

We used ty and included a CI job that checks for annotation issues

I feel the need to echo (#45919 (reply in thread))

both are designed to be much faster for large codebases, but neither is production-ready.

and again in (https://github.com/astral-sh/ty):

An extremely fast Python type checker and language server, written in Rust.

Warning

ty is in preview and is not ready for production use.

We're working hard to make ty stable and feature-complete, but until then, expect to encounter bugs, missing features, and fatal errors.

I'm looking forward to using ty when it is ready, but there are major gaps in what is supported vs either pyright or mypy.

There are two milestones that give a helpful overview of support:

• ty Beta
• ty GA

Some select issues that seem relevant to pyarrow-stubs that I think could be a problem:

• Check for overlapping overloads astral-sh/ty#103
• Support narrowing using TypeGuard and TypeIs astral-sh/ty#117
• support Protocols astral-sh/ty#161
• support typing.TypeAlias astral-sh/ty#544
• improve typevar solving astral-sh/ty#623
• Overload argument type expansion leads to combinatoric explosion astral-sh/ty#868
• Consider checking for subtyping for ambiguous overload matching astral-sh/ty#665

[yangdanny97]

Given that arrow is a popular library, it's best to use multiple typecheckers in CI, to make sure the types work for users regardless of the tool they're using.

So that means for now starting with Mypy/pyright, and then adding ty/pyrefly whenever they're ready (Pyrefly and ty are both running in CI for pandas-stubs, so it may already be possible today).

Another good tool to use is stubtest, which will check the stubs against values at runtime to make sure that the stubs are complete & accurate. It has an allowlist functionality so that you can add it to CI without being fully passing, and then clean up the allowlist going forward.

[paddyroddy]

Does feel like we should use mypy_primer as well, which NumPy use in their type checking CI. Note, NumPy previously had a separate stubs library and moved them in.