array-api-extra requirements brainstorming

[lucascolley]

The idea of a package named something like array-api-extra (xpx?) has been floated in various places. In a sentence, the purpose would be to house array-agnostic implementations of functions which are likely to be useful in a variety of domains (in this sense, the functions belong on or close to the array library level), but are not a part of the standard.

Proposed functions so far include kron, atleast_nd, cov, moveaxis_to_end, vector_norm, ravel, just from SciPy alone. Some of them can be relatively trivial to implement for end-users, but there is still value in having drop-in replacements for existing non-standard functions, especially when converting large codebases. Forming a collection of functions that are needed as consumer libraries convert to array-agnostic implementations should help inform potential additions to the standard, too.

These functions would take an xp kwarg and perform all computation with that namespace, assuming that it is compatible with the standard.

---

In scope

• Functions written purely in terms of the standard (and Python)
• Functions that may be generally useful across domains
• Functions from standard extensions (if they may be implemented in terms of functions from the base standard)
• Adding functionality on top of standard functions e.g. adding a kwarg to mean

Out of scope

• Functions which consume specific array libraries or helpers for compatibility with existing libraries - they belong in array-api-compat
• Anything that would introduce any dependencies other than array-api-compat

General package requirements

• Releases on PyPI & conda-forge like array-api-compat
• Tested against array-api-strict
• Documented API

Design choices

• Version(s) of the standard
  • I think it makes sense to just consume the most recent version of the standard
• Backwards-compatibility guarantees
  • I think that we could start off without making any guarantees and say that if you absolutely need stability, just copy the implementations.
• API structure
  • I think we could start off with just one flat namespace.

cc @asmeurer @rgommers @izaid, x-ref gh-837, scipy/scipy#21054

[izaid]

I think this is great! And actually xpx kind of nice to use...

[asmeurer]

These functions would take an xp kwarg and perform all computation with that namespace, assuming that it is compatible with the standard.

Why do they need to take an xp argument? Unless it's an array creation function, shouldn't any function just be able to use array_namespace() on its arguments?

[lucascolley]

In SciPy we currently do both - try to pass xp around to avoid redundant array_namespace calls, but call array_namespace if it isn't provided.

If we were to just always call array_namespace, then we might have to add some extra xp.asarrays into SciPy to turn array-likes into NumPy arrays. But I think the main concern is just avoiding redundant calls to array_namespace.

[asmeurer]

I think that we could start off without making any guarantees and say that if you absolutely need stability, just copy the implementations.

Indeed. We should also make it vendorable. And I agree we should make each function isolated so people can easily just copy the functions directly.

[asmeurer]

My biggest concern with the package is making sure we have good quality assurance, so that we know the function implementations are correct. That means good tests (we can test against known implementations like NumPy), and also ideally, we can get enough people who are familiar with the various array functions to help review, or even maintain, the package.

I would also add that specifically in-scope is implementations or branches in implementations that are specific to a given array library or set of array libraries. For instance, if some libraries already have some fast versions of a given function, we should just dispatch to those, or if some function can be implemented faster for libraries that have access to stride tricks, we should use those when we can. This is of particular importance when it has an impact on performance.

[lucascolley]

I would also add that specifically in-scope is implementations or branches in implementations that are specific to a given array library or set of array libraries. For instance, if some libraries already have some fast versions of a given function, we should just dispatch to those, or if some function can be implemented faster for libraries that have access to stride tricks, we should use those when we can.

Hmm, I was actually thinking of declaring that out of scope, at least to start with. I think that really broadens the scope a lot - not necessarily a bad thing, but I think it is more than we need to commit to right now. This delegation to existing libraries could still be done downstream for now. My fear is that this potentially moves in the direction of making the package a one-stop-shop for delegation to existing libraries, which could result in chasing lots of moving targets.

This is of particular importance when it has an impact on performance.

I think downstream libraries should be responsible for which array types (if any) they would like to optimise for. Then they can copy the agnostic implementations to customise to their needs.

[lucascolley]

My biggest concern with the package is making sure we have good quality assurance, so that we know the function implementations are correct.

restricting the scope as I suggest would make this part easier too

[asmeurer]

@lucascolley I went ahead and created an array-api-extra repo and gave you write access. https://github.com/data-apis/array-api-extra

Feel free to start prototyping stuff there. If it's helpful, you can copy stuff over from array-api-strict or array-api-compat.