Description
cudf-polars currently supports passing fixed-size Array columns through GPU. This feature request extends that support to is_null() and is_not_null() on Array columns.
Reproducer
import polars as pl
df = pl.DataFrame(
{
"embedding": pl.Series(
[
[1.0, 2.0],
None,
[None, None],
[3.0, None],
],
dtype=pl.Array(pl.Float32, 2),
)
}
)
query = df.lazy().select(
pl.col("embedding").is_null().alias("is_null"),
pl.col("embedding").is_not_null().alias("is_not_null"),
)
print(
query.collect(
engine=pl.GPUEngine(
executor="streaming",
raise_on_fail=True,
)
)
)
Output
NotImplementedError: Query execution with GPU not possible: unsupported operations.
Expected
Query should run on GPU and match Polars:
shape: (4, 2)
┌─────────┬─────────────┐
│ is_null ┆ is_not_null │
│ --- ┆ --- │
│ bool ┆ bool │
╞═════════╪═════════════╡
│ false ┆ true │
│ true ┆ false │
│ false ┆ true │
│ false ┆ true │
└─────────┴─────────────┘
Only outer validity of each Array row should be checked. [None, None] and [3.0, None] are valid Array rows.
Scope after Polars optimization
Polars supports all of expressions below. cudf-polars receives optimized plan, which may contain different operation from original Python expression. This issue covers direct outer null checks and expressions that preserve those checks after optimization.
| Python expression |
Optimized expression in Polars 1.44 |
This issue |
a.is_null().cast(pl.Int8) |
a.is_null().strict_cast(pl.Int8) |
Supported |
df.drop_nulls("a") |
Filter on a.is_not_null() |
Supported |
a.is_null().sum() |
a.null_count() |
Out of scope |
a.is_not_null().sum() |
a.len() - a.null_count() |
Out of scope |
a.is_null().any() |
a.has_nulls() |
Out of scope |
a.is_not_null().all() |
a.has_nulls().not() |
Out of scope |
Description
cudf-polars currently supports passing fixed-size Array columns through GPU. This feature request extends that support to
is_null()andis_not_null()on Array columns.Reproducer
Output
Expected
Query should run on GPU and match Polars:
Only outer validity of each Array row should be checked.
[None, None]and[3.0, None]are valid Array rows.Scope after Polars optimization
Polars supports all of expressions below. cudf-polars receives optimized plan, which may contain different operation from original Python expression. This issue covers direct outer null checks and expressions that preserve those checks after optimization.
a.is_null().cast(pl.Int8)a.is_null().strict_cast(pl.Int8)df.drop_nulls("a")a.is_not_null()a.is_null().sum()a.null_count()a.is_not_null().sum()a.len() - a.null_count()a.is_null().any()a.has_nulls()a.is_not_null().all()a.has_nulls().not()