Skip to content

[FEA] Support outer null checks for Array columns in cudf-polars #24235

Description

@0guban0v

Description

cudf-polars currently supports passing fixed-size Array columns through GPU. This feature request extends that support to is_null() and is_not_null() on Array columns.

Reproducer

import polars as pl

df = pl.DataFrame(
    {
        "embedding": pl.Series(
            [
                [1.0, 2.0],
                None,
                [None, None],
                [3.0, None],
            ],
            dtype=pl.Array(pl.Float32, 2),
        )
    }
)

query = df.lazy().select(
    pl.col("embedding").is_null().alias("is_null"),
    pl.col("embedding").is_not_null().alias("is_not_null"),
)

print(
    query.collect(
        engine=pl.GPUEngine(
            executor="streaming",
            raise_on_fail=True,
        )
    )
)

Output

NotImplementedError: Query execution with GPU not possible: unsupported operations.

Expected

Query should run on GPU and match Polars:

shape: (4, 2)
┌─────────┬─────────────┐
│ is_null ┆ is_not_null │
│ ---     ┆ ---         │
│ bool    ┆ bool        │
╞═════════╪═════════════╡
│ false   ┆ true        │
│ true    ┆ false       │
│ false   ┆ true        │
│ false   ┆ true        │
└─────────┴─────────────┘

Only outer validity of each Array row should be checked. [None, None] and [3.0, None] are valid Array rows.

Scope after Polars optimization

Polars supports all of expressions below. cudf-polars receives optimized plan, which may contain different operation from original Python expression. This issue covers direct outer null checks and expressions that preserve those checks after optimization.

Python expression Optimized expression in Polars 1.44 This issue
a.is_null().cast(pl.Int8) a.is_null().strict_cast(pl.Int8) Supported
df.drop_nulls("a") Filter on a.is_not_null() Supported
a.is_null().sum() a.null_count() Out of scope
a.is_not_null().sum() a.len() - a.null_count() Out of scope
a.is_null().any() a.has_nulls() Out of scope
a.is_not_null().all() a.has_nulls().not() Out of scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions