GSoC Idea Discussion: Native xarray Support in Lumen via XarraySource

[SanketMeghale]

Hi, I am preparing for the Lumen + xarray Integration GSoC idea and wanted to get feedback on the technical direction and MVP scope.

After reviewing Lumen's source architecture, my current view is that xarray support should be introduced as a native XarraySource built on Source, rather than on BaseSQLSource.

The main reason is that xarray is fundamentally coordinate-aware and multidimensional, while BaseSQLSource is centered around SQL expressions and SQL execution. Starting from a SQL-first abstraction seems likely to force premature flattening and lose important xarray semantics too early. My current thinking is that SQL interoperability, if needed, would fit better as a later interoperability layer rather than as the MVP foundation.

Current design:

• xarray.Dataset as the backend object
• data_vars exposed as logical tables
• 1D coordinates exposed as queryable/filterable fields
• dims and attrs preserved through metadata
• filtering handled natively in xarray
• results converted to pandas DataFrames at the compatibility boundary

Current MVP scope:

• get_tables()
• get_schema()
• get_metadata()
• get()
• in-memory and URI-based dataset loading
• coordinate-aware filtering with datetime/date coercion

I am currently treating aggregation as out of scope for get() and more appropriate for a later transform layer.

I have prototyped this locally with tests for schema, metadata, filtering, URI loading, and edge cases, and also validated it against xarray's real air_temperature tutorial dataset.

The main questions I would appreciate feedback on are:

1. Does exposing Dataset.data_vars as logical tables seem like the right MVP model?
2. Does it make sense to keep aggregation in a transform layer rather than in the source API?
3. Is native source-level support the right first milestone, with SQL/AI interoperability deferred to later work?
4. Should multi-file dataset support be part of the MVP, or a later extension?

Any feedback on whether this seems like the right direction would be very helpful.

[SanketMeghale]

Following up here with a small update: I now have the MVP prototype working locally with tests/examples covering schema, metadata, filtering, URI loading, and validation against xarray's air_temperature tutorial dataset.

@philippjfr, @ahuang11 when you have time, could you please share whether this direction seems reasonable for an MVP, especially:

• exposing Dataset.data_vars as logical tables
• keeping aggregation out of the source API for now
• starting with a native XarraySource and deferring SQL interoperability

If a draft PR would be more useful than design discussion at this stage, I can open one.

[SanketMeghale]

I've opened a draft PR with the current MVP implementation for early feedback:

• PR: Add experimental XarraySource #1751

It includes the native XarraySource, focused tests, and docs/examples, while still keeping aggregation and broader interoperability out of scope for now.

[ahuang11]

I'd like to re-iterate from https://holoviz.org/contribute.html#using-ai-readme

please refrain from tagging maintainers shortly after opening a PR/issue, as reviews will happen when time allows

Thank you

[SanketMeghale]

Thank you for the clarification and for pointing me to the guidelines. I apologize for the earlier tagging and will avoid doing that going forward. I appreciate the time maintainers take to review contributions.

[ghostiee-11]

Closing this, as i feel its done in #1741 ...