Find integration point with the existing duckdb_zarr

[alxmrs]

Xref: https://github.com/WayScience/duckdb_zarr

Cc: @d33bs

Hello, Dave!

It seems we have the same or very similar ideas about how Zarr can integrate with the DuckDB/RDBMS ecosystem. My collaborator and I, Jayendra, come from the pangeo/climate/weather side of the Zarr community. We have recently explored integrations between Nd arrays and DataFusion across two projects:

• https://alxmrs.github.io/xarray-sql/
• https://crates.io/crates/zarr-datafusion

I'm excited that this idea resonated with you enough to build it. I see no sense in reinventing the wheel; what do you think is the best path going forward to realize the potential of reinventing scientific data by bringing the relational model to gridded rasters?

Here is a talk I gave about this idea:

https://youtu.be/AtB_6c-GcJE?si=NDt9LyfYMjb4PC0b

I think working together would be ideal, as we come from similar but different parts of the Zarr community.

If you'll tolerate it, maybe a good next step is for us to meet.

Cheers,

Alex

[alxmrs]

Cc: @jayendra13

[jayendra13]

repo is public for https://crates.io/crates/zarr-datafusion here https://github.com/jayendra13/zarr-datafusion

[d33bs]

Thanks for opening this issue, Alex, and for sharing the related projects and talk! I definitely think there’s strong alignment here, even though we’re coming from different scientific domains.

One of the most exciting parts of this space right now is seeing similar ideas emerge independently across communities like climate/Pangeo and bioimaging. To me, that’s a strong signal that there’s a real need for better integration between ndarray-oriented scientific data and modern query/analytical systems.

I completely agree that we’re better off collaborating than reinventing the wheel in parallel. The cross-science perspective is especially valuable here since many of these challenges around scalability, interoperability, and queryability are shared across domains, even if the underlying data or science differs. In general I've been really inspired by a lot of the scientific software and infrastructure work coming out of the geoscience community in particular - it'd be exciting to see how we can improve things together!

A conversation definitely sounds worthwhile. I’d love to better understand where the projects overlap, where they differ, and where there may be opportunities to align efforts moving forward.

What do you think is the best way for us to coordinate further? Feel free to use the email address from my GitHub profile if you'd like.

Looking forward to continuing the discussion!

[alxmrs]

Sent you an email! Can't wait to meet :)

[d33bs]

Documenting our decisions from discussions outside of GitHub here: we decided to soft-archive work on duckdb_zarr and merge efforts within duckdb-zarr (this repo). As a first step, we worked on bioimage compatibility via #15 (which was a core focus of duckdb_zarr). Next steps involve eventually archiving duckdb_zarr and replacing the existing community extension for duckdb-zarr with this repo's implementation. I feel we should also rename the extension to simplify: zarr, once we get to that point.