Pyodide & Zarr

[jakirkham]

Learned recently that Pyodide is including Zarr, which is great to hear 😄 Also happy to learn folks are able to build Numcodecs.

Would be interested to learn more about how folks are using Pyodide & Zarr together. So opening this issue for that discussion.

[jakirkham]

cc @rth (in case you have insights here 🙂)

[martindurant]

I wonder how many fsspec implementations run in pyodide? Perhaps nothing that uses async...
Maybe we need a zarr JS frontend less than we thought :)

[jakirkham]

I didn't think async was an issue, but threads and sockets are ( dask/dask#7764 (comment) ). IDK how requests would work there, but it seems some effort may be needed ( pyodide/pyodide#1956 ), which seems like the bigger blocker to using fsspec with Pyodide.

Yeah this came up in the discussion about Zarr.js ( #24 ). Namely that WASM may eventually solve things here.

[davidbrochart]

Pyodide supports async, but not threads.
I'm wondering how far we are from having Xarray and Dask to work in JupyterLite. It would be nice for cloud-based data/processing, since the browser would just be used for Dask scheduling and visualization.

[martindurant]

Dask uses threads for sure, and fsspec also uses a dedicated IO thread for async.

[jakirkham]

One could use the single threaded scheduler with Dask. Think that is what people trying Dask with Pyodide have generally done. Some discussion in issue ( dask/dask#7764 ).

[oeway]

I think if zarr can somehow support async store natively, we will be able to create something like fsspec for pyodide.

Without that, it's going to be very hard and we will need to either wait for pyodide to support multi-threading, or using some tricks or dark magic described here.

[joshmoore]

@jakirkham : guess a discussion probably leaves my question at #14 (comment) open :)

@oeway : do you have an idea of what the minimum steps would be to having a working implementation? (Do I need to stroll through zarr-developers/zarr-python#536 and look to revive items?)

[martindurant]

Another idea from @d70-t was to enable a fully async mode for zarr, like x = await arr[:], which could then use an async backend such as the browser HTTP.

fsspec/filesystem_spec#960 is a small prototype of an async fsspec filesystem for HTTP; it works, but cannot be called by zarr currently, since the mapper interface makes blocking calls.

[d70-t]

My current understanding of how things in a Browser work roughly is: you simply can't do any blocking wait for doing any IO or waiting for other "threads" (web workers). Instead you always have to use either callbacks or async functions. Even if you do async IO in a separate worker, you can't do a blocking wait in your main thread for the worker to finish a job...

Running fsspec fully async is fine and @martindurant's backend just needs a little polishing but works already now. Using it, what you can do right now is: load all the data asynchronously and once data is there, use zarr synchronously (e.g. on a MemoryStore). This of course is very clumsy and what we really want is to delay loading data until e.g. .isel within xarray has happened. For this to work, I believe, we must have an async interface all the way down. That interface probably should be explicit, e.g. the equivalent of x = await arr[...] or even x = await ds.var.isel(...) as mentioned here in zarr-developers/zarr-python#536, but had been postponed as the second step (which seems to be a reasonable thing, because it will touch a lot of code...).

Maybe there's a way to hack around it using the dark magic mentioned by @oeway (which as far as I understand, basically tries to implement async without using the async-Keyword).

If it would be for the fully async implementaion (I can't really estimate how feasible that would be), another question would be if we even need something like x = await (await ds.var).isel(...) in stead of x = await ds.var.isel(...). Probably we don't want to have that (but there are ways to avoid this).

[oeway]

@d70-t I think you are right! And I like the idea of going for fully async implementation.

Not just for pyodide, it can also be super beneficial for running zarr in native Python on the server side, e.g. serving a large number of users. In any async programming (e.g. a FastAPI server app), we don't want blocking code in order to achieve higher concurrency. Having native async suppoert in zarr would be useful when combined with async IO libraries such as aiofiles or aiohttp.

For the current pyodide (without mutli-threading support), the easiest way for me is to use XMLHTTPRequest to "convert" async request to sync, and that works well in e.g. JupyterLite.

I have a working demo where I use Zarr-Python with a ZipStore to fetch and visualize a remote large tissue image with http requests:
https://jupyter.imjoy.io/lab/index.html?load=https://gist.github.com/oeway/391b4352ea57b5682366ce3dc2fa9174&open=1

However, this only works if the Pyodide code runs in a web-worker (like a thread in the browser), and it will either fail or block the web UI when used in the main thread.

PyScript would be a nice way for using zarr-python in the browser, but I am not sure whether it run python in a separate web-worker or not.

[martindurant]

Is there anything preventing the use of that sync file-like object in general pyscript code? Are there any restrictions, can I make an fsspec filesystem out of it? I don't immediately see where a webworker is used, and whether it is necessary for just the IO part. That would immediately enable intake.open, pandas.read_csv, etc. and anything else that currently depends on requests. Of course, async is better for zarr when accessing multiple chunks...

PyScript would be a nice way for using zarr-python in the browser, but I am not sure whether it run python in a separate web-worker or not.

I believe no, but the situation is rapidly changing, and the whole topic of async and/or threads is high on the priority list.

[d70-t]

I didn't know that a synchronous request actually exists in the Web (but XMLHttpRequest.open() with async=False clearly is one).

Based on this it should be relatively easy to build a synchronous HTTP fsspec-filesystem, which in turn should then work with current zarr and xarray. But it will be slow and likely have some bad user experience (e.g. due to blocking the user thread).

[d70-t]

So here's some discussion on synchronous XHR: it's deprecated. (And even more deprecated on the main thread). So it will work 😄, but we shouldn't do it 😬 ...