-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
no need to uncompress the zstd JSON to disk -- fsspec can handle it!
For example:
%%time
fs = fsspec.filesystem("reference", fo='s3://esip-qhub-public/ecmwf/era5_multivar_197901_202205.json.zst',
ref_storage_args={"compression": "zstd" ,'skip_instance_cache':True},
remote_protocol='s3', remote_options={'anon':True})
m = fs.get_mapper("")
ds = xr.open_dataset(m, engine="zarr", backend_kwargs={'consolidated':False}, chunks={'time0':288})
dsalso perhaps create an intake catalog for the smaller and larger ERA5 virtual datasets?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels