Open
Description
I'm trying to create fsspec files for netCDF 3 datasets on a local filesystem, but am encountering an error during the json reference file creation process:
AsyncGroup.create_array() got an unexpected keyword argument 'compression'
The error is coming from the netCDF3.py module NetCDF3toZarr translate method in the following block:
arr = z.create_dataset(
name=dim,
shape=shape,
dtype=var.data.dtype,
fill_value=fill,
chunks=shape,
compression=None,
)
I'm using Kerchunk v0.2.7.
compression is not included in the create_dataset parameter list according to the zarr documentation, but compressor is, so fixing the argument in this and other create_dataset calls should presumably solve the issue.
Steps to reproduce:
def write_fsspec(fs_read, input_file, output_dir):
with fs_read.open(input_file) as infile:
print(f"Running kerchunk generation for {input_file}...")
chunks = kerchunk.netCDF3.NetCDF3ToZarr(infile, inline_threshold=300)
file_name = os.path.basename(input_file)
file_name = file_name.replace('.nc', '.json')
out_file_name = output_dir + '/' + file_name
with open(out_file_name, "wb") as f:
f.write(json.dumps(chunks.translate()).encode()) # call to netCDF3.py originates here
print(f"Finished writing {out_file_name}")
return out_file_name
dir_path = config['input_dir']
dir_path += '**/*.nc'
file_paths = glob.glob(dir_path, recursive=True)
fs_read = fsspec.filesystem('local')
temp_dir = TemporaryDirectory(prefix=config['output_dir'])
output_files = [write_fsspec(fs_read, f, temp_dir.name) for f in file_paths]
Metadata
Metadata
Assignees
Labels
No labels