Change default netCDF engine to use h5netcdf and add netcdf_engine_order

[shoyer]

The default engine when reading/writing netCDF files is now h5netcdf or scipy, which are typically faster than the prior default of netCDF4-python. You can control this default behavior explicitly via the new netcdf_engine_order parameter in set_options(), e.g., xr.set_options(netcdf_engine_order=['netcdf4', 'scipy', 'h5netcdf']) to restore the prior defaults.

I've also updated the documentation page which misled @lesserwhirls about Xarray supporting invalid netCDF files without invalid_netcdf=True.

• Closes Should Xarray prefer h5netcdf and scipy to netCDF4? #10657
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[shoyer]

Looking at the test failures, it looks like we previously supported writing NCZarr with ds.to_netcdf(f"file://{filename}#mode=nczarr"). Now we require also passing engine='netcdf4' explicitly.

Should we try to auto-detect URLs like this and use netcdf4 as the backend? Or is it better to encourage users to make an explicit choice?

[dcherian]

in general I'm pro "explicit choice", but this would be a breaking change.

@malmans2 how common is nczarr use? I haven't really seen it.

[shoyer]

I went ahead and added automatic support for writing nczarr. This wasn't hard to check.

[malmans2]

in general I'm pro "explicit choice", but this would be a breaking change.

@malmans2 how common is nczarr use? I haven't really seen it.

I've never seen it actually used in python applications either. From a quick search on GitHub, it looks like the few packages that write to nczarr directly use netcdf4-python rather than xarray

[shoyer]

I added supports_groups to BackendEntrypoint. Otherwise, we have no way to check if a backend supports open_datatree() short of calling the open_datatree() method.

This turned up because scipy is now used in preference to netcdf4 when opening netcdf v3 files, but scipy doesn't support opening groups.

In principle we could add support for reading groups to the SciPy backend (netcdf3 files arguably contain a single group, at the root node), but in any case this will also come up for custom backends.

[shoyer]

I would love to get this in before the next release, to avoid needing repeated breaking changes.

[kmuehlbauer]

LGTM, Stephan. Nice to be able to parametrize this.