application of data compression via netCDF library

[lars2015]

• enable data compression features of netCDF library to compress meteo data (and particle data)

• newer version of netCDF library offers advanced data compression capabilities

  • lossy compression is enabled with the new quantize feature: https://docs.unidata.ucar.edu/netcdf-c/4.9.2/md__media_psf_Home_Desktop_netcdf_releases_v4_9_2_release_netcdf_c_docs_quantize.html
  • support for advanced compression methods (e.g. Zstandard) is supported via HDF5 filters: https://docs.unidata.ucar.edu/netcdf-c/4.9.2/filters.html
  • maybe look at NCZarr implementation? https://docs.unidata.ucar.edu/netcdf-c/current/md__media_psf_Home_Desktop_netcdf_releases_v4_9_2_release_netcdf_c_docs_nczarr.html
  • standard compression methods of netCDF library (zlib, szlib) are available via nc_def_var_deflate(), but these are slow

• Tasks:

  • review code for netCDF output of meteo data in MPTRAC
  • enable Zstd + quantize compression via netCDF library
  • define and evaluate test cases based on ERA5 and ERA6 data (compression rate? bandwidth?)