Welcome to wetterdienst Discussions!

[gutzbenj]

Hey everyone and welcome to our discussion board. If you have any ideas about what other data out there we could cover you may put your ideas here. Also you may as well lead some general discussions about the topic of enivronmental data.

[mhaberler]

Hello,

since I was kindly invited by @gutzbenj to join in on gribmagic but discussions seem to happen here, let me introduce myself and my use case:
Internet old hand, electronics/RF engineer and MBA by education, owner of an aborted academic career and avid hot air balloon pilot with a knack for winter Alps crossings.

My interest derived from my ballooning hobby and I focus on wind trajectory generation using the best available weather model(s) for this purpose, and for animation. Towards that end I have initiated and helped with the windy.com traj plugin which is now used throughout the ballooning community and is most helpful using the icon-EU model by DWD in our part of the world. The method is also applicable to parachute jump planning and a friend and mentor of mine is involved with this so I want to help with that too.

The cosmo-d2 short term forecast model is currently mutated into icon-d2 by DWD and is becoming a standard service. It sports much better resolution than icon-EU (2km grid) and superb vertical resolution but nobody seems to have used it for this purpose yet.

My data needs are:

• extract a subset of parameters from the currently valid forecast from DWD (wind, pressure, temperature, humidity etc) in lat/lon/level dimensions

• possibly extract a bounding box of relevant data - I am mostly interested in the Alps region

• make that data remotely accessible for programs for trajectory and sounding forecast. Clients would be in Python but potentially also in Javascript so access methods need to be compatible.

• archive some of these parameter sets for hindcasting flights given a GPX track

• low usage intensity: personal, potentially some community use

My current wisdom to solve the problem effectively is:

• selectively download gribs from DWD and process them into a/some netCDF files as these support remote lazy loading for indexed access (grib2 seems not to support that important aspect)
• possibly compress the files using netCDF discretization
• on the client side, use xarray to process these netCDF files with lazy loading of only the required sub-cubes of date

There is a "standard" way of serving netCDF with the OpenNdap protocol, the hyrax server which might do the job as well but it is a Java/Tomcat/Apache clunker all of which I tend to avoid.

This has led me to the pydap project which can handle both client and server side based on a pure Python library and it seems to do the job.

So this is where I stand. I need to solve the downloading & extraction problem, and the remote indexed access/lazy loading problem in a simple, automatically manageable way. Hence my interest in gribmagic and pydap.

[amotl]

Dear Michael,

thanks for joining our collaborative development efforts. Your suggestions and needs are both very reasonable, so let's try to bring all ideas together and we will see how we can push the code base into a direction to make working with GRIB data effortless in the context of what you are aiming at.

As you can see I am currently busy to get gribmagic into a shape that it can be published on PyPI. Maybe I will be able to push an early preview version of gribmagic-0.1.0 before Christmas Eve. However, there is still much refactoring needed before I want to do that. We also might want to remedy some bugs and bring in new features.

In order to parallelize development and research, I will be happy to dip into early exploration of some of the technologies you are suggesting here and humbly ask you to join me on that in order to share some workload.

I am looking forward to all things we might conceive together.

With kind regards,
Andreas.

[amotl]

P.S.: Also, while I appreciate in general that GitHub gained a discussion section just recently, I am also humbly asking you to join us for further discussions around those topics over at the Panodata Community Forum [1] and use that as a collaborative workspace beyond code.

While the number of people writing there is yet low, using the right tool for the job is very important to me. We are running Discourse there and I strongly believe it is the absolute best forum software available these days. Benjamin and me talked about this just yesterday and he will also be happy following us to have fruitful discussions there.

Saying that, I've just created a separate category section [2] which should be all about weather data and corresponding tooling, mostly coming from the community but also from our own pen.

[1] https://community.panodata.org/
[2] https://community.panodata.org/c/weather-data/15

[mhaberler]

fair enough. I signed up on [1] as mhaberler, pending approval

will install gribmagic and see how I fare

what is the relation of wetterdienst and gribmagic? I played with wetterdienst and that looks standalone so far

[amotl]

Just sharing a concise "About ICON" page at the DWD.

https://www.dwd.de/EN/research/weatherforecasting/num_modelling/01_num_weather_prediction_modells/icon_description.html

[amotl]

Dear Michael,

So this is where I stand. I need to solve the downloading & extraction problem, and the remote indexed access/lazy loading problem in a simple, automatically manageable way. Hence my interest in gribmagic and pydap.

We just gave GribMagic [1] some love and finally published the package to PyPI [2]. You can read about more the details and join the discussion at [3].

With kind regards,
Andreas.

[1] https://github.com/earthobservations/gribmagic
[2] https://pypi.org/project/gribmagic/
[3] https://community.panodata.org/t/developing-gribmagic/233

[mhaberler]

@amotl really appreciated!

my current goal is a forecast vertical wind profile given a location, based on ICON-Global, ICON-EU and ICON-D2 depending on location

delivery as a JSON API, possibly REST

it looks like we have a diploma thesis candidate doing that using your code as a basis, goal time spring

and if the chap don't perform I'll do it myself as we really need it ;)