Weather data from Environment and Climate Change Canada (ECCC)

[amotl]

Hi there,

based on some implementations by @Zeitsperre et al., we are looking at importing weather data from Environment and Climate Change Canada (ECCC) [1].

With kind regards,
Andreas.

[1] https://www.canada.ca/en/environment-climate-change.html

[Zeitsperre]

Honoured to be mentioned! It's interesting that you are looking into this. I know my work in https://github.com/Zeitsperre/canada-climate-python is a bit old, but I work on ECCC data collection/management as part of @Ouranosinc. I'm curious to learn more about your project here!

[gutzbenj]

Hi there! I'd say we share the curiosity when looking at Ouranos! Important topics at the right time! I'm surprised by the sheer size of that organization serving as a non-profit!

With our service we want to enable people accessing weather data in an easy way to remove the necessity of writing their own extraction scripts but rather focus on the analysis. So far we only cover the German Weather Service but we are almost ready to broaden our range of data products by some other services like ECCC.

[amotl]

Dear Trevor,

thanks for answering so quickly. I created this discussion thread as an introduction to #286 which I whipped up the other day and just pushed. Happy to see you joining.

I know my work in https://github.com/Zeitsperre/canada-climate-python is a bit old.

This is exactly where I found the essential gist to get this started. Thank you so much for publishing this as open source software the other day. As you can see, and also to second Benjamin, we are focusing on the acquisition aspects and are aiming to do everything in-memory while looking at state-of-the-art caching mechanisms like fsspec as suggested by @kmuehlbauer at #243 (comment) ff. within the next iterations.

We will be happy to see you adding any relevant suggestions to this discussion. Also, there might be a chance that our clean-room implementation might make it back to pieces of software you are conceiving these day by means of just having to import wetterdienst, if you like it?

With kind regards,
Andreas.

P.S.: Benjamin is also looking to get hydrological data acquisition aspects into Wetterdienst as well. Am I right to assume the best way to do that within the ECCC/@Ouranosinc ecosystem would be to use birdy or rooki in order to acquire data from a public Raven instance you are running over at @Ouranosinc, as roughly outlined at [1]?

[1] https://pavics-sdi.readthedocs.io/projects/raven/en/latest/getting_started.html

[Zeitsperre]

With our service we want to enable people accessing weather data in an easy way to remove the necessity of writing their own extraction scripts but rather focus on the analysis. So far we only cover the German Weather Service but we are almost ready to broaden our range of data products by some other services like ECCC.

Very cool. I haven't played much with it, but I know that ECCC and the MSC (Meteorological Service of Canada) maintains an FTP-like service for accessing station data on-the-fly: https://eccc-msc.github.io/open-data/msc-datamart/readme_en/

This is exactly where I found the essential gist to get this started. Thank you so much for publishing this as open source software the other day. As you can see, and also to second Benjamin, we are focusing on the acquisition aspects and are aiming to do everything in-memory while looking at state-of-the-art caching mechanisms like fsspec as suggested by @kmuehlbauer at #243 (comment) ff. within the next iterations.

No problem. I know that I was partway through some refactoring for Python3 and have since learned much better methods for dealing with the data outputs from ECCC. Having everything in memory is a very different approach, but I'm curious to see where you go with this.

One issue that we constantly have had to deal with are the unit conversions and ensuring that the data is compliant with the climate data standards we use at Ouranos (NetCDF CF-Conventions). I can point you to some work I've contributed to on this if it's of interest to you.

We will be happy to see you adding any relevant suggestions to this discussion. Also, there might be a chance that our clean-room implementation might make it back to pieces of software you are conceiving these day by means of just having to import wetterdienst, if you like it?

Yes possibly. I can't guarantee I can help too much, but depending on how things play out, perhaps this could be an interesting side project for my data processing/management needs. I'll chime in where I can.

P.S.: Benjamin is also looking to get hydrological data acquisition aspects into Wetterdienst as well. Am I right to assume the best way to do that within the ECCC/@Ouranosinc ecosystem would be to use birdy or rooki in order to acquire data from a public Raven instance you are running over at @Ouranosinc, as roughly outlined at [1]?

Yes, that's very much the approach to consider! Raven is part of the Bird-house project (https://bird-house.github.io/), which is a series of WPS services that aid in performing climate, hydrology, terrain, and EO-like workflows using remote data/servers. If you're interested in seeing some examples, Ouranos develops it as the PAVICS platform/project (https://www.ouranos.ca/publication-scientifique/FicheArsenault2020_EN.pdf). We have a JupyerLab based frontend that contains many examples of how to use birdy and run analyses. If you want an account, email pavics @ ouranosinc [dot] com, and we can set you up with a login to try it out.

Best,

[amotl]

Hi Trevor,

ECCC and the MSC (Meteorological Service of Canada) maintains an FTP-like service for accessing station data on-the-fly at https://eccc-msc.github.io/open-data/msc-datamart/readme_en/.

Thanks a stack!

One issue that we constantly have had to deal with are the unit conversions and ensuring that the data is compliant with the climate data standards we use at Ouranos (NetCDF CF-Conventions). I can point you to some work I've contributed to on this if it's of interest to you.

Sure, I am all ears. I am assuming you already know about cfgrib, if that is actually relevant on this matter?

Raven is part of the Bird-house project, which is a series of WPS services that aid in performing climate, hydrology, terrain, and EO-like workflows using remote data/servers.

Thanks for your elaboration, that looks very smart. I've also created #287 for further discussions about this on our end.

@Ouranosinc develops Raven as the PAVICS platform/project. We have a JupyerLab based frontend that contains many examples of how to use birdy and run analyses.

Nice!

If you want an account...

Thank you so much. I believe we will definitively come back to your offer. Maybe @gutzbenj currently has more time available to chime in here? As he is a hydrologist by profession, I believe this might spark his interest.

With kind regards,
Andreas.

[gutzbenj]

Thanks Andreas.

I'd be delighted to test the environment, but would probably come back to you @Zeitsperre next year after fixing some of the problems we still have with this library.

[amotl]

In this context, @gutzbenj just also shared two interesting links:

• https://climate.weather.gc.ca/FAQ_e.html#Q3
• https://drive.google.com/drive/folders/1WJCDEU34c60IfOnG4rv5EPZ4IhhW9vZH