SYKE (corine)

[MarkusKonk]

This is a template to structure the integration of the use cases' analyses into AquaINFRA's virtual research environment.
Please copy this template and assign all those who should be involved using the "Assignees" - feature on the right, for example, any developers or use case members from your side.

And please use the comment section below. No mailing will be necessary and the entire communication is in one place. Individual persons can be pinged using, for instance, @MarkusKonk

Description

• Rather a technical than a thematic description
• How to run the analysis?
• Which programming languages are used?
• How does the workflow look like?
• Which functions are used and how are they connected?
• Anything else needed to run the analysis?

Two tools are already deployed:

• https://aquainfra.ogc.igb-berlin.de/pygeoapi/processes/syke-corine-stats-real-areas
• https://aquainfra.ogc.igb-berlin.de/pygeoapi/processes/riverload

---

Plotting tool

(Started to work on this 2025-10-06)

URLs

• GitHub Repo:
• Deployed tool on AquaINFRA server: not yet

Status / blocking

What was done, what remains to do, and which task is blocking the further work?

• @eebruun sent GitHub repo
• Merret tried to build the conda env, failed, it is windows-specific
• [] @eebruun to generate a new, not platform-specific one... BLOCKING

---

Riverload tool

• Deployed: https://aquainfra.ogc.igb-berlin.de/pygeoapi/processes/riverload
• GitHub original: https://github.com/eebruun/aquainfra_TOC_loads
• GitHub aquabranch: https://github.com/eebruun/aquainfra_TOC_loads/tree/aquabranch

Status / blocking

What was done, what remains to do, and which task is blocking the further work?

• TODO

---

CORINE tool

• Deployed: https://aquainfra.ogc.igb-berlin.de/pygeoapi/processes/syke-corine-stats-real-areas
• GitHub original: https://github.com/eebruun/aquainfra_corine
• GitHub our fork: https://github.com/AquaINFRA/aquainfra_corine

Status / blocking

What was done, what remains to do, and which task is blocking the further work?

• (...)
• Documentation: Output file is hard to understand - does it make sense? Please add a better explanation of the outputs (asked by email 22 May, email thread "AquaINFRA SYKE CORINE tool: Progress and questions")
• Empty results, maybe due to CRS? Test whether their tool depends on the input data being in a specific CRS (asked by email 22 May), as with WGS84 data I got no results. Probably need to: Test, document, maybe check in the code... (Email thread "AquaINFRA SYKE CORINE tool: Progress and questions")
• Merge to SYKE GitHub repo (Discussion in thread "VS: VS: DDAS: ProviderTypeError when trying to access ..." since 14 May. Markus re-asked 02 June in Email thread "AquaINFRA SYKE CORINE tool: Progress and questions")

Input data (CORINE tool)

• Input datasets: Test datasets are included in the Github repo:
  • Harmonized_Corine_level1_headers.csv
  • Harmonized_Corine_level4_headers.csv
  • aquainfra_catchment areas.shp
  • (Plus large rasters which are not on GitHub)

Commands for running R/Python scripts

Commands for running Docker commands:

git clone git@github.com:AquaINFRA/aquainfra_corine.git
cd aquainfra_corine
docker build -t syke_corine:20250524 .
docker run --name syketest -v "./readonly:/readonly" -v "./out:/out" syke_corine:20250524 https://aqua.igb-berlin.de/download/some-shape-file.zip input_path_rasters input_path_headers output_path randomstring contact_email

cURL Commands for running OGC API Processes:

See here: https://github.com/AquaINFRA/aquainfra_corine/blob/aquabranch/pygeoapi_processes/process_corine_stats_real_areas.py

Checklist CORINE tool

• Input data is identified and/or accessible via DDAS
  • Input data is made accessible under an open license
• Analysis code is available on GitHub
  • Analysis code is made accessible under an open license
• Workflow is sketched: Does not apply, just one tool - or will there be more?
• Analysis is split into reusable functions: Does not apply, just one tool - or will there be more?
  • Input parameters are identified
  • functions follow input - processing - output - logic
  • input data can be passes via URL: Only the regions...
  • output data is stored in file, e.g., csv, json etc. (several csv files, one per input region, so we won't know beforehand how many)
• Reusable functions are saved in separate files
• Runtime, libraries, and library versions identified
• R commands are added to README
• Analysis is dockerized
• Docker commands are added to README
• OGC API Processes created locally
  • python scripts created
  • titles and descriptions for input parameters added to json (but should be improved)
• OGC API Processes deployed on server
• Tests successful
• Galaxy tool created locally
• Galaxy tool deployed on server
• Tests successful
• Workflow is complete

[merretbuurman]

Note: I only realized today that we have one issue for SYKE; but two very different tools so far (riverloads, and corine). Today I received the link to a third one, the plotting (https://github.com/eebruun/aquainfra_riverload_plot), which seems to combine both tools, so I guess we can keep this issue for communication on all three tools.

Plotting tool (3rd tool)

@eebruun Some questions on the new tool

• This tool plots combined results from both CORINE and riverload tool, correct?
• Basically, it is the main() function that you want us to put on Galaxy, right?

Inputs

I understand that you want it to have the inputs:

• url_riverload (output of Galaxy tool riverload)
• url_corine (output of Galaxy tool Corine)
• url_vemala (something external)
• min_date
• max_date
• area_name
• (You also define a url_vesla somewhere, but you do not use it as a parameter to main()

Outputs

I understand that you want to provide these outputs to the user:

• output_path, eventually a URL (pointing to a JPG file "Riverload_{area_name_out}.jpg")
• output_MK_path, eventually a URL (pointing to a CSV file "Mann-Kendall_{area_name_out}.csv")

TODO for new tool

• @eebruun Please provide a cleaned conda yml file without windows-specific stuff
• @eebruun Please provide a title and description for the tool, and a title and description for each input and each output

[merretbuurman]

Plotting tool

I had an initial look at the code and then I tried to install the conda environment, I ran into various errors that seem to be linked to the conda being exported on windows. I found the suggestion to make a cleaner yml file by:

conda activate aquainfra
conda env export --no-builds | grep -v "prefix:" > aquainfra.yml

[eebruun]

We have updated new conda yml file 20251015_riverload_plot.yml in https://github.com/eebruun/aquainfra_riverload_plot.

[merretbuurman]

Thanks! I managed to create the environment and build a Docker container to run the analysis. When I run it, I run into an error saying ZeroDivisionError: float division by zero. Can you reproduce this error and figure out what is going wrong? More details below...

This text got a bit long, because I will be out of office or the next weeks, so I wrote more details so you can hopefully work with it, and so I can remember what happens next, once I am back. (But I am available for questions, at least until Oct 24th).

Status

What I did, what worked, what did not

• Succeeded to build and activate the conda environment
• Succeeded to build a Docker image containing the analysis
• Modified the script to allow users to change and pass the inputs (so far they are hard-coded inside the script)
• But I ran into that error...
• I committed my modifications to a new branch aquabranch and tried to push that to GitHub for you to see, but I couldn't, apparently I don't have permissions, right?

What is needed next

• Permissions to push my modifications to GitHub (unless you prefer me to make a fork and then create a pull request, which is more complicated, but we can do that too of course). Then I can push my modifications and you can try it out.
• Solve that error, that "division by zero" problem. Maybe my input data is not correct?
• Provide a working set of test input data
• Confirm or correct the input parameters (three URLs: riverload, vemala, corine; and start date and end date and area name)
• What is the output? Some CSV? Where is it stored?
• For each input and each output, I need a title and a little description (short). Also for the entire tool (title, description)

Modifications of the script

At the moment the three URLs that the main() function needs (riverload, vemala and corine), and the start date, end date and area name are hard-coded in the script, so a user cannot modify them. I changed the script so that the input parameters can be passed from the command line (which is necessary to make a tool out of it). Then, you can run the script in python like this:

# These are the three URLs that I use:
URL_RIVERLOAD="https://aquainfra.ogc.igb-berlin.de/download/out/riverload-b31204fd-aa8f-11f0-8941-fa163e42fba0.csv"
URL_VEMALA="https://sykedata.ymparisto.fi/earthobservation-1/project/aquainfra/vemala/tocm_21_011.csv"
URL_CORINE="https://aquainfra.ogc.igb-berlin.de/download/out/outputs_339a4ec3-aab7-11f0-ac3e-fa163e42fba0/corine_stats_real_areas_Pitkkoski_339a4ec3-aab7-11f0-ac3e-fa163e42fba0.csv"
python 20251006_aquainfra_plot.py ${URL_RIVERLOAD} ${URL_VEMALA} ${URL_CORINE} "20251001" "20251003" "Myarea"

But when I do that, I run into this error: ZeroDivisionError: float division by zero:

# These are the three URLs that I use:
URL_RIVERLOAD="https://aquainfra.ogc.igb-berlin.de/download/out/riverload-b31204fd-aa8f-11f0-8941-fa163e42fba0.csv"
URL_VEMALA="https://sykedata.ymparisto.fi/earthobservation-1/project/aquainfra/vemala/tocm_21_011.csv"
URL_CORINE="https://aquainfra.ogc.igb-berlin.de/download/out/outputs_339a4ec3-aab7-11f0-ac3e-fa163e42fba0/corine_stats_real_areas_Pitkkoski_339a4ec3-aab7-11f0-ac3e-fa163e42fba0.csv"

# This is how I run the script, inside a docker container...
# This is equivalent to running:
# python 20251006_aquainfra_plot.py ... ... 

docker run sykeplot:20251016 ${URL_RIVERLOAD} ${URL_VEMALA} ${URL_CORINE} "20251001" "20251003" "Myarea"
syketest
Thu Oct 16 17:51:58 UTC 2025
Traceback (most recent call last):
  File "/app/20251006_aquainfra_plot.py", line 682, in <module>
Plotting riverload for Vantaan suualue
Riverload data processed!
processing corine data
corine data processed!
Processing Vesla data
Vesla data processed
Vemala data processed!
    output_path,output_MK_path, df_results = main(url_riverload, url_vemala, url_corine, min_date, max_date, area_name= 'Vantaan suualue')
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/20251006_aquainfra_plot.py", line 293, in main
    res = mk.original_test(series)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/riverload_plot/lib/python3.11/site-packages/pymannkendall/pymannkendall.py", line 253, in original_test
    Tau = s/(.5*n*(n-1))
          ~^^~~~~~~~~~~~
ZeroDivisionError: float division by zero
ERROR conda.cli.main_run:execute(125): `conda run python 20251006_aquainfra_plot.py https://aquainfra.ogc.igb-berlin.de/download/out/riverload-b31204fd-aa8f-11f0-8941-fa163e42fba0.csv https://sykedata.ymparisto.fi/earthobservation-1/project/aquainfra/vemala/tocm_21_011.csv https://aquainfra.ogc.igb-berlin.de/download/out/outputs_339a4ec3-aab7-11f0-ac3e-fa163e42fba0/corine_stats_real_areas_Pitkkoski_339a4ec3-aab7-11f0-ac3e-fa163e42fba0.csv 20251001 20251003 Myarea` failed. (See above for error)

[merretbuurman]

Update:
I could push to the GitHub repo this morning, here is my branch:
https://github.com/eebruun/aquainfra_riverload_plot/tree/aquabranch

Here is the commit that lets users pass their own URL, which you could try and then see whether you get the same error and why..
https://github.com/eebruun/aquainfra_riverload_plot/commit/c7e231df06cf32d67ee042679a1f372b113b624b

[eebruun]

1. Thanks! I managed to create the environment and build a Docker container to run the analysis. When I run it, I run into an error saying ZeroDivisionError: float division by zero. Can you reproduce this error and figure out what is going wrong? More details below...
   The error was caused within the mann-kendall test, when the script tried to perform the test with empty data set. More below.

This text got a bit long, because I will be out of office or the next weeks, so I wrote more details so you can hopefully work with it, and so I can remember what happens next, once I am back. (But I am available for questions, at least until Oct 24th).

Status
What I did, what worked, what did not
[X] Succeeded to build and activate the conda environment
[X] Succeeded to build a Docker image containing the analysis
[X] Modified the script to allow users to change and pass the inputs (so far they are hard-coded inside the script)
[ ] But I ran into that error...
[ ] I committed my modifications to a new branch aquabranch and tried to push that to GitHub for you to see, but I couldn't, apparently I don't have permissions, right?
What is needed next
[ ] Permissions to push my modifications to GitHub (unless you prefer me to make a fork and then create a pull request, which is more complicated, but we can do that too of course). Then I can push my modifications and you can try it out.

Permissions granted? If you can push the modifications to the main branch, let's keep that as our working version. We updated the script with your implemented user-inputs and additional improvements. The working script is now https://github.com/eebruun/aquainfra_riverload_plot/blob/main/Aquainfra_Riverload_plotting.py

[ ] Solve that error, that "division by zero" problem. Maybe my input data is not correct?
Indeed, input data was not correct. The maximum date where data is currently available is 31.12.2024. The Riverload data from aquainfra_TOC_loads seems to cover TOC-load until 12.11.2024 -> we only filter the modelled Vemala data to that period. The Vemala data is currently available till 31.12.2024.
[ ] Provide a working set of test input data
This should be fixed with correct time period
[ ] Confirm or correct the input parameters (three URLs: riverload, vemala, corine; and start date and end date and area name)
User is now informed if dates are outside of valid time range, or if there are errors reading the dataframes, or if the dataframes are empty. Also, we modified the script, so that if user inputs too late starting date (>31.12.2024), the script prints an error message and exits. If user inputs too late end date, the maximum date is hardcoded to 31.12.2024. This needs to be adjusted if we get the riverload data and modelled data automatically updated for further dates. Further, user is informed if the conditions for performing M-K test are fulfilled (over 2 year time period, starting from YYYY-01-01, ending to YYYY-12-31). If the conditions are not met, the script will skip the M-K test
[ ] What is the output? Some CSV? Where is it stored?
The outputs are a plot: .jpg image file saved to Results/date_today/Riverload_areaname.jpg. This should be modified for compatible format with Galaxy-system. Second output is the M-K-test .csv file, saved to Results/date_today/Mann-Kendall_areaname.csv. This should also be modified to proper Galaxy output. The output paths are currently created here:

DEFINE THE OUTPATH FOR GALAXY TEMPORARY OUTPUT TO SAVE FIGURE AND GENERATE URL

results_folder = os.path.join("Results", date_now)
if not os.path.exists(results_folder):
    os.makedirs(results_folder)
output = os.path.join(results_folder, f"Riverload_{area_name_out}.jpg")
output_mk = os.path.join(results_folder, f"Mann-Kendall_{area_name_out}.csv")

[ ] For each input and each output, I need a title and a little description (short). Also for the entire tool (title, description)
This is almost finished
At the moment the three URLs that the main() function needs (riverload, vemala and corine), and the start date, end date and area name are hard-coded in the script, so a user cannot modify them. I changed the script so that the input parameters can be passed from the command line (which is necessary to make a tool out of it). Then, you can run the script in python like this:
Thanks for these changes! They were under consideration, and are now implemented from your version. This approach work very well and user is now informed from invalid inputs

[eebruun]

Syke Plotting tool description.docx
Here is the Plotting tool description. Are you expecting anything from us? What happens next?

[merretbuurman]

Plotting tool

Hi,
I have tried your updated script in the Docker and it works. I also created a first pygeoapi process for it. Please test it using different dates and different inputs and let me know how it works, ok?

ALso, I noticed that the script does output info when the user provided non-functioning inputs, but as it does not throw any error, the script terminates normally, and pretends to be successful. When running on the server, the user does not see the log, and as there is no error, the tool will say it succeeded, but then there is no result. So please make sure the script throws a proper error if something is wrong.

Status

• Updated script seems to work in the Docker.
• First pygeoapi process created
• Pushed all to main: https://github.com/eebruun/aquainfra_riverload_plot/tree/main
• Please review the descriptions at https://aquainfra.ogc.igb-berlin.de/pygeoapi-dev/processes/riverload-plot
• Please test it using curl or some other HTTP tools, like you did with the other tools
• Please **modify the description of the parameter url_corine. It seems to want an URL for a CSV file, outcome of the CORINE tool (it is called url_corine inside your script). But you described it (with title csv_files_per_area) as: Output of CORINE tools: A JSON object containing links to the output result csv file of each polygon. Which does not really describe what the user needs to pass... I guess the user just picks one of the URLs that are returned by the CORINE tool, right? Could you update the description, so that a user who is not you would understand what to input?
• Please make sure the script throws a proper error if something is wrong.
• Bonus: Could you also make it so that the user can provide the names of the output files, and the directory? I would prefer it to save the results to /out/Mann-Kendal_Myarea.csv (usw.) and not to ./Results/2025_11_09/Mann-Kendall_Myarea.csv (usw) (Less urgent than the rest)
• @merretbuurman TODO: Improve the log extraction (it is R-specific right now, I need to make it python-specific to extract better error messages to send to the user)
• When this is done, @merretbuurman to move this from dev to prod
• When this is done, @MarkusKonk to have a look at this for Galaxy

Links

• Current tool description: https://aquainfra.ogc.igb-berlin.de/pygeoapi-dev/processes/riverload-plot
• Docker build and run instructions: https://github.com/eebruun/aquainfra_riverload_plot/blob/main/README.md
• Pygeoapi curl example requests: https://github.com/eebruun/aquainfra_riverload_plot/blob/main/pygeoapi_processes/README.md

[eeroa1]

Hi,
thanks for the feedback. We modified and tested the script for different time periods and wrong input links etc., and got the expected results/errors. The errors should now be logged correctly. Overall, the script is working as expected.

• Please review the descriptions at https://aquainfra.ogc.igb-berlin.de/pygeoapi-dev/processes/riverload-plot
  We clarified the descriptions on riverload-plot document 'Syke Plotting tool description.docx'

• Please test it using curl or some other HTTP tools, like you did with the other tools
  Tested with -curl commands on WSL successfully for unique requests of riverload-table, corine(GeoJson and shape), and for riverload-plotting function with the results generated from -curl requests. The plotting test was performed for Vantaa-area.

• Please **modify the description of the parameter url_corine. It seems to want an URL for a CSV file, outcome of the CORINE tool (it is called url_corine inside your script). But you described it (with title csv_files_per_area) as: Output of CORINE tools: A JSON object containing links to the output result csv file of each polygon. Which does not really describe what the user needs to pass... I guess the user just picks one of the URLs that are returned by the CORINE tool, right? Could you update the description, so that a user who is not you would understand what to input?
  Yes the CORINE tool generates csv-links for different areas (Lepsämä, Pitkäkoski, Vantaa, and Kerava). Description is updated.

• Please make sure the script throws a proper error if something is wrong.
  We changed all the printing lines to loggin.info, and added logging.errors and raising errors for the different datasources. The script was developed to be used for the first day of the year -> last day of the year to be also compatitable with seasonal Mann-Kendall analysis. The un-optimal temporal selection is currently logged only as info, not as an error, because it's possible to run the script for other time periods as well. The temporal constraint are presented in the descriptions document. Should this un-recommended temporal user input be logged as an error, or is info enough?. The Mann-Kendall constraints are currently logged as info as well. The changes were pushed to 'Aquainfra_Riverload_plotting.py'

• Bonus: Could you also make it so that the user can provide the names of the output files, and the directory? I would prefer it to save the results to /out/Mann-Kendal_Myarea.csv (usw.) and not to ./Results/2025_11_09/Mann-Kendall_Myarea.csv (usw) (Less urgent than the rest)
  This is still hardcoded, but we changed the results dir to 'out' to follow the format '/out/Mann-Kendal_Myarea.csv'. We are not exactly sure how this should be modified for the needs of the Galaxy. We can add the user inputs as output-folders or output-filenames