HELCOM HEAT

[merretbuurman]

This is a template to structure the integration of the use cases' analyses into AquaINFRA's virtual research environment.
Please copy this template and assign all those who should be involved using the "Assignees" - feature on the right, for example, any developers or use case members from your side.

And please use the comment section below. No mailing will be necessary and the entire communication is in one place. Individual persons can be pinged using, for instance, @MarkusKonk

Description

• Rather a technical than a thematic description
• How to run the analysis?
• Which programming languages are used?
• How does the workflow look like?
• Which functions are used and how are they connected?
• Anything else needed to run the analysis?

URLs

URL to input datasets:

• Default data is downloaded by the original HEAT code: https://github.com/AquaINFRA/HEAT/blob/aquabranch/R/all_heat_functions.R#L9
• Additional input data to be downloaded from ICES portal

URL to GitHub:

• Original from HELCOM: https://github.com/ices-tools-prod/HEAT
• Code was modularized, extended, updated by Merret: https://github.com/AquaINFRA/HEAT/tree/aquabranch
• Added pygeoapi processes: https://github.com/AquaINFRA/HEAT/tree/aquabranch_pygeoapi

Commands for running R scripts

• Here

Commands for running Docker commands:

• Here

cURL Commands for running OGC API Processes:

• HOLAS Assessment workflow
• Customizable HEAT workflow

Checklist

• Input data is identified and/or accessible via DDAS
  • Description of how to download ICES for this data is written
  • Input data is made accessible under an open license
• Analysis code is available on GitHub
  • Analysis code is made accessible under an open license: https://github.com/ices-tools-prod/HEAT/blob/master/LICENSE
• Workflow is sketched
• Analysis is split into reusable functions (by Merret, to be reviewed for usefulness by HELCOM people)
  • Input parameters are identified (by Merret, to be reviewed for usefulness by HELCOM people)
  • functions follow input - processing - output - logic (by Merret, to be reviewed for usefulness by HELCOM people)
  • [x input data can be passed via URL
  • output data is stored in file, e.g., csv, json etc. (by Merret, to be reviewed for usefulness by HELCOM people)
• Reusable functions are saved in separate files
• Runtime, libraries, and library versions identified
• R commands are added to README
• Analysis is dockerized
• Docker commands are added to README
• OGC API Processes created locally
  • python scripts created
  • titles and descriptions for input parameters added to json (by Merret, to be reviewed for usefulness by HELCOM people)
• OGC API Processes deployed on server (WIP)
• Tests by Merret successful (WIP)
  • HOLAS1
  • HOLAS2
  • HOLAS3
  • HOLAS4
  • HOLAS5
  • HEAT1
  • HEAT2
  • HEAT3
  • HEAT4
  • HEAT5
• Tests by HELCOM successful
  • HOLAS1
  • HOLAS2
  • HOLAS3
  • HOLAS4
  • HOLAS5
  • HEAT1
  • HEAT2
  • HEAT3
  • HEAT4
  • HEAT5
• Galaxy tool created locally
• Galaxy tool deployed on server
• Tests successful
• Workflow is complete

[merretbuurman]

Fine-grained TODOs...

WP4

• Fix dependency problem during Docker image build
• Enable downloading outside input files inside the container
• Send them the error I found when using ICES input data

HELCOM

• Testing
• Provide GitHub user name for this issue
• Feedback on whether users should be able to omit one type of data

HELCOM but not urgent

• Ideally, more explanatory error messages...
• Keep the modularizaion, pull it into HEAT GitHub?

[merretbuurman]

Tests of OGC API by HELCOM

Dmitry has tested the processes and ran into issues in the custom ones:

• [SOLVED] Providing custom input files stored on google drive did not work --> solved.
• [WONTFIX] Providing custom input files stored on HELCOM Sharepoint did not work --> Sharepoint requires log in, so it will never work on our platform, as we don't provide any authentication forwarding.
• [WORKAROUND WORKS] The test polygon in question led to an error Geometry type of '3D Polygon' not supported in shapefiles. --> this will be solved, but in the meantime, manually store the shapefile e.g. with SHPT=POLYGON before using it as input (e.g. in QGIS under Export - Save Features As - Layer Options).

Merrets TODOs:

• Still: Move the download+unzip into the R part, not the python part - copy from specleanr getdata.R (https://github.com/AnthonyBasooma/specleanr/blob/aquabranch/src/getdata.R#L62)
• (Maybe clean up file names of downloaded files, to avoid difficult-to-handle ones like /job_xyz/unzipped_download?id=17o252m1oKsycLrPi434scwVoAJkZCQU3&export=download&authuser=0/test polygon.shp )
• Manually force the SHPT=POLYGON when storing the shapefile in heat-custom-1. Investigate which types work, or which don't, and how to do that in a smart way.

[MarkusKonk]

Heat-Workflow: https://aqua.usegalaxy.eu/u/markus.konkol/w/workflow-constructed-from-history-heat-1
Heat-History: https://aqua.usegalaxy.eu/u/markus.konkol/h/heat

HeatAdvanced-Workflow: https://aqua.usegalaxy.eu/u/markus.konkol/w/workflow-constructed-from-history-heatadvanced-workflow-2
HeatAdvanced-History: https://aqua.usegalaxy.eu/u/markus.konkol/h/heatadvanced-workflow