Malta

[merretbuurman]

This is a template to structure the integration of the use cases' analyses into AquaINFRA's virtual research environment.
Please copy this template and assign all those who should be involved using the "Assignees" - feature on the right, for example, any developers or use case members from your side.

And please use the comment section below. No mailing will be necessary and the entire communication is in one place. Individual persons can be pinged using, for instance, @MarkusKonk

Description

• Rather a technical than a thematic description
• How to run the analysis?
• Which programming languages are used?
• How does the workflow look like?
• Which functions are used and how are they connected?
• Anything else needed to run the analysis?

Malta use case will have two main steps (plus maybe visualisation):

1. SEAWAT groundwater model (FORTRAN?? C?? compiled, Joel already dockerized, but so far no user parameters possible). Or was this called ModFlow? What is ModFlow? What is iMod?
2. mitgcm model (FORTRAN, compiled, Joel will attempt to dockerize), probably including post-processing so that users receive NetCDF or similar "readable" files instead of cryptic binary mitgcm output files?

@ Sarah, maybe you can explain a little bit what the steps do (not too much into water details), and what are the inputs and outputs (again, not too much into water details, but rather technical: Are inputs/outputs files (and which formats) or numbers or ... ?

During our meeting on 5 June 2025 we decided to go for dockerizing both steps. Dockerized model will have less performance than non-dockerized, as we cannot use OpenMPI etc., but if we reduce spatial and temporal resolution and timespan, it will be ok for a demo.

Compilation: I think we agreed to start with a pre-compiled model. Whether we'll try to provide compilation-on-request, let's see (need to check security aspects, but if we do it inside a container and strictly restrict what users can pass as input, maybe it can be ok. Tbd).

Status and next steps

Currently the docker image for SEAWAT fails to build, some file is missing in the repo! @ Sarah @ Joel

• Next meeting: Thu 26 June, 11 AM CEST in IGB's AquaINFRA zoom channel
• Next meeting: Wed 16 Juli, 11 AM CEST in Google Meet channel provided by Joel
• Sarah and Joel to be added to this repo and assigned to this issue
• TODOs until then, mitgcm side (let's see how far we get):
  • Joel: Try dockerize mitgcm, simple case.
  • Merret: Provide Sarah+Joel with GitHub repo that contains the work done with Savitri (mitgcm fork and postprocessing repo)
• TODO until then, SEAWAT side (let's see how far we get)
  • Sarah: Provide Merret with GitHub repo with the SEAWAT work (Joel's Dockerfile, ...)
  • Merret to commit her changes to the Dockerfile, so Joel can then add user params.
  • Sarah/Joel to update repo so that the Docker image build does not fail anymore
  • Merret to commit the pygeoapi python stuff to the repo too
  • Sarah/Joel: to work on user parameters (so users can change what happens in SEAWAT)
  • Joel to include those user params in the entrypoint/dockerfile (ask Merret in case of questions)
  • Sarah/Joel: To somehow make the example run quicker for testing (so far takes 2.5 - 3 hrs for one test run...)
  • Sarah/Joel: Would it be possible to write the outputs that you want the user to receive into a directory called "/out" inside the container? We can use "/app/malta_simulation", but if we use "/out" we can reuse standard code which we've been using in all the use cases so far.
  • Merret to add the user params to the process
  • Sarah/Joel to test these

Done

• On May 2nd, Joel sent the Dockerfile:

The Docker image for the Malta Ground Water Model is found here: [...]
Once the Docker image is setup, one needs to:
cd /app/SCRIPTS
./run_model.sh
The results appear in /app/malta_simulation

• On May 23rd, Merret modified so that the run_model.sh gets run from the entrypoint, and added the whole thing to the AquaINFRA server as a first demo version:

[...] I already put the current version on the server, to test out a few things, and also to show you how this will look eventually. I had to adapt the Dockerfile a little to add a CMD/ENTRYPOINT that our server can run, and I added a python file (to allow our server calling the docker) and a JSON file (containing the descriptions of the model and the parameters). If you want to try it out, see below. But running it on our dev server takes between 2 hrs and 3.5 hrs, so there is no quick testing

URLs

URL to input datasets: ???
URL to GitHub:

• SEAWAT: https://github.com/sarahfiala/Malta_Groundwater

Commands for running R/Python scripts

• SEAWAT: See README.md in https://github.com/sarahfiala/Malta_Groundwater
• MITgcm: TODO

Commands for running Docker commands:

• SEAWAT: See README.md in https://github.com/sarahfiala/Malta_Groundwater
• MITgcm: TODO

cURL Commands for running OGC API Processes:

• SEAWAT: See README.md in https://github.com/sarahfiala/Malta_Groundwater
• MITgcm: TODO

Checklist

• Assign all involved people:
  • Savitri
  • Sarah
  • Merret
  • Joel
  • Markus
  • Quim??
• Input data is identified and/or accessible via DDAS
  • Input data is made accessible under an open license
• Analysis code is available on GitHub
  • Analysis code is made accessible under an open license
  • Dependency's license is verified - are we allowed to provide this as a service?
    • SEAWAT/iMOD/MOFLOW dependency (Deltares or USGS?)
    • MITgcm dependency
• Workflow is sketched (see Oulu presentation by Sarah - maybe extend with file formats / user params)
• Analysis is split into reusable functions
  • Input parameters are identified
  • functions follow input - processing - output - logic
  • input data can be passes via URL
  • output data is stored in file, e.g., csv, json etc.
• Reusable functions are saved in separate files
• Runtime, libraries, and library versions identified
• R commands are added to README
• Analysis is dockerized
• Docker commands are added to README
• OGC API Processes created locally
  • python scripts created
  • titles and descriptions for input parameters added to json
• OGC API Processes deployed on server
• Tests successful
• Galaxy tool created locally
• Galaxy tool deployed on server
• Tests successful
• Workflow is complete

[sarahfiala]

@merretbuurman @SavitriGaliana Joel requires the script and any additional material in order to dockerize the MITgcm model. Could you please provide him with this. Emails have been sent but no reply.

[SavitriGaliana]

@sarahfiala I thought the e-mail of Joel was answered, I am sorry. I think I do not have access to the github where the are the scripts for the execution of the MITgcm as a pygeopai process. If I understood it correctly, this is what we wanted to dockerize for now. @merretbuurman could you please confirm what I am saying and provide access to this github to Joel.

[merretbuurman]

Hi @sarahfiala (I guess Joel is not on this issue yet?),
the scripts for executing MITgcm as a pygeoapi process are in Savitri's repo at CSIC:
https://github.com/SavitriGaliana/MITgcm_CATSEA_chain/blob/dev/pygeoapi_processes/mitgcm_process.py

FORTRAN model as a process

The pygeoapi process is defined in this file:
https://github.com/SavitriGaliana/MITgcm_CATSEA_chain/blob/dev/pygeoapi_processes/mitgcm_process.py

Basically, two things happen in there (which would have to happen inside the Docker):

• the FORTRAN model is called here: https://github.com/SavitriGaliana/MITgcm_CATSEA_chain/blob/dev/pygeoapi_processes/mitgcm_process.py#L96
• the postprocessing of the file happens (using the modified bin2netcdf script here) here: https://github.com/SavitriGaliana/MITgcm_CATSEA_chain/blob/dev/pygeoapi_processes/mitgcm_process.py#L152

NetCDF visualisation as a process

The pygeoapi process is defined in this file:
https://github.com/SavitriGaliana/MITgcm_CATSEA_chain/blob/dev/pygeoapi_processes/mitgcm_plotting.py

This one only contains python code and is already dockerized, which might provide some insight how dockers are called from inside processes.

---

Old: Baroclinic Gyre (MITgcm Tutorial) as a process

(This was my answer from some minutes ago, but it described the first test, the process running the baroclinic gyre tutorial. In case that's of any help, I'll leave it here...)

Hi, the scripts for executing MITgcm as a pygeoapi process are here - I think you all should have access? https://github.com/AquaINFRA/MITgcm/tree/aquabranch (it is a fork of the original MITgcm git repo, as it tooks scripts from them (gendata.py, gluemncbig) and modified them.

The pygeoapi process is defined in this file:
https://github.com/AquaINFRA/MITgcm/blob/aquabranch/pygeoapi_processes/mitgcm_baroclinic_gyre.py

Basically, three things happen in there (which would have to happen inside the Docker):

• the modified gendata.py (this) is called in this line: https://github.com/AquaINFRA/MITgcm/blob/aquabranch/pygeoapi_processes/mitgcm_baroclinic_gyre.py#L143
• then the FORTRAN model is called in this line: https://github.com/AquaINFRA/MITgcm/blob/aquabranch/pygeoapi_processes/mitgcm_baroclinic_gyre.py#L155
• and then the results are glued together using the modified gluemncbig bash script (this), in this line: https://github.com/AquaINFRA/MITgcm/blob/aquabranch/pygeoapi_processes/mitgcm_baroclinic_gyre.py#L211

[merretbuurman]

Need to check the licenses to see whether we are allowed to expose functionality. Or replace by that other modelling code that Sarah/Joel mentioned: https://content.oss.deltares.nl/imod/iMOD_Software_License_Agreement.pdf

[merretbuurman]

Decisions/outcome from zoom call:

• Joel will provide dockerized groundwater model and dockerized MITgcm by (latest) 8 August. If something is ready earlier, it would be welcome.
• As downloading data from Copernicus from inside the services is not possible due to that login, we will proceed with pre-downloaded demo data (or users download their own data, put it on some platform that provides a publicly accessible link, and uses that link).

[joelazz]

The SeaWAT groundwater dockerised model is available here
As input, one can pass the sea level values, Interval duration in years between each sea level in the list and recharge values
It produces an output netcdf file - /out/salt_flow.nc
This only uses freeware programs available directly from USGS (i.e. no software that requires registration)

The Dockerised MitGCM setup for the Spanish coast is found here
As input, it receives the simulation start date as YYYY-MM-DD, and a Google Drive link to a publicly accessible zip drive that contains the input files. The Readme on the git contains example parameters that can be used
As output, it produces a zip containing a services of netcdf files: /out/MitGCMoutput.zip

[merretbuurman]

Groundwater Model (status, update)

• GitHub: https://github.com/joelazz/aquainfra_MaltaGW (I pushed my updates to the main branch - sorry, I should have made a Pull Request first - but I don't think the stuff will interfere with anything).
• Process Description: https://aquainfra.ogc.igb-berlin.de/pygeoapi-dev/processes/malta-groundwater

Status

The Groundwater model is running as a docker, and as a pygeoapi process on the AquaINFRA dev server. It takes three optional parameters. It takes 20-30 minutes to finish.

• Dockerized by Joel
• First version of process is running

Questions, next steps

• @sarahfiala Please test the tool as it is now, thanks!
• @sarahfiala Does the output NetCDF file look all right?
• @sarahfiala Do the users need to get the log file / command line output too?
• @sarahfiala Working on the visualisation tool (either extra tool, or included in model run)
• Easy task @sarahfiala : We need the Process Description, i.e. title+descriptions for all input parameters, for the output, and for the tool itself! (See details below)
• @merretbuurman Write a README about the pygeoapi part
• @merretbuurman Eventually push to prod instance (currently running on /pygeoapi-dev).

Some useful info

Curl request

curl -i -X POST "https://${PYSERVER}/processes/malta-groundwater/execution" --header "Content-Type: application/json" --header "Prefer: respond-async" --data '{
  "inputs": {
    "sealevel_int": 250,
    "user_recharge": 0.002 
  }                                     
}'

How-To for asynchronous requests: https://nimbus.igb-berlin.de/index.php/apps/files/files/12371371?dir=/AquaINFRA_public_docs&editing=false&openfile=true

Process description

See the placeholders here: https://aquainfra.ogc.igb-berlin.de/pygeoapi-dev/processes/malta-groundwater
You could maybe directly modify the JSON file: https://github.com/joelazz/aquainfra_MaltaGW/blob/main/pygeoapi_processes/groundwater.json

Duration

Two example runs:

2025-11-07 08:48:03,538 - pygeoapi.process.aquainfra_MaltaGW.pygeoapi_processes.groundwater:203 - DEBUG - Start running docker container (image: maltagw:20251106, name: maltagw_78e3d9ed-bbb6-11f0-91d1-fa163e42fba0)
...
2025-11-07 09:09:45,301 - pygeoapi.process.aquainfra_MaltaGW.pygeoapi_processes.groundwater:207 - DEBUG - Finished running docker container (image: maltagw:20251106, name: maltagw_945e89e1-bbb5-11f0-b90d-fa163e42fba0)

2025-11-10 14:24:41,221 - pygeoapi.process.aquainfra_MaltaGW.pygeoapi_processes.groundwater:207 - DEBUG - Start running docker container (image: maltagw:20251106, name: maltagw_fedcef38-be40-11f0-9206-fa163e42fba0)
...
2025-11-10 14:52:18,618 - pygeoapi.process.aquainfra_MaltaGW.pygeoapi_processes.groundwater:212 - DEBUG - Finished running docker container (image: maltagw:20251106, name: maltagw_fedcef38-be40-11f0-9206-fa163e42fba0)

[merretbuurman]

Hi, update today:

• I enabled running the pygeopi process without any input params, as requested. In that case, the defaults are used (by the bash script that calls the model).
• And I sent the output of the process/docker without any input params to @sarahfiala for her to verify, as the previous NetCDF was generated using the example values from te README, so it could not be used for comparison.