Transition AM3 CI work into model-config-tests

[jo-basevi]

Investigate how to generalise existing codes to accommodate payu/cylc/non-cylc models in the existing framework.

[jo-basevi]

I've put some thoughts below, though please let me know if I need to add more context as I ended up writing about a lot of things..

CI with model build

From looking through the existing tests in am3_ci repo, the tests are a bit different from the current CI in this repository. It modifies a suite to use a particular path of the model source repositories that are cloned to NCI.

While the payu model configurations use a release/pre-release version of the model executables already built by spack and deployed to /g/data/vk83 on gadi. So I think a common workflow to test a payu model configuration once there is a prerelease deployment of a model is:

• Create a feature branch starting from a dev- development branch on the model configuration repository
• Manually update the model version used in the configuration (for payu this is modifying the model module version used in the config.yaml configuration file)
• Open a Pull Request to the dev- branch. As reproducibility tests are run automatically on PRs from dev-* -> release-* branches, triggering reproducibility tests requires !test repro comment on the PR from feature-branch -> dev-*.

There was some talk a while back about automatically modifying one model configuration branch that updates the model version to a prerelease version just built. This would open a draft PR and trigger the repro tests. This would provide a link from a Model Deployment Repository PR, (e.g. https://github.com/ACCESS-NRI/ACCESS-ESM1.6/pulls) to a model configuration repository PR (e.g. https://github.com/ACCESS-NRI/access-esm1.6-configs/pulls).

There are also scheduled tests that run repro tests on a release model configuration every month/week. If these fail, it creates an issue on the model configuration repository. These were mostly to check if anything on NCI had changed.

From my very limited understanding of rose+cylc suites, is the source model code are compiled/built at runtime? I also don't know how much of the model source code will be eventually built and deployed via a model deployment repository using the Build-CD workflows and how that will be configured in a suite.

Payu logic in CI in this repository

Currently, payu-version is referenced through several github CI workflows. This is because when the repro PR or scheduled tests are run, the payu module is used to create the test virtual environment and pip install the test package (also building an environment on top of a payu environments also allows the test virtual environments to be quite small):

model-config-tests/.github/workflows/test-repro.yml

Lines 142 to 149 in 6246f28

	# Load payu module
	if [ "${{ inputs.payu-version }}" == "dev" ]; then
	echo "::warning::Using the prerelease module payu/dev for testing"
	module use ${{ vars.PRERELEASE_MODULE_LOCATION }}
	else
	module use ${{ vars.MODULE_LOCATION }}
	fi
	module load payu/${{ inputs.payu-version }}

The payu-version is parsed from a CI configuration file (config/ci.json in the main branch of the model configuration repository - see README section). This file includes other configuration such as which tests to run and the version of tests themselves. This allows different tags/branches of a configuration repository to have tests run with different versions of payu.

I imagine python-version, model-config-tests-version and markers (e.g. markers are used to select tests in model-config-tests), would be useful for CI testing for cylc configurations. It would be useful to figure out if there's any other branch specific CI configuration required in ci.json, e.g would cylc-version ever be different for different configurations(?), e.g. https://github.com/ACCESS-NRI/am3_ci/blob/41376b393b7b8ad139c5f8526864b749c7023cfd/submit.sh#L10C13-L10C24. The payu module could still be used to get dependencies such as python and hpcpy when it's added to payu environment.

As the CI deals with versioning of configurations, it uses a metadata.yaml file to store/update the version in addition to creating tags, e.g.
https://github.com/ACCESS-NRI/access-esm1.6-configs/blob/2a33443c4a1c50f3f2b175fd7190bd2cf6fe8338/metadata.yaml#L27. The schema for these experiment metadata files are found here. Would it make sense for the suites have these metadata files or something similar?

The other thing that the CI does is store historical parsed checksums/output on the configuration, (e.g. https://github.com/ACCESS-NRI/access-om2-configs/blob/release-1deg_jra55_ryf/testing/checksum/historical-3hr-checksum.json). This is required for the historical reproducibility tests and versioning logic.

At a first guess of the changes required for CI, would be a rose/cylc version of the workflow that sets up a run and runs the repro tests - test-repro.yml and have some logic/configuration that determines if it's a rose/cylc configuration.
In model-config-tests repro pytests, it submits several different payu experiments at the start of the pytest call, and then waits on the login node, polling the PBS scheduler every minute or so until all the payu PBS jobs have completed, and then compares the output.
This is OK for repro tests currently as they aren't running for that long, though it may become an future issue for longer running experiments (see related issue #15). Would it be possible to wait for a suite run to complete and how long does it take for the suite to run?

The pytests in model-config-tests have a quite a bit of payu logic. If the pytests were used for submitting the suite runs, could have a class similar to ExpTestHelper but for cylc configurations rather than Payu. This class sets up an experiment, uses a model class to modify model runtimes, submits the experiment, and waits for it to complete. Then there are model class methods for extracting checksums/output based on the model used in the configuration.

Depending on how different the tests required are, we could fully separate the payu tests and cylc tests in different directories and have a separate entry-point test command. If the full configuration CI pipelines (e.g. config versioning, storing historical checksums) are not required initially, we could start with a separate custom scheduled Github workflow that just tests the suite runs.

For me, a positive of including rose+cylc suite tests in this repository is if there are improvements to CI workflows and tests, such as parsing of checksums and outputs, how the tests are configured and run, then these can be shared. If we do include rose+cylc tests and CI, and then there ends up being no cross over then maybe it does need to be in a separate repository and package to model-config-tests.

[bschroeter]

Meeting 08/05/2025

• Tests are different, rose/cylc is built at runtime rather than using executables.
• More discussion needed, Aidan back next week.
• More to follow after AM3 Spack build.

[aidanheerdegen]

There was some talk a while back about automatically modifying one model configuration branch that updates the model version to a prerelease version just built. This would open a draft PR and trigger the repro tests

It's something we want to work towards, but I don't think is a top priority for this initial work.

From my very limited understanding of rose+cylc suites, is the source model code are compiled/built at runtime? I also don't know how much of the model source code will be eventually built and deployed via a model deployment repository using the Build-CD workflows and how that will be configured in a suite.

We have modified the regional model to use spack built and deployed executables. The aim would be to do the same for AM3.

would cylc-version ever be different for different configurations

Yes we need to support this. In the past it has been necessary to "port" suites to newer versions of rose+cylc, so they aren't guaranteed to run with new versions, although that might only be major version changes, and it is also necessary to be able to update that version and check the suite still runs and produces the same answers.

Would it make sense for the suites have these metadata files or something similar?

Yes. I don't know if anything like this already exists, but yes we would, and if we're interfacing with the intake catalogue it definitely makes sense to create a file such as this, even if it duplicates some information that is already available in the suite output.

For me, a positive of including rose+cylc suite tests in this repository is if there are improvements to CI workflows and tests, such as parsing of checksums and outputs, how the tests are configured and run, then these can be shared.

Agreed. It reduces duplication and I think would reduce the support burden and is the preferred option if there is significant overlap in functionality.

[bschroeter]

Will try to add a rose/cylc approach into the model config tests.
Possible to use github API to create comment on the PR to raise if the test has failed.
Can possibly sit on a login node and wait, but this might chew up runner minutes.

Run length of one day should only take a few minutes. Queue time is the biggest factor but could run on express if needed.

@jo-basevi to investigate.

[aidanheerdegen]

Possible to use github API to create comment on the PR to raise if the test has failed.

This is certainly possible, but isn't considered necessary for the other CI as it runs fairly quickly and reports the result directly into the relevant PR.

Nightly or weekly "CI" that run longer tests could make use of this functionality however.

[bschroeter]

Blocked by spack build, updating issue relationships.

[bschroeter]

@jo-basevi, we have a semi-working spack build. How is this issue tracking?

[aidanheerdegen]

@jo-basevi, we have a semi-working spack build. How is this issue tracking?

There hasn't been any work on this as we waiting for a spack build to be available through the CD system.

Unfortunately we're now in the position where we don't have resources to port the AM3 tests in model-config-tests, so I am proposing an interim solution where we copy the existing am3ci into https://github.com/ACCESS-NRI/access-am3-configs, strip out the build part, modify the suites to use a pre-built spack exe and run whichever am3_ci test is appropriate for repro testing.

In the other config repos we're not running repro tests as automatic CI, except when doing PRs from dev- into release-. So we could run the repro tests as scheduled, like now, or/and using a ChatOps !test repro

Does that sound do-able @bschroeter? I have run this past @ccarouge and we were in broad agreement subject to consulting you.