Add repro payu setup check - this should pick up changes to manifests

[jo-basevi]

This PR adds a "repro_payu_setup" test marker, which selects a test that runs "payu setup" with "--reproduce" flag - This is equivalent to setting the following in config.yaml:

manifests:
  reproduce:
    inputs: true
    exe: true
    restart: true

So setup will raise an error if any full hashes in the generated manifests do not match.

Tested this with running model-config-tests -m repro_payu_setup on a configuration with unchanged manifests, and with a changed restart manifest (removed values from a binhash and md5), the error is:

E           Failed: Failed to run payu setup with --reproduce. Error: Loading access-esm1p6/2025.11.002
E             Loading requirement: cice5/access-esm1.6-2025.11.001-6umbsy5 mom5/2025.05.000-tzlv4hw um7/2025.11.000-nwpmrxg
E           Run cannot reproduce: manifest manifests/restart.yaml is not correct
E
E           Full output: laboratory path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab
E           binary path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/bin
E           input path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/input
E           work path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/work
E           archive path:  /scratch/tm70/jb4202/tmp/test-model-repro/lab/archive
E           Experiment name is configured in config.yaml:  dev-preindustrial+concentrations-repro_payu_setup
E           payu: Found modules in /opt/Modules/v4.3.0
E           Loading input manifest: manifests/input.yaml
E           Loading restart manifest: manifests/restart.yaml
E           Loading exe manifest: manifests/exe.yaml
E           Setting up atmosphere
E           Setting up ocean
E           Setting up ice
E           Setting up coupler
E           Checking exe, input and restart manifests
E           Manifest path: stored hash != calculated hash
E             work/atmosphere/restart_dump.astart: md5: f65a5a631c516907330e7f2d4479d5e != f65a5a631c516907330e7f2d4479d5e2

An issue is that Payu raises an error when restart manifests are empty, e.g. ACCESS-OM2:

 Failed: Failed to run payu setup with --reproduce. Error: Restart manifest must exist and be populated if reproduce is configured to True

I wonder if payu could not error here and only error if there was no pre-existing manifest and a new restart manifest was generated?

Related to #9

[codecov]

Codecov Report

❌ Patch coverage is 80.35714% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.43%. Comparing base (b881d91) to head (f277c4c).
⚠️ Report is 18 commits behind head on main.

Files with missing lines	Patch %	Lines
...fig_tests/config_tests/test_bit_reproducibility.py	44.44%	10 Missing ⚠️
src/model_config_tests/exp_test_helper.py	97.14%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #190       +/-   ##
===========================================
+ Coverage   68.39%   85.43%   +17.04%     
===========================================
  Files          14       22        +8     
  Lines         829     1298      +469     
===========================================
+ Hits          567     1109      +542     
+ Misses        262      189       -73     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jo-basevi]

Thinking about this more, it could just check if any manifests/input.yaml, manifests/restart.yaml, manifests/exe.yaml had changed..

Originally I thought it is not a good idea as it'll pick up changes to binhashes which can change even if the file is exactly the same. But I think it'll still be useful to update the binhashes on the configurations as it can lead to longer payu setup time in future, as it needs to recalculate the md5 hashes.

[Qian-HuiChen]

Thanks @jo-basevi for the previous work and suggestions!
I change the setup_reproduce to run payu setup (no --reproduce flag) and then check git diff. If the output is not empty, then returns an error.

In this case, empty manifests will not raise an issue.

Also test model-config-tests -m repro_payu_setup on unchanged manifests --> pass.
I changed manifests on git and run model-config-tests -m repro_payu_setup again, an error is raised as expected:

Manifests have been modified. The modified files include: restart.yaml

I am not sure if this is what we want. Happy to chat and change it.

[jo-basevi]

Thanks for adding this change in!!

Thinking about this more, I am wondering whether this should be two tests:

test_setup_reproduce_manifests

• Uses payu setup --repro (will need to fix payu failing on empty manifests first...). This will pick up if md5 hashes have changed (so the files are different).

test_setup_unchanged_manifests

• Catches if paths or binhashes have changed using the git diff logic you have added. If the md5's are the same, it will just need the manifests to be updated with payu setup. This wouldn't be as large of an issue as the configurations are still using the same files but the initial payu setup would take longer as it needs to recalculate all the md5 hashes.

For scheduled tests for tags, only test_setup_reproduce_manifests should run, as if it fails, it will indicate something on NCI has changed.
But for PRs or scheduled tests for branches, both manifest tests should run so the manifests are always kept up to date.

[jo-basevi]

If there are exceptions or sys.exit(1) should result in non-zero code. If there is a zero code, and there's an "error" in setup_result.stderr.lower(), I wonder if it would've coming from a warning or similar but not an error that would prevent the payu run. Probably fine to leave this, but I'm not 100% sure we should fail the test for that case.

[Qian-HuiChen]

Good point. I agree that we shouldn't let exception fail the test. In the latest commit, the code will

• raise error when payu setup returns non-zero code,
• print a warning when error is in stderr and payu setup exits with zero.

[jo-basevi]

Could get an index error if len(diff_lines) < 12 here

I am wondering whether it should just show the full git diff here? If it did show the full error, it should show the all the names of modified files at the top.

Changes detected in manifest file(s):
 - manifests/input.yml

If md5 hashes have changed, this indicates file contents being different.
If binhashes/paths have changed but md5's are the same, this will mean the configuration can reproduce the manifests but `payu setup` will take longer to run as it needs to re-calculate all the md5 hashes. 

Full git diff:
...

Also if you prefer the truncated sections, feel free to keep it too! Could start with truncated diff then if someone asks for the full diff, then it'll be easy enough to add it later

[Qian-HuiChen]

Could get an index error if len(diff_lines) < 12 here

I asked myself the same question before. So I tested the top_lines = "\n".join(diff_lines[2:12]) when diff_lines are less than 12 rows. It reads in all lines without raising error.
I even took an extreme test top_lines = "\n".join(diff_lines[2:300]). It reads in all output of git diff, no add-in empty lines and still didn't raise error. IndexError should not be an issue in this case.

I do agree that it should list all file names at the top, and then details of each file underneath. This is implemented in the latest commit.

[jo-basevi]

This git diff may run from a different directory, so would be safer to use git -C <path/to/repo> e.g.

"git diff -C {self.control_path} {file}"

[Qian-HuiChen]

Thanks for pointing this out! Good point. It is implemented in the latest commit.

[Qian-HuiChen]

For scheduled tests for tags, only test_setup_reproduce_manifests should run, as if it fails, it will indicate something on NCI has changed.
But for PRs or scheduled tests for branches, both manifest tests should run so the manifests are always kept up to date.

I agree that it is good to split test_setup_reproduce_manifests and test_setup_unchanged_manifests for different usages and triggered differently. I have opened a payu issue to solve the empty manifest handling issue first, and then come back to modify the model-config-tests.