Add a test that compares the restart files against a known-good output

[manodeep]

This issue uncovered a case where even though the bitwise repro checks passed, the restart files were actually corrupted. Normally such a situation would be caught by a test that compares, say 1 run with 2 days against the output from 2 consecutive runs with 1 day each; however, ESM1.6 is known to be not restart-reproducible - so we don't have that test.

In absence of restart reproducibility, such a case would have been caught if we compared the restart at the end of 1-day (say checksum) against a known-good checksum that's committed into the relevant config-branch.

[anton-seaice]

Some options

1. Presumably its simplest, to just change the historical repro test to be two consecutive runs with a restart in the middle (and save the checksums from the end of the run)

2. It would be better to be saving two sets of checksums, one unchanged from the current historical test, and one for the restart tests.

3. (A third option would be to calculate checksums for all restart fields in the historical tests (rather than only the ocean fields), which also would catch this problem)

[aidanheerdegen]

Can we just use the MD5 hash in the manifest?

Related: #86

[anton-seaice]

In my daydreams we add metadata (like the UUID, or the author/rundate/git repo etc) to restart files which breaks that (we may already in some cases)

[aidanheerdegen]

In my daydreams we add metadata (like the UUID, or the author/rundate/git repo etc) to restart files which breaks that (we may already in some cases)

Hrm. Yeah real plusses and minuses territory there.

So we'd probably have to go down the route of adding variable level MD5 checksums to the metadata. netCDF4 supports HDF5 Fletcher 32 data integrity checksums, but they are calculated at the chunk level, so might be a bit of overkill for this use case.

[anton-seaice]

I got the impression that they were quite slow / overkill ? Although don't really remember now

The mom approach of calculating a global sum, or a global checksum for each restart field seems like the best balance to me

[manodeep]

@blimlim and I have uncovered yet another interesting bug with the AMIP config. One run produces bitwise identical results and restart files for the first year, but then diverges immediately upon startup for the second year. The difference seems to come from the land-use change code path, which only gets executed after the first year, i.e., we would need to run a 1 year experiment, followed by a restart to catch this bug.

May be a starting point would be to figure out whether the configs execute different code-paths depending on the simulated period.

[manodeep]

More details are here