Skip to content

Add restart capability to the fire behavior model - #39

Merged
mefrediani merged 13 commits into
NCAR:developfrom
masih-e:restart-pr
Oct 21, 2026
Merged

mefrediani merged 13 commits into
NCAR:developfrom
masih-e:restart-pr

Conversation

@masih-e

@masih-e masih-e commented Aug 26, 2026 •

Copy link
Copy Markdown
Collaborator

Description of changes:

The CFBM has had no way to stop a simulation and resume it.
This feature adds a restart capability. A run can write its dynamic fire state to a
NetCDF file at a fixed interval, and a later run can be initialized from one of
those files instead of a cold start. It covers standalone runs, serial and MPI,
and runs coupled to WRF.

Two options are added to the &time namelist block:

Option Type Default Meaning
restart logical .false. Resume from fire_restart_<start datetime>.nc
restart_interval integer -1 Restart write interval in seconds; -1 disables writes

Both default to the previous behavior, so existing namelists are unaffected.

Changes by file:

  • state/state_mod.F90 carries the Read_restart and Write_restart
    type-bound methods, Build_restart_file_name, and the MPI helpers they need.
    A restart read validates the grid metadata stored in the file against the
    current configuration before touching any state, so a mismatched restart file
    fails early with a message instead of producing a silently wrong run.
  • io/netcdf_mod.F90 adds attribute writers (Add_netcdf_att) and
    MPI-aware readers and writers for distributed 2D and 3D fields.
  • driver/initialize_mod.F90 routes the real-case restart path through a
    new Init_fire_state_from_restart, leaving the existing geogrid
    initialization untouched. The WRF-coupled path reads the restart after the
    fire components are initialized.
  • driver/fire_behavior.F90 reads the restart state at startup in place of
    Save_state, and writes a restart every restart_interval seconds.
    restart_interval must map to a whole number of time steps or the run stops
    with a message.
  • io/namelist_mod.F90 adds the two options, their broadcast, and a range
    check.
  • physics/fmc_wrffire_mod.F90 drops its unused state_mod and
    namelist_mod imports. This one is load-bearing rather than cosmetic:
    state_mod now uses fmc_wrffire_mod for the moisture restart fields, so
    leaving those imports in place would create a circular module dependency.

Type of change

  • New feature (non-breaking change which adds functionality)

Tests conducted:

All testing on Derecho (NCAR) using the repository environment
env/derecho/gnu-12.2.0: GCC 12.2.0, cray-mpich 8.1.25, NetCDF-Fortran 4.9.2,
HDF5 1.12.2, ESMF 8.5.0, CMake 3.26.3.

Builds. Three configurations, all clean with no errors or new warnings:
serial (--mpi-off), MPI (default), and --esmx (which includes NUOPC).

Registered ctest suite, run on the --esmx build so every registered case
is reachable:

test7 test8 test7esmf test8esmf test7esmx test8esmx testx
pass pass pass pass pass pass pass

7 of 7. The NUOPC cases matter here because the cap uses both initialize_mod
and state_mod, and this PR touches both.

Restart regressions. These run a case straight through, then run it again
stopping and restarting partway, and compare the two. Restart test scripts are
not included in this PR (see below), but the results are:

Serial build, 20 of 20 checks:

test7_restart test8_restart test9 test9_restart
5/5 5/5 5/5 5/5

MPI build on a compute node, 30 of 30 checks:

ranks test7_restart test8_restart
1 5/5 5/5
2 5/5 5/5
4 5/5 5/5

The multi-rank runs are the ones that exercise the distributed restart read and
write and the halo exchange following a restart read. Restarting at 2 and 4
ranks reproduces the continuous run, so the domain decomposition is handled
correctly on the restart path.

57 passing checks in total across the three build configurations.

Not covered. Two gaps, stated explicitly:

  1. The WRF-coupled restart path (Init_fire_state_within_wrf calling
    Read_restart) is exercised by a separate WRF harness, and not tested here.
  2. Rank counts above 4 have not been tried.

On the test scripts. The restart test scripts (test7_restart.s,
test8_restart.s, test9.s, test9_restart.s, the test9 case directory, and
the tests/CMakeLists.txt registration) are deliberately held out of this PR
pending the separate discussion about how tests should be organized. They are
ready and can be added here or in a follow-up PR, whichever is preferred.

Documentation:

  • Documentation has been updated for these changes
  • No documentation update needed

Documentation was left out to keep the diff focused on
the code. Happy to add it to this PR if preferred, otherwise it will follow
separately.

Contributors:

@masih-e (Masih Eghdami:masih@ucar.edu)
@mefrediani (Maria Frediani:frediani@ucar.edu)
@pedro-jm (Pedro Jimenez:jimenez@ucar.edu; reviewer)

LLM/coding assistant technology used: the restart implementation was developed
with assistance from Codex GPT-5.5 and Claude Opus 4.7. Claude Opus 5 assisted
with the pre-review cleanup of this branch.

Further information:

state/state_mod.F90 grows from 1063 to 1965 lines in this PR, which is more
than that file should carry. A restart-specific module is worth doing as a follow-up. This
change is not possible with the current code because the restart calls depend on the fire
state type.

masih-e and others added 2 commits August 26, 2026 13:57
Lets a simulation write its dynamic fire state to a NetCDF restart file
at a fixed interval and resume from one of those files, for standalone
runs (serial and MPI) and for runs coupled to WRF.

Namelist (time block):
  restart          - resume from fire_restart_<start datetime>.nc
  restart_interval - restart write interval in seconds, -1 disables

state_mod:
  Read_restart and Write_restart type-bound methods, plus
  Build_restart_file_name and the MPI helpers they need. A restart read
  validates the grid metadata stored in the file against the current
  configuration before touching any state, so a mismatched restart file
  fails early instead of producing a silently wrong run.

netcdf_mod:
  Attribute writers (Add_netcdf_att) and MPI-aware readers and writers
  for distributed 2D and 3D fields.

initialize_mod:
  Init_fire_state_from_restart takes the real-case restart path, so the
  existing geogrid initialization is left as is. The WRF-coupled path
  reads the restart after the fire components are initialized.

fire_behavior:
  Reads the restart state at startup instead of Save_state, and writes a
  restart every restart_interval seconds. restart_interval must map to a
  whole number of time steps or the run stops with a message.

fmc_wrffire_mod:
  Drops the unused state_mod and namelist_mod imports. state_mod now uses
  fmc_wrffire_mod for the moisture restart fields, so these imports would
  otherwise form a circular module dependency.

The code was created with assistance from Codex GPT-5.5 and Claude Opus 4.7.

Co-authored-by: mefrediani <frediani@ucar.edu>
Move restart-specific state code and helpers into a new
state_restart_submod.F90 submodule.
@masih-e

masih-e commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Updated the PR with the state restart refactor. Restart-related implementations were moved out of state_mod.F90 into a state submodule without changing the public state_mod interfaces or restart logic.

Move Distribute_global_var2d out of netcdf_mod and into mpi_mod so the
domain-distribution helper lives with the other MPI utilities. Keep the
restart-specific variable names at the restart read call sites, but use
generic naming and messages in the MPI helper itself.
Move standalone restart metadata initialization into state initialization and
remove the initialize_mod restart wrapper.

Move restart output interval validation and write timing into
state_fire_t%Handle_restart so the main driver loop stays focused on
high-level model actions.
Move the standalone restart read/save decision into Init_fire_state so the
main fire_behavior driver stays focused on top-level model control.

Move the per-step model-clock consistency check into
state_fire_t%Check_model_clock. Keep it separate from restart scheduling
because the clock check is a general model consistency check, not restart-only
logic.
@masih-e

masih-e commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator Author

Updated with review cleanups:

  • moved generic 2D distribution helper into mpi_mod
  • moved standalone restart initialization/read-save flow out of the main driver
  • moved restart output scheduling into state_fire_t%Handle_restart
  • moved model clock consistency check into state_fire_t%Check_model_clock

pedro-jm
pedro-jm previously approved these changes Oct 2, 2026
@mefrediani

Copy link
Copy Markdown
Collaborator

@masih-e I checked the restart branch against the changes we've made recently for the regression. This is what I found:

For the current conflict, you can accept both (keep the comment and the call to check_model_clock)

However, I found two substantive interactions with the recent merges:

  1. Observed-perimeter activation is not restored. The new fire_perimeter_ignited flag defaults to false, and Add restart capability to the fire behavior model #39 neither saves nor restores it. After restarting an already ignited perimeter case, the first advance would reinstall lfn_hist, overwriting the
    evolved front. That activation state must survive restart. Activation logic (

    ! Install an aligned perimeter at the inclusive interval end. It begins
    ! propagating next step; zero-time perimeters were installed at initialization.
    ! Mid-step activation would age fuel without proportional propagation.
    ! Check_nml rejects it pending a separate review of split advances.
    if (config_flags%fire_is_real_perim .and. .not. grid%fire_perimeter_ignited) then
    ignition_time = anint (config_flags%fire_ignition_start_time1 / grid%dt) * grid%dt
    if (ignition_time <= time_end) call grid%Ignite_fire_perimeter (ignition_time)
    ).

  2. Restart I/O has incompatible fuel-array types. Recent changes made fuel_frac and fuel_frac_burnt_dt REAL64, but Add restart capability to the fire behavior model #39’s restart read, write, and halo helpers still accept default real. The combined source therefore has argument-kind mismatches. Restart
    storage must also preserve double precision; converting to single precision would discard the accuracy that the recent fix introduced. Current declarations
    (

    ! Retain small consumption increments when subtracting nearly-one fractions.
    real(REAL64), dimension(:, :), allocatable :: fuel_frac ! "fuel remaining" "1"
    real, dimension(:, :), allocatable :: fire_area ! "fraction of cell area on fire" "1"
    real(REAL64), dimension(:, :), allocatable :: fuel_frac_burnt_dt ! "fraction of fuel burnt on current dt" "-"
    ), restart reader
    (
    subroutine Read_restart_field (file_name, var_name, var)
    implicit none
    character (len = *), intent (in) :: file_name, var_name
    real, dimension(this%ifms:this%ifme, this%jfms:this%jfme), intent (in out) :: var
    call Get_netcdf_var_mpi (file_name, this%cfbm_comm, this%nx, this%ny, &
    this%ifps, this%ifpe, this%jfps, this%jfpe, var_name, var(this%ifps:this%ifpe, this%jfps:this%jfpe))
    end subroutine Read_restart_field
    ).

Beyond the two restart defects identified earlier, there are configuration and numerical changes to account for:

Change Consequence for restart
[#47: timing and perimeter initialization](#47) Preserve the corrected interval calculation and restore fire_perimeter_ignited. New namelist checks also apply to restart runs: valid ignition configuration, timestep-aligned perimeter activation, and interval_atm equal to an integer multiple of dt.
[#56: fuel and coordinate precision](#56) Restart read/write and MPI helpers need to preserve the two REAL64 fuel arrays. The new projection coefficients are reconstructed by the projection constructor, so they do not require additional checkpoint fields.
[#57: standalone wind interpolation](#57) Atmospheric fields now follow the revised horizontal/vertical interpolation order. Restart already saves uf, vf, and roughness; I found no additional missing wind-state field. Continuity should nevertheless be checked across the first atmospheric update after restart.
[#49: interpolation precision, wind bounds, Strict builds, harness](#49) Preserve these fixes and compare continuous/restarted runs using the same build configuration. The current regression matrix explicitly excludes restart continuity.
[#55: NUOPC wind imports](#55) Changes coupled wind mapping; I found no additional standalone restart-file requirement.

Thus, the concrete source incompatibilities remain the missing perimeter flag and the fuel-array precision mismatch. The other changes require appropriate configuration and validation.

Write and read fire_perimeter_ignited as an integer restart attribute and
broadcast it on every rank.
Read and write fuel_frac and fuel_frac_burnt_dt as NetCDF doubles.
Halo exchange for the variables as doubles.
@mefrediani
mefrediani merged commit ead119e into NCAR:develop Oct 21, 2026
2 checks passed
mefrediani added a commit that referenced this pull request Oct 22, 2026
With `fmoist_freq > 1`, the first scheduled moisture update could miss
the step-1 initialization, leaving class moisture and forcing history
undefined. Initialize once from the first valid atmospheric state
independently of cadence, apply zero elapsed moisture time during
initialization, and restore that state on restart.

For example, with `dt=4` and `fmoist_freq=3`, initialization is at 4 s,
the update at 12 s uses 8 s elapsed time, and later updates use 12 s
intervals. Time-based updates retain their existing phase. Invalid or
unavailable current/previous forcing fails explicitly. The NUOPC
rainfall accumulator is initialized before its first import.

Checkpoints now record initialization status/time and moisture
activation/cadence. Loading restores the lifecycle and validates class
state, forcing history, and matching settings. Prescribed legacy
checkpoints remain readable. PR #39 time-based checkpoints are accepted
when their metadata establishes compatible initialized state; ambiguous
legacy step-based checkpoints fail with a specific diagnostic.
Configuration documentation and twelve focused lifecycle scenarios cover
the correction. No new namelist options, ordinary diagnostic fields, or
humidity-formula changes are included.

Refs #42. Its remaining extinction, controls, and diagnostic work
remains open.

### Review base and dependencies

- This draft targets `develop` at
`ead119eca6dfb576d181a5560273d70035f780a3`, which includes the squash
merge of #39. The original local restart merge `bc205bc` and upstream
`ead119e` have identical file trees but different ancestry.
- Head `a87b23dae1c8b4a282ed503202cbb34579f0536f` contains six commits
and six files, without historical restart commits in its commit range.
The final atmospheric defaults use the original patch bounds
(`ifps:ifpe`, `jfps:jfpe`) in `Set_vars_to_default`;
`state/state_mod.F90` now matches develop exactly and drops out of the
feature diff.
- The review follow-up sets the four atmospheric fill values in
`Set_vars_to_default`, initializes the precipitation-rate accumulator in
NUOPC `Realize` while preserving restored rainfall, and groups
external/intrinsic imports before CFBM modules. The component member
remains `initialized`; its NetCDF attribute remains `fmc_initialized`.
- P00 / #61 at `ed18f91` remains an explicit validation dependency
absent from this feature's ancestry. The remaining foundation merge
order is this PR, then #61. Update and validate P00 separately before
its merge.
- The earlier replayed implementation `dec7eb2` was byte-for-byte
identical to validated `3e472d8`. Initialization-placement validation
used separate local integration
`d144a898a436efb7ef411c781368e6853c0415b3`, combining P00 and P02 at
`439474f`; all seven feature files matched that revision exactly. The
subsequent patch-bound restoration changes only the four atmospheric
default assignments. Original branches and validation evidence remain
preserved.

### Validation

Earlier validation at combined revision `554d150` used GNU Strict builds
on Derecho with GNU 12.2.0, Cray MPICH 8.1.25, NetCDF 4.9.2, and ESMF
8.5.0. Serial, OpenMP, MPI, hybrid, and ordinary NUOPC/ESMX execution
passed.

- Twelve focused initialization/forcing/timing scenarios per build; 45
Python tests; formatting and diff checks passed.
- All 15 restart cases passed, including the five delayed-moisture cases
previously failing in P00. There were 65 continuous/segmented model runs
and 30 loader probes.
- All 320 file comparisons and 12 cross-layout comparisons had zero
maximum absolute error; 450 checkpoint checks passed. Loader probes
comprised five accepted schema-compatible resumes and 25 expected
rejections.
- Existing PR suite: 36 model runs, 40 execution comparisons, and 36
approved-reference comparisons passed. Largest cross-driver absolute
difference was `fgrnhfx = 0.125 W m-2`, consistent with the existing
baseline. No reference or tolerance changed.
- Original PR #39-era checkpoints generated with P00 were also checked:
prescribed/time-based resumes agreed exactly with corresponding P02
checkpoints, and ambiguous step-based state was rejected.

PBS jobs: `7890398.desched1`, `7890541.desched1`, `7890596.desched1`.
Full retained evidence:
`/glade/derecho/scratch/frediani/tmp/cfbm-P02-20261021T034953Z/`,
including `attempt-02/results/restart-z8yehdcl/summary.json`,
`attempt-02/results/pr-hw75p1af/summary.json`, and
`legacy-originals/summary.json`.

P01/P02/P04 combined acceptance remains pending, including shared clock
precision and fractional/long-duration timing. Direct WRF-host
execution/restart, NUOPC/ESMX restart, and transfer of checkpoints
between MPI layouts remain untested here. Ordinary NUOPC/ESMX execution
and comparisons among independent MPI layouts do not establish those
restart paths. GitHub CI must pass on the final PR revision before
merge; its normal PR suite does not run P00's optional restart suite.

### Initialization-placement follow-up validation

Feature `439474f` matches tested P00/P02 integration
`d144a898a436efb7ef411c781368e6853c0415b3` exactly. Four fresh GNU
Strict builds (serial, OpenMP, MPI/NUOPC/ESMX, hybrid/NUOPC/ESMX) passed
with the environment above.

- All 15 restart cases pass: 65 continuous/segmented runs, 30 loading
probes, 320 zero-error file comparisons (315 continuity and five
accepted legacy resumes), 12 zero-error cross-layout comparisons, and
450 checkpoint checks. The restart suite's 67 CTest invocations include
the twelve focused P02 lifecycle scenarios in each build.
- The existing 36-run PR matrix, 40 execution comparisons, and 36
approved-reference comparisons pass. Maximum cross-driver difference
remains `fgrnhfx = 0.125 W m-2` (maximum percentage error
`0.0006368705741347137%`), within unchanged tolerances.
- A connected, nonzero precipitation-rate ESMX_Data check passes with
one MPI rank, four MPI ranks, and four ranks by four OpenMP threads. All
33 output-file comparisons against the previously validated executable
are exact. Thirty accumulated-rain checks have maximum absolute error
`1.862645149230957e-09` in model accumulation units.
- Formatting and regression both passed in [GitHub workflow
39690399727](https://github.com/NCAR/fire_behavior/actions/runs/39690399727)
on `439474f`. Local formatting and whitespace checks pass. No approved
reference or regression tolerance changed.

PBS job `7900048.desched1` completed the builds and both suites
successfully, then exited 1 because the scratch rate-check script looked
for its completion marker in stdout instead of the ESMF log. After
correcting only that checker, job `7900164.desched1` reran the six
rate-check model executions and exited 0. Failed-attempt evidence and
original scripts remain preserved.

Evidence:
`/glade/derecho/scratch/frediani/tmp/cfbm-P02-init-review-20261021T225746Z/`,
including `attempt-01/results/restart-hl_i2yr_/summary.json`,
`attempt-01/results/pr-jpmyi5r2/summary.json`, and
`attempt-01/rate-probe-02/summary.json`. The combined P01/P04,
coupled-restart, and direct WRF-host validation gaps above remain.

### Patch-bound restoration

The four atmospheric defaults retain their original `ifps:ifpe,
jfps:jfpe` patch sections. The temporary explicit memory-bound spelling
in `f6a6206` is superseded by `a87b23d`. The final assignments match the
original expressions, and `state/state_mod.F90` matches develop exactly.
Defaults remain in `Set_vars_to_default`; the NUOPC cold-start rainfall
initialization in `Realize` is unchanged.

Whitespace checks pass. The detailed HPC results above are from
`439474f`; the optional restart suite has not been rerun after restoring
patch bounds. GitHub CI on `a87b23d` is pending.
mefrediani added a commit that referenced this pull request Oct 23, 2026
The unrestricted Rothermel moisture cubic produced nonzero reaction at
extinction and negative reaction above it, affecting `iboros`, base
spread and flame diagnostics before the later spread clamp. This change
sets moisture damping to exactly zero at or above extinction, preserves
the polynomial below extinction, and leaves physical moisture uncapped.
An explicit moisture threshold is retained because a lower clamp on the
polynomial alone can leave positive roundoff at exact extinction. The
moisture-damping comment identifies `etam`. The independent empirical
chaparral spread law remains unchanged. Refs #42; this addresses its
extinction scope only.

Validation follows the source and lifetime of each quantity:

- `Check_nml` requires finite, nonnegative prescribed `fuelmc_g`. Zero
is valid; units are water mass per dry fuel mass.
- `fuel_t%Check_fuels`, called once during Anderson initialization,
requires finite, positive `fuelmce` for burnable categories.
- Moisture initialization validates the fuel loads and fixed averaging
weights. The component then checks each evolving class after its update,
before averaging. Valid classes with valid weights do not require a
repeated check of the computed mean. Empty categories retain zero
weights.
- The restart reader checks effective and class moisture separately
through the FMC component. `Check_fmc` is a common type-bound procedure
of `fmc_t`, inherited by the moisture implementation. A source comment
explains why `Check_fmc` remains: `fmc_g` is read independently from the
checkpoint and used directly, and prescribed-moisture restarts have no
class state. This is an input check, including for supported legacy
files; an unchanged checkpoint from the same validated revision would
already satisfy it.

Computed-class checks execute after the OpenMP loop on the initial
thread, avoiding concurrent fatal diagnostics across tiles. Spatial
checks cover owned burnable cells. ROS owns the extinction calculation.
No namelist option or checkpoint schema is added; P02 initialization and
restart compatibility checks are retained. Full live/dead SB40
combustion and gridded-input readers remain separate work.

The review base is `develop` at `622e191`, including merged #39 and #62.
Head `af2ca03` contains a 12-file P03 diff. P00/#61 is a test dependency
only in separate integration `3b56260`; its commits are absent from this
feature branch. Production files, focused Fortran tests and
configuration documentation in that integration match the PR head
exactly.


Validation of `af2ca03` / integration `3b56260`:

- [GitHub workflow
39997883832](https://github.com/NCAR/fire_behavior/actions/runs/39997883832)
passes formatting and regression on the published head.
- Derecho PBS job `7922406` passes four GNU Strict builds: serial,
OpenMP, MPI and hybrid, including ordinary NUOPC/ESMX execution. All 35
focused scenarios pass in every build, covering extinction boundaries
and diagnostics, namelist/table inputs, class state, initialization
loads, and independently restored effective moisture. The bound checker
is also exercised without initialized class arrays, matching
prescribed-moisture restarts.
- All 378 PR CTests, 36 model runs, 40 execution comparisons and 36
approved-reference comparisons pass. Maximum reference error is zero;
maximum execution difference is 0.125 W m-2 in `fgrnhfx`, within the
existing tolerance.
- All 15 restart cases and 30 loading probes pass. The 315 continuity
file comparisons have zero error; 450 checkpoint checks and 12
independent layout comparisons also pass.
- An additional 22 checkpoint mutation/control probes pass in serial and
hybrid configurations. These include invalid `fmc_g` with unchanged
valid classes, invalid classes, and accepted zero/wet states.
- A separate Fortran probe with the GNU Strict arithmetic flags confirms
that, at a moisture ratio of one, the polynomial and `max(etam, 0.0)`
both return `2.384185791015625e-7` in single precision. This small
positive residual explains retaining the explicit extinction condition.
The below-extinction arithmetic is unchanged.

No reference or tolerance was changed. Direct WRF-host
execution/restart, NUOPC/ESMX restart, cross-layout checkpoint transfer
and combined P01/P04 acceptance remain untested. Legacy readability does
not establish numerical equivalence across the extinction correction.
Evidence, exact commands, source snapshots and PBS logs are retained
under
`/glade/derecho/scratch/frediani/tmp/cfbm-P03-type-bound-review-20261023T174535Z`.
mefrediani added a commit that referenced this pull request Oct 23, 2026
Adds continuous-versus-restarted regression coverage to the existing
harness and runs it in GitHub CI alongside the existing PR suite. The
cases exercise prescribed and evolving moisture, delayed moisture
updates, checkpoints around observed-perimeter activation, and
preservation of REAL64 fuel state. All 15 restart cases pass on the
merged P02 initialization correction, including the five cases that
failed on the original restart foundation.

## Scope and review base

- Base: `develop` at `622e191fac104e1a73ae99a769e5bd3dfee2946c`,
including merged #39 and #62.
- Head: `56b832037a8e73d198d395e1c194afbdb46ed205`.
- The 23-file diff contains the restart harness, tests/documentation,
suite help and CI selection. Production code and P02's focused unit
tests are identical to develop. No scientific tolerance or reference
changes.
- Merge `ce551ee` incorporates develop into the published P00 history.
The moisture component and restart-reader conflicts retain develop's
merged P02 implementation. This preserves the original P00 commits and
the common ancestor used by stacked #65.
- The harness includes the initialization metadata checks and 30
restart-loading probes previously validated in the P00/P02 integration.
These check supported legacy inference and explicit rejection of
ambiguous or damaged state.

CI builds serial, OpenMP, MPI and hybrid configurations once, then runs
both `pr` and `restart`. Both suites execute even if one fails; either
failure fails the regression check. The existing test-evidence artifact
retains both result directories. The approved `github-pr` baseline
remains unchanged, and optional candidate generation still uses only the
`pr` results. The configuration guide includes a restart YAML example
and explains suite selection, segment timing and comparison policy.

## Validation

Derecho PBS `7915414.desched1` exits zero after fresh serial, OpenMP,
MPI and hybrid GNU Strict builds. MPI/hybrid builds include NUOPC and
ESMX. Environment: GNU 12.2, Cray MPICH 8.1.25, NetCDF 4.9.2, ESMF 8.5,
shared NPL Python.

- All 67 restart CTests pass: 15 restart cases, 48 P02 lifecycle checks,
and four Python harness registrations.
- All 65 continuity model invocations and 30 restart-loading probes
complete as expected. The 315 continuity file comparisons plus five
accepted-legacy checkpoint comparisons are bitwise equal, with zero
maximum error.
- All 450 checkpoint validity checks and 12 cross-layout comparisons
pass; cross-layout error is zero.
- All 238 existing PR CTests pass, including 36 model runs. All 40
execution comparisons and 36 approved-reference comparisons pass.
Reference error is zero; maximum cross-driver error is 0.125 W m-2 in
`fgrnhfx`, within the unchanged tolerance.
- All 45 Python tests pass; YAPF 0.43.0, shell syntax and whitespace
checks pass.
- Local workflow checks confirm both suites execute and the final status
is correct when neither, either, or both suites fail. The documented
restart configuration matches the production YAML; its 28 s split
retains the next output at 36 s.
- [GitHub workflow
39970138677](https://github.com/NCAR/fire_behavior/actions/runs/39970138677)
passes on implementation revision `5de15ed`: formatting, all four
builds, all 238 PR CTests and all 67 restart CTests. Downloaded reports
confirm all 15 restart cases, 320 zero-error file comparisons and 30
loading probes pass in CI. Both suite summaries and eight JUnit reports
are retained in the uploaded test-evidence artifact; the 36
approved-reference comparisons also have zero error.

The PBS run used clean candidate `9a64864`, whose complete tree equals
merge `ce551ee`. Later `c8aeeaa` changes documentation, and `5de15ed`
changes CI selection and documentation; production code, model cases,
harness and approved baseline remain identical to the PBS-validated
revision. PBS evidence is preserved under
`/glade/derecho/scratch/frediani/tmp/cfbm-P00-develop-20261022T230324-0600/`;
CI-update evidence is under
`/glade/derecho/scratch/frediani/tmp/cfbm-P00-restart-ci-20261023T082618-0600/`.
Repository validation history retains the original failures and records
the successful PBS rerun.

Documentation-only follow-up `2c6b5b9` adds the [test-count
breakdown](https://github.com/NCAR/fire_behavior/blob/test-restart-continuity/tests/regression/README.md#how-test-counts-are-reported):
checkpoint validity, file comparisons, the per-build PR CTest table, and
restart invocation counts. Counts and links were checked against the
downloaded reports from workflow 39970138677. Its [CI run
39974535865](https://github.com/NCAR/fire_behavior/actions/runs/39974535865)
uses the unchanged workflow; no executable, harness, case, or baseline
changed.

Documentation follow-up `b2da936` replaces development labels in the
user-facing regression guides with descriptions of moisture
initialization, legacy restart metadata, and coverage limits.
`tests/AGENTS.md` requires this distinction and keeps development
provenance in validation records. Follow-up `56b8320` consolidates all
test guidance there, retains a pointer at `tests/regression/AGENTS.md`,
and requires informative unit-test comments describing the behavior,
setup and expected result. The historical validation file is unchanged.
Guide-label, relative-link, whitespace, and pinned YAPF checks pass.
Workflows 39974535865 and 39977209659 on the preceding documentation
revisions passed both formatting and regression checks.

## Coverage limits

Restart comparisons use the standalone driver and the same
revision/execution layout. NUOPC/ESMX integrations pass in the existing
PR suite, but coupled restart, native WRF-host restart, cross-layout
checkpoint transfer, and combined P01/P04 behavior remain separate
validation work. Synthetic legacy checkpoints test schema compatibility,
not numerical equivalence across scientific revisions. No production
correction for #42 is included in this test PR.

---------

Co-authored-by: masih <masih@ucar.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants