Repository navigation
Add restart capability to the fire behavior model - #39
Conversation
Lets a simulation write its dynamic fire state to a NetCDF restart file at a fixed interval and resume from one of those files, for standalone runs (serial and MPI) and for runs coupled to WRF. Namelist (time block): restart - resume from fire_restart_<start datetime>.nc restart_interval - restart write interval in seconds, -1 disables state_mod: Read_restart and Write_restart type-bound methods, plus Build_restart_file_name and the MPI helpers they need. A restart read validates the grid metadata stored in the file against the current configuration before touching any state, so a mismatched restart file fails early instead of producing a silently wrong run. netcdf_mod: Attribute writers (Add_netcdf_att) and MPI-aware readers and writers for distributed 2D and 3D fields. initialize_mod: Init_fire_state_from_restart takes the real-case restart path, so the existing geogrid initialization is left as is. The WRF-coupled path reads the restart after the fire components are initialized. fire_behavior: Reads the restart state at startup instead of Save_state, and writes a restart every restart_interval seconds. restart_interval must map to a whole number of time steps or the run stops with a message. fmc_wrffire_mod: Drops the unused state_mod and namelist_mod imports. state_mod now uses fmc_wrffire_mod for the moisture restart fields, so these imports would otherwise form a circular module dependency. The code was created with assistance from Codex GPT-5.5 and Claude Opus 4.7. Co-authored-by: mefrediani <frediani@ucar.edu>
Move restart-specific state code and helpers into a new state_restart_submod.F90 submodule.
|
Updated the PR with the state restart refactor. Restart-related implementations were moved out of state_mod.F90 into a state submodule without changing the public state_mod interfaces or restart logic. |
Move Distribute_global_var2d out of netcdf_mod and into mpi_mod so the domain-distribution helper lives with the other MPI utilities. Keep the restart-specific variable names at the restart read call sites, but use generic naming and messages in the MPI helper itself.
Move standalone restart metadata initialization into state initialization and remove the initialize_mod restart wrapper. Move restart output interval validation and write timing into state_fire_t%Handle_restart so the main driver loop stays focused on high-level model actions.
Move the standalone restart read/save decision into Init_fire_state so the main fire_behavior driver stays focused on top-level model control. Move the per-step model-clock consistency check into state_fire_t%Check_model_clock. Keep it separate from restart scheduling because the clock check is a general model consistency check, not restart-only logic.
|
Updated with review cleanups:
|
|
@masih-e I checked the restart branch against the changes we've made recently for the regression. This is what I found: For the current conflict, you can accept both (keep the comment and the call to check_model_clock) However, I found two substantive interactions with the recent merges:
Beyond the two restart defects identified earlier, there are configuration and numerical changes to account for:
Thus, the concrete source incompatibilities remain the missing perimeter flag and the fuel-array precision mismatch. The other changes require appropriate configuration and validation. |
Write and read fire_perimeter_ignited as an integer restart attribute and broadcast it on every rank.
Read and write fuel_frac and fuel_frac_burnt_dt as NetCDF doubles. Halo exchange for the variables as doubles.
With `fmoist_freq > 1`, the first scheduled moisture update could miss the step-1 initialization, leaving class moisture and forcing history undefined. Initialize once from the first valid atmospheric state independently of cadence, apply zero elapsed moisture time during initialization, and restore that state on restart. For example, with `dt=4` and `fmoist_freq=3`, initialization is at 4 s, the update at 12 s uses 8 s elapsed time, and later updates use 12 s intervals. Time-based updates retain their existing phase. Invalid or unavailable current/previous forcing fails explicitly. The NUOPC rainfall accumulator is initialized before its first import. Checkpoints now record initialization status/time and moisture activation/cadence. Loading restores the lifecycle and validates class state, forcing history, and matching settings. Prescribed legacy checkpoints remain readable. PR #39 time-based checkpoints are accepted when their metadata establishes compatible initialized state; ambiguous legacy step-based checkpoints fail with a specific diagnostic. Configuration documentation and twelve focused lifecycle scenarios cover the correction. No new namelist options, ordinary diagnostic fields, or humidity-formula changes are included. Refs #42. Its remaining extinction, controls, and diagnostic work remains open. ### Review base and dependencies - This draft targets `develop` at `ead119eca6dfb576d181a5560273d70035f780a3`, which includes the squash merge of #39. The original local restart merge `bc205bc` and upstream `ead119e` have identical file trees but different ancestry. - Head `a87b23dae1c8b4a282ed503202cbb34579f0536f` contains six commits and six files, without historical restart commits in its commit range. The final atmospheric defaults use the original patch bounds (`ifps:ifpe`, `jfps:jfpe`) in `Set_vars_to_default`; `state/state_mod.F90` now matches develop exactly and drops out of the feature diff. - The review follow-up sets the four atmospheric fill values in `Set_vars_to_default`, initializes the precipitation-rate accumulator in NUOPC `Realize` while preserving restored rainfall, and groups external/intrinsic imports before CFBM modules. The component member remains `initialized`; its NetCDF attribute remains `fmc_initialized`. - P00 / #61 at `ed18f91` remains an explicit validation dependency absent from this feature's ancestry. The remaining foundation merge order is this PR, then #61. Update and validate P00 separately before its merge. - The earlier replayed implementation `dec7eb2` was byte-for-byte identical to validated `3e472d8`. Initialization-placement validation used separate local integration `d144a898a436efb7ef411c781368e6853c0415b3`, combining P00 and P02 at `439474f`; all seven feature files matched that revision exactly. The subsequent patch-bound restoration changes only the four atmospheric default assignments. Original branches and validation evidence remain preserved. ### Validation Earlier validation at combined revision `554d150` used GNU Strict builds on Derecho with GNU 12.2.0, Cray MPICH 8.1.25, NetCDF 4.9.2, and ESMF 8.5.0. Serial, OpenMP, MPI, hybrid, and ordinary NUOPC/ESMX execution passed. - Twelve focused initialization/forcing/timing scenarios per build; 45 Python tests; formatting and diff checks passed. - All 15 restart cases passed, including the five delayed-moisture cases previously failing in P00. There were 65 continuous/segmented model runs and 30 loader probes. - All 320 file comparisons and 12 cross-layout comparisons had zero maximum absolute error; 450 checkpoint checks passed. Loader probes comprised five accepted schema-compatible resumes and 25 expected rejections. - Existing PR suite: 36 model runs, 40 execution comparisons, and 36 approved-reference comparisons passed. Largest cross-driver absolute difference was `fgrnhfx = 0.125 W m-2`, consistent with the existing baseline. No reference or tolerance changed. - Original PR #39-era checkpoints generated with P00 were also checked: prescribed/time-based resumes agreed exactly with corresponding P02 checkpoints, and ambiguous step-based state was rejected. PBS jobs: `7890398.desched1`, `7890541.desched1`, `7890596.desched1`. Full retained evidence: `/glade/derecho/scratch/frediani/tmp/cfbm-P02-20261021T034953Z/`, including `attempt-02/results/restart-z8yehdcl/summary.json`, `attempt-02/results/pr-hw75p1af/summary.json`, and `legacy-originals/summary.json`. P01/P02/P04 combined acceptance remains pending, including shared clock precision and fractional/long-duration timing. Direct WRF-host execution/restart, NUOPC/ESMX restart, and transfer of checkpoints between MPI layouts remain untested here. Ordinary NUOPC/ESMX execution and comparisons among independent MPI layouts do not establish those restart paths. GitHub CI must pass on the final PR revision before merge; its normal PR suite does not run P00's optional restart suite. ### Initialization-placement follow-up validation Feature `439474f` matches tested P00/P02 integration `d144a898a436efb7ef411c781368e6853c0415b3` exactly. Four fresh GNU Strict builds (serial, OpenMP, MPI/NUOPC/ESMX, hybrid/NUOPC/ESMX) passed with the environment above. - All 15 restart cases pass: 65 continuous/segmented runs, 30 loading probes, 320 zero-error file comparisons (315 continuity and five accepted legacy resumes), 12 zero-error cross-layout comparisons, and 450 checkpoint checks. The restart suite's 67 CTest invocations include the twelve focused P02 lifecycle scenarios in each build. - The existing 36-run PR matrix, 40 execution comparisons, and 36 approved-reference comparisons pass. Maximum cross-driver difference remains `fgrnhfx = 0.125 W m-2` (maximum percentage error `0.0006368705741347137%`), within unchanged tolerances. - A connected, nonzero precipitation-rate ESMX_Data check passes with one MPI rank, four MPI ranks, and four ranks by four OpenMP threads. All 33 output-file comparisons against the previously validated executable are exact. Thirty accumulated-rain checks have maximum absolute error `1.862645149230957e-09` in model accumulation units. - Formatting and regression both passed in [GitHub workflow 39690399727](https://github.com/NCAR/fire_behavior/actions/runs/39690399727) on `439474f`. Local formatting and whitespace checks pass. No approved reference or regression tolerance changed. PBS job `7900048.desched1` completed the builds and both suites successfully, then exited 1 because the scratch rate-check script looked for its completion marker in stdout instead of the ESMF log. After correcting only that checker, job `7900164.desched1` reran the six rate-check model executions and exited 0. Failed-attempt evidence and original scripts remain preserved. Evidence: `/glade/derecho/scratch/frediani/tmp/cfbm-P02-init-review-20261021T225746Z/`, including `attempt-01/results/restart-hl_i2yr_/summary.json`, `attempt-01/results/pr-jpmyi5r2/summary.json`, and `attempt-01/rate-probe-02/summary.json`. The combined P01/P04, coupled-restart, and direct WRF-host validation gaps above remain. ### Patch-bound restoration The four atmospheric defaults retain their original `ifps:ifpe, jfps:jfpe` patch sections. The temporary explicit memory-bound spelling in `f6a6206` is superseded by `a87b23d`. The final assignments match the original expressions, and `state/state_mod.F90` matches develop exactly. Defaults remain in `Set_vars_to_default`; the NUOPC cold-start rainfall initialization in `Realize` is unchanged. Whitespace checks pass. The detailed HPC results above are from `439474f`; the optional restart suite has not been rerun after restoring patch bounds. GitHub CI on `a87b23d` is pending.
The unrestricted Rothermel moisture cubic produced nonzero reaction at extinction and negative reaction above it, affecting `iboros`, base spread and flame diagnostics before the later spread clamp. This change sets moisture damping to exactly zero at or above extinction, preserves the polynomial below extinction, and leaves physical moisture uncapped. An explicit moisture threshold is retained because a lower clamp on the polynomial alone can leave positive roundoff at exact extinction. The moisture-damping comment identifies `etam`. The independent empirical chaparral spread law remains unchanged. Refs #42; this addresses its extinction scope only. Validation follows the source and lifetime of each quantity: - `Check_nml` requires finite, nonnegative prescribed `fuelmc_g`. Zero is valid; units are water mass per dry fuel mass. - `fuel_t%Check_fuels`, called once during Anderson initialization, requires finite, positive `fuelmce` for burnable categories. - Moisture initialization validates the fuel loads and fixed averaging weights. The component then checks each evolving class after its update, before averaging. Valid classes with valid weights do not require a repeated check of the computed mean. Empty categories retain zero weights. - The restart reader checks effective and class moisture separately through the FMC component. `Check_fmc` is a common type-bound procedure of `fmc_t`, inherited by the moisture implementation. A source comment explains why `Check_fmc` remains: `fmc_g` is read independently from the checkpoint and used directly, and prescribed-moisture restarts have no class state. This is an input check, including for supported legacy files; an unchanged checkpoint from the same validated revision would already satisfy it. Computed-class checks execute after the OpenMP loop on the initial thread, avoiding concurrent fatal diagnostics across tiles. Spatial checks cover owned burnable cells. ROS owns the extinction calculation. No namelist option or checkpoint schema is added; P02 initialization and restart compatibility checks are retained. Full live/dead SB40 combustion and gridded-input readers remain separate work. The review base is `develop` at `622e191`, including merged #39 and #62. Head `af2ca03` contains a 12-file P03 diff. P00/#61 is a test dependency only in separate integration `3b56260`; its commits are absent from this feature branch. Production files, focused Fortran tests and configuration documentation in that integration match the PR head exactly. Validation of `af2ca03` / integration `3b56260`: - [GitHub workflow 39997883832](https://github.com/NCAR/fire_behavior/actions/runs/39997883832) passes formatting and regression on the published head. - Derecho PBS job `7922406` passes four GNU Strict builds: serial, OpenMP, MPI and hybrid, including ordinary NUOPC/ESMX execution. All 35 focused scenarios pass in every build, covering extinction boundaries and diagnostics, namelist/table inputs, class state, initialization loads, and independently restored effective moisture. The bound checker is also exercised without initialized class arrays, matching prescribed-moisture restarts. - All 378 PR CTests, 36 model runs, 40 execution comparisons and 36 approved-reference comparisons pass. Maximum reference error is zero; maximum execution difference is 0.125 W m-2 in `fgrnhfx`, within the existing tolerance. - All 15 restart cases and 30 loading probes pass. The 315 continuity file comparisons have zero error; 450 checkpoint checks and 12 independent layout comparisons also pass. - An additional 22 checkpoint mutation/control probes pass in serial and hybrid configurations. These include invalid `fmc_g` with unchanged valid classes, invalid classes, and accepted zero/wet states. - A separate Fortran probe with the GNU Strict arithmetic flags confirms that, at a moisture ratio of one, the polynomial and `max(etam, 0.0)` both return `2.384185791015625e-7` in single precision. This small positive residual explains retaining the explicit extinction condition. The below-extinction arithmetic is unchanged. No reference or tolerance was changed. Direct WRF-host execution/restart, NUOPC/ESMX restart, cross-layout checkpoint transfer and combined P01/P04 acceptance remain untested. Legacy readability does not establish numerical equivalence across the extinction correction. Evidence, exact commands, source snapshots and PBS logs are retained under `/glade/derecho/scratch/frediani/tmp/cfbm-P03-type-bound-review-20261023T174535Z`.
Adds continuous-versus-restarted regression coverage to the existing harness and runs it in GitHub CI alongside the existing PR suite. The cases exercise prescribed and evolving moisture, delayed moisture updates, checkpoints around observed-perimeter activation, and preservation of REAL64 fuel state. All 15 restart cases pass on the merged P02 initialization correction, including the five cases that failed on the original restart foundation. ## Scope and review base - Base: `develop` at `622e191fac104e1a73ae99a769e5bd3dfee2946c`, including merged #39 and #62. - Head: `56b832037a8e73d198d395e1c194afbdb46ed205`. - The 23-file diff contains the restart harness, tests/documentation, suite help and CI selection. Production code and P02's focused unit tests are identical to develop. No scientific tolerance or reference changes. - Merge `ce551ee` incorporates develop into the published P00 history. The moisture component and restart-reader conflicts retain develop's merged P02 implementation. This preserves the original P00 commits and the common ancestor used by stacked #65. - The harness includes the initialization metadata checks and 30 restart-loading probes previously validated in the P00/P02 integration. These check supported legacy inference and explicit rejection of ambiguous or damaged state. CI builds serial, OpenMP, MPI and hybrid configurations once, then runs both `pr` and `restart`. Both suites execute even if one fails; either failure fails the regression check. The existing test-evidence artifact retains both result directories. The approved `github-pr` baseline remains unchanged, and optional candidate generation still uses only the `pr` results. The configuration guide includes a restart YAML example and explains suite selection, segment timing and comparison policy. ## Validation Derecho PBS `7915414.desched1` exits zero after fresh serial, OpenMP, MPI and hybrid GNU Strict builds. MPI/hybrid builds include NUOPC and ESMX. Environment: GNU 12.2, Cray MPICH 8.1.25, NetCDF 4.9.2, ESMF 8.5, shared NPL Python. - All 67 restart CTests pass: 15 restart cases, 48 P02 lifecycle checks, and four Python harness registrations. - All 65 continuity model invocations and 30 restart-loading probes complete as expected. The 315 continuity file comparisons plus five accepted-legacy checkpoint comparisons are bitwise equal, with zero maximum error. - All 450 checkpoint validity checks and 12 cross-layout comparisons pass; cross-layout error is zero. - All 238 existing PR CTests pass, including 36 model runs. All 40 execution comparisons and 36 approved-reference comparisons pass. Reference error is zero; maximum cross-driver error is 0.125 W m-2 in `fgrnhfx`, within the unchanged tolerance. - All 45 Python tests pass; YAPF 0.43.0, shell syntax and whitespace checks pass. - Local workflow checks confirm both suites execute and the final status is correct when neither, either, or both suites fail. The documented restart configuration matches the production YAML; its 28 s split retains the next output at 36 s. - [GitHub workflow 39970138677](https://github.com/NCAR/fire_behavior/actions/runs/39970138677) passes on implementation revision `5de15ed`: formatting, all four builds, all 238 PR CTests and all 67 restart CTests. Downloaded reports confirm all 15 restart cases, 320 zero-error file comparisons and 30 loading probes pass in CI. Both suite summaries and eight JUnit reports are retained in the uploaded test-evidence artifact; the 36 approved-reference comparisons also have zero error. The PBS run used clean candidate `9a64864`, whose complete tree equals merge `ce551ee`. Later `c8aeeaa` changes documentation, and `5de15ed` changes CI selection and documentation; production code, model cases, harness and approved baseline remain identical to the PBS-validated revision. PBS evidence is preserved under `/glade/derecho/scratch/frediani/tmp/cfbm-P00-develop-20261022T230324-0600/`; CI-update evidence is under `/glade/derecho/scratch/frediani/tmp/cfbm-P00-restart-ci-20261023T082618-0600/`. Repository validation history retains the original failures and records the successful PBS rerun. Documentation-only follow-up `2c6b5b9` adds the [test-count breakdown](https://github.com/NCAR/fire_behavior/blob/test-restart-continuity/tests/regression/README.md#how-test-counts-are-reported): checkpoint validity, file comparisons, the per-build PR CTest table, and restart invocation counts. Counts and links were checked against the downloaded reports from workflow 39970138677. Its [CI run 39974535865](https://github.com/NCAR/fire_behavior/actions/runs/39974535865) uses the unchanged workflow; no executable, harness, case, or baseline changed. Documentation follow-up `b2da936` replaces development labels in the user-facing regression guides with descriptions of moisture initialization, legacy restart metadata, and coverage limits. `tests/AGENTS.md` requires this distinction and keeps development provenance in validation records. Follow-up `56b8320` consolidates all test guidance there, retains a pointer at `tests/regression/AGENTS.md`, and requires informative unit-test comments describing the behavior, setup and expected result. The historical validation file is unchanged. Guide-label, relative-link, whitespace, and pinned YAPF checks pass. Workflows 39974535865 and 39977209659 on the preceding documentation revisions passed both formatting and regression checks. ## Coverage limits Restart comparisons use the standalone driver and the same revision/execution layout. NUOPC/ESMX integrations pass in the existing PR suite, but coupled restart, native WRF-host restart, cross-layout checkpoint transfer, and combined P01/P04 behavior remain separate validation work. Synthetic legacy checkpoints test schema compatibility, not numerical equivalence across scientific revisions. No production correction for #42 is included in this test PR. --------- Co-authored-by: masih <masih@ucar.edu>
Description of changes:
The CFBM has had no way to stop a simulation and resume it.
This feature adds a restart capability. A run can write its dynamic fire state to a
NetCDF file at a fixed interval, and a later run can be initialized from one of
those files instead of a cold start. It covers standalone runs, serial and MPI,
and runs coupled to WRF.
Two options are added to the
&timenamelist block:restart.false.fire_restart_<start datetime>.ncrestart_interval-1-1disables writesBoth default to the previous behavior, so existing namelists are unaffected.
Changes by file:
state/state_mod.F90carries theRead_restartandWrite_restarttype-bound methods,
Build_restart_file_name, and the MPI helpers they need.A restart read validates the grid metadata stored in the file against the
current configuration before touching any state, so a mismatched restart file
fails early with a message instead of producing a silently wrong run.
io/netcdf_mod.F90adds attribute writers (Add_netcdf_att) andMPI-aware readers and writers for distributed 2D and 3D fields.
driver/initialize_mod.F90routes the real-case restart path through anew
Init_fire_state_from_restart, leaving the existing geogridinitialization untouched. The WRF-coupled path reads the restart after the
fire components are initialized.
driver/fire_behavior.F90reads the restart state at startup in place ofSave_state, and writes a restart everyrestart_intervalseconds.restart_intervalmust map to a whole number of time steps or the run stopswith a message.
io/namelist_mod.F90adds the two options, their broadcast, and a rangecheck.
physics/fmc_wrffire_mod.F90drops its unusedstate_modandnamelist_modimports. This one is load-bearing rather than cosmetic:state_modnow usesfmc_wrffire_modfor the moisture restart fields, soleaving those imports in place would create a circular module dependency.
Type of change
Tests conducted:
All testing on Derecho (NCAR) using the repository environment
env/derecho/gnu-12.2.0: GCC 12.2.0, cray-mpich 8.1.25, NetCDF-Fortran 4.9.2,HDF5 1.12.2, ESMF 8.5.0, CMake 3.26.3.
Builds. Three configurations, all clean with no errors or new warnings:
serial (
--mpi-off), MPI (default), and--esmx(which includes NUOPC).Registered ctest suite, run on the
--esmxbuild so every registered caseis reachable:
7 of 7. The NUOPC cases matter here because the cap uses both
initialize_modand
state_mod, and this PR touches both.Restart regressions. These run a case straight through, then run it again
stopping and restarting partway, and compare the two. Restart test scripts are
not included in this PR (see below), but the results are:
Serial build, 20 of 20 checks:
MPI build on a compute node, 30 of 30 checks:
The multi-rank runs are the ones that exercise the distributed restart read and
write and the halo exchange following a restart read. Restarting at 2 and 4
ranks reproduces the continuous run, so the domain decomposition is handled
correctly on the restart path.
57 passing checks in total across the three build configurations.
Not covered. Two gaps, stated explicitly:
Init_fire_state_within_wrfcallingRead_restart) is exercised by a separate WRF harness, and not tested here.On the test scripts. The restart test scripts (
test7_restart.s,test8_restart.s,test9.s,test9_restart.s, thetest9case directory, andthe
tests/CMakeLists.txtregistration) are deliberately held out of this PRpending the separate discussion about how tests should be organized. They are
ready and can be added here or in a follow-up PR, whichever is preferred.
Documentation:
Documentation was left out to keep the diff focused on
the code. Happy to add it to this PR if preferred, otherwise it will follow
separately.
Contributors:
@masih-e (Masih Eghdami:masih@ucar.edu)
@mefrediani (Maria Frediani:frediani@ucar.edu)
@pedro-jm (Pedro Jimenez:jimenez@ucar.edu; reviewer)
LLM/coding assistant technology used: the restart implementation was developed
with assistance from Codex GPT-5.5 and Claude Opus 4.7. Claude Opus 5 assisted
with the pre-review cleanup of this branch.
Further information:
state/state_mod.F90grows from 1063 to 1965 lines in this PR, which is morethan that file should carry. A restart-specific module is worth doing as a follow-up. This
change is not possible with the current code because the restart calls depend on the fire
state type.