Compare to epicast

[stevenhofmeyr]

Pull Request: compare-to-epicast → development

Summary

This branch adds infrastructure and bug fixes to enable quantitative comparison of ExaEpi simulation outputs against Epicast reference data. It introduces a new Python script for parsing Epicast binary outputs, significantly enhances the comparison and parameter-fitting utilities, fixes several correctness bugs in the disease progression model, and improves the work-neighborhood interaction model.

---

Changes

New Features

utilities/read_epicast_events.py (new file)

• Parses Epicast binary event output files into structured data.
• Supports splitting output by run and plotting multiple Epicast and ExaEpi time series on the same axes for side-by-side comparison.

utilities/compare_to_epicast.py (major update)

• Enables plotting of multiple random ExaEpi runs alongside multiple Epicast runs.
• Improved layout, labeling, and series aggregation for clearer visual comparison.

utilities/calc_epicast_trans_probs.py (major update)

• Replaced manual parameter tuning with a proper optimization pipeline:
  • Converts cumulative transition probability arrays to PMFs using a survival/hazard model.
  • Fits gamma distributions to empirical PMFs via differential evolution (global search) followed by Nelder-Mead local refinement.
• Added a loc (location/shift) parameter to all gamma distributions for latent, infectious, incubation, and immune periods, enabling better fits to Epicast's discrete transition tables.

---

Bug Fixes

src/AgentDefinitions.H — Fix isNewlyPresymptomatic counting

• The previous definition fired one step before isNewlySymptomatic/isNewlyAsymptomatic, causing a mismatch in aggregate counts.
• Fixed so that isNewlyPresymptomatic fires on the same step as the symptomatic/asymptomatic transition (disease_counter == incb_period), ensuring sum(NewP) == sum(NewS) + sum(NewA) over a complete run.

src/AgentContainer.cpp & src/InitializeInfections.cpp — Pass symptomatic pointer to setInfected

• setInfected now receives the agent's symptomatic status pointer so it can be initialized correctly at infection time, fixing a missing initialization.

src/InteractionModWorkNborhood.H — Fix work-neighborhood interaction model

• Corrected a variable shadowing bug (nborhood_ptr was declared twice).
• Fixed the community vs. neighborhood infection count split: community counts now correctly exclude the agent's own neighborhood (previously they included it, double-counting neighborhood infections).
• Applied asymp_relative_inf scaling consistently to both community-level and neighborhood-level asymptomatic contributions.

src/DiseaseParm.H / src/DiseaseParm.cpp — Add loc parameter to gamma distributions

• Added immune_length_loc, infectious_length_loc, incubation_length_loc, and hospital_delay_length_loc shift parameters to all gamma-distributed disease period samplers.
• These are readable from input files via pp.query(...).
• Updated setInfected to apply the loc offset when sampling latent, infectious, incubation, and hospital delay periods.
• Fixed the incubation period clamp: changed > to >= and adjusted the floor to - 1 to avoid an off-by-one that could produce a zero-length infectious window.

src/DiseaseStatus.H & src/HospitalModel.H — Apply loc to immune period sampling

• Both the standard recovery path and the hospital discharge path now add immune_length_loc when drawing the immune period duration.

---

Behavior / Configuration Changes

src/main.cpp

• Removed the #ifndef COMPARE_TO_EPICAST guard around the standard daily interaction sequence (morning commute, daytime interactions, evening commute, evening interactions). All interaction steps now run unconditionally, making the COMPARE_TO_EPICAST build mode behave identically to the standard build for the interaction loop.
• Minor reformatting of the debug agent dump header string.

CMakeLists.txt

• Disabled AMReX_TINY_PROFILE (was ON).
• Added GIT_SUBMODULES "" to the AMReX FetchContent_Declare to avoid fetching unnecessary submodules (fixes build issues in the devcontainer).
• Added a commented-out add_compile_definitions(COMPARE_TO_EPICAST) line as a convenience toggle.

src/InitializeInfections.cpp

• Removed the #warning preprocessor directive that fired on every COMPARE_TO_EPICAST build.
• NTRY defaults to NTRY+10 (i.e., 10) when the Epicast comparison mode is not defined.

---

Infrastructure / Housekeeping

• requirements.txt: Added scipy and matplotlib as explicit dependencies (required by the comparison and fitting scripts).
• .gitignore: Added __pycache__ to the root ignore list.
• data/.gitignore: Added ignore rules for large external datasets (EducationData, LODES7, census block group shapefiles, UrbanPop binaries) while preserving the committed NM test fixtures.
• utilities/UrbanPop-scripts/plot_timeseries.py: Minor plotting fix for updated output format.
• utilities/custom-clang-format.py: Updated to handle additional edge cases in the clang-format wrapper script.

---

Testing

• The standard build (without COMPARE_TO_EPICAST defined) is unaffected by the interaction-loop guard removal.
• The isNewlyPresymptomatic fix can be verified by checking that aggregate NewP, NewS, and NewA counts balance over a full simulation run.
• The work-neighborhood double-counting fix is verifiable by confirming that num_infected_community >= 0 assertions no longer fire (the old code could produce negative values).

---

Dependencies

• Python utilities now require scipy and matplotlib (added to requirements.txt).

[atmyers]

Could you merge or rebase with development? Some of these commits are already on there.

I think the CI failure could be fixed by only changing the nborhood_size and workgroup_size for the UrbanPop initialization type. (I know the plan is to remove the Census data option and only use UrbanPop going forward, but I think would be a quick fix and we could merge this faster, then remove the census data option and change the CI tests to use UrbanPop in a separate PR.)

[stevenhofmeyr]

It's already merged with development. I've reset the nborhood_size and workgroup_size to the previous values.