Skip to content

DEVLOG.md

Latest commit

 

History

History
86 lines (68 loc) · 5.09 KB

DEVLOG.md

File metadata and controls

86 lines (68 loc) · 5.09 KB
Raw

AquaNexus development log

The log for the nationwide dissolved-oxygen rebuild. The log of the earlier two-river version (September 2026) is in docs/archive/DEVLOG_legacy.md; that code is at the git tag legacy-ayase-naka.

2026-10-16 to 2026-10-18: rebuilding around a nationwide dataset

Why

An honest review of the first version found that its real signal was 138 dissolved-oxygen labels from two rivers; everything else was infrastructure, a synthetic habitat index (circular by construction) and a hydraulic model that did not move the score. The rebuild keeps real labels only and trains on the Ministry of the Environment's nationwide record.

Data (details in docs/DATA_SOURCES.md)

  • 565,674 DO samples, 5,883 stations, 47 prefectures, FY2013-22. Gate asked for about 50,000.
  • Weather joined from NASA POWER (193 grid cells). Open-Meteo ERA5 was tried first and abandoned because its free tier rate-limits long histories; ERA5-Land's daily endpoint returns only temperature.
  • Station elevation from GSI; relief among stations within 10 km derived from it. Open-Meteo's elevation API was also rate-limited and dropped.
  • File quirks worth remembering: NENDO is fiscal year and FISCALYEAR is calendar year of the sample; DO_3 is label-derived and must never be a feature.

Method

Four evaluation regimes (temporal, unseen station, both at once, unseen prefecture), baselines (climatology, persistence, ridge), LightGBM with early stopping on carved-out stations, Optuna tuning scored by station-grouped CV on training rows only, split-conformal intervals. Seven feature sets from weather-only to weather plus neighbours plus the station's own history.

Results

See README.md and docs/RESULTS.md. In short: strict-regime R² 0.48 (geo), 0.55 (nowcast), 0.73 (full) against 0.38 for the best baseline.

What did and did not work:

  • Station identity is worth about 0.25 R². Weather alone loses to climatology at stations the climatology has seen, and beats it at stations it has not.
  • Elevation and relief added about 0.06-0.07 R² for weather-only models.
  • Neighbour features from stations within 25 km added little (at most 0.05 R²) and hurt on the Ayase.
  • Adding neighbours within 5 km and stations on a same-named river within 60 km (matched on the river name with segment marks stripped) lifted nowcast from 0.516 to 0.549 and full from 0.715 to 0.728 on the strict regime, and nowcast on the Ayase from 0.33 to 0.63. On the temporal regime nowcast moved the other way (0.632 to 0.612), so the gain is not uniform.
  • Land use (国土数値情報, 100 m mesh, shares within 1 and 5 km of each station; 113 files, 920 MB, downloaded with approval): strict-regime R² geo 0.484 to 0.504, nowcast 0.549 to 0.560, full 0.728 to 0.726 (no gain). On unseen stations the gain for geo is larger (0.502 to 0.536) and recall of DO below 5 mg/L for geo rose from 20% to 23%. Ayase geo 0.51 to 0.60, Naka 0.34 to 0.43. Served geo and nowcast use land use; full does not. Shares are circular buffers around the station, not upstream catchments, and the mesh is FY2016.
  • A dedicated below-5 classifier ranks low-oxygen samples better than thresholding the regression (AUC 0.91/0.92/0.97 against 0.86/0.88/0.95) and its raw probabilities are well calibrated. Isotonic recalibration hurt, so it is not used.
  • Tuning moved held-out R² by less than 0.01 on unseen stations. Defaults are served.
  • Tail weighting (weights 3, 10, 30) raises recall of DO below 5 mg/L at a steep cost in precision; weight 10 is served as full_alert.
  • Interval coverage: about 0.89 at 0.90 nominal overall, 0.40-0.57 for samples below 5 mg/L (geo to full).

Integrity check: neighbour features and held-out stations

add_neighbors initially used every station's readings, so a training row's neighbour features could contain a held-out station's label. That does not reach a test row's own prediction, but it is test labels inside training inputs. scripts/check_neighbor_leak.py rebuilds the frame with the held-out stations excluded from every neighbour pool. Result (experiments/do_neighbor_leak_check.csv): R² moved by at most 0.011 in either direction (strict nowcast 0.507 to 0.518, full 0.715 to 0.717, geo 0.484 to 0.482), so the published numbers stand. The pooled version is kept as the default because it reflects deployment, where the neighbours of a new station are the stations that exist.

Cleanup and rebuild of the repository

  • Removed HEC-RAS, point-cloud, GNN, Bayesian, deep-learning, transfer, remote-sensing, fish survey and synthetic-data code, with their tests and scripts. Old documents moved to docs/archive/. Tag legacy-ayase-naka preserves the old state.
  • New API (/predict, /models, /stations) and frontend (overview, predict, models, about).
  • Freed about 80 GB by deleting regenerable extracted LiDAR files and caches (with confirmation).

Open items

  • Upstream catchment (needs flow direction) and distance to coast would need further data.
  • Low-oxygen events remain poorly predicted; a classifier and adaptive intervals are the next things to try.