NEMWEB 403 resilience: exponential backoff + Retry-After + request-volume limits

[purcell-lab]

Follow-up to #20 (added a User-Agent to the STPASA/PD7DAY/TradingIS clients).

Why

NEMWEB (Akamai) intermittently returns 403 Forbidden, which leaves STPASA/PD7DAY data stale (and was the trigger for the log flood fixed in #19). Investigation finding: from an unaffected IP, every User-Agent (including HA's default and a browser UA) returns 200 — so the worst 403s are rate/IP-based, not purely UA-based. The UA in #20 is necessary hygiene but likely insufficient on its own.

What's missing today

• coordinator._fetch_all_with_retry does a single 5 s retry on 403 for PD7DAY only.
• StpasaClient.fetch_all_regions has no retry/backoff — any error returns {} (best-effort), so a transient 403 immediately drops the whole cycle.
• TradingIS / dispatch likewise have no backoff.
• The shared semaphore caps concurrency (2) but not request frequency.

Proposed work

1. Exponential backoff with jitter on 403/429 (e.g. 5s → 15s → 45s, ±jitter), shared helper used by all NEMWEB clients — not just PD7DAY.
2. Honor Retry-After when present on the 403/429 response.
3. Cap attempts (e.g. 3) then fall through to the existing stale-cache fallback (no change to that behaviour).
4. Request-volume limiting — a minimum interval between NEMWEB requests (token-bucket / min-gap), in addition to the concurrency semaphore, to stay under Akamai's rate threshold across the 5 region coordinators + report types.
5. Optional: distinguish 403 (bot/rate block) from 429 (explicit rate limit) in logging for diagnosis.

Acceptance

• A transient 403 no longer drops a whole STPASA/PD7DAY cycle (retried with backoff first).
• Sustained 403s degrade gracefully to stale-cache with one warning per cycle (already true after fix(stpasa): log staleness once per cycle, not per forecast interval #19).
• No increase in steady-state NEMWEB request rate.

Context / log evidence

On the live instance this was ~212k stale warnings/day (symptom, fixed in #19) driven by continuous STPASA staleness from these 403s. Verify against the NEMWEB STPASA + PD7DAY directory endpoints.