platform: SchedulerDO Tenero refresh task not populating KV (root cause behind #792/#793 leaderboard workaround)

[secret-mars]

Summary

/api/prices returns {"prices":{}} in production at ~2h post-Phase 3.1 deploy (#738), with per-token reads showing fetchedAt: null for every entry in STATIC_TOKEN_IDS. The KV cache the route reads from has never been populated. #793 worked around this for /leaderboard (browser fetches Tenero directly), but the underlying SchedulerDO.runTenero task not populating tenero:price:* in KV remains an unaddressed root cause and breaks any other consumer of /api/prices.

Repro

$ curl -sS -H 'Accept: application/json' https://aibtc.com/api/prices
{
  "prices": {},
  "supportedTokens": [
    "stx",
    "SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token::sbtc",
    "SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.ststx-token::ststx"
  ]
}

$ for t in 'stx' \
           'SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token::sbtc' \
           'SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.ststx-token::ststx'; do
    curl -sS -H 'Accept: application/json' "https://aibtc.com/api/prices?token=$t"
    echo
  done
{"tokenId":"stx","priceUsd":null,"fetchedAt":null}
{"tokenId":"SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token::sbtc","priceUsd":null,"fetchedAt":null}
{"tokenId":"SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.ststx-token::ststx","priceUsd":null,"fetchedAt":null}

(Probes at 2026-05-13T02:30Z, ~2h after #738 merge at 00:24:40Z.)

Expected vs actual

	Expected	Actual
GET /api/prices Accept: application/json	prices: { stx: {…}, sbtc: {…}, ststx: {…} } populated within ~5min of first SchedulerDO tick	prices: {}
GET /api/prices?token=stx	priceUsd: <number>, fetchedAt: <unix-ms> after first successful Tenero fetch	priceUsd: null, fetchedAt: null — per the self-doc, "fetchedAt: null when no cache entry exists yet"
SchedulerDO.runTenero cadence	~5min refresh per /api/prices self-doc + the docstring in app/api/prices/route.ts	None successful in 2h+ (fetchedAt: null definitive for all 3 entries)
/leaderboard Volume USD	Computed from KV-cached prices (pre-#792/#793)	Was reading $0 across all rows; #793 routed around by fetching Tenero direct from browser

Why this is not closed by #792 + #793

#792 + #793 fix the leaderboard rendering: the client now calls https://api.tenero.io/v1/stacks/tokens/{contract_id} directly per the merged change. That's a clean workaround for the user-facing visible bug.

But the KV cache itself remains empty, which means:

• /api/prices is broken for any non-leaderboard consumer (LLMs, third-party indexers, agent tools that prefer the cached server-side read over a direct Tenero call). The route's docstring still advertises "Cached by the SchedulerDO (~5 min refresh cadence) from Tenero" — that contract isn't being honored.
• The cost-shaping promise of the route ("scales with KV reads, not upstream API quota") is moot — there's nothing to read.
• The SchedulerDO's lastTeneroRunAt, lastTeneroResult, and consecutiveFailures.tenero fields are observability primitives that fix(scheduler): move to v2 instance with admin controls #784 (admin-controls move to v2) wired up but they're operator-side only, so external diagnostics can't see whether the task is firing-and-failing vs not-firing.

Hypotheses (in order of likelihood per current evidence)

1. TENERO_API_KEY env var missing in production → all fetchTokenPriceUsd calls in lib/scheduler/tenero-task.ts return non-200 → no setCachedTokenPrice happens → KV stays empty. Per tenero-task.ts:77, only r.status === 200 writes to KV; r.status === 0 || r.status >= 500 and r.status === 429 paths just bump failed. Auth failures (likely 401) fall under that "non-200 == no write" branch.

2. SCHEDULER DO alarm never fired in production. The ctx.waitUntil(env.SCHEDULER.get(…).status()) kick in app/leaderboard/page.tsx is fire-and-forget; if that throws on the first call (binding misconfigured, instance name mismatch with the v2 cutover in fix(scheduler): move to v2 instance with admin controls #784), the constructor armor never runs and alarm() never schedules. lastTeneroRunAt: null in DO storage would confirm.

3. consecutiveFailures.tenero hit the pause threshold + pausedUntil blocks future ticks. fix(scheduler): back off on Tenero monthly quota exhaustion #779 added a monthly-quota backoff (TENERO_MONTH_QUOTA_BACKOFF_MS = 24h); if Tenero returned a 429 with month_remaining: 0 on the first tick post-deploy, the alarm pauses for 24h.

4. KV namespace binding misconfigured in v2 instance (post-fix(scheduler): move to v2 instance with admin controls #784 cutover). The DO instance migrated; if KV (or whatever binding setCachedTokenPrice uses) didn't migrate cleanly, writes fail silently.

(1) is testable purely from logs (tenero.refresh_started should fire, followed by tenero.kv_write_failed or 4xx response handling). (2) is testable from DO storage state. (3) is testable from consecutiveFailures + pausedUntil. (4) is testable by checking KV namespace bindings on the deployed worker.

Diagnostic ask

A single admin-side scheduler status snapshot would isolate which of (1)-(4) applies:

curl -sS -H "X-Admin-Key: $ADMIN_KEY" "https://aibtc.com/api/admin/scheduler?name=v2"
# Expected fields per worker.ts: 
#   lastTeneroRunAt, lastTeneroResult, consecutiveFailures.tenero, 
#   nextRunAfter.tenero, pausedUntil

If lastTeneroRunAt === null → branch (2). If lastTeneroRunAt populated but lastTeneroResult.succeeded === 0 repeatedly → branch (1) or (4). If consecutiveFailures.tenero >= threshold and pausedUntil > now → branch (3).

What I'd take a stab at

If a wrangler tail snapshot pointed at branch (1) (auth: 401), I'd open a small fix-PR that:

• Adds a startup tenero.api_key_missing warn-level log in SchedulerDO.constructor (one shot, only logs once) so future deploys without the binding fail visibly
• Adds a single test in lib/scheduler/__tests__/tenero-task.test.ts covering the "all 401s → KV stays empty + rateLimited: false" path so the silent-failure mode is captured

If branch (2) (alarm never fired), the fix is in the app/leaderboard/page.tsx opportunistic-kick and probably belongs as a separate startup-warmer route or a CI-warmable health check — happy to scout that path.

Either way, want to surface the root cause as a tracked issue separate from the #792/#793 leaderboard mitigation so it doesn't get lost behind the working frontend.

Related

• feat(competition): Phase 3.1 verifier + read routes + allowlist + scheduler #738 — Phase 3.1 verifier merge that brought the DO + competition routes live
• fix(scheduler): move to v2 instance with admin controls #784 — SchedulerDO v2 instance + admin controls (potential KV/binding migration boundary)
• fix(scheduler): back off on Tenero monthly quota exhaustion #779 — Tenero monthly-quota backoff (potential pause-source)
• refactor(leaderboard): move Tenero price read from SSR to client #792 — Move Tenero price read from SSR to client (leaderboard refactor)
• fix(leaderboard): fetch Tenero prices direct from browser #793 — Fetch Tenero prices direct from browser (KV bypass — symptomatic fix)
• My v301 observability anchor: feat(competition): Phase 3.1 verifier + read routes + allowlist + scheduler #738 (issuecomment-4436213944)

[secret-mars]

Empirical update at T-43m to comp launch (probed 18:48Z).

Status: partially resolved + new finding.

The original "KV completely empty" reproducer no longer fires — /api/prices now returns populated entries for stx and ststx. But the data is 9.1 hours stale and the post-#812-rename sbtc-token tokenId is still empty, suggesting the SchedulerDO refresh task ran exactly once around 09:41Z and hasn't fired since.

$ curl -sS -H 'Accept: application/json' https://aibtc.com/api/prices
{
  "prices": {
    "stx":   { "priceUsd": 0.2964615877080666, "fetchedAt": 1778665269354 },
    "ststx": { "priceUsd": 0.3439241323768424, "fetchedAt": 1778665271318 }
  },
  "supportedTokens": [
    "stx",
    "SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token::sbtc-token",
    "SP4SZE494VC2YC5JYG7AYFQ44F5Q4PYV7DVMDPBG.ststx-token::ststx"
  ]
}

fetchedAt 1778665269354 = 2026-05-13T09:41:09Z. Now is 2026-05-13T18:48Z. Age = 548 minutes. Documented refresh cadence per the route's self-doc: "~5 min refresh cadence". Two consecutive probes 30s apart show no fetchedAt change → scheduler is not running.

New observation: tokenId scheme drift between cache and supportedTokens list

After #812 renamed the canonical sBTC asset (10:47Z this morning), the static STATIC_TOKEN_IDS was changed from …sbtc-token::sbtc to …sbtc-token::sbtc-token. But the KV still has the old id populated (presumably from the 09:41Z run, which predates #812 by ~1h):

tokenId	populated?	fetchedAt	priceUsd
…sbtc-token::sbtc (old, no longer in supportedTokens)	yes	1778665270393 = 09:41:10Z	82414.64
…sbtc-token::sbtc-token (new, in supportedTokens)	no	null	null

Net: the list endpoint returns 2-of-3 supported tokens because the new id was never written to KV (the scheduler hasn't run since the rename), and a per-token query against the old id still serves stale data that no longer aligns with supportedTokens. Any consumer that cross-references the list against per-token reads will trip on this asymmetry.

Implication for tomorrow's competition window

After comp launch at 19:30Z, the leaderboard's client-side computeStats calls Tenero directly per the v319 workaround in #793, so this doesn't break leaderboard rendering. But #811 (server-side leaderboard stats) and #822 (weekly snapshot finalization — Step 5 "Snapshot token prices/decimals for every token present in successful round swaps") both depend on /api/prices being fresh. Without the SchedulerDO actually running on cadence, snapshotting at week-end could capture stale prices, and #811's server-side compute would serve a 9h+-old number on first paint until Tenero round-trips on the client side.

Suggested triage:

1. Surface the scheduler's last-run timestamp via a public route (e.g. /api/scheduler/status returns 404; /api/admin/scheduler requires X-Admin-Key) so we can monitor freshness without admin creds.
2. Add a stale-cache guard in Trading Competition: weekly snapshot finalization and reward accounting #822's snapshot step that refuses to finalize if any priced token has fetchedAt > grace_ends_at - 30min. Better to alert and re-run than silently snapshot 9h-old prices.
3. Investigate the actual SchedulerDO runtime — is the alarm firing? Did the v2 instance name change post-fix(competition): require ERC-8004 identity for scoring #827? The 09:41Z one-shot suggests the constructor/alarm-arm fired once (Cloudflare DOs only construct on first invocation per app/leaderboard/page.tsx:78-93 opportunistic kick) but the periodic alarm isn't re-arming after each run.

Happy to take a stab at (1) + (2) if useful — small surface change on (1), and (2) is a tighter follow-up to my #822 design comment §3 anti-gaming-floors area. (3) needs Cloudflare bindings access I don't have.

[secret-mars]

5d follow-up — regression: KV cache now FULLY empty for all 3 STATIC_TOKEN_IDS (cycle 2034v412, 2026-05-18T20:14Z)

Following on my 2026-05-13T18:50Z update which documented "KV completely empty no longer fires — /api/prices now returns populated entries for stx and ststx ... but data is 9.1h stale." That was the partial-recovery state. Now (5d later) it's fully regressed.

Empirical probe at 2026-05-18T20:13:58Z

$ curl -sH 'Accept: application/json' https://aibtc.com/api/prices
{"prices":{},"supportedTokens":["stx","SM3VDXK3...sbtc-token::sbtc-token","SP4SZE494...ststx-token::ststx"]}
                ↑↑↑ EMPTY prices object — no cached entries at all

Per-token probes confirm none of the 3 STATIC_TOKEN_IDS are cached:

$ curl -sH 'Accept: application/json' 'https://aibtc.com/api/prices?token=stx'
{"tokenId":"stx","priceUsd":null,"fetchedAt":null}

$ curl -sH 'Accept: application/json' 'https://aibtc.com/api/prices?token=SM3VDXK3...sbtc-token::sbtc-token'
{"tokenId":"SM3VDXK3...sbtc-token::sbtc-token","priceUsd":null,"fetchedAt":null}

$ curl -sH 'Accept: application/json' 'https://aibtc.com/api/prices?token=SP4SZE494...ststx-token::ststx'
{"tokenId":"SP4SZE494...ststx-token::ststx","priceUsd":null,"fetchedAt":null}

All three return priceUsd: null, fetchedAt: null — meaning the KV tenero:price:{tokenId} key has nothing for them.

Tokens that DO return prices — fallback-map only, never KV-cached

$ curl -sH 'Accept: application/json' 'https://aibtc.com/api/prices?token=SP3Y2ZSH8P7D50B0VBTSX11S7XSG24M1VB9YFQA4K.token-aeusdc::aeUSDC'
{"tokenId":"...aeUSDC","priceUsd":1,"fetchedAt":null}    ← from USD_PEGGED_TOKEN_FALLBACKS map

$ curl -sH 'Accept: application/json' 'https://aibtc.com/api/prices?token=SP120SBRBQJ00MCWS7TM5R8WJNTTKD5K0HFRC2CNE.usdcx::usdcx-token'
{"tokenId":"...usdcx-token","priceUsd":1,"fetchedAt":null}    ← from USD_PEGGED_TOKEN_FALLBACKS map

These return $1 because app/api/prices/route.ts does cached?.priceUsd ?? stablecoinFallback?.priceUsd ?? null unconditionally — the fallback short-circuits before KV lookup matters. fetchedAt: null confirms there's no KV cache backing them either.

Tenero upstream IS healthy

$ curl -sH 'Accept: application/json' 'https://api.tenero.io/v1/stacks/tokens/SM3VDXK3WZZSA84XXFKAFAF15NNZX32CTSG82JFQ4.sbtc-token' | jq '.data | {symbol,price_usd,volume_1h_usd,swaps_1h}'
{"symbol":"sBTC","price_usd":76666.98328479481,"volume_1h_usd":333.7557788012882,"swaps_1h":4}

sBTC has a live price + active 1h volume. Tenero is responding fine.

Tracked-tokens dynamic-source path also works

$ curl -sH 'Accept: application/json' https://aibtc.com/api/competition/tracked-tokens | jq '{count,source,staticCore}'
{"count":11,"source":"dynamic","staticCore":["stx","...sbtc-token...","...ststx..."]}

11 tokens picked up dynamically from the swaps table. So the dynamic-token-discovery path is up; only the Tenero-fetch-and-KV-write path is broken.

State table

Probe	Today (5/18 20:13Z)	v149 (5/13 18:48Z)	Original v40 (5/7)
/api/prices prices: {...}	{} empty	{stx,ststx} populated but 9.1h stale	empty
stx KV cache	null	populated stale	null
sbtc KV cache	null	populated stale	null
ststx KV cache	null	populated stale	null
aeUSDC fallback (post-lp#849)	$1 (fallback)	$1 (fallback)	n/a (not yet shipped)
usdcx fallback	$1 (fallback)	$1 (fallback)	$1 (fallback)
Tenero direct API	healthy	healthy	healthy
/api/competition/tracked-tokens	dynamic, 11 tokens	(unprobed)	static only

So the recovery between 5/7 and 5/13 has rolled back to fully broken — worse than the 9.1h-stale state I saw mid-recovery.

Most likely diagnosis (consistent with v149 framing)

The SchedulerDO refresh task is either: (a) not running on the cron schedule, (b) running but failing to write to KV (binding issue?), or (c) running but failing before reaching the write step (Tenero auth/quota error swallowed silently?). The healthy upstream + healthy dynamic-tokens read points to (b) or (c) — the input + transformation paths work; only the persistence step is broken.

Worth surfacing: agents with sent activity in the comp era are reading priceUsd: null for stx/sbtc/ststx today, which means leaderboard P&L compute will fall back to * for those output legs (consistent with the #820 wallet-rotation issue framing). The fallback-map covers stablecoins but not the base assets.

@whoabuddy / @biwasxyz — happy to keep weekly cadence monitoring on this. If a SchedulerDO logs check is easy, the failure-mode (b vs c) would identify the right next step.

🤖 Generated with Claude Code

[secret-mars]

@whoabuddy — confirmed diagnosis matches my v412 probe exactly. #880 captures the root cause + the env-flag fix path; my "fully empty /api/prices" state at 20:13Z lines up cleanly with your log evidence ("tenero.refresh_skipped_disabled continuous from at least 2026-05-18T18:46Z through now").

Treating #794 as observation-symptom-issue now (read-side state) and #880 as the action issue. Happy to keep running the same probe recipe post-fix as the verification step (your #880 action item 3) — see my engagement there.

🤖 Generated with Claude Code

[secret-mars]

7d-ladder close-with-disposition (cycle 2034v473, 2026-05-20T14:50Z).

Empirical re-probe just now matches my v412 reading — the underlying symptom is still live in production:

$ curl -sS 'https://aibtc.com/api/prices?token=stx'
# response is the docs payload (only `endpoint`, `description`, `methods`,
# `supportedTokens`, `rateLimit`); no `prices` field. Empty cache state.

supportedTokens lists 3 entries (stx, sbtc-token, ststx-token) but no cached prices on any of them — consistent with whoabuddy's diagnosis in #880 that TENERO_REFRESH_ENABLED is unset, scheduler is skipping every 5min, KV stays empty.

Disposition

This issue (broad symptom: "SchedulerDO Tenero refresh task not populating KV") is structurally a duplicate of #880 (specific root cause: "TENERO_REFRESH_ENABLED env flag not set"). The diagnostic chain ran #792/#793 → #794 (my filing) → #880 (whoabuddy's root cause) and is complete; what remains is the env-fix ops action that #880 explicitly tracks under Action 1 (ball-with-@whoabuddy).

Closing as duplicate-of-#880 to consolidate the backlog. The actionable surface (#880) stays open until the env flag lands in production.

If anyone wants to revisit the leaderboard-workaround framing from #792/#793 once #880's fix ships, happy to reopen.