Stage 3: differentiators — live external data, evolving profile, spatial page portals - #3
Merged
Conversation
…ting) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…server-side suggestion filter Carried Stage-2b items (Stage-3 Task 1): - orchestrator.py: _MAX_PROMPT_CHARS (~12000) caps the composed system message after seed-JSON + module-context + exchange fold + conversation block are stacked. Over budget, the conversation block is truncated first, then module-context detail — the raw user prompt and exchange answers are never touched. - routes/transcribe.py: a reusable in-memory _RateLimiter (sliding window) caps transcription at 20/5min per owner → 429; the generate/preview routes are the next customer for the same helper. - db.py: the 📎-prefix / refine-join / refine-imperative / <3-word filter moves from frontend/src/lib/suggestions.ts into db.suggestion_prompts, so GET /api/suggestions never returns noise regardless of consumer (frontend filter stays as belt-and-braces). - routes/modules.py: POST /api/modules/generate_from_file gains a `preview` form field (default false) mirroring /modules/preview — the file/sketch caller can get `previews` back instead of a direct insert (R-223). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Carried Stage-2b items (Stage-3 Task 1): - lib/sketchExport.ts: rasterScale() clamps the offscreen snap raster to ~2048px per side, returning a downscale factor (<=1) applied to both the canvas dimensions and stroke coordinates; Canvas.tsx's rasterizeSketch now applies it (R-221). - EntryScreen.tsx: a Tab/Shift+Tab focus trap + opener-restore, mirroring ConfirmDialog's pattern — EntryScreen only mounts while open, so a plain mount/unmount effect matches ConfirmDialog's open-gated lifecycle. - lib/api.ts + PromptBar.tsx: generateModuleFromFile gains an optional `preview` flag; the file-attach path now requests a preview and routes the result through the existing preview-confirm stack instead of inserting straight onto the canvas (R-223). The sketch-snap caller (Canvas.tsx) is deliberately left on direct-insert — see the Stage-3 Task 1 report for the scope decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t edge test; rate-limiter eviction Fix Round 1 (Stage-3 Task 1 review): - Swept 26 untracked iCloud "* 2.*" duplicate files (test/source/pyc/fixture copies) that pytest was auto-discovering, inflating the headline count. None were git-tracked. Real clean-tree gate: 463 passed / 94.39% (461 baseline + this round's 2 new tests). - orchestrator token cap: added the untested safety-critical branch — test_cap_composed_prompt_keeps_all_protected_content_when_it_alone_exceeds_cap pins that a ~15000-char head + non-empty exchange (protected content alone > _MAX_PROMPT_CHARS) is NEVER truncated; conversation + module-context drop to zero, result overshoots the cap by design. - _RateLimiter.allow now evicts a key whose hits trim to empty before re-adding so idle per-owner entries never accumulate; new test_rate_limiter_evicts_idle_keys pins "the map never holds an empty list", existing limiter tests stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…R-704)
Adds the last-mile actionability differentiator's first domain: Metric/Kpi/
Ring/Gauge/ProgressBar may now carry an optional `data_source` binding
(schema.py's new DataSource model, mirrored in frontend/src/lib/types.ts).
A new src/services/live_data.py fetches real values keyless via Open-Meteo
(lat/lon or place→geocode), zero-dep urllib mirroring llm.py's style — a
network/parse failure never raises, it degrades to a stale cached payload or
an honest null-value error. Fetches are cached server-side per provider+query
(new `live_cache` table, NOT per-owner — public data) with the caller's
refresh_secs as TTL.
New GET /api/live/{provider} (routes/live.py) is owner-gated and rate-limited
by reusing Task 1's `_RateLimiter`; TRUS_LIVE_DATA=off returns a disabled
marker so components fall back to manual entry (added to .env.example +
conftest isolation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an Open Food Facts fetcher to the live-data framework (Task 2),
dispatched via the existing provider map so ALLOWED_PROVIDERS picks it
up automatically. query = {food: "..."}, returns kcal/100g with the
same cache/TTL and honest-error contract as weather. GET
/api/live/{provider} now validates a `food` param for the nutrition
provider (422 if missing).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…s out-of-domain (R-702/R-705) DECOMPOSE_SYSTEM_PROMPT now teaches the two launched live-data domains (nutrition for calorie/food intent, weather for weather/trip/hike intent) with a worked example each, and explicitly forbids emitting a data_source for any other domain (stocks, flights, etc.) — R-705 honesty seam. _parse_modules gains defense-in-depth: a bad or out-of-domain data_source (strict provider Literal, bounded refresh_secs/query) is stripped from the raw dict BEFORE ModuleConfig.model_validate, so the model's mistake costs only that binding — the component survives as manual entry rather than the whole module being rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_parse_module_config (refine/single-module parse) now applies the same _sanitize_module_data_sources strip-defense as _parse_modules before ModuleConfig.model_validate. REFINE_SYSTEM_PROMPT hands the model a config that may already carry a valid data_source; a model corrupting that nested field would otherwise fail Pydantic for the whole module. Now the bad binding is stripped and the module survives as manual entry. Also pins the sanitizer against garbage input (non-dict data_source, missing provider) → stripped to None, no crash. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… + graceful failure (R-701/R-703)
New useLiveValue hook polls GET /api/live/{provider} (mount + refresh_secs,
cleaned up on unmount) and feeds MetricField/KpiField/RingField/GaugeField/
ProgressBarField a live-value path: freshness ("as of N ago") + provenance
("via Open-Meteo"/"via Open Food Facts") when fresh; last value + a muted
stale/error badge on degrade, with every manual control staying editable
(R-703 — a dead provider never breaks the module); TRUS_LIVE_DATA=off falls
back to the plain manual field with no live chrome at all. Pure logic
(relative-time formatting, the query→URL-params builder, the provider→
display-name map, shared number formatting) extracted to lib/liveFormat.ts
with vitest coverage. Folds in the 3-2 deferred Minor: the friendly source
name is computed from the known provider rather than trusting the backend's
raw-path echo on the disabled marker.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…01/R-802)
- New user_profile table (id, owner, kind, text, source, created_at,
updated_at); db.profile_list/add/update/delete/clear, all owner-scoped
(R-903). profile_add dedups (case-insensitive, same owner+kind) and
self-curates at a 50-fact/owner cap by pruning the oldest row.
- Routes: GET/POST/PATCH/DELETE /api/profile(+/{id}), all gated via
_owner_id; bad kind / over-long text -> 422.
- Accretion (R-802, Option A - verbatim, no extra model call): on
/modules/generate and /modules/preview, once the orchestrator call
resolves an exchange into modules (no ClarifyingQuestion/refusal/error),
up to 3 answered turns become visible interview-sourced profile facts,
tagged goal/fact by a light keyword heuristic. Never fires without an
exchange; best-effort so a profile write can't break a generation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…803/R-804) generate_modules composes a bounded "What I know about you:" block from db.profile_list(owner) into the seeded system message, mirroring the Stage-2b conversation block's fetch point, budgeting, and cache-key exclusion exactly (profile lives in the composed message, never the semantic-cache key, which stays the raw prompt). _cap_composed_prompt gains a profile_block param in the same lowest-priority tier as conversation, never at the expense of the user prompt or exchange answers. Never reached by the grounded-file path (no owner threaded through there — doc content already dominates). Erasure (R-804/R-1003): confirmed db.profile_clear is a hard SQL DELETE, not a soft flag — a distinctive fact is gone from profile_list AND every future generation context immediately. No full-account erasure path exists yet (grepped routes/db); DELETE /api/profile is documented as the erasure surface for now, with a note that full-account erasure is a Stage-4 item. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move the interview→profile accretion seam off /generate and /preview onto the CONFIRMED insert (POST /api/modules). InsertModulesRequest carries an optional `exchange`; when present, insert_modules accretes the interview facts (source="interview") via the existing _accrete_profile_facts helper. So accretion fires only on a proposal the user actually accepted — a discarded preview draft never enters the profile. Tests moved to the insert path; preview/generate now assert non-accretion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add ProfilePanel — a slide-in dialog (mirrors ArchivedPanel/SnapshotsPanel anatomy) listing the owner's profile facts grouped by kind, each inline- editable (PATCH) and deletable (optimistic DELETE), with a manual add (kind selector + text → POST), an empty state, and a "clear everything" behind ConfirmDialog rendered as a SIBLING of the aside (2a-3 containing-block lesson). R-1306 floor: role=dialog/aria-modal, Escape closes, focus enters the panel and is trapped inside. R-1305: matte, accent rationed to Add only. Opened from a new Sidebar "Profile" button; new `user` icon. api.ts: profileList/Add/Update/Delete/Clear. Part B seam: insertModules now carries the accepted interview `exchange`, and PromptBar passes the exchange that produced the current preview stack on accept — so profile accretion fires only on a confirmed insert, never a fresh build or a discarded draft. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(R-503/R-504) R-504: pages gain nullable portal_x/portal_y (world coords) storing a CHILD page's placement on its PARENT's canvas as an enterable portal tile. Added via the existing additive-column migration pattern (idempotent; also in the fresh CREATE TABLE), threaded through _PAGE_COLS, _page_from_row, the Page schema, and update_page (owner-scoped by the WHERE session_id — a partial update like the existing name/icon/parent_id fields). RenamePageRequest + the PATCH route pass them through. R-503 (orphan fix): parent_id is a bare column with NO FK cascade, so deleting a parent orphaned its children (parent_id → a deleted row → they vanished from the sidebar tree, which renders from root). delete_page now REPARENTS the deleted page's direct children to its OWN parent (grandparent, or NULL/root when top-level) before deleting — children move up one level, never disappear; their modules stay intact (only the deleted page's own modules cascade). Owner-scoped. Added page_module_counts(session_id) — one grouped COUNT for the portal tiles' cheap "N tools" preview without loading child module configs — + GET /api/pages/counts (owner-gated). Tests: portal persist + read-back, owner-scoped isolation, reparent-to- grandparent, reparent-to-root, child modules survive, counts owner-scoped, migration idempotent on a legacy table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Child pages (parent_id === activePageId) now render as enterable WORLD-COORD portal tiles on the parent's canvas — transformed by the SAME pan/zoom as modules and the sketch overlay — so nesting is a spatial object, not just a sidebar row (the "digital clay" feel). - New pure lib lib/portalLayout.ts: PORTAL_W/H, autoPlacePortal (grid stack near origin for un-placed tiles), portalPosition (stored portal_x/portal_y overrides auto; a half-set pair or 0,0 handled). vitest covers grid wrap + stored-vs-auto. - Canvas: a portal layer rendered BELOW the module layer (modules paint on top; a module drag wins any overlap). Pointer discrimination mirrors module drag — WINDOW listeners for the gesture, stopPropagation so a portal pointerdown never pans the canvas or reaches a module. A pointerdown that moves past a 3px threshold is a DRAG (persists placement on drop, R-504); one that doesn't is a CLICK (enter the page). R-1305: a matte, dashed "place you can enter", distinct from a solid module card, accent rationed. R-1306: each tile is focusable + Enter/Space enters. - page.tsx: childPages memo + childCounts (GET /api/pages/counts, cheap "N tools" preview, refreshed on navigation), onEnterPortal reuses the page switch (the breadcrumb is the spatially-obvious return path), onPortalMove persists optimistically. Page-delete now reparents children locally to the grandparent, mirroring the server's R-503 reparent-not-orphan so the tree stays correct. - types.ts Page mirror + api.ts gain portal_x/portal_y + pageModuleCounts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e test Fix 1 (portal occlusion, R-502): auto-placed portals shared the module grid's origin (both at world (32,96)), so portal[0] landed at the exact point of module[0] — and since modules paint on top and capture the pointer, that first tile was invisible/un-clickable until the user panned. autoPlacePortal now places tiles in a negative-Y SHELF above the module grid (rows stack upward); modules only ever occupy y >= 96, so no collision regardless of module count. Canvas.contentBounds now includes the portal shelf so fit-to-content frames it, and page.tsx defers a fit on the FIRST visit of a page that has child portals (no saved view yet) so the shelf is visible without a manual fit — a page the user has already arranged keeps its saved view (no regression). vitest updated + a guard that no auto-placed portal ever lands in the module band. Fix 2 (de-flake): test_concurrent_migration_on_stale_db_does_not_double_alter flaked red under CI load. Root cause was the workers racing the one-time `PRAGMA journal_mode = WAL` switch (not the migration lock under test), raising spurious 'database is locked'. Pre-enable WAL on the stale file so the assertion depends only on the OUTCOME (no double-ALTER / no error), which the double-checked _schema_lock guarantees regardless of interleaving; widened the race window to 0.1s for margin. Also moved test_migration_adds_portal_columns_ idempotently onto its own tmp_path DB (never the shared _db_path()) so it can't perturb the race test's isolation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
STATUS.md gains a Stage 3 section (live external data, evolving user profile, visible spatial nesting, TRUS_LIVE_DATA env) plus fresh gate numbers from this run: pytest 567 passed/2 skipped/94.80% coverage, identical across 3 consecutive runs (confirms the de-flaked migration race test is stable); mypy/ruff clean; frontend 85 passed, tsc clean, build clean. API-level smoke against an isolated spare-port backend (claim flow, real Open-Meteo + Open Food Facts fetches, weather 422 on missing params, profile CRUD + cross-owner isolation, a 3-level page tree with portal-position persistence and reparent-not-orphan on parent delete) all passed — full transcript in .superpowers/sdd/stage3-task-10-report.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-free invariant _RateLimiter.allow ran a check-then-`del self._hits[key]` eviction plus a non-atomic setdefault→trim→append on a shared dict. /transcribe and /live are sync (threadpool) routes and /live fires parallel same-owner calls on page load, so the check-then-del could race two callers into a KeyError → 500. Replace `del` with idempotent `pop(key, None)` and wrap the read-modify-write of allow() in a per-instance threading.Lock. New threaded test reproduces the old race and passes under the fix. Also pin R-903's live-cache-is-shared half (the Stage-Exit checklist claimed it was test-pinned but nothing asserted it): two different owners requesting the same provider+query share one cache row (one upstream fetch serves both) and the live_cache table carries no owner/session column — owner-free by construction. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… portal move (R-701 honesty) MetricField rendered the "live" corner tag even when the provider was down with nothing cached (showLive true, liveActive false) — badging a manual/computed value "live" is dishonest. Show "live" only when an actual live value is present or loading, else fall back to the formula label; LiveMeta's honest "via X — unavailable" line stays. Verified the other four tag-carrying primitives (KpiField already guards on live.value !== null; Gauge/Ring/ProgressBar carry no "live" text tag) are already honest. handlePortalMove logged a failed portal-position PATCH but never rolled back the optimistic move, so a drag silently reverted on next load. Capture the prior position and, on failure, restore it plus flash a low-drama notice (mirrors the module saver's conflict surface). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…age 4 R-504: cross-device spatial persistence covers portal/page positions only; the per-page viewport (pan/zoom) is still client-only localStorage — a deliberate Stage-4 item. R-802: profile accretion currently covers interview answers only; prompt + workspace-activity accretion (the reserved source enum values) is a Stage-4 item. Both added to the Stage-4/Next backlog line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the three brief differentiators onto main — 20 commits, 10 quality-gated tasks each adversarially reviewed with a final whole-branch review + fix wave.
Live external data (R-701/702/703/704/705)
data_source on Metric/Kpi/Ring/Gauge/ProgressBar; keyless weather (Open-Meteo) + nutrition (Open Food Facts) with per-provider TTL cache and honest error->stale/last-cached; GET /api/live/{provider} owner-gated + rate-limited; orchestrator emits bindings for the two launch domains only and strips out-of-domain on every parse path; frontend renders live values with freshness + provenance, degrading to a stale badge while staying manually editable. Verified against the real APIs (London 27.8C, banana 88.1 kcal).
Evolving user profile (R-801/802/803/804)
Owner-scoped user_profile store (cap50, dedup); accretion of verbatim interview answers on confirmed insert; profile feeds generation (bounded block, cache-key untouched, cross-owner isolated); inspectable/editable ProfilePanel; real erasure.
Spatial nesting (R-502/503/504)
World-coord page-portal tiles on the parent canvas (drag-persist, click-to-enter, breadcrumb return); reparent-not-orphan on parent delete.
Quality
Requirements: docs/MVP-SPEC.md. Plan: docs/superpowers/plans/2026-07-03-stage3-differentiators.md.
Generated with Claude Code