AGENT.md

Data UI – Agent Overview

Data UI compiles a small nested-plist model into a complete RBAC-backed application (PostgreSQL + generic backend + REST API + React frontend). The model is the DNA of the app; the compiler guarantees the expansion is consistent. Tagline: "Your whole app, in an email."

Consequence for agents: the model format is an API for a non-human consumer. An AI does not write arbitrary code into a Data UI app — it selects from a defined vocabulary (the hook registry) and fills parameters.

Where to Look (read before acting)

Detail lives in the docs, not here. Consult the right one before touching the corresponding area:

• Framing, thesis, two tiers, goals, marketplace, current status, road to MVP → README.md
• Model vocabulary — every top-level key, type key, field key, :ui subkey, forms, views, write-through, rollups, trees, join tables, known gaps → docs/model-reference.md (read before editing any model or fixture). M2M list fields: join-table row-display :source :agg is :distinct (:list there is a compile error; omitted :agg defaults to :distinct).
• Hooks and the registry — all three contracts, registry API, action hooks, the no-transactions caveat → docs/hook-registry.md (read before authoring any hook or touching hook code)
• REST endpoints — all routes with parameters → docs/rest.md
• Deployment — every step, traps, admin password location, TLS, troubleshooting, clean-slate recovery → docs/deployment.md; session-by-session history in ~/.debug/deployment-work.md. Environments (development / staging / production, :domain vs :domain-stg) are defined in its "Environments" section — read it before any deploy / expose work.
• CLI verbs — scripts/data-ui (repl, db, psql, profile, tests, deploy, delete, deploy-bootstrap, traffic, snapshot, e-demo / demo, staging update / activate-modelbank / modelbank-orphans, host profiles, environment variables; routine staging verbs route to the staging checkout) and scripts/publish-data-ui → docs/cli.md. scripts/README.md is a stub pointing there.
• Lisp style and error reporting — u: preference, small functions, report-e / report-ve, valid-* naming → docs/lisp-style.md (read before writing any Lisp: tests, helpers, or engine code with permission). When dealing with report-ve (or report-e), read the macro documentation in lisp/aux.lisp first — the tilde-prefixed var-spec convention is easy to misread as a typo and both macros signal (they do not return).
• Dead-code decisions — docs/model-accessors.md (check before removing anything that looks unused)
• Backlog — docs/todo.org
• Demo video suggestions → ~/workbench/video-plan.org — running list of beats to capture (shots, pauses, points to land). Video work starts December 15; whenever a moment in the UI or flow would demo well, add it there.
• Mel's app → ~/workbench/mels-app-notes.org — the film continuity tracker (first expressly requested app): data model with settled decisions, what works today, build checklist. Read before any "Mel's app" work.

Key Architecture

• lisp/model.lisp – model compilation and the RBAC base model (*base-model*); heal-model — idempotent schema/RBAC re-sync for a live image (the tail of set-model without the table drops)
• lisp/backend.lisp – backend functions (be-list, be-insert, be-update, be-delete, be-action, be-landing-page, ...)
• lisp/public-pages.lisp – request-time public renderer and routes (/p/, /p/files/, /sitemap.xml, /robots.txt, the :public-url wire key); pure render functions plus one thin Hunchentoot adapter; the styling sanitizer (sanitize-styling-html, the cl-sanitize-html policy plus the authoritative pass) lives here too
• lisp/rest.lisp – REST API layer (Hunchentoot handlers)
• :type-roles vs row roles — two different resources. :type-roles exist because a row that does not exist yet has no resource to check, and they answer who may create one. They do not decide access to an existing row, and they are not copied onto new rows. Read docs/roles.md before reasoning about either.
• lisp/predicates.lisp – type/field predicates; :button field type
• lisp/database.lisp – database initialization and table creation
• lisp/aux.lisp – helpers (report-e, report-ve, path utilities)
• lisp/plist-json.lisp, lisp/deployment.lisp, lisp/startup.lisp, lisp/data-ui.lisp, lisp/data-ui-package.lisp, lisp/eval-safely.lisp – guarded eval helper (safe eval returning output/values/error strings; never enters the debugger)
• models/ – example models, one per file (e.g. todos.lisp, modelbank.lisp, widgets.lisp), each a bare model plist. Load with (set-model "todos") — bare file name, no path, no .lisp extension. Test fixtures under models/test/; set-model checks models/ first, then models/local/ (VIP deploys, gitignored, shadows same-named fixtures), then falls back to models/test/. list-models returns top-level models only.
• web/ – React frontend (Vite + TypeScript), intentionally minimal and schema-driven: consumes list-form / add-form / update-form, records, and allowed-values from the API
• tests/ – FiveAM suites (30+ files: predicate-, backend-, rest-, scoping-, action-, hook-registry-, rollup, M2M, compose, sortable/searchable, guest/api-roles, generator, ...; see data-ui.asd) plus helpers.lisp and model-template.lisp

Non-frontend code is SBCL Common Lisp, written by a human.

Interaction Conventions

• AI agents must not modify any Lisp source files outside web/ and tests/helpers.lisp without special permission from a human. That code is complex and largely outside of AI's current capabilities.

• The React frontend in web/ was built with AI assistance; future frontend work will also involve AI.

• Do not refactor, clean up, or "improve" Lisp code unless explicitly instructed.

• Before any code is written, a thorough discussion of the goals must happen.

• Some functions with no in-codebase callers are intentionally retained; see docs/model-accessors.md before removing "dead" code.

• Prose in .md and .org files (docs, backlog items, plans) is never hard-wrapped: one paragraph per line, one list item per line — a newline ends a paragraph or a list item, never a sentence fragment; viewers wrap for display. See ~/r/elisp/ai/global.md → Prose Line Length.

• Emacs buffers visiting project files typically run auto-revert-mode + auto-save-visited-mode. Agent-side edits to a file whose visiting buffer is unmodified are safe — auto-revert syncs it within seconds. A modified buffer is the hazard: auto-save-visited writes it back over the disk edit. Before editing any file that has a live Emacs buffer (e.g. docs/todo.org), check with the helper (in ~/r/elisp/dc-ai.el), and if it returns non-nil, flag it to the human instead of editing:

  (eval "(ai-buffer-edit-hazard-p \"PATH\")") ; nil = safe, t = flag it
  

  The helper exists because the obvious inline forms are traps: (buffer-modified-p (find-buffer-visiting PATH)) with no visiting buffer checks the current buffer (the gptel chat buffer, always modified) — a false hazard; and hand-inlining the correct if-let shape is easy to mangle into another false hazard (a binding, not a call). One named function, no paren shape to get wrong.

Live Introspection: eval-in-data-ui

The Elisp helper eval-in-data-ui (in ~/r/elisp/dc-ai.el) evaluates a Common Lisp form against a running Data UI instance (over Slime, in the :data-ui package) and returns the result. Invoke it via the Eval tool:

(eval-in-data-ui "(a:list-role-names *rbac*)")

Package-local nicknames like a: (the rbac library) are available. Read-most: prefer it for verification and introspection over guessing from source; treat state-mutating forms with REPL-level care.

Reload before testing. After editing a Lisp file, reload it into the live image before running tests or introspecting, or you test stale code:

(eval-in-data-ui "(load \"~/common-lisp/data-ui/lisp/model.lisp\")")

Full reload: (hard-reset) (from lisp/startup.lisp) runs asdf:load-system :force t + init-database + reset-database in one call. Use it (e.g. when asdf:load-system resets *rbac* to nil) instead of the three steps manually.

Gotcha — reloading rest.lisp with the server running can nil out *http-server* while the OS still holds the port; the next set-model → start-web-server fails with ADDRESS-IN-USE-ERROR. After reloading rest.lisp, verify *http-server*; if nil, (stop-web-server) before the next set-model.

Gotcha — cross-version snapshot restore on a live image. A restore replaces the entire public schema, so tables and RBAC rows introduced by newer code than the dump (e.g. usage after the action-log work) silently vanish. Heal with (heal-model) — never set-model, whose reset-tables drops the loaded model's user tables and destroys the just-restored data. Fresh processes self-heal at startup (*compiled-model* nil → the drop is skipped), so restarting the app also works.

Gotcha — 60s Eval timeout on long forms. eval-in-data-ui times out on the Emacs side after 60s; the form keeps running in SBCL and its output is lost. (run-tests) runs all 30+ suite groups and takes several minutes. Never poll with sleep loops — after a timeout, probe with (eval-in-data-ui "(* 2 3)") before assuming the image is busy. Run long work in a thread that writes a file, then read the file:

(eval-in-data-ui "(sb-thread:make-thread (lambda ()
  (with-open-file (s \"/tmp/data-ui-test-run.txt\"
                 :direction :output :if-exists :supersede)
    (let ((*standard-output* s))
      (run-tests)
      (format s \"~%RUN-TESTS-DONE~%\")))))")

then check the file (grep RUN-TESTS-DONE). Short suites (run-m2m-tests, run-backend-tests, ...) fit the timeout directly.

Running Tests

Use the helpers in tests/helpers.lisp — never call fiveam:run! directly. They handle model loading, database reset, and suite selection.

Full suite runs go through the Bash tool: scripts/data-ui tests. After implementing any plan — or finishing any nontrivial change — run the full suite this way as the final gate before calling the work done. It uses a dedicated test database (port 5445, isolated from the REPL's 5444), runs against the code on disk rather than the live image, and fits the Bash tool's 600s timeout (~2–4 min cold load + suite). In-image (run-tests) below is the fallback.

• (run-tests) — all suite groups (backend, predicates, scoping, hook registry, lifecycle, actions, secrets, widgets, M2M, rollups, compose, sortable/searchable, guest, API-roles, generator, and more). Each group runs with-model on its own fixture (test-model, m2m-test, modelbank-test, spawn-test, measure-rollup-test, ...) — see run-tests in helpers.lisp for the full list. Resets the database and loads the fixture per group. Takes several minutes — use the background-thread pattern above.
• (run-action-tests) — action hook suite (buttons, be-action, status transitions, in-progress guard, permissions, form exclusions)
• (run-scoping-tests) — scoping suite on the modelbank-test fixture under models/test/ (not the top-level modelbank demo)

For a focused run, add a run-* helper to tests/helpers.lisp — agents are explicitly permitted to modify that file for this. Do not modify the test suites themselves without human permission.

Writing a New Test Suite

1. Create tests/<name>-tests.lisp following the existing pattern: in-package, def-suite, in-suite, then test forms using FiveAM primitives (is, is-true, is-false, signals, finishes).
2. Register the file in data-ui.asd in the tests module, after the existing test files.
3. Add a run-* helper in tests/helpers.lisp, wrapped in (with-model "test-model" nil ...).
4. Add the helper to run-tests in the same file.
5. (load "...") the file (and helpers.lisp, if changed) before running.

Rules:

• Tests run inside with-model, which resets the DB and recompiles. Base types (from *base-model*) are present in every model.
• Test models come only from models/test/ (e.g. test-model, m2m-test, static-select-test). Never point with-model / set-model at top-level demo models (todos, books, chores, modelbank, ...) — they change with product work and break tests. If a needed shape only exists in a demo, copy a minimal stable fixture into models/test/. Manual REPL smoke against a demo is fine.
• Clean up any inserted rows so tests are order-independent.
• be-types returns plists, not alists — search with (getf entry :name) as the :key, not #'car.

Feature and Issue Tracking

docs/todo.org is the canonical backlog (org TODO / PLANNING / IN-PROGRESS / READY / DONE states + tags). Refer to it when deciding what to work on next.

When docs/todo.org grows too large, DONE items are moved (cut, not copied) from its * Done section to docs/done.org, the archive. New DONE items still land in docs/todo.org first, with their :COMPLETED: drawer, and are pruned to the archive in batches. Drawer dates move with the item; fix-data-ui-todo-dates (see below) targets only docs/todo.org.

Lifecycle: TODO → PLANNING (a plan is being drafted) → READY (plan complete and approved; waiting to implement) → IN-PROGRESS (being implemented) → DONE. Plan completion and review happen at READY, not IN-PROGRESS; flip to IN-PROGRESS only when implementation actually begins.

• Every item carries a :PROPERTIES: drawer with :CREATED: (and, once done, :COMPLETED:) org inactive timestamps, e.g. [2026-08-10 Sun].
• New items: include the drawer with :CREATED: set to today. Marking DONE: set :COMPLETED: to today.
• Titles are as short as possible — a noun phrase naming the ask, long enough only to disambiguate among siblings. Spec details (:widget :markdown, :ui (:help ...), dev/stg/prod) belong in the item body, never the title: "Markdown widget", not "Markdown widget (=:widget :markdown=)".
• After any change to docs/todo.org, run (fix-data-ui-todo-dates) (defined in ~/r/elisp/dc-ai.el, no parameters). It enforces the date rules; just write today's date and run it. Currently disabled (dc-ai-fix-todo-dates-enabled is nil) — it returns 0 and touches nothing. Skip it until it is re-enabled.

Org outline hierarchy: * section, ** parent todo, *** its direct children (children inherit tags/context). Never mix — do not put ** children under a ** parent; always step to ***:

* Post MVP
** TODO Lifecycle side effects
*** TODO Cross-table upsert
*** TODO HTTP fetch/post

Working with the Frontend

• Terminology: the Home / Settings / Admin buttons in the top-right header are the view switcher (they select viewMode via switchViewMode; Home calls returnToLanding). Distinct from the type tab row below the header, which picks a type within the current view.
• Build: npm install && npm run build in web/. Hunchentoot serves web/dist/; there is no dev server. Without npm run build, frontend changes are invisible (it runs tsc, then vite build).
• The app is intentionally simple — avoid heavy routing, state libraries, or styling until MVP is proven. All forms render from the schema returned by /api/list; the frontend does not hard-code field lists.
• :ui is the extension point, passed through verbatim — new :widget values need one keyword added to the *widgets* whitelist in lisp/model.lisp (plus a frontend branch); everything downstream of the whitelist is widget-agnostic. :widget, :read-only, :not-null, and :table are consumed by the React components.
• Permission flags (create / delete / update) from /api/list control visibility of Add, Delete, and Edit controls.
• :widget :button renders a button on the edit form; onClick posts to /api/actions; disabled while status is running.
• :widget :image / :image-list render thumbnail grids with a modal lightbox; the :table key tells the frontend which type to use for /api/file URLs.

Status Digest (September 2026)

• End-to-end proven: scripts/data-ui deploy todos → https://todo.demo.data-ui.com (k3d, HAProxy, TLS) — production, always built from the production clone at /data/data-ui/repos/production. Staging (host profiles + profile expose, serving each model's :domain-stg, derived as -stg on the first DNS label of :domain) is the flagship environment during MVP: it hosts Model Bank and has the Deploy button. Demo apps run as systemd units via the e-demo / demo verbs with nightly golden resets (scripts/data-ui, ops/); host apps load the staging clone at /data/data-ui/repos/staging, never a development checkout (updated by staging update; Model Bank activates separately via staging activate-modelbank). Development Lisp work happens in the development repo.
• Full CRUD on all types (built-in RBAC types included); JWT auth (plus guest login via :guest-allowed / :guest-auto / :api-roles); view-level and field-level scoping; write-through core path; action hooks (:deploy-model, :generate-model, :spawn); rollups (Phase A); file upload / list / delete (two-phase upload: multipart to /api/upload, then JSON /api/insert with the file-token).
• Frontend: type selector with categories, dynamic lists / forms, inline edit, sortable columns, debounced search with negative ("Not…") terms, filter chips, pagination driven by total, rollup lists (server-side sortable measures), role management, image lightbox.
• Known gaps: :agg fields on regular types are not sortable (use a rollup); file update unimplemented; one flaky scoping test; UI polish pending.
• Full catalog: README → Current Status; docs/model-reference.md → Known gaps and gotchas.

Current Focus (MVP)

Deadline: complete MVP, including the demo video, by December 31, 2026.

Model Bank is the priority function — the MVP's fitness function. Gaps surfaced by building it are, by definition, the highest-priority work. Polish waits; capability gaps that block real apps do not.

1. Build Model Bank (relationships, ownership, images, ratings)
2. Write-through edge cases (e.g. clear-to-NULL)
3. UI polish (the video shows the UI)
4. The 30-second video (nothing → deployed app)
5. File update (only if time permits)

Explicitly post-MVP (do not quietly pull in): transactions / rollback around primary write + hooks + write-through; idempotent database initialization; single-statement ON CONFLICT upserts; YAML/JSON model input and the hosted AI front door.

Critical invariant — no transactions today. Hooks, write-through, and actions are NOT atomic with the primary write. A failing hook fails the operation without rolling back prior side effects. Design accordingly; see docs/hook-registry.md → MVP caveat.