Merge develop into main#2
Open
niblyx-malnus wants to merge 142 commits into
Open
Conversation
…sistence spawn-proc now carries over existing next/skip queues instead of discarding them, so pending events from persistent subscriptions survive process restarts. MCP request handler checks for existing subscriptions via %kept before re-subscribing, preventing the wait loop from missing tool completion events after reload.
- Telegram parent nexus (telegram.hoon) manages multiple bot instances
via nested nexuses with a management UI at /ui/manage.html
- Telegram-bot child nexus: long-polls getUpdates, per-chat message
storage in messages/{chat-id}.json with embedded chat names,
sidebar chat UI, send/receive via Telegram API
- Oneshot nexus for one-off HTTP request/response patterns
- Loader: add same-file/same-fold identity transforms, %load row type
- JSON mark: crash on bad JSON instead of silently returning empty
- Root nexus: register oneshot and telegram, add %load row
- Nexus lib: reorder mana/mury type definitions
%manu in load routes through the node system — road resolves the nexus and derives the query. Root-level %manu takes neck+mana directly for callers that already know which nexus to ask. Also fix view-to-chat-data to read files from fil.contents instead of dir (which is subdirectories only).
All compiled artifacts (nexuses, MCP tools, marks, daises, naves, tubes) now flow through a single pipeline: gub/ source in Clay → build-code → bins (axal of compiled vases). Runtime lookup uses %code darts through fiberio, replacing the scattered compilation approaches that preceded it. MCP nexus rewrite: - Remove the builder process, /bin/ directory, tool-subject, and all compile-lib/write-bin/process-changes machinery (~300 lines) - get-dynamic-tools peeks ball mirror for source filenames, then looks up each compiled tool from bins via %code dart - await-tool does direct %code lookup with underscore-to-hyphen filename conversion — no subscription polling - MCP tools move from lib/nex/mcp/tools/ (compiled by builder with custom subject) to gub/lib/mcp/ (compiled by build-code with /< imports through the standard pipeline) - list-tools, call-tool, add-mcp-tool rewritten to use %code darts Nexus compilation: - Nexuses move from /sys/nexuses/ in the ball (as %temp cages) to bins at /nex/ (as compiled vases) - build-nexus reads exclusively from bins, no ball fallback - get-nexus in fiberio uses %code dart instead of ball peek - Remove rebuild-nexuses, rebuild-nexuses-incremental, rebuild-changed-caches, and the Clay sync cache rebuild step - Remove /sys/nexuses/ directory from root nexus on-load - All nexuses ported to gub/nex/ with /< imports Mark infrastructure cleanup: - Remove rebuild-tubes, rebuild-daises, rebuild-nexuses, and their helpers (build-mark-cores, build-tubes, try-build-tube, collect-nex-files) from lib/marks.hoon — all dead code since build-code handles compilation - Add dais/nave/tube convenience arms that accept a dep list directly, eliminating the cores map boilerplate in generated source - Generated dais/nave source uses tall-form %+ calls - Tube source generated with proper %mark sigils on mars pair Full %temp purge: - Remove clear-temp from lib/tarball.hoon and its tests - Remove %temp skip/reject logic from validate-new-cage, clam-cage, record-hist, collect-marks, collect-marks-shallow, cage-to-mime (in both fiberio and tarball) - Remove %temp filtering from ball-to-tarball export - No %temp references remain anywhere in the codebase New diagnostic tool: - check-bin (gub/lib/mcp/check-bin.hoon) looks up any bins entry via %code dart and returns compilation status or error tang
desk/mar/ now contains only marks needed at the Gall agent level: - Agent interface (grubbery-action, grubbery-ack) - Eyre (handle-http-request, handle-http-cancel) - On-peek returns (kids, tree, sand, born, gain, subs) - System sync (dill-told, dill-blit, jael-private-keys, jael-public-keys-result) - Common utility marks (json, mime, txt, noun, sig, hoon, ud, etc.) desk/gub/mar/ now contains only marks used within the gub/ namespace: - Domain marks (claude-*, server-*, peers-*, tool-state, etc.) - Shared utility marks Removed from both: bill, fiber-ack, jam, kelvin, keys, rails, udon, tool-args, tool-config, tool-execute (zero references anywhere). Removed dead libs: feather, feather-2, feather-icons, mcp, oneshot, s3, sigil/symbols, nex/clurd, nex/mcp, nex/server, nex/tools. Removed nex-tools import from app/grubbery.hoon. Removed dead sur/claude.hoon. 55 files, -19682 lines.
Introduce +$ built tagged union (%vase, %tang, %mime) replacing (each vase tang) across the entire codebase. Move built, bins, lode types to lib/nexus.hoon. Add six interleaved import runes to lib/build.hoon: /< name path — hoon/mime file with face /< * path — file without face (bare) /& name path — raw mime (file or dir with trailing /) /* name %mark path — file converted to mark type via tube /$ name %from %to — tube gate import /% name %mark — nave (mark core) import All imports interleaved in source order = subject order. Mark file dependencies tracked in topo-sort. Mimes are self-compiled artifacts seeded into results before the compilation loop. sync-gub now converts non-hoon Clay files to mime via tube scry. build-code passes full ball (hoon + mimes) to build-all.
Silo entries are now [refs=@ud =page] instead of [refs=@ud =cage], avoiding stale types after code recompilation. Pages are clamped back to cages via vale:dais on read (peek/peep). Delete the reload arm — on-init and on-load now inline the minimal sequence: sync-gub → validate-ball → load-ball-changes. validate-names is handled by pub in lib/tarball.hoon, and audit-weir/run-on-loads are already covered by the build-code cascade through validate-nexuses.
Instead of bricking the agent when on-load crashes or on-file/spool fails, store error tangs in boom maps and make affected processes inert. Processes skip (not black-hole) takes during boom, preserving them for replay when code heals. Also adds hardcoded %boom mark for self-healing broken cages from mark changes.
Replace single lode with code map (path -> lode) supporting multiple code nexuses. bins-get replaced by get-built which walks up the tree checking for /code children. code-in added to find which code nexus a path lives inside (used by save-file/delete to trigger rebuilds). build-tube-lazy walks up to find a code nexus with both marks in the same lode. /sys/code/ moved to /code/. All mark/tube/dais lookups, validation, and nexus building take pax=path for scoped resolution. Fix infinite loop in validate-marks: vale the noun first, then pass the old vase and vale result to validate-vase with force=%.n. When the old type nests the new, the old type is preserved — no spurious cage change, no save-file trigger, no rebuild cascade. Boom grubs extract [tang page] and vale the inner noun directly. reload-nexus-at now calls reload-new-nexuses to recursively discover and reload child nexuses created by a parent's on-load. Walks full subtree since nexuses may be deeply nested. Debug prints added to build-code, save-file, and delete for code nexus rebuild tracing.
Replace boom type from two flat maps (fold/file) with (axal [fol=(unit tang) fil=(map @ta tang)]). Add +bm door in lib/nexus.hoon with put-fold, put-file, del-file, clear, has-fold. clear uses +lop to delete the entire subtree. All boom operations in app/grubbery.hoon go through the door.
on-init and on-load were using =. state +:abet: which extracts only the state tail and discards all cards from abet, silently killing Clay/Dill/Jael subscriptions. Refactored to =^ cards state abet: pattern so all subscription cards are properly returned. Added sys.kelvin compatibility check in build-code: mismatched kelvins cause every hoon file in the nexus to compile as a tang crash with diagnostic output. Bootstrap %kelvin mark added to validate-new-cage. Graceful handling when dais is missing (returns tang instead of crash).
- Change process-take validation failure from restart to boom-file, preventing infinite fiber loops when mark infrastructure is broken - Add has-file to boom door for file-level boom checks - Hardcode bootstrap daises (%hoon, %mime, %kelvin) via Clay %cb scry so sync-clay-desk works even during kelvin mismatch - Separate cod/here in validate-ball so recursive paths stay correct - Remove validate-ball from on-init and on-load (sync handles it)
Tool source peek and write paths referenced /sys/code/lib/mcp but the code nexus lives at /code/lib/mcp. Also remove unused /cus/ directory from MCP nexus on-load and on-manu.
Thread cod=road:tarball through fiberio helpers (cage-to-mime, get-mark-conversions, get-tube, get-vale, get-nexus) so code lookups target a specific code nexus instead of using the caller's implicit scope. Migrate %code from a top-level dart type to a %node load alongside new %boom and %font loads. Add extend-road to tarball for building absolute roads from a base + inner path.
Adds +build-vale to marks.hoon which extracts noun:grab as a standalone $-(* vase) gate. build-code now builds vale gates alongside daises/naves and stores them under /val in bins.
validate-vase now takes $-(* vase) instead of dais:clay. get-dais -> get-vale, reads from /val bins instead of /das. All callers updated: validate-new-cage, clam-page, sync-clay-desk, validate-marks. validate-build now checks /val entries.
Remove dais and nave construction from the mark-cores roll. Only vale gates are built now. Remove /das and /nav validation from validate-build.
… step 4) Drop +grad requirement from validate-build mark check. Strip +grad from all 40 gub/mar/ mark files. desk/mar/ marks (Clay-facing) keep their +grad as Clay requires it.
…nation step 5) marks.hoon: remove build-nave, build-dais, nave-from, dais-from, and convenience wrappers dais/nave. Only build-vale, tube, and cast logic remains. build.hoon: remove /% nave import directive — parser, resolver, dependency tracking, and compilation. No nexus used it.
Collapse %diff into %over — they were identical except %diff enforced same-mark (which %over handles naturally). Remove from nexus types, fiberio, grubbery handler, server HTTP route, and claude nexus. Simplify %writ to unit payload. Add lazy tube building: when %code handler misses on a /tub path, call build-tube-lazy to build and cache on demand.
Replace separate /val and /tub bin paths with a single marc:tarball core per mark. Each marc bundles vale, grow, and grab into one dispatch core built at compile time. Kills build-tube-lazy and the runtime slap/slob dance for tube resolution.
- Marc is now a proper lead core (vale/grow/grab arms) instead of a triple, matching the nexus pattern. Built as a pure function of mark source inside the compile loop — no second pass, cache stores marcs directly, bins-to-cache reconstruction works correctly. - Neck type changed from @tas to rail ([path @ta]) for hierarchical nexus identity. Added rail-to-arm helper for display. - Removed dead %cast/%tube import machinery, transitive mark resolution (build-cast, compose-casts, tube-from), and the separate marc building step from grubbery.hoon. - Removed +grab/+grow validate-build check for marks — marc building in build.hoon is the real validation now.
Mark identities are now rails ([path @ta]), stored at their natural path in bins. Kills flatten-rail — marks, nexuses, and files all use the same hierarchical path model. - Added +$ blot, +$ bars, ++arm-to-rail to tarball.hoon - Marc grow/grab take blot instead of mark, use rail-to-arm for arm name resolution - validate-marks and validate-nexuses walk subtrees instead of flat node lookups - All Clay boundary call sites wrap marks as [/ mark]
…lback seek-built stops at nearest /code lode — lower namespaces must copy marks/libs they need. find-built returns namespace + source separately. %font dart returns bend (relative to asker) + source rail within code namespace. Explorer resolves to absolute paths for URL construction. Explorer: neck and mark columns link to source files via %font. Boom state displayed inline — nexus crashes as banner, file crashes as expandable details on the row. get-tube now tries target.grab(source) when source.grow(target) fails, fixing mime->mark conversions like mime->ud. check-bin MCP tool accepts optional code namespace parameter.
validate-marks and validate-nexuses now detect any change in built entries (vase→tang, tang→vase, etc) instead of only tracking vase changes. Failed marks use crash gates to propagate real error tangs through the uniform clam path. validate-nexuses booms per-directory instead of crashing the whole sync. build-nexus returns (each nexus tang) with explicit errors for all failure modes. validate-ball booms files on mark validation failure instead of silently skipping. Explorer shows nexus boom tangs in expandable details elements.
- validate-marks/validate-nexuses detect any code change (not just vase) - governed-dirs/governed-files walk ball pruning at child code boundaries - seek-built uses sibling lookup (snoc (snip pax) %code) - Root nexus hardcoded in lib/root.hoon, removed from gub/nex/ - boom-nexus sets fil booms on all files + stays all processes - File deletion clears file booms - Explorer boom rendering: modal popups instead of inline details
Each agent gets its own weir on creation instead of inheriting from the parent directory. This allows granular per-agent sandboxing. - agents-weir: standard agent sandbox (2 steps up for apis/channels) - main-agent-weir: extends agents-weir with make/poke on /agents so the main agent can create/delete sibling agents - Add default agents/main and channels/telegram/main-bot to on-load
Timer service in claw/app: receives timer-set pokes, proxies to behn (bypassing weir restrictions), pokes back with timer-wake. Encodes sender rail+wire in the behn wire for direct decode on wake — no scanning, no stale entries. State tracks [rail wire] → @da for cancellation. Cron jobs: schedule recurring messages to chats via cron expressions. cron_add/cron_list/cron_remove tools. Fibers at /proc/cron/{id} loop through timer service. Explorer: fix stab crash (split-fas helper), fix relative road parsing (parse-road-input counts ../ steps), fix render-lane double-slash and file/dir display. Fiberio: print destination and load type in veto error messages.
Behn timers, iris HTTP client, and clay desk sync now go through runtime-hooked namespace nodes instead of raw send-card/take-arvo. Fibers poke /sys/behn/, /sys/iris/, /sys/clay/, /sys/scry/ and get responses back as pokes. Runtime intercepts in handle-dart and on-arvo bridge to actual arvo cards. - /sys/behn/main.timer-state: set-timer:io, send-wait:io - /sys/iris/main.iris-state: send-request:io, take-client-response:io - /sys/clay/main.clay-state: mount/unmount via namespace pokes - /sys/clay/desks/: desk mirrors moved from /sys/clay/[desk] - /sys/scry/main.sig: scry:io as namespace alternative to do-scry - remove warp:io, take-arvo:io, cancel-request:io, get-agent:io - remove %mount-desk/%unmount-desk gall pokes - MCP clay tools use direct .^ scries instead of warp - all custom iris response matchers updated (claude, anthropic, wallet, telegram) - timer-state/iris-state/clay-state marks with JSON+mime conversion
All MCP tools and grub libraries now use scry:io (which pokes /sys/scry/main.sig) instead of .^ or do-scry:io. This ensures all scries in grub code go through the runtime namespace service.
- Dill belt service: grubs send belts via /sys/dill/main.sig instead of direct %arvo %d cards. run-dojo uses send-belt:io. - Clay new-desk service: runtime handles file scrying, desk creation, and desk.bill writing. new-desk tool reduced to 3 lines. - Remove do-scry:io (no remaining callers) - Refactor all /sys/ intercepts into handle-sys-poke dispatch arm with clean ?+ switch on service name
Gall pokes now go through the namespace runtime hook instead of direct %agent cards. The runtime encodes sender identity in the wire, emits the actual gall poke, and routes poke-acks back via the namespace. Also moves gall subscription paths from /sys/gall/[ship]/[agent]/... to /sys/gall/subs/[ship]/[agent]/... to avoid collisions with main.sig and future gall services.
Callers send pages (mark + noun) instead of cages (mark + vase). The runtime clams through the mark's dais to reconstruct typed vases before forwarding to gall. Tries our desk first for the mark, falls back to the destination agent's desk. Adds kiln mark files (commit, info, mount, nuke, permission, install, revive) to grubbery's desk so the runtime can clam hood pokes.
Clay writes now go through clay-info:io which pokes /sys/clay/ directly instead of proxying through hood. The runtime emits %c %info cards, eliminating the need for kiln-info marks (which can't exist because toro contains vases). Callers just pass [desk [path %ins mark content]~] — no vases anywhere.
…helpers Move HTTP request/cancel routing from server-state fiber into grubbery agent directly. Add get-server-state, save-server-state, find-eyre-binding, cull-if-exists helpers. Delete unused fiberio eyre card helpers (give-response-header, give-sse-*, etc). Strip debug traces.
HTTP request dispatch and cancel handling now live in the agent directly. The server-state spool retains only eyre-action (bind/unbind/send). Also removes orphaned find-eyre-binding arm.
Intercept bind/unbind/send pokes to /sys/eyre/main.server-state in the agent's dart handler, same pattern as timer-set. The server-state spool is now stay:m — pure data file, no fiber. Register /grubbery/api binding in sync-eyre instead of fiber startup.
Replace all send-cards:io calls in ball-api with http-res:io helpers that route through %eyre-action %send pokes to /sys/eyre/main.server-state. Fix %send handler to emit cards unconditionally — conns tracking is only for cancel routing, not for gating responses.
…ve gives Grubs are not gall agents — they can't watch or be watched directly, and all external IO goes through /sys/ namespace services. Remove the entire proc-wire wrapping layer and all fiber-level vane interaction. Deleted: $soup/$scry types, %soup/%scry darts, %soup/%scry/%arvo/ %agent/%watch/%leave gives, make-soup, get-soup, take-soup, take-arvo, take-agent, take-watch, take-leave, wrap-wire, unwrap-wire, wrap-watch-path, unwrap-watch-path, clean. on-arvo/on-agent catch-all fallthroughs now warn instead of forwarding to fibers.
Reorder service arms: dispatch router first, then behn, clay, eyre, gall, iris, scry. Drop stale "runtime-hooked" language — these are services, not hooks.
handle-gall-poke now looks up marks from /mar/clay/[desk]/ with fallback to /mar/clay/base/ in the code nexus. Uses segments:clay to split hyphenated marks into candidate paths. Adds base marc files (grubbery format, no +grad) for the common marks needed by gall pokes to %base agents.
All grubbery-action pokes (local and foreign) now route through /sys/ames/ships/[src]/ship.sig as darts. Our ship has no weir (full access); foreign ships get weir from usergroups. Clam vases at the weir boundary for poke, over, and make loads. - Wire-safe action chapter with bask (noun) instead of sage (vase) - Weir marc gains mime grab/grow for text-based usergroup config - Remote-make MCP tool for cross-ship file creation with mark support - Runtime services (/sys/) crash on make/cull failure instead of silently swallowing errors, so pokes properly nack
+remote core in lib/nexus.hoon holds load and intake types for cross-ship communication, mirroring internal dart/take pattern.
Every mark in nexus types was immediately wrapped in [/ mark] to become a blot at every callsite. This removes that conversion layer — blots flow end-to-end from the API boundary through types to internal logic. 36 files: nexus types, fiberio, grubbery core, all 22 nexus on-file implementations, MCP tools, and all peek/keep/make callers.
New nexus for syncing grubbery directories to/from S3 buckets. Supports mapping local paths to S3 prefixes, live sync via file watching, pull/push with a known-mark whitelist (text, wain, mime), and single-ball pull that preserves nexus structure. Also converts all nexuses from %manx to %html mark and fixes snip/type-narrowing compilation bugs across the codebase.
Add %fall entry for /sync directory in spin:loader so sync processes survive manual nexus reload. Remove debug ~& traces from bridge and grubbery.
Each code nexus now stores lightweight refs (ckey hashes) instead of full compiled vases. A shared global bins map holds the actual artifacts with refcounts. Same source + same deps = one entry in memory regardless of how many code namespaces reference it.
Global bins is content-addressed (keyed by output hash), but build-all's cache needs input-based keys. bins-to-cache now joins through keys (rail → input ckey), refs (stem → content ckey), and bins (content ckey → built) to reconstruct the correct mapping. Also: pull auto-culls destination before make, confirm dialog on pull buttons.
Unknown file extensions (jpg, png, etc) now pull as mime instead of being skipped. Adds /boom mark for error-wrapped grub content with rendered tang, blot info, and truncated data in JSON/mime output.
Switch from content-addressed ckeys (sham of built vase) to input-based ckeys from build-all. Eliminates 64s jam overhead on 824 compiled artifacts. bins-to-cache simplified from three-way join to direct lookup. Kelvin mismatch crash path removed — kelvin info stays in sys.kelvin for UI surfacing. Added bout profiling hints throughout build-code.
Stop parsing directory extensions into nexus necks on pull. Fixes .git directories getting interpreted as nexus types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keep main up to date with develop.