feat(introspection): slim, just-in-time engine introspection worker

introspection/Cargo.toml

@@ -0,0 +1,26 @@
+[workspace]
+
+[package]
+name = "iii-introspection"
+version = "0.1.0"
+edition = "2021"
+description = "Slim, streamable, just-in-time engine introspection worker for iii"
+license = "Apache-2.0"
+authors = ["Rohit Ghumare <ghumare64@gmail.com>"]
+publish = false
+
+[[bin]]
+name = "iii-introspection"
+path = "src/main.rs"
+
+[dependencies]
+iii-sdk = "=0.11.6"
+tokio = { version = "1", features = ["rt-multi-thread", "macros", "sync", "signal"] }
+serde = { version = "1", features = ["derive"] }
+serde_json = "1"
+anyhow = "1"
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+clap = { version = "4", features = ["derive"] }
+reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }
+serde_yaml = "0.9"

introspection/README.md

@@ -0,0 +1,59 @@
+# iii-introspection
+
+Slim, streamable, just-in-time engine introspection worker.
+
+Wraps `engine::workers::list` with progressive disclosure: slim by default, full schema only on `describe`. Solves Mike's "context bloat from dumping every function schema" gripe (May 8 sync).
+
+## Functions
+
+| Function ID | Purpose |
+|---|---|
+| `introspection::workers::list` | Slim worker list (name, status, function_count, description). `include=full` for raw `engine::workers::list` graph. |
+| `introspection::workers::describe` | One worker, full detail. |
+| `introspection::functions::list` | Slim function list (id + description). Optional `worker` filter, `filter` substring. |
+| `introspection::functions::describe` | Just-in-time full schema for one function id. |
+| `introspection::stream::subscribe` | Snapshot today. Live stream lands when engine emits on pubsub channel `introspection.registrations`. |
+| `introspection::registry::query` | Search `workers.iii.dev/registry/index.json`. |
+
+## Why this worker
+
+Mike (May 8 sync): *"introspection is tell me everything static about my engine and stream new things about my engine"*.
+
+Today the agent calls `engine::workers::list` → response includes every function's full request/response schema. Context bloat. Mike's gripe: *"only the function names and descriptions is necessary, and then it dives deeper if we find the right function"*.
+
+This worker enforces progressive disclosure at the introspection boundary:
+
+1. `introspection::functions::list` → slim ids only.
+2. Agent picks one. 
+3. `introspection::functions::describe id` → full schema only for the chosen function.
+4. Agent calls.
+
+Same shape as the progressive-disclosure pattern used elsewhere: descriptions in context, full schemas on demand.
+
+## Build
+
+```bash
+cd workers/introspection
+cargo build --release
+```
+
+## Run
+
+```bash
+./target/release/iii-introspection --url ws://127.0.0.1:49134 --config ./config.yaml
+```
+
+## SDK
+
+Pinned to `iii-sdk = "=0.11.6"` (max stable on crates.io as of 2026-05-11). Engine HEAD is `0.11.7-next.1` but unreleased.
+
+## Config
+
+```yaml
+registry_url: https://workers.iii.dev
+default_timeout_ms: 5000
+```
+
+## License
+
+Apache-2.0.

introspection/iii.worker.yaml

@@ -0,0 +1,8 @@
+iii: v1
+name: introspection
+language: rust
+deploy: binary
+manifest: Cargo.toml
+bin: iii-introspection
+description: Slim, just-in-time engine introspection — workers, functions, registrations, registry query.
+dependencies: {}

introspection/skill.md

@@ -0,0 +1,45 @@
+---
+id: iii://introspection
+title: Engine introspection (slim + just-in-time)
+description: Discover workers and functions on the running iii engine without dumping every schema into context.
+---
+
+# introspection
+
+Use this worker when you need to know what is alive on the engine. It does
+progressive disclosure so the agent context stays small.
+
+## When to use
+
+- The user asks what capabilities are available.
+- You need to find a function by keyword before calling it.
+- You want to discover new functions registered at runtime.
+- You need to search the public registry (`workers.iii.dev`) for a capability
+  that is not installed yet.
+
+## Functions
+
+| Function | Use it for |
+|---|---|
+| `introspection::workers::list` | One-line per worker: name, status, function_count, description. Default slim. Pass `{"include":"full"}` only when raw graph is required. |
+| `introspection::workers::describe` | Full detail for one named worker. |
+| `introspection::functions::list` | Slim function index `{id, worker, description}[]`. Optional `worker` filter and `filter` substring. |
+| `introspection::functions::describe` | Just-in-time. Returns full request and response schemas for one function id. |
+| `introspection::stream::subscribe` | Snapshot of current registrations. Switches to live deltas when the engine emits on the `introspection.registrations` channel. |
+| `introspection::registry::query` | Search `workers.iii.dev/registry/index.json` by name and description. |
+
+## Recommended flow
+
+1. `introspection::functions::list` with the user's keyword in `filter`.
+2. Pick one id from the slim list.
+3. `introspection::functions::describe` with that id to get full schemas.
+4. Call the function.
+
+This is the progressive-disclosure pattern. Descriptions stay in context,
+heavy schemas load only on demand.
+
+## Anti-pattern
+
+Do not call `engine::workers::list` directly when you only need ids and
+descriptions. That dumps every request and response schema into context for
+every function on the engine. Use `introspection::functions::list` instead.

introspection/src/config.rs

@@ -0,0 +1,38 @@
+use std::path::Path;
+
+use serde::Deserialize;
+
+#[derive(Debug, Clone, Deserialize)]
+pub struct Config {
+    #[serde(default = "default_registry")]
+    pub registry_url: String,
+    #[serde(default = "default_timeout")]
+    pub default_timeout_ms: u64,
+}
+
+fn default_registry() -> String {
+    "https://workers.iii.dev".to_string()
+}
+
+fn default_timeout() -> u64 {
+    5000
+}
+
+impl Default for Config {
+    fn default() -> Self {
+        Self {
+            registry_url: default_registry(),
+            default_timeout_ms: default_timeout(),
+        }
+    }
+}
+
+pub fn load(path: &str) -> anyhow::Result<Config> {
+    let p = Path::new(path);
+    if !p.exists() {
+        anyhow::bail!("config not found: {}", path);
+    }
+    let text = std::fs::read_to_string(p)?;
+    let cfg: Config = serde_yaml::from_str(&text).unwrap_or_default();
+    Ok(cfg)
+}

introspection/src/functions/context.rs

@@ -0,0 +1,187 @@
+//! `introspection::context::bootstrap` — compact one-shot context for an
+//! agent at session start. Designed to replace the harness's giant skill
+//! body dump in the system prompt: < 4KB instead of 50-80KB.
+//!
+//! Returns:
+//!   - connected worker names + brief description
+//!   - engine builtins with activation hints (so agent stops trying to
+//!     `sandbox::create` when iii-sandbox config block isn't enabled)
+//!   - top-level skill index (ids only, no bodies)
+//!   - canonical discovery flow ("call introspection::functions::list first")
+
+use std::sync::Arc;
+
+use iii_sdk::{IIIError, III};
+use serde_json::{json, Value};
+
+use super::{builtin_hint, ENGINE_BUILTINS};
+
+pub async fn bootstrap(iii: Arc<III>, _payload: Value) -> Result<Value, IIIError> {
+    let raw = super::call(&iii, "engine::workers::list", json!({}))
+        .await
+        .map_err(|e| IIIError::Handler(format!("engine::workers::list failed: {e}")))?;
+
+    let workers = raw
+        .get("workers")
+        .and_then(|v| v.as_array())
+        .cloned()
+        .unwrap_or_default();
+
+    let mut connected_map: std::collections::HashMap<String, Value> =
+        std::collections::HashMap::new();
+    let mut live_names: std::collections::HashSet<String> = Default::default();
+
+    for w in workers {
+        let name = w
+            .get("name")
+            .and_then(|n| n.as_str())
+            .unwrap_or("")
+            .to_string();
+        let status = w.get("status").and_then(|s| s.as_str()).unwrap_or("");
+        if name.is_empty() || is_anonymous_name(&name) {
+            continue;
+        }
+        let live = matches!(status, "connected" | "available");
+        if !live {
+            continue;
+        }
+        live_names.insert(name.clone());
+        let entry = json!({
+            "name": name.clone(),
+            "status": status,
+            "fn_count": w
+                .get("function_count")
+                .cloned()
+                .or_else(|| {
+                    w.get("functions")
+                        .and_then(|f| f.as_array())
+                        .map(|a| json!(a.len()))
+                })
+                .unwrap_or(json!(0)),
+            "description": w.get("description").cloned(),
+        });
+        // Dedup: prefer the row with the highest fn_count for the same name.
+        match connected_map.get(&name) {
+            Some(prev) => {
+                let prev_count = prev
+                    .get("fn_count")
+                    .and_then(|v| v.as_u64())
+                    .unwrap_or(0);
+                let new_count = entry.get("fn_count").and_then(|v| v.as_u64()).unwrap_or(0);
+                if new_count > prev_count {
+                    connected_map.insert(name, entry);
+                }
+            }
+            None => {
+                connected_map.insert(name, entry);
+            }
+        }
+    }
+    let mut connected: Vec<Value> = connected_map.into_values().collect();
+    connected.sort_by(|a, b| {
+        a.get("name")
+            .and_then(|v| v.as_str())
+            .unwrap_or("")
+            .cmp(b.get("name").and_then(|v| v.as_str()).unwrap_or(""))
+    });
+
+    // Truly disabled engine builtins: known builtins that are NOT present in
+    // workers list at all. Core builtins (iii-state, iii-pubsub, iii-stream,
+    // iii-queue, iii-engine-functions, iii-worker-manager, iii-console) are
+    // always available so they appear in `connected` above; only iii-sandbox /
+    // iii-http / iii-cron (config-block-gated) typically land here.
+    let mut not_registered_builtins: Vec<Value> = Vec::new();
+    for (n, hint) in ENGINE_BUILTINS {
+        if !live_names.contains(*n) {
+            not_registered_builtins.push(json!({
+                "name": n,
+                "status": "not_registered",
+                "activation_hint": hint,
+            }));
+        }
+    }
+
+    // Top-level skill index (no bodies).
+    let skills = super::call(&iii, "skills::list", json!({}))
+        .await
+        .ok()
+        .and_then(|v| v.get("skills").and_then(|s| s.as_array()).cloned())
+        .unwrap_or_default();
+    let skill_index: Vec<Value> = skills
+        .iter()
+        .filter_map(|s| {
+            let id = s.get("id").and_then(|v| v.as_str())?;
+            // root-level only — `auth-credentials/get_token` etc are children;
+            // agent fetches them via skill::fetch when needed.
+            if id.contains('/') {
+                return None;
+            }
+            Some(json!({"id": id, "uri": format!("iii://{id}")}))
+        })
+        .collect();
+
+    // Highlight a few critical hints inline so the agent doesn't have to
+    // bootstrap a long discovery sequence on simple asks.
+    let pinned_tips = json!([
+        "Use `introspection::functions::list { filter: \"<keyword>\" }` for discovery; do NOT call `engine::functions::list` (52KB+).",
+        "Once you pick an id, call `introspection::functions::describe { id: \"…\" }` for its schema.",
+        "If `iii-sandbox` is missing, you cannot `sandbox::*` — fall back to `shell::bash::exec` or `shell::exec`.",
+        "If a worker name appears in `engine_builtins_disabled` here, the engine config doesn't enable it yet; suggest the user re-launch the engine with `--config <yaml>` and the matching block.",
+        "todo functions DO NOT exist by default — build CRUD using `state::set`/`state::get`/`state::list`/`state::delete` keyed under `todo/*`.",
+    ]);
+
+    Ok(json!({
+        "tips": pinned_tips,
+        "connected_workers": connected,
+        "engine_builtins_disabled": not_registered_builtins,
+        "skill_index": skill_index,
+        "discovery_flow": [
+            "introspection::context::bootstrap",
+            "introspection::functions::list { filter: '<keyword>' }",
+            "introspection::functions::describe { id }",
+            "agent_call { function, payload }",
+        ],
+    }))
+}
+
+fn is_anonymous_name(name: &str) -> bool {
+    let bytes = name.as_bytes();
+    if let Some(pos) = name.rfind(':') {
+        let after = &name[pos + 1..];
+        if !after.is_empty() && after.chars().all(|c| c.is_ascii_digit()) {
+            if pos == 0 || bytes[pos - 1] != b':' {
+                return true;
+            }
+        }
+    }
+    false
+}
+
+pub async fn worker_status(iii: Arc<III>, payload: Value) -> Result<Value, IIIError> {
+    let name = payload
+        .get("name")
+        .and_then(|v| v.as_str())
+        .ok_or_else(|| IIIError::Handler("missing required field: name".into()))?;
+    let raw = super::call(&iii, "engine::workers::list", json!({}))
+        .await
+        .map_err(|e| IIIError::Handler(format!("engine::workers::list failed: {e}")))?;
+    let workers = raw
+        .get("workers")
+        .and_then(|v| v.as_array())
+        .cloned()
+        .unwrap_or_default();
+    let row = workers.iter().find(|w| {
+        w.get("name").and_then(|n| n.as_str()) == Some(name)
+    });
+    let status = row
+        .and_then(|w| w.get("status").and_then(|s| s.as_str()))
+        .unwrap_or("not_registered")
+        .to_string();
+    let hint = builtin_hint(name);
+    Ok(json!({
+        "name": name,
+        "status": status,
+        "builtin": hint.is_some(),
+        "activation_hint": hint,
+    }))
+}