Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
4e4c8ff
refactor(modules): load TinyJuice outside dependency graph
senamakel Aug 13, 2026
af92714
chore(vendor): remove TinyJuice submodule
senamakel Aug 13, 2026
3521b22
Merge remote-tracking branch 'origin/main' into tinyjuice-module
senamakel Aug 13, 2026
971d250
test(tokenjuice): preserve module-free pass-through guards
senamakel Aug 13, 2026
32930f3
fix(flows): adapt to advanced graph contract
senamakel Aug 13, 2026
66d61f0
chore(tauri): lock TinyFlows HTTP client
senamakel Aug 13, 2026
024aa3c
fix(modules): preload native test fixtures
senamakel Aug 13, 2026
39b0506
fix(ci): fetch module fixture without gh
senamakel Aug 13, 2026
768c96d
test(tokenjuice): exercise released module in CI
senamakel Aug 13, 2026
b9ee475
chore(deps): recalibrate removal simulator
senamakel Aug 13, 2026
1a7db29
fix(tokenjuice): enable explicit test module override
senamakel Aug 13, 2026
37c6c66
docs(tokenjuice): explain test module opt-in
senamakel Aug 13, 2026
2d36fed
Merge remote-tracking branch 'origin/main' into tinyjuice-module
senamakel Aug 13, 2026
7fff9f2
test(tokenjuice): isolate released module config
senamakel Aug 13, 2026
163f217
test(tokenjuice): admit release fixture before middleware
senamakel Aug 14, 2026
9a75004
ci(modules): normalize native fixture ownership
senamakel Aug 14, 2026
cdd938e
ci(modules): stage fixtures outside runner-owned checkout
senamakel Aug 14, 2026
6c99690
ci(modules): export absolute fixture paths
senamakel Aug 14, 2026
f78405f
test(tokenjuice): remove vendored fixture integration
senamakel Aug 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 40 additions & 1 deletion .github/workflows/ci-lite.yml
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,7 @@ jobs:
#
# This asserts the calibration still holds. If it fails, every
# projection built on the simulator is suspect until it is fixed.
run: python3 scripts/dep-sim.py --cut-nothing --expect-names 284
run: python3 scripts/dep-sim.py --cut-nothing --expect-names 281

- name: Guard — new feature-gated test modules must be acknowledged
# Self-maintaining coverage: the set of source files that #[cfg]-gate a test on
Expand Down Expand Up @@ -681,6 +681,45 @@ jobs:
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov

- name: Install checksum-pinned native test modules
run: |
# tinybus validates every directory ancestor. GitHub mounts the
# checkout as the host runner uid while this container runs as root,
# so a module below $GITHUB_WORKSPACE is correctly refused. Keep
# native fixtures in a root-owned, container-ephemeral tree instead.
module_root="/opt/openhuman-test-modules/${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
memory_dir="$module_root/tinymemory"
juice_dir="$module_root/tinyjuice"
memory_archive="$memory_dir/tinymemory-module-1.0.1-ubuntu-22.04-x86_64.tar.gz"
juice_archive="$juice_dir/tinyjuice-module-0.2.2-ubuntu-22.04-x86_64.tar.gz"
rm -rf "$memory_dir" "$juice_dir"
mkdir -p "$memory_dir" "$juice_dir"
curl --fail --location --silent --show-error \
"https://github.com/tinyhumansai/tinymemory/releases/download/v1.0.1/$(basename "$memory_archive")" \
--output "$memory_archive"
curl --fail --location --silent --show-error \
"https://github.com/tinyhumansai/tinyjuice/releases/download/v0.2.2/$(basename "$juice_archive")" \
--output "$juice_archive"
echo "35d13463041f455bebd833a71ba370d891a4e59d1b8e576e81635e2777a0c3dd $memory_archive" \
| sha256sum --check
echo "fd8caf7fccb53328870fd26922aa9768d253cd4b3bf758967847d6512df03863 $juice_archive" \
| sha256sum --check
# Do not restore the release builder's uid into the trusted tree.
tar --no-same-owner -xzf "$memory_archive" -C "$memory_dir"
tar --no-same-owner -xzf "$juice_archive" -C "$juice_dir"
echo "TINYMEMORY_TEST_MODULE=$memory_dir/libtinymemory_module.so" \
>> "$GITHUB_ENV"
echo "TINYJUICE_TEST_MODULE=$juice_dir/libtinyjuice_module.so" \
>> "$GITHUB_ENV"

- name: Run TinyJuice host-module regression
run: |
cargo test --lib --features modules \
openhuman::agent::tinyagents::middleware::tests::tool_output_tabulates_a_large_graph_for_a_non_exempt_tool \
-- --ignored --exact
env:
CARGO_BUILD_JOBS: "1"

- name: Run cargo llvm-cov for openhuman core (changed modules only)
run: bash scripts/ci/rust-coverage-changed.sh
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -401,7 +401,7 @@ jobs:
# fork the core image doesn't need. The Dockerfile COPYs vendor/ because
# [patch.crates-io] resolves Rust SDK crates from vendor/.
- name: Init vendored Rust submodules
run: git submodule update --init --recursive vendor/tinyagents vendor/tinyflows vendor/tinycortex vendor/tinyjuice vendor/tinychannels vendor/tinyplace vendor/tinyhumans-sdk vendor/tinybus vendor/tinymemory
run: git submodule update --init --recursive vendor/tinyagents vendor/tinyflows vendor/tinycortex vendor/tinychannels vendor/tinyplace vendor/tinyhumans-sdk vendor/tinybus vendor/tinymemory
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Log in to GHCR
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ jobs:
# fork the core image doesn't need. The Dockerfile COPYs vendor/ because
# [patch.crates-io] resolves Rust SDK crates from vendor/.
- name: Init vendored Rust submodules
run: git submodule update --init --recursive vendor/tinyagents vendor/tinyflows vendor/tinycortex vendor/tinyjuice vendor/tinychannels vendor/tinyplace vendor/tinyhumans-sdk vendor/tinybus vendor/tinymemory
run: git submodule update --init --recursive vendor/tinyagents vendor/tinyflows vendor/tinycortex vendor/tinychannels vendor/tinyplace vendor/tinyhumans-sdk vendor/tinybus vendor/tinymemory
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
- name: Build image (no push)
Expand Down
3 changes: 0 additions & 3 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@
[submodule "vendor/tinycortex"]
path = vendor/tinycortex
url = https://github.com/tinyhumansai/tinycortex
[submodule "vendor/tinyjuice"]
path = vendor/tinyjuice
url = https://github.com/tinyhumansai/tinyjuice
[submodule "vendor/tinychannels"]
path = vendor/tinychannels
url = https://github.com/tinyhumansai/tinychannels
Expand Down
35 changes: 3 additions & 32 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 1 addition & 6 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -159,10 +159,6 @@ tinyplace = "2.0"
# Optional: exclusive to the default-ON `flows` feature (#4797). A slim build
# without `flows` drops this crate and its `jaq-*` JSON-query stack entirely.
tinyflows = { version = "0.6", features = ["mock"], optional = true }
# TinyJuice — host-agnostic TokenJuice compression engine. OpenHuman keeps
# config/RPC/tool/runtime adapters in `src/openhuman/tokenjuice/` and patches
# this dependency to the vendored submodule below.
tinyjuice = { version = "0.2.1", default-features = false }
# TinyAgents — Rust LLM orchestration framework (LangGraph/LangChain-style):
# durable state graphs, agent-loop harness, model/tool registries, REPL +
# `.rag` workflow language. openhuman's agent engine + orchestration run on this
Expand Down Expand Up @@ -1048,7 +1044,7 @@ motosan-ai-oauth = { path = "vendor/motosan-ai-oauth" }
# version in lockstep with the `tinyagents` requirement above. After cloning:
# `git submodule update --init vendor/tinyagents` (worktrees included).
tinyagents = { path = "vendor/tinyagents" }
# TinyFlows, TinyCortex, TinyJuice, TinyChannels, and TinyPlace are vendored beside
# TinyFlows, TinyCortex, TinyChannels, and TinyPlace are vendored beside
# TinyAgents so integration work can test crate changes against OpenHuman before
# publishing.
tinyflows = { path = "vendor/tinyflows" }
Expand All @@ -1057,7 +1053,6 @@ tinycortex = { path = "vendor/tinycortex" }
# depends on it by path, above). Patch it onto the same checkout so both see one
# copy of the contract types and the trait identities unify.
tinycortex-api = { path = "vendor/tinycortex/api" }
tinyjuice = { path = "vendor/tinyjuice" }
tinychannels = { path = "vendor/tinychannels" }
tinyplace = { path = "vendor/tinyplace/sdk/rust" }

Expand Down
26 changes: 0 additions & 26 deletions app/src-tauri/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 1 addition & 2 deletions app/src-tauri/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ motosan-ai-oauth = { path = "../../vendor/motosan-ai-oauth" }
# released tag) — same patch as the root Cargo world so both resolve the
# in-tree SDK source. `git submodule update --init vendor/tinyagents` first.
tinyagents = { path = "../../vendor/tinyagents" }
# TinyFlows, TinyCortex, TinyJuice, TinyChannels, and TinyPlace are vendored beside
# TinyFlows, TinyCortex, TinyChannels, and TinyPlace are vendored beside
# TinyAgents so integration work can test crate changes against OpenHuman before
# publishing.
tinyflows = { path = "../../vendor/tinyflows" }
Expand All @@ -263,7 +263,6 @@ tinycortex = { path = "../../vendor/tinycortex" }
# patch it onto the same checkout here too, or this independent Cargo world
# tries to resolve it from crates.io and fails (it was never published).
tinycortex-api = { path = "../../vendor/tinycortex/api" }
tinyjuice = { path = "../../vendor/tinyjuice" }
tinychannels = { path = "../../vendor/tinychannels" }
tinyplace = { path = "../../vendor/tinyplace/sdk/rust" }

Expand Down
8 changes: 7 additions & 1 deletion scripts/kernel-floor.limits
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,12 @@
# Simulate with: scripts/dep-sim.py --cut <crates>
#
# History
# 304/281/2 2026-08-14 TinyJuice moved behind the TinyBus module boundary
Comment thread
coderabbitai[bot] marked this conversation as resolved.
# (-3 packages / -3 names). The `tinyjuice`,
# `tinyjuice-tokenizer`, and `tinyjuice-vector`
# crates no longer enter the always-on `flows` graph;
# OpenHuman keeps only its stable wire types and host
# adapter. Measured with `scripts/kernel-floor.sh flows`.
# 308/284/2 2026-08-13 tinyflows' host-stack advance (PR #5537) moved its
# direct HTTP client from reqwest 0.12 to 0.13 while the
# rest of the kernel still uses 0.12. That creates one
Expand Down Expand Up @@ -255,4 +261,4 @@
# (libsqlite3-sys, ring) — see docs/plans MIGRATION-PLAN G6.
# 307/284 2026-08-12 Re-baseline after the upstream lockfile resolution;
# `flows` remains at two native packages.
flows:308:284:2
flows:304:281:2
6 changes: 0 additions & 6 deletions src/core/jsonrpc.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2058,12 +2058,6 @@ fn register_domain_subscribers(
// battery-powered hosts).
crate::openhuman::cron::scheduler_gate::init_global(&config);

// Install the TokenJuice content-router runtime config (compressor
// toggles + CCR cache limits + optional on-disk tier). Compaction runs on
// every agent's tool output, so this must be set before any agent loop
// executes a tool.
crate::openhuman::inference::tokenjuice::install_from_config(&config);

// Seed the scheduler-gate signed-out override from the on-disk session.
// Without this, a sidecar that boots with no stored JWT would happily
// spin up cron / channel loops and fire LLM requests that all 401.
Expand Down
10 changes: 10 additions & 0 deletions src/openhuman/agent/tinyagents/middleware.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3654,10 +3654,20 @@ mod tests {
}

#[tokio::test]
#[ignore = "requires a built TinyJuice module"]
Comment thread
coderabbitai[bot] marked this conversation as resolved.
async fn tool_output_tabulates_a_large_graph_for_a_non_exempt_tool() {
// Sanity baseline proving this test's payload actually exercises real
// tinyjuice tabulation (and isn't just below-threshold): a tool name
// NOT in COMPACTION_EXEMPT_TOOLS loses the `"type"` marker.
// Resolve the explicit release fixture before `after_tool` performs
// ambient config initialisation. A pristine CI workspace otherwise
// exercises the production fail-open path before the test override is
// admitted, hiding a usable module behind unchanged output.
crate::openhuman::inference::tokenjuice::install_from_config(
&crate::openhuman::config::Config::default(),
)
.await
.expect("released TinyJuice module must load and accept host configuration");
let mw = compaction_enabled_mw();
let payload = large_workflow_proposal_json();
assert!(
Expand Down
4 changes: 2 additions & 2 deletions src/openhuman/config/schema/tokenjuice.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
//!
//! Controls the TinyJuice content-aware tool-output compaction engine: which
//! compressors are enabled, the Compress-Cache-Retrieve (CCR) store limits, and
//! the opt-in Python/ML plain-text compressor. Installed into the runtime at
//! startup via [`crate::openhuman::inference::tokenjuice::install_from_config`].
//! the opt-in Python/ML plain-text compressor. The host applies these settings
//! before module calls via [`crate::openhuman::inference::tokenjuice::install_from_config`].

use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
Expand Down
28 changes: 15 additions & 13 deletions src/openhuman/inference/tokenjuice/README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
# OpenHuman TokenJuice Adapter

The reusable compression engine now lives in the vendored `tinyjuice` crate at
`vendor/tinyjuice` and is patched through Cargo. This directory is the
OpenHuman adapter layer.
The reusable compression engine ships as the separately released `tinyjuice`
TinyBus module. It is not linked into OpenHuman's dependency graph. This
directory is the host adapter and shared wire-contract layer.

OpenHuman-owned files:

| Path | Role |
| --- | --- |
| `mod.rs` | Stable OpenHuman module seam, TinyJuice re-exports, config-to-engine install hook, ML/savings callback wiring. |
| `mod.rs` | TinyBus calls, config installation, pass-through fallback, and savings wiring. |
| `types.rs` | Dependency-free copy of the stable JSON wire contract. |
| `schemas.rs` | JSON-RPC controller schemas and handlers. |
| `config_patch.rs` | Partial update shape for the `[tokenjuice]` config block. |
| `tools.rs` | OpenHuman agent tool implementation for `tokenjuice_retrieve`. |
Expand All @@ -17,14 +18,15 @@ OpenHuman-owned files:

TinyJuice-owned engine pieces:

| TinyJuice path | Role |
| TinyJuice repository path | Role |
| --- | --- |
| `vendor/tinyjuice/src/compress.rs` | Content router entry point. |
| `vendor/tinyjuice/src/compressors/` | JSON, code, log, search, diff, HTML, ML slot, and generic compressors. |
| `vendor/tinyjuice/src/cache/` | CCR store, retrieval markers, disk tier, ranged retrieval helpers. |
| `vendor/tinyjuice/src/rules/` | Rule loader/compiler and embedded rule table. |
| `vendor/tinyjuice/src/vendor/rules/*.json` | Vendored upstream rule JSON files. |
| `vendor/tinyjuice/src/detect/`, `text/`, `tokens.rs`, `types.rs` | Detection, text helpers, token estimates, public types. |
| `src/compress.rs` | Content router entry point. |
| `src/compressors/` | JSON, code, log, search, diff, HTML, ML slot, and generic compressors. |
| `src/cache/` | CCR store, retrieval markers, disk tier, ranged retrieval helpers. |
| `src/rules/` | Rule loader/compiler and embedded rule table. |
| `src/vendor/rules/*.json` | Vendored upstream rule JSON files. |
| `src/detect/`, `text/`, `tokens.rs`, `types.rs` | Detection, text helpers, token estimates, public types. |

Do not add OpenHuman runtime dependencies to TinyJuice. Runtime services,
settings persistence, JSON-RPC, tools, and pricing stay in this adapter.
Do not add the `tinyjuice` crate back to OpenHuman. Runtime services, settings
persistence, JSON-RPC, tools, pricing, and the optional ML callback stay here;
engine behavior stays behind the loadable module boundary.
Loading
Loading