Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ provider over some other base URL need a rule for it, and a rule that names a
port matches only that port:

```bash
pxpipe warp --route '127.0.0.1:8082/v1/*=http://127.0.0.1:47821' -- codex
pxpipe warp --route '127.0.0.1:9090/v1/*=http://127.0.0.1:47821' -- codex
```

## Offline export (no proxy)
Expand Down Expand Up @@ -175,6 +175,7 @@ is confabulations, so lower is better.
| `grok-4.5` | **100/100** | **97/98** | 17/18 | **0/16** | 0/15 | native 14px/84 quality suite (live profile); [quality](eval/grok-density/QUALITY_RESULTS.md), [native-sweep](eval/grok-density/native-sweep/RESULTS.md) |
| `grok-4.6` high | **100/100** | **97/98** | 17/18 | **0/16** | 0/15 | native 14px/84, reasoning high; [quality](eval/grok-profile/QUALITY_RESULTS.md) |
| `moonshotai/kimi-k3` | 79/100 | 84/98 | 15/18 | 1/16 | 0/15 | generic GPT profile: [quality results](eval/sol-profile/KIMI_K3_QUALITY_RESULTS.md) |
| `qwen-3.8` (`@cf/qwen/qwen3.8-27b`) | 98/100 | 72/98 | 11/18 | **0/16** | 0/15 | prior 5×8 broad suite (0/15 hex); native 14px pilot: 8/8 exact, 0 inventions, 11/15 hex: [pilot & quality](eval/qwen-profile/QUALITY_RESULTS.md) |

### Native-profile cost check

Expand Down
3 changes: 2 additions & 1 deletion eval/gemini-profile/gemini-client.mjs
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// Google AI Studio eval client through the local Cloudflare AI Gateway.

function gatewayOrigin() {
const base = (process.env.OPENAI_BASE_URL || 'http://127.0.0.1:8082/v1').replace(/\/$/, '');
const base = (process.env.OPENAI_BASE_URL ?? '').replace(/\/$/, '');
if (!base) throw new Error('OPENAI_BASE_URL is required (point it at your gateway)');
return new URL(base).origin;
}

Expand Down
2 changes: 1 addition & 1 deletion eval/gist-recall/OPUS_14PX.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ density, matching how fable's matrix cells are scored at fable's `5x8`.
pageWidthPx}`; `run*.py` **preflight-aborts before any API call** if the
corpus manifest model ≠ the model under test (guards silent profile/model
mismatch).
- Model reached via **upstream (`:8082`), pxpipe-bypassed** — so pxpipe's own
- Model reached via **the gateway upstream, pxpipe-bypassed** — so pxpipe's own
text→image rendering does not contaminate the arms; the model still gets the
images. Concurrency 3, `CCI_READY_TIMEOUT=120`, `CCI_TIMEOUT=210`.
- Raw results: `work/results.opus-14px.jsonl`, `work2/…`, `work3/…`.
Expand Down
2 changes: 1 addition & 1 deletion eval/glyph-matrix/PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
## What this measures

Per-character confusion matrix for 12-char hex IDs read back through the
pixelpipe render path, across render-style arms — the precision-tier
pxpipe render path, across render-style arms — the precision-tier
failure mode that gist-recall (98/98) structurally cannot surface and
needle-haystack only bracketed (~87% exact-match on worst-case hex at
prod style).
Expand Down
2 changes: 1 addition & 1 deletion eval/grok-density/QUALITY_SUITE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Grok is evaluated through the OpenAI-compatible Responses endpoint used by
Codex. Fable and Opus use the Claude harnesses.

```bash
export OPENAI_BASE_URL=http://127.0.0.1:8082/v1
export OPENAI_BASE_URL=http://127.0.0.1:<GATEWAY_PORT>/v1
export OPENAI_API_KEY=…
export SOL_QUALITY_MODEL=grok-4.5
export SOL_QUALITY_LIVE=1
Expand Down
2 changes: 1 addition & 1 deletion eval/grok-density/native-sweep/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ atlas swap → rebuild → render, live ask locks answers before truth is scored
```bash
node eval/grok-density/native-sweep/gen-fixture.mjs
node eval/grok-density/native-sweep/render-ladder.mjs
OPENAI_BASE_URL=http://127.0.0.1:8082/v1 OPENAI_API_KEY=… \
OPENAI_BASE_URL=http://127.0.0.1:<GATEWAY_PORT>/v1 OPENAI_API_KEY=… \
bash eval/grok-density/native-sweep/_ask_all.sh
node eval/grok-density/native-sweep/score-all.mjs
```
Expand Down
4 changes: 2 additions & 2 deletions eval/grok-density/native-sweep/RESULTS.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Grok 4.5 native JetBrains Mono 8–16px blind sweep

Date: 2026-07-23. Model: `grok-4.5` via direct Responses (`OPENAI_BASE_URL`
upstream `:8082`, pxpipe port 47821 rejected by the client).
upstream local gateway; pxpipe port 47821 rejected by the client).

## Protocol (same spirit as Opus / Sol native sweeps)

Expand Down Expand Up @@ -58,7 +58,7 @@ evidence; this sweep is the native-size negative result.
# atlases already built under eval/opus-density/native-sweep/atlases (symlinked)
node eval/grok-density/native-sweep/gen-fixture.mjs # fresh truth (optional)
node eval/grok-density/native-sweep/render-ladder.mjs # swaps atlas*, rebuilds, restores
OPENAI_BASE_URL=http://127.0.0.1:8082/v1 OPENAI_API_KEY=… \
OPENAI_BASE_URL=http://127.0.0.1:<GATEWAY_PORT>/v1 OPENAI_API_KEY=… \
bash eval/grok-density/native-sweep/_ask_all.sh # locks answers-*.json
node eval/grok-density/native-sweep/score-all.mjs
```
Expand Down
2 changes: 1 addition & 1 deletion eval/grok-density/native-sweep/ask.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Live-call grok-4.5 on one rendered rung. Blind: never opens truth.json.
// Writes answers-<label>.json with per-token answer + conf.
// Run: OPENAI_BASE_URL=http://127.0.0.1:8082/v1 OPENAI_API_KEY=… \
// Run: OPENAI_BASE_URL=http://127.0.0.1:<GATEWAY_PORT>/v1 OPENAI_API_KEY=… \
// node eval/grok-density/native-sweep/ask.mjs jbmono14
import { readFileSync, writeFileSync, readdirSync } from 'node:fs';
import { dirname, join } from 'node:path';
Expand Down
2 changes: 1 addition & 1 deletion eval/grok-profile/QUALITY_RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Receipts:
- `verbatim-hex-grok-4.6-results.json`

```bash
export OPENAI_BASE_URL=http://127.0.0.1:8082/v1
export OPENAI_BASE_URL=http://127.0.0.1:<GATEWAY_PORT>/v1
export OPENAI_API_KEY=…
GROK_QUALITY_LIVE=1 N=100 node eval/grok-profile/novel-arithmetic.mjs
GROK_QUALITY_LIVE=1 node eval/grok-profile/gist-recall.mjs
Expand Down
33 changes: 33 additions & 0 deletions eval/qwen-profile/QUALITY_RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Qwen 3.8 quality results

Model: `workers-ai/@cf/qwen/qwen3.8-27b` (`qwen 3.8`) through Cloudflare Workers AI (via a local gateway).

## Benchmark Summary

| Geometry | Arithmetic (N=100) | Gist (N=98) | State (N=18) | Never-Stated (N=16) | Dense Hex (N=15) | Paired Pilot (8 facts) |
|---|---:|---:|---:|---:|---:|---:|
| **Spleen 5×8 (152 cols)** | 98/100 | 72/98 | 11/18 | **0/16** | 0/15 | 0/8 exact (unreadable / confabulated) |
| **JetBrains Mono 14px (84 cols)** | — | — | — | — | **11/15** | **8/8 exact (100%)**, 2/2 gist, 2/2 guard |

## Paired Pilot (Alpha / Beta)

Evaluated against the paired synthetic terminal pilot (Alpha & Beta sessions with 4 exact-fact extractions + 1 gist + 1 unstated guard each):

| Profile | Fixture | Exact Facts (4) | Gist | Guard (NOT STATED) | Notes |
|---|---|---:|:---:|:---:|---|
| Spleen 5×8 (152 cols) | Alpha | 0/4 | fail | fail | Raw output empty; font unreadable |
| Spleen 5×8 (152 cols) | Beta | 0/4 | pass | pass | Confabulated `path` and `port` |
| **JetBrains Mono 14px (84 cols)** | Alpha | **4/4** | **pass** | **pass** | `fingerprint`, `camelCase`, `path`, `port` exact |
| **JetBrains Mono 14px (84 cols)** | Beta | **4/4** | **pass** | **pass** | `fingerprint`, `camelCase`, `path`, `port` exact |

## Verbatim Hex (15 trials)

- **Spleen 5×8 (152 cols)**: 0/15 exact matches. 5×8 bitmap glyphs are too dense for Qwen 3.8's vision encoder.
- **JetBrains Mono 14px (84 cols)**: **11/15** exact 12-char hex matches (vs Sol 7/8, Grok 4/8).

## Receipts

- `novel-arithmetic-results.json`
- `gist-recall-results.json`
- `verbatim-hex-results.json` (Spleen 5×8 baseline)
- `verbatim-hex-14px-results.json` (JetBrains Mono 14px pilot)
Loading
Loading