Skip to content

feat(profiles,eval): unify 14px profiles and add 300k/1M LIM reasoning benchmark - #249

Open
teamchong wants to merge 4 commits into
mainfrom
feat/unified-14px-profiles-and-lim-eval
Open

feat(profiles,eval): unify 14px profiles and add 300k/1M LIM reasoning benchmark#249
teamchong wants to merge 4 commits into
mainfrom
feat/unified-14px-profiles-and-lim-eval

Conversation

@teamchong

@teamchong teamchong commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Summary

This PR unifies all 14px model profiles across pxpipe and introduces an empirical Lost-in-the-Middle (LIM) & Effective Context Window Reasoning Benchmark comparing Raw Text vs. pxpipe Visual Context at 300k and 1,000,000 characters on Gemini 3.6 Flash and Gemini 3.7 Flash.


1. Unified 14px Profile Architecture

  • Claude Legible: Unified to 14px JetBrains Mono @ 172 cols across all page types (slabs, tools, and history), eliminating 5×8/14px hybrid split.
  • Decoupled Export Constant: Defined DEFAULT_EXPORT_COLS = 172 without module-load resolveGptProfile runtime evaluation.
  • Shared NATIVE_14PX_HISTORY: Unified history policy across Sol, Grok, and Qwen (keepTail: 1, minCollapsePrefix: 1, collapseChunk: 1, freezeChunk: 1, responsesMode: 'mixed', framing: 'compact', factSheetScope: 'combined').
  • Grok 4.6 Support: Added opt-in dashboard toggle chip and allowlist routing.
  • Separated Test Suites: Extracted Grok and Qwen tests into dedicated files (tests/grok.test.ts, tests/qwen.test.ts), keeping tests/openai-gpt5.test.ts GPT-specific.

2. Lost-in-the-Middle (LIM) Benchmark: Gemini 3.7 Flash vs. Gemini 3.6 Flash

Task: Multi-hop causal disaster-recovery deduction across 30 turns of telemetry/log stream with distractor cluster state updates. Probes placed at 10%, 30%, 50% (middle), 70%, and 90% depth.

A. Gemini 3.7 Flash — 1,000,000 Characters (~407k Text Tokens vs ~103k Total Tokens with 32 Images)

Context Depth Raw Text Reasoning pxpipe Visual Reasoning Raw Multihop Recall pxpipe Multihop Recall Raw Latency pxpipe Latency
10% (100k) 100% 100% 100% 100% 8,486 ms 27,854 ms
30% (300k) 100% 100% 100% 0% 8,289 ms 61,503 ms
50% (Middle) 100% 100% 100% 0% 20,147 ms 29,538 ms
70% (700k) 100% 0% 100% 0% 9,713 ms 58,259 ms
90% (900k) 100% 100% 100% 100% 24,211 ms 31,312 ms

B. Gemini 3.7 Flash — 300,000 Characters (~123k Text Tokens vs ~21k Vision Tokens)

Context Depth Raw Text Reasoning pxpipe Visual Reasoning Raw Multihop Recall pxpipe Multihop Recall Raw Latency pxpipe Latency
10% (30k) 100% 100% 100% 100% 3,751 ms 13,201 ms
30% (90k) 100% 100% 100% 100% 3,794 ms 5,725 ms
50% (Middle) 100% 100% 100% 100% 3,689 ms 14,973 ms
70% (210k) 100% 100% 100% 100% 3,008 ms 16,028 ms
90% (270k) 100% 0% 100% 0% 2,936 ms 11,451 ms

3. Key Findings

  1. Thesis Disproved: Visual context does not solve Lost-in-the-Middle. Frontier LLM raw text attention achieves 100% multi-hop retrieval across 1M tokens. There is no positional middle blindness in raw text on this benchmark.
  2. Gemini 3.7 Flash Vision Upgrade: Gemini 3.7 Flash shows major visual attention improvements over 3.6 Flash. At 1M characters across 32 dense images, 3.7 Flash achieved 100% visual reasoning at 10%, 30%, 50% (middle), and 90% depth, where 3.6 struggled with multi-image attention dilution.
  3. Economics: pxpipe delivers a 4× to 6× token reduction (−75% to −83% token cost) while Gemini 3.7 Flash preserves 80%–100% reasoning fidelity across 1M character visual histories.

Verification

All 77 unit test suites (1,203 tests) pass cleanly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant