Skip to content

README.md

Latest commit

 

History

History
316 lines (227 loc) · 16 KB

README.md

File metadata and controls

316 lines (227 loc) · 16 KB
Raw

fm-bench

CI npm

Benchmark Apple's fm command on macOS 27+.

Measure latency, throughput, streaming smoothness, stability, and goodput across Apple Foundation Models — with repeatable prompt suites and JSON/CSV reports for automation.

Why fm-bench exists

Apple's Foundation Models run on-device, and the fm CLI is the terminal and script interface to them. That makes raw model quality only half the story.

For real apps, the important questions are:

  • How fast does the first token arrive?
  • Does streaming stay smooth?
  • How stable is latency over repeated runs?
  • What happens under concurrency?
  • Does this Mac meet an interactive SLO?

fm-bench answers those questions with repeatable local benchmarks. Think of it as GeekBench for Apple Foundation Models — run it, get numbers, compare across hardware, models, and macOS updates.

It is deliberately honest about what it can measure: the installed fm is probed for its actual capabilities, and any metric that build cannot supply is reported as unavailable rather than estimated. See docs/compatibility.md.

Quick Start

npm install -g fm-bench
fm-bench

One command discovers your models, runs the standard prompt suite, and prints a full benchmark report (real output from a macOS 27.0 Apple M5 Pro):

fm-bench 0.7.0 | darwin/arm64 | fm
prompts 3 | runs 3 | concurrency 1 | stream on | measured 9 | failed 0 | skipped 0 | elapsed 6.37s | SLO
TTFT<=1.00s,E2E<=3.00s
unavailable: quota unavailable — this fm build exposes no quota command

┌───┬────────┬────────┬─────┬──────┬──────────┬───────┬───────┬─────────┬────────┬───────┬─────┬──────┐
│ C │ MODEL  │ STATUS │ OK  │ GOOD │ GOOD RPS │ TTFT  │ E2E   │ E2E P95 │ USER/S │ SYS/S │ CV  │ NOTE │
├───┼────────┼────────┼─────┼──────┼──────────┼───────┼───────┼─────────┼────────┼───────┼─────┼──────┤
│ 1 │ system │ ok     │ 9/9 │ 100% │      1.7 │ 377ms │ 487ms │   749ms │   34.0 │  32.8 │ 25% │      │
└───┴────────┴────────┴─────┴──────┴──────────┴───────┴───────┴─────────┴────────┴───────┴─────┴──────┘

┌───┬────────┬────────┬─────────┬───────────┬──────────┬───────────┬─────────┬────────┐
│ C │ MODEL  │ IN AVG │ OUT AVG │ PREFILL/S │ DECODE/S │ CHUNK P95 │ E2E P99 │ REPEAT │
├───┼────────┼────────┼─────────┼───────────┼──────────┼───────────┼─────────┼────────┤
│ 1 │ system │     13 │      20 │      35.9 │      113 │     197ms │   749ms │   100% │
└───┴────────┴────────┴─────────┴───────────┴──────────┴───────────┴─────────┴────────┘

Default profile is standard (3 prompts). Use --runs 5, --sweep-concurrency 1,2, or --profile client for heavier suites.

Wide terminals add TTFT P95, TPOT, and RPS columns; medium terminals tighten the table; narrow terminals switch to compact model cards automatically. --width <n> previews any layout.

Install

npm install -g fm-bench

Install directly from GitHub (always latest):

npm install -g --install-links git+https://github.com/devinoldenburg/fm-bench.git

Local development:

npm install && npm link
fm-bench doctor   # verify your setup

Requirements: macOS 27+, Node.js 20+, Apple Intelligence enabled.

Commands

Command What it does
fm-bench Run the full benchmark (default)
fm-bench models Show the detected fm capabilities, discovered models, availability, and quota when supported
fm-bench compare <a.json> <b.json> Regression diff with suite/hardware/macOS warnings; --strict exits 2 when suites differ
fm-bench history [dir] Trend table from saved reports (sorted by time, tags visible)
fm-bench validate <report.json> Verify report JSON (schema v1) before sharing; --json for CI
fm-bench export <report.json> Standalone HTML report with embedded JSON
fm-bench legend Definitions, provenance (measured/proxy/derived), and color rules for every column
fm-bench doctor Environment and fm capability check; --json for scripts
fm-bench metrics Alias for legend

Exit codes: 0 success, 1 operational failure (failed --ci gate, invalid reports), 2 usage or environment error (bad flags, unsupported macOS, unusable fm, no runnable model). Interrupting a run with Ctrl+C terminates in-flight fm processes.

Common Recipes

# Quick smoke test
fm-bench --profile quick

# Standard 5-run benchmark with SLO budgets and warmups
fm-bench --runs 5 --warmup 1 --slo-ttft-ms 750 --slo-e2e-ms 4000

# Sweep concurrency to find your throughput ceiling
fm-bench --sweep-concurrency 1,2,4 --runs 3

# Reasoning and coding workloads
fm-bench --profile reasoning --runs 5
fm-bench --profile coding --runs 3 --histogram

# Archive runs and compare before/after a macOS update
fm-bench --output-dir reports/ --tag before-update --export-html
fm-bench --output-dir reports/ --tag after-update --export-html
fm-bench validate reports/*.json
fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json --strict

# Fail CI when SLOs regress or any run fails
fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5

# Save JSON for automation
fm-bench --json --out bench.json
fm-bench --format csv --out bench.csv

Options Reference

Workload

Flag Default Description
-m, --models <list> discovered Comma-separated or repeated model names
-r, --runs <n> 1 Measured runs per prompt/model
--warmup <n> 0 Unmeasured warmup runs per model before measurement
-c, --concurrency <n> 1 Parallel fm processes
--sweep-concurrency <list> — Separate operating points, e.g. 1,2,4
--request-rate <rps> — Pace request starts at a target rate
--ramp-up-ms <n> 0 Gradually ramp pacing over n ms
--timeout-ms <n> 60000 Timeout per fm call
--retry <n> 0 Retry failed calls with exponential backoff (500ms–4s)
--profile <name> standard Built-in prompt suite (see Profiles)
-p, --prompt <text> — Custom prompt, repeatable
--prompt-file <file> — JSON, JSONL, or blank-line separated prompts
-i, --instructions <text> — Passed to fm respond when the build supports it
--use-case <case> — System model use case, when supported
--guardrails <level> — System model guardrail level, when supported

Quality Gates

Flag Description
--slo-ttft-ms <n> Count a run as good only if TTFT ≤ n ms
--slo-e2e-ms <n> Count a run as good only if E2E latency ≤ n ms
--slo-tpot-ms <n> Count a run as good only if TPOT ≤ n ms
--ci Exit 1 if any run fails or any SLO is violated (for pipelines)
--fail-fast Stop after the first failed run

Output

Flag Description
--json / --csv Output format (also --format table|json|csv)
-o, --out <file> Save JSON (.json), per-run CSV (.csv), or shareable HTML (.html)
--output-dir <dir> Auto-save timestamped JSON (and optional HTML with --export-html)
--export-html With --output-dir, also write a matching .html report
--tag <name> Label this run; repeatable; appears in payload and header
--note <text> Freeform annotation in payload and header
--histogram Print ASCII latency distribution chart after the report
--capture-output Include raw model output in JSON reports
-v, --verbose Append per-run CSV after the summary table

Display

Flag Description
--no-stream Disable streaming; TTFT and decode metrics are reported as unavailable
--greedy / --no-greedy Request or omit greedy sampling (default: greedy)
--available-only Hide unavailable discovered models
--color / --no-color Force or disable ANSI colors (auto on TTYs)
--ascii Plain ASCII table borders instead of Unicode
--compact Force narrow terminal layout
--width <n> Render as if the terminal is n columns wide
--progress / --no-progress Force or disable the live progress line
--fm-bin <path> fm binary to execute (default: FM_BIN or fm)

Prompt Profiles

Nine built-in suites, choose the one that matches your use case:

Profile Prompts Best for
quick 1 Smoke test, fast health check
standard 3 Default — short chat, JSON generation, medium output
interactive 3 Conversational latency (TTFT-heavy)
throughput 3 Longer generation, token throughput signal
client 5 Real-world mix: chat, content, extraction, summarization, code
stress 5 High-load mix with math and reasoning
reasoning 5 Multi-step logic, estimation, debugging — capability + speed
coding 5 Code review, refactoring, algorithms, system design
creative 5 Product copy, analogies, commit messages, docs

Metrics

Latency — TTFT (p50/p95), E2E (p50/p95/p99), TPOT, 95% confidence interval, coefficient of variation (CV).

Throughput — prefill tokens/s, decode tokens/s, output tokens/s per request, aggregate system tokens/s, requests per second.

Streaming quality — second-chunk delay, chunk-gap p95, captured from stdout chunk arrival during streaming runs.

Reliability — success rate, goodput rate and RPS against SLO budgets, repeatability (most common output hash frequency across repeated runs).

Every metric is labelled by provenance in JSON and in fm-bench legend:

  • measured — process wall clock, exit codes, chunk arrival, fm token counts
  • proxy — TTFT and chunk gaps (chunk granularity, not token timestamps); prefill tokens/s
  • derived — TPOT, decode tokens/s, throughput, CV, confidence intervals, goodput

Token counts come from the fm build's own token-counting command (count-tokens, or token-count on older builds). If the build cannot count tokens, token metrics render as -, JSON carries null, and metrics.promptTokens.available is false. Spread statistics (CV, 95% CI) need at least two successful samples; with one sample they are unavailable rather than 0. See docs/methodology.md.

fm compatibility

fm-bench probes fm --help and fm respond --help once per run and adapts:

  • Subcommand names (count-tokens vs legacy token-count) are detected, not hardcoded.
  • Flags the build does not document (--stream, --use-case, --guardrails, --model) are not passed through.
  • Unsupported models are rejected before a benchmark starts, with the supported list in the error.
  • Raw fm argument-error text never reaches reports, tables, or JSON.

fm-bench doctor shows exactly what was detected. Policy and the verified-build table: docs/compatibility.md.

Sharing and comparing results

Reports from 0.6.0+ include schema v1: reportId, hardware fingerprint, a suite key so you can tell if two JSON files used the same prompts and run settings, plus the detected fm capabilities and per-metric availability. See docs/report-format.md.

  • Share HTML with teammates who do not use the CLI: fm-bench export bench.json -o bench.html
  • Gate uploads in CI: fm-bench validate artifact.json
  • Apples-to-apples regressions: same --profile and --runs, then fm-bench compare a.json b.json

Regression Tracking

Track performance across macOS updates, model changes, or hardware swaps:

fm-bench --profile coding --runs 5 --output-dir reports/ --tag before
fm-bench --profile coding --runs 5 --output-dir reports/ --tag after
fm-bench compare reports/fm-bench_*before*.json reports/fm-bench_*after*.json
fm-bench history reports/

The compare output shows each model/concurrency row with the before value, a color-coded percent delta (green = improvement, red = regression), and the after value — for TTFT, E2E, TPOT, tokens/s, RPS, success rate, and CV. It warns when hardware, macOS version/build, or the prompt suite differ.

CI Integration

Gate deployments or model updates on benchmark quality:

# Fails with exit code 1 if TTFT > 750ms, E2E > 4s, or any run fails
fm-bench --ci --slo-ttft-ms 750 --slo-e2e-ms 4000 --runs 5

Prints fm-bench ci: PASS or fm-bench ci: FAIL — <reason> to stderr. Designed for GitHub Actions, Buildkite, or any shell-based pipeline. --json and --csv write only data to stdout; progress and diagnostics go to stderr.

Prompt Files

JSON array:

[
  { "id": "tiny", "prompt": "Reply with exactly: ok" },
  { "id": "json", "prompt": "Convert alpha, beta, gamma into JSON." }
]

JSONL:

{"id":"tiny","prompt":"Reply with exactly: ok"}
{"id":"latency","prompt":"Explain p95 latency in one sentence."}

Plain text files are split on blank lines. Parse errors name the file and, for JSONL, the line number.

Colors and Legend

Table output is color-coded on interactive terminals — green is better/passing, yellow is marginal/partial, red is failing/unstable. Fixed thresholds apply to success rate, goodput, CV, and repeatability. Latency uses SLO thresholds when set, otherwise lower-is-better relative ranking. Throughput uses higher-is-better relative ranking.

fm-bench legend          # definitions, provenance, and color rules
fm-bench legend --json   # machine-readable

NO_COLOR=1 disables color; FORCE_COLOR=1 or --color enables it. --ascii switches to plain ASCII borders for log systems. A live single-line progress indicator runs on stderr during interactive sessions; the final report always goes to stdout.

Requirements

  • macOS 27.0 or newer (Apple's fm CLI is preinstalled there).
  • Node.js 20 or newer.
  • Apple Intelligence enabled on the device.

Benchmark commands refuse to start on older macOS versions and report the detected version plus the latest supported macOS — see docs/supported-platforms.md.

Development

npm install
npm test          # node --test (unit + integration against a fake fm)
npm run lint      # node --check on every source file
npm run check     # lint + tests + npm pack integrity — run this before pushing

No runtime npm dependencies. Integration tests drive the real CLI against test/fixtures/fake-fm.mjs, which emulates normal, streaming, slow, malformed, failing, timed-out, partial, and interrupted fm behaviour, so the suite runs on machines without fm.

Documentation: methodology · report format · compatibility · supported platforms · releasing.

License

MIT