The CLI is a thin wrapper over these functions; both produce the same ProfileDocument
(document-schema.md).
import {
time, profile, bench, ab, group, task, sweep, range, run, keep,
compareDocuments, DEFAULT_THRESHOLDS, defineConfig,
createDocument, loadDocument, saveDocument, OstiaDocumentError,
renderers, MINIMAL_PROTOCOL_VERSION,
} from "ostia"
import type {
ProfileDocument, Workload, Comparison, Warning, WarningCode, CompareResult, Thresholds,
TaskOptions, GroupOptions, RunOptions, OstiaConfig, OstiaConfigInput, WorkloadConfig,
TimeOptions, CommandSpec, PrepareHook, PrepareFn, PrepareRun, TimeSource, TimeUnit,
BenchOptions, AbOptions, ProfileOptions, ProfileResult,
MinimalEvent, MinimalRenderOptions, MinimalProtocolContext, MermaidOptions, VizOptions,
} from "ostia"Subprocess timing; the same behavior as ostia time (cli.md).
const controller = new AbortController()
const doc = await time({
commands: ["bun a.ts", ["bun", "-e", "console.log('a b')"]],
prepare: "rm -rf dist", // before every trial; or ({ phase, index }) => ...
timeSource: { pattern: /in (\d+)ms/, unit: "ms" },
samples: 10, // or budgetMs / minSamples
warmup: 2,
interleave: true, // default with 2+ commands
cpu: true,
heap: false,
cpuIntervalUs: 1000, // default
cwd: "packages/app",
env: { NODE_ENV: "production" },
outDir: "node_modules/.cache/ostia",
noiseCheck: true,
timeoutMs: 30_000, // no default
ignoreExitCodes: [1],
signal: controller.signal,
})A command is a string (whitespace-split, no shell), an argv array, or an object
{ command, label?, prepare?, timeSource?, timeoutMs?, ignoreExitCodes? } whose fields
override the top-level ones for that command. That's how one command becomes several
labeled workloads:
const build = ["bun", "cli.ts", "build", "fixture"]
const inMs = { pattern: /in (\d+)ms/ }
const doc = await time({
commands: [
{ command: build, label: "warm", timeSource: inMs },
{ command: build, label: "incremental", timeSource: inMs, prepare: () => touchPost() },
{ command: build, label: "cold", timeSource: inMs, prepare: "rm -rf fixture/dist" },
{ command: build, label: "wall clock" },
],
samples: 5,
})A prepare function receives { phase, index } with phase one of "warmup",
"timing", "cpu", "heap". A RegExp timeSource pattern must not have the g, y
or d flag; time() rejects with a RangeError before running anything.
A command stops at its first non-ignored non-zero exit. time() still resolves; check
Trial.exitCode or the nonzero-exit warning. Aborting signal kills in-flight
children and resolves with the measurements completed so far, the last one carrying an
aborted warning.
CPU capture of a function in the current process. The default origin: "inspector"
produces CDP-shaped evidence; origin: "jsc" uses bun:jsc's sampling profiler and adds
JIT tier counts (LLInt / Baseline / DFG / FTL).
const { result, measurement, document } = await profile(() => hashLoop(8_000_000), {
origin: "jsc",
cpuIntervalUs: 100, // default 1000, as in time()/bench() and `--cpu-interval`
name: "hash loop", // optional: the workload's label and id
})
measurement.jit?.tiers // { llint: 0, baseline: 9, dfg: 37, ftl: 2825 }
const { files } = await renderers.collapsed.render(document, {})document holds the one workload and measurement. An already-aborted signal runs fn
without the profiler and returns an aborted warning in place of CPU evidence; a signal
can't interrupt fn once it's running.
By default the workload id hashes the function's name and source text, not the values
it closes over: () => work(n) profiled for two different n yields the same id twice,
and editing the function changes its id. Pass name to identify the workload yourself:
the id then hashes name alone and name is also the label, so differently-parameterised
closures stay distinct under createDocument, comparisons and renderers, and the id
survives edits to the function. name must be a non-empty string.
// suite.ts
import { group, task } from "ostia"
group("parse", () => {
task("small input", () => parse(smallBuf))
task("large input", () => parse(largeBuf))
task("full pipeline", () => build(), { budgetMs: 2000, minSamples: 10 })
})const doc = await bench({
suites: ["bench/parse.ts"],
budgetMs: 500, // per task; default 500
// samples: 50, // exact per-task count instead
minSamples: 20,
jobs: 1,
isolate: false,
gc: false,
cpu: false,
cpuIntervalUs: 100, // default
alloc: false,
peakMem: false,
filter: "parse/",
preload: ["bench/setup.ts"],
bunFlags: ["--conditions=browser"],
timeoutMs: 60_000,
outDir: "node_modules/.cache/ostia",
cwd: process.cwd(),
noiseCheck: true,
})bench() runs each suite file in a child process (and each isolated task in its own),
exactly like ostia bench (cli.md). It rejects if a suite fails, if a
suite file doesn't exist, or if jobs isn't an integer >= 1. Each call keeps its scratch files
in a directory of its own under outDir, so concurrent calls can share one. It also takes a signal: aborting kills in-flight suite processes and resolves with the
suites and tasks that had finished.
| Option | Meaning |
|---|---|
budgetMs, samples, minSamples |
Override the suite-wide sampling settings for this task. |
isolate, gc, cpu, alloc, peakMem |
Override the suite-wide flag (and the group's) for this task. |
baseline |
This task is the group's Relative reference (default: the group's fastest). |
description |
Stored as Workload.description and shown in minimal output. |
aliases |
Former names of the task in the same group. compare, ci and ab match a base measured under any of them, so a rename still compares. |
params |
Structured parameters; part of the workload id. Merged over the current sweep() point. |
before, after |
Run once, unmeasured, before the task's warmup and after its last trial, in the task's own process (works with isolate). |
GroupOptions takes description, isolate, gc, cpu, alloc, peakMem, before and after;
the flags are defaults for the group's tasks, and before/after run once around the
group's first and last measured task.
Groups nest. A task's group is the names of its enclosing groups joined with /, outermost
first, so a task t in group("a") > group("b") has id a/b/t and Workload.group a/b
(ids of tasks in a single group are unchanged). An inner group inherits what it doesn't set
from the group around it: description, isolate, gc, cpu, alloc and peakMem (the
inner value wins), and .skip/.only. Hooks don't merge, they nest: the outer group's before
runs once before the first measured task anywhere inside it, nested groups included, then each
inner group's own, and after runs innermost first.
group(
"editor",
() => {
let doc: Document
task("append", () => doc.append(node), {
before: () => { doc = mountDocument() },
after: () => doc.destroy(),
description: "worst case: full repaint at max document size",
})
},
{ before: setupSharedFixture, after: teardownSharedFixture, isolate: true },
)Module-scope code in a suite file runs once, before any task is sampled. There is no
per-trial hook, since that would defeat batching; use gc or isolate for per-trial
concerns. Because before/after run once per task, a suite that creates several
instances of something stateful (a mounted component, a server) should scope its queries
to the instance, not to a global lookup that assumes only one exists.
Task functions and hooks may be async. Awaiting costs a microtask per call, so an async task measures a few nanoseconds slower than the same synchronous body.
task.skip()/group.skip() register without measuring. The workload stays in the
document with skipped: true, prints as - skipped, and compare treats it as
unchanged with a skipped warning. task.only()/group.only() restrict the suite file
to the marked tasks (--filter still applies) and print bench: N task(s) selected by .only to stderr, once per suite file, so a forgotten .only is visible. A .skip or .only
group applies to the groups nested in it. Under ab, a .only in the candidate suite also
leaves out the tasks that exist only at the base ref, as --filter does.
A task's return value is already protected from dead-code elimination. Use keep() for
an intermediate value the task doesn't return:
task("parse then validate", () => {
const ast = parse(input)
keep(ast)
return validate(ast)
})sweep() calls fn once per point of the Cartesian product of dims; task() calls
inside inherit the point as params, so each point is its own workload and compare
matches points across runs.
group("parse", () => {
sweep({ size: range(100, 10_000), impl: ["current", "fast"] }, ({ size, impl }) => {
const input = buildInput(size) // once per point, unmeasured
task(impl, () => impls[impl](input))
})
})range() produces geometric points ending exactly on end: range(100, 10_000) is
[100, 800, 6400, 10000]. The markdown renderer pivots a group into a table (rows = first
dimension, columns = second) when all its tasks share the same two param keys; otherwise
params print as a key=value suffix.
Runs the tasks registered in the current file, in the current process, prints a report,
and returns the document. For a suite run directly with bun suite.ts:
import { group, run, task } from "ostia"
group("parse", () => { task("small", () => parse(smallBuf)) })
try {
await run({ filter: process.env.FILTER })
} finally {
await cleanupFixtures()
}Options: filter, budgetMs, samples, minSamples, warmup (a fraction of the
budget, default 0.1), gc, cpu, cpuIntervalUs, alloc, noiseCheck, quiet (don't
print), format (default "table"). There is no subprocess, so isolate and peakMem
are ignored; prefer ostia bench or bench() for numbers you will compare.
Paired A/B timing of suite files against a git ref; the same behavior as ostia ab
(cli.md). The suite files are the ones bench() runs, unchanged.
const doc = await ab({
suites: ["bench/parse.bench.ts"],
base: "origin/main", // default "HEAD"; the candidate is the working tree
baseSetup: ["bun scripts/generate.ts"], // run once in a freshly extracted base tree
keepTrees: 5, // cached base trees to keep; default
rounds: 15, // default
metric: "time", // default; "instructions" judges instructions retired per call
thresholdPct: 10, // default
geomeanThresholdPct: 1.5, // default
confirm: 2, // fresh-process repeats per flagged task; default
alloc: true, // also compare retained heap per call
peakMem: true, // also compare first-call RSS rise; both use 3 fresh processes per side
memThresholdPct: 10, // default
filter: "parse/",
preload: ["bench/setup.ts"],
bunFlags: ["--conditions=browser"],
timeoutMs: 120_000,
outDir: "node_modules/.cache/ostia",
cwd: process.cwd(),
noiseCheck: true,
onProgress: (p) => console.error(p), // { phase: "setup" | "measure" | "confirm" | "memory", ... }
})
doc.ab // { base, matched, regressed, improved, unchanged, unconfirmed, outputDiffers, notComparable, threw, memory?, geomeanPct, verdict, ... }
doc.measurements // one phase: "paired" measurement per paired task; `threw` instead of `timing` when it threw
if (doc.ab!.verdict === "fail") process.exitCode = 1It rejects with a RangeError for bad settings, with an AbBaseError when cwd isn't
in a git repository or base isn't a commit, with an AbSetupError (its message
ends with the command's output) when a baseSetup command fails, and with an
AbSuiteError when one side's suite fails outside any task, such as on import; its side
is "base" or "cand". A task that returns its
result gets an output comparison between the two sides for free
(paired.sameOutput).
The comparison behind ostia compare and ostia ci (statistics.md).
thresholds defaults to DEFAULT_THRESHOLDS and must be complete; spread over the
defaults to change one field.
const result = compareDocuments(base, cand, { ...DEFAULT_THRESHOLDS, timingPct: 3 })
result.comparisons // Comparison[], one per workload id in both documents
result.unmatched // { baseOnly: Workload[], candOnly: Workload[] }
result.summary // { matched, regressed, improved, unchanged, geomeanPct, effectiveTimingPct, verdict }ostia compare stores these on the candidate document as comparisons,
comparisonSummary and unmatched (ids only).
const doc = createDocument(workloads, measurements) // optional 3rd arg: environment
await saveDocument(doc, "doc.json") // atomic write
const loaded = await loadDocument("doc.json") // throws OstiaDocumentErrorcreateDocument stamps versions, platform, timestamp and git metadata. It's useful for
combining several profile() results:
const a = await profile(() => taskA())
const b = await profile(() => taskB())
const doc = createDocument(
[a.document.workloads[0]!, b.document.workloads[0]!],
[a.measurement, b.measurement],
)loadDocument accepts schemaVersion: 2 only.
Each renderer is { name, render(doc, options) } returning { text? } and/or
{ files? }. Names: table, markdown, json, jsonl, minimal, collapsed,
mermaid, speedscope, cpuprofile. Each renderer's options are typed per format:
collapsed, speedscope and cpuprofile take VizOptions ({ measurementId? }),
mermaid takes MermaidOptions (VizOptions plus topN?), minimal takes
MinimalRenderOptions ({ protocol? }, a MinimalProtocolContext, to emit the
unmatched and summary events), and the rest take no options (pass {}).
The table and markdown renderers agree on grouping (a task's group, else the part of
its id before the last /), on percent deltas (+3.2%, -1.5%, and 0.0% with no sign
when a delta rounds to zero), on byte counts (1024-based B/KiB/MiB), and on naming a
comparison or capture whose workload is missing from the document by its raw id.
const { text } = await renderers.markdown.render(doc, {})
const { files } = await renderers.speedscope.render(doc, { measurementId })Returns its argument unchanged; it exists to type-check ostia.config.ts.
See config.md.
| mitata / hyperfine | ostia |
|---|---|
bench("name", fn) |
task("name", fn) |
run() at the bottom of the file |
run() at the bottom of the file |
baseline() |
task(name, fn, { baseline: true }) |
.range(name, start, end, mult) |
sweep({ name: range(start, end, mult) }, ...) |
generator setup (function* () { ...; yield fn }) |
task(name, fn, { before, after }) |
do_not_optimize(value) |
keep(value) |
hyperfine --runs N --warmup N |
ostia time --samples N --warmup N |
hyperfine --prepare CMD |
ostia time --prepare CMD |
hyperfine -i / --ignore-failure |
ostia time --ignore-failure |
hyperfine -L var a,b cmd-{var} |
one command per value, or sweep() for in-process code |
hyperfine --export-json / --export-markdown |
--export-json PATH / --format markdown |