Skip to content

Latest commit

 

History

History
376 lines (314 loc) · 15.9 KB

library.md

File metadata and controls

376 lines (314 loc) · 15.9 KB

Library API

The CLI is a thin wrapper over these functions; both produce the same ProfileDocument (document-schema.md).

import {
  time, profile, bench, ab, group, task, sweep, range, run, keep,
  compareDocuments, DEFAULT_THRESHOLDS, defineConfig,
  createDocument, loadDocument, saveDocument, OstiaDocumentError,
  renderers, MINIMAL_PROTOCOL_VERSION,
} from "ostia"
import type {
  ProfileDocument, Workload, Comparison, Warning, WarningCode, CompareResult, Thresholds,
  TaskOptions, GroupOptions, RunOptions, OstiaConfig, OstiaConfigInput, WorkloadConfig,
  TimeOptions, CommandSpec, PrepareHook, PrepareFn, PrepareRun, TimeSource, TimeUnit,
  BenchOptions, AbOptions, ProfileOptions, ProfileResult,
  MinimalEvent, MinimalRenderOptions, MinimalProtocolContext, MermaidOptions, VizOptions,
} from "ostia"

time(opts) → Promise<ProfileDocument>

Subprocess timing; the same behavior as ostia time (cli.md).

const controller = new AbortController()
const doc = await time({
  commands: ["bun a.ts", ["bun", "-e", "console.log('a b')"]],
  prepare: "rm -rf dist",       // before every trial; or ({ phase, index }) => ...
  timeSource: { pattern: /in (\d+)ms/, unit: "ms" },
  samples: 10,                  // or budgetMs / minSamples
  warmup: 2,
  interleave: true,             // default with 2+ commands
  cpu: true,
  heap: false,
  cpuIntervalUs: 1000,          // default
  cwd: "packages/app",
  env: { NODE_ENV: "production" },
  outDir: "node_modules/.cache/ostia",
  noiseCheck: true,
  timeoutMs: 30_000,            // no default
  ignoreExitCodes: [1],
  signal: controller.signal,
})

A command is a string (whitespace-split, no shell), an argv array, or an object { command, label?, prepare?, timeSource?, timeoutMs?, ignoreExitCodes? } whose fields override the top-level ones for that command. That's how one command becomes several labeled workloads:

const build = ["bun", "cli.ts", "build", "fixture"]
const inMs = { pattern: /in (\d+)ms/ }
const doc = await time({
  commands: [
    { command: build, label: "warm", timeSource: inMs },
    { command: build, label: "incremental", timeSource: inMs, prepare: () => touchPost() },
    { command: build, label: "cold", timeSource: inMs, prepare: "rm -rf fixture/dist" },
    { command: build, label: "wall clock" },
  ],
  samples: 5,
})

A prepare function receives { phase, index } with phase one of "warmup", "timing", "cpu", "heap". A RegExp timeSource pattern must not have the g, y or d flag; time() rejects with a RangeError before running anything.

A command stops at its first non-ignored non-zero exit. time() still resolves; check Trial.exitCode or the nonzero-exit warning. Aborting signal kills in-flight children and resolves with the measurements completed so far, the last one carrying an aborted warning.

profile(fn, opts?) → Promise<{ result, measurement, document }>

CPU capture of a function in the current process. The default origin: "inspector" produces CDP-shaped evidence; origin: "jsc" uses bun:jsc's sampling profiler and adds JIT tier counts (LLInt / Baseline / DFG / FTL).

const { result, measurement, document } = await profile(() => hashLoop(8_000_000), {
  origin: "jsc",
  cpuIntervalUs: 100,  // default 1000, as in time()/bench() and `--cpu-interval`
  name: "hash loop",   // optional: the workload's label and id
})
measurement.jit?.tiers // { llint: 0, baseline: 9, dfg: 37, ftl: 2825 }
const { files } = await renderers.collapsed.render(document, {})

document holds the one workload and measurement. An already-aborted signal runs fn without the profiler and returns an aborted warning in place of CPU evidence; a signal can't interrupt fn once it's running.

By default the workload id hashes the function's name and source text, not the values it closes over: () => work(n) profiled for two different n yields the same id twice, and editing the function changes its id. Pass name to identify the workload yourself: the id then hashes name alone and name is also the label, so differently-parameterised closures stay distinct under createDocument, comparisons and renderers, and the id survives edits to the function. name must be a non-empty string.

Suites: group(), task(), bench()

// suite.ts
import { group, task } from "ostia"

group("parse", () => {
  task("small input", () => parse(smallBuf))
  task("large input", () => parse(largeBuf))
  task("full pipeline", () => build(), { budgetMs: 2000, minSamples: 10 })
})
const doc = await bench({
  suites: ["bench/parse.ts"],
  budgetMs: 500,       // per task; default 500
  // samples: 50,      // exact per-task count instead
  minSamples: 20,
  jobs: 1,
  isolate: false,
  gc: false,
  cpu: false,
  cpuIntervalUs: 100,  // default
  alloc: false,
  peakMem: false,
  filter: "parse/",
  preload: ["bench/setup.ts"],
  bunFlags: ["--conditions=browser"],
  timeoutMs: 60_000,
  outDir: "node_modules/.cache/ostia",
  cwd: process.cwd(),
  noiseCheck: true,
})

bench() runs each suite file in a child process (and each isolated task in its own), exactly like ostia bench (cli.md). It rejects if a suite fails, if a suite file doesn't exist, or if jobs isn't an integer >= 1. Each call keeps its scratch files in a directory of its own under outDir, so concurrent calls can share one. It also takes a signal: aborting kills in-flight suite processes and resolves with the suites and tasks that had finished.

TaskOptions

Option Meaning
budgetMs, samples, minSamples Override the suite-wide sampling settings for this task.
isolate, gc, cpu, alloc, peakMem Override the suite-wide flag (and the group's) for this task.
baseline This task is the group's Relative reference (default: the group's fastest).
description Stored as Workload.description and shown in minimal output.
aliases Former names of the task in the same group. compare, ci and ab match a base measured under any of them, so a rename still compares.
params Structured parameters; part of the workload id. Merged over the current sweep() point.
before, after Run once, unmeasured, before the task's warmup and after its last trial, in the task's own process (works with isolate).

GroupOptions takes description, isolate, gc, cpu, alloc, peakMem, before and after; the flags are defaults for the group's tasks, and before/after run once around the group's first and last measured task.

Groups nest. A task's group is the names of its enclosing groups joined with /, outermost first, so a task t in group("a") > group("b") has id a/b/t and Workload.group a/b (ids of tasks in a single group are unchanged). An inner group inherits what it doesn't set from the group around it: description, isolate, gc, cpu, alloc and peakMem (the inner value wins), and .skip/.only. Hooks don't merge, they nest: the outer group's before runs once before the first measured task anywhere inside it, nested groups included, then each inner group's own, and after runs innermost first.

group(
  "editor",
  () => {
    let doc: Document
    task("append", () => doc.append(node), {
      before: () => { doc = mountDocument() },
      after: () => doc.destroy(),
      description: "worst case: full repaint at max document size",
    })
  },
  { before: setupSharedFixture, after: teardownSharedFixture, isolate: true },
)

Module-scope code in a suite file runs once, before any task is sampled. There is no per-trial hook, since that would defeat batching; use gc or isolate for per-trial concerns. Because before/after run once per task, a suite that creates several instances of something stateful (a mounted component, a server) should scope its queries to the instance, not to a global lookup that assumes only one exists.

Task functions and hooks may be async. Awaiting costs a microtask per call, so an async task measures a few nanoseconds slower than the same synchronous body.

.skip and .only

task.skip()/group.skip() register without measuring. The workload stays in the document with skipped: true, prints as - skipped, and compare treats it as unchanged with a skipped warning. task.only()/group.only() restrict the suite file to the marked tasks (--filter still applies) and print bench: N task(s) selected by .only to stderr, once per suite file, so a forgotten .only is visible. A .skip or .only group applies to the groups nested in it. Under ab, a .only in the candidate suite also leaves out the tasks that exist only at the base ref, as --filter does.

keep(value)

A task's return value is already protected from dead-code elimination. Use keep() for an intermediate value the task doesn't return:

task("parse then validate", () => {
  const ast = parse(input)
  keep(ast)
  return validate(ast)
})

sweep(dims, fn) and range(start, end, multiplier = 8)

sweep() calls fn once per point of the Cartesian product of dims; task() calls inside inherit the point as params, so each point is its own workload and compare matches points across runs.

group("parse", () => {
  sweep({ size: range(100, 10_000), impl: ["current", "fast"] }, ({ size, impl }) => {
    const input = buildInput(size) // once per point, unmeasured
    task(impl, () => impls[impl](input))
  })
})

range() produces geometric points ending exactly on end: range(100, 10_000) is [100, 800, 6400, 10000]. The markdown renderer pivots a group into a table (rows = first dimension, columns = second) when all its tasks share the same two param keys; otherwise params print as a key=value suffix.

run(opts?) → Promise<ProfileDocument>

Runs the tasks registered in the current file, in the current process, prints a report, and returns the document. For a suite run directly with bun suite.ts:

import { group, run, task } from "ostia"

group("parse", () => { task("small", () => parse(smallBuf)) })

try {
  await run({ filter: process.env.FILTER })
} finally {
  await cleanupFixtures()
}

Options: filter, budgetMs, samples, minSamples, warmup (a fraction of the budget, default 0.1), gc, cpu, cpuIntervalUs, alloc, noiseCheck, quiet (don't print), format (default "table"). There is no subprocess, so isolate and peakMem are ignored; prefer ostia bench or bench() for numbers you will compare.

ab(opts) → Promise<ProfileDocument>

Paired A/B timing of suite files against a git ref; the same behavior as ostia ab (cli.md). The suite files are the ones bench() runs, unchanged.

const doc = await ab({
  suites: ["bench/parse.bench.ts"],
  base: "origin/main",        // default "HEAD"; the candidate is the working tree
  baseSetup: ["bun scripts/generate.ts"], // run once in a freshly extracted base tree
  keepTrees: 5,               // cached base trees to keep; default
  rounds: 15,                 // default
  metric: "time",             // default; "instructions" judges instructions retired per call
  thresholdPct: 10,           // default
  geomeanThresholdPct: 1.5,   // default
  confirm: 2,                 // fresh-process repeats per flagged task; default
  alloc: true,                // also compare retained heap per call
  peakMem: true,              // also compare first-call RSS rise; both use 3 fresh processes per side
  memThresholdPct: 10,        // default
  filter: "parse/",
  preload: ["bench/setup.ts"],
  bunFlags: ["--conditions=browser"],
  timeoutMs: 120_000,
  outDir: "node_modules/.cache/ostia",
  cwd: process.cwd(),
  noiseCheck: true,
  onProgress: (p) => console.error(p), // { phase: "setup" | "measure" | "confirm" | "memory", ... }
})
doc.ab            // { base, matched, regressed, improved, unchanged, unconfirmed, outputDiffers, notComparable, threw, memory?, geomeanPct, verdict, ... }
doc.measurements  // one phase: "paired" measurement per paired task; `threw` instead of `timing` when it threw
if (doc.ab!.verdict === "fail") process.exitCode = 1

It rejects with a RangeError for bad settings, with an AbBaseError when cwd isn't in a git repository or base isn't a commit, with an AbSetupError (its message ends with the command's output) when a baseSetup command fails, and with an AbSuiteError when one side's suite fails outside any task, such as on import; its side is "base" or "cand". A task that returns its result gets an output comparison between the two sides for free (paired.sameOutput).

compareDocuments(base, cand, thresholds?) → CompareResult

The comparison behind ostia compare and ostia ci (statistics.md). thresholds defaults to DEFAULT_THRESHOLDS and must be complete; spread over the defaults to change one field.

const result = compareDocuments(base, cand, { ...DEFAULT_THRESHOLDS, timingPct: 3 })
result.comparisons  // Comparison[], one per workload id in both documents
result.unmatched    // { baseOnly: Workload[], candOnly: Workload[] }
result.summary      // { matched, regressed, improved, unchanged, geomeanPct, effectiveTimingPct, verdict }

ostia compare stores these on the candidate document as comparisons, comparisonSummary and unmatched (ids only).

Documents

const doc = createDocument(workloads, measurements) // optional 3rd arg: environment
await saveDocument(doc, "doc.json")          // atomic write
const loaded = await loadDocument("doc.json") // throws OstiaDocumentError

createDocument stamps versions, platform, timestamp and git metadata. It's useful for combining several profile() results:

const a = await profile(() => taskA())
const b = await profile(() => taskB())
const doc = createDocument(
  [a.document.workloads[0]!, b.document.workloads[0]!],
  [a.measurement, b.measurement],
)

loadDocument accepts schemaVersion: 2 only.

renderers

Each renderer is { name, render(doc, options) } returning { text? } and/or { files? }. Names: table, markdown, json, jsonl, minimal, collapsed, mermaid, speedscope, cpuprofile. Each renderer's options are typed per format: collapsed, speedscope and cpuprofile take VizOptions ({ measurementId? }), mermaid takes MermaidOptions (VizOptions plus topN?), minimal takes MinimalRenderOptions ({ protocol? }, a MinimalProtocolContext, to emit the unmatched and summary events), and the rest take no options (pass {}).

The table and markdown renderers agree on grouping (a task's group, else the part of its id before the last /), on percent deltas (+3.2%, -1.5%, and 0.0% with no sign when a delta rounds to zero), on byte counts (1024-based B/KiB/MiB), and on naming a comparison or capture whose workload is missing from the document by its raw id.

const { text } = await renderers.markdown.render(doc, {})
const { files } = await renderers.speedscope.render(doc, { measurementId })

defineConfig(config)

Returns its argument unchanged; it exists to type-check ostia.config.ts. See config.md.

Coming from mitata or hyperfine

mitata / hyperfine ostia
bench("name", fn) task("name", fn)
run() at the bottom of the file run() at the bottom of the file
baseline() task(name, fn, { baseline: true })
.range(name, start, end, mult) sweep({ name: range(start, end, mult) }, ...)
generator setup (function* () { ...; yield fn }) task(name, fn, { before, after })
do_not_optimize(value) keep(value)
hyperfine --runs N --warmup N ostia time --samples N --warmup N
hyperfine --prepare CMD ostia time --prepare CMD
hyperfine -i / --ignore-failure ostia time --ignore-failure
hyperfine -L var a,b cmd-{var} one command per value, or sweep() for in-process code
hyperfine --export-json / --export-markdown --export-json PATH / --format markdown