inference-labs

Official JavaScript / TypeScript client for Inference Labs — a vendor-neutral router for the major cloud LLMs (OpenAI / Azure, Anthropic, Google, AWS Bedrock, RunwayML). One endpoint, one billing surface, automatic failover, semantic caching, and policy-based model selection (cost-first, quality-first, latency-first, balanced, judge).

Install

Until the package lands on the npm registry, install directly from GitHub:

npm install github:bosslesss/inference-labs-js
# or pin to a release:
npm install https://github.com/bosslesss/inference-labs-js/releases/download/v0.1.0/inference-labs-0.1.0.tgz

Modern Node (≥18) ships fetch natively — no extra HTTP dependency.

Quickstart

import { InferenceLabs } from "inference-labs";

const client = new InferenceLabs({ apiKey: "il_live_..." });  // or INFERENCE_LABS_API_KEY env

const out = await client.generate({
  prompt: "Summarize: 'The cache hit rate determines spend more than model choice.'",
  strategy: "cost-first",
  maxCostUsd: 0.01,
});

console.log(out.text);
console.log(`routed via ${out.provider}/${out.model} -- $${out.costUsd.toFixed(5)} (${out.latencyMs} ms)`);

Streaming:

for await (const chunk of client.stream({ prompt: "Write a haiku about caching." })) {
  process.stdout.write(chunk);
}

Routing options

All options below are per-call and optional. Defaults are sensible.

Option	Type	What it does
prompt	string	Required. The text to route.
modality	"text" | "image" | "video"	Default "text".
strategy	"balanced" | "cost-first" | "quality-first" | "latency-first" | "judge"	Policy used to pick the model.
maxCostUsd	number	Hard per-request cost cap.
maxLatencyMs	number	Latency budget in milliseconds.
allowModels	string[]	Restrict to a subset of model IDs.
denyModels	string[]	Exclude specific model IDs.
workspaceId	string	Override the API key's default workspace.
collectTrace	boolean	Persist a redacted trace for evals (default true).
redactPii	boolean	Run the PII redactor before storage (default true).

The returned GenerationResult:

interface GenerationResult {
  text: string;
  model: string;        // e.g. "claude-sonnet-4-5"
  provider: string;     // e.g. "Anthropic"
  costUsd: number;
  latencyMs: number;
  cached: boolean;
  traceId: string | null;
  raw: Record<string, unknown>;  // full response payload
}

Errors

import {
  InferenceLabsError,
  AuthenticationError,
  RateLimitError,
  InsufficientCreditsError,
  APIError,
} from "inference-labs";

All errors inherit from InferenceLabsError.

Configuration

const client = new InferenceLabs({
  apiKey: "il_live_...",                    // or INFERENCE_LABS_API_KEY
  baseUrl: "https://app.inference-labs.com", // override for staging / self-hosted
  timeoutMs: 60_000,
  fetch: customFetch,                        // override the global fetch
});

License

Apache-2.0. See LICENSE.

Links

• Marketing: https://inference-labs.com
• App / dashboard: https://app.inference-labs.com
• Python SDK: https://github.com/bosslesss/inference-labs-python
• MCP server (Claude Desktop / Cursor / Windsurf): https://github.com/bosslesss/inference-labs-mcp
• OpenAPI spec: https://inference-labs.com/api/openapi.yaml