diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/.gitignore b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/.gitignore new file mode 100644 index 000000000000..2a2371d2f294 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/.gitignore @@ -0,0 +1,13 @@ +# The skill tool is intentionally NOT registered as a pnpm workspace member +# (its `package.json` would otherwise be picked up by the repo-root `sdk/**` +# glob in pnpm-workspace.yaml, and Turbo would try to build it during CI +# without any way to install its deps). Instead, `package.json.template` is +# the tracked file; the skill scripts copy it into place before running +# `npm install`. See README.md for the exact sequence. +package.json + +# Skill tool dependencies — installed per-user via `npm install`, never committed. +node_modules/ +package-lock.json +dist/ +.tshy/ diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/README.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/README.md new file mode 100644 index 000000000000..c0ea9a9dd2c2 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/README.md @@ -0,0 +1,81 @@ +# `_shared/` — library, not a skill + +Pure-TypeScript helpers used by the authoring skill scripts under +`.github/skills/cu-sdk-author-analyzer*/`. + +The leading underscore marks this as a **library directory**, not a skill. It +is intentionally excluded from the Copilot skill picker. + +Rules for `schemaValidator.ts` (the validator): + +- **No `@azure/*` imports.** No network calls. No I/O beyond reading / + parsing caller-provided JSON. +- **No new runtime dependencies.** Native Node `fs` only. +- **Stable, small, well-tested.** Anything here is referenced by multiple skill + scripts; breakage cascades. + +The CLI command modules (`extractLayoutCommand.ts`, `createAndTestCommand.ts`, +`createAndTestRouterCommand.ts`) wrap the `ContentUnderstandingClient` from +`@azure/ai-content-understanding`. They are allowed to import `@azure/*` since +they're the bridge between the validator and the service. + +Current modules: + +- [`schemaValidator.ts`](src/schemaValidator.ts) — validates analyzer schema + JSON before any service call (catches `baseAnalyzerId` typos, missing + `fieldSchema`, missing `contentCategories` analyzer routes, etc.). Pure + TypeScript. +- [`cli.ts`](src/cli.ts) — subcommand dispatcher. +- [`clientHelpers.ts`](src/clientHelpers.ts) — client builder + raw-response + capture policy + JS-SDK serializer translation (`items` → + `itemDefinition`). +- [`extractLayoutCommand.ts`](src/extractLayoutCommand.ts) — Stage 1: extract + document layout. +- [`createAndTestCommand.ts`](src/createAndTestCommand.ts) — Stage 2 + (single-type). +- [`createAndTestRouterCommand.ts`](src/createAndTestRouterCommand.ts) — + Stage 2 (classify-and-route). + +## Install + +The tool is **intentionally NOT part of the pnpm workspace** — it's a +standalone CLI that lives outside the published source tree under +`.github/skills/_shared/`, so it has zero effect on the published +`@azure/ai-content-understanding` artifact. + +To keep pnpm's repo-root `sdk/**` workspace glob from picking this directory +up (which would fail CI because the tool's deps aren't in the top-level +lockfile), the tracked file is `package.json.template` — the runtime +`package.json` is `.gitignore`d and generated locally on install: + +```bash +(cd .github/skills/_shared && \ + [ -f package.json ] || cp package.json.template package.json && \ + npm install) +``` + +The tool depends on the published `@azure/ai-content-understanding` package +from npm, not the in-tree source, so contributors can iterate without first +running a repo-root build. + +## Run + +```bash +(cd .github/skills/_shared && \ + node_modules/.bin/tsx src/cli.ts extract-layout \ + --input --output ) +``` + +Or, with the npm script wrapper: + +```bash +(cd .github/skills/_shared && \ + npm run cli -- extract-layout \ + --input --output ) +``` + +## Run tests + +```bash +(cd .github/skills/_shared && npm test) +``` diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/package.json.template b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/package.json.template new file mode 100644 index 000000000000..76a50c83cc46 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/package.json.template @@ -0,0 +1,23 @@ +{ + "name": "cu-skill", + "version": "0.1.0", + "description": "Companion CLI for the cu-sdk-author-analyzer / cu-sdk-author-analyzer-classify-route GitHub Copilot skills. Wraps the @azure/ai-content-understanding SDK with three subcommands (extract-layout, create-and-test, create-and-test-router) and a pure-TypeScript schema validator.", + "type": "module", + "private": true, + "scripts": { + "build": "tsc -p tsconfig.json", + "cli": "tsx src/cli.ts", + "test": "tsx --test src/*.test.ts" + }, + "dependencies": { + "@azure/ai-content-understanding": "^1.2.0-beta.2", + "@azure/core-auth": "^1.5.0", + "@azure/core-rest-pipeline": "^1.13.0", + "@azure/identity": "^4.0.0" + }, + "devDependencies": { + "@types/node": "^22.0.0", + "tsx": "^4.22.4", + "typescript": "~5.7.0" + } +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/cli.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/cli.ts new file mode 100644 index 000000000000..d9a6bd51021d --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/cli.ts @@ -0,0 +1,59 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Subcommand dispatcher for the cu-skill tool. Mirrors Python's + * extract_layout.py / create_and_test.py / create_and_test_router.py + * entry points, the .NET Program.cs, and the Java Cli.java. + */ + +import { runExtractLayout } from "./extractLayoutCommand.js"; +import { runCreateAndTest } from "./createAndTestCommand.js"; +import { runCreateAndTestRouter } from "./createAndTestRouterCommand.js"; + +async function main(): Promise { + const args = process.argv.slice(2); + if (args.length === 0 || isHelp(args[0])) { + printUsage(); + process.exit(args.length === 0 ? 1 : 0); + } + const subcommand = args[0]; + const subArgs = args.slice(1); + let exit = 0; + switch (subcommand) { + case "extract-layout": + exit = await runExtractLayout(subArgs); + break; + case "create-and-test": + exit = await runCreateAndTest(subArgs); + break; + case "create-and-test-router": + exit = await runCreateAndTestRouter(subArgs); + break; + default: + console.error(`unknown subcommand: ${subcommand}`); + printUsage(); + exit = 1; + } + process.exit(exit); +} + +function isHelp(arg: string): boolean { + return arg === "-h" || arg === "--help" || arg === "help"; +} + +function printUsage(): void { + console.log("cu-skill — Content Understanding analyzer-authoring tool."); + console.log(); + console.log("Subcommands:"); + console.log(" extract-layout extract document layout (stage 1)"); + console.log(" create-and-test validate, create, batch-test a single-type analyzer"); + console.log(" create-and-test-router classify-and-route variant (N inner + 1 outer)"); + console.log(); + console.log("Use ' --help' for per-command flags."); +} + +main().catch((err) => { + console.error(err); + process.exit(1); +}); diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.test.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.test.ts new file mode 100644 index 000000000000..6d42983fe1b2 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.test.ts @@ -0,0 +1,150 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Unit tests for the pure `translateForJsSerializer` helper in + * clientHelpers.ts. The JS SDK's typed model serializer expects + * `field.itemDefinition` on the wire but reads `field.items` from our + * schema files, so we pre-rename before handing off. These tests pin + * down the scoping rules so the rename does NOT clobber user field + * names that happen to be `items` or `$ref`. + */ + +import { strict as assert } from "node:assert"; +import { describe, it } from "node:test"; +import { _translateForJsSerializer as translate } from "./clientHelpers.js"; + +describe("translateForJsSerializer", () => { + it("renames array-field `items` to `itemDefinition` on the field descriptor", () => { + const input = { + type: "array", + method: "extract", + items: { type: "object", properties: {} }, + }; + const out = translate(input) as Record; + assert.equal(out["itemDefinition"] !== undefined, true, "should be renamed to itemDefinition"); + assert.equal("items" in out, false, "original `items` key should be dropped"); + }); + + it("does NOT rename a user field literally named `items`", () => { + // A very common analyzer schema pattern: a field for line items named + // "items" (extract-method array). The rename must NOT clobber that + // user-chosen key, because `fieldSchema.fields.` and + // `properties.` are keyed by user names, not by our wire vocab. + const input = { + fieldSchema: { + fields: { + items: { + type: "array", + method: "extract", + description: "the line-item table", + items: { + type: "object", + properties: { + sku: { type: "string", method: "extract", description: "SKU column" }, + }, + }, + }, + }, + }, + }; + + const out = translate(input) as { + fieldSchema: { fields: Record }; + }; + const fields = out.fieldSchema.fields; + + // User field name preserved. + assert.equal("items" in fields, true, "user field name `items` must survive"); + assert.equal( + "itemDefinition" in fields, + false, + "must not have created an itemDefinition sibling", + ); + + // The array-descriptor level DID get renamed (that's the whole point). + const desc = fields["items"] as Record; + assert.equal( + desc["itemDefinition"] !== undefined, + true, + "array descriptor should have itemDefinition", + ); + assert.equal("items" in desc, false, "array descriptor `items` should be renamed"); + }); + + it("does not rename anything under `properties.` (user field names)", () => { + // The children of `properties` are user field names, not the wire + // vocabulary. They must never be renamed. + const input = { + type: "object", + method: "extract", + properties: { + items: { type: "string", method: "extract", description: "not an array" }, + $ref: { type: "string", method: "extract", description: "user named it $ref" }, + ordinary: { type: "string", method: "extract" }, + }, + }; + const out = translate(input) as { properties: Record }; + assert.equal("items" in out.properties, true); + assert.equal("$ref" in out.properties, true); + assert.equal("ordinary" in out.properties, true); + // No spurious renames at property-map level. + assert.equal("itemDefinition" in out.properties, false); + assert.equal("ref" in out.properties, false); + }); + + it("renames `$ref` to `ref` on a field descriptor", () => { + const input = { type: "string", $ref: "#/definitions/foo" }; + const out = translate(input) as Record; + assert.equal(out["ref"], "#/definitions/foo"); + assert.equal("$ref" in out, false); + }); + + it("recurses into nested object properties and array items", () => { + // Full schema, three levels deep. Confirms recursion works AND scoping + // holds under recursion. + const input = { + fieldSchema: { + fields: { + invoice: { + type: "object", + method: "extract", + properties: { + lineItems: { + type: "array", + method: "extract", + items: { + type: "object", + properties: { + itemCode: { type: "string", method: "extract" }, + }, + }, + }, + }, + }, + }, + }, + }; + const out = translate(input) as { + fieldSchema: { + fields: { + invoice: { + properties: { + lineItems: { itemDefinition?: { properties: unknown }; items?: unknown }; + }; + }; + }; + }; + }; + const lineItems = out.fieldSchema.fields.invoice.properties.lineItems; + assert.equal(lineItems.itemDefinition !== undefined, true); + assert.equal(lineItems.items, undefined); + }); + + it("passes through primitives and arrays of primitives unchanged", () => { + assert.equal(translate("hello"), "hello"); + assert.equal(translate(42), 42); + assert.equal(translate(null), null); + assert.deepEqual(translate([1, 2, 3]), [1, 2, 3]); + }); +}); diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.ts new file mode 100644 index 000000000000..d658efad8f7c --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/clientHelpers.ts @@ -0,0 +1,283 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Shared helpers used by the CLI command modules. + * + * - buildClient(): creates a ContentUnderstandingClient using credentials from + * environment variables (CONTENTUNDERSTANDING_ENDPOINT / CONTENTUNDERSTANDING_KEY, + * or `az login` via DefaultAzureCredential with ManagedIdentity / Workload + * identity excluded to avoid the slow IMDS probe on dev boxes). + * - capturedAnalyzeBinary() / capturedCreateAnalyzer(): wrap the typed SDK + * methods with a pipeline policy that captures the raw service JSON so we + * can write on-disk output in the same shape as the Python and .NET skills + * (valueString / valueNumber / ... rather than the SDK's deserialised + * typed model). The LRO envelope `{id,status,result,usage}` is unwrapped + * to match `poller.pollUntilDone()` semantics in Python and .NET. + * - guessContentType(): file-extension lookup for the analyzeBinary contentType. + * - readEnv(): trim one layer of surrounding quotes (dotenv strips them but + * raw `export` from a shell does not). + * - enumerateInputs(): single-file or folder-of-files enumeration with the + * same supported extension list as Python. + */ + +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { extname, join } from "node:path"; +import { + ChainedTokenCredential, + AzureCliCredential, + EnvironmentCredential, + type TokenCredential, +} from "@azure/identity"; +import { AzureKeyCredential } from "@azure/core-auth"; +import { ContentUnderstandingClient } from "@azure/ai-content-understanding"; +import type { + PipelinePolicy, + PipelineRequest, + PipelineResponse, + SendRequest, +} from "@azure/core-rest-pipeline"; + +const SUPPORTED_EXTENSIONS = new Set([ + ".pdf", + ".png", + ".jpg", + ".jpeg", + ".tif", + ".tiff", + ".bmp", + ".heif", + ".heic", + ".wav", + ".mp3", + ".m4a", + ".mp4", + ".mov", +]); + +/** + * Reads an env var and strips one layer of surrounding single or double + * quotes — dotenv strips them by default but raw `export` from a shell does + * not, so users who source .env manually still get a clean value. + */ +export function readEnv(name: string): string | undefined { + const value = process.env[name]; + if (value === undefined || value === "") { + return value; + } + if ( + value.length >= 2 && + ((value.startsWith('"') && value.endsWith('"')) || + (value.startsWith("'") && value.endsWith("'"))) + ) { + return value.substring(1, value.length - 1); + } + return value; +} + +export function buildClient(): ContentUnderstandingClient { + let endpoint = readEnv("CONTENTUNDERSTANDING_ENDPOINT"); + if (!endpoint || endpoint.trim() === "") { + throw new Error( + "CONTENTUNDERSTANDING_ENDPOINT is not set. Configure your .env file (see cu-sdk-setup).", + ); + } + // Strip trailing slash to match the convention from samples — avoids + // double-slash URLs when the env var was copy-pasted from the portal. + while (endpoint.endsWith("/")) { + endpoint = endpoint.substring(0, endpoint.length - 1); + } + + const key = readEnv("CONTENTUNDERSTANDING_KEY"); + if (key && key.trim() !== "") { + return new ContentUnderstandingClient(endpoint, new AzureKeyCredential(key)); + } + // Build a focused chain: Environment first (CI), then Azure CLI (dev + // boxes). DefaultAzureCredential would probe IMDS first and block for + // ~30 s on WSL / laptops before falling through. + const credential: TokenCredential = new ChainedTokenCredential( + new EnvironmentCredential(), + new AzureCliCredential(), + ); + return new ContentUnderstandingClient(endpoint, credential); +} + +/** + * Wraps `client.analyzeBinary(...)` in a pipeline policy that captures the + * raw service response, then unwraps the LRO envelope so the returned object + * matches the Python / .NET on-disk shape (`{analyzerId, contents: [...]}`). + */ +export async function capturedAnalyzeBinary( + client: ContentUnderstandingClient, + analyzerId: string, + bytes: Uint8Array, + contentType: string, +): Promise> { + let rawResponse: PipelineResponse | undefined; + const capturePolicy: PipelinePolicy = { + name: "cuSkillCaptureRawResponse", + async sendRequest(req: PipelineRequest, next: SendRequest): Promise { + const res = await next(req); + // Only keep the final LRO status response — its body is what we want + // for the on-disk shape. The submission response (202) has an empty + // body, so this filter keeps things simple. + if (res.bodyAsText && res.bodyAsText.length > 0) { + rawResponse = res; + } + return res; + }, + }; + client.pipeline.addPolicy(capturePolicy); + try { + const poller = client.analyzeBinary(analyzerId, bytes, contentType); + await poller.pollUntilDone(); + } finally { + client.pipeline.removePolicy({ name: "cuSkillCaptureRawResponse" }); + } + + if (!rawResponse || !rawResponse.bodyAsText) { + throw new Error("analyzeBinary completed but no raw response body was captured"); + } + const envelope = JSON.parse(rawResponse.bodyAsText); + // Unwrap `{id, status, result, usage}` so callers see just the analysis + // result. Matches Python's `poller.result()` and .NET's `op.result`. + if ( + envelope !== null && + typeof envelope === "object" && + !Array.isArray(envelope) && + "result" in envelope && + typeof envelope["result"] === "object" && + envelope["result"] !== null + ) { + return envelope["result"] as Record; + } + return envelope as Record; +} + +/** + * Create an analyzer using a raw JSON schema. The typed SDK signature only + * accepts `ContentAnalyzer`, but we want to pass whatever the user put in + * their schema file (including custom properties that may not be in the + * model). + * + * The JS SDK's serializer has a small wire-format quirk for nested field + * definitions: it reads `itemDefinition` and writes `items` (the wire shape + * we want). Our raw JSON already uses the wire shape (`items` directly), so + * we pre-translate to the typed shape before handing the schema off — that + * way the serializer doesn't strip the `items` body when it builds the PUT. + * Mirrors the way Python's typed sdk requires no remapping because Python's + * model is permissive. Same translation applies to `$ref` → `ref`. + */ +export async function createAnalyzerRaw( + client: ContentUnderstandingClient, + analyzerId: string, + schema: Record, +): Promise { + const adapted = translateForJsSerializer(schema); + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const poller = client.createAnalyzer(analyzerId, adapted as any); + await poller.pollUntilDone(); +} + +/** + * Recursively rename wire-format keys that the JS typed model serializer + * expects under different names: + * - `items` → `itemDefinition` — but ONLY on field-descriptor objects + * (siblings of `"type": "array"`), NOT on arbitrary map keys. Users + * may legitimately name a field literally "items" (very common for + * line-item arrays) and that key must be preserved. + * - `$ref` → `ref` — likewise only on field descriptors. + * + * A field descriptor is any object that has a `type` string property. The + * children of `fieldSchema.fields`, `properties`, and (renamed) + * `itemDefinition.properties` are keyed by user-chosen field names and must + * not be renamed. + */ +function translateForJsSerializer(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(translateForJsSerializer); + } + if (value && typeof value === "object") { + const src = value as Record; + const isFieldDescriptor = typeof src["type"] === "string"; + const out: Record = {}; + for (const [k, v] of Object.entries(src)) { + let key = k; + if (isFieldDescriptor) { + if (k === "items") { + key = "itemDefinition"; + } else if (k === "$ref") { + key = "ref"; + } + } + out[key] = translateForJsSerializer(v); + } + return out; + } + return value; +} + +// Exported for unit testing. The public entry point is the internal +// `createAnalyzer` above; this is exposed only so tests can assert the +// key-renaming rules without needing to mock the Azure client. +export const _translateForJsSerializer = translateForJsSerializer; + +export function guessContentType(filePath: string): string { + const ext = extname(filePath).toLowerCase(); + switch (ext) { + case ".pdf": + return "application/pdf"; + case ".png": + return "image/png"; + case ".jpg": + case ".jpeg": + return "image/jpeg"; + case ".tif": + case ".tiff": + return "image/tiff"; + case ".bmp": + return "image/bmp"; + case ".heif": + case ".heic": + return "image/heif"; + case ".wav": + return "audio/wav"; + case ".mp3": + return "audio/mpeg"; + case ".m4a": + return "audio/mp4"; + case ".mp4": + return "video/mp4"; + case ".mov": + return "video/quicktime"; + default: + return "application/octet-stream"; + } +} + +export function enumerateInputs(inputPath: string): string[] { + if (!existsSync(inputPath)) { + return []; + } + const stats = statSync(inputPath); + if (stats.isFile()) { + return [inputPath]; + } + if (stats.isDirectory()) { + return readdirSync(inputPath) + .map((name) => join(inputPath, name)) + .filter((p) => statSync(p).isFile()) + .filter((p) => SUPPORTED_EXTENSIONS.has(extname(p).toLowerCase())) + .sort(); + } + return []; +} + +export function stripExtension(name: string): string { + const ext = extname(name); + return ext ? name.substring(0, name.length - ext.length) : name; +} + +export function readFileBytes(filePath: string): Uint8Array { + return new Uint8Array(readFileSync(filePath)); +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.test.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.test.ts new file mode 100644 index 000000000000..37a17790c484 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.test.ts @@ -0,0 +1,84 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Unit tests for the pure helpers exported from createAndTestCommand.ts. + * Mirrors Python's tests/test_skills_create_and_test.py. The CLI entry + * point (runCreateAndTest) and the network-dependent helpers (ensureAnalyzer, + * createAnalyzer, analyzeFile) are not covered here — they require an + * Azure client and live recording infrastructure. + */ + +import { strict as assert } from "node:assert"; +import { describe, it } from "node:test"; +import { summarize } from "./createAndTestCommand.js"; + +function scalar(value: string, conf: number): Record { + return { type: "string", valueString: value, confidence: conf }; +} + +function num(value: number, conf: number): Record { + return { type: "number", valueNumber: value, confidence: conf }; +} + +function arrayOfObjects(items: Record[]): Record { + return { type: "array", valueArray: items }; +} + +function obj(o: Record): Record { + return { type: "object", valueObject: o }; +} + +describe("summarize — leaf-row flattening", () => { + it("flattens nested valueArray and valueObject fields to leaf rows", () => { + const doc = { + contents: [ + { + fields: { + invoiceNumber: scalar("INV-100", 0.95), + lineItems: arrayOfObjects([ + obj({ + itemCode: scalar("A123", 0.8), + amount: num(60, 0.92), + }), + obj({ + itemCode: scalar("B456", 0.7), + amount: num(30, 0.9), + }), + ]), + address: obj({ + street: scalar("123 Main St", 0.88), + }), + }, + }, + ], + }; + + const out = summarize([{ name: "docX", doc }]); + + // Leaf rows present. + assert.ok(out.includes("lineItems[].itemCode"), "leaf row lineItems[].itemCode missing"); + assert.ok(out.includes("lineItems[].amount"), "leaf row lineItems[].amount missing"); + assert.ok(out.includes("address.street"), "leaf row address.street missing"); + assert.ok(out.includes("invoiceNumber"), "leaf row invoiceNumber missing"); + + // The old aggregate-only behaviour would emit a `lineItems` row with `n/a` + // confidence and no children. The new behaviour must not emit a bare + // `lineItems ` or `address ` row. + for (const line of out.split("\n")) { + const stripped = line.trim(); + assert.ok( + !(stripped.startsWith("lineItems ") || stripped.startsWith("address ")), + `aggregate-only row leaked: ${line}`, + ); + } + + // Lowest-confidence list should rank the 0.700 leaf at the top. + assert.ok(out.includes("0.700"), "missing 0.700 confidence row"); + const lowestSection = out.split("lowest-confidence")[1] ?? ""; + assert.ok( + lowestSection.includes("lineItems[].itemCode"), + "lowest-confidence section should mention lineItems[].itemCode", + ); + }); +}); diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.ts new file mode 100644 index 000000000000..3c53af141a56 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestCommand.ts @@ -0,0 +1,553 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Stage 2 of the analyzer-authoring loop (single-document-type variant): + * validate the schema locally, create the analyzer, batch-test inputs, dump + * per-document JSON + LLM-friendly markdown (best-effort), and print a + * stdout summary with per-field fill-rate + avg-confidence. Mirrors Python's + * create_and_test.py, .NET's CreateAndTestCommand.cs, and Java's + * CreateAndTestCommand.java. + */ + +import { createHash } from "node:crypto"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import type { ContentUnderstandingClient } from "@azure/ai-content-understanding"; +import { toLlmInput } from "@azure/ai-content-understanding"; +import { + buildClient, + capturedAnalyzeBinary, + createAnalyzerRaw, + enumerateInputs, + guessContentType, + readFileBytes, + stripExtension, +} from "./clientHelpers.js"; +import { validateFile } from "./schemaValidator.js"; + +export interface CreateAndTestOptions { + schema: string; + input: string; + output: string; + analyzerId: string; + iterations: number; + ephemeral: boolean; + reuse: boolean; +} + +interface NamedDoc { + name: string; + doc: Record; +} + +export async function runCreateAndTest(args: string[]): Promise { + const opts = parseArgs(args); + if (!opts) { + return 2; + } + return runCore(opts); +} + +export async function runCore(opts: CreateAndTestOptions): Promise { + // 1. Validate schema locally. + const validation = validateFile(opts.schema); + if (!validation.ok) { + for (const e of validation.errors) { + console.error(`[VALIDATE] ${e}`); + } + return 2; + } + + const rawJson = readFileSync(opts.schema, "utf-8"); + const rawSchema = JSON.parse(rawJson) as Record; + const schema = stripComments(rawSchema) as Record; + + // Pre-flight warning: fieldSchema without models.completion. + if ("fieldSchema" in schema) { + const models = schema["models"]; + const completion = + models && typeof models === "object" && !Array.isArray(models) + ? (models as Record)["completion"] + : undefined; + if (typeof completion !== "string" || completion.trim() === "") { + console.error( + "[WARN] schema has fieldSchema but no models.completion; " + + "this will fail unless resource defaults are configured " + + "(see samples-dev/updateDefaults.ts).", + ); + } + } + + const inputs = enumerateInputs(opts.input); + if (inputs.length === 0) { + console.error(`no supported documents found under ${opts.input}`); + return 2; + } + mkdirSync(opts.output, { recursive: true }); + + let analyzerId = opts.analyzerId; + if (!analyzerId) { + const stem = stripExtension( + opts.schema.substring(opts.schema.lastIndexOf("/") + 1), + ); + analyzerId = opts.reuse + ? `${stem}_${schemaHash(schema)}` + : `${stem}_${Math.floor(Date.now() / 1000)}`; + } + + const client = buildClient(); + let reused = false; + let fail = 0; + const results: NamedDoc[] = []; + try { + if (opts.reuse) { + reused = await ensureAnalyzer(client, analyzerId, schema); + } else { + await createAnalyzer(client, analyzerId, schema); + } + for (const file of inputs) { + for (let iter = 1; iter <= opts.iterations; iter++) { + const suffix = + opts.iterations > 1 ? `_iter${String(iter).padStart(3, "0")}` : ""; + const stem = stripExtension( + file.substring(file.lastIndexOf("/") + 1), + ); + const outPath = join(opts.output, `${stem}${suffix}.json`); + try { + console.log(`[ANALYZE] ${file} -> ${outPath}`); + const r = await analyzeFile(client, analyzerId, file); + writeFileSync(outPath, JSON.stringify(r.doc, null, 2)); + if (r.llmMarkdown) { + writeFileSync( + join(opts.output, `${stem}${suffix}.llm.md`), + r.llmMarkdown, + ); + } + results.push({ + name: stripExtension(outPath.substring(outPath.lastIndexOf("/") + 1)), + doc: r.doc, + }); + } catch (ex) { + const msg = ex instanceof Error ? ex.message : String(ex); + console.error(`[FAIL] ${file}: ${msg}`); + fail++; + } + } + } + } finally { + await cleanup(client, analyzerId, opts.ephemeral, reused); + } + + console.log(summarize(results)); + return fail === 0 ? 0 : 1; +} + +// --------------------------------------------------------------------------- +// Service interaction +// --------------------------------------------------------------------------- + +export async function ensureAnalyzer( + client: ContentUnderstandingClient, + analyzerId: string, + schema: Record, +): Promise { + try { + await client.getAnalyzer(analyzerId); + console.log(`[REUSE] analyzer ${analyzerId} already exists`); + return true; + } catch { + // Treat any error as "not found" — the next createAnalyzer call will + // surface the real reason if it's something else (auth, etc.). Mirrors + // Python's broad-except in ensure_analyzer. + } + await createAnalyzer(client, analyzerId, schema); + return false; +} + +export async function createAnalyzer( + client: ContentUnderstandingClient, + analyzerId: string, + schema: Record, +): Promise { + console.log(`[CREATE] analyzer_id=${analyzerId}`); + await createAnalyzerRaw(client, analyzerId, schema); + console.log(`[CREATE] ${analyzerId} ready`); +} + +export async function analyzeFile( + client: ContentUnderstandingClient, + analyzerId: string, + filePath: string, +): Promise<{ doc: Record; llmMarkdown: string | null }> { + const bytes = readFileBytes(filePath); + const contentType = guessContentType(filePath); + const doc = await capturedAnalyzeBinary(client, analyzerId, bytes, contentType); + + // Best-effort LLM-friendly markdown. The toLlmInput helper expects the + // typed model; cast through unknown since our `doc` is the raw shape. + let llmMarkdown: string | null = null; + try { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + llmMarkdown = toLlmInput(doc as any); + } catch { + // Raw JSON is always written; LLM text is optional. + } + return { doc, llmMarkdown }; +} + +async function cleanup( + client: ContentUnderstandingClient, + analyzerId: string, + ephemeral: boolean, + reused: boolean, +): Promise { + if (ephemeral) { + try { + console.log(`[CLEANUP] delete analyzer ${analyzerId}`); + await client.deleteAnalyzer(analyzerId); + } catch (ex) { + const msg = ex instanceof Error ? ex.message : String(ex); + console.error(`[CLEANUP] delete failed: ${msg}`); + } + } else if (reused) { + console.log(`[KEEP] reused analyzer ${analyzerId} retained`); + } else { + console.log( + `[KEEP] analyzer ${analyzerId} retained (use --ephemeral to delete)`, + ); + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +/** Recursively drop any object key whose name starts with `_`. */ +export function stripComments(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(stripComments); + } + if (value && typeof value === "object") { + const out: Record = {}; + for (const [k, v] of Object.entries(value as Record)) { + if (k.startsWith("_")) { + continue; + } + out[k] = stripComments(v); + } + return out; + } + return value; +} + +/** Stable 8-char sha1 over canonical JSON form. */ +export function schemaHash(schema: Record): string { + const canonical = canonicalize(schema); + return createHash("sha1").update(canonical).digest("hex").substring(0, 8); +} + +export function canonicalize(value: unknown): string { + return JSON.stringify(sortKeys(value)); +} + +function sortKeys(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(sortKeys); + } + if (value && typeof value === "object") { + const obj = value as Record; + const out: Record = {}; + for (const k of Object.keys(obj).sort()) { + out[k] = sortKeys(obj[k]); + } + return out; + } + return value; +} + +// --------------------------------------------------------------------------- +// Summary +// --------------------------------------------------------------------------- + +interface RowEntry { + docName: string; + value: unknown; + confidence: number | null; +} + +interface FieldRecord { + category: string; + fieldPath: string; + fieldVal: Record; +} + +export function summarize(results: NamedDoc[]): string { + // category -> field -> rows + const table = new Map>(); + for (const nd of results) { + for (const f of iterFields(nd.doc)) { + let perCat = table.get(f.category); + if (!perCat) { + perCat = new Map(); + table.set(f.category, perCat); + } + let rows = perCat.get(f.fieldPath); + if (!rows) { + rows = []; + perCat.set(f.fieldPath, rows); + } + rows.push({ + docName: nd.name, + value: fieldValue(f.fieldVal), + confidence: fieldConfidence(f.fieldVal), + }); + } + } + if (table.size === 0) { + return "[SUMMARY] no fields extracted across any document."; + } + + const lines: string[] = ["", "=".repeat(72), "[SUMMARY]"]; + for (const [category, perField] of table.entries()) { + const docNames = new Set(); + for (const rows of perField.values()) { + for (const r of rows) { + docNames.add(r.docName); + } + } + const nDocs = docNames.size; + const catLabel = category === "" ? "(single)" : category; + const header = `category: ${catLabel} (${nDocs} document${nDocs === 1 ? "" : "s"})`; + lines.push("", header, "-".repeat(header.length)); + lines.push( + ` ${"field".padEnd(40)} fill rate avg conf`, + ); + for (const [fname, rows] of perField.entries()) { + const denom = rows.length; + const filled = rows.filter((r) => r.value !== null); + const fillRate = denom === 0 ? 0 : filled.length / denom; + const confidences = filled + .map((r) => r.confidence) + .filter((c): c is number => c !== null); + const confStr = + confidences.length === 0 + ? " n/a" + : (confidences.reduce((a, b) => a + b, 0) / confidences.length).toFixed(3); + lines.push( + ` ${fname.padEnd(40)} ${(fillRate * 100).toFixed(1).padStart(5)}% ${confStr}`, + ); + } + } + + // Lowest-confidence triples across all categories. + const lows: { conf: number; cat: string; field: string; doc: string }[] = []; + for (const [category, perField] of table.entries()) { + for (const [fname, rows] of perField.entries()) { + for (const r of rows) { + if (r.value !== null && r.confidence !== null) { + lows.push({ conf: r.confidence, cat: category, field: fname, doc: r.docName }); + } + } + } + } + if (lows.length > 0) { + lows.sort((a, b) => a.conf - b.conf); + lines.push("", "lowest-confidence fields:"); + for (const row of lows.slice(0, 3)) { + const catTag = row.cat === "" ? "" : `[${row.cat}] `; + lines.push( + ` ${row.conf.toFixed(3)} ${catTag}${row.field} (${row.doc})`, + ); + } + } + lines.push("=".repeat(72)); + return lines.join("\n"); +} + +export function iterFields(doc: Record): FieldRecord[] { + const out: FieldRecord[] = []; + const contents = doc["contents"]; + if (!Array.isArray(contents)) { + return out; + } + for (const content of contents) { + if (!content || typeof content !== "object" || Array.isArray(content)) { + continue; + } + const cObj = content as Record; + const category = typeof cObj["category"] === "string" ? (cObj["category"] as string) : ""; + const fields = cObj["fields"]; + if (!fields || typeof fields !== "object" || Array.isArray(fields)) { + continue; + } + for (const [fname, fval] of Object.entries(fields as Record)) { + if (!fval || typeof fval !== "object" || Array.isArray(fval)) { + continue; + } + for (const pl of recurse(fname, fval as Record)) { + out.push({ category, fieldPath: pl.path, fieldVal: pl.leaf }); + } + } + } + return out; +} + +function recurse( + prefix: string, + fieldVal: Record, +): { path: string; leaf: Record }[] { + const out: { path: string; leaf: Record }[] = []; + const arr = fieldVal["valueArray"]; + if (Array.isArray(arr)) { + for (const item of arr) { + if ( + item && + typeof item === "object" && + !Array.isArray(item) && + (item as Record)["valueObject"] && + typeof (item as Record)["valueObject"] === "object" && + !Array.isArray((item as Record)["valueObject"]) + ) { + const vobj = (item as Record)["valueObject"] as Record; + for (const [childName, childVal] of Object.entries(vobj)) { + if (childVal && typeof childVal === "object" && !Array.isArray(childVal)) { + out.push( + ...recurse(`${prefix}[].${childName}`, childVal as Record), + ); + } + } + } else { + let wrap: Record; + if (typeof item === "string") { + wrap = { valueString: item }; + } else if (item && typeof item === "object" && !Array.isArray(item)) { + wrap = { ...(item as Record) }; + } else { + wrap = {}; + } + out.push({ path: prefix, leaf: wrap }); + } + } + return out; + } + const obj = fieldVal["valueObject"]; + if (obj && typeof obj === "object" && !Array.isArray(obj)) { + for (const [childName, childVal] of Object.entries(obj as Record)) { + if (childVal && typeof childVal === "object" && !Array.isArray(childVal)) { + out.push( + ...recurse(`${prefix}.${childName}`, childVal as Record), + ); + } + } + return out; + } + out.push({ path: prefix, leaf: fieldVal }); + return out; +} + +function fieldValue(field: Record): unknown { + for (const key of [ + "valueString", + "valueNumber", + "valueInteger", + "valueBoolean", + "valueDate", + "valueTime", + ]) { + const v = field[key]; + if (v !== undefined && v !== null && !(typeof v === "string" && v === "")) { + return v; + } + } + const arr = field["valueArray"]; + if (Array.isArray(arr) && arr.length > 0) { + return arr; + } + const obj = field["valueObject"]; + if (obj && typeof obj === "object" && !Array.isArray(obj) && Object.keys(obj).length > 0) { + return obj; + } + return null; +} + +function fieldConfidence(field: Record): number | null { + const c = field["confidence"]; + return typeof c === "number" ? c : null; +} + +// --------------------------------------------------------------------------- +// CLI parsing +// --------------------------------------------------------------------------- + +function parseArgs(args: string[]): CreateAndTestOptions | null { + const opts: CreateAndTestOptions = { + schema: "", + input: "", + output: "", + analyzerId: "", + iterations: 1, + ephemeral: false, + reuse: false, + }; + for (let i = 0; i < args.length; i++) { + switch (args[i]) { + case "--schema": + opts.schema = args[++i]; + break; + case "--input": + opts.input = args[++i]; + break; + case "--output": + opts.output = args[++i]; + break; + case "--analyzer-id": + opts.analyzerId = args[++i]; + break; + case "--iterations": + opts.iterations = parseInt(args[++i], 10); + break; + case "--ephemeral": + opts.ephemeral = true; + break; + case "--reuse": + opts.reuse = true; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + console.error(`unknown argument: ${args[i]}`); + printUsage(); + return null; + } + } + if (!opts.schema || !opts.input || !opts.output) { + console.error("--schema, --input, and --output are required"); + printUsage(); + return null; + } + if (opts.iterations < 1) { + console.error("--iterations must be >= 1"); + return null; + } + return opts; +} + +function printUsage(): void { + console.error("Usage:"); + console.error(" cu-skill create-and-test"); + console.error(" --schema "); + console.error(" --input "); + console.error(" --output "); + console.error(" [--analyzer-id ]"); + console.error(" [--iterations N]"); + console.error(" [--ephemeral]"); + console.error(" [--reuse]"); + console.error(""); + console.error( + "Stage 2 (single-type): validate, create, batch-test a custom analyzer,", + ); + console.error("print a per-field fill-rate + avg-confidence summary."); +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.test.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.test.ts new file mode 100644 index 000000000000..dbbe635e08d1 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.test.ts @@ -0,0 +1,423 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Unit tests for pure helpers in createAndTestRouterCommand.ts. Mirrors + * the portion of Python's tests/test_skills_classify_route_router.py that + * does not require mocking the Azure client. + */ + +import { strict as assert } from "node:assert"; +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, it } from "node:test"; +import { + discoverInnerFromDir, + summarizeRouted, + versionSortKey, + wireInnerIds, +} from "./createAndTestRouterCommand.js"; + +function field(value: string, confidence: number): Record { + return { valueString: value, confidence }; +} + +function segment(category: string, fields: Record): Record { + return { category, fields }; +} + +describe("summarizeRouted — per-category denominator", () => { + it("does not dilute invoice fill rate by other categories' segments", () => { + const results = [ + { + name: "packet_a", + doc: { + contents: [ + segment("invoice", { InvoiceNumber: field("INV-1", 0.9) }), + segment("invoice", { InvoiceNumber: field("INV-2", 0.91) }), + segment("invoice", { InvoiceNumber: field("INV-3", 0.92) }), + segment("bank_statement", { AccountNumber: field("12345", 0.8) }), + ], + }, + }, + ]; + + const text = summarizeRouted(results); + + // Invoice: 3 segments, 3 filled → 100% + assert.ok(text.includes("category: invoice (3 segments)"), "invoice segment count wrong"); + assert.ok(text.includes("InvoiceNumber") && text.includes("100.0%"), "invoice not at 100%"); + // Bank statement: 1 segment, 1 filled → 100% + assert.ok( + text.includes("category: bank_statement (1 segment)") || + text.includes("category: bank_statement (1 segments)"), + "bank_statement segment count wrong", + ); + assert.ok(text.includes("AccountNumber"), "AccountNumber missing"); + // Packet-wide denominator must NOT leak through. + assert.ok(!text.includes("33.3%"), "33.3% leaked (packet-wide denom)"); + assert.ok(!text.includes("25.0%"), "25.0% leaked (packet-wide denom)"); + }); + + it("reports zero fill for a missing field in some segments", () => { + const results = [ + { + name: "packet", + doc: { + contents: [ + segment("invoice", { + InvoiceNumber: field("INV-1", 0.9), + TotalAmount: field("$100", 0.7), + }), + segment("invoice", { InvoiceNumber: field("INV-2", 0.91) }), + ], + }, + }, + ]; + + const text = summarizeRouted(results); + assert.ok(text.includes("category: invoice (2 segments)"), "segment count wrong"); + // InvoiceNumber appears in both segments → 100% + // TotalAmount appears in 1 of 2 → 50% + assert.ok(text.includes("InvoiceNumber") && text.includes("100.0%"), "InvoiceNumber not 100%"); + assert.ok(text.includes("TotalAmount") && text.includes(" 50.0%"), "TotalAmount not 50%"); + }); +}); + +describe("wireInnerIds — alias-to-id substitution", () => { + it("substitutes analyzerId for matching aliases (keyed by value, not category name)", () => { + // Category name is deliberately different from the analyzerId value + // to catch the "keyed off cat name instead of alias" regression. + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice_bucket: { description: "d", analyzerId: "invoice" }, + loan_bucket: { description: "d", analyzerId: "loan_application" }, + }, + }, + }; + + const wired = wireInnerIds( + outer, + new Map([ + ["invoice", "real-invoice-id"], + ["loan_application", "real-loan-id"], + ]), + ); + + assert.deepEqual(wired.errors, [], "expected no errors"); + const cats = (wired.patched["config"] as Record)[ + "contentCategories" + ] as Record>; + assert.equal(cats["invoice_bucket"]["analyzerId"], "real-invoice-id"); + assert.equal(cats["loan_bucket"]["analyzerId"], "real-loan-id"); + // Outer schema must not be mutated in place. + assert.equal( + ( + (outer.config as Record)["contentCategories"] as Record< + string, + Record + > + )["invoice_bucket"]["analyzerId"], + "invoice", + "wireInnerIds mutated its input", + ); + }); + + it("keeps 'prebuilt-*' analyzerId values as-is (no --inner-schema required)", () => { + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "prebuilt-invoice" }, + receipt: { description: "d", analyzerId: "prebuilt-receipt" }, + }, + }, + }; + + const wired = wireInnerIds(outer, new Map()); + assert.deepEqual(wired.errors, [], "prebuilt-* values must not require aliases"); + const cats = (wired.patched["config"] as Record)[ + "contentCategories" + ] as Record>; + assert.equal(cats["invoice"]["analyzerId"], "prebuilt-invoice"); + assert.equal(cats["receipt"]["analyzerId"], "prebuilt-receipt"); + }); + + it("returns an error when a category references an unknown alias", () => { + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "invoice" }, + loan: { description: "d", analyzerId: "loan_application" }, + }, + }, + }; + + const wired = wireInnerIds(outer, new Map([["invoice", "real-invoice-id"]])); + assert.ok(wired.errors.length > 0, "expected an error for missing alias"); + assert.ok( + wired.errors.some((e) => e.includes("loan_application") && e.includes("loan")), + `expected error to name missing alias + category, got: ${JSON.stringify(wired.errors)}`, + ); + }); + + it("returns an error when an inner-schema alias is supplied but never used", () => { + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "invoice" }, + }, + }, + }; + + const wired = wireInnerIds( + outer, + new Map([ + ["invoice", "real-invoice-id"], + ["extra", "real-extra-id"], + ]), + ); + assert.ok(wired.errors.length > 0, "expected an error for unused alias"); + assert.ok( + wired.errors.some((e) => e.includes("extra") && e.includes("no category")), + `expected error to name unused alias, got: ${JSON.stringify(wired.errors)}`, + ); + }); + + it("allows categories without analyzerId (classification-only bucket)", () => { + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "invoice" }, + other: { description: "catch-all classification bucket" }, + }, + }, + }; + + const wired = wireInnerIds(outer, new Map([["invoice", "real-invoice-id"]])); + assert.deepEqual(wired.errors, [], "categories without analyzerId must be allowed"); + const cats = (wired.patched["config"] as Record)[ + "contentCategories" + ] as Record>; + assert.ok(!("analyzerId" in cats["other"]), "'other' must remain analyzerId-less"); + }); +}); + +describe("versionSortKey — pure key extractor", () => { + it("returns group 0 for the bare alias (no suffix)", () => { + const [g, v] = versionSortKey("invoice", "invoice"); + assert.equal(g, 0); + assert.equal(v, 0); + }); + + it("returns group 1 with numeric version for `_v` suffix (v10 > v9)", () => { + const v9 = versionSortKey("invoice_v9", "invoice"); + const v10 = versionSortKey("invoice_v10", "invoice"); + assert.deepEqual(v9, [1, 9, ""]); + assert.deepEqual(v10, [1, 10, ""]); + // The whole point of the fix. + assert.ok(v10[1] > v9[1], "v10 must sort higher than v9 by numeric version"); + }); + + it("also recognises `_` (no `v` prefix) as a numeric version", () => { + assert.deepEqual(versionSortKey("invoice_42", "invoice"), [1, 42, ""]); + }); + + it("returns group 2 with lexicographic suffix for non-numeric suffixes", () => { + assert.deepEqual(versionSortKey("invoice_draft", "invoice"), [2, 0, "draft"]); + }); +}); + +describe("discoverInnerFromDir — alias-to-file resolution", () => { + // Each test writes a fresh temp dir, sets up files, exercises the helper, + // then rms. Mirrors the discovery behaviour of Python's + // `_discover_inner_from_dir` and .NET's `DiscoverInnerFromDir`. + + function outerWith(aliases: Array): Record { + // Build an outer classifier schema with N categories, each carrying the + // supplied analyzerId (or none if `null`). + const cats: Record> = {}; + aliases.forEach((a, i) => { + const entry: Record = { description: `d${i}` }; + if (a !== null) entry["analyzerId"] = a; + cats[`cat_${i}`] = entry; + }); + return { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: cats, + }, + }; + } + + function withTempDir(fn: (dir: string) => void): void { + const dir = mkdtempSync(join(tmpdir(), "cu-skill-discover-")); + try { + fn(dir); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } + + it("resolves aliases by exact-match filename stem", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + writeFileSync(join(dir, "bank_statement.json"), "{}"); + + const outer = outerWith(["invoice", "bank_statement"]); + const resolved = discoverInnerFromDir(outer, dir); + + assert.ok(resolved !== null, "should not return null on success"); + assert.equal(resolved!.size, 2); + assert.equal(resolved!.get("invoice"), join(dir, "invoice.json")); + assert.equal(resolved!.get("bank_statement"), join(dir, "bank_statement.json")); + }); + }); + + it("picks natural version max, so `_v10` beats `_v9` (regression: alphabetical was wrong)", () => { + // Regression: the previous implementation did `.sort()` and took the + // last element, so alphabetical order applied — but '1' < '9' char- + // by-char, meaning `invoice_v10.json` sorted BEFORE `invoice_v9.json`. + // "Alphabetical last" then picked v9, silently loading the older + // schema. Copilot flagged this on the .NET PR (#60394); the natural + // version sort fix mirrors what .NET / Java / Python ship now. + withTempDir((dir) => { + writeFileSync(join(dir, "invoice_v1.json"), "{}"); + writeFileSync(join(dir, "invoice_v2.json"), "{}"); + writeFileSync(join(dir, "invoice_v9.json"), "{}"); + writeFileSync(join(dir, "invoice_v10.json"), "{}"); + + const resolved = discoverInnerFromDir(outerWith(["invoice"]), dir); + assert.ok(resolved !== null); + assert.equal( + resolved!.get("invoice"), + join(dir, "invoice_v10.json"), + "v10 must beat v9 (natural version order, not alphabetical)", + ); + }); + }); + + it("prefers versioned suffix over bare alias baseline", () => { + // A bare `.json` is group 0 (baseline); versioned files are + // group 1 (numeric) or group 2 (other suffix). So any versioned + // file beats the bare baseline as "newer". + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + writeFileSync(join(dir, "invoice_v1.json"), "{}"); + + const resolved = discoverInnerFromDir(outerWith(["invoice"]), dir); + assert.ok(resolved !== null); + assert.equal(resolved!.get("invoice"), join(dir, "invoice_v1.json")); + }); + }); + + it("skips categories whose analyzerId starts with `prebuilt-`", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + // No `prebuilt-invoice.json` needed on disk — it's a service alias. + + const outer = outerWith(["invoice", "prebuilt-invoice"]); + const resolved = discoverInnerFromDir(outer, dir); + + assert.ok(resolved !== null, "prebuilt-* must not cause a failure"); + assert.equal(resolved!.size, 1); + assert.equal(resolved!.has("invoice"), true); + assert.equal(resolved!.has("prebuilt-invoice"), false); + }); + }); + + it("skips categories without an `analyzerId` (classification-only buckets)", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + + const outer = outerWith(["invoice", null]); + const resolved = discoverInnerFromDir(outer, dir); + + assert.ok(resolved !== null); + assert.equal(resolved!.size, 1); + }); + }); + + it("returns null and reports every missing alias in the error message", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + + const errs: string[] = []; + const origError = console.error; + console.error = (msg: string): void => { + errs.push(msg); + }; + try { + const resolved = discoverInnerFromDir( + outerWith(["invoice", "bank_statement", "loan_application"]), + dir, + ); + assert.equal(resolved, null, "missing aliases must yield null"); + } finally { + console.error = origError; + } + + const joined = errs.join(" "); + assert.ok(joined.includes("bank_statement"), "error must name bank_statement"); + assert.ok(joined.includes("loan_application"), "error must name loan_application"); + assert.ok(!joined.includes("[invoice]"), "resolved alias must not be listed as missing"); + }); + }); + + it("ignores unrelated JSON files in the directory", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + writeFileSync(join(dir, "notes.json"), "{}"); // not a category alias + writeFileSync(join(dir, "settings.json"), "{}"); // not a category alias + + const resolved = discoverInnerFromDir(outerWith(["invoice"]), dir); + assert.ok(resolved !== null); + assert.equal(resolved!.size, 1); + assert.equal(resolved!.get("invoice"), join(dir, "invoice.json")); + }); + }); + + it("treats a category with a non-string analyzerId as skippable", () => { + withTempDir((dir) => { + writeFileSync(join(dir, "invoice.json"), "{}"); + + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "invoice" }, + broken: { description: "d", analyzerId: 42 as unknown as string }, + }, + }, + }; + const resolved = discoverInnerFromDir(outer, dir); + assert.ok(resolved !== null, "non-string analyzerId must be tolerated"); + assert.equal(resolved!.size, 1); + }); + }); + + it("returns an empty map when the outer schema has no categories at all", () => { + withTempDir((dir) => { + const outer = { + baseAnalyzerId: "prebuilt-document", + config: { enableSegment: true, contentCategories: {} }, + }; + const resolved = discoverInnerFromDir(outer, dir); + assert.ok(resolved !== null); + assert.equal(resolved!.size, 0); + }); + }); +}); diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.ts new file mode 100644 index 000000000000..c832aff16684 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/createAndTestRouterCommand.ts @@ -0,0 +1,637 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Stage 2 (classify-and-route variant): given an outer classifier schema and + * N inner field-extractor schemas, create all of them, wire the outer's + * `contentCategories[*].analyzerId` to the real inner analyzer IDs, + * batch-test inputs, and print a category-aware stdout summary. Mirrors + * Python's create_and_test_router.py, .NET's CreateAndTestRouterCommand.cs, + * and Java's CreateAndTestRouterCommand.java. + */ + +import { createHash } from "node:crypto"; +import { existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { + analyzeFile, + canonicalize, + createAnalyzer, + ensureAnalyzer, + iterFields, + stripComments, +} from "./createAndTestCommand.js"; +import { buildClient, enumerateInputs, stripExtension } from "./clientHelpers.js"; +import { validateFile } from "./schemaValidator.js"; + +interface RouterOptions { + outerSchema: string; + innerSchemas: string[]; + schemaDir: string | null; + input: string; + output: string; + analyzerId: string; + ephemeral: boolean; + reuse: boolean; +} + +interface NamedDoc { + name: string; + doc: Record; +} + +export async function runCreateAndTestRouter(args: string[]): Promise { + const opts = parseArgs(args); + if (!opts) { + return 2; + } + if (!existsSync(opts.outerSchema)) { + console.error(`outer schema not found: ${opts.outerSchema}`); + return 2; + } + if (!existsSync(opts.input)) { + console.error(`input not found: ${opts.input}`); + return 2; + } + + // 1. Resolve inner schemas. + const aliasToPath = new Map(); + for (const spec of opts.innerSchemas) { + const eq = spec.indexOf("="); + if (eq <= 0) { + console.error(`--inner-schema must be ALIAS=PATH; got: ${spec}`); + return 2; + } + const alias = spec.substring(0, eq); + const path = spec.substring(eq + 1); + if (!existsSync(path)) { + console.error(`inner schema not found: ${path}`); + return 2; + } + aliasToPath.set(alias, path); + } + if (opts.schemaDir) { + if (!statSync(opts.schemaDir).isDirectory()) { + console.error(`--schema-dir is not a directory: ${opts.schemaDir}`); + return 2; + } + // Read the outer schema so we know which aliases to look for. Matches + // Python's `_discover_inner_from_dir` and .NET's `DiscoverInnerFromDir`: + // for every category whose `analyzerId` is a non-`prebuilt-*` string, + // find a file whose stem is exactly `` or starts with `_`, + // and pick the highest-numbered version (so `invoice_v10.json` wins + // over `invoice_v9.json`). + let outerPreview: Record; + try { + outerPreview = stripComments(JSON.parse(readFileSync(opts.outerSchema, "utf-8"))) as Record< + string, + unknown + >; + } catch (e) { + console.error(`outer schema is not valid JSON: ${(e as Error).message}`); + return 2; + } + const resolved = discoverInnerFromDir(outerPreview, opts.schemaDir); + if (resolved === null) { + return 2; + } + for (const [alias, path] of resolved.entries()) { + if (!aliasToPath.has(alias)) { + aliasToPath.set(alias, path); + } + } + const summary = [...resolved.entries()] + .map(([a, p]) => `${a}=${p.substring(p.lastIndexOf("/") + 1)}`) + .join(", "); + console.log(`[SCHEMA-DIR] resolved: ${summary}`); + } + if (aliasToPath.size === 0) { + console.error("provide at least one --inner-schema or --schema-dir"); + return 2; + } + + // 2. Validate outer + each inner locally. + const outerVal = validateFile(opts.outerSchema); + if (!outerVal.ok) { + for (const e of outerVal.errors) { + console.error(`[VALIDATE outer] ${e}`); + } + return 2; + } + const outerSchema = stripComments(JSON.parse(readFileSync(opts.outerSchema, "utf-8"))) as Record< + string, + unknown + >; + + const aliasToSchema = new Map>(); + for (const [alias, path] of aliasToPath.entries()) { + const r = validateFile(path); + if (!r.ok) { + for (const msg of r.errors) { + console.error(`[VALIDATE inner ${alias}] ${msg}`); + } + return 2; + } + aliasToSchema.set( + alias, + stripComments(JSON.parse(readFileSync(path, "utf-8"))) as Record, + ); + } + + // Cross-check is handled by wireInnerIds() below in one place: every + // outer contentCategories[].analyzerId must either start with "prebuilt-" + // (a service prebuilt) or resolve to an --inner-schema alias, and no + // --inner-schema may be unused. + + const inputs = enumerateInputs(opts.input); + if (inputs.length === 0) { + console.error(`no supported documents found under ${opts.input}`); + return 2; + } + mkdirSync(opts.output, { recursive: true }); + + // 3. Compute deterministic IDs (or timestamp IDs when --reuse not set). + let outerId = opts.analyzerId; + if (!outerId) { + const stem = stripExtension(opts.outerSchema.substring(opts.outerSchema.lastIndexOf("/") + 1)); + // Include the inner hashes in the outer hash so any inner-schema edit + // forces a fresh outer ID. Otherwise --reuse could pick up an outer + // that points at stale inner IDs. + const innerKeysSorted = [...aliasToSchema.keys()].sort(); + let combined = canonicalize(outerSchema); + for (const alias of innerKeysSorted) { + combined += `|${alias}=${canonicalize(aliasToSchema.get(alias)!)}`; + } + outerId = opts.reuse + ? `${stem}_${sha1Prefix(combined)}` + : `${stem}_${Math.floor(Date.now() / 1000)}`; + } + + const aliasToId = new Map(); + for (const [alias, schema] of aliasToSchema.entries()) { + const suffix = opts.reuse + ? sha1Prefix(canonicalize(schema)) + : String(Math.floor(Date.now() / 1000)); + aliasToId.set(alias, `${outerId}_inner_${alias}_${suffix}`); + } + + // 4. Patch outer schema to point at the real inner IDs. This also + // validates that every outer contentCategories[].analyzerId resolves + // to an --inner-schema alias (or starts with "prebuilt-") and that + // no --inner-schema is unused. + const wired = wireInnerIds(outerSchema, aliasToId); + if (wired.errors.length > 0) { + for (const e of wired.errors) { + console.error(`[VALIDATE] ${e}`); + } + return 2; + } + const wiredOuter = wired.patched; + + const client = buildClient(); + let reused = false; + let fail = 0; + const results: NamedDoc[] = []; + try { + // 5. Create all inner analyzers first. + for (const [alias, schema] of aliasToSchema.entries()) { + const id = aliasToId.get(alias)!; + if (opts.reuse) { + await ensureAnalyzer(client, id, schema); + } else { + await createAnalyzer(client, id, schema); + } + } + // 6. Then the outer classifier. + if (opts.reuse) { + reused = await ensureAnalyzer(client, outerId, wiredOuter); + } else { + await createAnalyzer(client, outerId, wiredOuter); + } + + // 7. Analyze inputs through the outer. + for (const file of inputs) { + const stem = stripExtension(file.substring(file.lastIndexOf("/") + 1)); + const outPath = join(opts.output, `${stem}.json`); + try { + console.log(`[ANALYZE] ${file} -> ${outPath}`); + const r = await analyzeFile(client, outerId, file); + writeFileSync(outPath, JSON.stringify(r.doc, null, 2)); + if (r.llmMarkdown) { + writeFileSync(join(opts.output, `${stem}.llm.md`), r.llmMarkdown); + } + results.push({ name: stem, doc: r.doc }); + } catch (ex) { + const msg = ex instanceof Error ? ex.message : String(ex); + console.error(`[FAIL] ${file}: ${msg}`); + fail++; + } + } + } finally { + if (opts.ephemeral) { + const toDelete = [outerId, ...aliasToId.values()]; + for (const id of toDelete) { + try { + console.log(`[CLEANUP] delete ${id}`); + await client.deleteAnalyzer(id); + } catch (ex) { + const msg = ex instanceof Error ? ex.message : String(ex); + console.error(`[CLEANUP] delete failed for ${id}: ${msg}`); + } + } + } else if (reused) { + console.log(`[KEEP] reused analyzers retained`); + } else { + console.log(`[KEEP] analyzers retained (use --ephemeral to delete)`); + console.log(` outer: ${outerId}`); + for (const [alias, id] of aliasToId.entries()) { + console.log(` inner [${alias}]: ${id}`); + } + } + } + + console.log(summarizeRouted(results)); + return fail === 0 ? 0 : 1; +} + +export interface WireResult { + patched: Record; + errors: string[]; +} + +/** + * Auto-build `{alias: path}` from a directory of inner schema files. For + * every category in the outer schema whose `analyzerId` is a non-prebuilt + * alias, find a matching JSON file: filename stem equals the alias, or the + * stem starts with `_`. When multiple files match, pick the highest- + * numbered version — so `invoice_v10.json` beats `invoice_v9.json` + * (plain alphabetical sort broke as soon as version numbers hit two digits). + * Missing aliases are logged and null is returned. + * + * Mirrors Python's `_discover_inner_from_dir` and .NET's `DiscoverInnerFromDir`. + */ +export function discoverInnerFromDir( + outerSchema: Record, + schemaDir: string, +): Map | null { + const config = outerSchema["config"]; + const cats = + config && typeof config === "object" && !Array.isArray(config) + ? (config as Record)["contentCategories"] + : undefined; + const aliases: string[] = []; + if (cats && typeof cats === "object" && !Array.isArray(cats)) { + for (const entry of Object.values(cats as Record)) { + if (!entry || typeof entry !== "object" || Array.isArray(entry)) { + continue; + } + const aliasVal = (entry as Record)["analyzerId"]; + if (typeof aliasVal !== "string" || aliasVal.startsWith("prebuilt-")) { + continue; + } + aliases.push(aliasVal); + } + } + + const jsonFiles = readdirSync(schemaDir).filter((n) => n.endsWith(".json")); + + const resolved = new Map(); + const missing: string[] = []; + for (const alias of aliases) { + const matches = jsonFiles + .filter((n) => { + const stem = stripExtension(n); + return stem === alias || stem.startsWith(`${alias}_`); + }) + // Sort so the LAST element is the "newest" per the version key. + .sort((a, b) => + compareVersionKeys( + versionSortKey(stripExtension(a), alias), + versionSortKey(stripExtension(b), alias), + ), + ); + if (matches.length === 0) { + missing.push(alias); + continue; + } + resolved.set(alias, join(schemaDir, matches[matches.length - 1]!)); + } + + if (missing.length > 0) { + console.error( + `--schema-dir could not resolve inner schemas for: [${missing.join(", ")}]. ` + + `Looked in ${schemaDir} for files named .json or _*.json.`, + ); + return null; + } + return resolved; +} + +/** + * Sort key for `[_suffix]` filename stems. Higher tuple = newer. + * + * Group 0: bare `` (no suffix) — oldest baseline. + * Group 1: numeric suffix `_v` or + * `_` — sorted by N as an integer, so v10 beats v9. + * Group 2: any other suffix `_` — lexicographic tiebreaker. + * + * Exported for unit testing; the compare helper below turns the tuple into + * an `Array.prototype.sort` comparator. + */ +export function versionSortKey(stem: string, alias: string): [number, number, string] { + if (stem === alias) return [0, 0, ""]; + // We only get here for stems that already matched the `_` filter, + // so the underscore is guaranteed to be at index alias.length. + const suffix = stem.substring(alias.length + 1); + const m = /^v?(\d+)$/.exec(suffix); + if (m) { + const n = Number.parseInt(m[1]!, 10); + if (Number.isFinite(n)) return [1, n, ""]; + } + return [2, 0, suffix]; +} + +function compareVersionKeys(a: [number, number, string], b: [number, number, string]): number { + if (a[0] !== b[0]) return a[0] - b[0]; + if (a[1] !== b[1]) return a[1] - b[1]; + return a[2] < b[2] ? -1 : a[2] > b[2] ? 1 : 0; +} + +/** + * Replace each `contentCategories[].analyzerId` placeholder in the outer + * schema with the real inner analyzer ID from `aliasToId`. + * + * Rules (parity with Python's `_patch_outer_analyzer_ids` and .NET's + * `WireInnerIds`): + * + * - Categories that omit `analyzerId` are left as-is (a classification-only + * "other" bucket). + * - `analyzerId` values that start with `"prebuilt-"` are kept as-is — those + * are Azure-side prebuilt analyzers and don't need an `--inner-schema`. + * - Any other value must match an alias in `aliasToId`; otherwise a + * validation error is returned. + * - Every alias in `aliasToId` must be referenced by some category (typo + * check); otherwise an "unused" error is returned. + */ +export function wireInnerIds( + outerSchema: Record, + aliasToId: Map, +): WireResult { + const errors: string[] = []; + const patched = JSON.parse(JSON.stringify(outerSchema)) as Record; + const config = patched["config"]; + if (!config || typeof config !== "object" || Array.isArray(config)) { + return { patched, errors }; + } + const cats = (config as Record)["contentCategories"]; + if (!cats || typeof cats !== "object" || Array.isArray(cats)) { + return { patched, errors }; + } + + const usedRealIds = new Set(); + for (const [catName, catEntry] of Object.entries(cats as Record)) { + if (!catEntry || typeof catEntry !== "object" || Array.isArray(catEntry)) { + continue; + } + const entryObj = catEntry as Record; + const aliasVal = entryObj["analyzerId"]; + if (aliasVal === undefined || aliasVal === null) { + continue; + } + const alias = String(aliasVal); + if (alias.startsWith("prebuilt-")) { + usedRealIds.add(alias); + continue; + } + const real = aliasToId.get(alias); + if (real === undefined) { + const known = [...aliasToId.keys()].sort(); + errors.push( + `category '${catName}' references analyzerId='${alias}', but no ` + + `--inner-schema entry matches alias '${alias}'. Known aliases: [${known.join(", ")}]`, + ); + continue; + } + entryObj["analyzerId"] = real; + usedRealIds.add(real); + } + + // Catch unused inner schemas (cheap typo check). + for (const [alias, real] of aliasToId.entries()) { + if (!usedRealIds.has(real)) { + errors.push( + `--inner-schema '${alias}' was supplied but no category in the outer schema routes to it`, + ); + } + } + + return { patched, errors }; +} + +interface RowEntry { + docName: string; + value: unknown; + confidence: number | null; +} + +export function summarizeRouted(results: NamedDoc[]): string { + const table = new Map>(); + const segmentsPerCategory = new Map(); + for (const nd of results) { + const segCounts = new Map(); + const contents = nd.doc["contents"]; + if (Array.isArray(contents)) { + for (const c of contents) { + const cat = + c && + typeof c === "object" && + !Array.isArray(c) && + typeof (c as Record)["category"] === "string" + ? ((c as Record)["category"] as string) + : ""; + segCounts.set(cat, (segCounts.get(cat) ?? 0) + 1); + } + } + for (const [cat, count] of segCounts.entries()) { + segmentsPerCategory.set(cat, (segmentsPerCategory.get(cat) ?? 0) + count); + } + for (const f of iterFields(nd.doc)) { + let perCat = table.get(f.category); + if (!perCat) { + perCat = new Map(); + table.set(f.category, perCat); + } + let rows = perCat.get(f.fieldPath); + if (!rows) { + rows = []; + perCat.set(f.fieldPath, rows); + } + rows.push({ + docName: nd.name, + value: fieldValueRouted(f.fieldVal), + confidence: fieldConfRouted(f.fieldVal), + }); + } + } + if (table.size === 0) { + return "[SUMMARY] (category-aware) no fields extracted across any document."; + } + const lines: string[] = ["", "=".repeat(72), "[SUMMARY] (category-aware)"]; + const cats = [...table.keys()].sort(); + for (const cat of cats) { + const label = cat === "" ? "(uncategorized)" : cat; + const segCount = segmentsPerCategory.get(cat) ?? 0; + const header = `category: ${label} (${segCount} segment${segCount === 1 ? "" : "s"})`; + lines.push("", header, "-".repeat(header.length)); + lines.push(` ${"field".padEnd(30)} fill rate avg conf`); + const perField = table.get(cat)!; + const fieldEntries = [...perField.entries()].sort((a, b) => a[0].localeCompare(b[0])); + for (const [fname, rows] of fieldEntries) { + // Denominator is the per-category segment count so a field that's + // only meaningful in one category isn't penalised by other + // categories' segment counts, but a field that's missing from some + // segments in this category still reports the correct fraction. + // Mirrors Python's summarize_routed and .NET's SummarizeRouted. + const denom = segCount; + const filled = rows.filter((r) => r.value !== null); + const fillRate = denom === 0 ? 0 : filled.length / denom; + const confidences = filled.map((r) => r.confidence).filter((c): c is number => c !== null); + const confStr = + confidences.length === 0 + ? " n/a" + : (confidences.reduce((a, b) => a + b, 0) / confidences.length).toFixed(3); + lines.push( + ` ${fname.padEnd(30)} ${(fillRate * 100).toFixed(1).padStart(5)}% ${confStr}`, + ); + } + } + // Lowest-confidence triples. + const lows: { conf: number; cat: string; field: string; doc: string }[] = []; + for (const [cat, perField] of table.entries()) { + for (const [fname, rows] of perField.entries()) { + for (const r of rows) { + if (r.value !== null && r.confidence !== null) { + lows.push({ conf: r.confidence, cat, field: fname, doc: r.docName }); + } + } + } + } + if (lows.length > 0) { + lows.sort((a, b) => a.conf - b.conf); + lines.push("", "lowest-confidence fields across all categories:"); + for (const row of lows.slice(0, 3)) { + lines.push(` ${row.conf.toFixed(3)} [${row.cat}] ${row.field} (${row.doc})`); + } + } + lines.push("=".repeat(72)); + return lines.join("\n"); +} + +function fieldValueRouted(field: Record): unknown { + for (const key of [ + "valueString", + "valueNumber", + "valueInteger", + "valueBoolean", + "valueDate", + "valueTime", + ]) { + const v = field[key]; + if (v !== undefined && v !== null && !(typeof v === "string" && v === "")) { + return v; + } + } + const arr = field["valueArray"]; + if (Array.isArray(arr) && arr.length > 0) { + return arr; + } + const obj = field["valueObject"]; + if (obj && typeof obj === "object" && !Array.isArray(obj) && Object.keys(obj).length > 0) { + return obj; + } + return null; +} + +function fieldConfRouted(field: Record): number | null { + const c = field["confidence"]; + return typeof c === "number" ? c : null; +} + +function sha1Prefix(s: string): string { + return createHash("sha1").update(s).digest("hex").substring(0, 8); +} + +function parseArgs(args: string[]): RouterOptions | null { + const opts: RouterOptions = { + outerSchema: "", + innerSchemas: [], + schemaDir: null, + input: "", + output: "", + analyzerId: "", + ephemeral: false, + reuse: false, + }; + for (let i = 0; i < args.length; i++) { + switch (args[i]) { + case "--outer-schema": + opts.outerSchema = args[++i]; + break; + case "--inner-schema": + opts.innerSchemas.push(args[++i]); + break; + case "--schema-dir": + opts.schemaDir = args[++i]; + break; + case "--input": + opts.input = args[++i]; + break; + case "--output": + opts.output = args[++i]; + break; + case "--analyzer-id": + opts.analyzerId = args[++i]; + break; + case "--ephemeral": + opts.ephemeral = true; + break; + case "--reuse": + opts.reuse = true; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + console.error(`unknown argument: ${args[i]}`); + printUsage(); + return null; + } + } + if (!opts.outerSchema || !opts.input || !opts.output) { + console.error("--outer-schema, --input, and --output are required"); + printUsage(); + return null; + } + if (opts.innerSchemas.length === 0 && !opts.schemaDir) { + console.error("provide at least one --inner-schema ALIAS=PATH or --schema-dir DIR"); + printUsage(); + return null; + } + return opts; +} + +function printUsage(): void { + console.error("Usage:"); + console.error(" cu-skill create-and-test-router"); + console.error(" --outer-schema "); + console.error(" (--inner-schema ALIAS=PATH [--inner-schema ...] | --schema-dir )"); + console.error(" --input "); + console.error(" --output "); + console.error(" [--analyzer-id ]"); + console.error(" [--ephemeral]"); + console.error(" [--reuse]"); + console.error(""); + console.error("Classify-and-route Stage 2: validate, create N inner analyzers + 1"); + console.error("outer classifier, batch-test, print a category-aware summary."); +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/extractLayoutCommand.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/extractLayoutCommand.ts new file mode 100644 index 000000000000..dc0556ffd1b4 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/extractLayoutCommand.ts @@ -0,0 +1,138 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Stage 1 of the analyzer-authoring loop: extract document layout into + * .layout.{json,md} files. Mirrors Python's extract_layout.py, the + * .NET ExtractLayoutCommand.cs, and the Java ExtractLayoutCommand.java. + * + * Defaults to analyzerId = prebuilt-documentSearch (richer markdown than + * prebuilt-document) so the layout output is useful as Copilot context. + */ + +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { + buildClient, + capturedAnalyzeBinary, + enumerateInputs, + guessContentType, + readFileBytes, + stripExtension, +} from "./clientHelpers.js"; + +interface Options { + input: string; + output: string; + analyzerId: string; +} + +export async function runExtractLayout(args: string[]): Promise { + const opts = parseArgs(args); + if (!opts) { + return 2; + } + const inputs = enumerateInputs(opts.input); + if (inputs.length === 0) { + console.error(`no supported documents found under ${opts.input}`); + return 2; + } + mkdirSync(opts.output, { recursive: true }); + + const client = buildClient(); + let ok = 0; + let fail = 0; + for (const file of inputs) { + const stem = stripExtension(file.substring(file.lastIndexOf("/") + 1)); + try { + console.log( + `[RUN ] ${file} -> ${opts.output}/${stem}.layout.{json,md}`, + ); + const bytes = readFileBytes(file); + const contentType = guessContentType(file); + const payload = await capturedAnalyzeBinary( + client, + opts.analyzerId, + bytes, + contentType, + ); + + writeFileSync( + join(opts.output, `${stem}.layout.json`), + JSON.stringify(payload, null, 2), + ); + + const markdown = extractMarkdown(payload); + writeFileSync(join(opts.output, `${stem}.layout.md`), markdown); + ok++; + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.error(`[FAIL] ${file}: ${msg}`); + fail++; + } + } + + console.log(); + console.log( + `[DONE] ${ok} ok, ${fail} failed; output -> ${opts.output}`, + ); + return fail === 0 ? 0 : 1; +} + +function extractMarkdown(payload: Record): string { + const contents = payload["contents"]; + if (!Array.isArray(contents)) { + return ""; + } + for (const c of contents) { + if (c && typeof c === "object" && !Array.isArray(c)) { + const md = (c as Record)["markdown"]; + if (typeof md === "string" && md.length > 0) { + return md; + } + } + } + return ""; +} + +function parseArgs(args: string[]): Options | null { + let input: string | undefined; + let output: string | undefined; + let analyzerId = "prebuilt-documentSearch"; + for (let i = 0; i < args.length; i++) { + switch (args[i]) { + case "--input": + input = args[++i]; + break; + case "--output": + output = args[++i]; + break; + case "--analyzer-id": + analyzerId = args[++i]; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + console.error(`unknown argument: ${args[i]}`); + printUsage(); + return null; + } + } + if (!input || !output) { + console.error("--input and --output are required"); + printUsage(); + return null; + } + return { input, output, analyzerId }; +} + +function printUsage(): void { + console.error("Usage:"); + console.error( + " cu-skill extract-layout --input --output [--analyzer-id ]", + ); + console.error(); + console.error("Stage 1: extract layout JSON + markdown for each input document."); +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.test.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.test.ts new file mode 100644 index 000000000000..a1a705ae763d --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.test.ts @@ -0,0 +1,300 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Unit tests for schemaValidator.ts. Mirrors Python's + * tests/test_skills_shared_schema_validator.py, the .NET + * SkillSchemaValidatorTests.cs, and the Java SchemaValidatorTest.java. + * Pure native — no @azure/* deps, enforced by a purity guard test below. + */ + +import { strict as assert } from "node:assert"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, it } from "node:test"; +import { + KNOWN_BASE_ANALYZER_IDS, + validate, + validateFile, + validateString, +} from "./schemaValidator.js"; + +describe("SchemaValidator — valid schemas", () => { + it("accepts a valid single-type schema", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { + fields: { + invoiceNumber: { + type: "string", + method: "extract", + description: "Invoice number printed at the top right.", + }, + }, + }, + }); + assert.equal(r.ok, true, `Errors: ${r.errors.join("; ")}`); + assert.deepEqual(r.errors, []); + }); + + it("accepts a valid classify-and-route schema", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { + description: "Pages whose top heading is 'Invoice'.", + analyzerId: "invoice_extractor_v1", + }, + bank_statement: { + description: "Pages whose top heading is 'Bank Statement'.", + analyzerId: "bank_statement_extractor_v1", + }, + }, + }, + }); + assert.equal(r.ok, true, `Errors: ${r.errors.join("; ")}`); + }); + + it("allows classify-route category without analyzerId (catch-all 'other' bucket)", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "Invoices.", analyzerId: "inv" }, + other: { description: "Anything else." }, + }, + }, + }); + assert.equal(r.ok, true, `Errors: ${r.errors.join("; ")}`); + }); +}); + +describe("SchemaValidator — single-type rejections", () => { + it("rejects unknown baseAnalyzerId (catches prebuilt-documentAnalyzer typo)", () => { + // The service returns InvalidBaseAnalyzerId without a useful message, + // so we catch it locally with the actual allow-list. + const r = validate({ + baseAnalyzerId: "prebuilt-documentAnalyzer", + fieldSchema: { fields: { x: { type: "string" } } }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("baseAnalyzerId"))); + assert.ok(r.errors.some((e) => e.includes("prebuilt-documentAnalyzer"))); + }); + + it("rejects missing fieldSchema on non-classifier", () => { + const r = validate({ baseAnalyzerId: "prebuilt-document" }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("fieldSchema"))); + }); + + it("rejects empty fields object", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { fields: {} }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("at least one field"))); + }); + + it("rejects unknown field type", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { fields: { x: { type: "float" } } }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("'float'"))); + }); + + it("rejects unknown field method", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { fields: { x: { type: "string", method: "infer" } } }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("'infer'"))); + }); + + it("recurses into object.properties", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { + fields: { + billTo: { + type: "object", + properties: { + name: { type: "bogus" }, + }, + }, + }, + }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("billTo"))); + assert.ok(r.errors.some((e) => e.includes("'bogus'"))); + }); + + it("recurses into array.items", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { + fields: { + lineItems: { + type: "array", + items: { type: "nope" }, + }, + }, + }, + }); + assert.equal(r.ok, false); + // Path uses bracketed notation: fieldSchema.fields['lineItems'].items.type + assert.ok(r.errors.some((e) => e.includes("'lineItems'"))); + assert.ok(r.errors.some((e) => e.includes(".items.type"))); + assert.ok(r.errors.some((e) => e.includes("'nope'"))); + }); +}); + +describe("SchemaValidator — classify-and-route rejections", () => { + it("rejects classify-route with top-level fieldSchema", () => { + // Field extraction belongs in inner analyzers, not the outer classifier. + const r = validate({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { fields: { x: { type: "string" } } }, + config: { + enableSegment: true, + contentCategories: { + invoice: { description: "d", analyzerId: "a" }, + }, + }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("fieldSchema") && e.includes("inner"))); + }); + + it("rejects classify-route without enableSegment", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + config: { + contentCategories: { + invoice: { description: "d", analyzerId: "a" }, + }, + }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("enableSegment"))); + }); + + it("rejects empty category description", () => { + const r = validate({ + baseAnalyzerId: "prebuilt-document", + config: { + enableSegment: true, + contentCategories: { + invoice: { description: " ", analyzerId: "a" }, + }, + }, + }); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("description"))); + }); +}); + +describe("SchemaValidator — file loading", () => { + it("reports missing file", () => { + const missing = join(tmpdir(), `definitely-not-there-${Date.now()}.json`); + const r = validateFile(missing); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("not found"))); + }); + + it("reports invalid JSON", () => { + const tmp = mkdtempSync(join(tmpdir(), "skill-validator-")); + try { + const p = join(tmp, "broken.json"); + writeFileSync(p, "{ this is not json"); + const r = validateFile(p); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("not valid JSON"))); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } + }); + + it("validates a valid schema on disk", () => { + const tmp = mkdtempSync(join(tmpdir(), "skill-validator-")); + try { + const p = join(tmp, "valid.json"); + writeFileSync( + p, + JSON.stringify({ + baseAnalyzerId: "prebuilt-document", + fieldSchema: { fields: { x: { type: "string" } } }, + }), + ); + const r = validateFile(p); + assert.equal(r.ok, true, `Errors: ${r.errors.join("; ")}`); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } + }); + + it("validateString rejects invalid JSON", () => { + const r = validateString("not json"); + assert.equal(r.ok, false); + assert.ok(r.errors.some((e) => e.includes("not valid JSON"))); + }); +}); + +describe("SchemaValidator — allow-list surface", () => { + it("KNOWN_BASE_ANALYZER_IDS contains only modality prebuilt analyzers", () => { + // Sanity check: the allow-list must NOT include `*Search` variants, + // `prebuilt-invoice`, or `prebuilt-receipt` — these return + // `InvalidBaseAnalyzerId` if used as `baseAnalyzerId` for a custom + // analyzer. Only modality-level prebuilt analyzers are valid. + const expected = new Set([ + "prebuilt-document", + "prebuilt-audio", + "prebuilt-video", + "prebuilt-image", + ]); + assert.equal(KNOWN_BASE_ANALYZER_IDS.size, expected.size); + for (const id of expected) { + assert.ok(KNOWN_BASE_ANALYZER_IDS.has(id), `missing: ${id}`); + } + }); +}); + +describe("SchemaValidator — purity guard", () => { + it("schemaValidator.ts source does not import @azure/* or http modules", () => { + // The validator is intentionally pure-TypeScript so it can be unit-tested + // without spinning up the Azure SDK, and so it can be reused from any + // context (CI, scripts, samples). Drift would creep in if a future + // change accidentally pulls in @azure/* or an HTTP client. + const src = locateSchemaValidatorSource(); + const text = readFileSync(src, "utf-8"); + const forbidden = [ + 'from "@azure/', + "from '@azure/", + 'import "@azure/', + "import '@azure/", + "node:http", + "node:https", + "node:net", + ]; + for (const pattern of forbidden) { + assert.ok( + !text.includes(pattern), + `schemaValidator.ts must not contain \`${pattern}\` — see _shared/README.md`, + ); + } + }); +}); + +function locateSchemaValidatorSource(): string { + // The test runs from the package root (cwd) — the file is next to this one. + return join(import.meta.dirname ?? "src", "schemaValidator.ts"); +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.ts b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.ts new file mode 100644 index 000000000000..2e3d4daa7d46 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/src/schemaValidator.ts @@ -0,0 +1,291 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT License. + +/* + * Pure-TypeScript validator for Content Understanding analyzer schema JSON. + * + * Catches structural mistakes (missing keys, unknown baseAnalyzerId values, + * malformed contentCategories routes) BEFORE any call to the Content + * Understanding service. Failing fast here gives users an actionable error + * message and avoids a wasted service round-trip. + * + * Design rules (see README.md in this directory): + * * No `@azure/*` imports — pure native JavaScript. + * * No network calls. + * * Self-contained — drop-in for any tool or test. + * + * Public surface: + * * validate(value) — validate an already-parsed schema object + * * validateString(json) — convenience wrapper that parses JSON first + * * validateFile(path) — convenience wrapper that reads a file + * * KNOWN_BASE_ANALYZER_IDS — allow-list of baseAnalyzerId values + */ + +import { existsSync, readFileSync } from "node:fs"; + +/** + * Valid `baseAnalyzerId` values for custom analyzers. Only modality-level + * prebuilt analyzers are accepted by the service for `baseAnalyzerId`; `*Search` + * variants and task-specific prebuilt analyzers (`prebuilt-invoice`, + * `prebuilt-receipt`) return `InvalidBaseAnalyzerId` if used here. See + * https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/analyzer-reference#baseanalyzerid + */ +export const KNOWN_BASE_ANALYZER_IDS: ReadonlySet = new Set([ + "prebuilt-document", + "prebuilt-audio", + "prebuilt-video", + "prebuilt-image", +]); + +const ALLOWED_FIELD_TYPES: ReadonlySet = new Set([ + "string", + "number", + "integer", + "boolean", + "date", + "time", + "array", + "object", +]); + +const ALLOWED_FIELD_METHODS: ReadonlySet = new Set(["extract", "generate", "classify"]); + +export interface ValidationResult { + ok: boolean; + errors: string[]; +} + +function isObject(v: unknown): v is Record { + return typeof v === "object" && v !== null && !Array.isArray(v); +} + +/** Validate a parsed analyzer schema object. */ +export function validate(schema: unknown): ValidationResult { + const errors: string[] = []; + + if (!isObject(schema)) { + return { ok: false, errors: ["schema must be a JSON object at the top level"] }; + } + + // baseAnalyzerId + const baseEl = schema["baseAnalyzerId"]; + if (baseEl === undefined || baseEl === null) { + errors.push("missing required key: baseAnalyzerId"); + } else if (typeof baseEl !== "string") { + errors.push("baseAnalyzerId must be a string"); + } else if (!KNOWN_BASE_ANALYZER_IDS.has(baseEl)) { + const known = [...KNOWN_BASE_ANALYZER_IDS] + .sort() + .map((s) => `'${s}'`) + .join(", "); + errors.push(`unknown baseAnalyzerId: '${baseEl}'. Known values: [${known}]`); + } + + // config (optional, but if present must be an object) + const config = schema["config"]; + if (config !== undefined && config !== null) { + if (!isObject(config)) { + errors.push("config, if present, must be an object"); + // Bail out — without a well-typed config we can't tell whether this is + // a single-type or classify-and-route schema, and falling through would + // emit a confusing cascade of "missing fieldSchema" errors rooted in + // the same problem. + return { ok: false, errors }; + } + } + + const isClassifyRoute = isObject(config) && config["contentCategories"] !== undefined; + + if (isClassifyRoute) { + errors.push(...validateClassifyRoute(config as Record)); + if (schema["fieldSchema"] !== undefined) { + errors.push( + "classify-and-route schemas should not declare fieldSchema at " + + "the top level; field extraction belongs in inner analyzers", + ); + } + } else { + errors.push(...validateSingleType(schema)); + } + + return { ok: errors.length === 0, errors }; +} + +/** Validate a raw JSON string. */ +export function validateString(json: string, sourceLabel = ""): ValidationResult { + try { + return validate(JSON.parse(json)); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + return { + ok: false, + errors: [`schema file is not valid JSON (${sourceLabel}): ${msg}`], + }; + } +} + +/** Validate a schema stored in a JSON file. */ +export function validateFile(filePath: string): ValidationResult { + if (!existsSync(filePath)) { + return { ok: false, errors: [`schema file not found: ${filePath}`] }; + } + let text: string; + try { + text = readFileSync(filePath, "utf-8"); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + return { + ok: false, + errors: [`failed to read schema file ${filePath}: ${msg}`], + }; + } + return validateString(text, filePath); +} + +// --------------------------------------------------------------------------- +// Internal helpers +// --------------------------------------------------------------------------- + +function validateSingleType(schema: Record): string[] { + const errors: string[] = []; + const fieldSchema = schema["fieldSchema"]; + if (fieldSchema === undefined || fieldSchema === null) { + errors.push( + "missing required key: fieldSchema " + "(single-type schemas must declare fields to extract)", + ); + return errors; + } + if (!isObject(fieldSchema)) { + errors.push("fieldSchema must be an object"); + return errors; + } + const fields = fieldSchema["fields"]; + if (fields === undefined || fields === null) { + errors.push("fieldSchema.fields is required"); + return errors; + } + if (!isObject(fields)) { + errors.push("fieldSchema.fields must be an object mapping field names to definitions"); + return errors; + } + const keys = Object.keys(fields); + if (keys.length === 0) { + errors.push("fieldSchema.fields must declare at least one field"); + return errors; + } + for (const name of keys) { + errors.push(...validateFieldDefinition(name, fields[name], null)); + } + return errors; +} + +function validateFieldDefinition(name: string, definition: unknown, path: string | null): string[] { + const errors: string[] = []; + const prefix = path ?? `fieldSchema.fields['${name}']`; + + if (!isObject(definition)) { + errors.push(`${prefix} must be an object`); + return errors; + } + + let fieldType: string | null = null; + const typeEl = definition["type"]; + if (typeEl === undefined || typeEl === null) { + errors.push(`${prefix}.type is required`); + } else if (typeof typeEl !== "string") { + errors.push(`${prefix}.type must be a string`); + } else { + fieldType = typeEl; + if (!ALLOWED_FIELD_TYPES.has(fieldType)) { + const allowed = [...ALLOWED_FIELD_TYPES] + .sort() + .map((s) => `'${s}'`) + .join(", "); + errors.push(`${prefix}.type '${fieldType}' is not one of [${allowed}]`); + } + } + + const methodEl = definition["method"]; + if (methodEl !== undefined && methodEl !== null) { + if (typeof methodEl !== "string") { + errors.push(`${prefix}.method must be a string`); + } else if (!ALLOWED_FIELD_METHODS.has(methodEl)) { + const allowed = [...ALLOWED_FIELD_METHODS] + .sort() + .map((s) => `'${s}'`) + .join(", "); + errors.push(`${prefix}.method '${methodEl}' is not one of [${allowed}]`); + } + } + + const descEl = definition["description"]; + if (descEl !== undefined && descEl !== null && typeof descEl !== "string") { + errors.push(`${prefix}.description must be a string`); + } + + if (fieldType === "object") { + const propsEl = definition["properties"]; + if (propsEl !== undefined && propsEl !== null) { + if (!isObject(propsEl)) { + errors.push(`${prefix}.properties must be an object`); + } else { + for (const [childName, childDef] of Object.entries(propsEl)) { + errors.push( + ...validateFieldDefinition(childName, childDef, `${prefix}.properties['${childName}']`), + ); + } + } + } + } else if (fieldType === "array") { + const itemsEl = definition["items"]; + if (itemsEl !== undefined && itemsEl !== null) { + if (!isObject(itemsEl)) { + errors.push(`${prefix}.items must be an object`); + } else { + errors.push(...validateFieldDefinition("items", itemsEl, `${prefix}.items`)); + } + } + } + + return errors; +} + +function validateClassifyRoute(config: Record): string[] { + const errors: string[] = []; + + const enableEl = config["enableSegment"]; + if (enableEl !== true) { + errors.push("classify-and-route schemas must set config.enableSegment = true"); + } + + const categories = config["contentCategories"]; + if (!isObject(categories)) { + errors.push("config.contentCategories must be an object"); + return errors; + } + + const keys = Object.keys(categories); + if (keys.length === 0) { + errors.push("config.contentCategories must declare at least one category"); + return errors; + } + + for (const catName of keys) { + const entry = categories[catName]; + const prefix = `config.contentCategories['${catName}']`; + if (!isObject(entry)) { + errors.push(`${prefix} must be an object`); + continue; + } + const desc = entry["description"]; + if (typeof desc !== "string" || desc.trim() === "") { + errors.push(`${prefix}.description is required and must be a non-empty string`); + } + const analyzerId = entry["analyzerId"]; + if (analyzerId !== undefined && analyzerId !== null && typeof analyzerId !== "string") { + errors.push(`${prefix}.analyzerId, if present, must be a string`); + } + } + + return errors; +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/tsconfig.json b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/tsconfig.json new file mode 100644 index 000000000000..1aef295bb2eb --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared/tsconfig.json @@ -0,0 +1,19 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "resolveJsonModule": true, + "rootDir": "src", + "outDir": "dist", + "declaration": false, + "sourceMap": false, + "lib": ["ES2022"], + "types": ["node"] + }, + "include": ["src/**/*.ts"], + "exclude": ["node_modules", "dist"] +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md new file mode 100644 index 000000000000..7921a32a22ff --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md @@ -0,0 +1,494 @@ +--- +name: cu-sdk-author-analyzer-classify-route +description: Create and test a classify-and-route Azure AI Content Understanding pipeline for packets that contain multiple document types (e.g. invoice + bank statement + loan application in one PDF). Walks per-type schema authoring → outer classifier wiring → batch test → category-aware stdout summary using the typed ContentUnderstandingClient (JS). Use when the user has mixed-document packets. +--- + +# Author a Classify-and-Route Analyzer (mixed document packets) + +Build a classify-and-route pipeline: one **outer classifier analyzer** that +segments and labels a multi-document packet, plus one **inner extractor +analyzer per document type**. The packet flows through the outer analyzer +once; each segment is automatically routed to the matching inner analyzer +for field extraction. + +**This is an iterative, human-in-the-loop workflow.** You will typically run +the schema → test → review cycle multiple times to refine both the outer +classifier descriptions and each inner schema's field descriptions before +you're happy with both classification accuracy and extraction quality. + +> **[USE INSTEAD]:** If every page in the user's documents is the **same +> type** (only invoices, only contracts, etc.), use +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) instead. +> Classify-and-route is for **mixed** packets. + +> **[ASK USER] Modality check (first thing to confirm):** +> +> "Are you working with **document** files — PDFs or images of pages? This +> skill currently supports document modalities only. Audio, video, and +> image classifiers are planned for a future update." +> +> - If the user says **document** → continue with this skill. +> - If the user says **audio**, **video**, or **image** → stop this skill. +> Audio/video classify-and-route is on the roadmap; for now point them at +> the [REST tutorial](https://learn.microsoft.com/azure/ai-services/content-understanding/tutorial/create-custom-analyzer). + +> **[COPILOT INTERACTION MODEL]:** At each step marked with **[ASK USER]**, +> pause execution and prompt the user before proceeding. + +## Prerequisites + +Required: Node.js 22+ and npm (the skill tool is a standalone TypeScript +CLI under `.github/skills/_shared/` that depends on the published +`@azure/ai-content-understanding` package from npm, which declares +`engines.node: ">=22.0.0"`), `.env` or environment +variables with `CONTENTUNDERSTANDING_ENDPOINT` (plus `CONTENTUNDERSTANDING_KEY` +or `az login`), and the model defaults configured for this resource (see +[`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts)). + +> **[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.** +> +> ```bash +> node --version >/dev/null 2>&1 && echo 'node: ok' || echo 'node: MISSING' +> npm --version >/dev/null 2>&1 && echo 'npm: ok' || echo 'npm: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty' +> az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in' +> ``` +> +> | Failure | Route to | +> | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | +> | `node: MISSING` | install Node.js 22+ from https://nodejs.org | +> | `npm: MISSING` | install npm (bundled with Node.js) | +> | endpoint `MISSING` | create or edit `.env` with `CONTENTUNDERSTANDING_ENDPOINT=https://.services.ai.azure.com/`, then resume | +> | endpoint `ok`, key `empty`, `az: not logged in` | run `az login` **or** add `CONTENTUNDERSTANDING_KEY` to `.env`, then resume | +> | All env checks pass but service calls fail with model errors | run [`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts) once for this resource, then resume | +> | All ok | ✅ Proceed to the Packet check below. | +> +> Never ask the user to paste an endpoint or API key into chat — they edit `.env` directly or run `az login`. + +> **[ASK USER] Packet check:** +> +> 1. "Does each document in your packet contain more than one type of form (e.g. an invoice page followed by a bank statement page)?" — if no, route to `cu-sdk-author-analyzer`. +> 2. "What types of documents appear in your packets?" — capture as the list of inner analyzers. + +## Architecture + +``` + ┌──────────────────────────────┐ + mixed packet ───► │ outer (classifier) analyzer │ + │ baseAnalyzerId: prebuilt-… │ + │ config.enableSegment: true │ + │ config.contentCategories: │ + │ invoice ────────►│──┐ + │ bank_statement ────────►│──┼──► per-segment fields + │ loan_application ────────►│──┘ + └──────────────────────────────┘ + │ + inner analyzers (1 per type) │ + ─────────────────────────── ▼ + invoice extractor ◄──── routes here for invoice pages + bank statement ext. ◄──── routes here for bank pages + loan app extractor ◄──── routes here for loan pages +``` + +Key rules (also captured in +[`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) § +"classify-and-route"): + +1. **Category descriptions reference text anchors**, not visual cues + (matches the two-stage pipeline rule for fields). +2. **`config.enableSegment` must be `true`** so the classifier can carve up + the packet before routing. +3. **Inner analyzers must exist before** the outer classifier is created. + The provided tool handles ordering automatically. +4. **Category fill rate is per-category**, not packet-wide. The tool's + stdout summary uses the right denominator. + +## Package directory + +``` +sdk/contentunderstanding/ai-content-understanding +``` + +## Scripts and templates + +``` +.github/skills/ +├── _shared/ # The skill tool (single npm package, three subcommands) +└── cu-sdk-author-analyzer-classify-route/ + ├── SKILL.md (this file) + └── templates/ + └── classifier_template.json # Starter outer-classifier schema for Step 3 +``` + +The skill tool builds against the local `@azure/ai-content-understanding (npm package)`, +so before first use (the `cp` copies the tracked template into place — +`package.json` itself is `.gitignore`d so the tool directory is not picked up +as a pnpm workspace member during CI): + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && \ + [ -f package.json ] || cp package.json.template package.json && \ + npm install) +``` + +## Workflow + +### Step 1 — Identify the document types + +Run layout extraction (same as the single-type skill) on a representative +packet to see the section headings: + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && node_modules/.bin/tsx src/cli.ts \ + extract-layout --input --output .local_only/layout/) +``` + +> **[COPILOT]** Read `.local_only/layout/.layout.md` **yourself** and +> identify section headings, page breaks, and content shifts that suggest +> different document types. Then present your analysis to the user for +> confirmation — do **not** ask the user to read the layout cold. +> +> Example presentation: +> +> > "Based on the layout in `.layout.md` I see these document types: +> > +> > - **Pages 1–2** — appears to be an _invoice_ (top heading 'Invoice', +> > contains 'Invoice #' label and a line-item table) +> > - **Pages 3–4** — appears to be a _bank statement_ (top heading +> > 'Account Statement', contains account number and a transaction table) +> > - **Page 5** — appears to be a _loan application_ (top heading 'Loan +> > Application', applicant fields) +> > +> > Does this look right? Anything to add, remove, or rename before I +> > draft schemas?" +> +> Only fall back to a blank `[ASK USER]` ("What types do you see?") if the +> layout is too ambiguous to suggest types confidently. + +### Step 2 — Draft one inner schema per type + +Treat each type as a single-doc-type analyzer (pick `baseAnalyzerId` from +the table in +[`cu-sdk-common-knowledge` § Choosing `baseAnalyzerId`](../cu-sdk-common-knowledge/SKILL.md#choosing-baseanalyzerid), +then add `fieldSchema.fields`). Field descriptions follow the +[two-stage pipeline rule](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline) +— reference text and structure, never visual appearance. See +[`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) Step 2 for +the full field-schema walkthrough. + +> **[COPILOT] Read best practices before drafting fields.** Before writing +> any field description (in any inner schema or the outer classifier's +> category descriptions), fetch the official CU best-practices page and +> apply its guidance: +> +> 🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices +> +> The same key principles apply here as in +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md#step-2--draft-a-json-field-schema): +> be specific and concrete, use text anchors (not visual cues), include +> alternative labels and format examples, prefer `extract` for verbatim +> values, and keep the field count focused. + +> **Reference**: +> [`createClassifier.ts`](../../../samples-dev/createClassifier.ts) +> ships a complete worked example using +> `samples/sample_files/mixed_financial_docs.pdf` with three categories — +> Invoice, Bank_Statement, Loan_Application. + +### Step 3 — Draft the outer classifier schema + +The outer schema has **no** `fieldSchema`. Its job is classification + routing. +Start from the template: + +```bash +mkdir -p .local_only/schemas +cp sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json \ + .local_only/schemas/_classifier_v1.json +``` + +Example after editing (descriptions intentionally generic so they survive +minor wording variants — see the **Category description rule** below the +example for guidance on writing your own): + +```json +{ + "baseAnalyzerId": "prebuilt-document", + "description": "Classify mixed financial packets and route to per-type extractors.", + "config": { + "enableSegment": true, + "omitContent": true, + "contentCategories": { + "invoice": { + "description": "A commercial invoice. Look for an invoice / bill heading or label, an invoice number, line-item table with quantities and prices, and a total amount.", + "analyzerId": "invoice" + }, + "bank_statement": { + "description": "A bank or account statement. Look for an account-statement heading, an account number, a statement period or date range, and a transaction table with running balance.", + "analyzerId": "bank_statement" + }, + "loan_application": { + "description": "A loan or credit application form. Look for an application heading, applicant or borrower fields, requested loan amount, and applicant signature.", + "analyzerId": "loan_application" + } + } + }, + "models": { + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + } +} +``` + +The `analyzerId` value in each category is **an alias** that the tool +resolves at runtime, matching the `--inner-schema alias=path` flags you +pass. Two exceptions skip alias resolution: + +- Values starting with `prebuilt-` (e.g. `prebuilt-invoice`) are used as-is + — no inner schema needed. Useful for routing a category at a service + prebuilt extractor. +- Categories without an `analyzerId` at all are classification-only — the + segment is labelled but no fields are extracted. + +> **Why `omitContent: true`?** When omitted, the service also returns the +> raw, un-segmented document content as an extra entry in `contents`. That +> entry has no category, no fields, and shows up in the summary as a +> confusing `(uncategorized)` row. Setting `omitContent: true` removes it. + +> **Category description rule — the example values above are demo-specific.** +> +> The category descriptions in the JSON above are tied to the demo packet +> `samples/sample_files/mixed_financial_docs.pdf` (which uses the literal +> headings `Invoice`, `Bank Statement`, `Loan Application`). **Do not copy +> them verbatim.** When authoring for the user's own packet, write +> descriptions based on what you observed in **the user's** layout output +> from Step 1 — use the actual headings, labels, and structural markers +> from their documents. +> +> Keep descriptions: +> +> - **Generic over surface form** — describe the _kind_ of content +> ("contains a transaction table with running balance") rather than +> hardcoding one specific header string, so minor wording variants still +> classify correctly. +> - **Concrete enough to be discriminative** — include at least one anchor +> that distinguishes this category from the others in the packet. +> - **Text-anchored, not visual** — reference headings, labels, and +> neighbouring text, never colour / font / position-without-text. Same +> reason as the field-description rule in +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline). + +### Step 4 — Validate, create, and batch-test + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && node_modules/.bin/tsx src/cli.ts \ + create-and-test-router --outer-schema .local_only/schemas/classifier.json --inner-schema invoice=.local_only/schemas/invoice.json --inner-schema bank_statement=.local_only/schemas/bank_statement.json --inner-schema loan_application=.local_only/schemas/loan_application.json --input samples/sample_files/mixed_financial_docs.pdf --output .local_only/test_results/v1) +``` + +> **Shortcut — `--schema-dir`:** name each inner schema file +> `.json` (bare) or `_.json`, where `` is the +> `analyzerId` value in the outer schema. Then replace every +> `--inner-schema alias=path` with a single `--schema-dir .local_only/schemas/`. +> The tool discovers a file per alias by matching stem `==` `` or +> stem starts-with `_`, and picks the **highest-numbered version** — +> `_v.json` and `_.json` sort by `N` as an integer, so +> `invoice_v10.json` beats `invoice_v9.json` (previous alphabetical sort +> silently picked v9 because `'1' < '9'` char-by-char). Non-numeric suffixes +> (e.g. `invoice_draft.json`) fall back to lexicographic order and beat the +> bare `invoice.json` baseline but lose to any versioned file. + +> **Iteration helper — `--reuse`:** add `--reuse` to name analyzers by a +> sha1 of their schema (`_`) and skip creation when an +> analyzer with that ID already exists. Re-running with the same schemas +> is a no-op on the create side, so you don't pile up stale analyzers while +> iterating. Edit a schema → hash changes → new analyzer is created. + +The tool: + +1. Validates every schema (exits with code **2** if any fails — no service + call made). +2. Errors out if the outer schema references an alias that has no matching + `--inner-schema`, or if you supply an `--inner-schema` that no category + uses. +3. Creates inner analyzers first, then patches and creates the outer + classifier. +4. Analyzes every input file, writing one JSON per file under `--output`. +5. Prints a **category-aware** stdout summary (per-category fill rate + uses each category's segment count, not the packet-wide total). + +### Step 5 — Read the category-aware summary + +Example output: + +``` +======================================================================== +[SUMMARY] (category-aware) + +category: bank_statement (1 segments) +-------------------------------------- + field fill rate avg conf + AccountNumber 100.0% 0.918 + StatementPeriod 100.0% 0.882 + +category: invoice (1 segments) +------------------------------- + field fill rate avg conf + InvoiceNumber 100.0% 0.962 + TotalAmount 100.0% 0.531 + +category: loan_application (1 segments) +---------------------------------------- + field fill rate avg conf + ApplicantName 100.0% 0.875 + LoanAmount 100.0% 0.799 + +lowest-confidence fields across all categories: + 0.531 [invoice] TotalAmount (mixed_financial_docs) + 0.799 [loan_application] LoanAmount (mixed_financial_docs) + 0.875 [loan_application] ApplicantName (mixed_financial_docs) +======================================================================== +``` + +For each input document the tool writes two files into `--output`: + +- `.json` — full `AnalysisResult` with all per-segment fields and grounding. +- `.llm.md` — the same result rendered via the SDK's + [`toLlmInput`](../../../samples-dev/toLlmInput.ts) helper. + For classify-and-route, the helper expands each classified segment into + its own block with the **category in the YAML front matter**, separated by + `*****` dividers — drop it into an LLM prompt or skim it in VS Code. + +> **Template comment keys**: any key whose name starts with `_` +> (e.g. `_comment`, `_optional_returnDetails`) is stripped from both the +> outer and inner schemas before the request body is sent. Use them freely +> as inline documentation. + +> **`models.completion` for inner schemas**: each inner schema (which has a +> `fieldSchema`) needs `models.completion` set unless the resource has +> defaults configured via +> [`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts). +> `create-and-test-router` prints a `[WARN]` per inner schema that is +> missing it, before any service call. + +### Step 6 — Agent review and iterate + +> **[COPILOT]** After the category-aware summary prints, do the following +> **automatically** before asking the user anything: +> +> 1. For each entry in `result.contents[]` in `.json`, verify the +> assigned `category` matches what the page actually is. Use +> `.local_only/layout/.layout.md` as ground truth. +> - **Misclassified segment** → the **outer classifier's** +> `contentCategories..description` needs strengthening, not the +> inner schema. Add a discriminating anchor from the layout to the +> category description and recreate the outer classifier. +> 2. For each correctly-classified segment, compare extracted field values +> against the layout the same way as the single-type skill (see +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md#step-5--agent-review-and-iterate)). +> Flag fields where: +> - The extracted value looks wrong or is `null` when a value should be present +> - Confidence is below 0.7 +> - The grounding location is unexpected +> 3. Present a per-category diff to the user: +> - "In the `invoice` segment, field `TotalAmount` extracted `'1234'` — +> I see `'$1,234.56'` near the 'Total' label. Should I tighten the +> description?" +> - "Page 5 was classified as `loan_application` but the heading is +> actually 'Borrower Agreement' — should I add that wording to the +> `loan_application` category description?" +> 4. Record user-confirmed corrections in per-category ground-truth files +> (`.local_only/ground_truth_.json`) so the agent remembers +> correct answers across iterations: +> ```json +> [ +> { +> "doc": "mixed_packet_1.pdf", +> "segment": 0, +> "field": "TotalAmount", +> "correct_value": "1234.56" +> } +> ] +> ``` +> 5. Update the affected schema(s) — outer classifier for classification +> fixes, inner schemas for field fixes — as a new version +> (`*_v2.json`). +> 6. Re-run Step 4 with the new schemas. `--reuse` rebuilds only the +> analyzers whose schema hash changed. +> +> Repeat until every category has: +> +> - All segments classified correctly +> - Key fields **fill rate ≥ 80%** and **avg confidence ≥ 0.85** (computed +> per-category, as the summary already does) +> +> Stop and report to the user when any of: +> +> - All targets met (success) — then proceed to **Step 7** to hand off. +> - Three consecutive iterations show no improvement on a given category +> (escalate — may need a different `baseAnalyzerId`, schema redesign, or +> a different category split). +> - The user signals they're done. + +### Step 7 — Hand off the finished pipeline + +This step is required when Step 6 succeeds. **Do not skip it.** Without a +clean handoff the user has a working classify-and-route pipeline in their +resource but no idea what the outer classifier ID is, which inner +analyzers it routes to, where the final schemas live, or how to call it +from their own code. + +> **[COPILOT]** When Step 6 reaches success, report the following to the +> user in one message before stopping: +> +> 1. **All analyzer IDs built** — list both the **outer classifier** and +> every **inner extractor**, with the actual IDs printed by +> `create-and-test-router` (e.g. +> `classifier_v2_a1b2c3d4` → routes to +> `classifier_v2_a1b2c3d4_inner_invoice_b2c3d4e5`, +> `..._inner_bank_statement_c3d4e5f6`, +> `..._inner_loan_application_d4e5f6a7`). The user only needs to call +> the **outer** analyzer in their own code; it routes to the inner +> ones automatically. +> 2. **All schema files** — list the path to every schema JSON in the +> final iteration: the outer classifier and each inner schema (e.g. +> `.local_only/schemas/classifier_v2.json`, +> `.local_only/schemas/invoice_v3.json`, etc.). Recommend the user save +> them somewhere outside `.local_only/` for future reference; they can +> also inspect any existing analyzer's schema directly via the SDK (see +> [`getAnalyzer.ts`](../../../samples-dev/getAnalyzer.ts)). +> 3. **Next-step samples** — point the user to: +> - [`createClassifier.ts`](../../../samples-dev/createClassifier.ts) +> — how to (re)build the full classify-and-route pipeline from schema +> JSON in their own code (handles inner-first creation ordering and +> `contentCategories.analyzerId` wiring the same way our tool did). +> - [`analyzeBinary.ts`](../../../samples-dev/analyzeBinary.ts) +> and +> [`analyzeUrl.ts`](../../../samples-dev/analyzeUrl.ts) +> — how to call the analyzer on real input from their own code. Use +> the **outer classifier's** analyzer ID as `analyzerId`. +> +> Remind the user that the analyzers you just built are **already +> deployed** to their resource and ready to use directly via their IDs +> — they only need to re-create them from the schemas if they want to +> bootstrap into another resource. + +### Step 8 — Clean up (optional) + +By default the tool leaves both the outer classifier **and** all inner +analyzers in your resource so you can re-use them. Pass `--ephemeral` to +delete all of them at the end of the run. + +## Exit codes + +| Code | Meaning | +| ---- | ------------------------------------------------------------------------------------------ | +| `0` | Every document analyzed successfully. | +| `1` | At least one service-side failure. | +| `2` | Local user error — schema validation, missing inner alias, bad path. No service call made. | + +## Related skills + +- [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) — single doc type. +- [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) — service concepts, two-stage pipeline, `baseAnalyzerId` table, classify-and-route rules. +- [`cu-sdk-sample-run`](../cu-sdk-sample-run/SKILL.md) — run individual SDK samples (e.g. `createClassifier.ts`) for deeper reference. +- [`cu-sdk-setup`](../cu-sdk-setup/SKILL.md) — install the SDK, configure env. diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json new file mode 100644 index 000000000000..de955f8a4b94 --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json @@ -0,0 +1,32 @@ +{ + "_comment": "Starter template for a classify-and-route outer classifier. Copy to .local_only/schemas/_classifier_v1.json and edit. See SKILL.md Step 3 for the rules. Each contentCategories entry's analyzerId is an alias resolved by --inner-schema alias=path at runtime; values starting with 'prebuilt-' (e.g. prebuilt-invoice, prebuilt-receipt) are used as-is and don't need an inner schema. Keys beginning with '_' (_comment, _optional_*) are stripped from the request body before submission.", + "baseAnalyzerId": "prebuilt-document", + "description": "REPLACE: one-line description of what this packet contains.", + "config": { + "_comment": "enableSegment=true is required for multi-document packets. omitContent=true keeps the outer response small \u2014 inner analyzers see the full content independently.", + "enableSegment": true, + "omitContent": true, + "_optional_returnDetails": true, + "contentCategories": { + "REPLACE_category_a": { + "_comment": "Category routed to a custom inner analyzer. The analyzerId value must match an --inner-schema alias=path passed on the command line.", + "description": "REPLACE: text-anchored description \u2014 reference labels, headings, table column names. Never visual cues.", + "analyzerId": "REPLACE_category_a" + }, + "REPLACE_category_b": { + "_comment": "Category routed to a service prebuilt analyzer. Any analyzerId starting with 'prebuilt-' (e.g. prebuilt-invoice, prebuilt-receipt) is used as-is and does NOT need a corresponding --inner-schema entry.", + "description": "REPLACE: text-anchored description for a document type covered by a prebuilt analyzer.", + "analyzerId": "prebuilt-invoice" + }, + "other": { + "_comment": "Classify-only category with no analyzerId \u2014 segments matching this category are labelled but no fields are extracted. Recommended catch-all; without it every page is forced into one of the defined categories.", + "description": "REPLACE: catch-all category for pages that match none of the above." + } + } + }, + "models": { + "_comment": "Required when contentCategories is set, unless the resource has defaults via samples-dev/updateDefaults.ts.", + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + } +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/SKILL.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/SKILL.md new file mode 100644 index 000000000000..86ba6141bddf --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/SKILL.md @@ -0,0 +1,406 @@ +--- +name: cu-sdk-author-analyzer +description: Iteratively author and test a custom Azure AI Content Understanding analyzer for a folder of **document** files (PDFs, page images) of a single type. Walks layout extraction → schema authoring → validation → batch test → agent review → refine cycle using the typed ContentUnderstandingClient (JS). Document modality only — audio, video, and image analyzers are planned for a later update. Use when the user wants to author a custom analyzer for invoices, contracts, forms, or any other single-type document set. +--- + +# Author a Custom Analyzer (single document type) + +Author a custom Content Understanding analyzer for one document type +end-to-end: extract layout, draft a field schema, validate locally, create the +analyzer, batch-test it on sample files, and read a quality summary. + +**This is an iterative, human-in-the-loop workflow.** You will typically run +the schema → test → review cycle multiple times to refine field descriptions +before you're happy with the extraction quality. + +The workflow uses the typed `ContentUnderstandingClient` shipped in this +package — the same client `createAnalyzer.ts` and +`analyzeBinary.ts` use. + +> **[COPILOT INTERACTION MODEL]:** This skill is designed to be interactive. +> At each step marked with **[ASK USER]**, pause execution and prompt the user +> for input or confirmation before proceeding. Do NOT silently skip these +> prompts. Use the `ask_questions` tool when available. + +> **[USE INSTEAD]:** If the user's packet contains **multiple different +> document types** (for example, an invoice, a bank statement, and a loan +> application in one PDF), route them to the +> [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) +> skill instead. This skill assumes one document type per analyzer. + +> **[ASK USER] Modality check (first thing to confirm):** +> +> "Are you working with **document** files — PDFs or images of pages? This +> skill currently supports document modalities only. Audio, video, and +> image analyzers are planned for a future update." +> +> - If the user says **document** → continue with this skill. +> - If the user says **audio**, **video**, or **image** → stop this skill and +> point them at the typed-model samples +> ([`createAnalyzer.ts`](../../../samples-dev/createAnalyzer.ts) +> with `prebuilt-audio` / `prebuilt-video` / `prebuilt-image`) or the +> [REST tutorial](https://learn.microsoft.com/azure/ai-services/content-understanding/tutorial/create-custom-analyzer). + +## Prerequisites + +Required: Node.js 22+ and npm (the skill tool is a standalone TypeScript +CLI under `.github/skills/_shared/` that depends on the published +`@azure/ai-content-understanding` package from npm, which declares +`engines.node: ">=22.0.0"`), `.env` or environment +variables with `CONTENTUNDERSTANDING_ENDPOINT` (plus `CONTENTUNDERSTANDING_KEY` +or `az login`), and the model defaults configured for this resource (see +[`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts)). + +> **[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.** +> +> ```bash +> node --version >/dev/null 2>&1 && echo 'node: ok' || echo 'node: MISSING' +> npm --version >/dev/null 2>&1 && echo 'npm: ok' || echo 'npm: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty' +> az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in' +> ``` +> +> | Failure | Route to | +> |---|---| +> | `node: MISSING` | install Node.js 22+ from https://nodejs.org | +> | `npm: MISSING` | install npm (bundled with Node.js) | +> | endpoint `MISSING` | create or edit `.env` at the repo root with `CONTENTUNDERSTANDING_ENDPOINT=https://.services.ai.azure.com/`, then resume | +> | endpoint `ok`, key `empty`, `az: not logged in` | run `az login` **or** add `CONTENTUNDERSTANDING_KEY` to `.env`, then resume | +> | All env checks pass but service calls fail with model errors | run [`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts) once for this resource, then resume | +> | All ok | ✅ Proceed to Step 1. | +> +> Never ask the user to paste an endpoint or API key into chat — they edit `.env` directly or run `az login`. + +> **[ASK USER]** "How many representative documents do you have, and where are they?" — fewer than 3 is fine, but more is better for testing coverage. + +## Package directory + +``` +sdk/contentunderstanding/ai-content-understanding +``` + +## Scripts and templates + +``` +.github/skills/ +├── _shared/ +│ ├── package.json.template # Tracked manifest; copied to package.json on install +│ ├── tsconfig.json +│ └── src/ +│ ├── cli.ts # Dispatcher (extract-layout | create-and-test | create-and-test-router) +│ ├── schemaValidator.ts # Pure-TS schema validator (no service calls) +│ ├── clientHelpers.ts # Client builder + raw-response helpers +│ ├── extractLayoutCommand.ts # Stage 1 +│ ├── createAndTestCommand.ts # Stage 2 (single-type) +│ └── createAndTestRouterCommand.ts # Stage 2 (classify-and-route) +└── cu-sdk-author-analyzer/ + ├── SKILL.md (this file) + └── templates/ + └── schema_template.json # Starter schema for Step 2 +``` + +The skill tool builds against the local `@azure/ai-content-understanding (npm package)`, +so before first use (the `cp` copies the tracked template into place — +`package.json` itself is `.gitignore`d so the tool directory is not picked up +as a pnpm workspace member during CI): + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && \ + [ -f package.json ] || cp package.json.template package.json && \ + npm install) +``` + +## Workflow + +### Step 1 — Extract layout for representative documents + +The model behind Content Understanding sees the **text and structure** the +service extracts from your file, not the original pixels. Reviewing the +layout output is the fastest way to know what labels and headings you can +anchor your field descriptions to. + +> **[ASK USER]** "Point me at one of your sample documents (or a folder of +> them). I'll run layout extraction so we can see what the model will be +> looking at." + +Run: + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && node_modules/.bin/tsx src/cli.ts \ + extract-layout --input --output .local_only/layout/) +``` + +This produces one `.layout.md` and one `.layout.json` per input. +Open the `.layout.md` file in VS Code and look for the **text anchors** you +want to extract from — labels (`"Invoice #:"`), section headings +(`"Bill To"`), table headers, etc. + +> **Reference**: this is the same call pattern as +> [`analyzeBinary.ts`](../../../samples-dev/analyzeBinary.ts) +> using `prebuilt-documentSearch`. + +### Step 2 — Draft a JSON field schema + +Start from the template instead of writing from scratch: + +```bash +mkdir -p .local_only/schemas +cp sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json \ + .local_only/schemas/_v1.json +``` + +Then edit `.local_only/schemas/_v1.json`: set `baseAnalyzerId`, replace every +`REPLACE:` placeholder, and add/remove fields. The schema is a JSON object +with two required top-level keys: + +- `baseAnalyzerId` — which prebuilt analyzer your custom analyzer extends. Use the table below. +- `fieldSchema.fields` — the named fields you want to extract. + +> **[COPILOT] Read best practices before drafting fields.** Before writing +> any field description, fetch the official CU best-practices page and apply +> its guidance: +> +> 🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices +> +> Key principles that affect schema quality: +> +> - Be specific and concrete in field descriptions — vague descriptions +> produce vague extractions. +> - Use **text anchors** (labels, headings, neighbouring fields) — never +> visual cues like colour, font, or position-without-text-context. This +> matches the two-stage pipeline rule in +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline). +> - Include **format examples** and **alternative label names** when a value +> can appear in multiple wordings or formats. +> - Prefer `"method": "extract"` when the value appears verbatim; use +> `"generate"` only when the model needs to synthesise (summary, +> classification label) and `"classify"` for fixed enumerations. +> - Keep the field count focused on what the user actually needs. Extra +> fields cost tokens and can dilute extraction quality on the fields that +> matter. + +The template demonstrates **all three extraction methods** (`extract`, +`generate`, `classify`) plus the **nested object** and **array of objects** +shapes. Delete the example fields you don't need. + +> **Template comment keys**: any key whose name starts with `_` +> (e.g. `_comment`, `_optional_enableOcr`) is stripped from the request body +> by `create-and-test` before it hits the service. Use them freely as +> inline documentation. + +> **`models.completion` is effectively required**: whenever `fieldSchema` is +> present, the service needs a completion model. Leave the `models` block in +> the template populated unless you've run +> [`updateDefaults.ts`](../../../samples-dev/updateDefaults.ts) +> to set resource defaults. Omitting it fails with `InvalidRequest: +> 'models.completion' is not set` *after* a misleadingly successful +> `[CREATE]` log line. `create-and-test` prints a `[WARN]` if the schema +> is missing it. + +#### Choosing `baseAnalyzerId` + +See the prebuilt-analyzer table in +[`cu-sdk-common-knowledge` § Choosing `baseAnalyzerId`](../cu-sdk-common-knowledge/SKILL.md#choosing-baseanalyzerid). +The local validator (Step 3) rejects any value not on that list. + +#### Example single-type schema + +```json +{ + "baseAnalyzerId": "prebuilt-document", + "description": "Extract invoice header and totals.", + "config": { + "estimateFieldSourceAndConfidence": true, + "returnDetails": true + }, + "models": { + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + }, + "fieldSchema": { + "name": "invoice_v1", + "description": "Invoice header fields.", + "fields": { + "invoiceNumber": { + "type": "string", + "method": "extract", + "description": "Invoice number printed near the 'Invoice #' label at the top of the page.", + "estimateSourceAndConfidence": true + }, + "totalAmount": { + "type": "number", + "method": "extract", + "description": "Grand total at the bottom of the document; typically labelled 'Total' or 'Amount Due'. Excludes any 'Subtotal' value.", + "estimateSourceAndConfidence": true + } + } + } +} +``` + +> **Reference**: see +> [`createAnalyzer.ts`](../../../samples-dev/createAnalyzer.ts) +> for the typed-model equivalent. The `create-and-test` tool sends the JSON +> directly so the schema may include any properties supported by the +> service, even if the typed model doesn't expose them. + +> **Field-description rule (two-stage pipeline):** descriptions must reference +> **text content and structure** (labels, headings, neighbouring fields), not +> visual appearance (colour, font, size). See +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) § +> "two-stage pipeline". + +### Step 3 — Validate the schema locally + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && node_modules/.bin/tsx src/cli.ts \ + create-and-test --schema .local_only/schemas/invoice_v1.json --input samples/sample_files/sample_invoice.pdf --output .local_only/test_results/v1) +``` + +The script runs the local validator first. If anything is wrong (unknown +`baseAnalyzerId`, missing `fieldSchema`, malformed entries) it exits with +code **2** *before* any service call. + +### Step 4 — Read the stdout summary + +After the script finishes you get something like: + +``` +======================================================================== +[SUMMARY] + +category: (single) (3 documents) +--------------------------------- + field fill rate avg conf + invoiceNumber 100.0% 0.962 + totalAmount 66.7% 0.481 + +lowest-confidence fields: + 0.461 totalAmount (mixed_financial_docs) + 0.732 invoiceNumber (mixed_financial_docs) +======================================================================== +``` + +For each input document the script writes two files into `--output`: + +- `.json` — full per-document `AnalysisResult` (fields, grounding, confidences). +- `.llm.md` — same result rendered via the SDK's + [`toLlmInput`](../../../samples-dev/toLlmInput.ts) helper: + YAML front matter (category, page range, fields) plus the document text. + Drop this straight into an LLM prompt, or skim it in VS Code for a fast + human review. + +### Step 5 — Agent review and iterate + +> **[COPILOT]** After the summary prints, do the following **automatically** +> before asking the user anything: +> +> 1. Open each `.llm.md` (and the underlying `.json` for grounding) +> in `.local_only/test_results/` and compare extracted field values +> against the source content. Use `.local_only/layout/.layout.md` +> as ground truth — it shows the text the model actually saw. +> 2. Flag any field where: +> - The extracted value looks wrong or is `null` when a value should be present +> - Confidence is below 0.7 +> - The grounding location is unexpected +> 3. Present findings to the user with concrete diffs: +> - "Field `invoiceNumber` extracted `'INV-001'` — does this look correct?" +> - "Field `totalAmount` was empty, but I see `'$1,234.56'` near the +> 'Total' label in the layout — should I tighten the description?" +> 4. For each correction the user confirms, append to +> `.local_only/ground_truth_.json`: +> ```json +> [ +> { "doc": "invoice_001.pdf", "field": "totalAmount", "correct_value": "1234.56" }, +> { "doc": "invoice_002.pdf", "field": "invoiceDate", "correct_value": "2026-01-15" } +> ] +> ``` +> This file is the agent's memory of correct answers across iterations. +> 5. Use the corrections to refine the field descriptions in a new schema +> version (`.local_only/schemas/_v2.json`). +> 6. Re-run Step 3–4 with the new schema. The `--reuse` flag on +> `create-and-test` names analyzers by schema hash, so unchanged +> schemas are a no-op on the create side and a changed schema gets a +> fresh analyzer automatically. +> +> Repeat until all key fields reach **fill rate ≥ 80%** and +> **avg confidence ≥ 0.85**, or the user is satisfied. +> +> Stop and report to the user when any of: +> - Targets are met (success) — then proceed to **Step 6** to hand off. +> - Three consecutive iterations show no improvement (need a different +> approach — different `baseAnalyzerId`, different `method`, or schema +> redesign — escalate to the user). +> - The user signals they're done. + +### Step 6 — Hand off the finished analyzer + +This step is required when Step 5 succeeds. **Do not skip it.** Without a +clean handoff the user has a working analyzer in their resource but no +idea what its ID is, where the final schema lives, or how to call it from +their own code. + +> **[COPILOT]** When Step 5 reaches success, report the following to the +> user in one message before stopping: +> +> 1. **Final analyzer ID** — the actual ID printed by `create-and-test` +> on its last successful run (e.g. `invoice_v3_a1b2c3d4` when `--reuse` +> is used). The user will need this to call the analyzer from their own +> code. +> 2. **Final schema file** — the path to the last iteration of the schema +> JSON (e.g. `.local_only/schemas/invoice_v3.json`). Recommend the user +> save it somewhere outside `.local_only/` for future reference; they +> can also inspect any existing analyzer's schema directly via the SDK +> (see +> [`getAnalyzer.ts`](../../../samples-dev/getAnalyzer.ts)). +> 3. **Next-step samples** — point the user to: +> - [`createAnalyzer.ts`](../../../samples-dev/createAnalyzer.ts) +> — how to (re)create a custom analyzer from a schema JSON in their own code. +> - [`analyzeBinary.ts`](../../../samples-dev/analyzeBinary.ts) +> and +> [`analyzeUrl.ts`](../../../samples-dev/analyzeUrl.ts) +> — how to call the analyzer on real input from their own code. +> +> Remind the user that the analyzer you just built is **already deployed** +> to their resource and ready to use directly via its ID — they only +> need to re-create it from the schema if they want to bootstrap it in +> another resource. + +### Step 7 — Clean up (optional) + +By default the analyzer is kept in your resource so you can re-use it. Pass +`--ephemeral` to delete it at the end of a run: + +```bash +(cd sdk/contentunderstanding/ai-content-understanding/.github/skills/_shared && node_modules/.bin/tsx src/cli.ts \ + create-and-test --schema .local_only/schemas/invoice_v1.json --input samples/sample_files/sample_invoice.pdf --output .local_only/test_results/v1 --ephemeral) +``` + +> **Iteration helper — `--reuse`:** add `--reuse` to name the analyzer by a +> sha1 of its schema (`_`) and skip creation when an +> analyzer with that ID already exists. Re-running with the same schema is +> a no-op on the create side, so you don't pile up stale analyzers while +> iterating. Edit the schema → hash changes → new analyzer is created. + +For explicit lifecycle management see +[`getAnalyzer.ts`](../../../samples-dev/getAnalyzer.ts), +[`listAnalyzers.ts`](../../../samples-dev/listAnalyzers.ts), +[`updateAnalyzer.ts`](../../../samples-dev/updateAnalyzer.ts), +and +[`deleteAnalyzer.ts`](../../../samples-dev/deleteAnalyzer.ts). + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | All documents analyzed successfully. | +| `1` | At least one service-side failure (network, throttling, invalid response). | +| `2` | Local user error — schema validator failure, missing flag, bad input path. No service call was made. | + +## Related skills + +- [`cu-sdk-setup`](../cu-sdk-setup/SKILL.md) — install the SDK, configure env. +- [`cu-sdk-sample-run`](../cu-sdk-sample-run/SKILL.md) — run one reference sample. +- [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) — service concepts and field-description rules. +- [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) — multi-doc-type packets. diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json new file mode 100644 index 000000000000..688f541944aa --- /dev/null +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json @@ -0,0 +1,81 @@ +{ + "_comment": "Starter template for a single-doc-type custom analyzer. Copy to .local_only/schemas/_v1.json, replace placeholders, then run the create-and-test skill script (invokes the cu-skill tool under .github/skills/_shared/). See SKILL.md Step 2 for the baseAnalyzerId table and field-description rules. Keys beginning with '_' (_comment, _optional_*) are documentation aids; the local validator strips them before submission.", + "baseAnalyzerId": "prebuilt-document", + "description": "REPLACE: one-line description of what this analyzer extracts.", + "config": { + "_comment": "estimateFieldSourceAndConfidence is required for any field with method=extract. Other knobs (enableOcr, enableLayout, enableFormula) are optional; defaults are usually fine.", + "estimateFieldSourceAndConfidence": true, + "returnDetails": true, + "_optional_enableOcr": true, + "_optional_enableLayout": true, + "_optional_enableFormula": false + }, + "models": { + "_comment": "Required when using fieldSchema unless the resource has defaults set via samples-dev/updateDefaults.ts. Leaving it set is always safe.", + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + }, + "fieldSchema": { + "name": "REPLACE_schema_name_v1", + "description": "REPLACE: schema purpose.", + "fields": { + "exampleExtractField": { + "_comment": "method=extract: copy text verbatim from a specific location. Requires estimateSourceAndConfidence=true.", + "type": "string", + "method": "extract", + "description": "REPLACE: value printed after the 'EXAMPLE LABEL:' anchor near the top of the page. Reference text anchors (labels, headings), not visual appearance.", + "estimateSourceAndConfidence": true + }, + "exampleGenerateField": { + "_comment": "method=generate: AI synthesizes a value from the whole document. Use for summaries, derived values, free-form interpretation. estimateSourceAndConfidence is N/A here.", + "type": "string", + "method": "generate", + "description": "REPLACE: a one-sentence summary of the document's purpose." + }, + "exampleClassifyField": { + "_comment": "method=classify: AI picks one of the enum values. Use for fixed categorical fields.", + "type": "string", + "method": "classify", + "description": "REPLACE: high-level document type.", + "enum": ["invoice", "receipt", "contract", "report", "other"] + }, + "exampleDateField": { + "type": "date", + "method": "extract", + "description": "REPLACE: date printed after the 'DATE:' label. Not to be confused with other dates such as a due date or service date.", + "estimateSourceAndConfidence": true + }, + "exampleAmountField": { + "type": "number", + "method": "extract", + "description": "REPLACE: amount on the row labelled 'TOTAL' in the totals table at the bottom. Excludes any 'Subtotal' or 'Previous Balance' values.", + "estimateSourceAndConfidence": true + }, + "exampleNestedObjectField": { + "_comment": "Nested object (no array). Use when a single logical entity has several sub-fields (e.g. an address block).", + "type": "object", + "method": "extract", + "description": "REPLACE: the multi-line address block under the 'BILL TO:' heading.", + "properties": { + "name": { "type": "string", "method": "extract", "description": "REPLACE: company or person name on the first line." }, + "street": { "type": "string", "method": "extract", "description": "REPLACE: street address on the second line." }, + "city": { "type": "string", "method": "extract", "description": "REPLACE: city on the third line." } + } + }, + "exampleArrayField": { + "_comment": "Array of objects (line items / table rows). Wire key is 'items' (verified against the typed SDK's item_definition serialization).", + "type": "array", + "method": "extract", + "description": "REPLACE: each row of the line-items table.", + "items": { + "type": "object", + "properties": { + "itemCode": { "type": "string", "method": "extract", "description": "REPLACE: value in the ITEM CODE column." }, + "description": { "type": "string", "method": "extract", "description": "REPLACE: value in the DESCRIPTION column." }, + "amount": { "type": "number", "method": "extract", "description": "REPLACE: value in the AMOUNT column." } + } + } + } + } + } +} diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-common-knowledge/SKILL.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-common-knowledge/SKILL.md index ee06e50d7e33..fd4e7d60aefb 100644 --- a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-common-knowledge/SKILL.md +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-common-knowledge/SKILL.md @@ -50,7 +50,89 @@ Always read the relevant page (via `fetch_webpage`) before answering if the refe > **Search tip:** If the above pages don't cover the user's question, search the doc tree at `https://learn.microsoft.com/azure/ai-services/content-understanding/`. +## Field-description rule: the two-stage pipeline + +Custom analyzer extraction is a **two-stage pipeline**: + +1. **Stage 1 — content extraction (OCR + layout).** The service reads the + file and produces structured text plus layout metadata (sections, tables, + headings). The original pixels are *not* what the LLM in stage 2 sees. +2. **Stage 2 — field extraction (LLM).** The LLM reads the stage-1 markdown + and uses your field descriptions to identify values. + +Implications for `fieldSchema.fields[*].description`: + +✅ Reference **text content and structure**: labels (`"Invoice #"`), +section headings (`"Bill To"`), adjacent labels, alternative phrasings, +format examples. + +❌ Do **not** reference visual appearance: colour, font, font size, bold or +italic, or "the box at the top-right" without text anchors. + +Good description: + +> "Invoice issue date, found near the 'Invoice #' label at the top right. +> May also be labelled 'Invoice Date', 'Date', or 'Issued'. Format is +> usually MM/DD/YYYY. Examples: '01/15/2024', 'January 15, 2024'." + +Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) +and [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md). + +## Choosing `baseAnalyzerId` + +Every custom analyzer extends a built-in prebuilt analyzer via +`baseAnalyzerId`. Pick the row that matches the **modality** of the content +you're analyzing (documents, audio, video, image). Typos here are a common +first-time error; the local schema validator (in +[`_shared/schemaValidator.ts`](../_shared/)) rejects any value not in this +table. + +| Content type | `baseAnalyzerId` | +|---|---| +| Documents (PDF, image of a page) | `prebuilt-document` | +| Audio (mp3, wav, m4a) | `prebuilt-audio` | +| Video (mp4, mov) | `prebuilt-video` | +| Image-only analyzer | `prebuilt-image` | + +> ⚠️ **Only modality-level prebuilt analyzers are valid as `baseAnalyzerId` for +> custom analyzers.** `*Search` variants (`prebuilt-documentSearch`, +> `prebuilt-audioSearch`, `prebuilt-videoSearch`), task-specific prebuilt analyzers +> (`prebuilt-invoice`, `prebuilt-receipt`, `prebuilt-idDocument`), and +> `prebuilt-layout` are **not** accepted here — the service returns +> `InvalidBaseAnalyzerId`. Those prebuilt analyzers can still be called directly as +> standalone analyzers via +> `client.analyze("prebuilt-invoice", ...)`. See the +> [analyzer-reference docs](https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/analyzer-reference#baseanalyzerid). + +Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) +(custom analyzer) and +[`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) +(both inner extractors and the outer classifier). + +## Classify-and-route rule + +When using `config.contentCategories` to classify and route mixed-document +packets: + +1. **Category descriptions follow the same text-anchored rule** as field + descriptions. Describe each category by the text that appears on its + pages (headings, labels), not by visual style. +2. **`config.enableSegment` must be `true`** so the classifier carves the + packet into segments before routing each one. +3. **Inner analyzers must already exist** before the outer classifier is + created. +4. **Category fill rate is per-category**, not packet-wide. A field that + only appears in invoice segments should be evaluated against the number + of invoice segments, not the total number of segments. +5. **No top-level `fieldSchema`** on the outer classifier. The outer + analyzer's job is classification + routing only; field extraction + belongs in the inner analyzers. + +Used by [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md). + ## Related Skills - `cu-sdk-setup` — Set up JavaScript environment and run samples - `cu-sdk-sample-run` — Run specific samples interactively +- `cu-sdk-author-analyzer` — Author + test a custom analyzer for one document type +- `cu-sdk-author-analyzer-classify-route` — Author + test a classify-and-route pipeline for mixed-document packets diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-sample-run/SKILL.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-sample-run/SKILL.md index 10ffa9117f3a..3d73f2459a6a 100644 --- a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-sample-run/SKILL.md +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-sample-run/SKILL.md @@ -76,12 +76,14 @@ Gets raw JSON response for custom processing. Creates custom analyzer with field schema for domain-specific extraction. - Key concepts: Field types (string, number, date, object, array), extraction methods (extract, generate, classify) +- **Next step:** to run this workflow on your own folder of documents (validate schema → create → batch test → stdout summary), use the [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) skill. #### `createClassifier` Creates classifier to categorize documents (Loan_Application, Invoice, Bank_Statement). - Key concepts: Content categories, segmentation, document routing +- **Next step:** to run this workflow on your own mixed-document packets, use the [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) skill. ### Analyzer Management diff --git a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-setup/SKILL.md b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-setup/SKILL.md index 64f414852e63..2b12019aff7a 100644 --- a/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-setup/SKILL.md +++ b/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-setup/SKILL.md @@ -9,11 +9,20 @@ Set up your JavaScript / Node.js environment to use the Azure AI Content Underst > **[COPILOT INTERACTION MODEL]:** This skill is designed to be interactive. At each step marked with **[ASK USER]**, pause execution and prompt the user for input or confirmation before proceeding. Do NOT silently skip these prompts. Use the `ask_questions` tool when available. +> **[COPILOT] Step numbering is stable contract.** Authoring skills +> ([`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) and +> [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md)) +> route the user to specific steps here on probe failures — e.g. "endpoint +> `MISSING` → Step 4 (env vars)". When renumbering or restructuring this +> skill, update the routing tables in those skills (or keep step numbers +> stable). + ## Prerequisites Before starting, ensure you have: -- **Node.js 20 or later** installed +- **Node.js 22 or later** installed (matches `@azure/ai-content-understanding`'s + `engines.node: ">=22.0.0"` constraint) - An **Azure subscription** ([create one for free](https://azure.microsoft.com/free/)) - A **Microsoft Foundry resource** in a [supported region](https://learn.microsoft.com/azure/ai-services/content-understanding/language-region-support) @@ -36,12 +45,12 @@ Before starting, ensure you have: > > | Finding | Action | > | ------------------------------------------------- | ---------------------------------------------------------------------------------- | -> | `node v20+` and `npm` present | ✓ Good to go. Proceed to Step 1. | -> | Node missing or `< v20` | Report the finding, then go to the **[ASK USER] Node install choice** block below. | +> | `node v22+` and `npm` present | ✓ Good to go. Proceed to Step 1. | +> | Node missing or `< v22` | Report the finding, then go to the **[ASK USER] Node install choice** block below. | > | `npm` missing (very rare — should ship with Node) | Reinstall Node.js LTS. | > > **[ASK USER] Node install choice (only when probe fails):** -> Ask the user: "Node.js is missing or older than v20. How would you like to proceed?" +> Ask the user: "Node.js is missing or older than v22. How would you like to proceed?" > > - **Option A: Install it for me** — Agent runs the platform-appropriate install command (see below), verifies, and continues. > - **Option B: I'll install it myself** — Agent prints the install command for the user's platform and stops. User runs it, re-opens the terminal, and tells the agent to resume. @@ -378,7 +387,7 @@ cd sdk/contentunderstanding/ai-content-understanding The script will: -1. Probe and (optionally) install Node.js (>= 20). +1. Probe and (optionally) install Node.js (>= 22). 2. Detect existing `.env` and ask before overwriting. 3. Probe the Foundry resource for existing model defaults and prefill prompts. 4. Collect endpoint, auth method, and model deployment names. @@ -429,7 +438,7 @@ cp .env samples/v1/javascript/.env | Error | Solution | | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -| `node: command not found` | Install Node.js 20+ from [nodejs.org](https://nodejs.org/) or via `winget install OpenJS.NodeJS.LTS` / `brew install node`. | +| `node: command not found` | Install Node.js 22+ from [nodejs.org](https://nodejs.org/) or via `winget install OpenJS.NodeJS.LTS` / `brew install node`. | | `Cannot find module '@azure/ai-content-understanding'` | Run `setup_user_env.sh` to install (with automatic local-build fallback if not yet on npm). | | `Missing environment variables` / `CONTENTUNDERSTANDING_ENDPOINT` | Ensure `.env` exists in `samples/v1/javascript/` and is populated. Re-copy with `cp .env samples/v1/javascript/.env`. | | `Access denied due to invalid subscription key` | Verify `CONTENTUNDERSTANDING_ENDPOINT` URL is correct. Check API key or run `az login`. | diff --git a/sdk/contentunderstanding/ai-content-understanding/CHANGELOG.md b/sdk/contentunderstanding/ai-content-understanding/CHANGELOG.md index 1e9a2104f3dd..47b47ec13fa5 100644 --- a/sdk/contentunderstanding/ai-content-understanding/CHANGELOG.md +++ b/sdk/contentunderstanding/ai-content-understanding/CHANGELOG.md @@ -10,6 +10,25 @@ ### Other Changes +- Added GitHub Copilot skills under `.github/skills/` to help users + iteratively author custom analyzers in VS Code with Copilot: + - **`cu-sdk-author-analyzer`** — author and refine a custom analyzer + for a single document type (layout extraction → schema drafting → + validation → batch test → agent review → refine cycle). Document + modality only today; audio/video/image are planned. + - **`cu-sdk-author-analyzer-classify-route`** — author and refine a + classify-and-route pipeline for mixed-document packets (e.g. + invoice + bank statement + loan application in one PDF), with + per-category agent review. + + Both skills delegate to a small `cu-skill` TypeScript tool under + `.github/skills/_shared/` that exposes three subcommands — + `extract-layout`, `create-and-test`, and `create-and-test-router` — + and a pure-TypeScript `SchemaValidator` (Node `fs` only, no + `@azure/*` deps) that catches structural mistakes (unknown + `baseAnalyzerId`, missing `fieldSchema`, malformed + `contentCategories` routes) before a service round-trip. + ## 1.2.0-beta.2 (2026-06-11) ### Bugs Fixed diff --git a/sdk/contentunderstanding/ai-content-understanding/README.md b/sdk/contentunderstanding/ai-content-understanding/README.md index ad667361091e..eaecf49e96fa 100644 --- a/sdk/contentunderstanding/ai-content-understanding/README.md +++ b/sdk/contentunderstanding/ai-content-understanding/README.md @@ -681,6 +681,21 @@ For full setup instructions and available samples, see: - [TypeScript samples README](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/samples/v1/typescript/README.md) - [JavaScript samples README](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/samples/v1/javascript/README.md) +## What's New + +See [CHANGELOG.md][changelog] for the full release history. + +Recent highlights: + +- **Authoring skills for GitHub Copilot.** Two new skills, + [`cu-sdk-author-analyzer`][cu_sdk_author_analyzer_skill] (single document + type) and + [`cu-sdk-author-analyzer-classify-route`][cu_sdk_author_analyzer_classify_route_skill] + (mixed-document packets), guide the user and Copilot through an iterative + schema → test → review cycle for authoring custom Content Understanding + analyzers. See the [GitHub Copilot Skills](#github-copilot-skills) section + below. + ## GitHub Copilot Skills This package includes [GitHub Copilot][github_copilot] skills under `.github/skills/` that provide interactive, AI-assisted workflows for common tasks. In VS Code, Copilot can use these skills to help with environment setup, running samples, and understanding the service. @@ -692,6 +707,8 @@ This package includes [GitHub Copilot][github_copilot] skills under `.github/ski | [**cu-sdk-setup**][cu_sdk_setup_skill] | Interactive environment setup — installs the SDK, configures `.env` with endpoint and credentials, and runs the one-time `updateDefaults.js` model configuration | In VS Code Copilot Chat, ask: *"Set up my JavaScript environment for Content Understanding"* or reference the skill directly | | [**cu-sdk-sample-run**][cu_sdk_sample_run_skill] | Guided sample runner — helps you choose and run specific JavaScript samples with Node.js | Ask: *"Run analyzeUrl sample"* or *"Run the invoice analysis sample"* | | [**cu-sdk-common-knowledge**][cu_sdk_common_knowledge_skill] | Domain knowledge reference — answers questions about Content Understanding concepts, analyzers, field schemas, API operations, and JavaScript SDK usage | Ask: *"What prebuilt analyzers are available?"* or *"How do I create a custom analyzer?"* | +| [**cu-sdk-author-analyzer**][cu_sdk_author_analyzer_skill] | Iteratively author and test a custom analyzer for a single **document** type — layout extraction, schema drafting, local validation, batch test, agent review, and refine cycle. Document modality only today; audio/video/image are planned. | Ask: *"Help me author a custom analyzer for these invoices"* | +| [**cu-sdk-author-analyzer-classify-route**][cu_sdk_author_analyzer_classify_route_skill] | Iteratively author and test a classify-and-route pipeline for mixed-document packets (e.g. invoice + bank statement + loan application in one PDF). Includes per-category agent review and refinement of both classifier descriptions and inner-schema field descriptions. | Ask: *"Help me author an analyzer for this mixed financial packet"* | ### Using Skills in VS Code @@ -752,4 +769,7 @@ If you'd like to contribute to this library, please read the [contributing guide [cu_sdk_setup_skill]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-setup [cu_sdk_sample_run_skill]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-sample-run [cu_sdk_common_knowledge_skill]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-common-knowledge +[cu_sdk_author_analyzer_skill]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer +[cu_sdk_author_analyzer_classify_route_skill]: https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/contentunderstanding/ai-content-understanding/.github/skills/cu-sdk-author-analyzer-classify-route +[changelog]: https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/contentunderstanding/ai-content-understanding/CHANGELOG.md [file_issue]: https://github.com/Azure/azure-sdk-for-js/issues/new?labels=Cognitive%20-%20Content%20Understanding&title=[ContentUnderstanding]%20&body=%23%23%20Library%20Version%0A%0A%23%23%20Repro%20Steps%0A%0A%23%23%20Expected%20Result%0A%0A%23%23%20Actual%20Result