diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/README.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/README.md new file mode 100644 index 000000000000..874404d40593 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/README.md @@ -0,0 +1,65 @@ +# `_shared/` — library, not a skill + +Pure-Java helpers used by the authoring skill scripts under +`.github/skills/cu-sdk-author-analyzer*/`. + +The leading underscore marks this as a **library directory**, not a skill. It +is intentionally excluded from the Copilot skill picker. + +Rules for code in `SchemaValidator.java` (the validator): + +- **No `com.azure.*` imports.** No network calls. No I/O beyond reading / + parsing caller-provided JSON. +- **No new runtime dependencies.** Jackson (`jackson-databind`) only. +- **Stable, small, well-tested.** Anything here is referenced by multiple skill + scripts; breakage cascades. + +The CLI command classes (`ExtractLayoutCommand.java`, +`CreateAndTestCommand.java`, `CreateAndTestRouterCommand.java`) wrap the +`ContentUnderstandingClient` from `azure-ai-contentunderstanding`. They are +allowed to import `com.azure.*` since they're the bridge between the +validator and the service. + +Current modules: + +- [`SchemaValidator.java`](src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java) — + validates analyzer schema JSON before any service call (catches + `baseAnalyzerId` typos, missing `fieldSchema`, missing + `contentCategories` analyzer routes, etc.). Pure Jackson. +- [`Cli.java`](src/main/java/com/azure/ai/contentunderstanding/skills/Cli.java) — + subcommand dispatcher. +- [`ExtractLayoutCommand.java`](src/main/java/com/azure/ai/contentunderstanding/skills/ExtractLayoutCommand.java) — + Stage 1: extract document layout. +- [`CreateAndTestCommand.java`](src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommand.java) — + Stage 2 (single-type). +- [`CreateAndTestRouterCommand.java`](src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommand.java) — + Stage 2 (classify-and-route). + +## Build + +The Maven module is **intentionally NOT a child of azure-client-sdk-parent** +and is NOT referenced from the package POM — it lives outside the published +source tree so it has zero effect on the published +`azure-ai-contentunderstanding` artifact. + +```bash +mvn -B -q -f .github/skills/_shared/pom.xml -DskipTests compile +``` + +Dependencies are pulled from Maven Central using published versions; the tool +builds against the public SDK rather than the local reactor so contributors +can iterate without first running `mvn install` on the parent. + +## Run + +```bash +mvn -B -q -f .github/skills/_shared/pom.xml exec:java \ + -Dexec.args="extract-layout --input --output " +``` + +Or, after one `mvn package`, invoke directly: + +```bash +java -cp .github/skills/_shared/target/cu-skill-0.1.0.jar: \ + com.azure.ai.contentunderstanding.skills.Cli extract-layout ... +``` diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml new file mode 100644 index 000000000000..9f7dfcd6a360 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml @@ -0,0 +1,101 @@ + + + + 4.0.0 + + com.azure.ai.contentunderstanding.skills + cu-skill + 0.1.0 + jar + + cu-skill — Content Understanding analyzer-authoring tool + + Companion CLI for the cu-sdk-author-analyzer / cu-sdk-author-analyzer-classify-route + GitHub Copilot skills. Wraps the azure-ai-contentunderstanding Java SDK with three + subcommands (extract-layout, create-and-test, create-and-test-router) and a pure-Java + schema validator that catches structural mistakes before a service round-trip. + + + + 17 + 17 + UTF-8 + + true + + + + + + com.azure + azure-ai-contentunderstanding + 1.1.0-beta.2 + + + com.azure + azure-identity + 1.18.4 + + + com.azure + azure-core + 1.58.1 + + + + com.fasterxml.jackson.core + jackson-databind + 2.18.7 + + + + + org.junit.jupiter + junit-jupiter + 5.11.4 + test + + + + + + + org.codehaus.mojo + exec-maven-plugin + 3.5.0 + + com.azure.ai.contentunderstanding.skills.Cli + + + + org.apache.maven.plugins + maven-surefire-plugin + 3.5.2 + + + + diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/Cli.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/Cli.java new file mode 100644 index 000000000000..c7eaf7029c17 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/Cli.java @@ -0,0 +1,61 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Subcommand dispatcher for the cu-skill tool. Mirrors Python's + * extract_layout.py / create_and_test.py / create_and_test_router.py + * entry points and the .NET Program.cs. + */ + +package com.azure.ai.contentunderstanding.skills; + +import java.io.IOException; +import java.util.Arrays; + +public final class Cli { + + private Cli() { + } + + public static void main(String[] args) throws IOException { + if (args.length == 0 || isHelp(args[0])) { + printUsage(); + System.exit(args.length == 0 ? 1 : 0); + return; + } + String subcommand = args[0]; + String[] subArgs = Arrays.copyOfRange(args, 1, args.length); + int exit; + switch (subcommand) { + case "extract-layout": + exit = ExtractLayoutCommand.run(subArgs); + break; + case "create-and-test": + exit = CreateAndTestCommand.run(subArgs); + break; + case "create-and-test-router": + exit = CreateAndTestRouterCommand.run(subArgs); + break; + default: + System.err.println("unknown subcommand: " + subcommand); + printUsage(); + exit = 1; + } + System.exit(exit); + } + + private static boolean isHelp(String arg) { + return "-h".equals(arg) || "--help".equals(arg) || "help".equals(arg); + } + + private static void printUsage() { + System.out.println("cu-skill — Content Understanding analyzer-authoring tool."); + System.out.println(); + System.out.println("Subcommands:"); + System.out.println(" extract-layout extract document layout (stage 1)"); + System.out.println(" create-and-test validate, create, batch-test a single-type analyzer"); + System.out.println(" create-and-test-router classify-and-route variant (N inner + 1 outer)"); + System.out.println(); + System.out.println("Use ' --help' for per-command flags."); + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommand.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommand.java new file mode 100644 index 000000000000..0e3304f95f67 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommand.java @@ -0,0 +1,741 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Stage 2 of the analyzer-authoring loop (single-document-type variant): + * validate the schema locally, create the analyzer, batch-test inputs, dump + * per-document JSON + LLM-friendly markdown, and print a stdout summary with + * per-field fill-rate + avg-confidence. Mirrors Python's create_and_test.py + * and the .NET CreateAndTestCommand.cs. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.azure.core.exception.HttpResponseException; +import com.azure.core.http.rest.RequestOptions; +import com.azure.core.util.BinaryData; +import com.azure.core.util.polling.SyncPoller; +import com.azure.ai.contentunderstanding.ContentUnderstandingClient; +import com.azure.ai.contentunderstanding.LlmInputHelper; +import com.azure.ai.contentunderstanding.models.AnalysisResult; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.databind.node.ObjectNode; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.MessageDigest; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; + +final class CreateAndTestCommand { + + private static final ObjectMapper MAPPER = new ObjectMapper() + .enable(SerializationFeature.INDENT_OUTPUT); + + private static final Set SUPPORTED_SUFFIXES = new LinkedHashSet<>( + Arrays.asList(".pdf", ".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".heif", ".heic", + ".wav", ".mp3", ".m4a", ".mp4", ".mov")); + + private CreateAndTestCommand() { + } + + static int run(String[] args) throws IOException { + Options opts = parseArgs(args); + if (opts == null) { + return 2; + } + if (opts.iterations < 1) { + System.err.println("--iterations must be >= 1"); + return 2; + } + Path schemaPath = Paths.get(opts.schema); + if (!Files.exists(schemaPath)) { + System.err.println("schema not found: " + schemaPath); + return 2; + } + Path inputPath = Paths.get(opts.input); + if (!Files.exists(inputPath)) { + System.err.println("input not found: " + inputPath); + return 2; + } + return runCore(opts, schemaPath, inputPath); + } + + static int runCore(Options opts, Path schemaPath, Path inputPath) throws IOException { + // 1. Validate schema locally. + SchemaValidator.Result validation = SchemaValidator.validateFile(schemaPath); + if (!validation.isOk()) { + for (String e : validation.getErrors()) { + System.err.println("[VALIDATE] " + e); + } + return 2; + } + + String rawJson = Files.readString(schemaPath); + ObjectNode rawSchema = (ObjectNode) MAPPER.readTree(rawJson); + ObjectNode schema = stripComments(rawSchema); + + // Pre-flight warning: fieldSchema without models.completion. + if (schema.has("fieldSchema")) { + JsonNode models = schema.get("models"); + JsonNode completion = models != null ? models.get("completion") : null; + if (completion == null || !completion.isTextual() || completion.asText().isBlank()) { + System.err.println( + "[WARN] schema has fieldSchema but no models.completion; " + + "this will fail unless resource defaults are configured " + + "(run the Sample00_UpdateDefaultsAsync sample)."); + } + } + + List inputs = enumerateInputs(inputPath); + if (inputs.isEmpty()) { + System.err.println("no supported documents found under " + inputPath); + return 2; + } + Files.createDirectories(Paths.get(opts.output)); + + String analyzerId = opts.analyzerId; + if (analyzerId == null || analyzerId.isEmpty()) { + String stem = stripExtension(schemaPath.getFileName().toString()); + analyzerId = opts.reuse + ? stem + "_" + schemaHash(schema) + : stem + "_" + (System.currentTimeMillis() / 1000L); + } + + ContentUnderstandingClient client = ExtractLayoutCommand.buildClient(); + + boolean reused = false; + int fail = 0; + List results = new ArrayList<>(); + try { + if (opts.reuse) { + reused = ensureAnalyzer(client, analyzerId, schema); + } else { + createAnalyzer(client, analyzerId, schema); + } + for (Path file : inputs) { + for (int iter = 1; iter <= opts.iterations; iter++) { + String suffix = opts.iterations > 1 ? String.format("_iter%03d", iter) : ""; + String stem = stripExtension(file.getFileName().toString()); + Path outPath = Paths.get(opts.output, stem + suffix + ".json"); + try { + System.out.println("[ANALYZE] " + file + " -> " + outPath); + AnalyzeResult r = analyzeFile(client, analyzerId, file); + Files.writeString( + outPath, + MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(r.doc)); + if (r.llmMarkdown != null && !r.llmMarkdown.isEmpty()) { + Path llmPath = outPath.resolveSibling( + stem + suffix + ".llm.md"); + Files.writeString(llmPath, r.llmMarkdown); + } + results.add(new NamedDoc(stripExtension(outPath.getFileName().toString()), r.doc)); + } catch (Exception ex) { + System.err.println("[FAIL] " + file + ": " + ex.getMessage()); + fail++; + } + } + } + } finally { + cleanup(client, analyzerId, opts.ephemeral, reused); + } + + System.out.println(summarize(results)); + return fail == 0 ? 0 : 1; + } + + // ----------------------------------------------------------------------- + // Service interaction + // ----------------------------------------------------------------------- + + static boolean ensureAnalyzer(ContentUnderstandingClient client, String analyzerId, ObjectNode schema) + throws IOException { + try { + client.getAnalyzer(analyzerId); + System.out.println("[REUSE] analyzer " + analyzerId + " already exists"); + return true; + } catch (HttpResponseException ex) { + // Treat any error as "not found" — the next createAnalyzer call + // will surface the real reason if it's something else (auth, etc.) + // Mirrors Python's broad-except in ensure_analyzer. + } catch (RuntimeException ex) { + // Same rationale. + } + createAnalyzer(client, analyzerId, schema); + return false; + } + + static void createAnalyzer(ContentUnderstandingClient client, String analyzerId, ObjectNode schema) + throws IOException { + System.out.println("[CREATE] analyzer_id=" + analyzerId); + // Use the protocol overload so we can pass arbitrary schema JSON + // (the typed ContentAnalyzer model might not expose every property). + BinaryData body = BinaryData.fromString(MAPPER.writeValueAsString(schema)); + SyncPoller poller = + client.beginCreateAnalyzer(analyzerId, body, new RequestOptions()); + // Block on completion so any service error surfaces here, not at + // first analyze call. + poller.waitForCompletion(); + System.out.println("[CREATE] " + analyzerId + " ready"); + } + + static AnalyzeResult analyzeFile(ContentUnderstandingClient client, String analyzerId, Path file) + throws IOException { + byte[] bytes = Files.readAllBytes(file); + String contentType = guessContentType(file); + // Protocol overload returns raw service JSON (LRO envelope). + SyncPoller poller = client.beginAnalyzeBinary( + analyzerId, + contentType, + BinaryData.fromBytes(bytes), + new RequestOptions()); + BinaryData raw = poller.getFinalResult(); + JsonNode envelope = MAPPER.readTree(raw.toString()); + // Unwrap `{id, status, result, usage}` so on-disk shape matches + // Python's `poller.result()` output. + ObjectNode payload = envelope.has("result") + ? (ObjectNode) envelope.get("result") + : (ObjectNode) envelope; + + // Best-effort LLM-friendly markdown. We need a typed AnalysisResult + // for LlmInputHelper, which requires re-deserializing the unwrapped + // payload through Jackson into the SDK's BinaryData model loader. + String llmText = null; + try { + AnalysisResult typed = BinaryData.fromString(MAPPER.writeValueAsString(payload)) + .toObject(AnalysisResult.class); + llmText = LlmInputHelper.toLlmInput(typed); + } catch (RuntimeException ex) { + // Raw JSON is always written; LLM text is optional. + } + return new AnalyzeResult(payload, llmText); + } + + static void cleanup(ContentUnderstandingClient client, String analyzerId, boolean ephemeral, boolean reused) { + if (ephemeral) { + try { + System.out.println("[CLEANUP] delete analyzer " + analyzerId); + client.deleteAnalyzer(analyzerId); + } catch (RuntimeException ex) { + System.err.println("[CLEANUP] delete failed: " + ex.getMessage()); + } + } else if (reused) { + System.out.println("[KEEP] reused analyzer " + analyzerId + " retained"); + } else { + System.out.println( + "[KEEP] analyzer " + analyzerId + " retained (use --ephemeral to delete)"); + } + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + private static List enumerateInputs(Path inputPath) throws IOException { + List result = new ArrayList<>(); + if (Files.isRegularFile(inputPath)) { + result.add(inputPath); + return result; + } + try (var stream = Files.list(inputPath)) { + stream + .filter(Files::isRegularFile) + .filter(p -> { + String n = p.getFileName().toString().toLowerCase(Locale.ROOT); + int dot = n.lastIndexOf('.'); + return dot >= 0 && SUPPORTED_SUFFIXES.contains(n.substring(dot)); + }) + .sorted() + .forEach(result::add); + } + return result; + } + + private static String stripExtension(String name) { + int dot = name.lastIndexOf('.'); + return dot > 0 ? name.substring(0, dot) : name; + } + + private static String guessContentType(Path file) { + String name = file.getFileName().toString().toLowerCase(Locale.ROOT); + if (name.endsWith(".pdf")) { + return "application/pdf"; + } + if (name.endsWith(".png")) { + return "image/png"; + } + if (name.endsWith(".jpg") || name.endsWith(".jpeg")) { + return "image/jpeg"; + } + if (name.endsWith(".tif") || name.endsWith(".tiff")) { + return "image/tiff"; + } + if (name.endsWith(".bmp")) { + return "image/bmp"; + } + if (name.endsWith(".heif") || name.endsWith(".heic")) { + return "image/heif"; + } + if (name.endsWith(".wav")) { + return "audio/wav"; + } + if (name.endsWith(".mp3")) { + return "audio/mpeg"; + } + if (name.endsWith(".m4a")) { + return "audio/mp4"; + } + if (name.endsWith(".mp4")) { + return "video/mp4"; + } + if (name.endsWith(".mov")) { + return "video/quicktime"; + } + return "application/octet-stream"; + } + + /** + * Recursively drop any object key whose name starts with {@code _}. + * Lets templates carry {@code _comment} documentation keys without + * poisoning the service request body. + */ + static ObjectNode stripComments(ObjectNode obj) { + ObjectNode out = MAPPER.createObjectNode(); + Iterator> it = obj.fields(); + while (it.hasNext()) { + Map.Entry e = it.next(); + if (e.getKey().startsWith("_")) { + continue; + } + out.set(e.getKey(), stripCommentsNode(e.getValue())); + } + return out; + } + + private static JsonNode stripCommentsNode(JsonNode node) { + if (node instanceof ObjectNode obj) { + return stripComments(obj); + } + if (node instanceof ArrayNode arr) { + ArrayNode out = MAPPER.createArrayNode(); + for (JsonNode item : arr) { + out.add(stripCommentsNode(item)); + } + return out; + } + return node; + } + + /** Stable 8-char sha1 hash over the canonical JSON form of the schema. */ + static String schemaHash(ObjectNode schema) { + try { + String canonical = canonicalize(schema); + MessageDigest md = MessageDigest.getInstance("SHA-1"); + byte[] digest = md.digest(canonical.getBytes(StandardCharsets.UTF_8)); + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < 4; i++) { + sb.append(String.format("%02x", digest[i])); + } + return sb.toString(); + } catch (Exception ex) { + throw new IllegalStateException("hash failed: " + ex.getMessage(), ex); + } + } + + /** Object keys sorted alphabetically; arrays preserve order — same as Python. */ + static String canonicalize(JsonNode node) { + try { + return MAPPER.writer().writeValueAsString(sortKeys(node)); + } catch (IOException ex) { + throw new IllegalStateException("canonicalize failed: " + ex.getMessage(), ex); + } + } + + private static JsonNode sortKeys(JsonNode node) { + if (node instanceof ObjectNode obj) { + List keys = new ArrayList<>(); + obj.fieldNames().forEachRemaining(keys::add); + keys.sort(Comparator.naturalOrder()); + ObjectNode out = MAPPER.createObjectNode(); + for (String k : keys) { + out.set(k, sortKeys(obj.get(k))); + } + return out; + } + if (node instanceof ArrayNode arr) { + ArrayNode out = MAPPER.createArrayNode(); + for (JsonNode item : arr) { + out.add(sortKeys(item)); + } + return out; + } + return node; + } + + // ----------------------------------------------------------------------- + // Summary + // ----------------------------------------------------------------------- + + /** Build the stdout summary string. */ + static String summarize(List results) { + // category -> field -> list of (docName, value, confidence) + Map>> table = new LinkedHashMap<>(); + for (NamedDoc nd : results) { + for (FieldRecord f : iterFields(nd.doc)) { + table + .computeIfAbsent(f.category, k -> new LinkedHashMap<>()) + .computeIfAbsent(f.fieldPath, k -> new ArrayList<>()) + .add(new RowEntry(nd.name, fieldValue(f.fieldVal), fieldConfidence(f.fieldVal))); + } + } + if (table.isEmpty()) { + return "[SUMMARY] no fields extracted across any document."; + } + + StringBuilder sb = new StringBuilder(); + sb.append("\n").append("=".repeat(72)).append("\n"); + sb.append("[SUMMARY]\n"); + for (Map.Entry>> catEntry : table.entrySet()) { + // Distinct docs across all fields in this category — used for the + // header count; for fill-rate we use each field's own row count + // (matches Python/.NET semantics so array leaves don't inflate + // the denominator). + Set docNames = new LinkedHashSet<>(); + for (List rows : catEntry.getValue().values()) { + for (RowEntry r : rows) { + docNames.add(r.docName); + } + } + int nDocs = docNames.size(); + String catLabel = catEntry.getKey().isEmpty() ? "(single)" : catEntry.getKey(); + String header = "category: " + catLabel + + " (" + nDocs + " document" + (nDocs == 1 ? "" : "s") + ")"; + sb.append("\n").append(header).append("\n"); + sb.append("-".repeat(header.length())).append("\n"); + sb.append(String.format(" %-40s %-10s %-10s%n", "field", "fill rate", "avg conf")); + for (Map.Entry> field : catEntry.getValue().entrySet()) { + List rows = field.getValue(); + int denom = rows.size(); + long filled = rows.stream().filter(r -> r.value != null).count(); + double fillRate = denom == 0 ? 0.0 : (double) filled / denom; + List confidences = new ArrayList<>(); + for (RowEntry r : rows) { + if (r.value != null && r.confidence != null) { + confidences.add(r.confidence); + } + } + String confStr = confidences.isEmpty() ? "n/a" + : String.format("%.3f", confidences.stream().mapToDouble(Double::doubleValue).average().orElse(0)); + sb.append(String.format( + " %-40s %-9s %s%n", + field.getKey(), + String.format("%.1f%%", fillRate * 100), confStr)); + } + } + + // Lowest-confidence across all (category, field, doc) triples. + List lows = new ArrayList<>(); + for (Map.Entry>> catEntry : table.entrySet()) { + for (Map.Entry> field : catEntry.getValue().entrySet()) { + for (RowEntry r : field.getValue()) { + if (r.confidence != null) { + lows.add(new LowConfRow(r.confidence, catEntry.getKey(), field.getKey(), r.docName)); + } + } + } + } + if (!lows.isEmpty()) { + lows.sort(Comparator.comparingDouble(x -> x.confidence)); + sb.append("\nlowest-confidence fields:\n"); + for (int i = 0; i < Math.min(3, lows.size()); i++) { + LowConfRow row = lows.get(i); + String catTag = row.category.isEmpty() ? "" : "[" + row.category + "] "; + sb.append(String.format( + " %.3f %s%s (%s)%n", + row.confidence, catTag, row.field, row.docName)); + } + } + sb.append("=".repeat(72)); + return sb.toString(); + } + + static List iterFields(ObjectNode doc) { + List out = new ArrayList<>(); + JsonNode contents = doc.get("contents"); + if (contents == null || !contents.isArray()) { + return out; + } + for (JsonNode content : contents) { + String category = content.has("category") && content.get("category").isTextual() + ? content.get("category").asText() : ""; + JsonNode fields = content.get("fields"); + if (fields == null || !fields.isObject()) { + continue; + } + Iterator> it = fields.fields(); + while (it.hasNext()) { + Map.Entry e = it.next(); + if (!(e.getValue() instanceof ObjectNode fieldObj)) { + continue; + } + for (PathLeaf pl : recurse(e.getKey(), fieldObj)) { + out.add(new FieldRecord(category, pl.path, pl.leaf)); + } + } + } + return out; + } + + private static List recurse(String prefix, ObjectNode fieldVal) { + List out = new ArrayList<>(); + JsonNode arr = fieldVal.get("valueArray"); + if (arr != null && arr.isArray()) { + for (JsonNode item : arr) { + if (item instanceof ObjectNode itemObj && itemObj.has("valueObject") + && itemObj.get("valueObject") instanceof ObjectNode valueObj) { + Iterator> it = valueObj.fields(); + while (it.hasNext()) { + Map.Entry e = it.next(); + if (e.getValue() instanceof ObjectNode childObj) { + out.addAll(recurse(prefix + "[]." + e.getKey(), childObj)); + } + } + } else { + ObjectNode wrap = MAPPER.createObjectNode(); + if (item.isTextual()) { + wrap.put("valueString", item.asText()); + } else if (item instanceof ObjectNode io) { + wrap = io.deepCopy(); + } + out.add(new PathLeaf(prefix, wrap)); + } + } + return out; + } + JsonNode obj = fieldVal.get("valueObject"); + if (obj instanceof ObjectNode objVal) { + Iterator> it = objVal.fields(); + while (it.hasNext()) { + Map.Entry e = it.next(); + if (e.getValue() instanceof ObjectNode childObj) { + out.addAll(recurse(prefix + "." + e.getKey(), childObj)); + } + } + return out; + } + out.add(new PathLeaf(prefix, fieldVal)); + return out; + } + + private static Object fieldValue(ObjectNode field) { + for (String key : new String[] { + "valueString", "valueNumber", "valueInteger", "valueBoolean", "valueDate", "valueTime" + }) { + JsonNode v = field.get(key); + if (v != null && !v.isNull() && !(v.isTextual() && v.asText().isEmpty())) { + return v; + } + } + JsonNode arr = field.get("valueArray"); + if (arr != null && arr.isArray() && arr.size() > 0) { + return arr; + } + JsonNode obj = field.get("valueObject"); + if (obj != null && obj.isObject() && obj.size() > 0) { + return obj; + } + return null; + } + + private static Double fieldConfidence(ObjectNode field) { + JsonNode c = field.get("confidence"); + if (c != null && (c.isNumber() || c.canConvertToExactIntegral())) { + return c.asDouble(); + } + return null; + } + + // ----------------------------------------------------------------------- + // CLI parsing + // ----------------------------------------------------------------------- + + private static Options parseArgs(String[] args) { + Options o = new Options(); + try { + for (int i = 0; i < args.length; i++) { + switch (args[i]) { + case "--schema": + o.schema = requireValue(args, i, "--schema"); + i++; + break; + case "--input": + o.input = requireValue(args, i, "--input"); + i++; + break; + case "--output": + o.output = requireValue(args, i, "--output"); + i++; + break; + case "--analyzer-id": + o.analyzerId = requireValue(args, i, "--analyzer-id"); + i++; + break; + case "--iterations": + String itersRaw = requireValue(args, i, "--iterations"); + try { + o.iterations = Integer.parseInt(itersRaw); + } catch (NumberFormatException nfe) { + throw new IllegalArgumentException( + "--iterations must be an integer, got: " + itersRaw); + } + i++; + break; + case "--ephemeral": + o.ephemeral = true; + break; + case "--reuse": + o.reuse = true; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + System.err.println("unknown argument: " + args[i]); + printUsage(); + return null; + } + } + } catch (IllegalArgumentException ex) { + System.err.println(ex.getMessage()); + printUsage(); + return null; + } + if (o.schema == null || o.input == null || o.output == null) { + System.err.println("--schema, --input, and --output are required"); + printUsage(); + return null; + } + return o; + } + + /** + * Returns the value following a flag ({@code args[i+1]}), or throws + * {@link IllegalArgumentException} with a usage-friendly message if the flag + * is at the end of {@code args}. Callers must {@code i++} themselves after + * consuming the value so the outer loop skips past it. + */ + private static String requireValue(String[] args, int i, String flag) { + if (i + 1 >= args.length) { + throw new IllegalArgumentException("missing value for " + flag); + } + return args[i + 1]; + } + + private static void printUsage() { + System.err.println("Usage:"); + System.err.println(" cu-skill create-and-test"); + System.err.println(" --schema "); + System.err.println(" --input "); + System.err.println(" --output "); + System.err.println(" [--analyzer-id ]"); + System.err.println(" [--iterations N]"); + System.err.println(" [--ephemeral]"); + System.err.println(" [--reuse]"); + System.err.println(); + System.err.println("Stage 2 (single-type): validate, create, batch-test a custom analyzer,"); + System.err.println("print a per-field fill-rate + avg-confidence summary."); + } + + static final class Options { + String schema; + String input; + String output; + String analyzerId = ""; + int iterations = 1; + boolean ephemeral; + boolean reuse; + } + + static final class AnalyzeResult { + final ObjectNode doc; + final String llmMarkdown; + + AnalyzeResult(ObjectNode doc, String llmMarkdown) { + this.doc = doc; + this.llmMarkdown = llmMarkdown; + } + } + + static final class NamedDoc { + final String name; + final ObjectNode doc; + + NamedDoc(String name, ObjectNode doc) { + this.name = name; + this.doc = doc; + } + } + + static final class FieldRecord { + final String category; + final String fieldPath; + final ObjectNode fieldVal; + + FieldRecord(String category, String fieldPath, ObjectNode fieldVal) { + this.category = category; + this.fieldPath = fieldPath; + this.fieldVal = fieldVal; + } + } + + private static final class PathLeaf { + final String path; + final ObjectNode leaf; + + PathLeaf(String path, ObjectNode leaf) { + this.path = path; + this.leaf = leaf; + } + } + + private static final class RowEntry { + final String docName; + final Object value; + final Double confidence; + + RowEntry(String docName, Object value, Double confidence) { + this.docName = docName; + this.value = value; + this.confidence = confidence; + } + } + + private static final class LowConfRow { + final double confidence; + final String category; + final String field; + final String docName; + + LowConfRow(double confidence, String category, String field, String docName) { + this.confidence = confidence; + this.category = category; + this.field = field; + this.docName = docName; + } + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommand.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommand.java new file mode 100644 index 000000000000..a16541fca7e7 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommand.java @@ -0,0 +1,798 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Stage 2 of the analyzer-authoring loop (classify-and-route variant): + * given an outer classifier schema and N inner field-extractor schemas, + * create all of them, wire the outer's `contentCategories[*].analyzerId` + * to the real inner analyzer IDs, batch-test inputs, and print a + * category-aware stdout summary. Mirrors Python's + * create_and_test_router.py and the .NET CreateAndTestRouterCommand.cs. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.azure.ai.contentunderstanding.ContentUnderstandingClient; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import com.fasterxml.jackson.databind.node.ObjectNode; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.MessageDigest; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Comparator; +import java.util.Iterator; +import java.util.LinkedHashMap; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.TreeMap; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.nio.charset.StandardCharsets; + +final class CreateAndTestRouterCommand { + + private static final ObjectMapper MAPPER = new ObjectMapper() + .enable(SerializationFeature.INDENT_OUTPUT); + + private static final Set SUPPORTED_SUFFIXES = new LinkedHashSet<>( + Arrays.asList(".pdf", ".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".heif", ".heic", + ".wav", ".mp3", ".m4a", ".mp4", ".mov")); + + private CreateAndTestRouterCommand() { + } + + static int run(String[] args) throws IOException { + Options opts = parseArgs(args); + if (opts == null) { + return 2; + } + Path outerPath = Paths.get(opts.outerSchema); + if (!Files.exists(outerPath)) { + System.err.println("outer schema not found: " + outerPath); + return 2; + } + Path inputPath = Paths.get(opts.input); + if (!Files.exists(inputPath)) { + System.err.println("input not found: " + inputPath); + return 2; + } + + // 1. Resolve inner schemas: either --inner-schema ALIAS=PATH ... or + // --schema-dir DIR (which auto-discovers *.json files and uses the + // file stem as the alias). + Map aliasToPath = new LinkedHashMap<>(); + for (String spec : opts.innerSchemas) { + int eq = spec.indexOf('='); + if (eq <= 0) { + System.err.println("--inner-schema must be ALIAS=PATH; got: " + spec); + return 2; + } + String alias = spec.substring(0, eq); + Path p = Paths.get(spec.substring(eq + 1)); + if (!Files.exists(p)) { + System.err.println("inner schema not found: " + p); + return 2; + } + aliasToPath.put(alias, p); + } + if (opts.schemaDir != null) { + Path dir = Paths.get(opts.schemaDir); + if (!Files.isDirectory(dir)) { + System.err.println("--schema-dir is not a directory: " + dir); + return 2; + } + // Read the outer schema so we know which aliases to look for. + // Matches Python's `_discover_inner_from_dir` and .NET's + // `DiscoverInnerFromDir`: for every category whose `analyzerId` is + // a non-`prebuilt-*` string, find a file whose stem is exactly + // `` or starts with `_`, and pick the + // alphabetically-last match (so `invoice_v2.json` wins over + // `invoice_v1.json`). + ObjectNode outerPreview; + try { + outerPreview = CreateAndTestCommand.stripComments( + (ObjectNode) MAPPER.readTree(Files.readString(outerPath))); + } catch (IOException | RuntimeException ex) { + System.err.println("outer schema is not valid JSON: " + ex.getMessage()); + return 2; + } + Map resolved = discoverInnerFromDir(outerPreview, dir); + if (resolved == null) { + return 2; + } + for (Map.Entry e : resolved.entrySet()) { + aliasToPath.putIfAbsent(e.getKey(), e.getValue()); + } + StringBuilder summary = new StringBuilder(); + for (Map.Entry e : resolved.entrySet()) { + if (summary.length() > 0) { + summary.append(", "); + } + summary.append(e.getKey()).append("=").append(e.getValue().getFileName()); + } + System.out.println("[SCHEMA-DIR] resolved: " + summary); + } + if (aliasToPath.isEmpty()) { + System.err.println("provide at least one --inner-schema or --schema-dir"); + return 2; + } + + // 2. Validate outer + each inner locally. + SchemaValidator.Result outerVal = SchemaValidator.validateFile(outerPath); + if (!outerVal.isOk()) { + for (String e : outerVal.getErrors()) { + System.err.println("[VALIDATE outer] " + e); + } + return 2; + } + ObjectNode outerSchema = CreateAndTestCommand.stripComments( + (ObjectNode) MAPPER.readTree(Files.readString(outerPath))); + + Map aliasToSchema = new LinkedHashMap<>(); + for (Map.Entry e : aliasToPath.entrySet()) { + SchemaValidator.Result r = SchemaValidator.validateFile(e.getValue()); + if (!r.isOk()) { + for (String msg : r.getErrors()) { + System.err.println("[VALIDATE inner " + e.getKey() + "] " + msg); + } + return 2; + } + aliasToSchema.put( + e.getKey(), + CreateAndTestCommand.stripComments( + (ObjectNode) MAPPER.readTree(Files.readString(e.getValue())))); + } + + // Cross-check is deferred until aliasToId is built (below): every + // outer contentCategories[].analyzerId must either start with + // "prebuilt-" (a service prebuilt) or resolve to an --inner-schema + // alias. wireInnerIds() returns any mismatches / unused inner + // schemas as validation errors. + + List inputs = enumerateInputs(inputPath); + if (inputs.isEmpty()) { + System.err.println("no supported documents found under " + inputPath); + return 2; + } + Files.createDirectories(Paths.get(opts.output)); + + // 3. Compute deterministic IDs (or timestamp IDs when --reuse not set). + String outerId = opts.analyzerId; + if (outerId == null || outerId.isEmpty()) { + String stem = stripExtension(outerPath.getFileName().toString()); + // Include the inner hashes in the outer hash so any inner-schema + // edit forces a fresh outer ID. Otherwise --reuse could pick up + // an outer that points at stale inner IDs. + StringBuilder combined = new StringBuilder(); + combined.append(CreateAndTestCommand.canonicalize(outerSchema)); + for (Map.Entry e : new TreeMap<>(aliasToSchema).entrySet()) { + combined.append("|").append(e.getKey()).append("=") + .append(CreateAndTestCommand.canonicalize(e.getValue())); + } + outerId = opts.reuse + ? stem + "_" + sha1Prefix(combined.toString()) + : stem + "_" + (System.currentTimeMillis() / 1000L); + } + + Map aliasToId = new LinkedHashMap<>(); + for (Map.Entry e : aliasToSchema.entrySet()) { + String suffix = opts.reuse + ? sha1Prefix(CreateAndTestCommand.canonicalize(e.getValue())) + : Long.toString(System.currentTimeMillis() / 1000L); + aliasToId.put(e.getKey(), outerId + "_inner_" + e.getKey() + "_" + suffix); + } + + // 4. Patch outer schema to point at the real inner IDs. This also + // validates that every outer contentCategories[].analyzerId + // resolves to an --inner-schema alias (or starts with "prebuilt-") + // and that no --inner-schema is unused. Mirrors Python's + // _patch_outer_analyzer_ids and .NET's WireInnerIds. + WireResult wired = wireInnerIds(outerSchema, aliasToId); + if (!wired.errors.isEmpty()) { + for (String e : wired.errors) { + System.err.println("[VALIDATE] " + e); + } + return 2; + } + ObjectNode wiredOuter = wired.patched; + + ContentUnderstandingClient client = ExtractLayoutCommand.buildClient(); + boolean reused = false; + int fail = 0; + List results = new ArrayList<>(); + try { + // 5. Create all inner analyzers first. + for (Map.Entry e : aliasToSchema.entrySet()) { + String id = aliasToId.get(e.getKey()); + if (opts.reuse) { + CreateAndTestCommand.ensureAnalyzer(client, id, e.getValue()); + } else { + CreateAndTestCommand.createAnalyzer(client, id, e.getValue()); + } + } + // 6. Then the outer classifier. + if (opts.reuse) { + reused = CreateAndTestCommand.ensureAnalyzer(client, outerId, wiredOuter); + } else { + CreateAndTestCommand.createAnalyzer(client, outerId, wiredOuter); + } + + // 7. Analyze inputs through the outer. + for (Path file : inputs) { + String stem = stripExtension(file.getFileName().toString()); + Path outPath = Paths.get(opts.output, stem + ".json"); + try { + System.out.println("[ANALYZE] " + file + " -> " + outPath); + CreateAndTestCommand.AnalyzeResult r = + CreateAndTestCommand.analyzeFile(client, outerId, file); + Files.writeString( + outPath, + MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(r.doc)); + if (r.llmMarkdown != null && !r.llmMarkdown.isEmpty()) { + Files.writeString( + outPath.resolveSibling(stem + ".llm.md"), + r.llmMarkdown); + } + results.add(new CreateAndTestCommand.NamedDoc(stem, r.doc)); + } catch (Exception ex) { + System.err.println("[FAIL] " + file + ": " + ex.getMessage()); + fail++; + } + } + } finally { + if (opts.ephemeral) { + List toDelete = new ArrayList<>(); + toDelete.add(outerId); + toDelete.addAll(aliasToId.values()); + for (String id : toDelete) { + try { + System.out.println("[CLEANUP] delete " + id); + client.deleteAnalyzer(id); + } catch (RuntimeException ex) { + System.err.println("[CLEANUP] delete failed for " + id + ": " + ex.getMessage()); + } + } + } else if (reused) { + System.out.println("[KEEP] reused analyzers retained"); + } else { + System.out.println("[KEEP] analyzers retained (use --ephemeral to delete)"); + System.out.println(" outer: " + outerId); + for (Map.Entry e : aliasToId.entrySet()) { + System.out.println(" inner [" + e.getKey() + "]: " + e.getValue()); + } + } + } + + System.out.println(summarizeRouted(results)); + return fail == 0 ? 0 : 1; + } + + /** + * Auto-build {@code {alias: path}} from a directory of inner schema files. + * For every category in the outer schema whose {@code analyzerId} is a + * non-{@code "prebuilt-"} alias, find a matching JSON file: filename stem + * equals the alias, or the stem starts with {@code "_"}. When + * multiple files match, pick the highest-numbered version — so + * {@code "invoice_v10.json"} beats {@code "invoice_v9.json"} (plain + * alphabetical sort broke as soon as version numbers hit two digits). + * If any alias has no match, logs the missing set to stderr and returns + * {@code null}. + * + *

Mirrors Python's {@code _discover_inner_from_dir} and .NET's + * {@code DiscoverInnerFromDir}. + */ + static Map discoverInnerFromDir(ObjectNode outerSchema, Path schemaDir) throws IOException { + List aliases = new ArrayList<>(); + JsonNode config = outerSchema.get("config"); + if (config instanceof ObjectNode configObj) { + JsonNode cats = configObj.get("contentCategories"); + if (cats instanceof ObjectNode catsObj) { + Iterator it = catsObj.fieldNames(); + while (it.hasNext()) { + JsonNode entry = catsObj.get(it.next()); + if (!(entry instanceof ObjectNode entryObj)) { + continue; + } + JsonNode aliasNode = entryObj.get("analyzerId"); + if (aliasNode == null || !aliasNode.isTextual()) { + continue; + } + String alias = aliasNode.asText(); + if (alias.startsWith("prebuilt-")) { + continue; + } + aliases.add(alias); + } + } + } + + List jsonFiles; + try (var stream = Files.list(schemaDir)) { + jsonFiles = stream + .filter(p -> p.getFileName().toString().endsWith(".json")) + .toList(); + } + + Map resolved = new LinkedHashMap<>(); + List missing = new ArrayList<>(); + for (String alias : aliases) { + List matches = new ArrayList<>(); + String prefix = alias + "_"; + for (Path p : jsonFiles) { + String stem = stripExtension(p.getFileName().toString()); + if (stem.equals(alias) || stem.startsWith(prefix)) { + matches.add(p); + } + } + if (matches.isEmpty()) { + missing.add(alias); + continue; + } + // Highest version key wins: v10 > v9 > bare baseline. + final String aliasFinal = alias; + matches.sort(Comparator.comparing( + p -> versionSortKey(stripExtension(p.getFileName().toString()), aliasFinal))); + resolved.put(alias, matches.get(matches.size() - 1)); + } + + if (!missing.isEmpty()) { + System.err.println( + "--schema-dir could not resolve inner schemas for: " + missing + + ". Looked in " + schemaDir + " for files named .json or _*.json."); + return null; + } + return resolved; + } + + /** + * Sort key for {@code [_suffix]} filename stems. Higher tuple = newer. + * + *

    + *
  • Group 0: bare {@code } (no suffix) — oldest baseline.
    • + *
  • Group 1: numeric suffix {@code _v} or {@code _} + * — sorted by N as an integer, so v10 beats v9.
    • + *
  • Group 2: any other suffix {@code _} — lexicographic + * tiebreaker.
    • + *
+ * + *

Package-private so tests can pin the key extractor directly. + */ + static VersionKey versionSortKey(String stem, String alias) { + if (stem.equals(alias)) { + return new VersionKey(0, 0, ""); + } + // We only get here for stems that already matched the `_` filter, + // so the underscore is guaranteed to be at index alias.length(). + String suffix = stem.substring(alias.length() + 1); + Matcher m = VERSION_SUFFIX.matcher(suffix); + if (m.matches()) { + try { + return new VersionKey(1, Integer.parseInt(m.group(1)), ""); + } catch (NumberFormatException ignore) { + // Fall through to group 2 if the number overflows int. + } + } + return new VersionKey(2, 0, suffix); + } + + private static final Pattern VERSION_SUFFIX = Pattern.compile("^v?(\\d+)$"); + + /** Sort key tuple for {@link #versionSortKey}. Higher = newer. */ + record VersionKey(int group, int version, String lex) implements Comparable { + @Override + public int compareTo(VersionKey o) { + int c = Integer.compare(this.group, o.group); + if (c != 0) { + return c; + } + c = Integer.compare(this.version, o.version); + if (c != 0) { + return c; + } + return this.lex.compareTo(o.lex); + } + } + + /** + * Result of {@link #wireInnerIds}: the (possibly patched) outer schema + * plus any validation errors. If {@code errors} is empty the wiring + * succeeded; otherwise the caller should surface each error and abort. + */ + static final class WireResult { + final ObjectNode patched; + final List errors; + + WireResult(ObjectNode patched, List errors) { + this.patched = patched; + this.errors = errors; + } + } + + /** + * Replace each {@code contentCategories[].analyzerId} placeholder in the + * outer schema with the real inner analyzer ID from {@code aliasToId}. + * + *

Rules (parity with Python's {@code _patch_outer_analyzer_ids} and + * .NET's {@code WireInnerIds}): + *

    + *
  • Categories that omit {@code analyzerId} are left as-is (a + * classification-only "other" bucket).
    • + *
  • {@code analyzerId} values that start with {@code "prebuilt-"} are + * kept as-is — those are Azure-side prebuilt analyzers and don't + * need an {@code --inner-schema}.
    • + *
  • Any other value must match an alias in {@code aliasToId}; + * otherwise a validation error is returned.
    • + *
  • Every alias in {@code aliasToId} must be referenced by some + * category (typo check); otherwise an "unused" error is returned.
    • + *
+ */ + static WireResult wireInnerIds(ObjectNode outerSchema, Map aliasToId) { + List errors = new ArrayList<>(); + ObjectNode patched = outerSchema.deepCopy(); + JsonNode config = patched.get("config"); + if (!(config instanceof ObjectNode configObj)) { + return new WireResult(patched, errors); + } + JsonNode cats = configObj.get("contentCategories"); + if (!(cats instanceof ObjectNode catsObj)) { + return new WireResult(patched, errors); + } + + Set usedRealIds = new LinkedHashSet<>(); + List catNames = new ArrayList<>(); + catsObj.fieldNames().forEachRemaining(catNames::add); + for (String catName : catNames) { + JsonNode entry = catsObj.get(catName); + if (!(entry instanceof ObjectNode entryObj)) { + continue; + } + JsonNode aliasNode = entryObj.get("analyzerId"); + if (aliasNode == null || aliasNode.isNull()) { + continue; + } + String alias = aliasNode.asText(); + if (alias.startsWith("prebuilt-")) { + usedRealIds.add(alias); + continue; + } + String real = aliasToId.get(alias); + if (real == null) { + List known = new ArrayList<>(aliasToId.keySet()); + java.util.Collections.sort(known); + errors.add( + "category '" + catName + "' references analyzerId='" + alias + + "', but no --inner-schema entry matches alias '" + alias + + "'. Known aliases: " + known); + continue; + } + entryObj.put("analyzerId", real); + usedRealIds.add(real); + } + + // Catch unused inner schemas (cheap typo check). + for (Map.Entry e : aliasToId.entrySet()) { + if (!usedRealIds.contains(e.getValue())) { + errors.add( + "--inner-schema '" + e.getKey() + "' was supplied but no " + + "category in the outer schema routes to it"); + } + } + + return new WireResult(patched, errors); + } + + /** + * Like {@link CreateAndTestCommand#summarize}, but groups by the + * {@code category} field that classify-and-route stamps on each segment + * and uses per-category segment counts as the fill-rate denominator. + */ + static String summarizeRouted(List results) { + // category -> field -> list[(docName, value, confidence)] + Map>> table = new LinkedHashMap<>(); + Map segmentsPerCategory = new LinkedHashMap<>(); + for (CreateAndTestCommand.NamedDoc nd : results) { + // Count segments per category in this document. + Map segCounts = new LinkedHashMap<>(); + JsonNode contents = nd.doc.get("contents"); + if (contents != null && contents.isArray()) { + for (JsonNode c : contents) { + String cat = c.has("category") && c.get("category").isTextual() + ? c.get("category").asText() : ""; + segCounts.merge(cat, 1, Integer::sum); + } + } + for (Map.Entry sc : segCounts.entrySet()) { + segmentsPerCategory.merge(sc.getKey(), sc.getValue(), Integer::sum); + } + for (CreateAndTestCommand.FieldRecord f : CreateAndTestCommand.iterFields(nd.doc)) { + table + .computeIfAbsent(f.category, k -> new LinkedHashMap<>()) + .computeIfAbsent(f.fieldPath, k -> new ArrayList<>()) + .add(new RowEntry(nd.name, fieldValueRouted(f.fieldVal), fieldConfRouted(f.fieldVal))); + } + } + if (table.isEmpty()) { + return "[SUMMARY] (category-aware) no fields extracted across any document."; + } + StringBuilder sb = new StringBuilder(); + sb.append("\n").append("=".repeat(72)).append("\n"); + sb.append("[SUMMARY] (category-aware)\n"); + // Sort categories alphabetically for stable output. + List cats = new ArrayList<>(table.keySet()); + cats.sort(Comparator.naturalOrder()); + for (String cat : cats) { + String label = cat.isEmpty() ? "(uncategorized)" : cat; + int segCount = segmentsPerCategory.getOrDefault(cat, 0); + sb.append("\ncategory: ").append(label).append(" (") + .append(segCount).append(" segment").append(segCount == 1 ? "" : "s").append(")\n"); + String header = "category: " + label; + sb.append("-".repeat(header.length())).append("\n"); + sb.append(String.format(" %-30s %-10s %-10s%n", "field", "fill rate", "avg conf")); + // Sort fields alphabetically. + List>> fieldEntries = + new ArrayList<>(table.get(cat).entrySet()); + fieldEntries.sort(Map.Entry.comparingByKey()); + for (Map.Entry> field : fieldEntries) { + // Denominator is the per-category segment count so a field + // that's only meaningful in one category isn't penalised by + // other categories' segment counts, but a field missing from + // some segments in this category still reports the correct + // fraction. Mirrors Python's summarize_routed and .NET's + // SummarizeRouted. + List rows = field.getValue(); + int denom = segCount; + long filled = rows.stream().filter(r -> r.value != null).count(); + double fillRate = denom == 0 ? 0.0 : (double) filled / denom; + List confidences = new ArrayList<>(); + for (RowEntry r : field.getValue()) { + if (r.value != null && r.confidence != null) { + confidences.add(r.confidence); + } + } + String confStr = confidences.isEmpty() ? "n/a" + : String.format("%.3f", confidences.stream().mapToDouble(Double::doubleValue).average().orElse(0)); + sb.append(String.format( + " %-30s %-9s %s%n", + field.getKey(), + String.format("%.1f%%", fillRate * 100), confStr)); + } + } + // Lowest-confidence triples (category, field, doc) across all results. + List lows = new ArrayList<>(); + for (Map.Entry>> catEntry : table.entrySet()) { + for (Map.Entry> field : catEntry.getValue().entrySet()) { + for (RowEntry r : field.getValue()) { + if (r.confidence != null) { + lows.add(new LowConfRow(r.confidence, catEntry.getKey(), field.getKey(), r.docName)); + } + } + } + } + if (!lows.isEmpty()) { + lows.sort(Comparator.comparingDouble(x -> x.confidence)); + sb.append("\nlowest-confidence fields across all categories:\n"); + for (int i = 0; i < Math.min(3, lows.size()); i++) { + LowConfRow row = lows.get(i); + sb.append(String.format( + " %.3f [%s] %s (%s)%n", + row.confidence, row.category, row.field, row.docName)); + } + } + sb.append("=".repeat(72)); + return sb.toString(); + } + + private static Object fieldValueRouted(ObjectNode field) { + // Same logic as CreateAndTestCommand but private to keep the + // helper graph easy to reason about. + for (String key : new String[] { + "valueString", "valueNumber", "valueInteger", "valueBoolean", "valueDate", "valueTime" + }) { + JsonNode v = field.get(key); + if (v != null && !v.isNull() && !(v.isTextual() && v.asText().isEmpty())) { + return v; + } + } + JsonNode arr = field.get("valueArray"); + if (arr != null && arr.isArray() && arr.size() > 0) { + return arr; + } + JsonNode obj = field.get("valueObject"); + if (obj != null && obj.isObject() && obj.size() > 0) { + return obj; + } + return null; + } + + private static Double fieldConfRouted(ObjectNode field) { + JsonNode c = field.get("confidence"); + if (c != null && (c.isNumber() || c.canConvertToExactIntegral())) { + return c.asDouble(); + } + return null; + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + private static List enumerateInputs(Path inputPath) throws IOException { + List result = new ArrayList<>(); + if (Files.isRegularFile(inputPath)) { + result.add(inputPath); + return result; + } + try (var stream = Files.list(inputPath)) { + stream + .filter(Files::isRegularFile) + .filter(p -> { + String n = p.getFileName().toString().toLowerCase(Locale.ROOT); + int dot = n.lastIndexOf('.'); + return dot >= 0 && SUPPORTED_SUFFIXES.contains(n.substring(dot)); + }) + .sorted() + .forEach(result::add); + } + return result; + } + + private static String stripExtension(String name) { + int dot = name.lastIndexOf('.'); + return dot > 0 ? name.substring(0, dot) : name; + } + + private static String sha1Prefix(String s) { + try { + MessageDigest md = MessageDigest.getInstance("SHA-1"); + byte[] digest = md.digest(s.getBytes(StandardCharsets.UTF_8)); + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < 4; i++) { + sb.append(String.format("%02x", digest[i])); + } + return sb.toString(); + } catch (Exception ex) { + throw new IllegalStateException("sha1 failed: " + ex.getMessage(), ex); + } + } + + // ----------------------------------------------------------------------- + // CLI parsing + // ----------------------------------------------------------------------- + + private static Options parseArgs(String[] args) { + Options o = new Options(); + try { + for (int i = 0; i < args.length; i++) { + switch (args[i]) { + case "--outer-schema": + o.outerSchema = requireValue(args, i, "--outer-schema"); + i++; + break; + case "--inner-schema": + o.innerSchemas.add(requireValue(args, i, "--inner-schema")); + i++; + break; + case "--schema-dir": + o.schemaDir = requireValue(args, i, "--schema-dir"); + i++; + break; + case "--input": + o.input = requireValue(args, i, "--input"); + i++; + break; + case "--output": + o.output = requireValue(args, i, "--output"); + i++; + break; + case "--analyzer-id": + o.analyzerId = requireValue(args, i, "--analyzer-id"); + i++; + break; + case "--ephemeral": + o.ephemeral = true; + break; + case "--reuse": + o.reuse = true; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + System.err.println("unknown argument: " + args[i]); + printUsage(); + return null; + } + } + } catch (IllegalArgumentException ex) { + System.err.println(ex.getMessage()); + printUsage(); + return null; + } + if (o.outerSchema == null || o.input == null || o.output == null) { + System.err.println("--outer-schema, --input, and --output are required"); + printUsage(); + return null; + } + if (o.innerSchemas.isEmpty() && o.schemaDir == null) { + System.err.println("provide at least one --inner-schema ALIAS=PATH or --schema-dir DIR"); + printUsage(); + return null; + } + return o; + } + + /** + * Returns the value following a flag ({@code args[i+1]}), or throws + * {@link IllegalArgumentException} with a usage-friendly message if the flag + * is at the end of {@code args}. Callers must {@code i++} themselves after + * consuming the value so the outer loop skips past it. + */ + private static String requireValue(String[] args, int i, String flag) { + if (i + 1 >= args.length) { + throw new IllegalArgumentException("missing value for " + flag); + } + return args[i + 1]; + } + + private static void printUsage() { + System.err.println("Usage:"); + System.err.println(" cu-skill create-and-test-router"); + System.err.println(" --outer-schema "); + System.err.println(" (--inner-schema ALIAS=PATH [--inner-schema ...] | --schema-dir )"); + System.err.println(" --input "); + System.err.println(" --output "); + System.err.println(" [--analyzer-id ]"); + System.err.println(" [--ephemeral]"); + System.err.println(" [--reuse]"); + System.err.println(); + System.err.println("Classify-and-route Stage 2: validate, create N inner analyzers + 1"); + System.err.println("outer classifier, batch-test, print a category-aware summary."); + } + + static final class Options { + String outerSchema; + List innerSchemas = new ArrayList<>(); + String schemaDir; + String input; + String output; + String analyzerId = ""; + boolean ephemeral; + boolean reuse; + } + + private static final class RowEntry { + final String docName; + final Object value; + final Double confidence; + + RowEntry(String docName, Object value, Double confidence) { + this.docName = docName; + this.value = value; + this.confidence = confidence; + } + } + + private static final class LowConfRow { + final double confidence; + final String category; + final String field; + final String docName; + + LowConfRow(double confidence, String category, String field, String docName) { + this.confidence = confidence; + this.category = category; + this.field = field; + this.docName = docName; + } + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/ExtractLayoutCommand.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/ExtractLayoutCommand.java new file mode 100644 index 000000000000..e69789b51116 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/ExtractLayoutCommand.java @@ -0,0 +1,293 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Stage 1 of the analyzer-authoring loop: extract document layout into + * .layout.{json,md} files. Mirrors Python's extract_layout.py and the + * .NET ExtractLayoutCommand.cs. + * + * Defaults to analyzerId = prebuilt-documentSearch (richer markdown than + * prebuilt-document) so the layout output is useful as Copilot context. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.azure.core.credential.AzureKeyCredential; +import com.azure.core.credential.TokenCredential; +import com.azure.core.http.rest.RequestOptions; +import com.azure.core.util.BinaryData; +import com.azure.core.util.polling.SyncPoller; +import com.azure.ai.contentunderstanding.ContentUnderstandingClient; +import com.azure.ai.contentunderstanding.ContentUnderstandingClientBuilder; +import com.azure.identity.AzureCliCredentialBuilder; +import com.azure.identity.ChainedTokenCredentialBuilder; +import com.azure.identity.EnvironmentCredentialBuilder; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Locale; +import java.util.Set; + +final class ExtractLayoutCommand { + + private static final ObjectMapper MAPPER = new ObjectMapper() + .enable(SerializationFeature.INDENT_OUTPUT); + + private static final Set SUPPORTED_SUFFIXES = new LinkedHashSet<>( + Arrays.asList(".pdf", ".png", ".jpg", ".jpeg", ".tif", ".tiff", ".bmp", ".heif", ".heic")); + + private ExtractLayoutCommand() { + } + + static int run(String[] args) throws IOException { + Options opts = parseArgs(args); + if (opts == null) { + return 2; + } + + Path input = Paths.get(opts.input); + Path output = Paths.get(opts.output); + if (!Files.exists(input)) { + System.err.println("input not found: " + input); + return 2; + } + Files.createDirectories(output); + + List files = enumerateInputs(input); + if (files.isEmpty()) { + System.err.println("no supported documents found under " + input); + return 2; + } + + ContentUnderstandingClient client = buildClient(); + int ok = 0; + int fail = 0; + for (Path file : files) { + String stem = stripExtension(file.getFileName().toString()); + try { + System.out.println("[RUN ] " + file + " -> " + output + "/" + stem + ".layout.{json,md}"); + byte[] bytes = Files.readAllBytes(file); + String contentType = guessContentType(file); + + SyncPoller poller = client.beginAnalyzeBinary( + opts.analyzerId, + contentType, + BinaryData.fromBytes(bytes), + new RequestOptions()); + // Unwrap the LRO envelope `{id, status, result, usage}` so the + // on-disk shape matches the Python skill output (just the + // analysis result, like `poller.result()` in Python). + JsonNode envelope = MAPPER.readTree(poller.getFinalResult().toString()); + JsonNode payload = envelope.has("result") ? envelope.get("result") : envelope; + + Files.writeString( + output.resolve(stem + ".layout.json"), + MAPPER.writerWithDefaultPrettyPrinter().writeValueAsString(payload)); + + String markdown = extractMarkdown(payload); + Files.writeString(output.resolve(stem + ".layout.md"), markdown); + ok++; + } catch (Exception ex) { + System.err.println("[FAIL] " + file + ": " + ex.getMessage()); + fail++; + } + } + + System.out.println(); + System.out.println("[DONE] " + ok + " ok, " + fail + " failed; output -> " + output); + return fail == 0 ? 0 : 1; + } + + private static String extractMarkdown(JsonNode payload) { + JsonNode contents = payload.get("contents"); + if (contents != null && contents.isArray()) { + for (JsonNode c : contents) { + JsonNode md = c.get("markdown"); + if (md != null && md.isTextual() && !md.asText().isEmpty()) { + return md.asText(); + } + } + } + return ""; + } + + private static String guessContentType(Path file) { + String name = file.getFileName().toString().toLowerCase(Locale.ROOT); + if (name.endsWith(".pdf")) { + return "application/pdf"; + } + if (name.endsWith(".png")) { + return "image/png"; + } + if (name.endsWith(".jpg") || name.endsWith(".jpeg")) { + return "image/jpeg"; + } + if (name.endsWith(".tif") || name.endsWith(".tiff")) { + return "image/tiff"; + } + if (name.endsWith(".bmp")) { + return "image/bmp"; + } + if (name.endsWith(".heif") || name.endsWith(".heic")) { + return "image/heif"; + } + return "application/octet-stream"; + } + + private static List enumerateInputs(Path inputPath) throws IOException { + List result = new ArrayList<>(); + if (Files.isRegularFile(inputPath)) { + result.add(inputPath); + return result; + } + try (var stream = Files.list(inputPath)) { + stream + .filter(Files::isRegularFile) + .filter(p -> { + String n = p.getFileName().toString().toLowerCase(Locale.ROOT); + int dot = n.lastIndexOf('.'); + return dot >= 0 && SUPPORTED_SUFFIXES.contains(n.substring(dot)); + }) + .sorted() + .forEach(result::add); + } + return result; + } + + private static String stripExtension(String name) { + int dot = name.lastIndexOf('.'); + return dot > 0 ? name.substring(0, dot) : name; + } + + static ContentUnderstandingClient buildClient() { + String endpoint = readEnv("CONTENTUNDERSTANDING_ENDPOINT"); + if (endpoint == null || endpoint.isBlank()) { + throw new IllegalStateException( + "CONTENTUNDERSTANDING_ENDPOINT is not set. Configure your .env file (see cu-sdk-setup)."); + } + // Strip trailing slash to match the convention from samples — avoids + // double-slash URLs when the env var was copy-pasted from the portal. + while (endpoint.endsWith("/")) { + endpoint = endpoint.substring(0, endpoint.length() - 1); + } + + ContentUnderstandingClientBuilder builder = new ContentUnderstandingClientBuilder() + .endpoint(endpoint); + + String key = readEnv("CONTENTUNDERSTANDING_KEY"); + if (key != null && !key.isBlank()) { + builder.credential(new AzureKeyCredential(key)); + } else { + // DefaultAzureCredentialBuilder in the Java SDK does not expose + // an exclude API (unlike .NET), and its ManagedIdentity probe + // blocks for ~30s on dev boxes (WSL, laptops) before timing out. + // Build a focused chain: Environment first (CI), then Azure CLI + // (dev boxes). This covers both worlds without the metadata-service stall. + TokenCredential cred = new ChainedTokenCredentialBuilder() + .addLast(new EnvironmentCredentialBuilder().build()) + .addLast(new AzureCliCredentialBuilder().build()) + .build(); + builder.credential(cred); + } + return builder.buildClient(); + } + + /** + * Reads an env var and strips one layer of surrounding single or double + * quotes — `python-dotenv` strips them by default but raw `export` from a + * shell does not, so users who source .env manually still get a clean value. + */ + static String readEnv(String name) { + String value = System.getenv(name); + if (value == null || value.isEmpty()) { + return value; + } + if (value.length() >= 2 + && ((value.charAt(0) == '"' && value.charAt(value.length() - 1) == '"') + || (value.charAt(0) == '\'' && value.charAt(value.length() - 1) == '\''))) { + return value.substring(1, value.length() - 1); + } + return value; + } + + private static Options parseArgs(String[] args) { + String input = null; + String output = null; + String analyzerId = "prebuilt-documentSearch"; + try { + for (int i = 0; i < args.length; i++) { + switch (args[i]) { + case "--input": + input = requireValue(args, i, "--input"); + i++; + break; + case "--output": + output = requireValue(args, i, "--output"); + i++; + break; + case "--analyzer-id": + analyzerId = requireValue(args, i, "--analyzer-id"); + i++; + break; + case "-h": + case "--help": + printUsage(); + return null; + default: + System.err.println("unknown argument: " + args[i]); + printUsage(); + return null; + } + } + } catch (IllegalArgumentException ex) { + System.err.println(ex.getMessage()); + printUsage(); + return null; + } + if (input == null || output == null) { + System.err.println("--input and --output are required"); + printUsage(); + return null; + } + Options o = new Options(); + o.input = input; + o.output = output; + o.analyzerId = analyzerId; + return o; + } + + /** + * Returns the value following a flag ({@code args[i+1]}), or throws + * {@link IllegalArgumentException} with a usage-friendly message if the flag + * is at the end of {@code args}. Callers must {@code i++} themselves after + * consuming the value so the outer loop skips past it. + */ + private static String requireValue(String[] args, int i, String flag) { + if (i + 1 >= args.length) { + throw new IllegalArgumentException("missing value for " + flag); + } + return args[i + 1]; + } + + private static void printUsage() { + System.err.println("Usage:"); + System.err.println(" cu-skill extract-layout --input --output [--analyzer-id ]"); + System.err.println(); + System.err.println("Stage 1: extract layout JSON + markdown for each input document."); + } + + private static final class Options { + String input; + String output; + String analyzerId; + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java new file mode 100644 index 000000000000..d55a52bce7f5 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java @@ -0,0 +1,335 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Pure-Java validator for Content Understanding analyzer schema JSON. + * + * Catches structural mistakes (missing keys, unknown baseAnalyzerId values, + * malformed contentCategories routes) BEFORE any call to the Content + * Understanding service. Failing fast here gives users an actionable error + * message and avoids a wasted service round-trip. + * + * Design rules (see README.md in this directory): + * * No com.azure.* imports — pure Jackson. + * * No network calls. + * * Self-contained — drop-in for any tool or test. + * + * Public surface: + * * SchemaValidator.validate(JsonNode) — validate a parsed schema node + * * SchemaValidator.validateFile(Path) — convenience wrapper that loads a JSON file + * * SchemaValidator.validateString(String) — validate a raw JSON string + * * SchemaValidator.KNOWN_BASE_ANALYZER_IDS — allow-list of baseAnalyzerId values + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Iterator; +import java.util.LinkedHashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeSet; + +/** + * Pure-Java validator for Content Understanding analyzer schemas. + */ +public final class SchemaValidator { + + /** + * Valid {@code baseAnalyzerId} values for custom analyzers. Only modality-level + * prebuilt analyzers are accepted by the service for {@code baseAnalyzerId}; {@code *Search} + * variants and task-specific prebuilt analyzers ({@code prebuilt-invoice}, {@code prebuilt-receipt}) + * return {@code InvalidBaseAnalyzerId} if used here. See + * + * the analyzer reference docs. + */ + public static final Set KNOWN_BASE_ANALYZER_IDS = Collections.unmodifiableSet( + new LinkedHashSet<>(Arrays.asList( + "prebuilt-document", + "prebuilt-audio", + "prebuilt-video", + "prebuilt-image"))); + + private static final Set ALLOWED_FIELD_TYPES = Collections.unmodifiableSet( + new LinkedHashSet<>(Arrays.asList( + "string", "number", "integer", "boolean", "date", "time", "array", "object"))); + + private static final Set ALLOWED_FIELD_METHODS = Collections.unmodifiableSet( + new LinkedHashSet<>(Arrays.asList( + "extract", "generate", "classify"))); + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + /** Validation result: {@link #ok} is true when the schema is structurally valid. */ + public static final class Result { + private final boolean ok; + private final List errors; + + Result(boolean ok, List errors) { + this.ok = ok; + this.errors = Collections.unmodifiableList(errors); + } + + public boolean isOk() { + return ok; + } + + public List getErrors() { + return errors; + } + } + + private SchemaValidator() { + // utility class + } + + /** Validate a parsed analyzer schema node. */ + public static Result validate(JsonNode schema) { + List errors = new ArrayList<>(); + + if (schema == null || !schema.isObject()) { + errors.add("schema must be a JSON object at the top level"); + return new Result(false, errors); + } + + // baseAnalyzerId + JsonNode baseEl = schema.get("baseAnalyzerId"); + if (baseEl == null || baseEl.isNull()) { + errors.add("missing required key: baseAnalyzerId"); + } else if (!baseEl.isTextual()) { + errors.add("baseAnalyzerId must be a string"); + } else { + String baseValue = baseEl.asText(); + if (!KNOWN_BASE_ANALYZER_IDS.contains(baseValue)) { + String known = String.join(", ", new TreeSet<>(KNOWN_BASE_ANALYZER_IDS) + .stream().map(s -> "'" + s + "'").toList()); + errors.add("unknown baseAnalyzerId: '" + baseValue + "'. Known values: [" + known + "]"); + } + } + + // config (optional, but if present must be an object) + JsonNode config = schema.get("config"); + if (config != null && !config.isNull()) { + if (!config.isObject()) { + errors.add("config, if present, must be an object"); + // Bail out — without a well-typed config we can't tell whether + // this is a single-type or classify-and-route schema, and + // falling through would emit a confusing cascade of + // "missing fieldSchema" errors rooted in the same problem. + return new Result(false, errors); + } + } + + boolean isClassifyRoute = config != null + && config.isObject() + && config.has("contentCategories"); + + if (isClassifyRoute) { + errors.addAll(validateClassifyRoute(config)); + if (schema.has("fieldSchema")) { + errors.add("classify-and-route schemas should not declare fieldSchema at " + + "the top level; field extraction belongs in inner analyzers"); + } + } else { + errors.addAll(validateSingleType(schema)); + } + + return new Result(errors.isEmpty(), errors); + } + + /** Validate a schema stored in a JSON file. */ + public static Result validateFile(Path path) { + if (path == null || !Files.exists(path)) { + return new Result(false, Collections.singletonList( + "schema file not found: " + (path == null ? "" : path.toString()))); + } + String text; + try { + text = Files.readString(path); + } catch (IOException ex) { + return new Result(false, Collections.singletonList( + "failed to read schema file " + path + ": " + ex.getMessage())); + } + return validateString(text, path.toString()); + } + + /** Validate a raw JSON string. */ + public static Result validateString(String json) { + return validateString(json, ""); + } + + private static Result validateString(String json, String sourceLabel) { + try { + return validate(MAPPER.readTree(json)); + } catch (JsonProcessingException ex) { + return new Result(false, Collections.singletonList( + "schema file is not valid JSON (" + sourceLabel + "): " + ex.getOriginalMessage())); + } + } + + // ----------------------------------------------------------------------- + // Internal helpers + // ----------------------------------------------------------------------- + + private static List validateSingleType(JsonNode schema) { + List errors = new ArrayList<>(); + JsonNode fieldSchema = schema.get("fieldSchema"); + if (fieldSchema == null || fieldSchema.isNull()) { + errors.add("missing required key: fieldSchema " + + "(single-type schemas must declare fields to extract)"); + return errors; + } + if (!fieldSchema.isObject()) { + errors.add("fieldSchema must be an object"); + return errors; + } + JsonNode fields = fieldSchema.get("fields"); + if (fields == null || fields.isNull()) { + errors.add("fieldSchema.fields is required"); + return errors; + } + if (!fields.isObject()) { + errors.add("fieldSchema.fields must be an object mapping field names to definitions"); + return errors; + } + boolean any = false; + Iterator> it = fields.fields(); + while (it.hasNext()) { + Map.Entry entry = it.next(); + any = true; + errors.addAll(validateFieldDefinition(entry.getKey(), entry.getValue(), null)); + } + if (!any) { + errors.add("fieldSchema.fields must declare at least one field"); + } + return errors; + } + + private static List validateFieldDefinition(String name, JsonNode definition, String path) { + List errors = new ArrayList<>(); + String prefix = path != null ? path : "fieldSchema.fields['" + name + "']"; + + if (!definition.isObject()) { + errors.add(prefix + " must be an object"); + return errors; + } + + String fieldType = null; + JsonNode typeEl = definition.get("type"); + if (typeEl == null || typeEl.isNull()) { + errors.add(prefix + ".type is required"); + } else if (!typeEl.isTextual()) { + errors.add(prefix + ".type must be a string"); + } else { + fieldType = typeEl.asText(); + if (!ALLOWED_FIELD_TYPES.contains(fieldType)) { + String allowed = String.join(", ", new TreeSet<>(ALLOWED_FIELD_TYPES) + .stream().map(s -> "'" + s + "'").toList()); + errors.add(prefix + ".type '" + fieldType + "' is not one of [" + allowed + "]"); + } + } + + JsonNode methodEl = definition.get("method"); + if (methodEl != null && !methodEl.isNull()) { + if (!methodEl.isTextual()) { + errors.add(prefix + ".method must be a string"); + } else { + String method = methodEl.asText(); + if (!ALLOWED_FIELD_METHODS.contains(method)) { + String allowed = String.join(", ", new TreeSet<>(ALLOWED_FIELD_METHODS) + .stream().map(s -> "'" + s + "'").toList()); + errors.add(prefix + ".method '" + method + "' is not one of [" + allowed + "]"); + } + } + } + + JsonNode descEl = definition.get("description"); + if (descEl != null && !descEl.isNull() && !descEl.isTextual()) { + errors.add(prefix + ".description must be a string"); + } + + if ("object".equals(fieldType)) { + JsonNode propsEl = definition.get("properties"); + if (propsEl != null && !propsEl.isNull()) { + if (!propsEl.isObject()) { + errors.add(prefix + ".properties must be an object"); + } else { + Iterator> it = propsEl.fields(); + while (it.hasNext()) { + Map.Entry child = it.next(); + errors.addAll(validateFieldDefinition( + child.getKey(), child.getValue(), + prefix + ".properties['" + child.getKey() + "']")); + } + } + } + } else if ("array".equals(fieldType)) { + JsonNode itemsEl = definition.get("items"); + if (itemsEl != null && !itemsEl.isNull()) { + if (!itemsEl.isObject()) { + errors.add(prefix + ".items must be an object"); + } else { + errors.addAll(validateFieldDefinition("items", itemsEl, prefix + ".items")); + } + } + } + + return errors; + } + + private static List validateClassifyRoute(JsonNode config) { + List errors = new ArrayList<>(); + + JsonNode enableEl = config.get("enableSegment"); + boolean hasEnableSegment = enableEl != null && enableEl.isBoolean() && enableEl.asBoolean(); + if (!hasEnableSegment) { + errors.add("classify-and-route schemas must set config.enableSegment = true"); + } + + JsonNode categories = config.get("contentCategories"); + if (categories == null || !categories.isObject()) { + errors.add("config.contentCategories must be an object"); + return errors; + } + + boolean any = false; + Iterator> it = categories.fields(); + while (it.hasNext()) { + Map.Entry cat = it.next(); + any = true; + String prefix = "config.contentCategories['" + cat.getKey() + "']"; + JsonNode entry = cat.getValue(); + + if (!entry.isObject()) { + errors.add(prefix + " must be an object"); + continue; + } + + JsonNode descEl = entry.get("description"); + if (descEl == null || !descEl.isTextual() || descEl.asText().isBlank()) { + errors.add(prefix + ".description is required and must be a non-empty string"); + } + + JsonNode analyzerIdEl = entry.get("analyzerId"); + if (analyzerIdEl != null && !analyzerIdEl.isNull() && !analyzerIdEl.isTextual()) { + errors.add(prefix + ".analyzerId, if present, must be a string"); + } + } + + if (!any) { + errors.add("config.contentCategories must declare at least one category"); + } + + return errors; + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommandTest.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommandTest.java new file mode 100644 index 000000000000..dca41e571135 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommandTest.java @@ -0,0 +1,112 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Unit tests for the pure helpers in CreateAndTestCommand. Mirrors Python's + * tests/test_skills_create_and_test.py. The CLI entry point (run) and the + * network-dependent helpers are not covered here — they require an Azure + * client and live recording infrastructure. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class CreateAndTestCommandTest { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private static ObjectNode scalar(String value, double conf) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("type", "string"); + n.put("valueString", value); + n.put("confidence", conf); + return n; + } + + private static ObjectNode number(double value, double conf) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("type", "number"); + n.put("valueNumber", value); + n.put("confidence", conf); + return n; + } + + private static ObjectNode arrayOfObjects(List items) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("type", "array"); + n.putArray("valueArray").addAll(items); + return n; + } + + private static ObjectNode objectField(ObjectNode inner) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("type", "object"); + n.set("valueObject", inner); + return n; + } + + @Test + void summarizeFlattensNestedArrayAndObjectFieldsToLeafRows() { + ObjectNode lineItem1 = MAPPER.createObjectNode(); + lineItem1.set("itemCode", scalar("A123", 0.80)); + lineItem1.set("amount", number(60.0, 0.92)); + + ObjectNode lineItem2 = MAPPER.createObjectNode(); + lineItem2.set("itemCode", scalar("B456", 0.70)); + lineItem2.set("amount", number(30.0, 0.90)); + + ObjectNode addressInner = MAPPER.createObjectNode(); + addressInner.set("street", scalar("123 Main St", 0.88)); + + ObjectNode fields = MAPPER.createObjectNode(); + fields.set("invoiceNumber", scalar("INV-100", 0.95)); + fields.set("lineItems", arrayOfObjects(List.of( + objectField(lineItem1), + objectField(lineItem2)))); + fields.set("address", objectField(addressInner)); + + ObjectNode content = MAPPER.createObjectNode(); + content.set("fields", fields); + + ObjectNode doc = MAPPER.createObjectNode(); + doc.putArray("contents").add(content); + + List results = new ArrayList<>(); + results.add(new CreateAndTestCommand.NamedDoc("docX", doc)); + + String out = CreateAndTestCommand.summarize(results); + + // Leaf rows present. + assertTrue(out.contains("lineItems[].itemCode"), "leaf row lineItems[].itemCode missing"); + assertTrue(out.contains("lineItems[].amount"), "leaf row lineItems[].amount missing"); + assertTrue(out.contains("address.street"), "leaf row address.street missing"); + assertTrue(out.contains("invoiceNumber"), "leaf row invoiceNumber missing"); + + // The old aggregate-only behaviour would emit a bare `lineItems` or + // `address` row with `n/a` confidence and no children. The new + // behaviour must not emit those. + for (String line : out.split("\n")) { + String stripped = line.trim(); + assertFalse(stripped.startsWith("lineItems ") || stripped.startsWith("address "), + "aggregate-only row leaked: " + line); + } + + // Lowest-confidence list should surface the 0.700 leaf. + assertTrue(out.contains("0.700"), "missing 0.700 confidence row"); + int lowIdx = out.indexOf("lowest-confidence"); + assertTrue(lowIdx >= 0, "no 'lowest-confidence' section found"); + String lowSection = out.substring(lowIdx); + assertTrue(lowSection.contains("lineItems[].itemCode"), + "lowest-confidence section should mention lineItems[].itemCode"); + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommandTest.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommandTest.java new file mode 100644 index 000000000000..b51d0aa6f0a9 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommandTest.java @@ -0,0 +1,497 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Unit tests for pure helpers in CreateAndTestRouterCommand. Mirrors the + * portion of Python's tests/test_skills_classify_route_router.py that does + * not require mocking the Azure client. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.databind.node.ObjectNode; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class CreateAndTestRouterCommandTest { + + private static final ObjectMapper MAPPER = new ObjectMapper(); + + private static ObjectNode field(String value, double confidence) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("valueString", value); + n.put("confidence", confidence); + return n; + } + + private static ObjectNode segment(String category, Map fields) { + ObjectNode n = MAPPER.createObjectNode(); + n.put("category", category); + ObjectNode fieldsObj = n.putObject("fields"); + for (Map.Entry e : fields.entrySet()) { + fieldsObj.set(e.getKey(), e.getValue()); + } + return n; + } + + private static ObjectNode docWithSegments(List segments) { + ObjectNode doc = MAPPER.createObjectNode(); + ArrayNode contents = doc.putArray("contents"); + for (ObjectNode s : segments) { + contents.add(s); + } + return doc; + } + + @Test + void summarizeRoutedUsesPerCategoryDenominator() { + // Three invoice segments (all filled) must report 100%, not be + // diluted by other categories' segments. + ObjectNode doc = docWithSegments(List.of( + segment("invoice", Map.of("InvoiceNumber", field("INV-1", 0.9))), + segment("invoice", Map.of("InvoiceNumber", field("INV-2", 0.91))), + segment("invoice", Map.of("InvoiceNumber", field("INV-3", 0.92))), + segment("bank_statement", Map.of("AccountNumber", field("12345", 0.8))))); + + List results = new ArrayList<>(); + results.add(new CreateAndTestCommand.NamedDoc("packet_a", doc)); + + String text = CreateAndTestRouterCommand.summarizeRouted(results); + + // Invoice: 3 segments, 3 filled → 100% + assertTrue(text.contains("category: invoice (3 segments)"), + "invoice segment count wrong: " + text); + assertTrue(text.contains("InvoiceNumber") && text.contains("100.0%"), + "invoice not at 100%: " + text); + // Bank statement: 1 segment, 1 filled → 100% + assertTrue( + text.contains("category: bank_statement (1 segment)") || + text.contains("category: bank_statement (1 segments)"), + "bank_statement segment count wrong: " + text); + // Packet-wide denominator must NOT leak through. + assertFalse(text.contains("33.3%"), "33.3% leaked: " + text); + assertFalse(text.contains("25.0%"), "25.0% leaked: " + text); + } + + @Test + void summarizeRoutedReportsZeroFillForMissingFieldInSomeSegments() { + // Two invoice segments, only one has TotalAmount → 50% fill. + Map segWithBoth = new HashMap<>(); + segWithBoth.put("InvoiceNumber", field("INV-1", 0.9)); + segWithBoth.put("TotalAmount", field("$100", 0.7)); + + ObjectNode doc = docWithSegments(List.of( + segment("invoice", segWithBoth), + segment("invoice", Map.of("InvoiceNumber", field("INV-2", 0.91))))); + + List results = new ArrayList<>(); + results.add(new CreateAndTestCommand.NamedDoc("packet", doc)); + + String text = CreateAndTestRouterCommand.summarizeRouted(results); + assertTrue(text.contains("category: invoice (2 segments)"), + "segment count wrong: " + text); + // InvoiceNumber appears in both segments → 100% + assertTrue(text.contains("InvoiceNumber") && text.contains("100.0%"), + "InvoiceNumber not 100%: " + text); + // TotalAmount appears in 1 of 2 → 50% + assertTrue(text.contains("TotalAmount") && text.contains(" 50.0%"), + "TotalAmount not 50%: " + text); + } + + @Test + void wireInnerIdsSubstitutesMatchingAliases() { + // Category name is deliberately different from the analyzerId value + // to catch the "keyed off cat name instead of alias" regression. + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + ObjectNode invCat = cats.putObject("invoice_bucket"); + invCat.put("description", "d"); + invCat.put("analyzerId", "invoice"); + ObjectNode loanCat = cats.putObject("loan_bucket"); + loanCat.put("description", "d"); + loanCat.put("analyzerId", "loan_application"); + + Map aliasToId = new LinkedHashMap<>(); + aliasToId.put("invoice", "real-invoice-id"); + aliasToId.put("loan_application", "real-loan-id"); + + CreateAndTestRouterCommand.WireResult wired = + CreateAndTestRouterCommand.wireInnerIds(outer, aliasToId); + + assertTrue(wired.errors.isEmpty(), "expected no errors, got: " + wired.errors); + ObjectNode patchedCats = (ObjectNode) wired.patched.get("config").get("contentCategories"); + assertEquals("real-invoice-id", + patchedCats.get("invoice_bucket").get("analyzerId").asText(), + "invoice_bucket should be patched to the aliasToId value for 'invoice'"); + assertEquals("real-loan-id", + patchedCats.get("loan_bucket").get("analyzerId").asText(), + "loan_bucket should be patched to the aliasToId value for 'loan_application'"); + // Input must not be mutated. + assertEquals("invoice", + outer.get("config").get("contentCategories").get("invoice_bucket").get("analyzerId").asText(), + "wireInnerIds mutated its input"); + } + + @Test + void wireInnerIdsKeepsPrebuiltAnalyzerIdsAsIs() { + // analyzerId values starting with "prebuilt-" resolve to Azure + // service prebuilts and must NOT require a matching --inner-schema. + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + ObjectNode invCat = cats.putObject("invoice"); + invCat.put("description", "d"); + invCat.put("analyzerId", "prebuilt-invoice"); + ObjectNode receiptCat = cats.putObject("receipt"); + receiptCat.put("description", "d"); + receiptCat.put("analyzerId", "prebuilt-receipt"); + + // No --inner-schema needed; both categories use service prebuilts. + Map aliasToId = new LinkedHashMap<>(); + + CreateAndTestRouterCommand.WireResult wired = + CreateAndTestRouterCommand.wireInnerIds(outer, aliasToId); + + assertTrue(wired.errors.isEmpty(), "prebuilt-* values must not need aliases; got: " + wired.errors); + ObjectNode patchedCats = (ObjectNode) wired.patched.get("config").get("contentCategories"); + assertEquals("prebuilt-invoice", + patchedCats.get("invoice").get("analyzerId").asText(), + "prebuilt-invoice must be preserved as-is"); + assertEquals("prebuilt-receipt", + patchedCats.get("receipt").get("analyzerId").asText(), + "prebuilt-receipt must be preserved as-is"); + } + + @Test + void wireInnerIdsReportsMissingAliasAsError() { + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + ObjectNode invCat = cats.putObject("invoice"); + invCat.put("description", "d"); + invCat.put("analyzerId", "invoice"); + ObjectNode loanCat = cats.putObject("loan"); + loanCat.put("description", "d"); + loanCat.put("analyzerId", "loan_application"); + + Map aliasToId = new LinkedHashMap<>(); + aliasToId.put("invoice", "real-invoice-id"); + // "loan_application" alias intentionally missing. + + CreateAndTestRouterCommand.WireResult wired = + CreateAndTestRouterCommand.wireInnerIds(outer, aliasToId); + + assertFalse(wired.errors.isEmpty(), "expected an error for missing alias"); + assertTrue( + wired.errors.stream().anyMatch(e -> e.contains("loan_application") && e.contains("loan")), + "expected error to name the missing alias and category, got: " + wired.errors); + } + + @Test + void wireInnerIdsReportsUnusedAliasAsError() { + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + ObjectNode invCat = cats.putObject("invoice"); + invCat.put("description", "d"); + invCat.put("analyzerId", "invoice"); + + Map aliasToId = new LinkedHashMap<>(); + aliasToId.put("invoice", "real-invoice-id"); + // "extra" is supplied but not referenced by any category — likely a typo. + aliasToId.put("extra", "real-extra-id"); + + CreateAndTestRouterCommand.WireResult wired = + CreateAndTestRouterCommand.wireInnerIds(outer, aliasToId); + + assertFalse(wired.errors.isEmpty(), "expected an error for unused alias"); + assertTrue( + wired.errors.stream().anyMatch(e -> e.contains("extra") && e.contains("no category")), + "expected error to name the unused alias, got: " + wired.errors); + } + + @Test + void wireInnerIdsAllowsCategoriesWithoutAnalyzerId() { + // Classification-only "other" bucket must not cause a wire error. + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + ObjectNode invCat = cats.putObject("invoice"); + invCat.put("description", "d"); + invCat.put("analyzerId", "invoice"); + ObjectNode otherCat = cats.putObject("other"); + otherCat.put("description", "catch-all classification bucket"); + // no analyzerId + + Map aliasToId = new LinkedHashMap<>(); + aliasToId.put("invoice", "real-invoice-id"); + + CreateAndTestRouterCommand.WireResult wired = + CreateAndTestRouterCommand.wireInnerIds(outer, aliasToId); + + assertTrue(wired.errors.isEmpty(), + "categories without analyzerId must be allowed; got: " + wired.errors); + ObjectNode patchedCats = (ObjectNode) wired.patched.get("config").get("contentCategories"); + assertFalse(patchedCats.get("other").has("analyzerId"), + "'other' bucket must remain analyzerId-less"); + } + + // ------------------------------------------------------------------- + // versionSortKey — pure key extractor + // ------------------------------------------------------------------- + + @Test + void versionSortKey_BareAlias_ReturnsGroupZero() { + CreateAndTestRouterCommand.VersionKey key + = CreateAndTestRouterCommand.versionSortKey("invoice", "invoice"); + assertEquals(0, key.group()); + assertEquals(0, key.version()); + } + + @Test + void versionSortKey_VPrefixedNumeric_ReturnsGroupOneWithVersion() { + CreateAndTestRouterCommand.VersionKey v9 + = CreateAndTestRouterCommand.versionSortKey("invoice_v9", "invoice"); + CreateAndTestRouterCommand.VersionKey v10 + = CreateAndTestRouterCommand.versionSortKey("invoice_v10", "invoice"); + assertEquals(1, v9.group()); + assertEquals(9, v9.version()); + assertEquals(1, v10.group()); + assertEquals(10, v10.version()); + // The whole point of the fix. + assertTrue(v10.compareTo(v9) > 0, "v10 must sort higher than v9"); + } + + @Test + void versionSortKey_BareNumeric_ReturnsGroupOneWithVersion() { + CreateAndTestRouterCommand.VersionKey key + = CreateAndTestRouterCommand.versionSortKey("invoice_42", "invoice"); + assertEquals(1, key.group()); + assertEquals(42, key.version()); + } + + @Test + void versionSortKey_NonNumericSuffix_ReturnsGroupTwoWithSuffix() { + CreateAndTestRouterCommand.VersionKey key + = CreateAndTestRouterCommand.versionSortKey("invoice_draft", "invoice"); + assertEquals(2, key.group()); + assertEquals(0, key.version()); + assertEquals("draft", key.lex()); + } + + // ------------------------------------------------------------------- + // discoverInnerFromDir — end-to-end filesystem-touching resolution + // ------------------------------------------------------------------- + + private static Path makeTempDir() throws IOException { + return Files.createTempDirectory("cu-skill-discover-"); + } + + private static void writeEmptyJson(Path dir, String name) throws IOException { + Files.writeString(dir.resolve(name), "{}"); + } + + private static void deleteRecursive(Path dir) throws IOException { + if (!Files.exists(dir)) { + return; + } + try (Stream paths = Files.walk(dir)) { + paths.sorted(Comparator.reverseOrder()).forEach(p -> { + try { + Files.deleteIfExists(p); + } catch (IOException ignore) { + // best-effort test cleanup + } + }); + } + } + + private static ObjectNode outerWithAliases(String... aliases) { + ObjectNode outer = MAPPER.createObjectNode(); + outer.put("baseAnalyzerId", "prebuilt-document"); + ObjectNode config = outer.putObject("config"); + config.put("enableSegment", true); + ObjectNode cats = config.putObject("contentCategories"); + for (int i = 0; i < aliases.length; i++) { + ObjectNode entry = cats.putObject("cat_" + i); + entry.put("description", "d"); + if (aliases[i] != null) { + entry.put("analyzerId", aliases[i]); + } + } + return outer; + } + + @Test + void discoverInnerFromDir_ResolvesExactMatchStem() throws IOException { + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + writeEmptyJson(dir, "bank_statement.json"); + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice", "bank_statement"), dir); + + assertNotNull(resolved); + assertEquals(2, resolved.size()); + assertEquals(dir.resolve("invoice.json"), resolved.get("invoice")); + assertEquals(dir.resolve("bank_statement.json"), resolved.get("bank_statement")); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_PicksNaturalVersionMaxNotAlphabeticalLast() throws IOException { + // Regression: the previous implementation `.sorted()` alphabetically + // and took the last element as "newest". But '1' < '9' char-by-char, + // so `invoice_v10.json` sorted BEFORE `invoice_v9.json` — "alphabetical + // last" then picked v9, silently loading the older schema. Copilot + // flagged this on the .NET PR (#60394); the natural version sort fix + // brings all four languages back in lockstep. + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice_v1.json"); + writeEmptyJson(dir, "invoice_v2.json"); + writeEmptyJson(dir, "invoice_v9.json"); + writeEmptyJson(dir, "invoice_v10.json"); + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice"), dir); + + assertNotNull(resolved); + assertEquals(dir.resolve("invoice_v10.json"), resolved.get("invoice"), + "v10 must beat v9 (natural version order, not alphabetical)"); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_PrefersVersionedOverBareAlias() throws IOException { + // Bare `.json` is group 0 (baseline); versioned files are + // group 1 (numeric) or group 2 (other suffix). Any versioned file + // beats the bare baseline as "newer". + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + writeEmptyJson(dir, "invoice_v1.json"); + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice"), dir); + + assertNotNull(resolved); + assertEquals(dir.resolve("invoice_v1.json"), resolved.get("invoice")); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_SkipsPrebuiltAliases() throws IOException { + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + // No prebuilt-invoice.json on disk — it's a service alias. + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice", "prebuilt-invoice"), dir); + + assertNotNull(resolved); + assertEquals(1, resolved.size()); + assertTrue(resolved.containsKey("invoice")); + assertFalse(resolved.containsKey("prebuilt-invoice")); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_SkipsCategoriesWithoutAnalyzerId() throws IOException { + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + + // Second element null → classification-only bucket, no schema needed. + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice", null), dir); + + assertNotNull(resolved); + assertEquals(1, resolved.size()); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_MissingAliases_ReturnsNull() throws IOException { + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice", "bank_statement", "loan_application"), dir); + + assertNull(resolved, "missing aliases must yield null"); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_UnrelatedJsonFilesIgnored() throws IOException { + Path dir = makeTempDir(); + try { + writeEmptyJson(dir, "invoice.json"); + writeEmptyJson(dir, "notes.json"); + writeEmptyJson(dir, "settings.json"); + + Map resolved = CreateAndTestRouterCommand.discoverInnerFromDir( + outerWithAliases("invoice"), dir); + + assertNotNull(resolved); + assertEquals(1, resolved.size()); + assertEquals(dir.resolve("invoice.json"), resolved.get("invoice")); + } finally { + deleteRecursive(dir); + } + } + + @Test + void discoverInnerFromDir_NonExistentDir_Throws() { + Path missing = Path.of(System.getProperty("java.io.tmpdir"), + "cu-skill-not-there-" + System.nanoTime()); + assertThrows(IOException.class, + () -> CreateAndTestRouterCommand.discoverInnerFromDir(outerWithAliases("invoice"), missing)); + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/SchemaValidatorTest.java b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/SchemaValidatorTest.java new file mode 100644 index 000000000000..ca2f6deceeea --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/SchemaValidatorTest.java @@ -0,0 +1,366 @@ +// Copyright (c) Microsoft Corporation. All rights reserved. +// Licensed under the MIT License. + +/* + * Unit tests for SchemaValidator. Mirrors Python's + * tests/test_skills_shared_schema_validator.py and the .NET + * SkillSchemaValidatorTests.cs. Pure Jackson — no Azure.* deps, + * enforced by a purity guard test below. + */ + +package com.azure.ai.contentunderstanding.skills; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.LinkedHashSet; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class SchemaValidatorTest { + + private static final ObjectMapper M = new ObjectMapper(); + + private static JsonNode parse(String json) throws IOException { + return M.readTree(json); + } + + // ------------------------------------------------------------------- + // Valid schemas + // ------------------------------------------------------------------- + + @Test + void validateValidSingleTypeSchemaReturnsOk() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { + "fields": { + "invoiceNumber": { + "type": "string", + "method": "extract", + "description": "Invoice number printed at the top right." + } + } + } + } + """)); + assertTrue(r.isOk(), () -> "Errors: " + String.join("; ", r.getErrors())); + assertTrue(r.getErrors().isEmpty()); + } + + @Test + void validateValidClassifyRouteSchemaReturnsOk() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "config": { + "enableSegment": true, + "contentCategories": { + "invoice": { + "description": "Pages whose top heading is 'Invoice'.", + "analyzerId": "invoice_extractor_v1" + }, + "bank_statement": { + "description": "Pages whose top heading is 'Bank Statement'.", + "analyzerId": "bank_statement_extractor_v1" + } + } + } + } + """)); + assertTrue(r.isOk(), () -> "Errors: " + String.join("; ", r.getErrors())); + } + + @Test + void validateClassifyRouteCategoryWithoutAnalyzerIdAllowedForOtherBucket() throws IOException { + // Category without analyzerId classifies only; the route is fine and + // is a documented pattern for an "other" catch-all. + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "config": { + "enableSegment": true, + "contentCategories": { + "invoice": { "description": "Invoices.", "analyzerId": "inv" }, + "other": { "description": "Anything else." } + } + } + } + """)); + assertTrue(r.isOk(), () -> "Errors: " + String.join("; ", r.getErrors())); + } + + // ------------------------------------------------------------------- + // Single-type rejections + // ------------------------------------------------------------------- + + @Test + void validateUnknownBaseAnalyzerIdRejected() throws IOException { + // Catches the prebuilt-documentAnalyzer typo class — the service + // returns InvalidBaseAnalyzerId without a useful message, so we + // catch it locally with the actual allow-list. + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-documentAnalyzer", + "fieldSchema": { "fields": { "x": { "type": "string" } } } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("baseAnalyzerId"))); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("prebuilt-documentAnalyzer"))); + } + + @Test + void validateMissingFieldSchemaOnNonClassifierRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { "baseAnalyzerId": "prebuilt-document" } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("fieldSchema"))); + } + + @Test + void validateEmptyFieldsObjectRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { "fields": {} } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("at least one field"))); + } + + @Test + void validateUnknownFieldTypeRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { "fields": { "x": { "type": "float" } } } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("'float'"))); + } + + @Test + void validateUnknownFieldMethodRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { "fields": { "x": { "type": "string", "method": "infer" } } } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("'infer'"))); + } + + @Test + void validateNestedObjectFieldRecursesIntoProperties() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { + "fields": { + "billTo": { + "type": "object", + "properties": { + "name": { "type": "bogus" } + } + } + } + } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("billTo"))); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("'bogus'"))); + } + + @Test + void validateArrayFieldRecursesIntoItems() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { + "fields": { + "lineItems": { + "type": "array", + "items": { "type": "nope" } + } + } + } + } + """)); + assertFalse(r.isOk()); + // Path uses bracketed notation: fieldSchema.fields['lineItems'].items.type + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("'lineItems'"))); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains(".items.type"))); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("'nope'"))); + } + + // ------------------------------------------------------------------- + // Classify-and-route rejections + // ------------------------------------------------------------------- + + @Test + void validateClassifyRouteWithTopLevelFieldSchemaRejected() throws IOException { + // Field extraction belongs in inner analyzers, not the outer + // classifier — catch this before the service does. + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { "fields": { "x": { "type": "string" } } }, + "config": { + "enableSegment": true, + "contentCategories": { + "invoice": { "description": "d", "analyzerId": "a" } + } + } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream() + .anyMatch(e -> e.contains("fieldSchema") && e.contains("inner"))); + } + + @Test + void validateClassifyRouteWithoutEnableSegmentRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "config": { + "contentCategories": { + "invoice": { "description": "d", "analyzerId": "a" } + } + } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("enableSegment"))); + } + + @Test + void validateClassifyRouteWithEmptyCategoryDescriptionRejected() throws IOException { + SchemaValidator.Result r = SchemaValidator.validate(parse(""" + { + "baseAnalyzerId": "prebuilt-document", + "config": { + "enableSegment": true, + "contentCategories": { + "invoice": { "description": " ", "analyzerId": "a" } + } + } + } + """)); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("description"))); + } + + // ------------------------------------------------------------------- + // validateFile + // ------------------------------------------------------------------- + + @Test + void validateFileMissingFileReturnsError() { + Path missing = Paths.get( + System.getProperty("java.io.tmpdir"), + "definitely-not-there-" + System.nanoTime() + ".json"); + SchemaValidator.Result r = SchemaValidator.validateFile(missing); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("not found"))); + } + + @Test + void validateFileInvalidJsonReturnsError(@TempDir Path tmp) throws IOException { + Path p = tmp.resolve("broken.json"); + Files.writeString(p, "{ this is not json"); + SchemaValidator.Result r = SchemaValidator.validateFile(p); + assertFalse(r.isOk()); + assertTrue(r.getErrors().stream().anyMatch(e -> e.contains("not valid JSON"))); + } + + @Test + void validateFileValidSchemaOnDiskRoundTrips(@TempDir Path tmp) throws IOException { + Path p = tmp.resolve("valid.json"); + Files.writeString(p, """ + { + "baseAnalyzerId": "prebuilt-document", + "fieldSchema": { "fields": { "x": { "type": "string" } } } + } + """); + SchemaValidator.Result r = SchemaValidator.validateFile(p); + assertTrue(r.isOk(), () -> "Errors: " + String.join("; ", r.getErrors())); + } + + // ------------------------------------------------------------------- + // Allow-list surface + // ------------------------------------------------------------------- + + @Test + void knownBaseAnalyzerIdsOnlyContainsModalityPrebuilts() { + // Sanity check: the allow-list must NOT include `*Search` variants, + // `prebuilt-invoice`, or `prebuilt-receipt` — these return + // `InvalidBaseAnalyzerId` if used as `baseAnalyzerId` for a custom + // analyzer. Only modality-level prebuilt analyzers are valid. + Set expected = new LinkedHashSet<>(); + expected.add("prebuilt-document"); + expected.add("prebuilt-audio"); + expected.add("prebuilt-video"); + expected.add("prebuilt-image"); + assertEquals(expected, SchemaValidator.KNOWN_BASE_ANALYZER_IDS); + } + + // ------------------------------------------------------------------- + // Purity guard + // ------------------------------------------------------------------- + + @Test + void schemaValidatorSourceDoesNotImportAzureOrHttpNamespaces() throws IOException { + // The validator is intentionally pure-Java so it can be unit-tested + // without spinning up the Azure SDK, and so it can be reused from + // any context (CI, scripts, samples). Drift would creep in if a + // future change accidentally pulls in com.azure.* or an HTTP client. + Path src = locateSchemaValidatorSource(); + assertTrue(Files.exists(src), "SchemaValidator.java not found at " + src); + String text = Files.readString(src); + for (String forbidden : new String[] { + "import com.azure.", + "import java.net.http", + "import java.net.HttpURLConnection", + "import java.net.Socket" + }) { + assertFalse(text.contains(forbidden), + "SchemaValidator.java must not contain `" + forbidden + + "` — see _shared/README.md"); + } + } + + private static Path locateSchemaValidatorSource() { + // Walk up from the test working directory to find the module root. + Path dir = Paths.get("").toAbsolutePath(); + for (int i = 0; i < 8 && dir != null; i++) { + Path candidate = dir.resolve("src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java"); + if (Files.exists(candidate)) { + return candidate; + } + // Also check the known relative path under the package skill tree. + Path nested = dir.resolve(".github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java"); + if (Files.exists(nested)) { + return nested; + } + dir = dir.getParent(); + } + return Paths.get("src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java"); + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md new file mode 100644 index 000000000000..f2a704818e49 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md @@ -0,0 +1,475 @@ +--- +name: cu-sdk-author-analyzer-classify-route +description: Create and test a classify-and-route Azure AI Content Understanding pipeline for packets that contain multiple document types (e.g. invoice + bank statement + loan application in one PDF). Walks per-type schema authoring → outer classifier wiring → batch test → category-aware stdout summary using the typed ContentUnderstandingClient (Java). Use when the user has mixed-document packets. +--- + +# Author a Classify-and-Route Analyzer (mixed document packets) + +Build a classify-and-route pipeline: one **outer classifier analyzer** that +segments and labels a multi-document packet, plus one **inner extractor +analyzer per document type**. The packet flows through the outer analyzer +once; each segment is automatically routed to the matching inner analyzer +for field extraction. + +**This is an iterative, human-in-the-loop workflow.** You will typically run +the schema → test → review cycle multiple times to refine both the outer +classifier descriptions and each inner schema's field descriptions before +you're happy with both classification accuracy and extraction quality. + +> **[USE INSTEAD]:** If every page in the user's documents is the **same +> type** (only invoices, only contracts, etc.), use +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) instead. +> Classify-and-route is for **mixed** packets. + +> **[ASK USER] Modality check (first thing to confirm):** +> +> "Are you working with **document** files — PDFs or images of pages? This +> skill currently supports document modalities only. Audio, video, and +> image classifiers are planned for a future update." +> +> - If the user says **document** → continue with this skill. +> - If the user says **audio**, **video**, or **image** → stop this skill. +> Audio/video classify-and-route is on the roadmap; for now point them at +> the [REST tutorial](https://learn.microsoft.com/azure/ai-services/content-understanding/tutorial/create-custom-analyzer). + +> **[COPILOT INTERACTION MODEL]:** At each step marked with **[ASK USER]**, +> pause execution and prompt the user before proceeding. + +## Prerequisites + +Required: Maven + JDK 17+ (the skill tool is a standalone Maven module +under `.github/skills/_shared/`), `.env` or environment variables with +`CONTENTUNDERSTANDING_ENDPOINT` (plus `CONTENTUNDERSTANDING_KEY` or +`az login`), and the model defaults configured for this resource (see +[`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java)). + +> **[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.** +> +> ```bash +> mvn -v >/dev/null 2>&1 && echo 'mvn: ok' || echo 'mvn: MISSING' +> java --version >/dev/null 2>&1 && echo 'java: ok' || echo 'java: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty' +> az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in' +> ``` +> +> | Failure | Route to | +> |---|---| +> | `mvn: MISSING` | install Maven from https://maven.apache.org/install.html | +> | `java: MISSING` | install JDK 17+ from https://adoptium.net | +> | endpoint `MISSING` | create or edit `.env` with `CONTENTUNDERSTANDING_ENDPOINT=https://.services.ai.azure.com/`, then resume | +> | endpoint `ok`, key `empty`, `az: not logged in` | run `az login` **or** add `CONTENTUNDERSTANDING_KEY` to `.env`, then resume | +> | All env checks pass but service calls fail with model errors | run [`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java) once for this resource, then resume | +> | All ok | ✅ Proceed to the Packet check below. | +> +> Never ask the user to paste an endpoint or API key into chat — they edit `.env` directly or run `az login`. + +> **[ASK USER] Packet check:** +> 1. "Does each document in your packet contain more than one type of form (e.g. an invoice page followed by a bank statement page)?" — if no, route to `cu-sdk-author-analyzer`. +> 2. "What types of documents appear in your packets?" — capture as the list of inner analyzers. + +## Architecture + +``` + ┌──────────────────────────────┐ + mixed packet ───► │ outer (classifier) analyzer │ + │ baseAnalyzerId: prebuilt-… │ + │ config.enableSegment: true │ + │ config.contentCategories: │ + │ invoice ────────►│──┐ + │ bank_statement ────────►│──┼──► per-segment fields + │ loan_application ────────►│──┘ + └──────────────────────────────┘ + │ + inner analyzers (1 per type) │ + ─────────────────────────── ▼ + invoice extractor ◄──── routes here for invoice pages + bank statement ext. ◄──── routes here for bank pages + loan app extractor ◄──── routes here for loan pages +``` + +Key rules (also captured in +[`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) § +"classify-and-route"): + +1. **Category descriptions reference text anchors**, not visual cues + (matches the two-stage pipeline rule for fields). +2. **`config.enableSegment` must be `true`** so the classifier can carve up + the packet before routing. +3. **Inner analyzers must exist before** the outer classifier is created. + The provided tool handles ordering automatically. +4. **Category fill rate is per-category**, not packet-wide. The tool's + stdout summary uses the right denominator. + +## Package directory + +``` +sdk/contentunderstanding/azure-ai-contentunderstanding +``` + +## Scripts and templates + +``` +.github/skills/ +├── _shared/ # The skill tool (single Maven module, three subcommands) +└── cu-sdk-author-analyzer-classify-route/ + ├── SKILL.md (this file) + └── templates/ + └── classifier_template.json # Starter outer-classifier schema for Step 3 +``` + +The skill tool builds against the local `azure-ai-contentunderstanding-1.1.0-beta.2.jar`, +so before first use: + +```bash +mvn -B -q -DskipTests -f sdk/contentunderstanding/azure-ai-contentunderstanding/pom.xml package +``` + +## Workflow + +### Step 1 — Identify the document types + +Run layout extraction (same as the single-type skill) on a representative +packet to see the section headings: + +```bash +mvn -B -q -f sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml exec:java \ + -Dexec.args="extract-layout --input --output .local_only/layout/" +``` + +> **[COPILOT]** Read `.local_only/layout/.layout.md` **yourself** and +> identify section headings, page breaks, and content shifts that suggest +> different document types. Then present your analysis to the user for +> confirmation — do **not** ask the user to read the layout cold. +> +> Example presentation: +> +> > "Based on the layout in `.layout.md` I see these document types: +> > +> > - **Pages 1–2** — appears to be an *invoice* (top heading 'Invoice', +> > contains 'Invoice #' label and a line-item table) +> > - **Pages 3–4** — appears to be a *bank statement* (top heading +> > 'Account Statement', contains account number and a transaction table) +> > - **Page 5** — appears to be a *loan application* (top heading 'Loan +> > Application', applicant fields) +> > +> > Does this look right? Anything to add, remove, or rename before I +> > draft schemas?" +> +> Only fall back to a blank `[ASK USER]` ("What types do you see?") if the +> layout is too ambiguous to suggest types confidently. + +### Step 2 — Draft one inner schema per type + +Treat each type as a single-doc-type analyzer (pick `baseAnalyzerId` from +the table in +[`cu-sdk-common-knowledge` § Choosing `baseAnalyzerId`](../cu-sdk-common-knowledge/SKILL.md#choosing-baseanalyzerid), +then add `fieldSchema.fields`). Field descriptions follow the +[two-stage pipeline rule](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline) +— reference text and structure, never visual appearance. See +[`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) Step 2 for +the full field-schema walkthrough. + +> **[COPILOT] Read best practices before drafting fields.** Before writing +> any field description (in any inner schema or the outer classifier's +> category descriptions), fetch the official CU best-practices page and +> apply its guidance: +> +> 🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices +> +> The same key principles apply here as in +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md#step-2--draft-a-json-field-schema): +> be specific and concrete, use text anchors (not visual cues), include +> alternative labels and format examples, prefer `extract` for verbatim +> values, and keep the field count focused. + +> **Reference**: +> [`Sample05_CreateClassifierAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample05_CreateClassifierAsync.java) +> ships a complete worked example using +> `samples/sample_files/mixed_financial_docs.pdf` with three categories — +> Invoice, Bank_Statement, Loan_Application. + +### Step 3 — Draft the outer classifier schema + +The outer schema has **no** `fieldSchema`. Its job is classification + routing. +Start from the template: + +```bash +mkdir -p .local_only/schemas +cp sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json \ + .local_only/schemas/_classifier_v1.json +``` + +Example after editing (descriptions intentionally generic so they survive +minor wording variants — see the **Category description rule** below the +example for guidance on writing your own): +```json +{ + "baseAnalyzerId": "prebuilt-document", + "description": "Classify mixed financial packets and route to per-type extractors.", + "config": { + "enableSegment": true, + "omitContent": true, + "contentCategories": { + "invoice": { + "description": "A commercial invoice. Look for an invoice / bill heading or label, an invoice number, line-item table with quantities and prices, and a total amount.", + "analyzerId": "invoice" + }, + "bank_statement": { + "description": "A bank or account statement. Look for an account-statement heading, an account number, a statement period or date range, and a transaction table with running balance.", + "analyzerId": "bank_statement" + }, + "loan_application": { + "description": "A loan or credit application form. Look for an application heading, applicant or borrower fields, requested loan amount, and applicant signature.", + "analyzerId": "loan_application" + } + } + }, + "models": { + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + } +} +``` + +The `analyzerId` value in each category is **an alias** that the tool +resolves at runtime, matching the `--inner-schema alias=path` flags you +pass. Two exceptions skip alias resolution: + +* Values starting with `prebuilt-` (e.g. `prebuilt-invoice`) are used as-is + — no inner schema needed. Useful for routing a category at a service + prebuilt extractor. +* Categories without an `analyzerId` at all are classification-only — the + segment is labelled but no fields are extracted. + +> **Why `omitContent: true`?** When omitted, the service also returns the +> raw, un-segmented document content as an extra entry in `contents`. That +> entry has no category, no fields, and shows up in the summary as a +> confusing `(uncategorized)` row. Setting `omitContent: true` removes it. + +> **Category description rule — the example values above are demo-specific.** +> +> The category descriptions in the JSON above are tied to the demo packet +> `samples/sample_files/mixed_financial_docs.pdf` (which uses the literal +> headings `Invoice`, `Bank Statement`, `Loan Application`). **Do not copy +> them verbatim.** When authoring for the user's own packet, write +> descriptions based on what you observed in **the user's** layout output +> from Step 1 — use the actual headings, labels, and structural markers +> from their documents. +> +> Keep descriptions: +> +> - **Generic over surface form** — describe the *kind* of content +> ("contains a transaction table with running balance") rather than +> hardcoding one specific header string, so minor wording variants still +> classify correctly. +> - **Concrete enough to be discriminative** — include at least one anchor +> that distinguishes this category from the others in the packet. +> - **Text-anchored, not visual** — reference headings, labels, and +> neighbouring text, never colour / font / position-without-text. Same +> reason as the field-description rule in +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline). + +### Step 4 — Validate, create, and batch-test + +```bash +mvn -B -q -f sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml exec:java \ + -Dexec.args="create-and-test-router --outer-schema .local_only/schemas/classifier.json --inner-schema invoice=.local_only/schemas/invoice.json --inner-schema bank_statement=.local_only/schemas/bank_statement.json --inner-schema loan_application=.local_only/schemas/loan_application.json --input samples/sample_files/mixed_financial_docs.pdf --output .local_only/test_results/v1" +``` + +> **Shortcut — `--schema-dir`:** if your inner schema filenames match the +> outer-schema category aliases (e.g. `.local_only/schemas/invoice_v1.json` for category +> `invoice`), replace every `--inner-schema alias=path` with a single +> `--schema-dir .local_only/schemas/`. The tool picks the newest matching file per +> alias (alphabetical sort, so `invoice_v2.json` wins over `invoice_v1.json`). + +> **Iteration helper — `--reuse`:** add `--reuse` to name analyzers by a +> sha1 of their schema (`_`) and skip creation when an +> analyzer with that ID already exists. Re-running with the same schemas +> is a no-op on the create side, so you don't pile up stale analyzers while +> iterating. Edit a schema → hash changes → new analyzer is created. + +The tool: + +1. Validates every schema (exits with code **2** if any fails — no service + call made). +2. Errors out if the outer schema references an alias that has no matching + `--inner-schema`, or if you supply an `--inner-schema` that no category + uses. +3. Creates inner analyzers first, then patches and creates the outer + classifier. +4. Analyzes every input file, writing one JSON per file under `--output`. +5. Prints a **category-aware** stdout summary (per-category fill rate + uses each category's segment count, not the packet-wide total). + +### Step 5 — Read the category-aware summary + +Example output: + +``` +======================================================================== +[SUMMARY] (category-aware) + +category: bank_statement (1 segments) +-------------------------------------- + field fill rate avg conf + AccountNumber 100.0% 0.918 + StatementPeriod 100.0% 0.882 + +category: invoice (1 segments) +------------------------------- + field fill rate avg conf + InvoiceNumber 100.0% 0.962 + TotalAmount 100.0% 0.531 + +category: loan_application (1 segments) +---------------------------------------- + field fill rate avg conf + ApplicantName 100.0% 0.875 + LoanAmount 100.0% 0.799 + +lowest-confidence fields across all categories: + 0.531 [invoice] TotalAmount (mixed_financial_docs) + 0.799 [loan_application] LoanAmount (mixed_financial_docs) + 0.875 [loan_application] ApplicantName (mixed_financial_docs) +======================================================================== +``` + +For each input document the tool writes two files into `--output`: + +- `.json` — full `AnalysisResult` with all per-segment fields and grounding. +- `.llm.md` — the same result rendered via the SDK's + [`LlmInputHelper.toLlmInput`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample_Advanced_ToLlmInput.java) helper. + For classify-and-route, the helper expands each classified segment into + its own block with the **category in the YAML front matter**, separated by + `*****` dividers — drop it into an LLM prompt or skim it in VS Code. + +> **Template comment keys**: any key whose name starts with `_` +> (e.g. `_comment`, `_optional_returnDetails`) is stripped from both the +> outer and inner schemas before the request body is sent. Use them freely +> as inline documentation. + +> **`models.completion` for inner schemas**: each inner schema (which has a +> `fieldSchema`) needs `models.completion` set unless the resource has +> defaults configured via +> [`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java). +> `create-and-test-router` prints a `[WARN]` per inner schema that is +> missing it, before any service call. + +### Step 6 — Agent review and iterate + +> **[COPILOT]** After the category-aware summary prints, do the following +> **automatically** before asking the user anything: +> +> 1. For each entry in `result.contents[]` in `.json`, verify the +> assigned `category` matches what the page actually is. Use +> `.local_only/layout/.layout.md` as ground truth. +> - **Misclassified segment** → the **outer classifier's** +> `contentCategories..description` needs strengthening, not the +> inner schema. Add a discriminating anchor from the layout to the +> category description and recreate the outer classifier. +> 2. For each correctly-classified segment, compare extracted field values +> against the layout the same way as the single-type skill (see +> [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md#step-5--agent-review-and-iterate)). +> Flag fields where: +> - The extracted value looks wrong or is `null` when a value should be present +> - Confidence is below 0.7 +> - The grounding location is unexpected +> 3. Present a per-category diff to the user: +> - "In the `invoice` segment, field `TotalAmount` extracted `'1234'` — +> I see `'$1,234.56'` near the 'Total' label. Should I tighten the +> description?" +> - "Page 5 was classified as `loan_application` but the heading is +> actually 'Borrower Agreement' — should I add that wording to the +> `loan_application` category description?" +> 4. Record user-confirmed corrections in per-category ground-truth files +> (`.local_only/ground_truth_.json`) so the agent remembers +> correct answers across iterations: +> ```json +> [ +> { "doc": "mixed_packet_1.pdf", "segment": 0, "field": "TotalAmount", "correct_value": "1234.56" } +> ] +> ``` +> 5. Update the affected schema(s) — outer classifier for classification +> fixes, inner schemas for field fixes — as a new version +> (`*_v2.json`). +> 6. Re-run Step 4 with the new schemas. `--reuse` rebuilds only the +> analyzers whose schema hash changed. +> +> Repeat until every category has: +> +> - All segments classified correctly +> - Key fields **fill rate ≥ 80%** and **avg confidence ≥ 0.85** (computed +> per-category, as the summary already does) +> +> Stop and report to the user when any of: +> +> - All targets met (success) — then proceed to **Step 7** to hand off. +> - Three consecutive iterations show no improvement on a given category +> (escalate — may need a different `baseAnalyzerId`, schema redesign, or +> a different category split). +> - The user signals they're done. + +### Step 7 — Hand off the finished pipeline + +This step is required when Step 6 succeeds. **Do not skip it.** Without a +clean handoff the user has a working classify-and-route pipeline in their +resource but no idea what the outer classifier ID is, which inner +analyzers it routes to, where the final schemas live, or how to call it +from their own code. + +> **[COPILOT]** When Step 6 reaches success, report the following to the +> user in one message before stopping: +> +> 1. **All analyzer IDs built** — list both the **outer classifier** and +> every **inner extractor**, with the actual IDs printed by +> `create-and-test-router` (e.g. +> `classifier_v2_a1b2c3d4` → routes to +> `classifier_v2_a1b2c3d4_inner_invoice_b2c3d4e5`, +> `..._inner_bank_statement_c3d4e5f6`, +> `..._inner_loan_application_d4e5f6a7`). The user only needs to call +> the **outer** analyzer in their own code; it routes to the inner +> ones automatically. +> 2. **All schema files** — list the path to every schema JSON in the +> final iteration: the outer classifier and each inner schema (e.g. +> `.local_only/schemas/classifier_v2.json`, +> `.local_only/schemas/invoice_v3.json`, etc.). Recommend the user save +> them somewhere outside `.local_only/` for future reference; they can +> also inspect any existing analyzer's schema directly via the SDK (see +> [`Sample06_GetAnalyzer.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample06_GetAnalyzer.java)). +> 3. **Next-step samples** — point the user to: +> - [`Sample05_CreateClassifierAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample05_CreateClassifierAsync.java) +> — how to (re)build the full classify-and-route pipeline from schema +> JSON in their own code (handles inner-first creation ordering and +> `contentCategories.analyzerId` wiring the same way our tool did). +> - [`Sample01_AnalyzeBinaryAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample01_AnalyzeBinaryAsync.java) +> and +> [`Sample02_AnalyzeUrl.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample02_AnalyzeUrl.java) +> — how to call the analyzer on real input from their own code. Use +> the **outer classifier's** analyzer ID as `analyzerId`. +> +> Remind the user that the analyzers you just built are **already +> deployed** to their resource and ready to use directly via their IDs +> — they only need to re-create them from the schemas if they want to +> bootstrap into another resource. + +### Step 8 — Clean up (optional) + +By default the tool leaves both the outer classifier **and** all inner +analyzers in your resource so you can re-use them. Pass `--ephemeral` to +delete all of them at the end of the run. + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | Every document analyzed successfully. | +| `1` | At least one service-side failure. | +| `2` | Local user error — schema validation, missing inner alias, bad path. No service call made. | + +## Related skills + +- [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) — single doc type. +- [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) — service concepts, two-stage pipeline, `baseAnalyzerId` table, classify-and-route rules. +- [`cu-sdk-sample-run`](../cu-sdk-sample-run/SKILL.md) — run individual SDK samples (e.g. `Sample05_CreateClassifierAsync.java`) for deeper reference. +- [`cu-sdk-setup`](../cu-sdk-setup/SKILL.md) — install the SDK, configure env. diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json new file mode 100644 index 000000000000..650eb35fc0d2 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json @@ -0,0 +1,32 @@ +{ + "_comment": "Starter template for a classify-and-route outer classifier. Copy to .local_only/schemas/_classifier_v1.json and edit. See SKILL.md Step 3 for the rules. Each contentCategories entry's analyzerId is an alias resolved by --inner-schema alias=path at runtime; values starting with 'prebuilt-' (e.g. prebuilt-invoice, prebuilt-receipt) are used as-is and don't need an inner schema. Keys beginning with '_' (_comment, _optional_*) are stripped from the request body before submission.", + "baseAnalyzerId": "prebuilt-document", + "description": "REPLACE: one-line description of what this packet contains.", + "config": { + "_comment": "enableSegment=true is required for multi-document packets. omitContent=true keeps the outer response small \u2014 inner analyzers see the full content independently.", + "enableSegment": true, + "omitContent": true, + "_optional_returnDetails": true, + "contentCategories": { + "REPLACE_category_a": { + "_comment": "Category routed to a custom inner analyzer. The analyzerId value must match an --inner-schema alias=path passed on the command line.", + "description": "REPLACE: text-anchored description \u2014 reference labels, headings, table column names. Never visual cues.", + "analyzerId": "REPLACE_category_a" + }, + "REPLACE_category_b": { + "_comment": "Category routed to a service prebuilt analyzer. Any analyzerId starting with 'prebuilt-' (e.g. prebuilt-invoice, prebuilt-receipt) is used as-is and does NOT need a corresponding --inner-schema entry.", + "description": "REPLACE: text-anchored description for a document type covered by a prebuilt analyzer.", + "analyzerId": "prebuilt-invoice" + }, + "other": { + "_comment": "Classify-only category with no analyzerId \u2014 segments matching this category are labelled but no fields are extracted. Recommended catch-all; without it every page is forced into one of the defined categories.", + "description": "REPLACE: catch-all category for pages that match none of the above." + } + } + }, + "models": { + "_comment": "Required when contentCategories is set, unless the resource has defaults set by running the Sample00_UpdateDefaultsAsync sample.", + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/SKILL.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/SKILL.md new file mode 100644 index 000000000000..ed93fcbe7a73 --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/SKILL.md @@ -0,0 +1,397 @@ +--- +name: cu-sdk-author-analyzer +description: Iteratively author and test a custom Azure AI Content Understanding analyzer for a folder of **document** files (PDFs, page images) of a single type. Walks layout extraction → schema authoring → validation → batch test → agent review → refine cycle using the typed ContentUnderstandingClient (Java). Document modality only — audio, video, and image analyzers are planned for a later update. Use when the user wants to author a custom analyzer for invoices, contracts, forms, or any other single-type document set. +--- + +# Author a Custom Analyzer (single document type) + +Author a custom Content Understanding analyzer for one document type +end-to-end: extract layout, draft a field schema, validate locally, create the +analyzer, batch-test it on sample files, and read a quality summary. + +**This is an iterative, human-in-the-loop workflow.** You will typically run +the schema → test → review cycle multiple times to refine field descriptions +before you're happy with the extraction quality. + +The workflow uses the typed `ContentUnderstandingClient` shipped in this +package — the same client `Sample04_CreateAnalyzerAsync.java` and +`Sample01_AnalyzeBinaryAsync.java` use. + +> **[COPILOT INTERACTION MODEL]:** This skill is designed to be interactive. +> At each step marked with **[ASK USER]**, pause execution and prompt the user +> for input or confirmation before proceeding. Do NOT silently skip these +> prompts. Use the `ask_questions` tool when available. + +> **[USE INSTEAD]:** If the user's packet contains **multiple different +> document types** (for example, an invoice, a bank statement, and a loan +> application in one PDF), route them to the +> [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) +> skill instead. This skill assumes one document type per analyzer. + +> **[ASK USER] Modality check (first thing to confirm):** +> +> "Are you working with **document** files — PDFs or images of pages? This +> skill currently supports document modalities only. Audio, video, and +> image analyzers are planned for a future update." +> +> - If the user says **document** → continue with this skill. +> - If the user says **audio**, **video**, or **image** → stop this skill and +> point them at the typed-model samples +> ([`Sample04_CreateAnalyzerAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample04_CreateAnalyzerAsync.java) +> with `prebuilt-audio` / `prebuilt-video` / `prebuilt-image`) or the +> [REST tutorial](https://learn.microsoft.com/azure/ai-services/content-understanding/tutorial/create-custom-analyzer). + +## Prerequisites + +Required: Maven + JDK 17+ (the skill tool is a standalone Maven module +under `.github/skills/_shared/`), `.env` or environment variables with +`CONTENTUNDERSTANDING_ENDPOINT` (plus `CONTENTUNDERSTANDING_KEY` or +`az login`), and the model defaults configured for this resource (see +[`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java)). + +> **[COPILOT] Probe first, then route on failure — do not duplicate setup logic here.** +> +> ```bash +> mvn -v >/dev/null 2>&1 && echo 'mvn: ok' || echo 'mvn: MISSING' +> java --version >/dev/null 2>&1 && echo 'java: ok' || echo 'java: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_ENDPOINT=https?://' .env >/dev/null && echo 'endpoint: ok' ) || echo 'endpoint: MISSING' +> ( [ -f .env ] && grep -E '^CONTENTUNDERSTANDING_KEY=.+' .env >/dev/null && echo 'key: set' ) || echo 'key: empty' +> az account show >/dev/null 2>&1 && echo 'az: ok' || echo 'az: not logged in' +> ``` +> +> | Failure | Route to | +> |---|---| +> | `mvn: MISSING` | install Maven from https://maven.apache.org/install.html | +> | `java: MISSING` | install JDK 17+ from https://adoptium.net | +> | endpoint `MISSING` | create or edit `.env` at the repo root with `CONTENTUNDERSTANDING_ENDPOINT=https://.services.ai.azure.com/`, then resume | +> | endpoint `ok`, key `empty`, `az: not logged in` | run `az login` **or** add `CONTENTUNDERSTANDING_KEY` to `.env`, then resume | +> | All env checks pass but service calls fail with model errors | run [`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java) once for this resource, then resume | +> | All ok | ✅ Proceed to Step 1. | +> +> Never ask the user to paste an endpoint or API key into chat — they edit `.env` directly or run `az login`. + +> **[ASK USER]** "How many representative documents do you have, and where are they?" — fewer than 3 is fine, but more is better for testing coverage. + +## Package directory + +``` +sdk/contentunderstanding/azure-ai-contentunderstanding +``` + +## Scripts and templates + +``` +.github/skills/ +├── _shared/ +│ ├── pom.xml # Single tool project — three subcommands +│ ├── Cli.java # Dispatcher (extract-layout | create-and-test | create-and-test-router) +│ ├── SchemaValidator.java # Pure-Java schema validator (no service calls) +│ ├── ExtractLayoutCommand.java # Stage 1 +│ ├── CreateAndTestCommand.java # Stage 2 (single-type) +│ └── CreateAndTestRouterCommand.java # Stage 2 (classify-and-route) +└── cu-sdk-author-analyzer/ + ├── SKILL.md (this file) + └── templates/ + └── schema_template.json # Starter schema for Step 2 +``` + +The skill tool builds against the local `azure-ai-contentunderstanding-1.1.0-beta.2.jar`, +so before first use: + +```bash +mvn -B -q -DskipTests -f sdk/contentunderstanding/azure-ai-contentunderstanding/pom.xml package +``` + +## Workflow + +### Step 1 — Extract layout for representative documents + +The model behind Content Understanding sees the **text and structure** the +service extracts from your file, not the original pixels. Reviewing the +layout output is the fastest way to know what labels and headings you can +anchor your field descriptions to. + +> **[ASK USER]** "Point me at one of your sample documents (or a folder of +> them). I'll run layout extraction so we can see what the model will be +> looking at." + +Run: + +```bash +mvn -B -q -f sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml exec:java \ + -Dexec.args="extract-layout --input --output .local_only/layout/" +``` + +This produces one `.layout.md` and one `.layout.json` per input. +Open the `.layout.md` file in VS Code and look for the **text anchors** you +want to extract from — labels (`"Invoice #:"`), section headings +(`"Bill To"`), table headers, etc. + +> **Reference**: this is the same call pattern as +> [`Sample01_AnalyzeBinaryAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample01_AnalyzeBinaryAsync.java) +> using `prebuilt-documentSearch`. + +### Step 2 — Draft a JSON field schema + +Start from the template instead of writing from scratch: + +```bash +mkdir -p .local_only/schemas +cp sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json \ + .local_only/schemas/_v1.json +``` + +Then edit `.local_only/schemas/_v1.json`: set `baseAnalyzerId`, replace every +`REPLACE:` placeholder, and add/remove fields. The schema is a JSON object +with two required top-level keys: + +- `baseAnalyzerId` — which prebuilt analyzer your custom analyzer extends. Use the table below. +- `fieldSchema.fields` — the named fields you want to extract. + +> **[COPILOT] Read best practices before drafting fields.** Before writing +> any field description, fetch the official CU best-practices page and apply +> its guidance: +> +> 🔗 https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/best-practices +> +> Key principles that affect schema quality: +> +> - Be specific and concrete in field descriptions — vague descriptions +> produce vague extractions. +> - Use **text anchors** (labels, headings, neighbouring fields) — never +> visual cues like colour, font, or position-without-text-context. This +> matches the two-stage pipeline rule in +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md#field-description-rule-the-two-stage-pipeline). +> - Include **format examples** and **alternative label names** when a value +> can appear in multiple wordings or formats. +> - Prefer `"method": "extract"` when the value appears verbatim; use +> `"generate"` only when the model needs to synthesise (summary, +> classification label) and `"classify"` for fixed enumerations. +> - Keep the field count focused on what the user actually needs. Extra +> fields cost tokens and can dilute extraction quality on the fields that +> matter. + +The template demonstrates **all three extraction methods** (`extract`, +`generate`, `classify`) plus the **nested object** and **array of objects** +shapes. Delete the example fields you don't need. + +> **Template comment keys**: any key whose name starts with `_` +> (e.g. `_comment`, `_optional_enableOcr`) is stripped from the request body +> by `create-and-test` before it hits the service. Use them freely as +> inline documentation. + +> **`models.completion` is effectively required**: whenever `fieldSchema` is +> present, the service needs a completion model. Leave the `models` block in +> the template populated unless you've run +> [`Sample00_UpdateDefaultsAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample00_UpdateDefaultsAsync.java) +> to set resource defaults. Omitting it fails with `InvalidRequest: +> 'models.completion' is not set` *after* a misleadingly successful +> `[CREATE]` log line. `create-and-test` prints a `[WARN]` if the schema +> is missing it. + +#### Choosing `baseAnalyzerId` + +See the prebuilt-analyzer table in +[`cu-sdk-common-knowledge` § Choosing `baseAnalyzerId`](../cu-sdk-common-knowledge/SKILL.md#choosing-baseanalyzerid). +The local validator (Step 3) rejects any value not on that list. + +#### Example single-type schema + +```json +{ + "baseAnalyzerId": "prebuilt-document", + "description": "Extract invoice header and totals.", + "config": { + "estimateFieldSourceAndConfidence": true, + "returnDetails": true + }, + "models": { + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + }, + "fieldSchema": { + "name": "invoice_v1", + "description": "Invoice header fields.", + "fields": { + "invoiceNumber": { + "type": "string", + "method": "extract", + "description": "Invoice number printed near the 'Invoice #' label at the top of the page.", + "estimateSourceAndConfidence": true + }, + "totalAmount": { + "type": "number", + "method": "extract", + "description": "Grand total at the bottom of the document; typically labelled 'Total' or 'Amount Due'. Excludes any 'Subtotal' value.", + "estimateSourceAndConfidence": true + } + } + } +} +``` + +> **Reference**: see +> [`Sample04_CreateAnalyzerAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample04_CreateAnalyzerAsync.java) +> for the typed-model equivalent. The `create-and-test` tool sends the JSON +> directly so the schema may include any properties supported by the +> service, even if the typed model doesn't expose them. + +> **Field-description rule (two-stage pipeline):** descriptions must reference +> **text content and structure** (labels, headings, neighbouring fields), not +> visual appearance (colour, font, size). See +> [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) § +> "two-stage pipeline". + +### Step 3 — Validate the schema locally + +```bash +mvn -B -q -f sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml exec:java \ + -Dexec.args="create-and-test --schema .local_only/schemas/invoice_v1.json --input samples/sample_files/sample_invoice.pdf --output .local_only/test_results/v1" +``` + +The script runs the local validator first. If anything is wrong (unknown +`baseAnalyzerId`, missing `fieldSchema`, malformed entries) it exits with +code **2** *before* any service call. + +### Step 4 — Read the stdout summary + +After the script finishes you get something like: + +``` +======================================================================== +[SUMMARY] + +category: (single) (3 documents) +--------------------------------- + field fill rate avg conf + invoiceNumber 100.0% 0.962 + totalAmount 66.7% 0.481 + +lowest-confidence fields: + 0.461 totalAmount (mixed_financial_docs) + 0.732 invoiceNumber (mixed_financial_docs) +======================================================================== +``` + +For each input document the script writes two files into `--output`: + +- `.json` — full per-document `AnalysisResult` (fields, grounding, confidences). +- `.llm.md` — same result rendered via the SDK's + [`LlmInputHelper.toLlmInput`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample_Advanced_ToLlmInput.java) helper: + YAML front matter (category, page range, fields) plus the document text. + Drop this straight into an LLM prompt, or skim it in VS Code for a fast + human review. + +### Step 5 — Agent review and iterate + +> **[COPILOT]** After the summary prints, do the following **automatically** +> before asking the user anything: +> +> 1. Open each `.llm.md` (and the underlying `.json` for grounding) +> in `.local_only/test_results/` and compare extracted field values +> against the source content. Use `.local_only/layout/.layout.md` +> as ground truth — it shows the text the model actually saw. +> 2. Flag any field where: +> - The extracted value looks wrong or is `null` when a value should be present +> - Confidence is below 0.7 +> - The grounding location is unexpected +> 3. Present findings to the user with concrete diffs: +> - "Field `invoiceNumber` extracted `'INV-001'` — does this look correct?" +> - "Field `totalAmount` was empty, but I see `'$1,234.56'` near the +> 'Total' label in the layout — should I tighten the description?" +> 4. For each correction the user confirms, append to +> `.local_only/ground_truth_.json`: +> ```json +> [ +> { "doc": "invoice_001.pdf", "field": "totalAmount", "correct_value": "1234.56" }, +> { "doc": "invoice_002.pdf", "field": "invoiceDate", "correct_value": "2026-01-15" } +> ] +> ``` +> This file is the agent's memory of correct answers across iterations. +> 5. Use the corrections to refine the field descriptions in a new schema +> version (`.local_only/schemas/_v2.json`). +> 6. Re-run Step 3–4 with the new schema. The `--reuse` flag on +> `create-and-test` names analyzers by schema hash, so unchanged +> schemas are a no-op on the create side and a changed schema gets a +> fresh analyzer automatically. +> +> Repeat until all key fields reach **fill rate ≥ 80%** and +> **avg confidence ≥ 0.85**, or the user is satisfied. +> +> Stop and report to the user when any of: +> - Targets are met (success) — then proceed to **Step 6** to hand off. +> - Three consecutive iterations show no improvement (need a different +> approach — different `baseAnalyzerId`, different `method`, or schema +> redesign — escalate to the user). +> - The user signals they're done. + +### Step 6 — Hand off the finished analyzer + +This step is required when Step 5 succeeds. **Do not skip it.** Without a +clean handoff the user has a working analyzer in their resource but no +idea what its ID is, where the final schema lives, or how to call it from +their own code. + +> **[COPILOT]** When Step 5 reaches success, report the following to the +> user in one message before stopping: +> +> 1. **Final analyzer ID** — the actual ID printed by `create-and-test` +> on its last successful run (e.g. `invoice_v3_a1b2c3d4` when `--reuse` +> is used). The user will need this to call the analyzer from their own +> code. +> 2. **Final schema file** — the path to the last iteration of the schema +> JSON (e.g. `.local_only/schemas/invoice_v3.json`). Recommend the user +> save it somewhere outside `.local_only/` for future reference; they +> can also inspect any existing analyzer's schema directly via the SDK +> (see +> [`Sample06_GetAnalyzer.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample06_GetAnalyzer.java)). +> 3. **Next-step samples** — point the user to: +> - [`Sample04_CreateAnalyzerAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample04_CreateAnalyzerAsync.java) +> — how to (re)create a custom analyzer from a schema JSON in their own code. +> - [`Sample01_AnalyzeBinaryAsync.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample01_AnalyzeBinaryAsync.java) +> and +> [`Sample02_AnalyzeUrl.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample02_AnalyzeUrl.java) +> — how to call the analyzer on real input from their own code. +> +> Remind the user that the analyzer you just built is **already deployed** +> to their resource and ready to use directly via its ID — they only +> need to re-create it from the schema if they want to bootstrap it in +> another resource. + +### Step 7 — Clean up (optional) + +By default the analyzer is kept in your resource so you can re-use it. Pass +`--ephemeral` to delete it at the end of a run: + +```bash +mvn -B -q -f sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml exec:java \ + -Dexec.args="create-and-test --schema .local_only/schemas/invoice_v1.json --input samples/sample_files/sample_invoice.pdf --output .local_only/test_results/v1 --ephemeral" +``` + +> **Iteration helper — `--reuse`:** add `--reuse` to name the analyzer by a +> sha1 of its schema (`_`) and skip creation when an +> analyzer with that ID already exists. Re-running with the same schema is +> a no-op on the create side, so you don't pile up stale analyzers while +> iterating. Edit the schema → hash changes → new analyzer is created. + +For explicit lifecycle management see +[`Sample06_GetAnalyzer.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample06_GetAnalyzer.java), +[`Sample07_ListAnalyzers.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample07_ListAnalyzers.java), +[`Sample08_UpdateAnalyzer.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample08_UpdateAnalyzer.java), +and +[`Sample09_DeleteAnalyzer.java`](../../../src/samples/java/com/azure/ai/contentunderstanding/samples/Sample09_DeleteAnalyzer.java). + +## Exit codes + +| Code | Meaning | +|---|---| +| `0` | All documents analyzed successfully. | +| `1` | At least one service-side failure (network, throttling, invalid response). | +| `2` | Local user error — schema validator failure, missing flag, bad input path. No service call was made. | + +## Related skills + +- [`cu-sdk-setup`](../cu-sdk-setup/SKILL.md) — install the SDK, configure env. +- [`cu-sdk-sample-run`](../cu-sdk-sample-run/SKILL.md) — run one reference sample. +- [`cu-sdk-common-knowledge`](../cu-sdk-common-knowledge/SKILL.md) — service concepts and field-description rules. +- [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) — multi-doc-type packets. diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json new file mode 100644 index 000000000000..f7b26906dbba --- /dev/null +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json @@ -0,0 +1,81 @@ +{ + "_comment": "Starter template for a single-doc-type custom analyzer. Copy to .local_only/schemas/_v1.json, replace placeholders, then run the create-and-test skill script (invokes the cu-skill tool under .github/skills/_shared/). See SKILL.md Step 2 for the baseAnalyzerId table and field-description rules. Keys beginning with '_' (_comment, _optional_*) are documentation aids; the local validator strips them before submission.", + "baseAnalyzerId": "prebuilt-document", + "description": "REPLACE: one-line description of what this analyzer extracts.", + "config": { + "_comment": "estimateFieldSourceAndConfidence is required for any field with method=extract. Other knobs (enableOcr, enableLayout, enableFormula) are optional; defaults are usually fine.", + "estimateFieldSourceAndConfidence": true, + "returnDetails": true, + "_optional_enableOcr": true, + "_optional_enableLayout": true, + "_optional_enableFormula": false + }, + "models": { + "_comment": "Required when using fieldSchema unless the resource has defaults set by running the Sample00_UpdateDefaultsAsync sample. Leaving it set is always safe.", + "completion": "gpt-4.1", + "embedding": "text-embedding-3-large" + }, + "fieldSchema": { + "name": "REPLACE_schema_name_v1", + "description": "REPLACE: schema purpose.", + "fields": { + "exampleExtractField": { + "_comment": "method=extract: copy text verbatim from a specific location. Requires estimateSourceAndConfidence=true.", + "type": "string", + "method": "extract", + "description": "REPLACE: value printed after the 'EXAMPLE LABEL:' anchor near the top of the page. Reference text anchors (labels, headings), not visual appearance.", + "estimateSourceAndConfidence": true + }, + "exampleGenerateField": { + "_comment": "method=generate: AI synthesizes a value from the whole document. Use for summaries, derived values, free-form interpretation. estimateSourceAndConfidence is N/A here.", + "type": "string", + "method": "generate", + "description": "REPLACE: a one-sentence summary of the document's purpose." + }, + "exampleClassifyField": { + "_comment": "method=classify: AI picks one of the enum values. Use for fixed categorical fields.", + "type": "string", + "method": "classify", + "description": "REPLACE: high-level document type.", + "enum": ["invoice", "receipt", "contract", "report", "other"] + }, + "exampleDateField": { + "type": "date", + "method": "extract", + "description": "REPLACE: date printed after the 'DATE:' label. Not to be confused with other dates such as a due date or service date.", + "estimateSourceAndConfidence": true + }, + "exampleAmountField": { + "type": "number", + "method": "extract", + "description": "REPLACE: amount on the row labelled 'TOTAL' in the totals table at the bottom. Excludes any 'Subtotal' or 'Previous Balance' values.", + "estimateSourceAndConfidence": true + }, + "exampleNestedObjectField": { + "_comment": "Nested object (no array). Use when a single logical entity has several sub-fields (e.g. an address block).", + "type": "object", + "method": "extract", + "description": "REPLACE: the multi-line address block under the 'BILL TO:' heading.", + "properties": { + "name": { "type": "string", "method": "extract", "description": "REPLACE: company or person name on the first line." }, + "street": { "type": "string", "method": "extract", "description": "REPLACE: street address on the second line." }, + "city": { "type": "string", "method": "extract", "description": "REPLACE: city on the third line." } + } + }, + "exampleArrayField": { + "_comment": "Array of objects (line items / table rows). Wire key is 'items' (verified against the typed SDK's item_definition serialization).", + "type": "array", + "method": "extract", + "description": "REPLACE: each row of the line-items table.", + "items": { + "type": "object", + "properties": { + "itemCode": { "type": "string", "method": "extract", "description": "REPLACE: value in the ITEM CODE column." }, + "description": { "type": "string", "method": "extract", "description": "REPLACE: value in the DESCRIPTION column." }, + "amount": { "type": "number", "method": "extract", "description": "REPLACE: value in the AMOUNT column." } + } + } + } + } + } +} diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge/SKILL.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge/SKILL.md index acf85279ac93..f3e478429773 100644 --- a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge/SKILL.md +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge/SKILL.md @@ -49,7 +49,89 @@ Always read the relevant page (via `fetch_webpage`) before answering if the refe > **Search tip:** If the above pages don't cover the user's question, search the doc tree at `https://learn.microsoft.com/azure/ai-services/content-understanding/`. +## Field-description rule: the two-stage pipeline + +Custom analyzer extraction is a **two-stage pipeline**: + +1. **Stage 1 — content extraction (OCR + layout).** The service reads the + file and produces structured text plus layout metadata (sections, tables, + headings). The original pixels are *not* what the LLM in stage 2 sees. +2. **Stage 2 — field extraction (LLM).** The LLM reads the stage-1 markdown + and uses your field descriptions to identify values. + +Implications for `fieldSchema.fields[*].description`: + +✅ Reference **text content and structure**: labels (`"Invoice #"`), +section headings (`"Bill To"`), adjacent labels, alternative phrasings, +format examples. + +❌ Do **not** reference visual appearance: colour, font, font size, bold or +italic, or "the box at the top-right" without text anchors. + +Good description: + +> "Invoice issue date, found near the 'Invoice #' label at the top right. +> May also be labelled 'Invoice Date', 'Date', or 'Issued'. Format is +> usually MM/DD/YYYY. Examples: '01/15/2024', 'January 15, 2024'." + +Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) +and [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md). + +## Choosing `baseAnalyzerId` + +Every custom analyzer extends a built-in prebuilt analyzer via +`baseAnalyzerId`. Pick the row that matches the **modality** of the content +you're analyzing (documents, audio, video, image). Typos here are a common +first-time error; the local schema validator (in +[`_shared/SchemaValidator.java`](../_shared/)) rejects any value not in this +table. + +| Content type | `baseAnalyzerId` | +|---|---| +| Documents (PDF, image of a page) | `prebuilt-document` | +| Audio (mp3, wav, m4a) | `prebuilt-audio` | +| Video (mp4, mov) | `prebuilt-video` | +| Image-only analyzer | `prebuilt-image` | + +> ⚠️ **Only modality-level prebuilt analyzers are valid as `baseAnalyzerId` for +> custom analyzers.** `*Search` variants (`prebuilt-documentSearch`, +> `prebuilt-audioSearch`, `prebuilt-videoSearch`), task-specific prebuilt analyzers +> (`prebuilt-invoice`, `prebuilt-receipt`, `prebuilt-idDocument`), and +> `prebuilt-layout` are **not** accepted here — the service returns +> `InvalidBaseAnalyzerId`. Those prebuilt analyzers can still be called directly as +> standalone analyzers via +> `client.beginAnalyze("prebuilt-invoice", ...)`. See the +> [analyzer-reference docs](https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/analyzer-reference#baseanalyzerid). + +Used by [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) +(custom analyzer) and +[`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) +(both inner extractors and the outer classifier). + +## Classify-and-route rule + +When using `config.contentCategories` to classify and route mixed-document +packets: + +1. **Category descriptions follow the same text-anchored rule** as field + descriptions. Describe each category by the text that appears on its + pages (headings, labels), not by visual style. +2. **`config.enableSegment` must be `true`** so the classifier carves the + packet into segments before routing each one. +3. **Inner analyzers must already exist** before the outer classifier is + created. +4. **Category fill rate is per-category**, not packet-wide. A field that + only appears in invoice segments should be evaluated against the number + of invoice segments, not the total number of segments. +5. **No top-level `fieldSchema`** on the outer classifier. The outer + analyzer's job is classification + routing only; field extraction + belongs in the inner analyzers. + +Used by [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md). + ## Related Skills - `cu-sdk-setup` — Set up environment variables for Java SDK samples - `cu-sdk-sample-run` — Run specific Java SDK samples interactively +- `cu-sdk-author-analyzer` — Author + test a custom analyzer for one document type +- `cu-sdk-author-analyzer-classify-route` — Author + test a classify-and-route pipeline for mixed-document packets diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run/SKILL.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run/SKILL.md index 9f79b8b641f7..e7bda6acc88e 100644 --- a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run/SKILL.md +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run/SKILL.md @@ -71,10 +71,12 @@ Gets raw JSON response for custom processing. #### `Sample04_CreateAnalyzer` Creates custom analyzer with field schema for domain-specific extraction. - Key concepts: Field types (string, number, date, object, array), extraction methods (extract, generate, classify) +- **Next step:** to run this workflow on your own folder of documents (validate schema → create → batch test → stdout summary), use the [`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) skill. #### `Sample05_CreateClassifier` Creates classifier to categorize documents (Loan_Application, Invoice, Bank_Statement). - Key concepts: Content categories, segmentation, document routing +- **Next step:** to run this workflow on your own mixed-document packets, use the [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md) skill. #### `Sample16_CreateAnalyzerWithLabels` Builds an analyzer using **labeled training data** loaded from Azure Blob Storage. The repo ships labeled receipt data at `src/samples/resources/receipt_labels/` (`*.jpg`, `*.jpg.labels.json`, optional `*.jpg.result.json`). diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup/SKILL.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup/SKILL.md index 2dcfcc91bc85..2928efc7ef16 100644 --- a/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup/SKILL.md +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup/SKILL.md @@ -9,6 +9,14 @@ Set up your Java environment to use the Azure AI Content Understanding SDK and r > **[COPILOT INTERACTION MODEL]:** This skill is designed to be interactive. At each step marked with **[ASK USER]**, pause execution and prompt the user for input or confirmation before proceeding. Do NOT silently skip these prompts. Use the `ask_questions` tool when available. +> **[COPILOT] Step numbering is stable contract.** Authoring skills +> ([`cu-sdk-author-analyzer`](../cu-sdk-author-analyzer/SKILL.md) and +> [`cu-sdk-author-analyzer-classify-route`](../cu-sdk-author-analyzer-classify-route/SKILL.md)) +> route the user to specific steps here on probe failures — e.g. "endpoint +> `MISSING` → Step 4 (env vars)". When renumbering or restructuring this +> skill, update the routing tables in those skills (or keep step numbers +> stable). + ## Prerequisites Before starting, ensure you have: diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md index 1d588e2f59f9..60ab39226773 100644 --- a/sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md @@ -10,6 +10,24 @@ ### Other Changes +- Added GitHub Copilot skills under `.github/skills/` to help users + iteratively author custom analyzers in VS Code with Copilot: + - **`cu-sdk-author-analyzer`** — author and refine a custom analyzer + for a single document type (layout extraction → schema drafting → + validation → batch test → agent review → refine cycle). Document + modality only today; audio/video/image are planned. + - **`cu-sdk-author-analyzer-classify-route`** — author and refine a + classify-and-route pipeline for mixed-document packets (e.g. + invoice + bank statement + loan application in one PDF), with + per-category agent review. + + Both skills delegate to a small `cu-skill` Maven tool under + `.github/skills/_shared/` that exposes three subcommands — + `extract-layout`, `create-and-test`, and `create-and-test-router` — + and a pure-Java `SchemaValidator` that catches structural mistakes + (unknown `baseAnalyzerId`, missing `fieldSchema`, malformed + `contentCategories` routes) before a service round-trip. + ## 1.1.0-beta.2 (2026-06-11) ### Features Added diff --git a/sdk/contentunderstanding/azure-ai-contentunderstanding/README.md b/sdk/contentunderstanding/azure-ai-contentunderstanding/README.md index 2f277f82f41d..d520bca50a23 100644 --- a/sdk/contentunderstanding/azure-ai-contentunderstanding/README.md +++ b/sdk/contentunderstanding/azure-ai-contentunderstanding/README.md @@ -552,6 +552,24 @@ ContentUnderstandingClient client = new ContentUnderstandingClientBuilder() For more information, see [Azure SDK for Java logging][logging]. +## What's New + +See [CHANGELOG.md][changelog] for the full release history. + +Recent highlights: + +- **Authoring skills for GitHub Copilot.** Two new skills, + [`cu-sdk-author-analyzer`][cu_sdk_author_analyzer_skill] (single document + type) and + [`cu-sdk-author-analyzer-classify-route`][cu_sdk_author_analyzer_classify_route_skill] + (mixed-document packets), guide the user and Copilot through an iterative + schema → test → review cycle for authoring custom Content Understanding + analyzers. See the [GitHub Copilot Skills](#github-copilot-skills) section + below. +- **`LlmInputHelper.toLlmInput` helper** for converting `AnalysisResult` into + LLM-friendly Markdown with YAML front matter. See + [Sample05_CreateClassifierAsync][sample05]. + ## GitHub Copilot Skills This package includes [GitHub Copilot][github_copilot] skills under `.github/skills/` that provide interactive, AI-assisted workflows for common tasks. In VS Code, Copilot can use these skills to help with environment setup, running samples, and understanding the service. @@ -563,6 +581,8 @@ This package includes [GitHub Copilot][github_copilot] skills under `.github/ski | [**cu-sdk-setup**][cu_sdk_setup_skill] | Interactive environment setup — creates and configures your `.env` file with endpoint, credentials, and model deployment settings | In VS Code Copilot Chat, ask: *"Set up my Java environment for Content Understanding"* or reference the skill directly | | [**cu-sdk-sample-run**][cu_sdk_sample_run_skill] | Guided sample runner — helps you build the SDK, configure credentials, and run specific samples with Maven | Ask: *"Run Sample02_AnalyzeUrl"* or *"Run the invoice analysis sample"* | | [**cu-sdk-common-knowledge**][cu_sdk_common_knowledge_skill] | Domain knowledge reference — answers questions about Content Understanding concepts, analyzers, field schemas, API operations, and Java SDK usage | Ask: *"What prebuilt analyzers are available?"* or *"How do I create a custom analyzer?"* | +| [**cu-sdk-author-analyzer**][cu_sdk_author_analyzer_skill] | Iteratively author and test a custom analyzer for a single **document** type — layout extraction, schema drafting, local validation, batch test, agent review, and refine cycle. Document modality only today; audio/video/image are planned. | Ask: *"Help me author a custom analyzer for these invoices"* | +| [**cu-sdk-author-analyzer-classify-route**][cu_sdk_author_analyzer_classify_route_skill] | Iteratively author and test a classify-and-route pipeline for mixed-document packets (e.g. invoice + bank statement + loan application in one PDF). Includes per-category agent review and refinement of both classifier descriptions and inner-schema field descriptions. | Ask: *"Help me author an analyzer for this mixed financial packet"* | ### Using Skills in VS Code @@ -635,4 +655,8 @@ This project has adopted the [Microsoft Open Source Code of Conduct][code_of_con [cu_sdk_setup_skill]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup [cu_sdk_sample_run_skill]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run [cu_sdk_common_knowledge_skill]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge +[cu_sdk_author_analyzer_skill]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer +[cu_sdk_author_analyzer_classify_route_skill]: https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route +[changelog]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md +[sample05]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/contentunderstanding/azure-ai-contentunderstanding/src/samples/java/com/azure/ai/contentunderstanding/samples/Sample05_CreateClassifierAsync.java [file_issue]: https://github.com/Azure/azure-sdk-for-java/issues/new?labels=Cognitive%20-%20Content%20Understanding&title=[ContentUnderstanding]%20&body=%23%23%20Library%20Version%0A%0A%23%23%20Repro%20Steps%0A%0A%23%23%20Expected%20Result%0A%0A%23%23%20Actual%20Result