feat(contentunderstanding): Copilot skills for authoring custom analyzers#49672
feat(contentunderstanding): Copilot skills for authoring custom analyzers#49672chienyuanchang wants to merge 8 commits into
Conversation
…nalyzers Add two GitHub Copilot skills under .github/skills/ that guide users through the iterative cycle of authoring custom Content Understanding analyzers in VS Code: - cu-sdk-author-analyzer — single-document-type authoring - cu-sdk-author-analyzer-classify-route — classify-and-route pipelines Both skills delegate to a small cu-skill Maven tool under .github/skills/_shared/ that exposes three subcommands (extract-layout, create-and-test, create-and-test-router) and a pure-Java SchemaValidator (no com.azure.* deps) so structural mistakes are caught before a service round-trip. Mirrors the .NET skills shipped in Azure/azure-sdk-for-net#60394 and the Python skills in Azure/azure-sdk-for-python#47218. Live-tested against a real CU resource with mixed_financial_docs.pdf: - extract-layout produces .layout.{json,md} - create-and-test reports per-field fill rate + avg confidence and cleans up the analyzer when --ephemeral is set - create-and-test-router creates N inner + 1 outer analyzer, prints a category-aware summary, and cleans up all four New unit tests: - SchemaValidatorTest (18 cases) mirrors Python's test_skills_shared_schema_validator.py and the .NET SkillSchemaValidatorTests.cs: valid single-type and classify-route schemas, every rejection path, validateFile error handling, KNOWN_BASE_ANALYZER_IDS allow-list sanity, and a purity guard that fails if the validator source ever pulls in com.azure.* or HTTP namespaces. README: - New 'What's New' section and two new entries in the existing 'GitHub Copilot Skills' table. CHANGELOG: - Entry under 1.1.0-beta.3 (Unreleased) describing the new skills. Build: - mvn -B -DskipTests compile: clean - mvn -B test: 18/18 tests pass The Maven module is intentionally NOT a child of azure-client-sdk-parent and is NOT referenced from the package POM, so it has zero effect on the published azure-ai-contentunderstanding artifact.
CI cspell check flagged abbreviated locals and a few domain acronyms in the skill-tooling sources. Replaces them with the full words used elsewhere in the file so the dictionary does not need a new entry. - confs -> confidences (CreateAndTestCommand, CreateAndTestRouterCommand) - fobj -> fieldObj, vobj -> valueObj (CreateAndTestCommand) - IMDS -> metadata-service (ExtractLayoutCommand comment) - prebuilts -> prebuilt analyzers (SchemaValidator javadoc) No behavior change. All 18 unit tests still pass.
…le paths Verify-Links flagged 15 broken relative links in the two analyzer skill docs. The Python source skills referenced 'samples/sample_*.py' files at the package root; the Java port mechanically copied the link structure but the Java layout is different: - Java samples live at src/samples/java/com/azure/ai/contentunderstanding/samples/, not samples/. - Java does not have the per-sample .md companion files Python ships (Sample06_GetAnalyzer.md, Sample02_AnalyzeUrl.md, etc.) — only the .java source files. Rewrites all 15 links to point at the correct .java source under src/samples/java/. All 10 distinct sample files referenced were verified to exist on disk before the rewrite.
…ts from Python Three changes for parity with Python PR #47218: 1. **summarizeRouted denominator bug fix** (was a port-time regression). The denominator for a per-field fill rate must be the per-category segment count, NOT the per-field row count. Two invoice segments where only one has TotalAmount correctly reports 50% (1/2 segments), not 100% (1/1 rows). This matches Python and .NET; the same bug existed in JS and is being fixed on its PR. The comment claiming 'Mirrors Python/.NET semantics' was misleading — fixed in this commit. 2. **New CLI-helper test classes** — CreateAndTestCommandTest and CreateAndTestRouterCommandTest. Together they cover the 4 pure-helper tests Python ships (summarize leaf-row flattening, summarizeRouted per-category denominator, summarizeRouted zero-fill, wireInnerIds alias substitution). The denom regression above was discovered by these tests. Python ships 10 CLI-helper tests total; the remaining 6 either test argparse-specific behaviour (--help smoke, monkey-patched client) or test code structured differently in Java (error-tuple API), so they don't translate 1:1. We cover the substantive behaviour. 3. **Cross-skill SKILL.md updates** that Python's PR #47218 also shipped: a two-stage-pipeline section + a baseAnalyzerId table in cu-sdk-common-knowledge, 'Next step' cross-refs in cu-sdk-sample-run, and a 'step numbering is contract' callout in cu-sdk-setup.
CI cspell caught 3 occurrences in cu-sdk-common-knowledge/SKILL.md introduced by the earlier Python cross-skill port. Also fix one occurrence in SchemaValidatorTest.java comment that was already there but at the same risk. Rephrase all as 'prebuilt analyzers' (matches the surrounding prose and the phrasing already used in SchemaValidator.java's javadoc after the earlier cspell round).
There was a problem hiding this comment.
Pull request overview
This PR adds GitHub Copilot skills and a small standalone Java CLI tool to support an iterative “layout → schema → validate → create → batch test → review” workflow for authoring Azure AI Content Understanding custom analyzers (single-type and classify-and-route) directly from VS Code.
Changes:
- Added two new Copilot skills (
cu-sdk-author-analyzer,cu-sdk-author-analyzer-classify-route) plus templates to guide analyzer authoring workflows. - Introduced a standalone Maven module under
.github/skills/_shared/that implements thecu-skillCLI (layout extraction + create/test loops) and a pure-JacksonSchemaValidator. - Updated package docs (README + CHANGELOG) and cross-linked existing skills to the new authoring workflows.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/contentunderstanding/azure-ai-contentunderstanding/README.md | Adds “What’s New” highlights and documents the two new authoring skills with links. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/CHANGELOG.md | Records the addition of the new authoring skills and shared CLI tooling. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-setup/SKILL.md | Notes step-numbering stability as a contract for the new authoring skills to route users correctly. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-sample-run/SKILL.md | Adds “next step” pointers to the new authoring skills from relevant samples. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-common-knowledge/SKILL.md | Adds domain guidance used by authoring skills (two-stage pipeline, baseAnalyzerId rules, classify-and-route rules). |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/templates/schema_template.json | Starter JSON schema template for single-type analyzer authoring. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer/SKILL.md | New skill that guides the single-document-type authoring loop. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/templates/classifier_template.json | Starter JSON schema template for outer classifier in classify-and-route. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/cu-sdk-author-analyzer-classify-route/SKILL.md | New skill that guides the mixed-packet classify-and-route authoring loop. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/SchemaValidatorTest.java | Unit tests for the pure-Jackson SchemaValidator (including purity guard). |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommandTest.java | Unit tests for router summary logic and schema wiring helper. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/test/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommandTest.java | Unit tests for single-type summary logic (leaf field flattening, lowest-confidence reporting). |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/SchemaValidator.java | Adds pure-Jackson schema validation to fail fast before service calls. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/ExtractLayoutCommand.java | Implements layout extraction via prebuilt-documentSearch, writing .layout.{json,md} files. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestRouterCommand.java | Implements create/test loop for classify-and-route (N inner + 1 outer) and category-aware summary. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/CreateAndTestCommand.java | Implements create/test loop for single-type analyzers and per-field summary output. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/src/main/java/com/azure/ai/contentunderstanding/skills/Cli.java | Adds CLI subcommand dispatcher for the cu-skill tool. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/README.md | Documents _shared as a library/tooling directory (not a skill), with build/run instructions. |
| sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/_shared/pom.xml | Adds standalone Maven module definition for building and running the cu-skill CLI. |
… template The template comment referenced create_and_test.py — the Python source-of-truth filename copied into the port. In this Java skill, the concrete tool is the cu-skill Maven module under .github/skills/_shared/, so the template now references the create-and-test skill script and points readers at the tool directory. Same drift being fixed in the parallel .NET (#60394) and JS (#39137) PRs. No functional or public-API change.
Addresses all 9 comments from the Copilot code reviewer on #49672. Correctness fixes: 1. CreateAndTestRouterCommand: rewrite wireInnerIds() to fix a bug introduced by keying the alias lookup on the *category name* instead of on the *analyzerId value*. Previous behavior: contentCategories: loan: analyzerId: loan_application <- looked up by "loan", missed caused wireInnerIds to silently leave the placeholder in place when the category name and analyzerId value differed. New behavior mirrors Python (_patch_outer_analyzer_ids) and .NET (WireInnerIds): - Skip categories that omit analyzerId (classification-only bucket). - Skip analyzerId values that start with "prebuilt-" — those are Azure-side prebuilts and don't need an --inner-schema. - Any other value must match an alias in aliasToId; otherwise return a validation error naming the missing alias + category. - Every alias in aliasToId must be referenced by some category (typo check); otherwise return an "unused" error. Signature changed from `ObjectNode wireInnerIds(...)` to `WireResult wireInnerIds(...)` where WireResult carries both the patched schema and any errors. Caller now checks errors before proceeding with analyzer creation. Removed the old "cross-check" block in run() — wireInnerIds() now handles that check in one place with the correct behavior. Compile-error fixes (High): 2. CreateAndTestRouterCommand.java: drop unused imports com.azure.core.util.polling.SyncPoller and com.azure.core.util.BinaryData. 3. CreateAndTestCommand.java: drop unused import java.util.HashMap. Documentation fixes (Medium — .NET-copy-paste drift): 4. cu-sdk-author-analyzer/SKILL.md — prerequisites: "built DLL" wording was copied from the .NET port. Java runs the tool via Maven, so changed to "Maven + JDK 17+". 5. cu-sdk-author-analyzer-classify-route/SKILL.md — same fix. 6. cu-sdk-author-analyzer/SKILL.md — cp path used the .NET package name (Azure.AI.ContentUnderstanding); changed to the Java package name (azure-ai-contentunderstanding). 7. cu-sdk-author-analyzer-classify-route/SKILL.md — same fix. 8. cu-sdk-author-analyzer/SKILL.md — reference to `AnalysisResultExtensions.ToLlmInput` (a .NET name) changed to `LlmInputHelper.toLlmInput` which is the actual Java sample name. 9. cu-sdk-author-analyzer-classify-route/SKILL.md — same fix. Test coverage: Rewrote wireInnerIdsSubstitutesMatchingAliases with category names deliberately different from analyzerId values (catches the category-name-vs-alias-value regression) and added: - wireInnerIdsKeepsPrebuiltAnalyzerIdsAsIs (prebuilt-invoice etc. bypass the aliasToId check) - wireInnerIdsReportsMissingAliasAsError - wireInnerIdsReportsUnusedAliasAsError - wireInnerIdsAllowsCategoriesWithoutAnalyzerId ("other" catch-all bucket must not error) Verified: mvn -B test in .github/skills/_shared/ passes 26/26.
Ports the correctness fix from the sibling Java PR (Azure/azure-sdk-for-java#49672, bca631f00b4) to the JS skill. Bug: wireInnerIds keyed the alias lookup on the *category name* instead of on the *analyzerId value*, so a schema like contentCategories: loan_bucket: analyzerId: loan_application silently left the placeholder in place because `aliasToId.has("loan_bucket")` returned false even when `aliasToId.has("loan_application")` was true. The cross-check in runCreateAndTestRouter had the same defect and also rejected valid `prebuilt-*` values (Azure service prebuilts don't need an --inner-schema). New behavior mirrors Python (_patch_outer_analyzer_ids), .NET (WireInnerIds), and Java (WireResult): - Skip categories that omit analyzerId (classification-only "other" bucket). - Skip analyzerId values that start with "prebuilt-" (service prebuilts). - Any other value must match an alias in aliasToId; otherwise return an error naming the missing alias + category. - Every alias in aliasToId must be referenced by some category (typo check); otherwise return an "unused" error. Signature changed from `wireInnerIds(...) -> Record<string, unknown>` to `wireInnerIds(...) -> WireResult` where WireResult carries both patched schema and any errors. Caller in runCreateAndTestRouter now checks errors before proceeding. Test coverage: rewrote wireInnerIdsSubstitutesMatchingAliases with category names deliberately different from analyzerId values, and added: - keeps 'prebuilt-*' analyzerId values as-is - returns error for unknown alias - returns error for unused alias - allows categories without analyzerId Verified: `npm test` in .github/skills/_shared/ passes 27/27 (up from 23).
|
Thanks Copilot. Pushed bca631f which addresses all 9 comments. Correctness fixes
Compile-error fixes
Documentation fixes (.NET → Java copy-paste drift) 4, 5. Prerequisites: Test coverage Rewrote
Verified: |
Audit turned up 3 places in the Java port where the Python-source
filename `samples/sample_update_defaults.py` was left in place:
- schema_template.json (`models._comment`)
- classifier_template.json (`models._comment`)
- CreateAndTestCommand.java (a user-facing [WARN] on missing
models.completion)
Replaced with `Sample00_UpdateDefaultsAsync`, the actual Java sample
that configures resource-level model defaults (matching the link
convention already used in the SKILL.md files). Same drift exists in
the sibling .NET and JS PRs and is being fixed there too.
Verified: `mvn -B test` in .github/skills/_shared/ still 26/26 passing.
| if (opts.schemaDir != null) { | ||
| Path dir = Paths.get(opts.schemaDir); | ||
| if (!Files.isDirectory(dir)) { | ||
| System.err.println("--schema-dir is not a directory: " + dir); | ||
| return 2; | ||
| } | ||
| try (var stream = Files.list(dir)) { | ||
| stream | ||
| .filter(p -> p.getFileName().toString().endsWith(".json")) | ||
| .sorted() | ||
| .forEach(p -> aliasToPath.putIfAbsent( | ||
| stripExtension(p.getFileName().toString()), p)); | ||
| } | ||
| } |
| case "--input": | ||
| input = args[++i]; | ||
| break; | ||
| case "--output": | ||
| output = args[++i]; | ||
| break; | ||
| case "--analyzer-id": | ||
| analyzerId = args[++i]; | ||
| break; |
| case "--schema": | ||
| o.schema = args[++i]; | ||
| break; | ||
| case "--input": | ||
| o.input = args[++i]; | ||
| break; | ||
| case "--output": | ||
| o.output = args[++i]; | ||
| break; | ||
| case "--analyzer-id": | ||
| o.analyzerId = args[++i]; | ||
| break; | ||
| case "--iterations": | ||
| o.iterations = Integer.parseInt(args[++i]); | ||
| break; |
| case "--outer-schema": | ||
| o.outerSchema = args[++i]; | ||
| break; | ||
| case "--inner-schema": | ||
| o.innerSchemas.add(args[++i]); | ||
| break; | ||
| case "--schema-dir": | ||
| o.schemaDir = args[++i]; | ||
| break; | ||
| case "--input": | ||
| o.input = args[++i]; | ||
| break; | ||
| case "--output": | ||
| o.output = args[++i]; | ||
| break; | ||
| case "--analyzer-id": | ||
| o.analyzerId = args[++i]; | ||
| break; |
| <!-- Skip Checkstyle / SpotBugs / javadoc enforcement; this is internal tooling. --> | ||
| <checkstyle.skip>true</checkstyle.skip> | ||
| <spotbugs.skip>true</spotbugs.skip> | ||
| <maven.javadoc.skip>true</maven.javadoc.skip> |
feat(contentunderstanding): Copilot skills for authoring custom analyzers
Summary
Adds two GitHub Copilot skills under
sdk/contentunderstanding/azure-ai-contentunderstanding/.github/skills/that guide users through the iterative cycle of authoring custom Content Understanding analyzers directly inside VS Code:cu-sdk-author-analyzercu-sdk-author-analyzer-classify-routeBoth skills delegate to a small
cu-skillMaven tool under.github/skills/_shared/that exposes three subcommands:extract-layout— pulls document structure into.layout.{json,md}so Copilot has ground-truth context for schema drafting.create-and-test— validates a schema locally, creates the analyzer, batch-tests inputs, prints a per-field fill-rate + avg-confidence summary, and (with--ephemeral) cleans up.create-and-test-router— same loop for classify-and-route: creates N inner extractors + 1 outer classifier, wirescontentCategories[*].analyzerId, runs, and prints a category-aware summary.A pure-Java
SchemaValidator(Jackson only, nocom.azure.*imports) catches structural mistakes (unknownbaseAnalyzerId, missingfieldSchema, malformedcontentCategoriesroutes) before a service round-trip.This mirrors the .NET skills shipped in Azure/azure-sdk-for-net#60394 and the Python skills in Azure/azure-sdk-for-python#47218.
Live test
Tested against a real CU resource with
mixed_financial_docs.pdf(an invoice + bank statement + loan application):The router variant created 3 inner + 1 outer analyzer, classified 3 segments, extracted all fields, and cleaned up all 4 analyzers (~59 s end-to-end).
What's in the PR
_shared/SchemaValidator.javacom.azure.*deps; purity-guarded by a unit test_shared/ExtractLayoutCommand.java_shared/CreateAndTestCommand.java_shared/CreateAndTestRouterCommand.java_shared/Cli.java_shared/pom.xml+_shared/README.mdcu-sdk-author-analyzer/SKILL.md+ templatecu-sdk-author-analyzer-classify-route/SKILL.md+ templateSchemaValidatorTest.javaREADME.mdCHANGELOG.mdTest results
Coverage:
analyzerId(catch-all "other" bucket)baseAnalyzerIdrejected (catches theprebuilt-documentAnalyzertypo class)fieldSchemaon non-classifierfieldsobject, unknown fieldtype/methodobject.propertiesandarray.itemsrecursionfieldSchema→ rejectedenableSegment: true→ rejecteddescription→ rejectedvalidateFile: missing file, invalid JSON, valid round-tripKNOWN_BASE_ANALYZER_IDSallow-list sanity checkSchemaValidator.javaever pulls incom.azure.*/java.net.http/HttpURLConnectionBugs fixed during live test
.envvalues typically wrap in"…"; rawexportkeeps the quotes. Added areadEnv()helper that strips one layer of surrounding quotes (mirrors the .NET fix).DefaultAzureCredentialBuilderdoes not exposeexcludeCredentialsin the Java SDK (unlike .NET). Built a focusedChainedTokenCredentialBuilderchain (EnvironmentCredential→AzureCliCredential) so the IMDS probe doesn't stall on dev boxes (~30 s wait on WSL).BinaryDataprotocol overload ofbeginAnalyzeBinary/beginCreateAnalyzerso we can keep the wire format (valueString/valueNumber/...) and unwrap the LRO envelope{id,status,result,usage}to match Python'spoller.result()output and the .NET skill output.results.size()(doc count) instead ofrows.size()(per-field row count), which inflated array-leaf fields to 1300%. Switched to per-field row count to match Python and .NET semantics.Backward compatibility
No public API changes. The skills tree is opt-in tooling that lives under
.github/skills/and is only loaded by Copilot when a user explicitly asks. Thecu-skillMaven module is standalone — not a child ofazure-client-sdk-parent, not referenced from the package POM — so it doesn't affect normal package builds or the publishedazure-ai-contentunderstandingartifact.Checklist
mvn -B -DskipTests compilecleanmvn -B test18/18 passing