Sync eng/common directory with azure-sdk-tools for PR 16052#49752
Merged
Conversation
Rewrite all eval pipeline glue scripts as dependency-free ESM with node:test unit tests, under eng/common/scripts/eval/: - collect-stimuli.js (+ lib/glob.js) <- Split-EvalSuite.ps1 - invoke-eval-shard.js + lib/verdict.js <- Invoke-EvalShard.ps1 + EvalVerdict.Helpers.ps1 - build-eval-summary.js <- Build-EvalSummary.ps1 - sync-eval-git-repo.js <- Sync-EvalGitRepo.ps1 - init-eval-git-fixtures.js <- Initialize-EvalGitFixtures.ps1 Delete all .ps1 + Pester tests; rewire vally-eval.yml to invoke the JS via node; update README/.vally.yaml/eval-comment references. 40 node:test cases pass.
Extract the monolithic vally-eval.yml into modular, parameterized templates under eng/common/pipelines/templates (synced to other repos), leaving a thin repo-local entrypoint: - steps/vally-build-mock.yml: build + publish the mock MCP DLL - steps/vally-collect-stimuli.yml: glob eval files -> shard matrix (repo + optional common root) - steps/vally-run-shard.yml: npm ci + download mock + prime git fixtures + run shard + publish JUnit/transcripts - steps/vally-summarize.yml: aggregate shard JUnit + gate rollup - stages/archetype-vally-eval.yml: Eval stage (BuildMock || Detect -> RunShard matrix, customMcpBuildJobs hook) + Summary stage Drop the redundant global 'npm install -g @github/copilot-sdk': it is a locked transitive dep of @microsoft/vally-cli, so npm ci in eng/skill-eval already installs the executor (D9: pin via lockfile, no global installs). Migrate all jobs to the managed pool (name: $(LINUXPOOL) / image: $(LINUXVMIMAGE) / os: linux); the repo-local entrypoint supplies the image.yml variables and the variable group.
…ng lib modules
Make the shared archetype/templates MCP-agnostic so the skill-compliance and
live workflow-scenario evals can reuse the same shard fan-out as the mock tier:
- collect-stimuli.js: derive file-mode shard names from the full relative path
so per-skill <skill>/evals/trigger.eval.yaml files no longer collide.
- vally-build-mock.yml -> vally-build-mcp.yml: build a LIST of MCP servers and
publish them under a common stage root.
- vally-collect-stimuli.yml: accept an explicit list of eval glob patterns.
- vally-run-shard.yml: rename mock* params to mcp*; optionally wrap the eval in
AzureCLI@2 (live tier) so the agent's DevOps calls inherit the service
connection, else run as a plain script (hermetic tiers).
- archetype-vally-eval.yml: swap mock-specific params for mcpBuilds/mcpArtifact/
mcpStageRoot/patterns/azureServiceConnection and wire them through.
Add two thin entrypoints:
- eng/pipelines/skill-eval-shards.yml: sharded variant of skill-eval.yml
(builds cli + mock, one shard per skill).
- eng/pipelines/live-eval.yml: nightly scheduled live run (builds cli,
authenticated via opensource-api-connection).
Fix the pipeline ERR_MODULE_NOT_FOUND failure: the root .gitignore's Python
`lib/` rule silently excluded eng/common/scripts/eval/lib/{glob,verdict}.js, so
the pipeline checkout was missing them. Add a targeted negation and track the
two modules.
Defaults preserve the existing mock vertical; vally-eval.yml only updates the
renamed build-template trigger paths.
Pipeline (1ES migration, localized to the archetype): - archetype-vally-eval.yml now extends /eng/pipelines/templates/stages/1es-redirect.yml (chained extends, like archetype-sdk-tool-dotnet.yml). Adds Use1ESOfficial param. - 1ES Official forbids the `publish:` shortcut, so the three artifact publishes move to job-level templateContext.outputs: MCP server artifact on BuildMcp; per-shard eval-result-$(shardName) (JUnit, always) and eval-debug-$(shardName) (transcripts, on failure) on the RunShard matrix job. - Drop the publish steps from vally-build-mcp.yml / vally-run-shard.yml (keep the staging script steps) and remove the now-unused stageRoot/artifactName params from vally-build-mcp.yml. Clears the 17 [1ES PT] warnings; CG is opt-in under 1ES. Shard gate (invoke-eval-shard.js): - Demote the two "passing verdict" warnings (hadExecutionErrors and non-zero vally exit) from ##vso build warnings to plain log lines. Both are the same expected post-run agent-teardown noise and say nothing about the eval; the results.jsonl verdict is authoritative. Removes 8 confusing build annotations. - Replace em-dash with ASCII hyphen in the task.logissue strings (rendered as mojibake in ADO annotations). Earlier review fixes (carried in this PR): - collect-stimuli.js: guard against area-mode shard-name collisions (matches file mode); add a regression test. - vally-run-shard.yml: prime git fixtures from this shard's own evalArgs (--pattern per -e file) instead of the built-in defaults.
The 1ES Official managed image blocks direct egress to registry.npmjs.org,
so the shard's `npm ci` failed with EPERM ("rejected by your operating
system"), which cascaded into the collect-JUnit find errors. Restore the
Vally CLI through the allowlisted azure-sdk Azure Artifacts npm mirror via
an authenticated .npmrc (the same pattern typespec-author-eval-setup uses).
…erridden params and mcpBuilds
Adds eng/common/vally-eval (renamed from skill-eval) so the pinned @microsoft/vally-cli lockfile syncs via eng/common. Repoints the Vally eval pipeline (vally-eval.yml, skill-eval-2.yml triggers; vally-run-shard skillEvalDir default; invoke-eval-shard comment) to the new path. eng/skill-eval is left intact for legacy GitHub Actions and the going-away skill-eval.yml; it can be removed later.
Co-locates the custom-MCP override step template with the other vally-* eval step templates in eng/common/pipelines/templates/steps and repoints the archetype useCustomMcp branch + comments.
…plates to eval-*, extract generate-eval-matrix + eval-shard jobs, merge eval package.json, drop variable group, add maxParallel
- Replace MCP build coupling with a single mcpSetupTemplate path param; move the actual builds to repo-local eng/pipelines/eval/mcp-mock.yml + mcp-live.yml and delete the shared eval-build-mcp.yml. - Remove azureServiceConnection param; the live tier's AzureCLI@2 wrapping now lives in a repo-local evalInvokeTemplate (eval-invoke-live.yml). Default is the hermetic node runner eval-invoke.yml. This single run-time hook also covers chatbot/custom-MCP startup. - Convert shardTimeoutInMinutes from a param to an entrypoint-declared variable (hermetic 20, live 45); document the required-variable contract in the archetype + shard job. - Pass vallyRoot explicitly from every entrypoint; update trigger path filters for the new templates. - Fix sync-eval-git-repo.js cold-cache clone: init + fetch <ref> + checkout FETCH_HEAD so branches, tags, and SHAs all resolve (matches the refresh path), not just the remote default branch.
Audit and remove pipeline parameters that no caller overrides, inlining the constant value instead. Make the archetype evalGlobs default explicit and verified equivalent to the collect-stimuli.js fallback. Fix latent shardBy and githubTokenVariable passthrough bugs.
Make archetype-eval.yml a pure orchestrator (only job-template references, no inline steps) and fold each single-use steps template into its owning job template. Add build-mcp.yml and eval-summarize.yml job templates; delete the merged eval-collect-stimuli, eval-run-shard, and eval-summarize step templates. Only the two swap-point step templates (mcpSetupTemplate, evalInvokeTemplate) remain. Update trigger path-filters to templates/jobs/**.
Reduce oversized comment blocks to terse basic summaries: collapse multi-paragraph JS file headers, drop ported-from-PowerShell references and why/teardown rationale, and remove Phase/D2/D3 jargon from the pipeline YAML headers.
Fail fast in collect-stimuli when a discovered eval path contains whitespace, so a bad path errors at matrix time instead of mis-splitting the space-delimited -e args downstream. Add a comment in build-eval-summary flagging the regex coupling to Vally's JUnit output.
1ES requires job timeoutInMinutes to be an integer at compile time, so it cannot read a runtime variable macro. Thread shardTimeoutInMinutes as a number parameter (default 20) from each entrypoint through the archetype to the shard job; live tier sets 45.
…hard/MCP seams - Fold eval-invoke-live.yml into eval-invoke.yml via UseAzSdkAuthentication flag (AzureCLI@2 vs hermetic script); delete the live copy - Derive npm --prefix from the script dir in invoke-eval-shard.js; drop the skill-eval-prefix arg/param everywhere - Make vallyRoot required (no default) in archetype + shard; thread UseAzSdkAuthentication through - Move checkout into the BuildMcp job; remove it from the MCP setup templates; make the example MCP template a mock/live if-else - Add pre/post eval step hooks and blank-line spacing in eval-shard.yml
- Add eval-hook-example.yml placeholder documenting pre/post-eval hooks; surface commented preEvalTemplate/postEvalTemplate in both entrypoints. - Parameterize the pass-rate gate as a threshold param (default 0.8) threaded entrypoint -> archetype -> eval-shard -> eval-invoke; surface it explicitly per tier. - Make mcpSetupTemplate required in archetype-eval.yml (example default was tools-repo specific).
- Merge mcp-mock.yml/mcp-live.yml into eval-mcp-setup.yml with an McpServer if/else selector threaded through build-mcp and the archetype. - Add opt-in toolsRepo param (default '') so repos outside azure-sdk-tools can side-checkout the tools sources to build the MCP; no-op for the tools repo.
The archetype forwarded toolsRepo/mcpServer without declaring toolsRepo (ADO 'Key not found' compile error) and left mcpServer without a default despite being optional. Declare toolsRepo (default '') and default mcpServer to ''.
…tifacts, TestType rename, exec.js note
…or on missing path
azure-sdk-automation
Bot
added
EngSys
helen229
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync eng/common directory with azure-sdk-tools for PR Azure/azure-sdk-tools#16052 See eng/common workflow