Skip to content

Commit 9409edc

Browse files
ashragrawalclaude
andcommitted
Document evaluate env vars (auth token + model overrides) in FAQ and setup guide
- FAQ Q2/Q3: A365_MCP_AUTH_TOKEN (preferred over --auth-token to avoid exposing the token in process listings / shell history) and the A365_EVAL_COPILOT_MODEL / A365_EVAL_CLAUDE_MODEL model overrides. - Setup instructions: an environment-variable table in Step 4, and RULE 7 now allows passing the token via the env var. Follow-up to PR #362 review [5] and [9]: surfaces the env vars where users actually look, not just in --help / code comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent cc704a4 commit 9409edc

2 files changed

Lines changed: 17 additions & 3 deletions

File tree

docs/agent365-guided-setup/a365-evaluate-faq.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,15 +27,17 @@ Microsoft's role is supplying the `a365` CLI that orchestrates steps 1–2. Step
2727
|---|---|---|
2828
| Runtime data from your MCP server (live payloads, user requests) | **No** | Only the static schema from `tools/list` |
2929
| Personal data of your MCP server's end users | **No** | Schemas describe tool APIs, not user data |
30-
| Auth token supplied via `--auth-token` | **No** (AI never sees it) | Token is held in memory, sent only as the HTTP `Authorization` header to your server, and never logged or passed to the coding agent |
30+
| Auth token supplied via `--auth-token` or `A365_MCP_AUTH_TOKEN` | **No** (AI never sees it) | Token is held in memory, sent only as the HTTP `Authorization` header to your server, and never logged or passed to the coding agent. Prefer the `A365_MCP_AUTH_TOKEN` environment variable over the flag — it keeps the token out of process listings (`ps` / Task Manager) and shell history; the CLI warns once if you use the flag. |
3131
| Telemetry or usage data | **No** | This feature adds no telemetry to the base CLI |
3232
| Output report or checklist JSON | **No** (not sent anywhere) | Files are written to the `--output-dir` you specify and stay on your machine |
3333

3434
---
3535

3636
## Q3. Which AI model performs the evaluation, and who controls it?
3737

38-
By default the evaluation uses **`claude-haiku-4-5`**, invoked through whichever coding agent CLI you have installed (GitHub Copilot CLI or Claude Code CLI). You can override the engine with `--eval-engine github-copilot` or `--eval-engine claude-code`.
38+
By default the evaluation uses **Claude Haiku 4.5**, invoked through whichever coding agent CLI you have installed (GitHub Copilot CLI or Claude Code CLI). You can override the engine with `--eval-engine github-copilot` or `--eval-engine claude-code`.
39+
40+
To run a **different model** without waiting for a CLI update, set an environment variable before the run: `A365_EVAL_COPILOT_MODEL` for GitHub Copilot (needs an exact model ID, e.g. `claude-haiku-4.5`) or `A365_EVAL_CLAUDE_MODEL` for Claude Code (accepts an alias, e.g. `haiku`).
3941

4042
The CLI that runs the model is **yours**, installed from npm by you, authenticated with your credentials. Microsoft does not host or resell the model API call — it is made directly by your CLI to the AI provider under your own terms of service and billing. The `a365` CLI specifies the model flag but does not mediate the API call.
4143

docs/agent365-guided-setup/a365-evaluate-instructions.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ After all three questions are answered, create all todos for the path and mark T
6161

6262
**RULE 6 — SILENT EXECUTION.** Work silently. Do NOT narrate what you are about to do, announce step transitions, print todo state, emoji checklists, or step completion summaries. Only speak to the user when you need input, have an error to report, or need confirmation before a destructive action.
6363

64-
**RULE 7 — DO NOT SHIP THE TOKEN.** If the user provided an auth token in Question 2, pass it only via the `--auth-token` flag at the moment you invoke the CLI. Do NOT write it to a file, paste it back into chat, or commit it to any config.
64+
**RULE 7 — DO NOT SHIP THE TOKEN.** If the user provided an auth token in Question 2, pass it only via the `--auth-token` flag or the `A365_MCP_AUTH_TOKEN` environment variable at the moment you invoke the CLI (the env var is preferred — it keeps the token out of process listings and shell history). Do NOT write it to a file, paste it back into chat, or commit it to any config.
6565

6666
---
6767

@@ -179,6 +179,18 @@ a365 develop-mcp evaluate \
179179
${authToken:+--auth-token "$authToken"}
180180
```
181181

182+
### Optional: configure via environment variables
183+
184+
These settings can come from environment variables instead of (or alongside) the flags:
185+
186+
| Env var | Purpose |
187+
|---|---|
188+
| `A365_MCP_AUTH_TOKEN` | Bearer token for the MCP server, used when `--auth-token` is not passed. **Preferred over the flag** — it keeps the token out of process listings (`ps` / Task Manager) and shell history. If you pass `--auth-token`, the CLI prints a one-time warning recommending this variable. |
189+
| `A365_EVAL_COPILOT_MODEL` | Override the GitHub Copilot model (exact model ID, e.g. `claude-haiku-4.5`). |
190+
| `A365_EVAL_CLAUDE_MODEL` | Override the Claude Code model (alias, e.g. `haiku`). |
191+
192+
The model defaults to Claude Haiku 4.5; override only to move to a newer model without waiting for a CLI release.
193+
182194
### What you will see
183195

184196
The CLI logs progress in numbered steps `[1/5]` through `[5/5]`:

0 commit comments

Comments
 (0)