refactor(red-team): align GOAT with Meta's paper — drop pre-generated plan and stage hints

[Aryansharma28]

Summary

Stacks on #306 (GOAT strategy). Moves GOAT from ~60% to ~85% faithful to Meta's ICML 2025 GOAT paper by stripping two Crescendo-derived artifacts that aren't in the paper:

1. No pre-generated attack plan. The paper's attacker reasons turn-by-turn from the technique catalogue + objective + conversation history. It doesn't pre-generate a plan via a separate metaprompt LLM call. We were, as a Crescendo carryover. Now gated behind a new `needs_metaprompt_plan` / `needsMetapromptPlan` property on the strategy — GoatStrategy returns `False`, so `_generate_attack_plan` is skipped entirely for GOAT (saves one LLM call on turn 1).
2. No stage hints in the attacker's system prompt. The paper has no early/mid/late concept — adaptation is driven entirely by the score/hint feedback in the attacker's private conversation history. Removed the `_STAGES` array, the `_get_stage` method, and the `Stage: EARLY — …` block from `build_system_prompt`. `get_phase_name` still returns `early`/`mid`/`late` for telemetry dashboards, but the label is no longer surfaced to the attacker.

What this changes for users

• `RedTeamAgent.goat(...)` skips the metaprompt LLM call on turn 1 → ~33% cheaper startup, and the description-keyed stale-plan failure mode (red team: _attack_plan cache survives across scenarios with different descriptions #327) disappears for GOAT.
• GOAT attacker prompt is smaller and more focused — no plan, no stage text. The attacker reasons off catalogue + score feedback alone, matching the paper's methodology.
• `GOAT_METAPROMPT_TEMPLATE` is dropped from the public API (it's no longer rendered anywhere). Removed from `scenario.init.py` and `javascript/src/agents/red-team/index.ts`.
• `goat()` / `redTeamGoat` factories simplified — no more `setdefault` / object-spread template dance, which also removes the foot-gun fixed in 81f9ef4.
• Crescendo behaviour unchanged.

Paper gap remaining (not in this PR)

The biggest remaining gap is structured chain-of-thought output (`observation` / `strategy` / `reply` JSON fields). Tracked in langwatch/langwatch#2142 and #330. That's the next follow-up (stacked branch `feat/red-team-structured-output`).

Test plan

• Python: 152 red-team tests pass (stage boundary tests rewritten as progress-bucket tests; prompt-contains-X assertions updated to assert ATTACK PLAN and Stage: are absent from GOAT prompts)
• TypeScript: 103 red-team tests pass (same treatment)
• Smoke test: `GoatStrategy.needs_metaprompt_plan is False`, `CrescendoStrategy.needs_metaprompt_plan is True`, GOAT prompt contains technique catalogue + turn number but no plan/stage blocks, `scenario.GOAT_METAPROMPT_TEMPLATE` no longer exists.
• Manual: run a 10-turn GOAT attack on bank-demo to confirm behaviour is reasonable (requires API keys).

Related

Part of EPIC: langwatch/langwatch#1713 (Scenarios Red Teaming)

Makes progress toward — but does not close — the following:

• red team: _attack_plan cache survives across scenarios with different descriptions #327 (attack_plan cache): bug no longer reachable via GOAT path (GOAT doesn't generate a plan); still live for Crescendo.
• red team: strategies should own their default metaprompt template #328 (strategies should own default template): sidesteps the issue for GOAT by making the template irrelevant entirely; proper refactor still pending for Crescendo.
• red team: edge cases — empty techniques list, empty metaprompt_plan, scorer failure cascade #333 (empty `metaprompt_plan` malformed prompt): no longer reachable for GOAT; still live for Crescendo.

Does not address:

• red team: injection_probability desyncs H_attacker from what the target saw #326 (injection H_attacker desync) — still present, still documented as a caveat.
• red team: capture which GOAT technique the attacker chose per turn (telemetry) #330 (technique telemetry), langwatch/scenario#2142 (structured output) — next PR.

🤖 Generated with Claude Code

[github-actions]

Automated low-risk assessment

This PR was evaluated against the repository's Low-Risk Pull Requests procedure and does not qualify as low risk.

The PR changes runtime behavior and public API surface: it removes the pre-generated metaprompt plan, eliminates stage hints from attacker prompts, adds structured JSON parsing for attacker output, and skips an LLM call for GOAT. These are functional and API changes (GOAT_METAPROMPT_TEMPLATE removed) rather than purely documentation/tests, so they fall outside the low-risk categories and should receive a normal review.

This PR requires a manual review before merging.