Skip to content

[Improve] Guide kickoff messages to end with a period#355

Merged
mrubens merged 3 commits into
developfrom
improve/kickoff-end-with-period-26eymqdber8fc
Jul 15, 2026
Merged

[Improve] Guide kickoff messages to end with a period#355
mrubens merged 3 commits into
developfrom
improve/kickoff-end-with-period-26eymqdber8fc

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Opened on behalf of Matt Rubens. Follow up by mentioning @roomote-roomote, in the web UI, or in Slack.

Related issue

N/A — follow-up after #353 still left kickoff messages without trailing punctuation in practice.

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

What changed

Prompt-only guidance plus eval coverage so router kickoffs are short complete sentences ending with a period.

  • Reword the kickoff section to ask for a complete sentence ending with a period
  • Update examples to include trailing periods
  • Align the structured-output field description and router eval contract
  • Add hasValidKickoffMessage to the router eval assertions (non-empty + trailing period)
  • Apply that assertion on representative basic and model-routing cases
  • Enable the production kickoff section in the Promptfoo eval prompt via includeKickoffMessage: true
  • Require a non-null kickoffMessage string in the eval output format

No runtime force-normalization or static-template punctuation changes.

How it was tested

Pre-push lint:fast, check-types:fast, and knip passed. Full pnpm eval:router was not run in this environment (needs live LLM provider keys); the assertion and prompt flags are wired into the existing Promptfoo suite.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

Impact

Router prompt + schema encourage sentence-ending periods on dynamic kickoffs, and the router eval suite regresses that contract with the production kickoff section enabled.

Guide the router to write short complete kickoff sentences ending with a period via prompt, schema, and eval wording only.
@roomote-roomote

roomote-roomote Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

No code issues found. See task

  • packages/cloud-agents/evals/router/prompts/routing.ts:152 (Low) The router eval suite does not inspect kickoffMessage, so it passes when the model omits it or returns text without a trailing period. Add a kickoff assertion and apply it to representative router cases so this prompt contract is regression-tested.
  • packages/cloud-agents/evals/router/datasets/basic.yaml:31 (Medium) The newly required assertion runs against the Promptfoo prompt built at routing.ts:163 without includeKickoffMessage: true; that prompt omits the production kickoff section, while its output contract still explicitly permits kickoffMessage: null. The eval can therefore fail even when the evaluated prompt follows its own contract. Enable the kickoff section for this suite and make the eval output field required/non-null.

Reviewed 23c4023

@roomote-roomote

roomote-roomote Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the remaining open review item in 23c4023:

  • Eval prompt now builds with includeKickoffMessage: true (production kickoff section enabled)
  • Eval output format requires non-null kickoffMessage string ending with a period

Previous item (assert + representative cases) remains in 057c836.

mrubens added 2 commits July 15, 2026 02:41
Cover representative basic and model-routing cases so missing or
period-less kickoffMessage values fail the router eval suite.
Pass includeKickoffMessage so the suite evaluates the production kickoff
contract, and require a non-null kickoffMessage field in the eval output format.
@mrubens mrubens marked this pull request as ready for review July 15, 2026 02:57
@mrubens mrubens merged commit 459ba1e into develop Jul 15, 2026
17 checks passed
@mrubens mrubens deleted the improve/kickoff-end-with-period-26eymqdber8fc branch July 15, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant