Problem
When the main agent gathers a decision through an interactive question tool
(in my install, ask_user_question from @juicesharp/rpiv-ask-user-question),
the user's selection lands in the session as a toolResult.
collectReviewEvidence summarizes tool results as TOOL <name> → success
without reading content, so the guardian sees that a dialog happened but not
what the user chose. Since only source: "user" records can authorize, it
correctly falls back to ask_user — and the user confirms the same decision
twice, seconds apart.
Real example: the agent asked which hosts should receive a deployment; the user
selected "Canary pair: instance1, instance2" in the dialog. The dispatch was then
held with "the user has not explicitly authorized those exact demo
instances". The rule worked as designed — the reviewer just couldn't see the
answer.
Reproduction
- Guard
git commit (anything outside a low-risk allowlist works).
- Prompt: "Ask me via the question dialog which files to commit and which
message to use, then commit exactly what I picked."
- Answer the dialog. The commit is reviewed with evidence containing only
TOOL ask_user_question → success → ask_user instead of approve.
Proposed fix: operator-allowlisted answer tools
Not a special case for any particular package — a config-gated, tool-agnostic
convention:
{ "reviewEvidence": { "userAnswerTools": ["ask_user_question"] } }In collectReviewEvidence, for a successful toolResult from an allowlisted
tool name whose details match a documented shape
(answers: [{ question, answer | selected[], notes? }], cancelled !== true,
no error), emit one record per answered question at the toolResult entry
(stable key, so delta mode and the evidence-prefix cache are unaffected):
{ source: "user", text: 'USER (dialog answer): "Which demo hosts should get v1.32.1 first?" -> "Canary pair: southern, dominion"' }
The shape above is what rpiv-ask-user-question already emits, but the
convention would let any interactive tool qualify. Default: empty allowlist —
current behavior unchanged.
Why gate it behind config
- The allowlist is the operator asserting "I trust the extension serving this
tool name in my install" — tool names can be shadowed, so this should never
be inferred.
- Question and option labels are model-authored; the user picks among them. A
selection is an affirmative user decision, but slightly weaker evidence than
free-typed text. The distinct USER (dialog answer): prefix lets the system
prompt weigh that if desired.
Alternatives considered
- A bridge extension that re-injects answers via
pi.sendUserMessage(..., { deliverAs: "steer" }) after the dialog resolves.
I run this today and it works (verified end-to-end: the guardian approves a
guarded command whose targets exist only in the dialog answer), but it
duplicates content into the main LLM context and works around your
structured-source model rather than with it.
- Asking the dialog extension to emit a user message itself — same context
pollution, and it puts an authorization-policy decision in a UI package.
Happy to send a PR if this shape looks right. Thanks for the extension — the
cumulative-evidence reviewer is excellent, and this is the only case I've hit
where it loses a decision the user actually made.
Problem
When the main agent gathers a decision through an interactive question tool
(in my install,
ask_user_questionfrom@juicesharp/rpiv-ask-user-question),the user's selection lands in the session as a
toolResult.collectReviewEvidencesummarizes tool results asTOOL <name> → successwithout reading content, so the guardian sees that a dialog happened but not
what the user chose. Since only
source: "user"records can authorize, itcorrectly falls back to
ask_user— and the user confirms the same decisiontwice, seconds apart.
Real example: the agent asked which hosts should receive a deployment; the user
selected "Canary pair: instance1, instance2" in the dialog. The dispatch was then
held with "the user has not explicitly authorized those exact demo
instances". The rule worked as designed — the reviewer just couldn't see the
answer.
Reproduction
git commit(anything outside a low-risk allowlist works).message to use, then commit exactly what I picked."
TOOL ask_user_question → success→ask_userinstead ofapprove.Proposed fix: operator-allowlisted answer tools
Not a special case for any particular package — a config-gated, tool-agnostic
convention:
{ "reviewEvidence": { "userAnswerTools": ["ask_user_question"] } }In
collectReviewEvidence, for a successfultoolResultfrom an allowlistedtool name whose
detailsmatch a documented shape(
answers: [{ question, answer | selected[], notes? }],cancelled !== true,no
error), emit one record per answered question at the toolResult entry(stable key, so delta mode and the evidence-prefix cache are unaffected):
The shape above is what
rpiv-ask-user-questionalready emits, but theconvention would let any interactive tool qualify. Default: empty allowlist —
current behavior unchanged.
Why gate it behind config
tool name in my install" — tool names can be shadowed, so this should never
be inferred.
selection is an affirmative user decision, but slightly weaker evidence than
free-typed text. The distinct
USER (dialog answer):prefix lets the systemprompt weigh that if desired.
Alternatives considered
pi.sendUserMessage(..., { deliverAs: "steer" })after the dialog resolves.I run this today and it works (verified end-to-end: the guardian approves a
guarded command whose targets exist only in the dialog answer), but it
duplicates content into the main LLM context and works around your
structured-source model rather than with it.
pollution, and it puts an authorization-policy decision in a UI package.
Happy to send a PR if this shape looks right. Thanks for the extension — the
cumulative-evidence reviewer is excellent, and this is the only case I've hit
where it loses a decision the user actually made.