fix(sew): validate refined prompts before adopting them in SEWOptimizer - #269
Open
lamost423 wants to merge 1 commit into
Open
fix(sew): validate refined prompts before adopting them in SEWOptimizer#269lamost423 wants to merge 1 commit into
lamost423 wants to merge 1 commit into
Conversation
SimplePromptBreeder.generate_prompt returns the LLM response verbatim, and
SEWOptimizer adopted it as the new task/operator prompt with no validation.
When the LLM echoes the refinement meta-instructions instead of producing a
refined instruction (observed with deepseek-chat), the echoed meta-prompt
replaces the node prompt and silently corrupts the workflow: in a HumanEval
run (optimize_mode=prompt, zero-order) the first node's prompt became the
literal text 'Given the above information, please refine the instruction for
the 1-th task...', losing the {question} placeholder, and test pass@1 dropped
from 0.4 to 0.1 after 3 optimization steps.
Add _validate_refined_prompt, applied in both _wfg_prompt_optimization_step
and _action_graph_prompt_optimization_step: reject empty responses, responses
echoing the refinement meta-instructions, and responses that drop input
placeholders present in the original prompt; fall back to the original prompt
with a warning instead of adopting an invalid refinement.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happened
While running
SEWOptimizer(optimize_mode="prompt",order="zero-order") on a HumanEval subset withdeepseek-chat, the optimized workflow regressed from pass@1 = 0.4 to 0.1 after 3 optimization steps.Inspecting the saved workflow showed the first node's prompt had been replaced by a verbatim echo of the optimizer's own meta-instructions:
The original
{question}placeholder was lost, so the node produced garbage for every downstream sample.Root cause
SimplePromptBreeder.generate_promptreturns the LLM response verbatim, and both_wfg_prompt_optimization_stepand_action_graph_prompt_optimization_stepadopt it directly:When the LLM echoes the refinement request instead of answering it (models occasionally do this with the nested mutation-prompt + meta-instruction structure), the echo silently becomes the node prompt.
Fix
Add
SEWOptimizer._validate_refined_prompt, applied at both adoption sites. A refined prompt is rejected (keeping the original prompt, with a warning) when it:{question}) that was present in the original prompt — matching the contract the meta-prompt itself states ("always use bracket to wrap the inputs").Tests
tests/src/optimizers/test_sew_prompt_validation.py— 7 cases covering the real-world echoed response above, empty responses, placeholder loss, legitimate refinements, and the fallback/adoption behavior of_wfg_prompt_optimization_step(via a stubbed prompt breeder, no API calls). All pass; existingtest_sew_workflow_scheme.pystill passes.🤖 Generated with Claude Code