Surfaced by
The eval harness from #65 (PR #149). Llama 3.2 (3B) consistently emits a leading "Here are the rewritten bullets:" line under the terse and examples-led prompt variants. Example from tests/fixtures/rewrite/reports/eval-rewrite-llama-3-2-3b-instruct-…json:
"perBullet": [
{ "index": 0, "text": "Here are the rewritten bullets:", "startsWithActionVerb": false, ... },
{ "index": 1, "text": "Spearheaded marketing task management, ...", ... },
...
]
The preamble survives cleanRewriteLine and ends up as outputBullets[0]. That inflates the output bullet count by 1, fails the action-verb check on cell 0, and dilutes every per-criterion aggregate for Llama.
Root cause
src/lib/webllm/post-process.ts has a PROMPT_ECHO_LINES set that drops scaffolding lines like "rewritten bullets:" and "rules:". Llama's exact phrase "Here are the rewritten bullets:" is not in that set, so the line passes through as a normal bullet.
Fix
Two reasonable shapes:
- Substring match — change the check from set-membership to substring-contains for a short list of known preamble openers (
"here are the rewritten", "here is the rewritten", "rewritten bullets:", "original bullets:").
- Pattern match — regex like
/^here (?:are|is) (?:the )?rewritten/i plus the existing exact-match set.
Either works. (1) is simpler; (2) is more precise. Either way, add a unit test in post-process.test.ts that asserts the line is dropped.
Tracking
Surfaced by
The eval harness from #65 (PR #149). Llama 3.2 (3B) consistently emits a leading
"Here are the rewritten bullets:"line under the terse and examples-led prompt variants. Example fromtests/fixtures/rewrite/reports/eval-rewrite-llama-3-2-3b-instruct-…json:The preamble survives
cleanRewriteLineand ends up asoutputBullets[0]. That inflates the output bullet count by 1, fails the action-verb check on cell 0, and dilutes every per-criterion aggregate for Llama.Root cause
src/lib/webllm/post-process.tshas aPROMPT_ECHO_LINESset that drops scaffolding lines like"rewritten bullets:"and"rules:". Llama's exact phrase"Here are the rewritten bullets:"is not in that set, so the line passes through as a normal bullet.Fix
Two reasonable shapes:
"here are the rewritten","here is the rewritten","rewritten bullets:","original bullets:")./^here (?:are|is) (?:the )?rewritten/iplus the existing exact-match set.Either works. (1) is simpler; (2) is more precise. Either way, add a unit test in
post-process.test.tsthat asserts the line is dropped.Tracking