Skip to content

fix(webllm): cleanRewriteLine should strip "Here are the rewritten bullets:" preamble line #150

Description

@Vaishnavi1709

Surfaced by

The eval harness from #65 (PR #149). Llama 3.2 (3B) consistently emits a leading "Here are the rewritten bullets:" line under the terse and examples-led prompt variants. Example from tests/fixtures/rewrite/reports/eval-rewrite-llama-3-2-3b-instruct-…json:

"perBullet": [
  { "index": 0, "text": "Here are the rewritten bullets:", "startsWithActionVerb": false, ... },
  { "index": 1, "text": "Spearheaded marketing task management, ...", ... },
  ...
]

The preamble survives cleanRewriteLine and ends up as outputBullets[0]. That inflates the output bullet count by 1, fails the action-verb check on cell 0, and dilutes every per-criterion aggregate for Llama.

Root cause

src/lib/webllm/post-process.ts has a PROMPT_ECHO_LINES set that drops scaffolding lines like "rewritten bullets:" and "rules:". Llama's exact phrase "Here are the rewritten bullets:" is not in that set, so the line passes through as a normal bullet.

Fix

Two reasonable shapes:

  1. Substring match — change the check from set-membership to substring-contains for a short list of known preamble openers ("here are the rewritten", "here is the rewritten", "rewritten bullets:", "original bullets:").
  2. Pattern match — regex like /^here (?:are|is) (?:the )?rewritten/i plus the existing exact-match set.

Either works. (1) is simpler; (2) is more precise. Either way, add a unit test in post-process.test.ts that asserts the line is dropped.

Tracking

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

featureNew functionality

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions