Hi @simonpcouch,

I am not 100% sure but I think there is still a minor issue with the logging and what is shown in the Inspect Viewer.

If I understand correctly, there are basically 8 (potentially) different pieces of text shown in the Viewer in various locations when using model_graded_qa():

• Solver
  • User
  • Assistant
• Scorer
  • User
  • Assistant
  • Target
  • Answer (same as Solver Assistant)
  • Explanation (same as Scorer Assistant)
  • Score (extracted from Scorer Assistant)

However, some models return their chain-of-thoughts inside <think>...</think> tags. I want all these CoT removed from the submitted answer prior to scoring, so I use a custom generate_rm_thoughts() function. I think this function works correctly to remove the thoughts prior to scoring, but the answer with thoughts still shows up in the Viewer.

It is kind of hard to describe, so here is a reproducible example (with images):

library(tibble)
library(ellmer)
library(vitals)

simple_addition <- tibble(
  input = c("What's 2+2?", "What's 2+3?", "What's 2+4?"),
  target = c("4", "5", "6")
)

generate_rm_thoughts <- function(solver_chat = NULL) {
  chat <- solver_chat
  function(inputs, ..., solver_chat = chat) {
    ch <- solver_chat$clone()
    res <- ellmer::parallel_chat(ch, as.list(inputs), ...)
    result <- purrr::map_chr(res, function(c) c$last_turn()@text)
    result_trim <- stringr::str_remove_all(
      result,
      # Remove thoughts and leading line breaks
      "^<think>[\\s\\S]*</think>\\n+"
    )
    list(result = result_trim, solver_chat = res)
  }
}

tsk <- Task$new(
  dataset = simple_addition, 
  solver = generate_rm_thoughts(chat_groq(model = "qwen/qwen3-32b")), 
  scorer = model_graded_qa()
)

tsk$eval()

tsk$view()

Note the Scorer Assistant, Scorer User (Submission component), and Scorer Answer texts.

Scorer Assistant

Scorer User

Scorer Answer

Expected Behaviour: The "Answer" text in the above Viewer screenshot is the same as the answer text used for interpolation in the Scorer Template (shown in the Scorer User message as "Submission").