[Task] Add STARE Task #893
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kcz358
reviewed
Nov 17, 2025
Comment on lines
+50
to
+76
| def build_query(sample): | ||
| """ | ||
| Construct a multiple-choice query. Inserts <image> placeholder if missing. | ||
| Appends either chain-of-thought (CoT) or direct instruction from config. | ||
| """ | ||
| question = sample["question"] | ||
| images = sample.get("images", []) | ||
| answer = sample["answer"].strip().upper() | ||
| strategy = config["metadata"]["strategy"] | ||
|
|
||
| # Ensure <image> placeholder if images are provided | ||
| if "<image>" not in question and images: | ||
| question += "\n<image>" | ||
|
|
||
| # Fill the template with the question | ||
| prompt_template = stare_config["multi_choice_format"] | ||
| filled_prompt = prompt_template.format(question=question) | ||
|
|
||
| # Append the selected strategy instruction | ||
| if strategy == "CoT": | ||
| query = f"{filled_prompt}\n{stare_config['Strategy_Instruction']['CoT']}" | ||
| else: | ||
| query = f"{filled_prompt}\n{stare_config['Strategy_Instruction']['Directly']}" | ||
|
|
||
| # Build the result dictionary | ||
| res_dict = {"query": query, "gt_content": answer, **sample} | ||
| return res_dict |
Collaborator
There was a problem hiding this comment.
Would recommend to use doc_to_messages so that no need to put the placeholder for chat model and the images can be correctly interleaved
kcz358
reviewed
Nov 17, 2025
Collaborator
kcz358
left a comment
kcz358
left a comment
There was a problem hiding this comment.
Hi, thank you for the contribution. I think most of the code looks nice. The build query should be fine to work for simple model as it is hard to come up with a unified solution with it. I would recommend you to use doc_to_messages api also so that the chat model can build the interleave format better. Feel free to refer to mmmu for more info. Thanks!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add STARE task from https://github.com/STARE-bench/STARE/