Vision Benchmarks for Atropos Evals

environments/eval_environments/vision_evals/README.md

@@ -0,0 +1,216 @@
+# Vision Evaluation Benchmarks
+
+This folder contains 27 vision and multimodal benchmarks for evaluating vision-language models. The implementations follow VLMEvalKit patterns where applicable and use the Atropos Eval class for consistent async evaluation.
+
+## Benchmarks
+
+| Benchmark | What it Tests | Dataset | Scoring |
+|-----------|---------------|---------|---------|
+| MMMU | Multi-discipline academic QA | MMMU/MMMU | MCQ accuracy |
+| MMMU-Pro | Harder MMMU with 10 choices | MMMU/MMMU_Pro | MCQ accuracy |
+| MMBench | General multimodal understanding | lmms-lab/MMBench | MCQ accuracy |
+| MMStar | Expert-level multimodal QA | Lin-Chen/MMStar | MCQ accuracy |
+| MMVet | Open-ended VLM capabilities | lmms-lab/MMVet | GPT scoring |
+| MMVP | CLIP blindspot detection | MMVP/MMVP | MCQ accuracy |
+| AI2D | Scientific diagram understanding | lmms-lab/ai2d | MCQ accuracy |
+| BLINK | Visual perception tasks | BLINK-Benchmark/BLINK | MCQ accuracy |
+| ChartQA | Chart question answering | ahmed-masry/ChartQA | Relaxed accuracy |
+| CharXiv | Scientific chart understanding | princeton-nlp/CharXiv | GPT judge |
+| CountBench | Object counting | nielsr/countbench | Numeric match |
+| DocVQA | Document understanding | lmms-lab/DocVQA | ANLS score |
+| DynaMath | Dynamic math reasoning | DynaMath/DynaMath_Sample | JSON extraction |
+| HallusionBench | Visual hallucination detection | lmms-lab/HallusionBench | Yes/No accuracy |
+| InfoVQA | Infographic QA | lmms-lab/InfoVQA | ANLS score |
+| LogicVista | Visual logical reasoning | Yijia-Xiao/LogicVista | GPT extraction |
+| MathVerse | Visual math (multi-version) | AI4Math/MathVerse | GPT extraction + scoring |
+| MathVision | Visual math problems | MathLLMs/MathVision | GPT extraction |
+| MathVista | Visual math reasoning | AI4Math/MathVista | GPT extraction |
+| MMT-Bench | Multi-task multimodal | OpenGVLab/MMT-Bench | MCQ accuracy |
+| OCRBench | OCR capabilities | echo840/OCRBench | Substring match |
+| POPE | Object hallucination | lmms-lab/POPE | Yes/No accuracy |
+| RealWorldQA | Real-world visual QA | xai-org/RealworldQA | Fuzzy match |
+| SEED-Bench2 | Visual understanding | lmms-lab/SEED-Bench-2 | MCQ accuracy |
+| VisuLogic | Visual logic puzzles | visulogic dataset | MCQ accuracy |
+| VLMBlind | Basic visual perception | XAI/vlmsareblind | Task-specific |
+| WeMath | Visual math with 4D metrics | We-Math/We-Math | 4D scoring |
+
+## Running an Evaluation
+
+All benchmarks use the same CLI pattern:
+
+```bash
+python mmmu_environment.py \
+    --model-name "gpt-4o" \
+    --server-url "https://api.openai.com/v1"
+```
+
+For local models with vLLM or Ollama:
+
+```bash
+python mmbench_environment.py \
+    --model-name "Qwen/Qwen2-VL-7B-Instruct" \
+    --server-url "http://localhost:8000/v1"
+```
+
+The evaluations use the `ServerManager` from atroposlib for making API calls.
+
+## Comparison with VLMEvalKit
+
+These implementations are aligned with VLMEvalKit where it makes sense, but simplified for standalone use. Here are the key differences and similarities:
+
+### Scoring Methods
+
+**ChartQA** uses relaxed accuracy with 5% tolerance. Percentages are converted to decimals before comparison (5% becomes 0.05). This matches VLMEvalKit behavior in `vqa_eval.py`.
+
+**DocVQA and InfoVQA** use ANLS (Average Normalized Levenshtein Similarity) with a 0.5 threshold. This is the standard metric from the original papers.
+
+**MathVista** uses GPT-based answer extraction with 5 in-context learning examples. There is a prefetch mechanism that tries regex first before calling GPT. The extraction prompt and ICL examples are taken from VLMEvalKit.
+
+**MathVerse** uses two-stage GPT evaluation. First GPT extracts the answer from the response, then GPT judges whether the extracted answer matches the ground truth. This matches the VLMEvalKit approach.
+
+**WeMath** computes 4-dimensional metrics beyond simple accuracy:
+- IK (Insufficient Knowledge): wrong on steps AND wrong on multi
+- IG (Inadequate Generalization): right on steps BUT wrong on multi
+- CM (Complete Mastery): right on steps AND right on multi
+- RM (Rote Memorization): wrong on steps BUT right on multi
+
+**MMVet** uses GPT to score open-ended responses on a 0-1 scale. Without an API key it falls back to substring matching.
+
+**OCRBench** uses category-specific scoring. For handwritten math expressions it compares without spaces. For other categories it does case-insensitive substring matching.
+
+### What We Changed
+
+**Simpler data loading**: We use HuggingFace datasets directly instead of VLMEvalKit's TSV preprocessing. This makes the code easier to understand but may load data slightly differently.
+
+**Async evaluation**: Everything runs async with tqdm progress bars. VLMEvalKit uses synchronous evaluation by default.
+
+**No circular evaluation**: VLMEvalKit supports "circular" MCQ evaluation where options are rotated and the model must get all rotations correct. We do not implement this, which means our MCQ scores may be slightly higher than VLMEvalKit on some benchmarks.
+
+**Unified CLI**: All benchmarks use the same `eval_runner` CLI instead of VLMEvalKit's `run.py` with config files.
+
+### Expected Score Differences
+
+Due to the differences above, you should expect:
+
+- MCQ benchmarks (MMMU, MMBench, MMStar, AI2D): Within 1-2% of VLMEvalKit
+- VQA benchmarks (DocVQA, ChartQA): Very close, same scoring methods
+- Math benchmarks (MathVista, MathVerse): Within 2-3%, depends on GPT extraction
+- Open-ended (MMVet): Can vary more, depends on GPT judge prompts
+
+## Benchmark Details
+
+### General Multimodal Understanding
+
+**MMMU** tests multi-discipline academic knowledge across 30 subjects from accounting to physics. Questions require understanding images and domain knowledge. The validation split has about 900 questions.
+
+**MMMU-Pro** is a harder version with 10 answer choices instead of 4. It has three variants: standard (10 options), standard_4 (4 options), and vision (question in image).
+
+**MMBench** is a comprehensive benchmark covering perception, reasoning, and knowledge. It has English and Chinese versions.
+
+**MMStar** focuses on expert-level questions that require both visual understanding and specialized knowledge.
+
+**SEED-Bench2** tests visual understanding across many categories including scene understanding, instance identity, and spatial relations. The dataset is large (24k samples) so we stream by default and limit to 1000 samples.
+
+**MMT-Bench** is a multi-task benchmark covering 32 different task types. Good for testing breadth of capabilities.
+
+### Document and Chart Understanding
+
+**DocVQA** tests understanding of document images like forms, receipts, and scientific papers. Uses ANLS scoring which allows for minor OCR errors.
+
+**InfoVQA** is similar to DocVQA but focuses on infographics with more complex layouts.
+
+**ChartQA** tests chart reading. Has human and augmented subsets. The human subset is harder. Uses relaxed accuracy (5% tolerance for numbers).
+
+**CharXiv** focuses on scientific charts from arXiv papers. Uses GPT as a judge with grading queries from the dataset.
+
+**OCRBench** tests pure OCR capabilities across 10 categories from regular text to handwritten math expressions.
+
+### Math and Reasoning
+
+**MathVista** is a visual math benchmark with multiple question types (free form, multiple choice) and answer types (integer, float, text, list). Uses the dataset's built-in query prompts.
+
+**MathVerse** has problems at different visual complexity levels from "text dominant" to "vision only". Uses two-stage GPT evaluation.
+
+**MathVision** is another visual math benchmark. Uses GPT extraction with fallback to regex.
+
+**DynaMath** tests dynamic math reasoning with JSON-formatted outputs. Has subject and difficulty level breakdowns.
+
+**WeMath** provides detailed 4D metrics to understand where models fail. Good for diagnosing reasoning vs memorization issues.
+
+**LogicVista** tests visual logical reasoning with 5 skill types. Supports multi-letter answers where multiple options can be correct.
+
+**VisuLogic** tests visual logic with diagram-based puzzles.
+
+### Perception and Hallucination
+
+**POPE** tests object hallucination with yes/no questions about whether objects exist in images. Has random, popular, and adversarial variants.
+
+**HallusionBench** tests visual hallucinations more broadly. Questions are designed to trick models into seeing things that are not there.
+
+**MMVP** tests visual perception on cases where CLIP-based models tend to fail. Useful for understanding encoder limitations.
+
+**BLINK** tests basic visual perception like counting, spatial relations, and similarity. Models often struggle on these "easy" tasks.
+
+**VLMBlind** (VLMs Are Blind) tests very basic visual tasks that humans find trivial but VLMs often fail. Includes counting grid cells, finding circled letters, and counting Olympic rings.
+
+**CountBench** is a simple object counting benchmark.
+
+### Real World
+
+**RealWorldQA** tests understanding of real-world images from XAI. Uses fuzzy matching for answers.
+
+**AI2D** tests understanding of scientific diagrams from AI2 (Allen Institute). Good for testing diagram reasoning.
+
+## GPT Judge Configuration
+
+Several benchmarks use GPT for answer extraction or scoring. To enable this:
+
+```bash
+export OPENAI_API_KEY="your-key"
+```
+
+You can also configure the judge model when instantiating:
+
+```python
+eval_env = MathVista(
+    use_gpt_extraction=True,
+    judge_model="gpt-4o-mini",
+    judge_base_url="https://api.openai.com/v1",
+)
+asyncio.run(eval_runner(eval_env))
+```
+
+Without an API key, benchmarks fall back to regex-based extraction which is less accurate but free.
+
+## Output Format
+
+Results are saved to the eval directory:
+
+```
+eval_results/
+    metrics.json     # Overall scores
+    samples.jsonl    # Per-item predictions
+```
+
+The metrics.json file contains accuracy and other metrics depending on the benchmark. The samples.jsonl file has one line per question with the prediction, answer, and whether it was correct.
+
+## Adding New Benchmarks
+
+To add a new vision benchmark:
+
+1. Create a new file like `new_benchmark_environment.py`
+2. Inherit from `EvalBase`
+3. Implement `setup_data()` to load the dataset
+4. Implement `run_item(self, server: ServerManager, data_item: dict)` to process one item
+5. Use `await self.chat_completion(server, messages)` for API calls
+6. Add image encoding using the standard `encode_image()` pattern
+
+See any existing benchmark for a template. The MMMU implementation is a good starting point for MCQ benchmarks. DocVQA is a good template for VQA benchmarks.
+
+## References
+
+- VLMEvalKit: https://github.com/open-compass/VLMEvalKit
+- OpenVLM Leaderboard: https://huggingface.co/spaces/opencompass/open_vlm_leaderboard
+- MMMU: https://mmmu-benchmark.github.io/
+- MathVista: https://mathvista.github.io/
+- DocVQA: https://www.docvqa.org/

environments/eval_environments/vision_evals/ai2d_environment.py

@@ -0,0 +1,159 @@
+"""AI2D (AI2 Diagrams) evaluation environment."""
+
+import asyncio
+import base64
+import io
+from string import ascii_uppercase
+from typing import List, Optional, Tuple
+
+from datasets import load_dataset
+from environments.eval_environments.eval import EvalBase, eval_runner
+from PIL import Image
+
+from atroposlib.envs.server_handling.server_manager import ServerManager
+from environments.eval_environments.eval_helpers import (
+    extract_letter_from_answer_tag,
+    extract_mcqa_answer_with_fallback,
+)
+
+
+class AI2D(EvalBase):
+    """AI2D evaluation - diagram understanding benchmark."""
+
+    def setup_data(self) -> list:
+        split = getattr(self, "split", "test")
+        use_mask = getattr(self, "use_mask", True)
+
+        try:
+            dataset = load_dataset("lmms-lab/ai2d", split=split)
+            print(f"Loaded {len(dataset)} examples from AI2D ({split})")
+            return list(dataset)
+        except Exception as e:
+            print(f"Warning: Could not load AI2D: {e}")
+            try:
+                dataset = load_dataset("allenai/ai2_diagrams", split=split)
+                print(f"Loaded {len(dataset)} examples from AI2D ({split})")
+                return list(dataset)
+            except Exception:
+                raise ValueError(f"Could not load AI2D dataset: {e}")
+
+    def encode_image(self, pil_image: Image.Image) -> str:
+        buffer = io.BytesIO()
+        pil_image.save(buffer, format="PNG")
+        return base64.b64encode(buffer.getvalue()).decode("utf-8")
+
+    def get_image_base64(self, item: dict) -> Optional[str]:
+        for key in ["image", "decoded_image"]:
+            if key in item and item[key] is not None:
+                if isinstance(item[key], Image.Image):
+                    return self.encode_image(item[key])
+        return None
+
+    def build_messages(self, item: dict) -> List[dict]:
+        image_base64 = self.get_image_base64(item)
+        question = item.get("question", "")
+
+        choices = item.get("choices", [])
+        if isinstance(choices, str):
+            try:
+                choices = eval(choices)
+            except Exception:
+                choices = []
+
+        options = {}
+        if choices:
+            for i, choice in enumerate(choices):
+                options[ascii_uppercase[i]] = choice
+        else:
+            for letter in ascii_uppercase[:6]:
+                if letter in item and item[letter] is not None:
+                    val = item[letter]
+                    if isinstance(val, str) and val.strip():
+                        options[letter] = val
+
+        prompt = f"Question: {question}\n"
+        if options:
+            prompt += "Options:\n"
+            for letter in sorted(options.keys()):
+                prompt += f"{letter}. {options[letter]}\n"
+            prompt += "\nPlease select the correct answer from the options above."
+
+        content = []
+        if image_base64:
+            content.append(
+                {
+                    "type": "image_url",
+                    "image_url": {"url": f"data:image/png;base64,{image_base64}"},
+                }
+            )
+        content.append({"type": "text", "text": prompt})
+
+        return [{"role": "user", "content": content}]
+
+    def extract_answer(
+        self, response: str, num_choices: int
+    ) -> Tuple[Optional[str], str]:
+        valid_letters = set(ascii_uppercase[:num_choices])
+
+        letter, method = extract_letter_from_answer_tag(response, valid_letters)
+        if letter:
+            return letter, method
+
+        letter, method = extract_mcqa_answer_with_fallback(response, num_choices)
+        return letter, method
+
+    async def run_item(
+        self, server: ServerManager, data_item: dict
+    ) -> Tuple[dict, dict]:
+        try:
+            messages = self.build_messages(data_item)
+            completion = await self.chat_completion(server, messages)
+
+            if not completion.choices:
+                return {"accuracy": 0.0}, {"error": "Empty response"}
+
+            message = completion.choices[0].message
+            response = message.content or ""
+
+            if not response:
+                return {"accuracy": 0.0}, {"error": "Empty response"}
+
+            answer = data_item.get("answer", "")
+
+            choices = data_item.get("choices", [])
+            if isinstance(choices, str):
+                try:
+                    choices = eval(choices)
+                except Exception:
+                    choices = []
+
+            num_choices = len(choices) if choices else 4
+
+            extracted, method = self.extract_answer(response, num_choices)
+
+            correct = False
+            if extracted and answer:
+                if str(answer).isdigit():
+                    answer_letter = ascii_uppercase[int(answer)]
+                else:
+                    answer_letter = str(answer).upper()
+                correct = extracted.upper() == answer_letter
+
+            sample = {
+                "id": data_item.get("index", data_item.get("id", "")),
+                "question": data_item.get("question", "")[:200],
+                "answer": answer,
+                "prediction": extracted,
+                "raw_response": response[:500],
+                "correct": correct,
+                "extraction_method": method,
+            }
+
+            return {"accuracy": 1.0 if correct else 0.0}, sample
+
+        except Exception as e:
+            return {"accuracy": 0.0}, {"error": str(e)}
+
+
+if __name__ == "__main__":
+    asyncio.run(eval_runner(AI2D(split="test", temperature=0.0, max_tokens=256)))