Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
587 lines (477 loc) · 29.1 KB

CONTRIBUTING.md

File metadata and controls

587 lines (477 loc) · 29.1 KB
Raw

Contributing to MD Arena

MD Arena packages MD problems as Harbor-style tasks under tasks/<task_id>/. Adding a new task requires bundling input files, an agent-facing prompt, and an automated verifier that grades the result. This guide covers the required file layout, the step-by-step workflow for building one, and how to validate it before opening a PR.

Task Layout

Each benchmark task should live under tasks/<task_id>/:

tasks/<task_id>/
├── task.toml
├── instruction.md
├── CANARY.md            # canary marker - copy verbatim from task_template/
├── environment/
│   ├── Dockerfile
│   └── inputs/
├── solution/            # optional - reference solution, used by the Oracle agent
│   └── solve.sh
└── tests/
    ├── test.sh
    ├── correctness/
    │   ├── checks.py
    │   ├── helpers/
    │   └── expected_outputs/
    └── process/
        └── judge.toml

Use expected_outputs/ only when the verifier needs reference files. Use helpers/ only for task-local verifier utilities that cannot be replaced by Reward Kit built-ins.

Nothing else may be committed at the task root: only task.toml, instruction.md, CANARY.md, README.md, and .gitignore, plus the environment/, tests/, solution/, and steps/ directories. That list mirrors what Harbor itself reads, so anything else there is a file the harness will never see. Inside those directories, name files however you like. The validator enforces this - see "Task Validation" below.

Required Files

  • task.toml: Harbor task metadata, including task name, description, authors, resource limits, and declared artifacts, plus the [metadata] block describing the task itself - task_type, system_type, system_name, packages and difficulty. See "Task Metadata" below.
  • instruction.md: the prompt shown to the agent. It should name the available input paths and required output paths exactly. Do not put canary markers here - see CANARY.md below.
  • CANARY.md: the task's canary marker. Copy it from task_template/; the content is identical for every task. See "Canary Markers" below.
  • environment/Dockerfile: the task image build context. Task Dockerfiles should start from mdarena:latest, copy inputs/ into /app/inputs/, create /app/outputs/, and make /app writable. validate_tasks.py enforces the base: a task building on anything else fails validation unless it is listed in BASE_IMAGE_EXEMPTIONS. If your task needs PLUMED's lambda-dynamics/energy coupling (e.g. alchemical metadynamics), which the base image's default GROMACS 2025.4 native interface does not provide, do not add another base image - put /opt/gromacs-plumed/bin on PATH in your own Dockerfile, as 48_cb7_10_alchemical_metadynamics does. That prefix holds GROMACS 2024.3 with PLUMED's classic patch, shipped in the same base image.
  • tests/test.sh: verifier entry point. It should invoke Reward Kit, which writes /logs/verifier/reward.json and /logs/verifier/reward-details.json.
  • tests/correctness/checks.py: Reward Kit criteria for the required outputs and, when relevant, final_answer.txt.
  • tests/process/judge.toml: LLM-as-judge rubric for anything that can't be checked deterministically (see "How to Write Tests" below).

Task Metadata

[metadata] in task.toml describes the task for the dataset-composition figures and the per-type performance breakdowns. Every field is a fixed vocabulary except system_name. Do not invent a value: if none of the listed ones fit, raise it in the pull request so the vocabulary is extended deliberately, for every task at once, rather than growing a category of one.

That is not a style preference. The previous scheme let authors "reuse an existing category where possible", and it drifted to twelve values, four of which had a single task each - enough to make a per-category figure uninterpretable, and to disagree with the manuscript's own separate list of method classes.

task_type - one value, what the task asks the agent to do

Value n Meaning
analysis 13 Compute a quantity from a structure or trajectory
system_preparation 12 Build, clean, convert, or assemble a simulation-ready system
free_energy 8 Set up, plan, or analyse an alchemical free-energy calculation
enhanced_sampling 7 Set up umbrella sampling, expanded ensemble, REXEE, or metadynamics, or choose and assess a collective variable for one
parameterization 5 Generate or edit force-field parameters for a ligand or residue
diagnostics 4 Judge whether something is correct, or diagnose and repair it
quantum_chemistry 1 Set up a QM or QM/MM calculation

Tie-break rule. Most tasks touch more than one of these, and no single label is fully correct for them - a task is a point in several dimensions at once (activity, domain, system, software) and task_type projects it onto one. The goal is therefore not a perfect assignment but a deterministic one: two people should reach the same answer without conferring.

When a task spans two values, pick the one whose expertise it is testing - the part a competent non-specialist would get wrong.

Worked examples, all of them real:

Task A non-specialist could... ...but would get wrong Value
22_..._pca_cv_suitability run PCA on a trajectory judging whether that component is a usable CV enhanced_sampling
20_fep_abfe_dhdl_analysis plot dhdl files MBAR, window overlap, convergence free_energy
15_meng_coupling_groups run grompp on the provided inputs reading the failure it reports and rebuilding what the protocol names diagnostics
08_clash_check measure interatomic distances knowing what counts as a clash for MD diagnostics
25_mglu5_membrane_setup parameterize a ligand assembling a correct membrane system system_preparation

Note what the rule is not: it is not "the principal deliverable". That reading sends 15_meng_coupling_groups to system_preparation, because what it hands back is a TPR - which misses that the task exists to test whether an agent can find the fault. This one is the closest call in the table, and it got closer when the task was rebuilt in v0.2: grompp now names the missing groups itself, so the diagnosis is cheaper than the judgement about which molecules share a bath. It stays diagnostics because the agent is told nothing is wrong and has to establish that from tool output; revisit it if that stops being the harder half.

The field is single-valued on purpose. Per-type success rates ("5 of 7 system preparation tasks solved") only have a denominator if each task is counted once.

Membrane system preparation is not a separate value. It is task_type = "system_preparation" with system_type = "membrane_protein". Encoding it twice is what previously let 09_holo2apo and 21_water_transfer_ox2r sit outside a membrane_protein_system_preparation category that 25_mglu5_membrane_setup was inside, despite 09 and 25 being the same receptor.

system_type - one value, what is being simulated

Value n Meaning
membrane_protein 29 Any protein in a lipid bilayer - receptor, transporter, channel, or membrane-associated enzyme
soluble_protein 10 Protein in solution, with or without a bound ligand
small_molecule_system 7 No biomolecule present: ligand-only, host-guest, or a small-molecule liquid
peptide 3 Peptide or glycopeptide, too small to be treated as a folded protein
none 1 No molecular system can be named

Two rules decide the awkward cases:

It describes the scientific system the task concerns, not the contents of environment/inputs/. 20_fep_abfe_dhdl_analysis ships only dhdl XVG files - no structure, no topology - but the calculation is an A2A receptor ABFE, so it is membrane_protein. none means no system can be named, not no structure file was provided. Exactly one task qualifies: 42_expanded_ensemble_mdp, whose MDP decouples a moltype called COM that is never defined anywhere.

Membrane-versus-soluble follows the system, not the task's wording. A task on a GPCR is membrane_protein even if the word "membrane" never appears in its instruction - this is what previously split 09_holo2apo from 25_mglu5_membrane_setup despite both being mGlu5.

system_name - free text

The specific system, e.g. "ATP8B1-CDC50A complex" or "Adenosine A2A receptor". The only free-text field, and what makes any finer regrouping possible later - by receptor family, by PDB entry - without editing task files.

Four conventions, one per situation:

Situation Convention Example
Several systems semicolon-separated "Chignolin; T4 lysozyme"
Real system, no established name descriptive phrase "Solvated ligand from SMILES"
Identity deliberately withheld from the agent "Unspecified <what is known>" "Unspecified membrane protein (POPE/POPG bilayer)"
No molecular system "none", mirroring system_type "none"

The third and fourth are different states and must not be conflated: an unspecified system exists and is being withheld; "none" means there is nothing to name. Note also that agents never see task.toml - Harbor reads it host-side and the task Dockerfiles copy only environment/inputs/ - so naming a withheld system here does not leak it into the task. The "Unspecified ..." entries are placeholders pending the task authors, not a privacy measure.

Counts of distinct systems split on "; ", so "Chignolin; T4 lysozyme" is two.

packages - zero or more

The MD software a correct solution uses. Where a task admits several routes, the reference solution's route is canonical. Multi-valued, and the only such field - which is safe precisely because performance is never broken down by it, so a package used by a single task is a useful fact rather than an unusable sample size.

Four rules decide what goes in:

  • Domain software only. GROMACS, MDAnalysis, OpenFE and the like. Not NumPy, matplotlib, pandas, or MPI - the coverage claim is about biomolecular MD software, and generic scientific Python inflates it without meaning anything.
  • Not verifier dependencies. 25_mglu5_membrane_setup and 29_pept2_beta_lactam_membrane_setup pull MDAnalysis in tests/test.sh purely so the checker can mda.Universe(...) the agent's output and count lipids; neither reference solution touches it. That is the checker's parser, not something the task exercises.
  • Nothing, when the task names interchangeable tools. 01_similes2sim lists only GROMACS: its judge criterion explicitly accepts "RDKit, ACPYPE, Open Babel, or another valid chemistry toolkit", so no single one is exercised.
  • Empty is a valid answer. 45_atp8b_insertion_depth's reference solution imports only csv and os; four tasks are legitimately empty.

Software the task targets counts even if it never runs. 38_pept2_qm_cluster_orca lists ORCA. The task is to write a valid ORCA input file and, since v0.2, nothing executes it - but producing correct ORCA input is exactly the expertise under test. Do not remove the tag on the grounds that ORCA is not installed.

Use canonical project names, not command names or aliases: GROMACS, not gmx or GROMACS / gmx. Keep ecosystem members distinct rather than folding them into a parent - openmmforcefields is separate from OpenMM, MDRestraintsGenerator separate from GROMACS - because the figure can group them and the data cannot ungroup itself.

The 18 values currently in use: GROMACS, MDAnalysis, OpenMM, OpenFF Toolkit, RDKit, OpenFE, ACPYPE, PLUMED, Open Babel, PDBFixer, ParmEd, ProLIF, alchemlyb, openmmforcefields, ORCA, MDRestraintsGenerator, ensemble_md, physical_validation.

difficulty - one value

easy, medium, or hard, assigned by the task author and confirmed in review, based on workflow length, required domain knowledge, and technical complexity for current coding agents.

Adding a New Task: Step by Step

  1. Scaffold the task. Copy task_template/ to tasks/<next_number>_<slug>/ and fill in the TODO placeholders:

    cp -r task_template tasks/<next_number>_<slug>

    task_template/ already matches this repo's actual conventions - the nested tests/correctness/+tests/process/ split, mdarena:latest base image, resource limits matching the rest of tasks/. harbor task init TASKNAME is Harbor's own generic scaffold and is not repo-specific: it produces a flatter layout (a single tests/test_outputs.py, plain pytest, a generic Ubuntu Dockerfile) that would need reworking to match the "Task Layout" section above - prefer task_template/ unless you have a reason to start from Harbor's scaffold instead.

  2. Write the task instructions. Complete TASKNAME/instruction.md and TASKNAME/task.toml.

  3. Build the task environment. Edit TASKNAME/environment/Dockerfile and add any input files under TASKNAME/environment/inputs/.

  4. Prototype a solution. Start an interactive environment and work out the actual commands that solve the task:

    harbor task start-env -p TASKNAME -e docker -a -i   # or use daytona or modal

    Make sure the solution works without requiring interactive input.

  5. Freeze the solution. Turn the commands you verified in step 4 into TASKNAME/solution/solve.sh. This is what the Oracle agent runs to prove the task is solvable.

  6. Write the verifier. Update tests/test.sh, tests/correctness/checks.py, and tests/process/judge.toml - see "How to Write Tests" below for the split between the two.

  7. Validate with the Oracle agent:

    harbor run -p TASKNAME -a oracle

    A successful run reports reward 1. If the Oracle agent fails, check:

    • the solution script has execute permissions,
    • the Dockerfile installs everything the solution needs,
    • the test script writes /logs/verifier/reward.json and /logs/verifier/reward-details.json,
    • paths referenced in tests/ match the paths solution/solve.sh actually writes to.

  8. (Optional) Test with a real agent, e.g.:

    harbor run -p TASKNAME -a terminus-2 -m anthropic/claude-haiku-4-5

  9. Inspect the results:

    harbor view ./jobs

    Open the latest job, pick the TASKNAME trial, and check the Verifier Logs tab for the checker output and final reward. The Trajectory tab walks through each step the agent took - useful when a real agent fails the task.

  10. Run python scripts/validate_tasks.py (see "Task Validation" below) and open your PR.

How to Write Tests

Each task has two tests/ subfolders, matching the two ways an output can be judged:

  • tests/correctness/ - deterministic checks. Put reference files under expected_outputs/ and task-local helper code under helpers/. For numerical outputs keep tolerances explicit and scientifically justified. Three rules, in order of how often they are got wrong:

    1. Every weighted criterion must be able to fail a plausible wrong answer. The usual way to break this is a presence check. rk.file_exists("outputs/result.csv") at weight 1.0 pays an agent for creating a file, which no wrong answer fails - and if a real criterion reads that file too, the same thing is scored twice. Measured in v0.1: creating one task's whole output tree with touch and nothing else scored 0.9675.

    So register presence and non-empty checks at weight=0.0:

    # Kept for diagnostics, scored at zero: `result_csv_matches_reference` reads
# this file and returns False when it is absent, so weighting the presence
# check as well would score the same file twice.
rk.file_exists("outputs/result.csv", weight=0.0)

    They still appear in reward-details.json, separating "wrote nothing" from "wrote the wrong answer" - the first thing worth knowing about a failed run - but contribute nothing to the score. tests/test_scan_correctness_weights.py enforces this across the dataset, so a weighted rk.file_exists, or a file_contains_regex(..., r"\S") that any single character satisfies, fails CI.

    That leaves the checks that carry the weight: rk.command_succeeds, a content-bearing rk.file_contains_regex, and @criterion functions comparing against expected_outputs/ or recomputing the expectation from the task's own inputs.

    2. A criterion must never raise. Reward Kit aborts the whole correctness dimension the moment one does and writes no reward.json at all, so the trial reports no score rather than zero - indistinguishable, in a results table, from a task nobody ran. Read agent output inside try/except and return False:

    @criterion
def result_csv_matches_reference(workspace: Path) -> bool:
    try:
        rows = _read_csv(workspace / "outputs" / "result.csv")
    except (OSError, ValueError) as exc:
        print(f"could not read result.csv: {exc}")
        return False
    ...

    3. Check both ends before opening a PR. The reference solution must score 1.0 and an empty workspace 0.0; neither is implied by the other.

    python scripts/sweep_verifiers.py --mode oracle --task <id>   # must be 1.0
python scripts/sweep_verifiers.py --mode nop    --task <id>   # must be 0.0

    Neither catches a criterion that passes a wrong answer, which is the failure these rules exist to prevent - so also break your own output on purpose (delete a column, shift a number past tolerance) and confirm the score drops.

  • tests/process/ - LLM-as-judge rubric. judge.toml lists the judge backend and a set of [[criterion]] entries (binary or likert) with a description and weight. Use this for things that can't be checked deterministically, e.g. whether the agent's stated reasoning is scientifically sound. The repo default is judge = "claude-code" plus model = "claude-opus-5", which grades by running the real claude CLI (it can browse the workspace itself, so there's no files list to maintain). You can instead set judge directly to a model string (e.g. "anthropic/claude-opus-5") to call that model's API once instead - use this if you specifically want metered API billing instead of Claude subscription billing for grading; see "Debugging a Task Run" below for the credential each mode needs.

tests/test.sh invokes Reward Kit against both, which writes the combined result to /logs/verifier/reward.json and /logs/verifier/reward-details.json.

Keep verifier code small and task-local. Do not reintroduce a central bespoke evaluation framework.

Task Validation

Every task is checked automatically by scripts/validate_tasks.py, which runs in CI on every push and PR (validate-tasks workflow). Run it locally before opening a PR:

python scripts/validate_tasks.py

It checks, per task:

  • all required files listed above are present, and environment/inputs/ is non-empty.
  • task.toml has the required [task], [metadata], [verifier], [agent], and [environment] fields, and [metadata].task_id matches the folder name.
  • [metadata].task_type, system_type and difficulty hold a value from their vocabulary, and every entry in packages is a known canonical name. An invented or misspelled value is an error, not a new category - see "Task Metadata" above. packages must be present but may be empty.
  • tasks/README.md lists every task, with a task type and difficulty matching its task.toml. The index is written by hand, so nothing else stops it drifting away from the tasks it describes.
  • [environment] uses no field Harbor has deprecated: network_mode rather than allow_internet, memory_mb/storage_mb rather than memory/storage. Harbor still accepts the old spellings, migrating them silently behind a DeprecationWarning you will never see in a run's output, so this check is the only thing that catches them. Note the v0.1 task files all use allow_internet, which makes copying a task.toml from an older clone the likely way it reappears.
  • every /app/inputs/... path referenced in instruction.md is tracked in git under environment/inputs/ (glob patterns like dhdl.coul.*.xvg are matched, not required to exist literally). Being tracked matters, not merely existing: a file that sits on your disk but was never committed passes locally and then fails in CI, which is exactly the trap this check closes. A reference to a directory is satisfied when git tracks at least one file inside it.
  • any input that genuinely cannot be committed must be declared in scripts/external_inputs.toml, with a note saying where to obtain it. Treat every entry there as temporary: the validator errors if a declared file later becomes tracked, or if it names a task directory that no longer exists, so the list cannot quietly rot into a permanent hole in the check.
  • every artifact declared in task.toml is mentioned in instruction.md.
  • every file tests/process/judge.toml asks the judge to inspect is either a real input or something instruction.md actually tells the agent to produce.
  • any expected_outputs/<file> referenced in tests/correctness/checks.py actually exists.
  • nothing unexpected is git-tracked at the task root (see "Task Layout" above). This checks git rather than your filesystem on purpose: the .loops/ authoring pipeline keeps writing gathering.md, preparation.md, and solution.md into task directories, and those are fine to have locally - committing them is the problem, since some record expected verifier values in prose. Untracked files never trip this check.
  • both canary GUIDs appear verbatim in CANARY.md, task.toml, tests/correctness/checks.py, tests/process/judge.toml, and - when the task ships a reference solution - solution/solve.sh. This is an exact substring match, so it never produces false positives. instruction.md is deliberately excluded; see "Canary Markers" below.

This only checks structure and cross-references - it does not run the verifier or grade anything. Unit tests for the validator itself live in tests/test_validate_tasks.py.

Note which question each tool answers. scripts/validate_tasks.py checks a task's structure - files, task.toml fields, cross-references. Whether a task's grading logic is correct is a different question, answered by inspecting or running the verifiers under tests/.

scripts/check_verifier_imports.py is the stronger version of one of those checks: rather than reading tests/test.sh statically, it actually resolves each task's verifier environment and imports its checks.py for real, so it catches a declared package that does not install on the platform Harbor runs on. Run it in a Linux container, not on macOS - a package with no prebuilt wheel for your laptop fails to build locally while resolving fine on Linux, which reports failures that are not there:

docker run --rm --volume "$PWD:/repo:ro" --workdir /repo \
  ghcr.io/astral-sh/uv:python3.12-bookworm \
  python3 scripts/check_verifier_imports.py

CI runs it this way on every push, so you normally only need it when debugging a verifier that imports cleanly for you and fails there.

CI runs the validator on Python 3.11. Some of its checks are sensitive to the interpreter version - which stdlib modules are frozen changed in 3.11, for instance - so if a local run disagrees with CI, re-run it on 3.11 before digging further:

uv run --no-project --python 3.11 python scripts/validate_tasks.py

Canary Markers

MDArena is evaluation data. Every task carries two canary GUIDs - the shared BIG-bench one that existing corpus-exclusion pipelines already filter on, and an MDArena-specific one that gives contamination evidence attribution. See CANARY.md at the repository root for the full statement.

When adding a task, copy task_template/CANARY.md verbatim and keep the comment headers that task_template/ already puts at the top of task.toml, tests/correctness/checks.py, tests/process/judge.toml, and solution/solve.sh. The validator enforces this, so a task cannot ship without them.

Do not add a canary to instruction.md. Harbor passes it to the agent verbatim as its prompt, so a marker there would tell the agent mid-task that it is being evaluated - a known behaviour-shifting signal that would contaminate the trajectories the benchmark is trying to measure. The sibling CANARY.md and task.toml carry the marker for that directory instead.

Treat canaries as a filtering and contamination-auditing aid, not as access control: they work only if model developers filter for them, and they do not prevent anyone from training on a public repository.

Debugging a Task Run

To inspect a run after the fact, keep its container alive instead of letting Harbor tear it down:

harbor run -p TASKNAME -a AGENT -m MODEL \
    --no-delete \
    --ek keep_containers=true

--no-delete alone is not enough: Harbor's local Docker backend treats delete=false as "don't delete images/volumes aggressively," not "keep the container" - it still runs docker compose down, which removes the container. Adding --ek keep_containers=true makes Harbor call docker compose stop instead, so the container stays visible under docker ps -a.

Once you have the container ID:

docker start CONTAINER_ID
docker exec -it CONTAINER_ID bash

Inside the container, set the credential the verifier needs. For the repo default (judge = "claude-code" in judge.toml), that's export CLAUDE_CODE_OAUTH_TOKEN=... (get one via claude setup-token, which requires a Claude subscription); for a judge.toml using a plain model string instead, it's export ANTHROPIC_API_KEY=.... Don't export both at once: the claude CLI prefers ANTHROPIC_API_KEY when it's present, which silently defeats subscription billing. Then re-run the verifier manually from the container root:

cd /
uvx --from 'harbor-rewardkit==0.1.*' rewardkit /tests

This runs the same check Harbor's own tests/test.sh runs, with output written to /logs/verifier - useful for iterating on a failing checker without waiting for a full agent run each time. Note the container's clock is UTC, which may be an hour off from local time depending on daylight saving.

Output Conventions

Tasks should declare and request artifacts consistently. The common artifact paths are:

/app/outputs
/app/final_answer.txt

Instructions and tests should use the same paths. Verifiers should avoid relying on files outside /app, /tests, and /logs/verifier.

Large Files

Keep large inputs as small as scientifically possible. MDArena does not use Git LFS - inputs are committed directly, so every megabyte you add is paid for by every clone, every CI run, and every contributor.

Here are a few tips about shrinking an input before you commit it.

  • Include only the atoms required by the task. Remove solvent, ions, membranes, or unrelated chains unless they are scientifically necessary.
  • Include only the required trajectory frames. Extract a short representative segment (e.g., via truncation) or a small set of frames (e.g., via subsampling) rather than committing the complete production trajectory.
  • Use an appropriate compressed format. Prefer formats such as XTC over uncompressed trajectory formats when full coordinate precision is unnecessary.
  • Reduce trajectory precision where appropriate. Do not preserve more decimal precision than the verifier or analysis requires.
  • Avoid duplicate representations. Do not commit the same system in several formats unless each file is explicitly needed.
  • Remove unused metadata and intermediate files. Logs, checkpoints, temporary outputs, and software-generated backup files should not be included as task inputs.
  • Generate large derived files during setup when practical. When an input can be reproduced quickly and deterministically from a smaller source file, commit the source and generation script instead.
  • Use the smallest scientifically valid example. A reduced system or shorter trajectory is preferable when it still tests the intended capability.
  • Test from a fresh clone. Confirm that the task and verifier work without relying on large files that exist only in your local working directory.

Any reduction must preserve the scientific behaviour being tested. Do not shrink an input merely to reduce its size if doing so changes the intended task or makes the verifier unrepresentative. If a task genuinely requires large input files, please discuss this with the maintainers before adding them to the repository.

Style

If you are adding or updating code, such as custom evaluation scripts or verifier helpers, please ensure that your code follows the project's coding style and is free from linting errors. Run ruff check . locally before opening a PR. For any new code, please include docstrings that follow the NumPy docstring standard.