Skip to content

TODOS.md

Latest commit

 

History

History
138 lines (111 loc) · 6.74 KB

TODOS.md

File metadata and controls

138 lines (111 loc) · 6.74 KB
Raw

TODOs

Numbering is stable: closed items are deleted rather than renumbered, so the gaps are expected and a reference to "item 15" means the same thing next month. What a closed item established is recorded in docs/DATASET_VERSIONS.md, in the commit that closed it.

Blocking the v0.2 release, but only at the very end. Item 18 must be the last thing done before the release is published, and it must come after the evaluation sweep rather than before it.

Blocked on something other than effort. Item 4 needs a live-agent run over all 50 tasks, which is the v0.2 evaluation itself. Item 11 needs runtime data from a sweep that does not hit the timeout ceiling, since a timed-out trial's duration is censored. Items 5 and 6 need input from whoever authored the affected tasks rather than a code change.

Neither blocking nor urgent. Item 10 is dataset-growth planning for v0.3 and beyond, not a defect.

Everything here is captured with enough context that it does not need re-investigating from scratch.

5. Cite the primary publications for PDB-derived structures

Status: identified 2026-07-31. Agreed to add the citations in the next version of the manuscript; the repo-side table is still to be written.

No legal issue. wwPDB/RCSB release structural data without copyright restriction, so both redistributing raw entries and shipping processed derivatives are fine. This is a scientific-credit item, not a licensing one.

11 of the 23 tracked .pdb files are unmodified RCSB downloads, each with full HEADER/REMARK/JRNL records (task 03's has 518 REMARK lines, task 23's has 862):

Task File PDB ID
03 environment/inputs/Receptor.pdb 3PWH
07 environment/inputs/tyk2_complex.pdb 6X8F
13 environment/inputs/1A6M.pdb and tests/correctness/expected_outputs/reference/1A6M.pdb 1A6M
21 environment/inputs/{4s0v,5wqc,5ws3}.pdb 4S0V, 5WQC, 5WS3
23 environment/inputs/structure.pdb 4EIY
25 environment/inputs/structure.pdb 5CGC
26 environment/inputs/structure.pdb 3PBL
27 environment/inputs/structure.pdb 6GT3
29 environment/inputs/protein.pdb 9BIS
37 solution/4EIY.pdb 4EIY

The citations for these can be extracted directly: grep '^JRNL' <file>.

The other 12 are processed (GROMACS/MDAnalysis/PDBFixer output, headers stripped, starting at CRYST1 or ATOM). Processing does not discharge the citation obligation - the obligation follows the scientific content, not the file format - and these actually need more attention, because the provenance is no longer recoverable from the file itself. Note 44_traj_ana_atp8b_rmsd/ environment/inputs/reference.pdb looks raw at a glance (bare HEADER line) but its second line is TITLE MDANALYSIS FRAME 0: Created by PDBWriter.

Some structure IDs appear only in instruction.md with no raw file shipped - e.g. task 30 uses 6NQ0. At least the DHODH (task 32), ATP8B1 (44/45), and PepT2 (22/28/29/38) structures are not named anywhere machine-readable and need tracing by whoever prepared them.

What to do:

  1. Extract JRNL records from the 11 raw files for the citation list.
  2. Ask task authors for the source PDB ID of each processed structure.
  3. Add a task -> PDB ID -> primary citation table to DATA_PROVENANCE.md, so a reader picking up e.g. task 32 can tell which structure it derives from.
  4. Cite the primary publications in the next manuscript version.

6. Trace the provenance of the FEP ligand sets

Status: identified 2026-07-31, not done. This is the highest-risk redistribution item in the repo and the one most likely to draw an actual complaint - unlike the force fields, these are somebody else's curated benchmark collection rather than a community-standard parameter set.

What is affected: 48 tracked .sdf/.mol2/.mol files across 8 tasks - 24_ligand_parameterization_smirnoff_openfe, 32_fep_openfe_rbfe_dhodh, 33_fep_openfe_rbfe_d3, 34_fep_openfe_rbfe_def_ox2_s1, 35_fep_openfe_abfe_d3, 36_fep_openfe_abfe_mglu5r, 37_fep_openfe_abfe_a2ar, 38_pept2_qm_cluster_orca.

What is already known: task 32's gathering.md (removed from the repo, archived in MDArena_authoring_artifacts/) traced its inputs to Uni-FEP-Benchmarks/uni_fep_benchmarks/ChEMBL|DHODH|CHEMBL879486/, i.e. a ChEMBL-derived FEP benchmark collection on the group's cluster. The TYK2 set (task 07) is plausibly the Schrodinger/OpenFE JACS benchmark set, but that has not been confirmed. Ligand names differ per task - 11a/11b/12a/12b/ 13a/13b for D3, numeric IDs like 42922/43289 for OX2R, 4g-4r for A2A, 6-17 for mGlu5 - which suggests several different upstream sources rather than one.

What to do: one question to whoever authored each of these tasks - "where did these ligand files come from?" - then record source, licence, and requested citation per task in DATA_PROVENANCE.md. That is the whole job; no legal research is needed unless an upstream source turns out to restrict redistribution.

10. System-type coverage gaps to fill as tasks are added

Status: identified 2026-08-06 while designing the [metadata] taxonomy, not acted on. This is a dataset-growth item, not a defect - nothing is broken, but the gaps below are the ones a reviewer is most likely to name.

Assigning all 50 tasks to the proposed system_type values gives:

system_type n %
membrane_protein 29 58
soluble_protein 10 20
small_molecule_system 7 14
peptide 3 6
none (task 42, no molecular system) 1 2

Missing entirely: nucleic acids. There is no DNA or RNA task, and no protein-nucleic acid complex. For a benchmark claiming breadth in biomolecular simulation this is the most visible omission - nucleic-acid MD has its own force fields, ion-treatment conventions, and analysis idioms that none of the 50 tasks exercise. nucleic_acid and protein_nucleic_acid_complex are the types to add first.

After that, roughly in order of how often a reviewer would ask:

  • protein_protein_complex / antibody - pharma-relevant, and ATP8B1-CDC50A (tasks 44, 45) is currently the only complex, filed under membrane_protein.
  • intrinsically_disordered_protein - a known hard case for MD.
  • glycan as a first-class type rather than incidental to the glycoinsulin tasks (11, 50).

Balance target. Do not chase a uniform distribution. Membrane-heaviness is the benchmark's genuine differentiator and carries the manuscript's headline finding, so grow the other types rather than trimming membrane. The real constraint is that a per-type performance figure needs about n>=5 to say anything; on that basis peptide (n=3) is already too thin and should either grow or fold into soluble_protein until it does.