Skip to content

First expert-review round: one blinded workbook for three reviews (#23) - #45

Open
NingyuSUN wants to merge 1 commit into
mainfrom
expert-round-1
Open

NingyuSUN wants to merge 1 commit into
mainfrom
expert-round-1

Conversation

@NingyuSUN

Copy link
Copy Markdown
Owner

Adds the first expert-review round for #23: one workbook that experts fill in, plus the importer that turns their answers into each review's own format.

The workbook

Each part matches a field of expertise, and an expert does only the parts in their field.

Part Items Feeds
A · Variants 40 ClinVar germline classifications, Sept 2023: 20 divergent and 20 control, not marked as such ClinVar review kit (#23), scoring of the model reviewers (#22)
B · Paper claims the 60-claim extraction expert sample natural extraction errors with taxonomy codes (#21)
C · Single-cell a fresh audit sample: 59 auto-admitted, 10 others, not marked as such audit of what bioevidence admits (#24)
  • Every answer is a dropdown with plain options: "yes / no / unsure", and "usable as is / needs a curator's check / not usable".
  • The options live on a hidden sheet, so options containing commas don't break the dropdown.
  • The start sheet gives six rules in English and Chinese.
  • Blinding: the workbook shows no model, route, validator decision or reference label. Parts B and C get new item codes. The seed, the item key and the audit manifest stay in the maintainer's private folder.

The importer

import_round.py writes:

  • part A as protocol annotations, through import_sheets.convert;
  • part B as rows in the extraction sample's labelling format;
  • part C as protocol annotations that bioevidence review audit-score scores against the private manifest.

It rejects half-answered rows, unknown options, a missing reviewer ID and a repeat import of the same reviewer. Nothing is written unless every part is clean.

Checks

  • A round trip on the real round passed: built, filled as an expert would, imported, then checked with review check and scored with audit-score. 120 ClinVar annotations, 60 claim labels and 69 audit annotations came back.
  • 3 new tests cover blinding, dropdowns, the 20/20 balance, the import, partial and invalid answers, and repeat imports.
  • ruff, mypy and pytest pass, with 98.6% coverage. mkdocs build --strict passes.

One blinded workbook for the three reviews that wait on experts: 40
ClinVar classifications (half divergent, half control, unmarked), the
60-claim extraction sample, and a fresh single-cell audit sample (59
auto-admitted, 10 others). Every answer is a dropdown with plain options,
the start sheet gives six rules in English and Chinese, and items of
parts B and C get new codes. import_round.py writes each part back into
its review's own format and refuses half-answered rows, unknown options
and a second import of the same reviewer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant