research: presentation changes must not orphan verified corrections - #77
Conversation
Separate answer rendering identity from factual revision identity without relaxing exact receipt bindings. Keep earlier-answer outcomes visible but explicitly attributed to that earlier answer. Add 20 boundary test methods, a pinned before/after audit (21 overlapping perturbations), an offline HTML report and a provider-neutral A/B/C input exporter. Original 52 tests remain unchanged. No LLM runs, participants, token savings, deployment or efficacy claim. Stacked continuation of draft PR #76; do not merge automatically.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review Please review candidate |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
base='0d83d3e767756e40b5fceab4fe207a524c152ede'
head='6380a8324b53e8b721386660fa80421028cbf85c'
echo '== Commit availability and metadata =='
git cat-file -e "${base}^{commit}"
git cat-file -e "${head}^{commit}"
git show -s --format='BASE%n%H%n%P%n%s%n%nHEAD%n%H%n%P%n%s' "$base" "$head"
echo '== Changed paths =='
git diff --name-status "$base" "$head"
echo '== Diff summary =='
git diff --stat "$base" "$head"
echo '== Relevant filename discovery =='
fd -HI -t f '^(paired|lineage_probe|prepare_abc|verify|LINEAGE)\.(py|md)$|^(.*\.(yml|yaml))$' . \
| sed -n '1,240p'
echo '== Python structural outlines =='
while IFS= read -r f; do
echo "--- $f"
ast-grep outline "$f" --items all --type function,class 2>/dev/null || true
done < <(git diff --name-only "$base" "$head" -- '*.py')
echo '== Workflow structural outlines =='
while IFS= read -r f; do
echo "--- $f"
sed -n '1,280p' "$f"
done < <(git diff --name-only "$base" "$head" -- '.github/workflows/*.yml' '.github/workflows/*.yaml')
echo '== Targeted diff: lineage implementation, tests, CI, and documentation =='
git diff --find-renames --find-copies --unified=80 "$base" "$head" -- \
'benchmarks/paired-feedback-v0.1/*.py' \
'.github/workflows/*.yml' '.github/workflows/*.yaml' \
'benchmarks/paired-feedback-v0.1/*.md' \
'reports/science/paired-feedback-lineage-2026-09-05.ru.md' \
| sed -n '1,2200p'
echo '== Static references to lineage and experimental-boundary terms =='
rg -n -i -C 3 \
'fact_revision|related_outcome_observations|answer_id|fingerprint|baseline|lineage_probe|prepare_abc|comparative|participants|preregister|inference adapter|efficacy|deployment|auto-merge' \
benchmarks .github reports 2>/dev/null \
| sed -n '1,2600p'Length of output: 7709 🏁 Script executed: #!/bin/bash
set -euo pipefail
base='0d83d3e767756e40b5fceab4fe207a524c152ede'
head='6380a8324b53e8b721386660fa80421028cbf85c'
echo '== Candidate Python outlines =='
for f in \
benchmarks/paired-feedback-v0.1/paired.py \
benchmarks/paired-feedback-v0.1/verify.py \
benchmarks/paired-feedback-v0.1/test_lineage.py \
benchmarks/paired-feedback-v0.1/lineage_probe.py \
benchmarks/paired-feedback-v0.1/prepare_abc.py
do
echo "--- $f"
ast-grep outline "$f" --items all --type function,class 2>/dev/null || true
done
echo '== Candidate implementation: paired.py =='
cat -n benchmarks/paired-feedback-v0.1/paired.py
echo '== Baseline implementation: paired.py =='
git show "${base}:benchmarks/paired-feedback-v0.1/paired.py" | cat -n
echo '== Candidate tests: test_lineage.py =='
cat -n benchmarks/paired-feedback-v0.1/test_lineage.py
echo '== Candidate verifier and workflow =='
cat -n benchmarks/paired-feedback-v0.1/verify.py
cat -n .github/workflows/paired-feedback.yml
echo '== Candidate lineage probe and A/B/C generator =='
cat -n benchmarks/paired-feedback-v0.1/lineage_probe.py
cat -n benchmarks/paired-feedback-v0.1/prepare_abc.py
echo '== Candidate boundaries documentation =='
cat -n benchmarks/paired-feedback-v0.1/LINEAGE.md
cat -n reports/science/paired-feedback-lineage-2026-09-05.ru.md
echo '== Exact candidate diff =='
git diff --find-renames --unified=120 "$base" "$head" -- \
.github/workflows/paired-feedback.yml \
benchmarks/paired-feedback-v0.1/paired.py \
benchmarks/paired-feedback-v0.1/verify.py \
benchmarks/paired-feedback-v0.1/test_lineage.py \
benchmarks/paired-feedback-v0.1/lineage_probe.py \
benchmarks/paired-feedback-v0.1/prepare_abc.py \
benchmarks/paired-feedback-v0.1/LINEAGE.md \
reports/science/paired-feedback-lineage-2026-09-05.ru.md
echo '== All direct calls to relevant APIs =='
rg -n -C 4 \
'\b(build_answer|apply|accept|receipt|outcome|fact_revision|related_outcome_observations|answer_id)\s*\(' \
benchmarks/paired-feedback-v0.1 \
-g '*.py'Length of output: 155483 🏁 Script executed: #!/bin/bash
set -euo pipefail
base='0d83d3e767756e40b5fceab4fe207a524c152ede'
head='6380a8324b53e8b721386660fa80421028cbf85c'
repo='safal207/RESONANCE'
echo '== Immutable source fingerprints =='
printf 'baseline paired.py SHA-256: '
git show "${base}:benchmarks/paired-feedback-v0.1/paired.py" | sha256sum
printf 'candidate paired.py SHA-256: '
git show "${head}:benchmarks/paired-feedback-v0.1/paired.py" | sha256sum
echo 'Declared baseline fingerprint:'
rg -n '^BASELINE_SHA256|^BASELINE_COMMIT' benchmarks/paired-feedback-v0.1/lineage_probe.py
echo '== Candidate ancestry and exact changed source identity =='
git merge-base --is-ancestor "$base" "$head" && echo 'baseline is an ancestor of candidate'
git diff --quiet "$base" "${head}^" && echo 'candidate parent equals baseline'
git diff --name-only "${head}^" "$head" -- benchmarks/paired-feedback-v0.1/paired.py
echo '== Pull request metadata =='
gh pr view 77 --repo "$repo" --json number,state,isDraft,headRefName,headRefOid,baseRefName,baseRefOid,url
echo '== Paired-feedback workflow runs for this pull request =='
gh run list --repo "$repo" --workflow paired-feedback.yml --branch "$(gh pr view 77 --repo "$repo" --json headRefName --jq .headRefName)" --limit 10 \
--json databaseId,headSha,event,status,conclusion,workflowName,url,createdAt,updatedAt
echo '== Generated artifact and ignore-rule checks =='
git ls-tree -r --name-only "$head" | rg -n '(^|/)(results|abc-smoke|lineage-audit)\b|operator-key\.json|assessor-oracle\.json|fingerprints\.json' || true
fd -HI -t f '^\.gitignore$' . -x sh -c 'echo "--- $1"; cat -n "$1"' sh {} \;Length of output: 3202
I found no blocking issues.
The review did not treat the deterministic fixture comparison as an independent human study or scientific replication.
|
Keep verification active under optimized Python; preserve distinct HTML/JSON output paths; exclude skipped/expected failures from pass counts; normalize fingerprint paths; disable checkout credential persistence. Add eight regression tests and normal/optimized CI runs. Original 52 tests and the pinned baseline are unchanged. 80/80 local tests passed in both interpreter modes; these are repeated developer checks, not model or human efficacy evidence.
Follow-up: validation must survive interpreter optimizationNew exact head: Verified the seven #76 review findings against local source. Factual-lineage handling was addressed in the first candidate. This follow-up disables checkout credential persistence, rejects non-HTML output paths before writing, uses unconditional validation in the bridge/report/new probe/exporter, records skipped/expected-failure counts separately, and normalizes fingerprint paths. Added eight regression methods: 80/80 local tests, with zero skips/expected failures, passed both normally and under The previous candidate @coderabbitai review |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
base='0d83d3e767756e40b5fceab4fe207a524c152ede'
previous='6380a8324b53e8b721386660fa80421028cbf85c'
head='b068a5d18a90b7215c6dbc7d91aa2359078c035c'
repo='safal207/RESONANCE'
echo '== Commit identity and ancestry =='
git cat-file -e "${base}^{commit}"
git cat-file -e "${previous}^{commit}"
git cat-file -e "${head}^{commit}"
git show -s --format='%H%nparents: %P%nsubject: %s' "$base" "$previous" "$head"
git merge-base --is-ancestor "$base" "$head" && echo 'baseline is ancestor of updated head'
git merge-base --is-ancestor "$previous" "$head" && echo 'previous candidate is ancestor of updated head'
echo '== Changed paths: previous candidate to updated head =='
git diff --name-status "$previous" "$head"
echo '== Changed paths: pinned baseline to updated head =='
git diff --name-status "$base" "$head"
echo '== Diff from previous candidate =='
git diff --find-renames --find-copies --unified=100 "$previous" "$head" -- \
.github/workflows \
benchmarks/paired-feedback-v0.1 \
reports/science \
| sed -n '1,3200p'
echo '== Python and workflow outlines for changed files =='
git diff --name-only "$previous" "$head" -- '*.py' '*.yml' '*.yaml' '*.md' \
| while IFS= read -r f; do
echo "--- $f"
case "$f" in
*.py) ast-grep outline "$f" --items all --type function,class 2>/dev/null || true ;;
*) cat -n "$f" | sed -n '1,900p' ;;
esac
done
echo '== Target implementations and tests =='
for f in \
benchmarks/paired-feedback-v0.1/paired.py \
benchmarks/paired-feedback-v0.1/verify.py \
benchmarks/paired-feedback-v0.1/lineage_probe.py \
benchmarks/paired-feedback-v0.1/prepare_abc.py \
benchmarks/paired-feedback-v0.1/test_lineage.py \
.github/workflows/paired-feedback.yml
do
if test -f "$f"; then
echo "--- $f"
cat -n "$f" | sed -n '1,1800p'
fi
done
echo '== Validation mechanisms and write sites =='
rg -n -i -C 5 \
'\bassert\b|raise\s+(ValueError|RuntimeError|SystemExit)|html|write_text|open\s*\(|Path\s*\(|BASELINE_SHA256|BASELINE_COMMIT|fingerprint|persist-credentials|python\s+-O|unittest|expectedFailure|skip' \
benchmarks/paired-feedback-v0.1 .github/workflows \
-g '*.py' -g '*.yml' -g '*.yaml'
echo '== Pull request and workflow-run metadata =='
gh pr view 77 --repo "$repo" --json number,state,isDraft,headRefOid,baseRefOid,headRefName,url
gh run list --repo "$repo" --commit "$head" --limit 20 \
--json databaseId,headSha,event,status,conclusion,workflowName,url,createdAt,updatedAtLength of output: 195171
I found no blocking issues.
This review treats the deterministic checks as code validation. It does not treat them as independent scientific replication or as evidence from comparative LLM runs or human participants. You are interacting with an AI system. |
1 participant
Stacked continuation of #76 — no merge or site deployment
Base pinned to
0d83d3e767756e40b5fceab4fe207a524c152ede. Current candidate:b068a5d18a90b7215c6dbc7d91aa2359078c035c. The #76 branch andmainare unchanged.Reproduced issue
A presentation-only a1 → a2 transition caused an otherwise bound accepted correction addressed to a1 to be ignored as stale. The exact same fixture worked when the answer was full from the start OR compact from the start. The failure was the presentation transition, not brevity.
Candidate repair
answer_idfromfact_revision; presentation and pending-review changes retain factual lineage.Review follow-up
CodeRabbit's #76 review examined the reported lineage issue and additional validation concerns. The candidate disables checkout credential persistence; rejects invalid HTML output suffixes before writing; keeps all runtime checks in bridge/report/probe/exporter active under Python
-O; excludes skipped/expected failures from passed counts; and uses POSIX fingerprint paths. Original 52 test methods are unchanged; 20 lineage and 8 validation methods were added. #76 review threads remain open because its branch has not been modified. This candidate still requires final review; automated code review is not independent scientific replication.Actually observed
Exact-head CI evidence verified
Run: https://github.com/safal207/RESONANCE/actions/runs/33965487680
Head:
b068a5d18a90b7215c6dbc7d91aa2359078c035cArtifact:
paired-feedback-dev-evidence, ID9969278057, 91,654 bytes.SHA-256:
8acce3bdfd7bda32bb5dee1919c3512009a0de85e664f47d8bd0d2330afe038c.Downloaded ZIP integrity and digest checked. Every reported Python source hash matches local bytes. Normal/optimized CI summaries and audit/bridge/A-B-C JSON match each other; audit/bridge/plan match local results. This is repeatability in another environment, not independent research replication.
Reproduce
Open
results/lineage-audit.html. SeeLINEAGE.md,REVIEW_FOLLOWUP.md, and the two Russian research notes underreports/science/paired-feedback-{lineage,validation-followup}-2026-09-05.ru.md.Boundaries
0 comparative LLM runs; 0 participants. Token-saving, pressure reduction, scientific novelty, adoption and commercial efficacy are unmeasured. Receipt registry/state are a trusted local test boundary, not source authentication or action authority. The A/B/C export is not an inference adapter or preregistered efficacy experiment. Existing persisted states need regeneration; no migration implemented. No auto-merge or journal-site deployment.