Repository navigation
Keep evidence for every part of a support question - #71
aryanorastar wants to merge 10 commits into
Conversation
|
October 24 release checkpoint: Railway already records exact main 1235379, and the requested narrow redirect/private-delivery smokes now pass. No duplicate deployment or new provider run was dispatched. The simultaneous comparison already completed October 23 against this same main and frozen candidate 55799ad (162 valid rows per arm); current c3b4b2b differs from that candidate only in documentation and has green exact-head CI. Reuse the saved evidence rather than silently rerun an unchanged candidate. The independent output grade remains an accuracy HOLD: a negative compatibility answer contradicts its cited official source, and citation/fallback/private-availability claims need review. Main is also inconsistent, so this is not proof that #71 introduced those failures. Aggregate pass-both 70→73, unwanted handoffs 13→10 and missing flags 5→3 do not certify factual accuracy. Response time alone is not a blocker under Aryan’s stated priority. The PR stays draft pending the remaining review decision; no merge or deployment approval is inferred. |
|
October 24 reviewer decision update: Aryan explicitly confirms the numerical latency waiver for p95 55.599 s → 63.167 s (+7.568 s versus the original +6 s limit), CONDITIONAL on the accuracy review passing. This is not accuracy approval or merge readiness. A private result-only review package is complete and independently integrity-checked: every flagged case has both runs from both arms verbatim with signals/problems, all 324 original rows retained, and occurrence by arm distinguished from causal attribution. No protected-input read or provider rerun. Accuracy HOLD remains; no merge or deployment. |
|
Reviewer grade received October 24: NOT APPROVED; keep draft. The result-only text review finds aggregate helpfulness improvements but more harmful candidate-specific outputs: unsupported compatibility/source-absence claims and customer-specific data reassurance. Main shares the false-absence family, so the per-part evidence spread is a plausible, unproven mechanism, not established causation. Additional losses include source attribution and fallback language. The conditional latency waiver remains recorded, but it does not clear accuracy. Approved order: two small deterministic production fixes first (orphaned redirect fragments and localized review-rejection fallback); then one general excerpt/absence/data-reassurance accuracy fix, independently reviewed before any provider run; then one simultaneous main | main+accuracy | main+accuracy+#71 comparison. Existing outputs stay preserved; no protected-input read or automatic unchanged rerun. |
|
October 25 final deciding run: Aryan explicitly authorises one simultaneous deployed-main d550dc1 versus #71-alone comparison, 81 cases × 2 runs per arm, concurrency 3 per arm. Main is integrated with merge commit 137cd47; local 992/992, offline 8/8 and exact-head Node 20/24/Postgres CI40172310231 pass. Independent launch review and 128 affected offline checks pass for the exact frozen source/wrapper/supervisor. The run is dispatched once; no paid preflight, selective rerun or further rework/rerun. No #82 excerpt contract is included. This is setup approval, not accuracy/readiness. Full independent output/source reading decides merge only if approved with no material finding absent from main; otherwise close, preserving branch/tests and #65. No automatic deployment or optional enablement. |
|
Closed without merging under Aryan’s one-final-attempt decision. Frozen deployed main d550dc1 versus standalone #71 at137cd47 each saved162 rows; the independent reviewer read all324 final replies plus four earlier-turn replies. The required generated-answer comparison cannot be accepted: every one of the81 run2 rows in each arm has no recorded model usage and finishes under100ms, with71 explicit unavailable fallbacks per arm. Recorded invalid=0 is not proof of valid generation. The supervisor also stopped before final validation. A separate saved-only audit verifies original hashes, source/input metadata, complete historical power coverage with no sleep events, and zero known-secret matches; that does not rescue the failed generation gate. Exact original execution-error causes were not retained, so this is NOT a claim that #71 caused the execution failure or a proven code regression. Local992/offline8 and exact-head CI still pass, but cannot replace accuracy approval. No further rework or rerun is authorised. Preserve branch fix/multipart-part-evidence, its tests and all comparison evidence; #65 stays open. Source-conflict work is tracked separately in #83, not implemented here. Production remains d550dc1; no merge, deploy or optional enablement. |
|
Final independent saved-output grade: HOLD, mergeApproval=false. All324 final plus4 earlier-turn replies read; all324 row hashes and three original file hashes rechecked unchanged. The full generated-quality gate remains invalid because every second-run row lacks recorded usage and71 per arm are explicit unavailable fallbacks. Additional observed candidate-only grounding/citation concerns reinforce the hold, without establishing deterministic patch causation. Source nuance: official documentation DOES support general device/source merging boundaries; it does not itself verify an individual duplicate’s cause. No claim that the mechanism is absent. Complete private grade/coverage and immutable result evidence are retained; no further rerun/rework, merge or deployment. |
Customer failure
Latest independent accuracy review, October25: HOLD remains. The single simultaneous frozen comparison is deployed main
d550dc1| Cfac7d17| C+#717c03601, each162 rows/zero invalid. All486 final outputs plus six earlier-turn outputs were read, with official-source checks. Both C and C+#71 receive HOLD for observed factual/completeness/citation/source-conflict problems, not latency. The C+#71 stack still contains unsupported feature exclusion/diagnosis and incorrect citation attribution; passed model claim checks missed them. This does not establish which code change caused an individual failure or verify every sealed question's intent.Original children finished normally with retained structural failures; the supervisor then stopped during power-history collection. Separate independent streamed historical power validation and post-hoc saved-result/source/row/date/hash checks pass; the original five artifacts/provenance are unchanged. Structural pass-both73→68→68 and unwanted flags11→21→19 are not a factual grade. No paid rerun, merge, deploy or optional enablement. Investigate general evidence/claim/citation handling with fresh original synthetic controls, never held-out wording/URLs; any paid comparison requires Aryan's approval. This checkpoint supersedes the earlier operational snapshots below, preserving their evidence.
Latest result/decision, October23: both simultaneous arms complete162 rows/81 cases×2, zero invalid/unscored, same UTC/IST day and11ms start skew. Structural pass-both70→73; unwanted13→10; missing5→3. Stage-total p50/p9534.096/55.598→34.417/63.164s. Original p95 limit fails, but Aryan now explicitly prioritizes accuracy/reliability over time: latency increases alone no longer block. Independent output review instead records an accuracy HOLD for evidence-negative/citation contradiction, case-specific recording-availability reassurance and fallback-language/citation gaps; patch causation is not established. No clipping regression observed in changed outputs; source-free output review cannot certify every hidden expected intent/part. No automatic rerun/merge/deploy. The completed result package is retained privately; no protected input was inspected.
Two- and three-part support questions can retrieve the right official page and still lose one requested part before answer generation. Reserve evidence for every distinct neutral part, not only questions with more than three parts. No phone-specific wording, hard-coded URLs or held-out-case tailoring.
General repair
Current verification
Latest publication c3b4b2b changes only five status/architecture Markdown files versus evaluated55799ad. Fresh972/offline8 and exact-head Node20/24/PostgreSQL CI39999319754 pass. Evaluated runtime and immutable result JSONs are unchanged. Independent grade remains accuracy HOLD; this is not readiness or deployment.
Current main123537974104649c4ecce9bee472029dcbbbd161 integrated by merge commits; candidate55799ad253d6bafc36b6bbf7946c55014497df89. Includes Bound provider requests over the whole response, not just headers #76/Recover abandoned unsent replies after missed heartbeats #70, merged Keep ordinary support and failed follow-ups in the same thread #67/Remove owned correction links when forgetting support history #74 and test-only Make provider deadline tests independent of scheduler timing #79; these later merges are not deployed. Live remains6b6d97d on the accepted reliability release.
Fresh972/972 and no-provider evaluation8/8 pass immediately before push. Exact-head Node20/24/PostgreSQL CI39991802703 passes.
Independent134 focused regressions and final integration review pass; required-chunk ordering and three-chunks-on-one-page findings were observed red, fixed and re-reviewed. The final Make provider deadline tests independent of scheduler timing #79 integration changes only its already-reviewed test and status documents, not the reviewed runtime repair.
Four deterministic live/stored fetch-contract checks pass both baseline and candidate. Six deliberate in-memory faults are rejected: seven-page amplification, serialized page fetch, changed timeout, serialized stored searches, increased row limit and unnecessary stored-path HTTP fallback.
Live docs remain≤6 fetched pages concurrently with10s per-page timeout; separate index fetch excluded. Stored retrieval retains≤5 queries, each3 parallel searches with12/all,5/help,5/GitHub limits and no HTTP fallback when stored results suffice. This verifies caps/parallelism/timeouts, not identical URLs or counts for every input.
Earlier Keep evidence for every part of a support question #71 full gate failed the retained deadline test's750ms oracle at1184ms. Separate test-only Make provider deadline tests independent of scheduler timing #79 now fixes the illustrative interval assumption AND that stopwatch assertion with client-observed partial-body/real HTTP cancellation checks. That failure is retained, not called a passed rerun.
Historical timing reading — not a proven code cause
The October20 run's largest tails are consecutive IDs gs-031/032/033 run1 (115–143s), with missing planner/review usage on two and docs-fetch failures2→23. IDs are not recorded wall-clock times. A transient docs-site/provider window is plausible, not proven. Page selection differs; unchanged caps alone cannot exclude source-specific failures. The old baseline predates #76/#70.
Per-stage historical p50/p95 seconds, main→candidate: planning7.755/12.994→7.793/14.039; retrieval1.091/3.933→1.347/9.999; answer5.816/10.749→6.120/10.918; review10.200/20.026→10.569/25.804. Per-row stage totals27.171/42.508→27.865/61.332 can combine multiple turns; percentile stages are not additive. Preserve all original slow/error rows and the failed+18.824s p95 gate. This patch is not presented as a latency fix.
Remaining gate
Complete, not approved: main1235379 / evaluated candidate55799ad each retain162 rows/concurrency3. Finishes17:44:36.870/17:44:35.968UTC, normal structural exits1; supervisor exit0, no signal/stop. Five preflight cases complete4/5 structural passes, zero invalid, max54.374s. Shared harness/judge/dependencies and host/source/same-day guards pass; zero known-secret matches. No raw child log text or production DB/Discord credentials. Original failed comparison preserved; do not rerun rows.
Repair the general evidence/negative-claim/citation/fallback cause using original synthetic regressions, not held-out phrases, feature names or hard-coded URLs. Complete protected-intent/part adjudication only with permitted evidence. Structural improvements do not clear the factual HOLD; latency alone is not the blocker. No new provider run follows automatically, and no merge/deployment/memory enablement is approved. Later status-only publication may change the PR head; evaluated runtime55799ad and the immutable JSON hashes stay recorded.