Repository navigation
Build a bounded controlled Java real-fault matrix #304
Description
Parent: #261. Milestone: M5 — Generalizable TCPCI benchmark.
Objective and bounded deliverable
Expand the existing controlled-observation/JUnit source producer beyond one Commons CLI pair. Investigate bounded Defects4J-compatible or direct-upstream source recipes; identify the actual reconstruction protocol rather than claiming framework equivalence.
- Pin independent buggy/fixed revisions, shared fixture, dependencies/notices, runtime/architecture/timezone and producer identity for every selected pair.
- Include materially independent faults/projects, preserve both-pass/both-fail/reversed/inconclusive outcomes and triggering evidence; repeated tests or seeds do not become faults.
- Emit deterministic typed observations/raw-capture provenance with interruption/replay and resource accounting. Source execution must remain isolated and outside Git.
- Admit only reproducible permitted controlled evidence. Constructed sequences retain their labels and never become longitudinal CI.
- Provide a real installed-package matrix workflow and reports separating producer correctness, triggering behavior and candidate policy evaluation. Reuse existing controlled contracts before adding capabilities.
Dependencies
Existing reviewed controlled/JUnit producer; no new child dependency.
Existing implementations remain the starting point: registry/profiles/materialization, RTPTorrent/LRTS readers, 4.7 LRTS ETL, controlled/JUnit producer, native policies/API/checkpoints and #251–#260 foundations. Inspect the actual code before proposing APIs. This issue covers only the remaining work above; already delivered slices are not reopened.
Shared acceptance gates
- Reuse the existing runner, RunSpec, cache, results, immutable artifacts and checkpoints; keep the domain independent of CLI/MCP/network transports. No unrelated refactor or replacement framework.
- Version new contracts; preserve legacy semantics, explicit units/missingness, deterministic identities/serialization and strict future-version rejection.
- Include meaningful unit/golden/compatibility/property or metamorphic tests as applicable, interruption/concurrency/idempotence/recovery checks, and a real installed-package workflow outside checkout without mocking the principal component. Normal tests stay network-free with owned tiny fixtures.
- Human documentation and Mermaid where appropriate must describe actual usable behavior, terms, provenance and evidence limits. No foreign large data/code is vendored.
- Before push: full merge-base review, OpenCodeReview 1.12.13 local delegation/rules plus manual coverage of all exclusions, all relevant tests/lint/format/types/Pylint/strict-docs/package checks and git diff --check. Fix confirmed critical/high/medium findings, review the exact pushed head and resolve valid PR threads before merge.
- Publish actual source/result evidence in the closing PR. Seeds do not establish independence; negative/inconclusive results remain valid.
Activity
jacksonpradolima commentedon Oct 17, 2026
Pre-execution bounded candidate selection (no new source test outcomes collected yet): reuse the unchanged pinned JUnit3 producer and generic controlled-observation contracts for three direct-upstream focal fixes across two codebases:
b0e1b80b6d4a10a9c9f46539bc4c7a3cce55886e→ fixed2ff9573ffb08dd52ec3a55a49f2d77a1e94efbdf. The upstream fix changes PosixParser and adds one test; common fixed fixture declares 96 tests. Exact upstream commit.9c0cabead7cf075308b11362172ae1a48d41321c→ fixed52d82d1dfff8c2b2ded9d843e0b03017af6d747c. Exact upstream fix. All 25 production Java files per role, LICENSE and NOTICE are pinned.Codec uses an explicitly partial common fixture selected before outcomes: retain 15 Java fixture files with original bytes, LICENSE and NOTICE. Omit DoubleMetaphoneTest because it embeds separately noticed aspell data and non-UTF8 bytes; omit SoundexTest because its original bytes are non-UTF8; omit AllTests because it references those excluded fixture classes. The existing source-view contract requires homogeneous reviewed terms and the unchanged compiler explicitly uses UTF-8. No umbrella Apache-2.0 claim is made for the omitted third-party data, and no silent transcoding is performed. The pinned NOTICE remains preserved.
Static pre-execution discovery declares 166 Codec tests; actual runtime discovery must agree. This is a bounded partial fixed-tree reproduction, not the full upstream suite or a complete Defects4J reconstruction. Missing focal-class fixture coverage stays visible.
Candidate recipe fingerprints frozen locally before execution:
sha256:e37b677032009b3fd06a3151adf23bf0bad798868243d8f9a00d26365ff9cb09.sha256:42147c77338ab43bf2142115090f27b3a55649d1d9438e1e9328969fda74892f.The existing Temurin 11.0.32.1 linux/arm64 image digest, explicit America/Los_Angeles timezone, harness and raw JAR pins remain unchanged. Acquisition and execution stay outside Git; normal tests stay network-free. Both CLI fixes share project history. Three curated fixes, multiple triggering methods and repeated trials do not establish statistical independence or additional projects/faults. Both-pass/both-fail/reversed/inconclusive outcomes and failed/interrupted collections will remain evidence. Protected policy evaluation and corpus admission remain separate work in #309 onward.
jacksonpradolima commentedon Oct 17, 2026
Source inspection after the initial qualification found unseeded
java.util.Randominputs in Codec's pinned Base64Test, HexTest and DigestUtilsTest. Their generated bytes/seeds are not captured by the existing producer. All 166 selected tests remain included; no new fixture exclusions, source edits, transcoding or deterministic seed injection were introduced.The pre-execution source/test selection above is unchanged. A conscious provenance-only correction appends this limitation to the Codec recipe and matrix plan:
sha256:42147c77338ab43bf2142115090f27b3a55649d1d9438e1e9328969fda74892fsha256:ca84d07e430d5397d697924b183e998cacb8501e9d84e7763fa9beedeef79fc7sha256:9b66051dcf34c386ddc550eb215bec9f2259a73d9d1e083f8955ca8e663c5d35sha256:872f7281dad67e8bacd3ad5f444e93f01a9d40d43eecb8f767503561138f4d9bTyped comparison confirms all source views, member hashes, dependencies, test IDs, runtime, harness, trial labels and fixture-selection text are unchanged. Older captured artifacts were retained and validated without migration/reinterpretation. Deterministic replay of saved observations is distinct from reconstructing identical fresh generated test inputs; scientific admission must assess that limitation.
The first frozen implementation passed 2,670 tests, one Linux-only skip and 95.26% coverage, including an actual installed three-source matrix and prior real CI workflows. The annotation/regression update passed 51 owned tests and all static/docs/package checks; the final exact-head full suite is running before any branch push. This remains local implementation/validation, not completed corpus admission or a claim of independent faults/policy efficacy.