Build a diversified, versioned benchmark corpus for adaptive CI test prioritization

Execution plan and delivered slices

This issue remains the public umbrella. Native GitHub sub-issues track the remaining implementation; their existence means planned work, not implementation or source admission. Work sequentially and review one bounded runnable slice at a time. Existing 4.0–4.7 foundations, source readers, controlled/JUnit producer and real LRTS ETL remain delivered and must be reused.

Order	Subtask	Dependencies
1	#301 — Add bounded ZIP64 source selection for real benchmark archives — published in 4.8 via #314 / #316	Existing reviewed main
2	#302 — Deliver real RTPTorrent Parquet ETL and native replay — published in 4.8 via #315 / #316	#301
3	#303 — Expand LRTS source slices and resolve stage and temporal admission evidence — published in 4.9 via #318	Existing reviewed main
4	#304 — Build a bounded controlled Java real-fault matrix	Existing reviewed main
5	#305 — Add a reproducible controlled non-Java real-fault source	Existing reviewed main
6	#306 — Add a permitted cross-language seeded-fault benchmark slice	Existing reviewed main
7	#307 — Expand HCS topology coverage and independent variant accounting	Existing reviewed main
8	#308 — Implement the deterministic interaction-covering CI stress laboratory	Existing reviewed main
9	#309 — Integrate evidence-backed source admission and production-reality coverage	#302, #303, #304, #305, #306, #307, #308
10	#310 — Validate corpus metric references and protected repeated-measures protocols	#309
11	#311 — Run a comparable baseline matrix with explicit external-method eligibility	#309, #310
12	#312 — Assess public GitHub Actions or GitLab prospective collection feasibility	#309
13	#313 — Freeze and execute the confirmatory corpus and publish the cross-corpus report	#309, #310, #311, #312

Start with #301, then #302, because the RTPTorrent real-data path needs an explicit bounded ZIP64 source profile. Java/non-Java/other sources do not acquire artificial dependencies on that archive profile. No overlapping implementations or duplicate experiment framework. Compatible source work may be integrated in a reviewed 4.8 source branch before publication; do not publish an infrastructure-only source pipeline as if the complete ETL had shipped. Security corrections remain independently shippable.

#301–#306 are delivered (6/13 native subtasks) in published versions 4.8–4.12. Source PRs #314/#315/#316, #318, #320, #322 and #324 and their reviewed documentation follow-ups implement the bounded public slices. The actual signed 4.12 wheel, real source/native/checkpoint/interruption workflows, release assets and deployed documentation were qualified. #325 publishes current verified installation links and examples; its automatic release correctly produced no bump. These are source-integrity and functional-reproduction deliveries, not scientific admission or policy-superiority results.

Next sequential implementation: #307. #307–#313 remain open. Keep one bounded implementation in progress, reuse existing contracts and preserve negative/inconclusive evidence. All seven diversity regimes and the original parent acceptance gates below remain unchanged. The umbrella remains open until those gates are satisfied; closing children alone is insufficient.

Parent acceptance traceability

Numbers below refer to the 25 acceptance checkboxes in their original order. A child may close only with its actual evidence; parent boxes require their complete scope, not merely an opened child.

Acceptance criteria	Owning subtasks
1: native inventory and source semantics	#309
2: RTPTorrent/LRTS admission or reproducible rejection	#302, #303, #309
3: controlled Java and non-Java corpora	#304, #305, #309
4: independent HCS grouping	#307, #310
5: synthetic interaction coverage	#308
6: component terms and pins	#301–#309, each relevant source
7–8: secure fetch and deterministic conversion	#301–#308, each relevant source
9–10: temporal protection and independent metrics	#309, #310
11: comparable candidate matrix	#311
12: raw/project/macro reports	#313
13–14: execution models and production reality	#309, #312
15: cost/latency/work distinctions	#310, #311
16: adverse confirmatory regimes	#308, #313
17: prospective feasibility	#312
18–19: dependence-aware statistical protocol	#310, #311
20: resources and incomplete/failed runs	#311, #313
21: frozen installed reproduction	#313
22–23: evidence layers and negative outcomes	#309, #311, #313
24: installed/add-a-source tutorials	#309, #313 and each source child
25: quality and review	Every child; final audit in #313

The seven required diversity regimes, source-admission and protected-confirmation requirements remain unchanged. Closing all bookkeeping items alone does not justify closing this umbrella if a mandatory evidence gate is still unmet.

Objective

Build a versioned, diverse, reproducible benchmark program that can determine where adaptive test-case prioritization and selection policies generalize, fail, drift, or become operationally uneconomic.

Milestone: M5 — Generalizable TCPCI benchmark.

Completion means that an installed COLEMAN package can validate benchmark manifests, materialize permitted datasets without silently vendoring large third-party archives, execute predeclared baselines under chronological protocols, and produce machine-readable and human-readable cross-corpus reports. Adding many rows from similar projects is not sufficient: the suite must cover materially different data-generating regimes.

Why this is required

The examples currently demonstrate the native schema with Alibaba Druid, Square Retrofit, Dune variants, and synthetic data. The public dataset section points to the Harvard Dataverse and extraction tools, while issue #255 covers mathematical/performance correctness and issue #257 covers one operational pilot. Those are necessary but do not establish external validity across project scale, languages, suite duration, failure prevalence, test volatility, flakiness, feedback availability, highly configurable systems, and distribution shift.

A benchmark result must never be described as universal or state of the art merely because it aggregates many builds. Reports must expose corpus composition and results per project/regime so a large easy dataset cannot hide failure on smaller or harder datasets.

Existing integration points

Inspect these interfaces before editing. They are starting points, not assertions that they are defective:

Reuse the native runner, run identity, result sink, checkpoint and protected-evaluation contracts. Do not introduce a second experiment framework.

Benchmark corpus layers

Implement a registry that can represent the following layers. Each source is admitted only after verifying current availability, license/terms, checksums, field semantics and reproducibility. A source that cannot be redistributed may still have a download/conversion recipe, but must not be committed or mirrored.

A. Native historical CI corpora

Register the existing public/authorized COLEMAN datasets and characterize every project independently. Record build count, time span, test count distribution, duration distribution, failure prevalence, verdict semantics, missingness, test churn, available context, variants, and whether outcomes represent failing tests or mapped unique faults.

These datasets preserve continuity with prior results but are not the sole evidence.

B. RTPTorrent historical regression-test corpus

Evaluate a version-pinned adapter for RTPTorrent, which covers 20 open-source Java systems and includes fine-grained builds, test results, version-control information and published baselines. Preserve its build/job identity and chronological order. Do not collapse job-level outcomes into build-level observations without a documented rule and a collision test.

C. Long-running test suites

Evaluate a version-pinned adapter for the LRTS artifact. LRTS is intentionally different from small/medium historical suites: it contains large, multi-module projects and long-running executions. Preserve project, pull request, build, stage and test-class identity. Treat its filtered dataset variants as distinct outcome definitions rather than interchangeable samples.

Where licensing and compatibility permit, reproduce recent simple and data-driven baselines such as those documented by the DANTE artifact. External baseline code must remain isolated and license-compatible; otherwise reproduce only the published algorithm from a clean specification and disclose that distinction.

D. Controlled real-fault corpora

Investigate adapters for Defects4J and BugsInPy. These provide reproducible real bugs and triggering tests across Java and Python, but they are not longitudinal CI histories. Use them to test fault diversity, exact triggering behavior, feature ablations and controlled cold-start conditions.

Reports must keep this layer separate from historical CI evidence. Constructed build sequences, repeated executions, or injected timestamps must be labeled synthetic and must never be presented as observed CI evolution.

E. Controlled seeded-fault, cross-language objects

Investigate a bounded subset of the Software-artifact Infrastructure Repository to cover languages and controlled fault matrices not present elsewhere. Seeded faults and older systems have different external-validity limitations; report them separately and include the source's stated validity limitations.

F. Highly configurable systems

Retain Dune-style variant-aware evaluation and add at least one materially different configuration topology when a redistributable source exists. Exercise WTS/VTS semantics, appearance/disappearance of variants, sparse variant matrices and cross-variant transfer. Do not treat duplicated test executions across variants as independent evidence.

G. Deterministic synthetic stress laboratory

Add a seeded scenario generator to isolate causal behavior that real data cannot cover densely. It must generate exact expected outcomes and support a fractional-factorial or covering-array experiment plan over:

suite size: tiny exact-oracle cases through at least 50,000 tests;
failure prevalence: zero, rare, moderate and bursty;
duration: constant, log-normal/heavy-tailed, failure-correlated and changing;
test churn: additions, removals, renames and identity collisions;
drift: sudden, gradual, recurring and context-dependent;
flakiness: independent, history-dependent and environment-correlated;
feedback: full, budget-censored, delayed, missing and corrected;
budgets: severe selection through full-suite execution;
context quality: informative, noisy, missing and adversarial;
variants: stable, sparse and rapidly changing;
parallelism: serial and bounded-worker scheduling.

Avoid a combinatorial explosion: generate a declared minimal interaction-covering design plus named adversarial cases. Synthetic results test mechanisms and robustness; they do not replace real-project evidence.

Versioned benchmark manifest

Define a schema before adapters. Illustrative shape:

schema_version: 1
benchmark_id: rtptorrent-v1.1
source:
  uri: https://doi.org/10.5281/zenodo.4046180
  version: "1.1"
  sha256: "<archive digest>"
  license_status: verified
  redistribution: recipe-only
adapter:
  name: rtptorrent
  version: 1
identity:
  observation: [project, build_id, job_id, test_id]
  group: project
time:
  event_field: started_at
  outcome_available_field: completed_at
outcome:
  unit: failing_test
  verdict_mapping: {passed: 0, failed: 1}
  unique_fault_mapping: unavailable
splits:
  protocol: expanding_window
  warmup_builds: 50
  validation_builds: 50
  test_builds: 100
budgets: [0.10, 0.25, 0.50, 1.00]
seeds: [11, 29, 47, 83, 131]
expected_profile:
  minimum_projects: 1
  permits_zero_failure_windows: true

Canonical manifests must include source/version/digest, license review status, redistribution rule, citation, adapter version, identities, timestamps and outcome availability, feature availability, outcome semantics, exclusions with counts/reasons, temporal protocol, budgets, seeds, metrics and resource limits. Hash canonical content into run identity.

Download, conversion and validation contract

Provide commands equivalent to:

coleman benchmark list
coleman benchmark fetch rtptorrent-v1.1
coleman benchmark validate rtptorrent-v1.1
coleman benchmark run benchmark-matrix.yaml
coleman benchmark report runs/<matrix-id>

Final names may change, but installed-package behavior must be documented and tested.

Default fetch performs no authenticated scraping and never requires secrets embedded in configuration.
Downloads use a cache outside the repository, verify digest before extraction, use bounded disk/network limits and reject unsafe archive paths.
Conversion is deterministic, restartable and writes provenance plus row-level exclusion summaries.
Offline validation works from an already materialized cache.
Large datasets and generated result archives are not committed to Git.
Tests use tiny licensed fixtures or schema-only fixtures, not network calls.
A source change, checksum mismatch or missing license status fails closed with an actionable message.

Production-reality gate

A corpus is not considered operationally representative merely because it came from an open-source repository. Record which parts of an actual CI decision are observed, reconstructed, missing or idealized.

The aggregate matrix must include at least two materially different CI providers or execution models (for example, Travis-style jobs and Jenkins-style stages) and must assess the feasibility of a GitHub Actions or GitLab CI prospective adapter without requiring private data. Cover:

pull-request, merge/mainline, release and scheduled-build triggers;
unit, integration and end-to-end test levels;
setup/shared-fixture cost and test-order dependencies;
serial, parallel, matrix, shard and rerun execution;
runner type, queue delay, concurrency and cancellation when available;
retries, quarantines, flaky classifications, timeouts and infrastructure failures;
generated/vendor files, monorepos, renamed paths and incomplete change-to-test traceability;
branch protection, mandatory suites and human overrides;
partial results produced when jobs stop early or selection omits tests;
delayed/corrected outcomes and retention gaps.

For each metric, state whether it measures ordering quality, selection quality, total runner work, developer-visible feedback latency or a modeled proxy. Do not derive wall-clock savings from serial test durations when the source executed tests in parallel. Do not count setup time as independently removable per test when it is shared by a suite or shard.

The frozen confirmatory profile must contain at least one long zero-failure interval, one flaky/retry regime, one high-churn interval, one parallel or sharded suite, one partial-feedback regime and one abrupt drift interval. When no real corpus provides a regime, use a declared synthetic stress case and preserve the uncovered real-evidence cell.

Diversity matrix and admission gates

Generate a machine-readable data card for each dataset and an aggregate coverage matrix containing at least:

observed versus constructed history;
real versus seeded/synthetic faults;
language and build system;
project count and time span;
build/test-suite scale quantiles;
suite-duration quantiles;
failure and zero-failure-window rates;
test churn and variant volatility;
flaky/unknown/timeout outcome handling;
full versus censored feedback;
contextual/change features;
serial/parallel execution evidence;
license and redistribution status.

Predeclare minimum diversity coverage. At least one admitted corpus must occupy each of these regimes: small volatile historical CI, medium historical CI, long-running/large-suite CI, highly configurable systems, controlled real faults in Java, controlled real faults outside Java, and deterministic adversarial simulation. If a source cannot be admitted, report the uncovered cell rather than silently substituting a similar Java corpus.

Baselines and candidate protocol

Every compatible corpus must include:

seeded random;
original/base order when meaningful;
shortest-test-first;
recent-failure and failure-frequency heuristics;
a simple recency-duration hybrid;
representative native stochastic and non-stationary bandits;
representative native contextual policy when compatible features exist;
the portfolio/meta-policy when implemented and eligible;
a published external baseline when reproducible and license-compatible;
an oracle only where all outcomes are legitimately known, clearly labeled unattainable.

Run candidates under identical folds, budgets, seeds, resource limits and outcome definitions. A candidate unable to consume a corpus must be marked incompatible with a reason, not assigned a worst score or silently omitted.

Separate policy development, model selection and protected confirmation. No test fold may tune hyperparameters, feature normalization, reward weights, stopping rules or candidate inclusion.

Measures and statistical protocol

Persist raw per-build observations before aggregation. At minimum report:

APFD/NAPFD only where their assumptions and outcome unit are satisfied;
APFDc or an independently verified cost-aware measure;
failing-test recall under budget;
unique-fault recall only when a genuine fault mapping exists;
time/rank to first failure with no-detection represented as censored or null, never zero;
selected and full-suite runner-seconds;
elapsed wall-clock only when scheduling/parallelism is modeled;
selection overhead, training time, inference latency and peak memory;
constraint violations, abstentions and fallback frequency;
stability across seeds, temporal folds and drift regimes;
per-project and macro-aggregated results.

For repeated candidates evaluated on the same projects/folds, use paired/repeated-measures methods such as Friedman/Iman-Davenport or a justified aligned/block alternative, followed by multiplicity-controlled post-hoc comparisons. Kruskal-Wallis is for independent groups and must not be used as a drop-in replacement for paired project results. Include paired effect sizes and confidence intervals, and use block/hierarchical bootstrap or another documented method when observations are serially dependent or clustered.

Predeclare a practical-equivalence region. Report wins, losses, ties and inconclusive comparisons. A statistically detectable but operationally trivial difference is not a promotion.

Independent correctness fixtures

Commit tiny fixtures whose answers are calculated without calling production policies or metric implementations:

a serial suite with known first-failure times and APFD/APFDc;
a budget boundary where one failing test is excluded but remains in recall denominator;
a parallel schedule where runner-seconds and elapsed time differ;
a late-arriving verdict that is unavailable during its build decision;
a rename/churn case that would duplicate test identity under naive matching;
a zero-failure window;
a flaky retry whose logical outcome rule is explicit;
a variant case where duplicated executions are not independent faults;
an archive with a traversal path that fetch must reject;
a checksum/source revision mismatch.

Mutation/fake implementations must demonstrate that the gates catch future leakage, metric denominator manipulation, unsafe extraction, invalid oracle use and input-order dependence.

Deliverables

Versioned benchmark registry and manifest schema.
Dataset data cards and aggregate diversity matrix.
Deterministic secure fetch/cache/validate adapters for admitted sources.
Seeded synthetic scenario generator and declared covering design.
Independent metric and temporal-leakage fixtures.
Reproducible baseline configuration matrix.
Machine-readable per-build/per-project/cross-corpus results.
Human report with limitations, failures and uncovered diversity cells.
Installed-package tutorial for adding a dataset without modifying the runner.
A frozen confirmatory benchmark profile suitable for later research comparison.

Acceptance criteria

Dependencies and sequence

Depends on:

Add versioned domain result records with explicit units and provenance #251 for versioned result/provenance records;
Introduce protected temporal evaluation and auditable statistical comparisons #252 for protected temporal evaluation and statistical comparison;
Create independent mathematical and performance benchmarks for decision extensions #255 for independent mathematical/performance benchmark foundations.

Complements #257: the operational pilot establishes value in a bounded deployment-like setting, while this issue establishes breadth and generalization. Neither substitutes for the other.

Recommended implementation order:

Benchmark schema, registry, data card and two tiny correctness fixtures.
Inventory current datasets and generate the first diversity matrix.
Secure fetch/cache/validate subsystem.
RTPTorrent adapter and baseline slice.
LRTS adapter and resource-bounded large-suite slice.
Controlled real-fault adapters.
HCS/variant expansion and synthetic stress generator.
Complete baseline matrix and statistical protocol.
Freeze and run the confirmatory profile.
Publish the report, limitations and exact reproduction commands.

Each phase must leave a runnable vertical slice. Do not postpone provenance, leakage tests or metric correctness until after expensive experiments.

Non-goals and boundaries

This issue does not claim that any policy is state of the art, upload third-party datasets without permission, require paid services, collect private repository data, equate seeded faults with production failures, infer unique faults from failing tests without mappings, or turn performance timing into correctness assertions. It does not implement a new learning algorithm; it creates the evidence infrastructure required to evaluate one honestly.

Dataset availability and repository contents may change. Pin and verify artifacts rather than assuming a URL remains authoritative. Any newly discovered license, privacy, integrity or reproducibility problem must fail closed and appear in the report.

Phase 1 is implemented and reviewed in #276, merged into the release/4.2 integration branch at ea1b59dd475d5302d487f5308cc6502841a04e8a. It is not yet part of a published release.

Delivered: schema-v1 source manifests, typed registry/data cards, two independent known-answer calculation fixtures checked against real native metrics/replay, metadata inventory of ten existing Parquet artifacts, deterministic immutable evidence, installed CLI/library/native examples, strict compatibility and source redistribution gates.

The two owned sources share one dependence group. All ten existing example registrations remain blocked pending source-specific admission evidence, and all seven minimum diversity regimes remain uncovered. This is functional correctness infrastructure, not completed cross-corpus evidence.

Validation: 1,363 tests passed; lint/format/two type checkers/Pylint/docstrings/structural cost/strict docs/build and installed SDK-free native smoke passed. Local-only OpenCodeReview delegation/rules plus manual exclusions covered all 19 changed files. Six valid review findings were reproduced and fixed; all six threads resolved, final current-head Codex review clean, Sonar issues zero, all 28 checks completed successfully/skipped.

This issue remains open. Next corpus slices still include secure fetch/cache/conversion, verified real-source admission/profiles, expanded temporal/identity/flaky fixtures and stress cases, baselines/statistics and frozen confirmatory evidence. The next sequential implementation is #258's reusable point-in-time observation foundation, which also supports this issue's late-feedback/correction requirements. No overlapping implementation is being started.

Phase 1 is now published in COLEMAN 4.2, alongside #258's point-in-time observation and decision-signal foundation. The actual released wheel, tagged source, signatures, SBOM and installed native workflows were verified after publication.

This issue remains open: the two owned correctness fixtures do not establish corpus diversity, all ten existing example registrations remain blocked pending source-specific admission evidence, and all seven required diversity regimes remain uncovered.

The next sequential implementation is #259, under review in #280 for the 4.3 integration branch, followed by #260. Remaining corpus phases still require secure materialization/conversion, verified real-source admission/profiles, expanded fixtures/stress cases, baseline/statistical protocols and frozen cross-corpus evidence. Published source/version and evidence boundaries are also tracked in the public roadmap.

The public 4.3 resource/shadow increments (#259/#260) are now published and verified. The next sequential corpus slice, phase 2, is under review in #284 against the 4.4 integration branch.

It adds bounded offline descriptions of all ten existing native Parquet artifacts, explicit units/missingness, identity collision counts, logical-build versus variant-execution groups, deterministic immutable profile/inventory artifacts, installed commands/examples and a real owned native CI fixture reopened through its learning checkpoint. The source-specific admission metadata is preserved: all ten existing registrations remain blocked for evaluation and all seven required diversity regimes remain uncovered.

Local validation before push: 1,710 tests passed, combined statement/branch coverage 95.14%, all repository quality/strict documentation/package checks passed, and local-only OpenCodeReview delegation/rules plus manual exclusions covered all 11 changed files. Current-head PR review and CI are pending.

#261 remains open. After this PR is reviewed and integrated, the next sequential implementation is phase 3's secure materialization foundation; external source admission, adapters, stress cases, baseline/statistical protocols, confirmatory execution and the final evidence report remain outstanding. No cross-corpus effectiveness or independent-observation claim follows from this descriptive inventory.

Phases 2 and 3 are now published in COLEMAN 4.4.0, through #284, #285 and the reviewed release aggregate #286. The released wheel/source, signatures, SHA-256 digests, SBOM, tag/lock consistency and installed native workflows outside the checkout were verified. Final release validation: 1,947 tests passed, 95.21% combined statement/branch coverage; repository quality checks and local-only delegated review passed.

This issue is not complete. The released functionality covers the initial manifest/registry, existing-source descriptive profiles/inventory, and secure materialization/cache/offline integrity foundation. All ten existing native registrations retain their admission blockers; all seven required diversity regimes remain uncovered. Source integrity is not scientific admission or generalization evidence.

The next sequential slice starts with a primary-source research/admission dossier for RTPTorrent, followed by its version-pinned adapter and bounded baseline slice. Source review also needs to distinguish the LRTS paper snapshot (21,255 builds / 57,437 suite runs) from the current repository README (32,199 / 108,366). These counts refer to different documented snapshots and must not be mixed.

Research will record exact versions/digests, source and component terms, field meanings/units, build/job/stage identity, outcome availability, exclusions and dependence/CI-model limitations before admission. RTPTorrent 1.1 archive size and published checksum need an explicit acquisition/verification recipe; SIR explicitly warns that maintenance has ended. Neither provider metadata nor a working download is enough to clear the admission gates.

Still outstanding: admitted RTPTorrent/LRTS slices or documented reproducible blockers; Java and non-Java real-fault adapters; variant expansion and the declared synthetic interaction-covering design; complete comparable baselines/statistics; production-reality and modern CI feasibility studies; frozen confirmatory execution; and raw/per-project/cross-corpus reports with limitations and exact reproduction commands. These remain sequential compatible 4.x increments. No external-corpus effectiveness claim or issue closure follows from 4.4.

Phase 4 integrated for the next compatible release

#290 is merged into release/4.5 at d7ce41a. The final reviewed head had all checks passing, completed Codex review without findings, no unresolved threads, zero open/confirmed Sonar issues and zero PR code-scanning alerts. Full local validation: 2,035 tests passed, 95.31% combined coverage; OpenCodeReview 1.12.13 ran only in local delegation, with 11/11 files reviewed, including all exclusions.

The source assessment and complete installed-package workflow now provide versioned raw-class conversion, explicit seconds/missingness, job/class/position identities, duplicate accounting, immutable provenance artifacts and an owned native Random/RNFail replay with checkpoint continuation. Actual pinned HikariCP conversion retains 46,995 executions across 1,662 jobs, including 23,302 repeated-class executions.

RTPTorrent remains blocked for protected temporal evaluation. Raw files lack verified event/completion clocks and complete build/job mapping; repeated class labels lack module/configuration disambiguation. The accessible metadata dump differs from the snapshot named in the paper. Full archive integrity is unverified, and ZIP64 / entry-count requirements exceed the existing v1 materializer. No timestamps were invented, no filtered baseline was substituted for complete history, and no third-party data rows were committed. Registry diversity gaps remain visible.

While reviewing this source, an existing exact-number validation defect was reproduced and corrected separately in #288, published and independently verified as 4.4.1; #289 pins current install commands to its actual wheel digest. The RTPTorrent adapter is in the integration branch, not that published wheel.

Next is the LRTS source-assessment slice, in dependency order. Primary-source research has pinned the paper-sized public metadata and a complete bounded Karaf raw-report archive; raw/processed snapshots and missing completion semantics remain distinct. Subsequent work still includes controlled real-fault corpora in Java and outside Java, configurable/synthetic stress, comparable baselines/statistics, protected frozen confirmation, cross-corpus reporting and modern-CI feasibility. This issue is not complete.

feat: preserve pinned LRTS metadata and raw Jenkins evidence (#291)

feat: release reproducible CI source assessments for 4.5 (#292)

docs: pin installation to verified 4.5.0 wheel (#293)

RTPTorrent and LRTS source assessments published in 4.5

The reviewed integrations #290 and #291 are now published in COLEMAN 4.5.0 through #292. #293 updates current installation commands to the actual verified wheel SHA-256 and the tagged example. Main is synchronized at 5ebea54; the merged trees exactly matched the reviewed heads.

Frozen source validation: 2,107 tests passed, 95.40% combined statement/branch coverage, all quality/strict-doc/package checks passed, and local-only OpenCodeReview plus manual exclusions covered 22/22 release files. Both confirmed Jenkins review findings were reproduced and fixed with regressions. A build/stage-scope finding was checked against pinned extraction code, the complete metadata and a distinct-stage reproduction; a fresh review then completed without findings. All PR checks passed, threads were resolved, actual Sonar issues and PR code-scanning alerts were zero.

All six published asset digests, complete wheel/source bytes against the tag, metadata/lock/SBOM consistency, reviewed en-US body and offline Sigstore identities were verified against the actual release trigger. The automatic bump changed only the project version in pyproject.toml and uv.lock. The actual base wheel ran outside checkout: owned native/cache/checkpoint workflows, both source readers, SDK documentation, the tagged point-in-time example and existing native profiles/inventory. Repeated runs retain their artifact and committed-result identities.

The LRTS guide documents complete original/Karaf archive verification, specific data terms, pinned metadata correspondence and one 228-suite / 931-method real slice with repeated labels retained. Original report totals are build-wide fields repeated across processed stages; processed stage statistics retain their separate definitions. The RTPTorrent guide preserves its class/job evidence and the unresolved full-archive/temporal-mapping limits. No third-party rows were committed.

This issue remains incomplete. Both historical sources remain blocked for protected temporal evaluation, and all seven required diversity regimes remain uncovered. Scheduled time is not outcome availability; reported cases do not establish the complete expected suite. Existing ZIP64/128-entry archive limits are unchanged. Functional owned replays establish no independent-project, effectiveness or large-suite performance evidence.

Next is controlled real-fault work. Pinned Defects4J documentation requires Java 11 and a specific timezone, with subject/dependency terms reviewed separately from its framework license. The pinned BugsInPy framework has no verified license; its existing upstream license issue remains open. Fresh paired outcomes, complete runtime/input identities and explicit controlled/reconstructed labels are required before admission. Variant/synthetic expansion, comparable baselines/statistics, modern-CI feasibility, frozen confirmation and cross-corpus reporting remain subsequent sequential work. No synthetic chronology, seed replication or metadata-only expectation will be promoted into observed CI evidence.

feat: retain controlled fault evidence and inconclusive comparisons (#…

Controlled observation foundation reviewed for 4.6

#294 is merged into release/4.6 at 6ad8f71809bf1ca5cd6f877d21b668b8732a7322. The merged tree exactly matches the reviewed head. This increment is not yet a published release; current main/latest release remain 4.5.0. The feature branch was removed after verification; the integration branch remains for related reviewed work.

Delivered: strict schema-v1 controlled contexts, common-test buggy/fixed observations, explicit monotonic integer-nanosecond units and missingness, stable context-scoped execution identity, source/revision/fixture/runtime provenance, negative/inconclusive paired comparisons, conserved duplication accounting and deterministic immutable artifacts. Existing source caches, materialization, runner, metrics, manifests and learning checkpoints are reused. The core executes no foreign source, process, container or transport. Collection clocks never become historical CI chronology; admission stays not_evaluated and generalization insufficient_evidence.

Primary-source research included the original Defects4J/BugsInPy papers, pinned documentation/source metadata and an independent BugsInPy reproduction study. Two direct official-upstream selected-test pairs were actually executed: Apache Commons CLI and Black each produced one assertion failure on the earlier revision and one pass on the corrected revision, with zero errors/skips. These are separate direct source-pair experiments, not complete Defects4J reconstruction or BugsInPy framework admission. Common-test transplantation, Black's injected executor failure/derived version metadata and emulated Python runtime are explicit. Foreign sources/dependencies/raw observations remain outside Git.

Final validation: 2,164 tests passed, 13 existing warnings, 95.47% combined statement/branch coverage; both new production modules have 100% combined coverage. All lint/format/Pyright/ty/Pylint-E/docstring/structural/strict-doc/build checks passed. Installed-package tests outside checkout exercised import boundaries, real native CI Random/RNFail, independently checked NAPFD/APFDc, checkpoint continuation and deterministic repeated artifacts. The rebuilt wheel also converted/re-published the actual Java observations with both original diagnostic logs pinned in the report package.

Local-only OpenCodeReview 1.12.13 delegation/rules plus manual review covered 12/12 files, including all exclusions and golden values. All 17 detailed Sonar test findings were fixed. A valid Codex P2 on unbound diagnostic hashes was reproduced with a failing regression and fixed on conversion and dataset reads, with four additional regression cases. Final-head review completed without findings; all checks passed, threads were resolved, actual Sonar issues and PR code-scanning alerts were zero. Current open dependency/code/secret security alerts are all zero, with no open bump PRs.

#261 remains incomplete. All seven required diversity regimes remain uncovered. The next sequential work is distributable pinned controlled-execution recipes and source/component-specific admission with a broader fresh test/fault matrix. BugsInPy framework terms remain unresolved; upstream subject permission does not clear the framework or every dependency. Source-specific admission stays distinct from a successful selected assertion.

Further outstanding work includes admitted native/controlled diversity, HCS/variant expansion, declared synthetic interaction-covering stress, comparable baseline/statistical profiles, modern-CI feasibility, frozen confirmatory execution and raw/per-project/cross-corpus reports. These remain compatible 4.x increments. No historical chronology, independent-project count, performance win or issue completion is inferred from this foundation.

feat(corpus): reproduce pinned JUnit3 faults with recoverable evidence (

Second reviewed controlled-source slice merged into the 4.6 integration branch: #295, following #294. This is not yet a published release or completion of #261.

The pinned direct Apache Commons CLI-13 recipe compiled both source revisions against the common fixed-tree fixture and collected 190 observations across 95 tests per role. It preserved 92 both-pass pairs, two both-fail pairs and one failure-to-pass pair. Source/fixture/dependency/runtime/producer provenance, explicit counter units, bounded raw evidence, interrupted cleanup and same-trial replay are versioned and verified. Negative/inconclusive results remain evidence.

Final source tree: bd037b0c44ea48c57d069a44828447dcd5840a31. Local verification: 2,243 tests, 95.26% combined coverage, actual installed-wheel Java reproduction/recovery, all quality/package gates and local-only OpenCodeReview/manual 32/32-file review. Confirmed review findings were reproduced, fixed with regressions and resolved; final-head review completed clean, all GitHub checks passed and actual Sonar/PR code-scanning findings were zero before merge.

Next is the reviewed 4.6 release aggregation, followed by one source/admission slice at a time. Remaining #261 work includes component-reviewed cross-language recipes, broader independent faults/projects, historical availability evidence, the seven diversity regimes, comparable predeclared baselines and the final cross-corpus report. These records retain not_evaluated/insufficient_evidence. A reproducible reader or a larger same-fault test matrix does not establish generalization.

feat(corpus): release reproducible controlled CI evidence for 4.6 (#296)

Delivery update: #301–#304 are complete (4 of the 13 native subtasks).

v4.10.0 publishes #304 through #320: a bounded CLI-13/CLI-51/CODEC-65 matrix, typed plans/indices, raw/source/context validation and interruption/offline recovery. The initial qualification retained 714 role observations / 357 same-test pairs across two Java codebases, including both-fail and inconclusive outcomes. Three curated fix labels and their tests do not establish independent faults/projects or policy efficacy.

The actual signed release wheel was installed and exercised outside checkout: all three Java sources, offline CLI validation/replay, existing real LRTS stage/Karaf and complete RTPTorrent analytical/native Parquet flows, native runs and checkpoint continuation. All six assets, source/wheel/tag correspondence, both signatures, SBOM, lock and reviewed en-US release body were verified. The implementation passed 2,680 local tests (one skip, 95.27% coverage) and 2,678 Linux tests (three skips). The release audit queried all 238 locked third-party packages with zero known vulnerabilities.

#321 updates the README/catalog and published matrix guide to the verified wheel/SHA, with direct installation, runtime preparation and complete CLI examples. The deployed site was verified; the docs-only follow-up correctly produced no additional bump.

#261 remains open. Existing admission/diversity/comparison gates are unchanged. Next is #305 (controlled non-Java source); #306–#308 complete the other source slices before #309–#313 integrate admission, metrics, baselines, prospective feasibility and the frozen report. Source conversion and a release number do not replace scientific evidence.

Public subtask progress: #301–#305 are implemented and integrated; 5 of 13 native sub-issues are closed. #306–#313 remain open, and the parent acceptance gates are not checked off by source collection alone.

#305 shipped through #322 in 4.11.0. The actual published wheel/source, all six asset digests, SBOM and lock were verified; both Sigstore signatures match the actual main push/release workflow. Installed outside checkout, the wheel reproduced the two frozen PySnooper contexts, validated raw evidence, replayed offline without Docker on PATH, and recovered a genuinely interrupted process without a false completion marker. Existing real Java, LRTS and RTPTorrent/native checkpoint workflows passed against the published wheel as well.

The bounded Python slice contains one curated upstream fix / one selected unchanged test, not two independent faults: ASCII UnicodeEncodeError→pass remains inconclusive, and UTF-8 is both_pass. Component attribution limitations, historical runtime conditions and constructed provenance remain explicit. This does not establish historical CI, cross-corpus diversity, policy efficacy or source admission. All seven required diversity regimes remain uncovered pending the evidence integration gates.

The documentation follow-up refreshes README/catalog and active commands to the actual 4.11 wheel SHA-256, retaining these limits. Next sequential subtask: #306, beginning with current public-source availability and component/data terms before downloading or promoting a seeded cross-language benchmark. No overlapping implementation is being started.

Public execution update: #301–#306 are delivered (6 of 13 native subtasks). #307–#313 remain open. The umbrella was closed while these acceptance gates were still pending; it has been reopened to keep tracking consistent with the actual work. No original acceptance checkbox is being marked complete by this update.

#306 shipped through #324 in 4.12.0. The installed published wheel reproduced a bounded Juliet C unsigned-32 addition oracle: six correlated inputs from one synthetic template, 12 role observations, five both-pass pairs and one expected failure-to-pass pair. This is neither six independent faults nor real CI history, and it establishes no policy advantage. The initial no-executable-scratch prototype and inconclusive evidence remain preserved.

The actual release was qualified outside checkout: C/Python interruption and offline replay, the three-fix Java matrix, real LRTS/RTPTorrent Parquet/native runs and checkpoint continuation, JUnit v2 relocated-output recovery, and original published 4.11 v1 marker compatibility. All six asset digests, source/wheel/tag/lock correspondence, SBOM, both actual main-push Sigstore signatures and reviewed en-US release notes were verified. The exact implementation passed 3,133 local tests (one skip; 95.123% combined line/branch coverage). Local-only OpenCodeReview plus manual review covered all 50 feature files; exact-head review, all CI, actual Sonar gate and PR security checks passed before merge.

#325 is merged and the published C guide, README and current installation links use the verified published 4.12 wheel/SHA. All three C/Python/Java API examples and both tagged owned ETL examples ran successfully. All 15 documentation files were reviewed, the actual deployed 15 pages were checked, and the documentation-only automatic release correctly produced no bump. Exact reviewed/merged trees were proved equal before branch cleanup.

#261 remains open. All seven diversity regimes and admission/confirmation gates remain unmet. Next is #307, verifying public historical HCS data terms, explicit time units, original identities, sparse variants and independent accounting before freezing a runnable protocol. Source expansion, interaction-covering stress (#308), admission (#309), protected metrics (#310), comparable baselines (#311), prospective feasibility (#312) and frozen cross-corpus reporting (#313) remain sequential work.

Navigation Menu

Repository navigation

Build a diversified, versioned benchmark corpus for adaptive CI test prioritization #261

Execution plan and delivered slices

Parent acceptance traceability

Objective

Why this is required

Existing integration points

Benchmark corpus layers

A. Native historical CI corpora

B. RTPTorrent historical regression-test corpus

C. Long-running test suites

D. Controlled real-fault corpora

E. Controlled seeded-fault, cross-language objects

F. Highly configurable systems

G. Deterministic synthetic stress laboratory

Versioned benchmark manifest

Download, conversion and validation contract

Production-reality gate

Diversity matrix and admission gates

Baselines and candidate protocol

Measures and statistical protocol

Independent correctness fixtures

Deliverables

Acceptance criteria

Dependencies and sequence

Non-goals and boundaries

Phase 4 integrated for the next compatible release

RTPTorrent and LRTS source assessments published in 4.5

Controlled observation foundation reviewed for 4.6

10 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Build a diversified, versioned benchmark corpus for adaptive CI test prioritization #261

Description

Execution plan and delivered slices

Parent acceptance traceability

Objective

Why this is required

Existing integration points

Benchmark corpus layers

A. Native historical CI corpora

B. RTPTorrent historical regression-test corpus

C. Long-running test suites

D. Controlled real-fault corpora

E. Controlled seeded-fault, cross-language objects

F. Highly configurable systems

G. Deterministic synthetic stress laboratory

Versioned benchmark manifest

Download, conversion and validation contract

Production-reality gate

Diversity matrix and admission gates

Baselines and candidate protocol

Measures and statistical protocol

Independent correctness fixtures

Deliverables

Acceptance criteria

Dependencies and sequence

Non-goals and boundaries

Activity

jacksonpradolima commented on Oct 15, 2026

jacksonpradolima commented on Oct 15, 2026

jacksonpradolima commented on Oct 15, 2026

jacksonpradolima commented on Oct 15, 2026

jacksonpradolima commented on Oct 16, 2026

Phase 4 integrated for the next compatible release

jacksonpradolima commented on Oct 16, 2026

RTPTorrent and LRTS source assessments published in 4.5

jacksonpradolima commented on Oct 16, 2026

Controlled observation foundation reviewed for 4.6

jacksonpradolima commented on Oct 16, 2026

10 remaining items

jacksonpradolima commented on Oct 17, 2026

jacksonpradolima commented on Oct 17, 2026

jacksonpradolima commented on Oct 17, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions