Skip to content

Reorganize tests into clause-addressable conformance and behavior suites #90

Description

@kjanat

Problem

The test suite is organized primarily by historical feature files, not by the behavior being proved or the standard that mandates it. Standards evidence is therefore difficult to discover, audit, run, and maintain.

Today:

  • test/ has 37 top-level suites and about 32.9k lines of suite code.
  • The five largest suites contain 56% of that code:
    • verify.test.ts: 5,048 lines
    • crl.test.ts: 4,132 lines
    • ocsp.test.ts: 3,735 lines
    • parse.test.ts: 3,648 lines
    • internals.test.ts: 1,956 lines
  • RFC 5280 assertions are distributed across builder, parser, internal encoder, verification, revocation, and helper suites.
  • Important RFC 5280 behavior in crl.test.ts, policy.test.ts, and name-constraints.test.ts often has no RFC/section in the test title.
  • Some citations exist only in comments inside generically named tests.
  • test/helpers.ts is a 500-line catch-all spanning DER utilities, mutations, chain issuance, CMS builders, OpenSSL detection, and filesystem paths.
  • Domain-surface smoke suites overlap with full feature suites without a clear ownership rule.

Answering “do we assert RFC 5280 §4.2.1.6?” should require opening one canonical file and finding:

describe('RFC 5280', () => {
	describe('§4.2.1.6 Subject Alternative Name', subjectAlternativeNameConformance);
});

It should not require repository-wide archaeology.

Goal

Reorganize tests so their paths and names answer three questions directly:

  1. What kind of evidence is this: standards conformance, public behavior, integration/interoperability, or package contract?
  2. Which standard and clause owns normative behavior?
  3. Which exact normative statement or API contract does the case prove?

This is an ownership and discoverability migration, not a rewrite of working assertions.

Target structure

test/
├── conformance/
│   ├── rfc-5280.test.ts
│   ├── rfc-5280/
│   │   ├── 4.1-certificate-profile.ts
│   │   ├── 4.2-extensions.ts
│   │   ├── 6.1-path-validation.ts
│   │   └── 6.3-crl-validation.ts
│   ├── rfc-6960.test.ts
│   ├── rfc-9618.test.ts
│   ├── x690.test.ts
│   └── pkits/
│       ├── pkits.test.ts
│       ├── manifest.ts
│       └── ...
├── behavior/
│   ├── x509/
│   ├── verify/
│   ├── revocation/
│   ├── keys/
│   ├── pkcs/
│   ├── pem/
│   └── der/
├── integration/
│   ├── lifecycle/
│   ├── differential/
│   └── fuzz/
├── contracts/
│   ├── root-exports.test.ts
│   ├── domain-entrypoints.test.ts
│   └── public-types.ts
├── support/
│   ├── builders/
│   ├── der/
│   ├── mutations/
│   ├── oracles/
│   └── paths.ts
└── fixtures/

Exact subdirectories may change during migration. The ownership categories and canonical standards entrypoints should not.

Standards convention

Each supported standard gets one canonical test/conformance/<standard>.test.ts front door. It must contain the ordered clause map, even when implementations live in smaller imported modules:

import { subjectAlternativeNameConformance } from './rfc-5280/4.2-extensions.ts';

describe('RFC 5280', () => {
	describe('§4.2.1.6 Subject Alternative Name', subjectAlternativeNameConformance);
});

Rules:

  • Sort clause describe blocks numerically.
  • Include the section title, not only the number.
  • Name cases after the normative behavior: MUST, MUST NOT, SHOULD, accepted profile behavior, or deliberate hardening.
  • Put positive and negative evidence under the same owning clause.
  • State updates/overrides explicitly, e.g. RFC 6818, RFC 9549, or RFC 9618.
  • A standards-mandated behavior has one canonical owner. API and integration suites may exercise the workflow, but must not become competing conformance sources.
  • Clause-specific malformed input belongs with its clause. Generic DER parser hardening belongs under behavior/der/.
  • PKITS remains a separate conformance harness because its section numbers, fixtures, expected outcomes, and fixed validation time are externally defined.

Non-standard test ownership

  • behavior/: public or internal behavior not owned by one normative clause.
  • integration/: cross-module lifecycle, OpenSSL differential, fuzz, and interoperability evidence.
  • contracts/: package exports, domain entrypoints, compile-time surface, and barrel checks.
  • support/: reusable test code only; no registered tests.
  • fixtures/: static or generated test material; files named *-fixtures.test.ts must become either actual tests or fixture modules, not both.

Trackability

Add focused commands:

{
	"test:conformance": "bun test test/conformance",
	"test:rfc5280": "bun test test/conformance/rfc-5280.test.ts",
	"test:pkits": "bun test test/conformance/pkits/pkits.test.ts",
	"test:differential": "bun test test/integration/differential"
}

Add a lightweight convention test that verifies:

  • every canonical conformance entrypoint has a top-level standard describe;
  • clause blocks use §<number> <title>;
  • no duplicate standard/clause ownership exists across canonical entrypoints;
  • support and fixture modules do not register tests;
  • obsolete top-level catch-all suites do not reappear.

Do not introduce a separate hand-maintained coverage matrix as the primary source of truth. The canonical entrypoints and executable clause blocks are the index. Authored scope docs may link to them.

Migration plan

1. Freeze the baseline

  • Record current discovered suites and runtime case counts.
  • Preserve 249 PKITS runs, the two DSA it.failing cases, the default differential fuzz corpus, it.each expansions, and inline malformed-input corpora.
  • Record current focused commands and CI artifacts before moving paths.

2. Establish taxonomy and conventions

  • Create the target directories and convention test.
  • Add canonical entrypoints for RFC 5280 first, then other RFC/ITU standards.
  • Add focused package scripts without changing broad bun test discovery.

3. Split support code

  • Decompose test/helpers.ts by responsibility.
  • Keep DER mutation/building logic single-owned; do not copy helpers into clause files.
  • Move OpenSSL wrappers under test/support/oracles/ while preserving availability gates and normalized verdicts.
  • Update #test/* imports, package.json, and deno.import_map.json atomically.

4. Migrate RFC 5280 evidence

  • Move normative cases from certificate.test.ts, csr.test.ts, internals.test.ts, parse.test.ts, verify.test.ts, crl.test.ts, policy.test.ts, name-constraints.test.ts, and revocation suites.
  • Organize by clause, not implementation module.
  • Start with known scattered areas:
    • §4.1.2.4 issuer DN
    • §4.2.1.6 Subject Alternative Name
    • §4.2.1.9 Basic Constraints
    • §4.2.1.10 Name Constraints
    • §4.2.1.13 CRL Distribution Points
    • §6.1 path validation
    • §6.3 CRL processing
    • Appendix A.1 attribute bounds
  • Convert comment-only citations into clause-owned test names.

5. Migrate remaining standards

  • RFC 6960 / RFC 5019 OCSP.
  • RFC 6125 / RFC 9525 service identity.
  • RFC 9618 policy processing.
  • PKCS/RFC suites for keys, PKCS#7, PKCS#8, PKCS#12, and PEM.
  • ITU-T X.680/X.690 DER/ASN.1 constraints.
  • RFC 3454 / RFC 4518 string preparation.

6. Split behavior and integration monoliths

  • Eliminate catch-all ownership in verify.test.ts, crl.test.ts, ocsp.test.ts, parse.test.ts, and internals.test.ts.
  • Keep lifecycle assertions under feature behavior directories.
  • Keep OpenSSL differential and fuzz suites isolated from deterministic behavior suites.
  • Consolidate overlapping domain-surface smoke tests under contracts/.

7. Update CI and documentation

  • Preserve broad coverage discovery and JUnit output.
  • Add a visible conformance CI step or summary.
  • Keep the dedicated OpenSSL environment and randomized stress workflow.
  • Update hard-coded test links in docs/PKIX-SCOPE.md, README.md, CONTRIBUTING.md, and test/AGENTS.md.

Acceptance criteria

  • test/conformance/rfc-5280.test.ts is the canonical RFC 5280 front door.
  • Searching that file for describe('§4.2.1.6 finds the Subject Alternative Name conformance owner.
  • Every RFC 5280 behavior claimed in docs/PKIX-SCOPE.md maps to an executable clause block.
  • bun test test/conformance/rfc-5280.test.ts runs all RFC 5280 conformance assertions.
  • Other supported standards have equivalent canonical entrypoints.
  • No normative assertion is discoverable only through a comment in an unrelated suite.
  • Each assertion has one canonical owner; migration does not inflate coverage through duplication.
  • The 249 PKITS runs, fixed time, expected-valid/invalid outcomes, and DSA expected failures are preserved.
  • Differential OpenSSL gates and fuzz behavior are preserved.
  • test/helpers.ts is removed or reduced to a narrow compatibility barrel during migration.
  • Catch-all internals.test.ts is removed.
  • Broad test discovery, focused scripts, coverage, JUnit, typecheck, docs lint, and randomized stress all pass.
  • Test-path links and agent guidance are updated in the same change that moves their targets.

Non-goals

  • Rewriting stable tests solely for stylistic consistency.
  • Replacing real cryptographic material with mocks.
  • Changing implementation behavior while moving tests.
  • Claiming unimplemented RFC coverage merely because a clause heading exists.
  • Moving or regenerating the PKITS corpus beyond path updates required by this reorganization.

Migration safety

Land this in reviewable slices, but each slice must leave one unambiguous owner for moved behavior. Temporary compatibility barrels are acceptable for imports; duplicate test registration and parallel old/new conformance ownership are not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    MigrateddxDeveloper experience, documentation UX, discoverability and permalinksenhancementNew feature or requestreliabilityrfc-conformanceRFC adherence and conformance evidencetestsTest coverage and harnesses

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions