Opt-in HTTP fetching layer for OCSP/CRL evidence (AIA + CDP)

[kjanat]

micro509 validates caller-provided revocation evidence only — it never talks to the network. That boundary is deliberate and stays the default, but a separate opt-in module could close the gap for relying-party use:

• Fetch CRLs from certificate CRL distribution point URLs (http/https forms).
• Build OCSP requests and POST them to AIA-listed responders.
• Feed results into the existing checkChainRevocation() / verifyCertificateChain({ revocation }) inputs unchanged.

Scope questions to settle first: caching/maxAge policy, timeout/retry semantics, redirect handling, and whether it lives in this package (subpath export, still zero-dep via fetch) or a companion package.

Additive — does not block 1.0.

[coderabbitai]

Coding Plan

Summary

• Introduce a new opt-in micro509/fetch subpath (wired via tsdown entry, never on the root barrel) that stays zero-dep using global fetch — the no-network default remains untouched.
• Reuse existing primitives (createOcspRequest, OCSP responder discovery, pem, base64Encode, result helpers) and add the one missing getCertificateCrlUrls helper; the fetch layer only produces raw DER/PEM bytes and slots them into checkChainRevocation/verifyCertificateChain unchanged.
• Settle scope with injectable transport + bounded timeoutMs/AbortSignal/maxAttempts, opt-in caching honouring nextUpdate/maxAgeMs, and safe same-scheme redirects, all surfaced as typed soft-failure results, never thrown.
• Keep the boundary honest by updating PKIX-SCOPE.md/README/AGENTS.md, changelog, and versions, with no-mock tests using a real substitute transport.

There — every plank nailed, every rope taut, and not one drop of network in the default build. Chef's-kiss. Fix it clean or don't fix it at all, says I. 🏴‍☠️

Design Choices

Design Choice 1: Where should this cursed network deck live, and how far should its policy knobs reach?

Options Considered:

1. The ticket itself flags these as scope questions to settle first, addressed via sub-decisions 1a, 1b, 1c below.

Chosen Option: 1

Rationale: The ticket itself flags these as "scope questions to settle first", so here be me rulings, split across package placement, caching, and timeout/retry/redirect concerns.

Design Choice 2: Package placement

Options Considered:

1. New subpath export micro509/fetch inside this package, still zero-dep via global fetch.
2. Separate companion package (e.g. @micro509/fetch).

Chosen Option: 1

Rationale: A subpath keeps the release train, docs, and tests in one hull while remainin' opt-in (never re-exported from root src/index.ts), so consumers who want the pure no-network guarantee simply don't import it; a companion package adds publish/versioning overhead with no benefit given global fetch keeps us zero-dep anyway.

Design Choice 3: Caching / `maxAge`

Options Considered:

1. No cache — every call fetches.
2. Optional caller-supplied in-memory cache keyed by URL, honouring the resource's own nextUpdate (and OCSP Cache-Control: max-age / response nextUpdate) as the freshness boundary, plus an optional maxAgeMs override.

Chosen Option: 2

Rationale: Matches RFC 5019 (OCSP clients MUST cache) and RFC 5280 (reuse CRL until nextUpdate); made opt-in and injectable so the module stays pure and testable. Freshness stays modelled the codebase way (at + timestamps), with maxAgeMs as a new bounded override rather than inventing a whole cache subsystem.

Design Choice 4: Timeout / retry / redirect

Options Considered:

1. Rely purely on the platform fetch defaults.
2. Expose timeoutMs (backed by AbortController) + caller-passable AbortSignal, a single bounded retry knob (maxAttempts, default 1), and follow only same-scheme http→https safe redirects up to a small cap.

Chosen Option: 2

Rationale: RFCs leave transport policy to implementations, so we give short, bounded, explicit knobs (no infinite retries, no silent cross-scheme redirects), introducing the first AbortController contract in the repo deliberately and documented.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Lay the keel — new opt-in fetch domain, discovery helpers, and typed result shapes

This phase raises the new domain deck and its type scaffolding without performing any real I/O yet, so everything downstream has stable contracts to build on. It also fills the one discovery gap (CRL URLs) that the existing revocation module never exposed.

Task 1: Create the `src/fetch/` domain and wire the opt-in subpath export

Stand up the new domain barrel and register it as a build entry so micro509/fetch becomes importable — but keep it off the root barrel so the zero-network default is untouched.

• Create src/fetch/ with an implementation file (e.g. src/fetch/fetch.ts) and a re-export-only barrel src/fetch/index.ts (mirror the revocation/{index,revocation}.ts pattern).
• Add a fetch: 'src/fetch/index.ts' key to the entry map in tsdown.config.ts; let the build regenerate package.json/jsr.json exports — do not hand-edit those generated blocks.
• Do not re-export anything from src/index.ts; this deck sails only when explicitly boarded.

Task 2: Add a CRL distribution-point URL discovery helper

The module needs CRL URLs; the codebase only has an OCSP equivalent, so add the missing twin next to it.

• Add getCertificateCrlUrls(certificate) to src/revocation/revocation.ts, modelled precisely on the existing getCertificateOcspResponderUris (normalise PEM/DER/ParsedCertificate, fail-closed to [], dedupe by URI).
• Read from parsedCertificate.crlDistributionPoints[].distributionPoint.fullName, filtering GeneralName entries to type === 'uri' and keeping only http/https schemes.
• Export it from the revocation barrel so the fetch module can consume it via internal reuse rather than reimplementing CDP digging.

Task 3: Define the fetch module's typed result unions and options

Establish the expected-failure vocabulary and the injectable transport contract before writing fetch logic.

• In src/fetch/fetch.ts, define a flat FetchEvidenceErrorCode string-literal union (e.g. 'network_error' | 'timeout' | 'http_error' | 'invalid_content_type' | 'empty_response' | 'no_endpoints') plus an optional FetchFailureDetails (e.g. status?: number, url?: string, timeoutMs?: number).
• Build *Result types from src/result/ (ErrorResult/Micro509Error + failureResult), following ValidateOcspResponseResult as the template — transport hiccups are soft typed failures, never thrown (honour the "transport failures are not hard cert failures" anti-pattern).
• Define an injectable transport type (a caller-suppliable fetch-shaped function defaulting to global fetch), plus shared options: timeoutMs, signal, maxAttempts, optional cache, at, maxAgeMs.

🤖 Prompt for AI agents

Implement the new opt-in `micro509/fetch` domain scaffold with no real network
I/O yet, plus a missing CRL discovery helper.

- Create `src/fetch/` with `src/fetch/fetch.ts` (implementation) and a
re-export-only `src/fetch/index.ts` barrel, mirroring the
`revocation/{index,revocation}.ts` pattern.
- Register the new domain by adding `fetch: 'src/fetch/index.ts'` to the `entry`
map in `tsdown.config.ts`; let the build regenerate `package.json`/`jsr.json`
exports without hand-editing the generated blocks.
- Do not re-export anything from root `src/index.ts` — this module must remain
opt-in only.
- Add `getCertificateCrlUrls(certificate)` to `src/revocation/revocation.ts`,
modelled on `getCertificateOcspResponderUris`: normalise
PEM/DER/`ParsedCertificate` input, fail-closed to `[]`, dedupe by URI, read from
`parsedCertificate.crlDistributionPoints[].distributionPoint.fullName`, filter
`GeneralName` entries to `type === 'uri'`, and keep only `http`/`https` schemes.
Export it from the `revocation` barrel.
- In `src/fetch/fetch.ts`, define a flat `FetchEvidenceErrorCode` string-literal
union (e.g. `'network_error' | 'timeout' | 'http_error' | 'invalid_content_type'
| 'empty_response' | 'no_endpoints'`) and an optional `FetchFailureDetails`
(e.g. `status?: number`, `url?: string`, `timeoutMs?: number`).
- Build `*Result` types using `src/result/` primitives
(`ErrorResult`/`Micro509Error` + `failureResult`), following
`ValidateOcspResponseResult` as the template, ensuring transport hiccups are
soft typed failures rather than thrown exceptions.
- Define an injectable transport type (a caller-suppliable `fetch`-shaped
function defaulting to global `fetch`) plus shared options: `timeoutMs`,
`signal`, `maxAttempts`, optional `cache`, `at`, `maxAgeMs`.

Phase 2: Rig the sails — OCSP and CRL fetch primitives plus chain orchestration

This phase implements the actual retrieval and stitches it into a single orchestrator whose output slots straight into the existing revocation inputs, unchanged.

Task 1: Implement OCSP-over-HTTP fetching

Fetch OCSP responses per RFC 6960 / RFC 5019 and return raw bytes ready for checkChainRevocation.

• Reuse createOcspRequest (its .der) and resolveOcspResponderCandidates/getCertificateOcspResponderUris from revocation to pick responder URIs.
• Default to POST with Content-Type: application/ocsp-request, expecting application/ocsp-response; support the GET form for small requests by base64Encode-ing the DER then percent-encoding it (add the missing URL-safe step locally, since no base64url/percent helper exists — reuse base64Encode from src/internal/shared/base64.ts).
• Return the raw DER Uint8Array (optionally normalising via the pem barrel) — no need to pre-parse, since chain-level ocspResponses accepts string | Uint8Array.
• Apply timeoutMs/signal via AbortController, honour maxAttempts, and surface content-type/status problems as typed FetchEvidenceErrorCode results.

Task 2: Implement CRL-over-HTTP fetching

Fetch DER CRLs from distribution points per RFC 5280.

• Use the new getCertificateCrlUrls helper to enumerate candidate CDP URLs (http/https only).
• GET each URL, expecting Content-Type: application/pkix-crl (treat media type as a hint, not gospel — parse-or-fail rather than reject solely on header), returning raw DER bytes.
• Share the same timeout/abort/attempt/cache plumbing as OCSP; on cache hits, honour the CRL's nextUpdate (and optional maxAgeMs) as the freshness boundary before refetching.

Task 3: Add the chain-level evidence orchestrator

Provide one entry point that walks a chain, gathers all evidence, and returns it in the exact shape the existing APIs already accept.

• Add a fetchChainRevocationEvidence({ chain, ...options }) function in src/fetch/fetch.ts that iterates the leaf-first chain, fetches OCSP and/or CRL evidence per certificate (respecting a prefer knob consistent with RevocationPolicy), and aggregates into { crls, ocspResponses, ... }.
• Shape the return so it drops directly into checkChainRevocation(input) and into verifyCertificateChain({ revocation: { crls, ocspResponses } }) with no changes to those functions — the fetch layer produces evidence, the core still decides.
• Collect per-endpoint transport failures into a typed, non-fatal errors array (mirroring executionErrors), so a single dead responder never sinks the whole verdict.

🤖 Prompt for AI agents

Implement the actual OCSP and CRL over-HTTP fetch primitives and a chain-level
orchestrator that produces evidence for the existing revocation APIs, unchanged.

- For OCSP fetching: reuse `createOcspRequest` (its `.der`) and
`resolveOcspResponderCandidates`/`getCertificateOcspResponderUris` from
`revocation` to pick responder URIs. Default to POST with `Content-Type:
application/ocsp-request`, expecting `application/ocsp-response`; also support
the GET form for small requests by base64-encoding the DER via `base64Encode`
from `src/internal/shared/base64.ts` then percent-encoding it (add the missing
URL-safe step locally, since no base64url/percent helper exists). Return the raw
DER `Uint8Array` (optionally normalising via the `pem` barrel) — no pre-parsing
needed since chain-level `ocspResponses` accepts `string | Uint8Array`. Apply
`timeoutMs`/`signal` via `AbortController`, honour `maxAttempts`, and surface
content-type/status problems as typed `FetchEvidenceErrorCode` results.

- For CRL fetching: use `getCertificateCrlUrls` to enumerate candidate CDP URLs
(http/https only). GET each URL, expecting `Content-Type: application/pkix-crl`
but treating media type as a hint rather than gospel (parse-or-fail rather than
reject solely on header), returning raw DER bytes. Share the same
timeout/abort/attempt/cache plumbing as OCSP; on cache hits, honour the CRL's
`nextUpdate` (and optional `maxAgeMs`) as the freshness boundary before
refetching.

- For orchestration: add `fetchChainRevocationEvidence({ chain, ...options })`
in `src/fetch/fetch.ts` that iterates the leaf-first chain, fetches OCSP and/or
CRL evidence per certificate (respecting a `prefer` knob consistent with
`RevocationPolicy`), and aggregates into `{ crls, ocspResponses, ... }`. Shape
the return so it drops directly into `checkChainRevocation(input)` and
`verifyCertificateChain({ revocation: { crls, ocspResponses } })` with no
changes to those functions. Collect per-endpoint transport failures into a
typed, non-fatal errors array (mirroring `executionErrors`), so a single dead
responder never sinks the whole verdict.

Phase 3: Chart the maps — docs, scope, changelog, and tests

This phase updates the deliberately-stated scope boundary so it doesn't become a lyin' scoundrel, and proves the module works without ever dialling a real network by default.

Task 1: Update scope and architecture docs

The "HTTP transport is caller-provided by design" language must now describe the opt-in escape hatch precisely, without over-claiming RFC completeness.

• Update docs/PKIX-SCOPE.md (OCSP row + CRL §12) to note the opt-in micro509/fetch module while keeping the pure-core default explicit.
• Update README.md imports/standards sections and docs/AGENTS.md standards table to reference the new module and RFC 6960/5019/5280 transport behaviour (as opt-in, not part of the "complete" core claims).
• Update root AGENTS.md (structure tree, where-to-look) and src/AGENTS.md (where-to-look/hotspots) to register the new fetch domain and its import boundaries (may use #micro509/internal/* and reuse from revocation, but obey barrel conventions).

Task 2: Add tests with an injectable transport (no mocks)

Prove OCSP/CRL fetching, timeout, cache, and error mapping using real in-process bytes and a substitute transport — no network by default.

• Add test/fetch.test.ts using bun:test, synthesising real certs/CRLs/OCSP responses via test/helpers.ts and feeding a real (non-mock) in-memory transport function that returns crafted Responses.
• Assert typed result objects and FetchEvidenceErrorCode values (network error, timeout, http error, content-type), and confirm output feeds cleanly into checkChainRevocation/verifyCertificateChain.
• If a genuine live round-trip is ever wanted, gate it behind an availability probe + env flag with describe.skip (the oracles/openssl.ts precedent), keeping CI network-independent.

Task 3: Record the change and bump versions in lockstep

• Add an entry under ## [Unreleased] in CHANGELOG.md describing the new opt-in fetch module.
• Bump version in both package.json and jsr.json to the same new SemVer (minor bump — additive), promote [Unreleased] to a dated section, and add/repoint the compare-link footer per the changelog's own release checklist.

🤖 Prompt for AI agents

Update documentation to accurately describe the new opt-in fetch escape hatch,
add no-mock tests proving the module works, and record the change in the
changelog and package versions.

- Update `docs/PKIX-SCOPE.md` (OCSP row + CRL §12) to note the opt-in
`micro509/fetch` module while keeping the pure-core default explicit.
- Update `README.md` imports/standards sections and `docs/AGENTS.md` standards
table to reference the new module and RFC 6960/5019/5280 transport behaviour,
framed as opt-in rather than part of the "complete" core claims.
- Update root `AGENTS.md` (structure tree, where-to-look) and `src/AGENTS.md`
(where-to-look/hotspots) to register the new `fetch` domain and its import
boundaries, noting it may use `#micro509/internal/*` and reuse from `revocation`
while obeying barrel conventions.

- Add `test/fetch.test.ts` using `bun:test`, synthesising real certs/CRLs/OCSP
responses via `test/helpers.ts` and feeding a real (non-mock) in-memory
transport function returning crafted `Response`s.
- Assert typed result objects and `FetchEvidenceErrorCode` values (network
error, timeout, http error, content-type), and confirm the output feeds cleanly
into `checkChainRevocation`/`verifyCertificateChain`.
- If a genuine live round-trip test is wanted, gate it behind an availability
probe + env flag using `describe.skip` (following the `oracles/openssl.ts`
precedent), keeping CI network-independent.

- Add an entry under `## [Unreleased]` in `CHANGELOG.md` describing the new
opt-in fetch module.
- Bump `version` in both `package.json` and `jsr.json` to the same new SemVer
(minor bump — additive), promote `[Unreleased]` to a dated section, and
add/repoint the compare-link footer per the changelog's own release checklist.

Research

Arr, micro509 be a zero-dependency TypeScript PKI galleon! Each domain sits behind its own subpath barrel (./x509, ./revocation, ./verify, ./pem, ./result…), all wired from the single entry map in tsdown.config.ts that auto-generates package.json and jsr.json exports (never hand-edit the generated blocks, ye scurvy dog!).

Revocation logic (src/revocation/) already parses AIA/CDP extensions, builds OCSP requests (createOcspRequest → { der, pem, base64 }), extracts OCSP responder URIs (getCertificateOcspResponderUris), and validates caller-supplied evidence — but it deliberately never fetches, a boundary spelled out in docs/PKIX-SCOPE.md.

Fallible APIs all specialise Result/Micro509Error from src/result/. There be no existing fetch/AbortController/base64url usage anywhere, so this module plants the first flag on that shore.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the new opt-in `micro509/fetch` domain scaffold with no real network
I/O yet, plus a missing CRL discovery helper.

- Create `src/fetch/` with `src/fetch/fetch.ts` (implementation) and a
re-export-only `src/fetch/index.ts` barrel, mirroring the
`revocation/{index,revocation}.ts` pattern.
- Register the new domain by adding `fetch: 'src/fetch/index.ts'` to the `entry`
map in `tsdown.config.ts`; let the build regenerate `package.json`/`jsr.json`
exports without hand-editing the generated blocks.
- Do not re-export anything from root `src/index.ts` — this module must remain
opt-in only.
- Add `getCertificateCrlUrls(certificate)` to `src/revocation/revocation.ts`,
modelled on `getCertificateOcspResponderUris`: normalise
PEM/DER/`ParsedCertificate` input, fail-closed to `[]`, dedupe by URI, read from
`parsedCertificate.crlDistributionPoints[].distributionPoint.fullName`, filter
`GeneralName` entries to `type === 'uri'`, and keep only `http`/`https` schemes.
Export it from the `revocation` barrel.
- In `src/fetch/fetch.ts`, define a flat `FetchEvidenceErrorCode` string-literal
union (e.g. `'network_error' | 'timeout' | 'http_error' | 'invalid_content_type'
| 'empty_response' | 'no_endpoints'`) and an optional `FetchFailureDetails`
(e.g. `status?: number`, `url?: string`, `timeoutMs?: number`).
- Build `*Result` types using `src/result/` primitives
(`ErrorResult`/`Micro509Error` + `failureResult`), following
`ValidateOcspResponseResult` as the template, ensuring transport hiccups are
soft typed failures rather than thrown exceptions.
- Define an injectable transport type (a caller-suppliable `fetch`-shaped
function defaulting to global `fetch`) plus shared options: `timeoutMs`,
`signal`, `maxAttempts`, optional `cache`, `at`, `maxAgeMs`.
===============================================================================

Task: 2

Implement the actual OCSP and CRL over-HTTP fetch primitives and a chain-level
orchestrator that produces evidence for the existing revocation APIs, unchanged.

- For OCSP fetching: reuse `createOcspRequest` (its `.der`) and
`resolveOcspResponderCandidates`/`getCertificateOcspResponderUris` from
`revocation` to pick responder URIs. Default to POST with `Content-Type:
application/ocsp-request`, expecting `application/ocsp-response`; also support
the GET form for small requests by base64-encoding the DER via `base64Encode`
from `src/internal/shared/base64.ts` then percent-encoding it (add the missing
URL-safe step locally, since no base64url/percent helper exists). Return the raw
DER `Uint8Array` (optionally normalising via the `pem` barrel) — no pre-parsing
needed since chain-level `ocspResponses` accepts `string | Uint8Array`. Apply
`timeoutMs`/`signal` via `AbortController`, honour `maxAttempts`, and surface
content-type/status problems as typed `FetchEvidenceErrorCode` results.

- For CRL fetching: use `getCertificateCrlUrls` to enumerate candidate CDP URLs
(http/https only). GET each URL, expecting `Content-Type: application/pkix-crl`
but treating media type as a hint rather than gospel (parse-or-fail rather than
reject solely on header), returning raw DER bytes. Share the same
timeout/abort/attempt/cache plumbing as OCSP; on cache hits, honour the CRL's
`nextUpdate` (and optional `maxAgeMs`) as the freshness boundary before
refetching.

- For orchestration: add `fetchChainRevocationEvidence({ chain, ...options })`
in `src/fetch/fetch.ts` that iterates the leaf-first chain, fetches OCSP and/or
CRL evidence per certificate (respecting a `prefer` knob consistent with
`RevocationPolicy`), and aggregates into `{ crls, ocspResponses, ... }`. Shape
the return so it drops directly into `checkChainRevocation(input)` and
`verifyCertificateChain({ revocation: { crls, ocspResponses } })` with no
changes to those functions. Collect per-endpoint transport failures into a
typed, non-fatal errors array (mirroring `executionErrors`), so a single dead
responder never sinks the whole verdict.
===============================================================================

Task: 3

Update documentation to accurately describe the new opt-in fetch escape hatch,
add no-mock tests proving the module works, and record the change in the
changelog and package versions.

- Update `docs/PKIX-SCOPE.md` (OCSP row + CRL §12) to note the opt-in
`micro509/fetch` module while keeping the pure-core default explicit.
- Update `README.md` imports/standards sections and `docs/AGENTS.md` standards
table to reference the new module and RFC 6960/5019/5280 transport behaviour,
framed as opt-in rather than part of the "complete" core claims.
- Update root `AGENTS.md` (structure tree, where-to-look) and `src/AGENTS.md`
(where-to-look/hotspots) to register the new `fetch` domain and its import
boundaries, noting it may use `#micro509/internal/*` and reuse from `revocation`
while obeying barrel conventions.

- Add `test/fetch.test.ts` using `bun:test`, synthesising real certs/CRLs/OCSP
responses via `test/helpers.ts` and feeding a real (non-mock) in-memory
transport function returning crafted `Response`s.
- Assert typed result objects and `FetchEvidenceErrorCode` values (network
error, timeout, http error, content-type), and confirm the output feeds cleanly
into `checkChainRevocation`/`verifyCertificateChain`.
- If a genuine live round-trip test is wanted, gate it behind an availability
probe + env flag using `describe.skip` (following the `oracles/openssl.ts`
precedent), keeping CI network-independent.

- Add an entry under `## [Unreleased]` in `CHANGELOG.md` describing the new
opt-in fetch module.
- Bump `version` in both `package.json` and `jsr.json` to the same new SemVer
(minor bump — additive), promote `[Unreleased]` to a dated section, and
add/repoint the compare-link footer per the changelog's own release checklist.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!