[phase-1] CloudEvents → S3 archive + analytics sink for PipelineRuns (stateless)

[mmgaggle]

Type

AFK — well-scoped: a small stateless HTTP service + a new S3
bucket + the TektonConfig event-listener wiring.

Why

ceph-tekton is keeping state to a minimum: artifacts in S3,
secrets in Vault, no project-owned databases. The Tekton Results
deploy issue (#59)
was filed initially but its postgres metadata store cuts against
that posture — we'd own a DB to back up, patch, and monitor for
a service whose state we'd rather hold in S3.

The lighter shape: catch the CloudEvents Tekton already emits
on PipelineRun lifecycle transitions (dev.tekton.event.pipelinerun.successful.v1

• .failed.v1 + .cancelled.v1) at a small stateless HTTP sink,
  and have the sink write:

1. One archive JSON per terminal event — the full PipelineRun
   payload that came in the CloudEvent — to
   s3://ceph-build-archive/pipelineruns/year=YYYY/month=MM/day=DD/<name>-<uid>.json
2. One analytics Parquet row per terminal event — the
   denormalised fields the build team queries (pipeline, branch,
   sha, distro, arch, status, durations, taskrun breakdown,
   artifact URIs + digests) — to a Hive-partitioned prefix under
   the same bucket.

This **collapses #58

• #59 into
  one component** that delivers both archive (replay a 6-month-old
  build) and analytics (DuckDB queries over Parquet for trend
  spotting) without postgres.

What to build

1. ceph-build-archive bucket

Add a 5th bucket to terraform/modules/s3-buckets/:

• No versioning, no object-lock
• No public read — PipelineRun payloads include step logs,
  internal URLs, and occasionally leaked secrets. First private
  bucket in the module (same posture [phase-1] Deploy Tekton Results with S3 backend on Sepia #59 had specified).
• Long retention: configurable, default 0 (never expire).
  Historical trend queries want as much history as we can afford.
• Wire through dev / dev-rgw / sepia envs.

Variables:

• build_archive_bucket_name (default ceph-build-archive)
• build_archive_public_read (default false)
• build_archive_expiration_days (default 0 = never expire)

2. The sink: ceph-build-archive-sink

A small Go HTTP server that:

• Listens on /cloudevents for incoming Tekton CloudEvents.
• Validates the event type is one of the terminal PipelineRun
  events (success / failed / cancelled).
• Pulls the full PipelineRun off the event payload
  (data.pipelineRun).
• Writes the archive JSON + analytics Parquet row to S3 via the
  AWS SDK.
• Authenticates to RGW via STS-OIDC (same pattern as
  generate-sbom's uploader step uses today).
• Idempotent: deduplicates on (pipelinerun.UID, pipelinerun.status.completionTime) so a re-delivered event
  doesn't write twice.

Deploys as a Deployment + Service (no Knative dependency).
Tekton's event-listener URI in TektonConfig points at the Service
DNS name.

3. Parquet schema (sketch — finalise during implementation)

One row per terminal PipelineRun. Columns:

• Identifiers: pipeline, pipelinerun, namespace, uid,
  event_type, event_time
• Source: branch, sha, event_source (push/pr/cron/tag)
• Status: status (Succeeded/Failed/Cancelled), failure_reason
• Timing: start_time, completion_time, duration_seconds
• TaskRun summary: taskruns_json (a nested JSON column listing
  task, status, start, end, distro, arch, step-durations) — keep
  nested rather than fan out so the per-pipeline schema is stable
• Artifacts: artifacts_json (the type-hinted IMAGES +
  ARTIFACT_OUTPUTS rolled up — one row per artifact emitted)
• Attestation pointers: rekor_uuids (array)

Partitioning: Hive-style year=YYYY/month=MM/day=DD/ so DuckDB's
partition pruner skips out-of-range dates.

4. TektonConfig wiring

Set spec.pipeline.default-cloud-events-sink (or per-PipelineRun
override via the same field) to point at the sink's Service URL.

5. Documentation

docs/build-archive.md covering:

• Architecture diagram (PipelineRun → CloudEvent → sink → S3
  archive + Parquet)
• DuckDB query recipes ("long pole task by distro", "queue
  time trend", "signed-Rekor latency", "build success rate by
  branch")
• How to replay an archived PipelineRun (aws s3 cp ... && kubectl create -f <archived.json> — the JSON is the full
  PipelineRun object the sink captured)
• Schema documentation (Parquet columns + types)
• Eventual-promotion path to Iceberg if scale ever demands it

6. e2e assertion

hack/e2e/assert-build-archive-sink.sh:

• Spins up the sink Deployment + Service on kind
• Configures TektonConfig to point at it
• Runs a smoke PipelineRun
• Asserts an archive JSON + Parquet row land in the test bucket
  (in-cluster MinIO or another httpd-mocked S3)

Acceptance criteria

• ceph-build-archive bucket terraform-managed across all
  three envs, private by default
• Sink Deployment + Service installed in
  kustomize/base/build-archive-sink/
• TektonConfig in the Sepia overlay points its cloud-events
  sink at the Service
• Terminal PipelineRuns drop one JSON + one Parquet row to
  the configured S3 path
• duckdb -c "SELECT pipeline, avg(duration_seconds) FROM 's3://ceph-build-archive/analytics/**/*.parquet' GROUP BY 1"
  returns expected shape against a populated bucket
• docs/build-archive.md covers the architecture, schema,
  query recipes, and the replay-from-archive procedure
• e2e assertion wired into hack/e2e/run-all.sh
• Sink can be idempotently re-deployed without losing or
  duplicating archived rows

Dependencies

• #60
  (TektonConfig pruner): the pruner is only safe to enable
  aggressively once this sink is reliably catching events.
  Order: deploy sink first → verify archive lands → only then
  flip the pruner on.
• RGW STS-OIDC trust must be configured (separate phase-1 work)
  so the sink can mint short-lived S3 creds from its projected
  SA token.

Supersedes

• #58 —
  the per-PipelineRun-Task that landed Parquet rows. Same goal,
  caught at the event boundary instead of from a finally: Task,
  so no per-Pipeline wiring needed.
• #59 —
  Tekton Results + postgres. The CloudEvents sink delivers the
  same archive coverage without a database.

Reference

• Tekton CloudEvents docs:
  https://tekton.dev/docs/pipelines/events/
• Tekton's cloud-events-sink config:
  https://tekton.dev/docs/pipelines/install/#customizing-the-pipelines-controller-behavior
• Red Hat scale post (the original prompt that surfaced this):
  https://www.redhat.com/en/blog/operating-tekton-scale-10-lessons-learned