[FEATURE] Complete the publish bundle loop (verifiable archive + run/pull consumers)

[andrewbudge]

The idea

publish already emits a manifest.json (format: "oxoflow-bundle-v1"), so the
scaffolding for shareable bundles exists — but the loop isn't closed, and a schema
mismatch makes it bundle nothing for most workflows today.

What I found in the code

1. Wrong environment schema. publish.rs scans each rule for [rules.env].file
   and a conda_env field. Neither exists — the real schema is [rules.environment]
   (EnvironmentSpec, rule.rs:203) with conda / pixi / docker / singularity
   / venv. A rule using [rules.environment]\nconda = "envs/fastp.yaml" matches
   nothing, so publish collects zero env files and bundles none — silently.
2. Output is a loose, uncompressed directory, and the manifest records env
   filenames with no checksums — nothing self-contained or verifiable to hand off.
3. No consumer. Nothing reads manifest.json: run has no bundle path and there's
   no fetch. The manifest is write-only.

Proposed solution

Close the loop in three pieces:

• Producer — read the real [rules.environment] schema; preserve source-relative
  paths (keep envs/); include bin/ (Nextflow auto-PATH convention); emit a single
  archive (.tar.zst) with a complete manifest: format version, entrypoint workflow,
  producing oxo-flow version, per-file SHA-256, and container image digests.
• Consumer — run accepts a bundle: extract → verify every file against the
  manifest → execute, refusing on checksum mismatch.
• Transport — a new oxo-flow pull <url> subcommand (there's no fetch path
  today) that resolves a remote bundle (GitHub release asset / raw URL /
  gh:owner/repo@ref), downloads it, verifies it against its manifest, and lands it
  locally ready to run.

End state: oxo-flow pull gh:user/repo@v1 && oxo-flow run <bundle> reproduces the
exact scripts + environments, checksum-verified. Because the bundle carries bin/,
this also makes it easy to share workflows that depend on custom helper scripts
the collaborator gets them automatically, instead of having to track down and place
loose scripts by hand.

Open questions:

1. Env/container vendoring. conda/pixi: spec files, lock files, or fully packed
   (conda-pack)? Containers: record digests + pull-on-run, or vendor image tarballs?
   This is the repro-vs-size tradeoff , and I want your input before making any design decisions.
2. Trust. pull + run is remote code execution. Checksums give integrity, not
   provenance — verify-before-run only, a confirmation gate, or signing later?
3. v1 scope. Stop at local archive round-trip (A–C), or include remote fetch (D)?

Suggested phasing (each a small PR)

• A. Fix the publish env-schema + path bugs — pure bugfix, independent of B–D.
• B. Emit a single verifiable archive + complete manifest.
• C. run consumes + verifies a local bundle.
• D. Add the new pull subcommand for remote bundles.

[ShixiangWang]

@andrewbudge

Great issue — the publish loop being half-open is one of those things where the scaffolding is there but nobody's walked through the whole thing yet.

A few thoughts on the proposal:

---

1. Phase A should ship immediately

Scanning [rules.env] while the schema has [rules.environment] is a silent bug — publish succeeds but the environment is never bundled, and the user has no idea. This isn't a feature, it's a bugfix. No reason to wait for B–D.

2. Platform scope: target Linux, be honest about the rest

Let's be realistic: Linux is the only platform where bioinformatics dependency support can be systemic. Windows is a lost cause for most bioinfo tools, and macOS conda/pixi behavior differs from Linux in subtle but painful ways. I'd recommend:

• Primary target: Linux. This is where publish→run round-trips get full testing and official support.
• macOS: best-effort. spec-only and lockfile strategies may work; packed almost certainly won't cross OS.
• Windows: documented as unsupported for the bundle loop (WSL users can run the Linux path).

This isn't a limitation — it's honesty about the bioinformatics dependency landscape.

3. Archive format: default .tar.gz, keep options open

I'd suggest .tar.gz as the default. It requires zero extra dependencies on either end, and for workflow bundles the bottleneck is environments, not script compression ratios. That said, supporting multiple formats (tar.gz / tar.zst / tar.xz) via a --format flag is cheap if the archive layer is abstracted early.

4. Environment vendoring: layered strategy

The "repro vs size" tension is real. Rather than picking one path, declare the strategy in the manifest:

Strategy	Mechanism	Bundle size	When to use
spec-only	env.yaml / pixi.toml	KB	Consumer has network and a working solver
lockfile	conda-lock / pixi.lock	MB	Reproducible resolve, still needs network
packed	conda-pack tarball	GB	True offline, same-OS only
digest	OCI image digest	0	Docker/Singularity already pulled separately

v1 should start with spec-only + digest. packed (conda-pack) can come later — it's valuable but the cross-platform failure modes are nasty and should not block the initial loop.

Caveat worth stating explicitly: even with packed or digest, cross-machine reproducibility is never guaranteed. conda-pack can break on glibc mismatches. Docker images pull successfully but OOM on different hardware. We're doing our best, not promising magic.

5. Security: checksums are necessary, not sufficient

pull + run is remote code execution. The v1 safety net should be:

• Manifest SHA-256 verified for every file (including bin/)
• Before execution: print bundle origin, key parameters, environment strategy
• Interactive confirmation prompt (skip with --yes)
• Reserve signatures field in manifest format — empty for now, but don't paint ourselves into a corner

No GPG, no key management, no trust-on-first-use. Just "here's what you're about to run and where it came from — proceed?"

6. Manifest should include rule-level resource declarations

PR #49 added pre-flight resource checks. The manifest should carry this forward: include each rule's resources { memory, threads } so oxo-flow run can warn "this bundle needs ≥ 32 GB RAM" before anything starts executing.

7. Phase D (pull) should not wait

Agreed that oxo-flow pull doesn't need to be deferred. The producer→consumer loop is the core value; pull is the natural distribution mechanism on top of it. As long as B+C are solid, D can follow immediately.

---

Suggested priority

A (fix schema bug)   → merge now, separate PR
B (producer) + C (consumer) → single or paired PR
D (pull)             → follow-up, no reason to delay

[andrewbudge]

1: Bug Fix

That bug fix will be implemented ASAP.

2: Focus on Linux

I fully agree Linux should be the fully supported environment. While macOS and Windows support would be interesting, implementation and maintenance would be huge and frankly not worth the effort. As you said, Linux covers the most common use case and development and implementation should be geared towards meeting that user base.

3: Archive format

Again, I agree with tar.gz being the canon archive/bundle option. It's common and well supported, which will ease implementation. I do think it should be a priority to ensure a level of abstraction so that if other formats are wanted, adding that feature will be easier. It may even be nice for oxo-flow to eventually detect the format of an archive and run it accordingly, but a --format flag is the right scope for a first step.

4: Environment

Like the archive, I like the idea of multiple options and agree that spec-only + digest should be the first step, since that's the most common use case. Same as with the archive format, I think a level of abstraction here will pay off when/if other environment management methods get added later. And agreed on the caveat — even with packed or digest strategies we shouldn't oversell reproducibility. conda-pack breaking on glibc mismatches and containers OOMing on different hardware are real failure modes, and I think it's worth stating that plainly somewhere users will see it (README/docs), not just leaving it as an internal assumption.

5: Security

Yes, making sure tools are tracked and safe to use is crucial. I agree that verbose and explicit confirmation before running a pipeline will ensure a user understands what's going to happen and what exactly will be pulled and run on their machine. Specifically:

• Include the --yes flag to skip the prompt for scripted/CI use — the confirmation gate shouldn't get in the way of automation once someone trusts a source.
• Go ahead and reserve the empty signatures field in the manifest format now. Costs nothing today and saves us a breaking schema change later if/when signing gets added.

6: Rule-level resource declarations

I like that idea! That will avoid OOM crashes and lost resources and time in running pipelines.

7: pull subcommand

Agreed, pull is a natural extension and will also help accelerate adoption and development of oxo-flow. And to close the loop on scope — I'm on board with your suggested PR structure: A merges now on its own, B+C as a single/paired PR, D as an immediate follow-up rather than something we sit on.

[ShixiangWang]

Great~