feat: Add initial support for DRS inputs/outputs (drs://)

[lbeckman314]

Overview 🌀

This PR adds initial support for using DRS URIs as inputs/outputs in Nextflow workflows!

Current Behavior ⚠️

• drs:// URIs are not recognized as valid input/output:

example.nf

workflow {
    align(file("drs://drs.example.org/object-id"))   // <---- Fails with `drs:// is an unknown scheme` error
}

New Behavior ✔️

• drs:// URIs are resolved automatically
• Bearer token auth is supported via drs.accessToken in nextflow.config

example.nf

workflow {
    align(file("drs://drs.example.org/object-id"))  // <---- nf-ga4gh resolves the DRS URI and streams the object
}

DRS Tests 🧪

Tested against a local Syfon DRS server backed by MinIO.

1. Run MinIO

➜ docker run -d --name minio -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minio-user \
  -e MINIO_ROOT_PASSWORD=minio-pass \
  minio/minio server /data --console-address ":9001"

➜ docker run --rm --network host --entrypoint sh minio/mc -c '
  mc alias set local http://localhost:9000 minio-user minio-pass &&
  mc mb -p local/local-bucket
'

2. Start Syfon

local.yaml

port: 8080
auth:
  mode: local
  allow_unauthenticated: true
database:
  sqlite:
    file: "drs_local.db"
s3_credentials:
  - bucket: "local-bucket"
    region: "us-east-1"
    access_key: "minio-user"
    secret_key: "minio-pass"
    endpoint: "http://localhost:9000"
    resources:
      - organization: "example"
        projects:
          - project: "example"

➜ syfon serve --config local.yaml

3. Upload a test object

➜ syfon upload --file README.md --org example --project example
<DID>

4. Build + install the plugin

➜ ./gradlew install

5. Run a workflow that consumes the DRS object

nextflow.config

plugins {
    id 'nf-ga4gh@1.4.2-rc.1'
}
drs {
    metadataScheme = 'http'   // <---- Local Syfon serves plain HTTP
}

test-drs.nf

process INSPECT {
    input:  path drs_file
    output: stdout
    script:
    """
    wc -l ${drs_file}
    head -1 ${drs_file}
    """
}

workflow {
    INSPECT(file('drs://localhost:8080/<DID>')).view()
}

➜ nextflow run test-drs.nf
executor >  local (1)
[af/c3de81] INSPECT | 1 of 1 ✔
     310 d6ee08ec-638e-52df-afa3-9459a22282f2

Breaking Changes? ❌

• No breaking changes to existing behavior/workflows — just adding additional support for DRS URI's!

Next Steps ⚙️

• Test against DRS implementations, possible options:
  • Syfon
  • ga4gh-starter-kit-drs
  • Others?
• Create Release Candidate for Plugin (1.5.0-rc.1)
• Document steps for testing + update here!
• Run final review + release 1.5.0!

[lbeckman314]

Important

Note to Self: Review the following 🔮

How drs:// URIs are resolved

The nf-ga4gh plugin lets you use GA4GH Data Repository Service
(DRS) objects as process inputs and outputs via drs://host/object_id URIs:

workflow {
    align(file("drs://drs.example.org/object-id"))
}

How that URI turns into actual bytes depends on which executor runs the task. There are
two distinct flows.

Flow A — Local / default execution (Nextflow resolves the object)

When a task runs locally — or whenever Nextflow itself needs to read a DRS object (e.g.
staging into a shared work directory) — the plugin's DRS filesystem provider resolves the
URI and streams the bytes. This uses the standard two-hop DRS protocol.

Given drs://<host>/<object_id>, the metadata API base URL is:

<scheme>://<host>/ga4gh/drs/v1

where <scheme> is https by default (configurable via drs.metadataScheme).

1. Hop 1 — fetch object metadata
   GET <base>/objects/<object_id> → returns a DrsObject containing an
   access_methods[] array.

2. Choose an access method
   The plugin picks the best method by type preference:
   https › s3 › gs › ftp, falling back to the first entry if none match.

3. Hop 2 — resolve the download URL (conditional)

   • If the chosen access method has an access_id:
     GET <base>/objects/<object_id>/access/<access_id> → returns a concrete
     (often presigned) url plus optional headers.
   • If it has only an inline access_url (no access_id): that URL is used directly.

4. Stream the bytes
   The resolved HTTP(S) URL is opened and streamed into the task work directory. Any
   headers returned with the access URL (e.g. presigned-URL auth) are forwarded on the
   download request.

Authentication (drs.accessToken Bearer, or drs.username / drs.password Basic) is
applied to the metadata hops (hops 1 and 2). See the
DRS configuration in the README.

Flow B — TES executor (the backend resolves the object)

Under the TES executor, the plugin does not resolve the DRS object. The
drs://host/object_id string is placed verbatim into the TES task input url, and
resolution is left entirely to the TES server. The two-hop protocol described above does
not run in this path — so drs.* auth settings are not applied by the plugin here, and any
credentials the DRS server requires must be available to the TES backend.

Resolution flow

flowchart TD
    A["file('drs://host/object_id')"] --> B{Executor?}

    B -->|"Local / default"| C["DrsFileSystemProvider<br/>newByteChannel / newInputStream"]
    C --> D["Hop 1: GET /ga4gh/drs/v1/objects/{object_id}<br/>→ access_methods[]"]
    D --> E["Choose access method<br/>https › s3 › gs › ftp"]
    E --> F{"access_id present?"}
    F -->|"Yes"| G["Hop 2: GET /objects/{object_id}/access/{access_id}<br/>→ url + headers"]
    F -->|"No"| H["Use inline access_url"]
    G --> I["Open URL, stream bytes → task work dir"]
    H --> I

    B -->|"TES"| J["TesInput.url = 'drs://host/object_id'<br/>(verbatim, unmodified)"]
    J --> K["TES backend resolves<br/>and fetches the DRS object"]

Loading

Notes and limitations

• The DRS filesystem is read-only — write operations are not supported.
• In the TES flow, DRS resolution (and its auth) is the responsibility of the TES backend,
  not this plugin.