Skip to content

Add support for concurrency via threads to fs mode#841

Merged
maddenp-cu merged 6 commits intoufs-community:mainfrom
maddenp-cu:fs-threads
Feb 12, 2026
Merged

Add support for concurrency via threads to fs mode#841
maddenp-cu merged 6 commits intoufs-community:mainfrom
maddenp-cu:fs-threads

Conversation

@maddenp-cu
Copy link
Collaborator

Synopsis

Add support for concurrency via threads to fs mode. Underlying iotaa supports threads, so this is just a matter of collecting the requested number of threads from CLI or API users and passing it on. Given a config.yaml specifying 3 GFS forecast GRIB files to download...

Old single-threaded behavior:

$ time uw fs copy -c config.yaml --target-dir ./grib
[2026-02-11T05:16:29]     INFO Validating config against internal schema: files-to-stage
[2026-02-11T05:16:29]     INFO Schema validation succeeded for fs config
[2026-02-11T05:16:29]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f001 -> grib/gfs.t12z.pgrb2.0p25.f001: Executing
[2026-02-11T05:16:51]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f001 -> grib/gfs.t12z.pgrb2.0p25.f001: Ready
[2026-02-11T05:16:51]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f002 -> grib/gfs.t12z.pgrb2.0p25.f002: Executing
[2026-02-11T05:17:21]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f002 -> grib/gfs.t12z.pgrb2.0p25.f002: Ready
[2026-02-11T05:17:21]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f003 -> grib/gfs.t12z.pgrb2.0p25.f003: Executing
[2026-02-11T05:17:47]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f003 -> grib/gfs.t12z.pgrb2.0p25.f003: Ready
[2026-02-11T05:17:47]     INFO File copies: Ready

real    1m19.354s
user    0m4.100s
sys     0m2.402s

New behavior, with three threads:

$ time uw fs copy -c config.yaml --target-dir ./grib --threads 3
[2026-02-11T05:17:58]     INFO Validating config against internal schema: files-to-stage
[2026-02-11T05:17:58]     INFO Schema validation succeeded for fs config
[2026-02-11T05:17:58]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f001 -> grib/gfs.t12z.pgrb2.0p25.f001: Executing
[2026-02-11T05:17:58]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f002 -> grib/gfs.t12z.pgrb2.0p25.f002: Executing
[2026-02-11T05:17:58]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f003 -> grib/gfs.t12z.pgrb2.0p25.f003: Executing
[2026-02-11T05:18:30]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f003 -> grib/gfs.t12z.pgrb2.0p25.f003: Ready
[2026-02-11T05:18:31]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f001 -> grib/gfs.t12z.pgrb2.0p25.f001: Ready
[2026-02-11T05:18:35]     INFO HTTP https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260209/12/atmos/gfs.t12z.pgrb2.0p25.f002 -> grib/gfs.t12z.pgrb2.0p25.f002: Ready
[2026-02-11T05:18:35]     INFO File copies: Ready

real    0m36.939s
user    0m4.200s
sys     0m2.316s

So, a reduction from 1m19s to 0m37s.

Type

  • Documentation
  • Enhancement (adds new functionality)

Impact

  • This is a non-breaking change (existing functionality continues to work as expected)

Checklist

  • I have added myself and any co-authors to the PR's Assignees list.
  • I have reviewed the documentation and have made any updates necessitated by this change.

@maddenp-cu maddenp-cu self-assigned this Feb 11, 2026
@maddenp-cu maddenp-cu marked this pull request as ready for review February 11, 2026 05:28
@maddenp-cu maddenp-cu merged commit 821fa54 into ufs-community:main Feb 12, 2026
2 checks passed
@maddenp-cu maddenp-cu deleted the fs-threads branch February 12, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants