Auto-add Slurm schedulers to workflows #57
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds automatic Slurm scheduler generation for HPC workflows, enabling users to submit workflows without manually configuring schedulers. The system uses HPC profiles to match job resource requirements to appropriate partitions and generates schedulers and workflow actions automatically.
Key changes:
- New
submit-slurmcommand that auto-generates Slurm schedulers from job resource requirements - HPC profile system with built-in Kestrel profile and support for custom profiles
- Automatic partition matching based on CPU, memory, GPU, and walltime requirements
- KDL format improvements including
depends_onanddepends_on_regexessupport
Reviewed changes
Copilot reviewed 33 out of 35 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_hpc.rs |
Comprehensive test suite for HPC profiles, partition matching, and scheduler generation |
tests/test_parameterization.rs |
Updated test to account for additional postprocess job in example |
src/main.rs |
Added SubmitSlurm command with auto-generation and submission logic |
src/cli.rs |
Added SubmitSlurm and Hpc commands to CLI structure |
src/config/mod.rs |
Exported HPC configuration types |
src/config/loader.rs |
Added HPC configuration section to default config template |
src/config/client.rs |
Added ClientHpcConfig with profile override and custom profile support |
src/client/workflow_spec.rs |
Added KDL serialization, improved KDL parsing with better error messages |
src/client/mod.rs |
Exported HPC profile types |
src/client/hpc/profiles.rs |
Core HPC profile system with partition matching algorithm |
src/client/hpc/mod.rs |
Added profile module exports |
src/client/hpc/kestrel.rs |
NREL Kestrel HPC profile with 15 partitions |
src/client/commands/mod.rs |
Added HPC commands module |
src/client/commands/hpc.rs |
HPC CLI commands and scheduler generation logic |
src/client/commands/workflows.rs |
Added create-slurm workflow command |
src/client/commands/slurm.rs |
Added generate command for scheduler generation |
examples/kdl/*.kdl |
Updated KDL examples with corrected syntax and new examples |
examples/yaml/resource_monitoring_demo.yaml |
Updated with distinct resource requirements |
docs/** |
Comprehensive documentation for HPC profiles, Slurm workflows, and custom profiles |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 49 out of 49 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
daniel-thom
force-pushed
the
feat/auto-add-slurm-schedulers
branch
from
2c997dd to
168aa16
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 67 out of 67 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
torc-dash/static/js/app-modals.js
Outdated
| const isLeaf = leafEvents.includes(event.id); | ||
| const jobs = event.jobs_becoming_ready || []; | ||
| const schedulers = event.scheduler_allocations || []; | ||
| const dependsOn = event.depends_on_events || []; |
There was a problem hiding this comment.
Unused variable dependsOn.
No description provided.