You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Update readme
* More updates
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Copy file name to clipboardExpand all lines: README.md
+41-25Lines changed: 41 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,11 +23,12 @@ with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futu
23
23
[ThreadPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) for parallel
24
24
execution of Python functions on a single computer. executorlib extends this functionality to distribute Python
25
25
functions over multiple computers within a high performance computing (HPC) cluster. This can be either achieved by
26
-
submitting each function as individual job to the HPC job scheduler - [HPC Submission Mode]() - or by requesting a
27
-
compute allocation of multiple nodes and then distribute the Python functions within this allocation - [HPC Allocation Mode]().
28
-
Finally, to accelerate the development process executorlib also provides a - [Local Mode]() - to use the executorlib
29
-
functionality on a single workstation for testing. Starting with the [Local Mode]() set by setting the backend parameter
30
-
to local - `backend="local"`:
26
+
submitting each function as individual job to the HPC job scheduler - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html) -
27
+
or by requesting a compute allocation of multiple nodes and then distribute the Python functions within this - allocation -
28
+
[HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html). Finally, to accelerate the
29
+
development process executorlib also provides a - [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html) -
30
+
to use the executorlib functionality on a single workstation for testing. Starting with the [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html)
31
+
set by setting the backend parameter to local - `backend="local"`:
31
32
```python
32
33
from executorlib import Executor
33
34
@@ -60,8 +61,7 @@ Python function. In addition to the compute cores `cores`, the resource dictiona
60
61
as `threads_per_core`, the GPUs per core as `gpus_per_core`, the working directory with `cwd`, the option to use the
61
62
OpenMPI oversubscribe feature with `openmpi_oversubscribe` and finally for the [Simple Linux Utility for Resource
62
63
Management (SLURM)](https://slurm.schedmd.com) queuing system the option to provide additional command line arguments
63
-
with the `slurm_cmd_args` parameter - [resource dictionary]().
64
-
64
+
with the `slurm_cmd_args` parameter - [resource dictionary](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#resource-dictionary
65
65
This flexibility to assign computing resources on a per-function-call basis simplifies the up-scaling of Python programs.
66
66
Only the part of the Python functions which benefit from parallel execution are implemented as MPI parallel Python
67
67
funtions, while the rest of the program remains serial.
@@ -87,7 +87,7 @@ with Executor(backend="slurm_submission") as exe:
87
87
```
88
88
In this case the [Python simple queuing system adapter (pysqa)](https://pysqa.readthedocs.io) is used to submit the
89
89
`calc()` function to the [SLURM](https://slurm.schedmd.com) job scheduler and request an allocation with two CPU cores
90
-
for the execution of the function - [HPC Submission Mode](). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html)
90
+
for the execution of the function - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html)
91
91
command is used to request the allocation to execute the Python function.
92
92
93
93
Within a given [SLURM](https://slurm.schedmd.com) allocation executorlib can also be used to assign a subset of the
@@ -116,23 +116,39 @@ In addition, to support for [SLURM](https://slurm.schedmd.com) executorlib also
116
116
to address the needs for the up-coming generation of Exascale computers. Still even on traditional HPC clusters the
117
117
hierarchical approach of the [flux](http://flux-framework.org) is beneficial to distribute hundreds of tasks within a
118
118
given allocation. Even when [SLURM](https://slurm.schedmd.com) is used as primary job scheduler of your HPC, it is
119
-
recommended to use [SLURM with flux]() as hierarchical job scheduler within the allocations.
119
+
recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm-with-flux)
120
+
as hierarchical job scheduler within the allocations.
![pre-commit-ci[bot]](https://avatars.githubusercontent.com/in/68672?v=4&size=40){kind=avatar}
0 commit comments