11# Development
2- The ` pympipool ` package is developed based on the need to simplify the up-scaling of python functions over multiple
2+ The ` executorlib ` package is developed based on the need to simplify the up-scaling of python functions over multiple
33compute nodes. The project is used for Exascale simualtion in the context of computational chemistry and materials
44science. Still it remains a scientific research project with the goal to maximize the utilization of computational
55resources for scientific applications. No formal support is provided.
@@ -41,31 +41,31 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
4141```
4242
4343## Integration
44- The key functionality of the ` pympipool ` package is the up-scaling of python functions with thread based parallelism,
44+ The key functionality of the ` executorlib ` package is the up-scaling of python functions with thread based parallelism,
4545MPI based parallelism or by assigning GPUs to individual python functions. In the background this is realized using a
4646combination of the [ zero message queue] ( https://zeromq.org ) and [ cloudpickle] ( https://github.com/cloudpipe/cloudpickle )
47- to communicate binary python objects. The ` pympipool .communication.SocketInterface` is an abstraction of this interface,
48- which is used in the other classes inside ` pympipool ` and might also be helpful for other projects. It comes with a
49- series of utility functions:
50-
51- * ` pympipool .communication.interface_bootup()` : To initialize the interface
52- * ` pympipool .communication.interface_connect()` : To connect the interface to another instance
53- * ` pympipool .communication.interface_send()` : To send messages via this interface
54- * ` pympipool .communication.interface_receive()` : To receive messages via this interface
55- * ` pympipool .communication.interface_shutdown()` : To shutdown the interface
56-
57- While ` pympipool ` was initially designed for up-scaling python functions for HPC, the same functionality can be leveraged
58- to up-scale any executable independent of the programming language it is developed in. This approach follows the design
59- of the ` flux.job.FluxExecutor ` included in the [ flux framework] ( https://flux-framework.org ) . In ` pympipool ` this approach
47+ to communicate binary python objects. The ` executorlib .communication.SocketInterface` is an abstraction of this
48+ interface, which is used in the other classes inside ` executorlib ` and might also be helpful for other projects. It
49+ comes with a series of utility functions:
50+
51+ * ` executorlib .communication.interface_bootup()` : To initialize the interface
52+ * ` executorlib .communication.interface_connect()` : To connect the interface to another instance
53+ * ` executorlib .communication.interface_send()` : To send messages via this interface
54+ * ` executorlib .communication.interface_receive()` : To receive messages via this interface
55+ * ` executorlib .communication.interface_shutdown()` : To shutdown the interface
56+
57+ While ` executorlib ` was initially designed for up-scaling python functions for HPC, the same functionality can be
58+ leveraged to up-scale any executable independent of the programming language it is developed in. This approach follows
59+ the design of the ` flux.job.FluxExecutor ` included in the [ flux framework] ( https://flux-framework.org ) . In ` executorlib ` this approach
6060is extended to support any kind of subprocess, so it is no longer limited to the [ flux framework] ( https://flux-framework.org ) .
6161
6262### Subprocess
6363Following the [ ` subprocess.check_output() ` ] ( https://docs.python.org/3/library/subprocess.html ) interface of the standard
64- python libraries, any kind of command can be submitted to the ` pympipool .SubprocessExecutor` . The command can either be
64+ python libraries, any kind of command can be submitted to the ` executorlib .SubprocessExecutor` . The command can either be
6565specified as a list ` ["echo", "test"] ` in which the first entry is typically the executable followed by the corresponding
6666parameters or the command can be specified as a string ` "echo test" ` with the additional parameter ` shell=True ` .
6767``` python
68- from pympipool import SubprocessExecutor
68+ from executorlib import SubprocessExecutor
6969
7070with SubprocessExecutor(max_workers = 2 ) as exe:
7171 future = exe.submit([" echo" " test" universal_newlines = True )
@@ -74,20 +74,20 @@ with SubprocessExecutor(max_workers=2) as exe:
7474```
7575>>> (False, "test", True)
7676```
77- In analogy to the previous examples the ` SubprocessExecutor ` class is directly imported from the ` pympipool ` module and
78- the maximum number of workers is set to two ` max_workers=2 ` . In contrast to the ` pympipool .Executor` class no other
77+ In analogy to the previous examples the ` SubprocessExecutor ` class is directly imported from the ` executorlib ` module and
78+ the maximum number of workers is set to two ` max_workers=2 ` . In contrast to the ` executorlib .Executor` class no other
7979settings to assign specific hardware to the command via the python interface are available in the ` SubprocessExecutor `
8080class. To specify the hardware requirements for the individual commands, the user has to manually assign the resources
8181using the commands of the resource schedulers like ` srun ` , ` flux run ` or ` mpiexec ` .
8282
83- The ` concurrent.futures.Future ` object returned after submitting a command to the ` pymipool .SubprocessExecutor` behaves
83+ The ` concurrent.futures.Future ` object returned after submitting a command to the ` executorlib .SubprocessExecutor` behaves
8484just like any other future object. It provides a ` done() ` function to check if the execution completed as well as a
8585` result() ` function to return the output of the submitted command.
8686
8787In comparison to the ` flux.job.FluxExecutor ` included in the [ flux framework] ( https://flux-framework.org ) the
88- ` pymipool .SubprocessExecutor` differs in two ways. One the ` pymipool .SubprocessExecutor` does not provide any option for
89- resource assignment and two the ` pymipool .SubprocessExecutor` returns the output of the command rather than just
90- returning the exit status when calling ` result() ` .
88+ ` executorlib .SubprocessExecutor` differs in two ways. One the ` executorlib .SubprocessExecutor` does not provide any
89+ option for resource assignment and two the ` executorlib .SubprocessExecutor` returns the output of the command rather
90+ than just returning the exit status when calling ` result() ` .
9191
9292### Interactive Shell
9393Beyond external executables which are called once with a set of input parameters and or input files and return one set
@@ -111,13 +111,13 @@ if __name__ == "__main__":
111111 count(iterations = int (user_input))
112112```
113113This example is challenging in terms of interfacing it with a python process as the length of the output changes depending
114- on the input. The first option that the ` pympipool .ShellExecutor` provides is specifying the number of lines to read for
114+ on the input. The first option that the ` executorlib .ShellExecutor` provides is specifying the number of lines to read for
115115each call submitted to the executable using the ` lines_to_read ` parameter. In comparison to the ` SubprocessExecutor `
116116defined above the ` ShellExecutor ` only supports the execution of a single executable at a time, correspondingly the input
117117parameters for calling the executable are provided at the time of initialization of the ` ShellExecutor ` and the inputs
118118are submitted using the ` submit() ` function:
119119``` python
120- from pympipool import ShellExecutor
120+ from executorlib import ShellExecutor
121121
122122with ShellExecutor([" python" " count.py" universal_newlines = True ) as exe:
123123 future_lines = exe.submit(string_input = " 4" lines_to_read = 5 )
@@ -130,10 +130,10 @@ The response for a given set of input is again returned as `concurrent.futures.F
130130execute other steps on the python side while waiting for the completion of the external executable. In this case the
131131example counts the numbers from ` 0 ` to ` 3 ` and prints each of them in one line followed by ` done ` to notify the user its
132132waiting for new inputs. This results in ` n+1 ` lines of output for the input of ` n ` . Still predicting the number of lines
133- for a given input can be challenging, so the ` pympipool .ShellExecutor` class also provides the option to wait until a
133+ for a given input can be challenging, so the ` executorlib .ShellExecutor` class also provides the option to wait until a
134134specific pattern is found in the output using the ` stop_read_pattern ` :
135135``` python
136- from pympipool import ShellExecutor
136+ from executorlib import ShellExecutor
137137
138138with ShellExecutor([" python" " count.py" universal_newlines = True ) as exe:
139139 future_pattern = exe.submit(string_input = " 4" stop_read_pattern = " done"
0 commit comments