Skip to content

Leowerneck/multiprocessing macos #57

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Leowerneck/multiprocessing macos #57

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89b33a4
Tutorial-ETK_thorn-BaikalETK.ipynb: updated multiprocessing import fo…
leowerneck Oct 6, 2022
666686a
Tutorial-ETK_thorn-WaveToyNRPy.ipynb: updated multiprocessing import …
leowerneck Oct 6, 2022
80f331f
Updated multiprocessing import and error check in various notebooks
leowerneck Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 70 additions & 56 deletions Tutorial-ETK_thorn-BaikalETK.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,70 +349,70 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...\n",
"\n",
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for BSSN constraints (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for BSSN constraints (Cartesian coords)...Generating symbolic expressions for BSSN constraints (Cartesian coords)...\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...Generating symbolic expressions for BSSN constraints (Cartesian coords)...\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords)...\n",
"Generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords)...\n",
"Finished generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords) in 0.1 seconds.\n",
"Finished generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords) in 0.0 seconds.\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 0.3 seconds. Next up: C codegen...Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 0.3 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 0.3 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=8) (Cartesian coords)...\n",
"Generating C code for 3-Ricci tensor (FD order=2) (Cartesian coords)...\n",
"Generating C code for 3-Ricci tensor (FD order=6) (Cartesian coords)...\n",
"Finished generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords) in 0.2 seconds.\n",
"Finished generating C code for Enforcing det(gammabar)=det(gammahat) constraint (Cartesian coords) in 0.2 seconds.\n",
"Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...\n",
"\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 0.4 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=8) (Cartesian coords)...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 0.4 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 1.3 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=2) (Cartesian coords)...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 1.4 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 0.5 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 1.2 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 0.5 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 1.5 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 0.6 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=8) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 0.6 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=6) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 0.6 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=6) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 0.7 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 0.8 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=2) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 0.8 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 3.8 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 1.0 seconds. Next up: C codegen...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 3.8 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=2) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 5.3 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=4) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 5.4 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=2) (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 1.2 seconds. Next up: C codegen...\n",
"Finished generating C code for BSSN constraints (FD order=2) (Cartesian coords) in 77.5 seconds.\n",
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...\n",
"Finished generating C code for BSSN constraints (FD order=4) (Cartesian coords) in 79.1 seconds.\n",
"Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 2.2 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=4) (Cartesian coords)...\n",
"Finished generating C code for BSSN constraints (FD order=4) (Cartesian coords) in 16.5 seconds.\n",
"Finished generating C code for BSSN constraints (FD order=6) (Cartesian coords) in 17.0 seconds.\n",
"Finished generating C code for BSSN constraints (FD order=2) (Cartesian coords) in 16.6 seconds.\n",
"Finished generating C code for BSSN constraints (FD order=4) (Cartesian coords) in 16.9 seconds.\n",
"Finished generating C code for BSSN constraints (FD order=8) (Cartesian coords) in 18.5 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=4) (Cartesian coords) in 37.9 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=6) (Cartesian coords) in 39.0 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=8) (Cartesian coords) in 41.5 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=2) (Cartesian coords) in 49.0 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=4) (Cartesian coords) in 49.8 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=2) (Cartesian coords) in 55.4 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords) in 55.6 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=6) (Cartesian coords) in 56.1 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords) in 56.1 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=8) (Cartesian coords) in 57.3 seconds.\n"
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 1.4 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=6) (Cartesian coords)...\n",
"Finished generating C code for BSSN constraints (FD order=4) (Cartesian coords) in 71.4 seconds.\n",
"Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 1.1 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=6) (Cartesian coords)...\n",
"Finished generating C code for BSSN_RHSs (FD order=4) (Cartesian coords) in 172.7 seconds.\n",
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 2.0 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=6) (Cartesian coords)...\n",
"Finished generating C code for BSSN_RHSs (FD order=2) (Cartesian coords) in 220.2 seconds.\n",
"Generating symbolic expressions for BSSN_RHSs (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN_RHSs (Cartesian coords) in 1.3 seconds. Next up: C codegen...\n",
"Generating C code for BSSN_RHSs (FD order=8) (Cartesian coords)...\n",
"Finished generating C code for BSSN_RHSs (FD order=4) (Cartesian coords) in 222.3 seconds.\n",
"Generating symbolic expressions for 3-Ricci tensor (Cartesian coords)...\n",
"Finished generating symbolic expressions for 3-Ricci tensor (Cartesian coords) in 1.1 seconds. Next up: C codegen...\n",
"Generating C code for 3-Ricci tensor (FD order=8) (Cartesian coords)...\n",
"Finished generating C code for BSSN constraints (FD order=6) (Cartesian coords) in 66.8 seconds.\n",
"Generating symbolic expressions for BSSN constraints (Cartesian coords)...\n",
"Finished generating symbolic expressions for BSSN constraints (Cartesian coords) in 2.1 seconds. Next up: C codegen...\n",
"Generating C code for BSSN constraints (FD order=8) (Cartesian coords)...\n",
"Finished generating C code for 3-Ricci tensor (FD order=2) (Cartesian coords) in 245.4 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords) in 246.1 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=4) (Cartesian coords) in 246.5 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=6) (Cartesian coords) in 168.2 seconds.\n",
"Finished generating C code for BSSN constraints (FD order=8) (Cartesian coords) in 40.0 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=6) (Cartesian coords) in 164.4 seconds.\n",
"Finished generating C code for BSSN_RHSs (FD order=8) (Cartesian coords) in 108.0 seconds.\n",
"Finished generating C code for 3-Ricci tensor (FD order=8) (Cartesian coords) in 138.0 seconds.\n"
]
}
],
Expand Down Expand Up @@ -465,17 +465,31 @@
"NRPyEnvVars = []\n",
"raised_exception = False\n",
"try:\n",
" if os.name == 'nt':\n",
"\n",
" import platform\n",
" os_name = platform.system() # Linux, Darwin, or Windows\n",
" # Step 2.d: Import the multiprocessing module.\n",
" if os_name == 'Windows':\n",
" # It's a mess to get working in Windows, so we don't bother. :/\n",
" # https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac\n",
" raise Exception(\"Parallel codegen currently not available in certain environments, e.g., Windows\")\n",
"\n",
" # Step 2.d: Import the multiprocessing module.\n",
" import multiprocessing\n",
" elif os_name == \"Darwin\":\n",
" # On Mac OS multiprocessing does not work as expected in Jupyter\n",
" # notebooks, so we use multiprocess instead\n",
" try:\n",
" import multiprocess as mp\n",
" except:\n",
" error = \"Could not load multiprocess. Please install it (e.g., `$ pip install multiprocess`) and try running this notebook again.\"\n",
" raise ImportError(error)\n",
" elif os_name == \"Linux\":\n",
" # On Linux using multiprocessing does not seem to be a problem\n",
" import multiprocessing as mp\n",
" else:\n",
" raise Exception(f\"Unknown system {os_name}. Reverting to serial code generation.\")\n",
"\n",
" # Step 2.e: Evaluate list of functions in parallel if possible;\n",
" # otherwise fallback to serial evaluation:\n",
" pool = multiprocessing.Pool()\n",
" pool = mp.Pool()\n",
" NRPyEnvVars.append(pool.map(master_func, range(len(BSSN_funcs))))\n",
" pool.terminate()\n",
" pool.join()\n",
Expand Down Expand Up @@ -1979,7 +1993,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
"version": "3.10.7"
}
},
"nbformat": 4,
Expand Down
62 changes: 35 additions & 27 deletions Tutorial-ETK_thorn-WaveToyNRPy.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,26 +409,23 @@
"name": "stdout",
"output_type": "stream",
"text": [
"***************************************\n",
"Starting parallel C kernel codegen...\n",
"***************************************\n",
"Generating symbolic expressions for WaveToy RHSs...\n",
"Generating symbolic expressions for WaveToy RHSs...Generating symbolic expressions for WaveToy RHSs...\n",
"\n",
"Generating symbolic expressions for WaveToy RHSs...\n",
"Generating symbolic expressions for WaveToy RHSs...\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.013369560241699219 seconds.\n",
"Generating C code for WaveToy RHSs (FD_order=2) in Cartesian coordinates.\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.017975807189941406 seconds.\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.019555091857910156 seconds.Generating C code for WaveToy RHSs (FD_order=4) in Cartesian coordinates.\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.03818702697753906 seconds.\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.04004979133605957 seconds.(BENCH) Finished WaveToy RHS symbolic expressions in 0.04361605644226074 seconds.(BENCH) Finished WaveToy RHS symbolic expressions in 0.04396963119506836 seconds.Generating C code for WaveToy RHSs (FD_order=6) in Cartesian coordinates.\n",
"\n",
"\n",
"\n",
"Generating C code for WaveToy RHSs (FD_order=4) in Cartesian coordinates.Generating C code for WaveToy RHSs (FD_order=2) in Cartesian coordinates.Generating C code for WaveToy RHSs (FD_order=8) in Cartesian coordinates.\n",
"\n",
"Generating C code for WaveToy RHSs (FD_order=8) in Cartesian coordinates.\n",
"(BENCH) Finished WaveToy RHS symbolic expressions in 0.02181220054626465 seconds.\n",
"Generating C code for WaveToy RHSs (FD_order=6) in Cartesian coordinates.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=2) in 0.07765388488769531 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=4) in 0.08968853950500488 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=6) in 0.14511585235595703 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=8) in 0.1823718547821045 seconds.\n",
"(BENCH) Finished C kernel codegen for WaveToy in 0.3171041011810303 seconds.\n"
"\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=2) in 0.1929771900177002 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=4) in 0.28936314582824707 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=6) in 0.39238786697387695 seconds.\n",
"(BENCH) Finished WaveToy_RHS C codegen (FD_order=8) in 0.5159783363342285 seconds.\n",
"(BENCH) Finished C kernel codegen for WaveToy in 0.8135979175567627 seconds.\n"
]
}
],
Expand Down Expand Up @@ -468,15 +465,26 @@
"start = time.time()\n",
"if __name__ == \"__main__\":\n",
" try:\n",
" if os.name == 'nt':\n",
" # Windows & Jupyter multiprocessing do not mix, so we run in serial on Windows.\n",
" # Here's why: https://stackoverflow.com/questions/45719956/python-multiprocessing-in-jupyter-on-windows-attributeerror-cant-get-attribut\n",
" raise Exception(\"Parallel codegen currently not available in Windows\")\n",
" # Step 3.d.ii: Import the multiprocessing module.\n",
" import multiprocessing\n",
" print(\"***************************************\")\n",
" print(\"Starting parallel C kernel codegen...\")\n",
" print(\"***************************************\")\n",
" import platform\n",
" os_name = platform.system() # Linux, Darwin, or Windows\n",
" # Step 2.d: Import the multiprocessing module.\n",
" if os_name == 'Windows':\n",
" # It's a mess to get working in Windows, so we don't bother. :/\n",
" # https://medium.com/@grvsinghal/speed-up-your-python-code-using-multiprocessing-on-windows-and-jupyter-or-ipython-2714b49d6fac\n",
" raise Exception(\"Parallel codegen currently not available in certain environments, e.g., Windows\")\n",
" elif os_name == \"Darwin\":\n",
" # On Mac OS multiprocessing does not work as expected in Jupyter\n",
" # notebooks, so we use multiprocess instead\n",
" try:\n",
" import multiprocess as mp\n",
" except:\n",
" error = \"Could not load multiprocess. Please install it (e.g., `$ pip install multiprocess`) and try running this notebook again.\"\n",
" raise ImportError(error)\n",
" elif os_name == \"Linux\":\n",
" # On Linux using multiprocessing does not seem to be a problem\n",
" import multiprocessing as mp\n",
" else:\n",
" raise Exception(f\"Unknown system {os_name}. Reverting to serial code generation.\")\n",
"\n",
" # Step 3.d.iii: Define master function for parallelization.\n",
" # Note that lambdifying this doesn't work in Python 3\n",
Expand All @@ -486,7 +494,7 @@
"\n",
" # Step 3.d.iv: Evaluate list of functions in parallel if possible;\n",
" # otherwise fallback to serial evaluation:\n",
" pool = multiprocessing.Pool() #processes=len(paramslist))\n",
" pool = mp.Pool() #processes=len(paramslist))\n",
" NRPyEnvVars.append(pool.map(master_func,range(len(paramslist))))\n",
" pool.terminate()\n",
" pool.join()\n",
Expand Down Expand Up @@ -1566,7 +1574,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
"version": "3.10.7"
}
},
"nbformat": 4,
Expand Down
Loading