Add integration tests for application notebooks (#619)

* Add integration tests for application notebooks

* remove pmix

* remove constraints

* fix notebook names

* increase time out

* remove ASE command

* Disable qe cluster test

* fix pseudo potential directory

* relative to working directory

* try absolute path

* Update pipeline.yml

* Test new debugging capability

* use flux run to call pw.x

* try p2.x with mpi

* mpi takes too long

* enable 3rd test

* always nest flux

* no more nesting

* no more nesting

* merge notebooks

* fix pseudo potential directory

* add quantum espresso

* Short introduction

* Add some more documentation

* Finish up documentation

Author: jan-janssen

.ci_support/environment-integration.yml

@@ -0,0 +1,21 @@
+channels:
+- conda-forge
+dependencies:
+- jupyter
+- papermill
+- numpy
+- openmpi
+- cloudpickle =3.1.1
+- mpi4py
+- pyzmq =26.3.0
+- flux-core =0.59.0
+- versioneer =0.28
+- h5py
+- matplotlib =3.10.0
+- networkx =3.4.2
+- pygraphviz =1.14
+- pysqa =0.2.4
+- ipython =9.0.2
+- atomistics =0.2.4
+- qe =7.2
+- gpaw =24.6.0

.github/workflows/pipeline.yml

@@ -173,6 +173,33 @@ jobs:
           flux start papermill notebooks/3-hpc-job.ipynb notebooks/3-hpc-job-out.ipynb -k python3
           papermill notebooks/4-developer.ipynb notebooks/4-developer-out.ipynb -k python3
 
+  notebooks_integration:
+    needs: [black]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Merge Notebook environment
+        run: |
+          echo -e "channels:\n  - conda-forge\n" > .condarc
+      - name: Setup Mambaforge
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          python-version: "3.12"
+          miniforge-version: latest
+          condarc-file: .condarc
+          environment-file: .ci_support/environment-integration.yml
+      - name: Install
+        shell: bash -l {0}
+        run: |
+          pip install versioneer[toml]==0.29
+          pip install . --no-deps --no-build-isolation
+      - name: Notebooks
+        shell: bash -l {0}
+        timeout-minutes: 20
+        run: |
+          flux start papermill notebooks/5-1-gpaw.ipynb notebooks/5-1-gpaw-out.ipynb -k python3
+          flux start papermill notebooks/5-2-quantum-espresso.ipynb notebooks/5-2-quantum-espresso-out.ipynb -k python3
+
   unittest_flux_mpich:
     needs: [black]
     runs-on: ubuntu-latest

README.md

@@ -139,6 +139,9 @@ as hierarchical job scheduler within the allocations.
   * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
   * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
   * [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux)
+* [Application](https://executorlib.readthedocs.io/en/latest/application.html)
+  * [GPAW](https://executorlib.readthedocs.io/en/latest/5-1-gpaw.html)
+  * [Quantum Espresso](https://executorlib.readthedocs.io/en/latest/5-2-quantum-espresso.html)
 * [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
   * [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
   * [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)

docs/_toc.yml

@@ -5,6 +5,10 @@ chapters:
 - file: 1-single-node.ipynb
 - file: 2-hpc-cluster.ipynb
 - file: 3-hpc-job.ipynb
+- file: application.md
+  sections:
+    - file: 5-1-gpaw.ipynb
+    - file: 5-2-quantum-espresso.ipynb
 - file: trouble_shooting.md
 - file: 4-developer.ipynb
 - file: api.rst

docs/application.md

@@ -0,0 +1,10 @@
+# Application
+While `executorlib` is designed to up-scale any Python function for high performance computing (HPC), it was initially
+developed to accelerate atomistic computational materials science simulation. To demonstrate the usage of `executorlib`
+in the context of atomistic simulation, it is combined with [atomistics](https://atomistics.readthedocs.io/) and the
+[atomic simulation environment (ASE)](https://wiki.fysik.dtu.dk/ase/) to calculate the bulk modulus with two density
+functional theory simulation codes [gpaw](https://gpaw.readthedocs.io/index.html) and [quantum espresso](https://www.quantum-espresso.org).
+The bulk modulus is calculated by uniformly deforming a supercell of atoms and measuring the change in total energy 
+during compression and elongation. The first derivative of this curve is the pressure and the second derivative is 
+proportional to the bulk modulus. Other material properties like the heat capacity, thermal expansion or thermal conductivity
+can be calculated in similar ways following the [atomistics](https://atomistics.readthedocs.io/) documentation.