Status of queue jobs in HPC

[rajorshichat]

Hello,

I am trying to install pyiron in a HPC system to submit jobs from my local computer. I am following the instructions in the documentation and am trying to test the installation in the remote HPC. While jobs with run_mode=modal runs flawlessly, I see a problem in the status of jobs with run_mode=queue.

Once the jobs are submitted, the status in the job table shows submitted. This status remains the same even after the job is finished and all the required output files are generated in the file system. However, the status under job['status'] shows initialized. Also the output group is not available in the HDF5 job file although the output files are generated. I run the following lines of code -

pr = Project("test_lammps")
lmp = pr.create.job.Lammps(job_name='my_job_6')
structure = pr.create.structure.bulk('Al', cubic=True).repeat(3)
lmp.structure = structure
pot = lmp.list_potentials()[0]
lmp.potential = pot
lmp.calc_md()
lmp.run(run_mode="queue", delete_existing_job=True)

lmp
{'groups': ['executable', 'input'], 'nodes': ['DICT_VERSION', 'HDF_VERSION', 'NAME', 'OBJECT', 'TYPE', 'VERSION', 'job_id', 'server', 'status']}

lmp=pr.load('my_job_6')
lmp['status']
'initialized'

Inspection of the file system reveals that unlike modal jobs, tar files are not being generated for queue jobs. Am I getting something wrong here ? I thank you for your suggestions.

[jan-janssen]

Hi @rajorshichat ,

We typically recommend setting up the remote installation in three steps:
https://pyiron.readthedocs.io/en/latest/source/installation_hpc_config.html#submit-to-remote-hpc

1. Install pyiron on the login node of the remote cluster and test if you a run jobs in modal mode.
2. Submit from the login node of the remote cluster to the queuing system of the cluster.
3. Submit from a workstation to the remote cluster using the SSH tunnel.

From your description I am a bit confused, did step 2 work for you?

Best,

Jan

[rajorshichat]

Hi @jan-janssen ,

Step 1 works perfectly for me. I am facing problems in step 2. When jobs are submitted to the queueing system of the cluster from the login node, I see a mismatch in the job status and the output group is missing in the HDF5 file although the required output files are generated.

[jan-janssen]

• Does the submission to the queuing system work? You can use the pysqa example to check if the queue.yaml file and the different submission files are configured correctly: https://pysqa.readthedocs.io/en/latest/example_config.html#submit-job-to-queue
• Is there an error.out file in the working directory of the LAMMPS calculation? This file could contain an error message from the queuing system.
• How is the ~/.pyiron configuration defined on the HPC? Please remove your password when posting the configuration file. Did you disable the database? DISABLE_DATABASE https://pyiron.readthedocs.io/en/latest/source/installation_advanced_config.html#custom-executables-and-parameter-files

[rajorshichat]

Hi @jan-janssen ,

• I think that the submission to the queuing system works well. A run_queue.shfile is created in the LAMMPS working directory (attached) which is same as the <queue_name>.shfile in the queues/directory. I used the pr.queue_table() function to check the submission and it seems to be working correctly:

jobid     user          jobname   status                                                        working_directory
0  332141  ra74san     run_queue.sh  running  /dss/dsshome1/0D/ra74san/pyiron_test/test_lammps/my_job_hdf5/my_job

• The error.out file contains some warnings about loading some modules. I have attached the file.

• The DISABLE_DATABASE was set to its default value. I set it to TRUE. The current configuration is as follows:

{'user': 'pyiron', 'resource_paths': ['/dss/dsshome1/0D/ra74san/pyiron/resources', '/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/share/pyiron', '/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/share/pyiron'], 'project_paths': [], 'connection_timeout': 60, 'sql_connection_string': None, 'sql_table_name': 'jobs_pyiron', 'sql_view_connection_string': None, 'sql_view_table_name': None, 'sql_view_user': None, 'sql_view_user_key': None, 'sql_file': '/dss/dsshome1/0D/ra74san/pyiron.db', 'sql_host': None, 'sql_type': 'SQLite', 'sql_user_key': None, 'sql_database': None, 'project_check_enabled': False, 'disable_database': True, 'credentials_file': None, 'write_work_dir_warnings': True, 'config_file_permissions_warning': True}

However, this does not seem to solve the problem. The output group is still missing in the HDF5 file and the job status in the job table and the HDF5 file shows initialized.

error.txt
run_queue.txt

[rajorshichat]

For completeness of information, I have used the attached script to install pyiron in the HPC as well as in my local system.

environment.txt

[jan-janssen]

The output is only written to the HDF5 file after the LAMMPS calculation is completed. To clarify, even after the job finished and is no longer listed in pr.queue_table() the output in the HDF5 file remains empty in your case, correct? Did you try pr.refresh_job_status() and afterwards calling pr.job_table()? Is the job still listed as initialized?

[jan-janssen]

The run_queue.sh script you attached looks a bit different from what pyiron expects, is this the script you created or is this the script that pyiron copied to the working directory? Can you show me the <queue_name>.sh script? I would expect something like:
https://pyiron.readthedocs.io/en/latest/source/installation_hpc_config.html#hpc-cluster-with-postgresql-database-and-jupyterhub

#!/bin/bash
#SBATCH --job-name={{job_name}}
#SBATCH --output=time.out
#SBATCH --error=error.out
#SBATCH --partition=cm4_inter
#SBATCH --nodes={{cores // 64}}
#SBATCH --ntasks={{cores}}
#SBATCH --time={{run_time_max // 60}}
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=R.Chattopadhyay@lmu.de
#SBATCH --get-user-env=L

{{command}}

The important part is the usage of the {{ }} to define the variables which pyiron can change at the time of submission. For example the{{command}} entry is going to be replaced by a call to load the pyiron object from the HDF5 file before calling the LAMMPS executable, this allows pyiron to monitor the execution of LAMMPS just like we do it when LAMMPS is running on the login node. In addition, the option #SBATCH --export=NONE might disable your Python environment, so I replaced it with #SBATCH --get-user-env=L.

[rajorshichat]

Hi @jan-janssen ,

Now I understand the issue. I missed out {{command}} in the template <queue_name.sh> script. I have now added this and the submission runs flawlessly. The issue with job status is also resolved.

However, now I face another issue. How to set the {{cores}}variable ? I find that this is always replaced by the cores_min entry in the queue configuration file. I tried setting job.server.cores to the desired number of cores. However this seems to be for multicore executables for modal jobs. For example -

queues:
  submit: {cores_max: 64, cores_min: 1, cores: 2, run_time_max: 1440, script: submit.sh}

replaces {{cores}} with 1 whereas,

queues:
  submit: {cores_max: 64, cores_min: 2, run_time_max: 1440, script: submit.sh}

replaces {{cores}} with 2.

Also, one question which is not related to this topic. Suppose I want to distribute one of my pyiron projects. In this case the project directory as well as the job table have to be distributed, right ? Is there also a function that can be used to create something like a tarball file for the project ?

Thank you once again for your help and suggestions.

[jan-janssen]

Can you share the code you use to submit the queue? Typically we would do something like:

job.server.queue = "submit"
job.server.cores = 64
job.run()

Also, one question which is not related to this topic. Suppose I want to distribute one of my pyiron projects. In this case the project directory as well as the job table have to be distributed, right ? Is there also a function that can be used to create something like a tarball file for the project ?

Yes, we provide a publication template for pyiron to help users with publishing their data:
https://github.com/pyiron/pyiron-publication-template

It basically uses the pr.pack() routine of the pyiron project object.

[rajorshichat]

Hi @jan-janssen,

Here is the code to submit the queue,

>>>job.server.run_mode
'modal'
job.server.queue = 'submit'
2025-08-08 08:49:12,709 - pyiron_log - WARNING - Updated the number of cores to: 2
>>> job.executable
'/p/home/jusers/chattopadhyay1/juwels/pyiron/resources/lammps/bin/run_lammps_2018.03.16.sh'
>>> job.server.cores = 3
>>> job.calc_md()
>>> job.server.run_mode
'queue'
>>> job.executable
'/p/home/jusers/chattopadhyay1/juwels/pyiron/resources/lammps/bin/run_lammps_2018.03.16.sh'
>>> job.run()
2025-08-08 08:50:34,053 - pyiron_log - WARNING - Updated the number of cores to: 2
/p/project1/hydrothermal2/rajorshi/micromamba/envs/MLP/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py:1446: RuntimeWarning: No multi core executable found falling back to the single core executable.
  warnings.warn(
The job my_job_2 was saved and received the ID: my_job_2
Queue system id:  12069531

The queue configuration file is,

queue_type: SLURM
queue_primary: submit
queues:
  submit: {cores_max: 64, cores_min: 2, run_time_max: 1440, script: submit.sh}

The template job script is,

#!/bin/bash -x
#SBATCH --nodes={{cores}}
#SBATCH --ntasks-per-node=48
#SBATCH --output=log.lammps
#SBATCH --error=error.out
#SBATCH --time=00:05:00
#SBATCH --partition=batch
#SBATCH --account=hydrothermal2
#SBATCH --get-user-env=L

{{command}}

module purge

module load Stages/2025  GCC/13.3.0  ParaStationMPI/5.10.0-1

srun /p/project1/hydrothermal2/rajorshi/micromamba/envs/MLP/bin/lmp_mpi -in control.inp

The job specific submission script is:

#!/bin/bash -x
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=48
#SBATCH --output=log.lammps
#SBATCH --error=error.out
#SBATCH --time=00:05:00
#SBATCH --partition=batch
#SBATCH --account=hydrothermal2
#SBATCH --get-user-env=L

python -m pyiron_base.cli wrapper -p /p/project1/hydrothermal2/rajorshi/pyiron_projects/test_lammps/my_job_2_hdf5/my_job_2 -f /p/project1/hydrothermal2/rajorshi/pyiron_projects/test_lammps/my_job_2.h5/my_job_2

module purge

module load Stages/2025  GCC/13.3.0  ParaStationMPI/5.10.0-1

srun /p/project1/hydrothermal2/rajorshi/micromamba/envs/MLP/bin/lmp_mpi -in control.inp

[jan-janssen]

The template should be only:

#!/bin/bash -x
#SBATCH --nodes={{cores}}
#SBATCH --ntasks-per-node=48
#SBATCH --output=log.lammps
#SBATCH --error=error.out
#SBATCH --time=00:05:00
#SBATCH --partition=batch
#SBATCH --account=hydrothermal2
#SBATCH --get-user-env=L

{{command}}

Meaning the whole following block should be removed:

module purge

module load Stages/2025  GCC/13.3.0  ParaStationMPI/5.10.0-1

srun /p/project1/hydrothermal2/rajorshi/micromamba/envs/MLP/bin/lmp_mpi -in control.inp

The idea is that these parts are moved to the executable script:
/p/home/jusers/chattopadhyay1/juwels/pyiron/resources/lammps/bin/run_lammps_2018.03.16_mpi.sh

#!/bin/bash
module purge
module load Stages/2025  GCC/13.3.0  ParaStationMPI/5.10.0-1

srun -n $1 /p/project1/hydrothermal2/rajorshi/micromamba/envs/MLP/bin/lmp_mpi -in control.inp

The way this works in pyiron is that pyiron replaces the {{command}} variable with the python command to load the pyiron object, as you showed in your example:

python -m pyiron_base.cli wrapper -p /p/project1/hydrothermal2/rajorshi/pyiron_projects/test_lammps/my_job_2_hdf5/my_job_2 -f /p/project1/hydrothermal2/rajorshi/pyiron_projects/test_lammps/my_job_2.h5/my_job_2

Then this python process creates a subprocess to call /p/home/jusers/chattopadhyay1/juwels/pyiron/resources/lammps/bin/run_lammps_2018.03.16_mpi.sh in the working directory:
/p/project1/hydrothermal2/rajorshi/pyiron_projects/test_lammps/my_job_2.h5/my_job_2

[rajorshichat]

Hi @jan-janssen

Thank you for your suggestions. Things work flawlessly now. But I have to explicitly choose the version every time using job.executable.version = '<version>_mpi' because I have also put a bash file containing the serial version in the same directory and by default pyiron chooses the serial version. Is there a work around for this ?

Also, I am trying to set up pyiron in another HPC system in our Institute. When I try to submit a queue, I get the following error -

File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/utils/deprecate.py", line 171, in decorated
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py", line 763, in run
    self._run_if_new(debug=debug)
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py", line 1281, in _run_if_new
    run_job_with_status_initialized(job=self, debug=debug)
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py", line 92, in run_job_with_status_initialized
    job.run()
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/utils/deprecate.py", line 171, in decorated
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py", line 765, in run
    self._run_if_created()
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py", line 1292, in _run_if_created
    return run_job_with_status_created(job=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py", line 139, in run_job_with_status_created
    job.run_if_scheduler()
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/generic.py", line 808, in run_if_scheduler
    return run_job_with_runmode_queue(job=self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pyiron_base/jobs/job/runfunction.py", line 400, in run_job_with_runmode_queue
    que_id = state.queue_adapter.submit_job(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pysqa/queueadapter.py", line 172, in submit_job
    return self._adapter.submit_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pysqa/utils/basic.py", line 181, in submit_job
    return self._commands.get_job_id_from_output(out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dss/dsshome1/0D/ra74san/miniconda3/envs/potentials/lib/python3.11/site-packages/pysqa/wrapper/slurm.py", line 36, in get_job_id_from_output
    return int(queue_submit_output.splitlines()[-1].rstrip().lstrip().split()[-1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '72539;cm4'

I changed int(queue_submit_output.splitlines()[-1].rstrip().lstrip().split()[-1])to int(queue_submit_output.splitlines()[-1].rstrip().lstrip().split()[-1].split(';')[0]) in the .../lib/python3.11/site-packages/pysqa/wrapper/slurm.pyscript and problem seems to be solved. But I am a bit afraid if this change breaks some other part of the code. Is it possible for you to make a comment on this ?

[jan-janssen]

But I have to explicitly choose the version every time using job.executable.version = '<version>_mpi' because I have also put a bash file containing the serial version in the same directory and by default pyiron chooses the serial version. Is there a work around for this ?

By default pyiron automatically shifts from the serial version to the MPI parallel version once the job.server.cores are changed. Are you sure the version name is exactly the same? Can you print job.executable before you set the executable just to make sure it already has the right version and is only missing the <version>_mpi part?

Also, I am trying to set up pyiron in another HPC system in our Institute. When I try to submit a queue, I get the following error -

ValueError: invalid literal for int() with base 10: '72539;cm4'

I changed int(queue_submit_output.splitlines()[-1].rstrip().lstrip().split()[-1])to int(queue_submit_output.splitlines()[-1].rstrip().lstrip().split()[-1].split(';')[0]) in the .../lib/python3.11/site-packages/pysqa/wrapper/slurm.pyscript and problem seems to be solved. But I am a bit afraid if this change breaks some other part of the code. Is it possible for you to make a comment on this ?

The challenge is that your job scheduler returns the job ID 72539 and some additional information ;cm4. So pyiron does not know how to handle this. The default is for the job scheduler to just return the job ID. Still this change also works on other systems which do not have the additional information. So I opened a pull request to include this change in the next version of pysqa: pyiron/pysqa#449

In particular as we had a similar case with a cluster in Belgium https://github.com/pyiron/pysqa/blob/main/pysqa/wrapper/gent.py#L38

By the way, do the other pyiron commands work, like getting the number of job IDs and so on? Because the people in Belgium basically had to add the information to every command.

[rajorshichat]

Hi @jan-janssen,

Okay I understand now. There were 3 different executable files in my resourcedirectory.

job.executable.list_executables()
['2018.03.03', '2020.03.03', '2020.03.03_mpi']

When I removed the 2018.03.03 version from the resource directory or selected job.executable.version='2020.03.03', pyiron switched to the mpi version automatically when job.server.cores was changed.

All my doubts seem to be solved for the time being. I thank you once again for your suggestions and help.