the record.dpgen did't work right and skip the train procedure

[maoxinxina]

when I set the record.dpgen from 1 0, I think the train procedure would begin, however, the file in the 00.train is 000 001 002 003 copied graph.000.pb graph.001.pb graph.002.pb graph.003.pb, the software didn't work right and skip the train procedure. the content of slurm is
gmx: error while loading shared libraries: libcufft.so.11: cannot open shared object file: No such file or directory
/data/share/software/dpgen/lib/python3.8/site-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
INFO:dpgen:-------------------------iter.000001 task 02--------------------------
INFO:dpgen:-------------------------iter.000001 task 03--------------------------
INFO:dpgen:-------------------------iter.000001 task 04--------------------------
NOTE: Some configuration directories are not set up yet:
/data/share/user10/.gromacswrapper
/data/share/user10/.gromacswrapper/qscripts
/data/share/user10/.gromacswrapper/templates
NOTE: You can create the configuration file and directories with:
>>> import gromacs
>>> gromacs.config.setup()
DeepModeling

Version: 0.10.6
Date: Jul-11-2022
Path: /data/share/software/dpgen/lib/python3.8/site-packages/dpgen

Dependency

 numpy     1.22.3   /data/share/software/dpgen/lib/python3.8/site-packages/numpy
dpdata      0.2.7   /data/share/software/dpgen/lib/python3.8/site-packages/dpdata

pymatgen unknown version or path
monty 2022.4.26 /data/share/software/dpgen/lib/python3.8/site-packages/monty
ase 3.22.1 /data/share/software/dpgen/lib/python3.8/site-packages/ase
paramiko 2.8.1 /data/share/software/dpgen/lib/python3.8/site-packages/paramiko
custodian 2022.5.26 /data/share/software/dpgen/lib/python3.8/site-packages/custodian

Reference

Please cite:
Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E,
DP-GEN: A concurrent learning platform for the generation of reliable deep learning
based potential energy models, Computer Physics Communications, 2020, 107206.

Description

2023-07-28 14:15:10,691 - INFO : info:check_all_finished: False
2023-07-28 14:15:10,728 - INFO : job: 7689fdd7d3d01dc7fb23adc129b252662544ab88 submit; job_id is 807282
2023-07-28 14:15:10,773 - INFO : job: fed5637530f1ac54471b1a3a6c5f00a528b2d1e4 submit; job_id is 807359
2023-07-28 14:15:10,791 - INFO : job: 0c59843203f8f1ce4eb7e29aa99919b9a6c4ee50 submit; job_id is 807385
2023-07-28 14:16:12,891 - INFO : job: 0c59843203f8f1ce4eb7e29aa99919b9a6c4ee50 807385 finished
2023-07-28 14:17:43,474 - INFO : job: 7689fdd7d3d01dc7fb23adc129b252662544ab88 807282 finished
2023-07-28 14:18:13,676 - INFO : job: fed5637530f1ac54471b1a3a6c5f00a528b2d1e4 807359 finished
INFO:dpgen:-------------------------iter.000001 task 05--------------------------
INFO:dpgen:-------------------------iter.000001 task 06--------------------------
INFO:dpgen:system 001 candidate : 0 in 671 0.00 %
INFO:dpgen:system 001 failed : 0 in 671 0.00 %
INFO:dpgen:system 001 accurate : 671 in 671 100.00 %
INFO:dpgen:system 001 accurate_ratio: 1.0000 thresholds: 1.0000 and 1.0000 eff. task min and max -1 0 number of fp tasks: 0
INFO:dpgen:-------------------------iter.000001 task 07--------------------------
INFO:dpgen:-------------------------iter.000001 task 08--------------------------
INFO:dpgen:=============================iter.000002==============================
INFO:dpgen:-------------------------iter.000002 task 00--------------------------
INFO:dpgen:-------------------------iter.000002 task 01--------------------------
INFO:dpgen:-------------------------iter.000002 task 02--------------------------
INFO:dpgen:-------------------------iter.000002 task 03--------------------------
INFO:dpgen:-------------------------iter.000002 task 04--------------------------
2023-07-28 14:18:13,741 - INFO : info:check_all_finished: False
2023-07-28 14:18:13,782 - INFO : job: 13af79f12d423db3dad063b825bb2b483878cb6b submit; job_id is 1344977
2023-07-28 14:18:13,794 - INFO : job: 7f8065eee7b14beaf5489c39e0ce9ae714edf30c submit; job_id is 1345058
2023-07-28 14:22:16,873 - INFO : job: 7f8065eee7b14beaf5489c39e0ce9ae714edf30c 1345058 finished
2023-07-28 14:23:17,103 - INFO : job: 13af79f12d423db3dad063b825bb2b483878cb6b 1344977 finished
INFO:dpgen:-------------------------iter.000002 task 05--------------------------
INFO:dpgen:-------------------------iter.000002 task 06--------------------------
INFO:dpgen:system 000 candidate : 34 in 1010 3.37 %
INFO:dpgen:system 000 failed : 5 in 1010 0.50 %
INFO:dpgen:system 000 accurate : 971 in 1010 96.14 %
INFO:dpgen:system 000 accurate_ratio: 0.9614 thresholds: 1.0000 and 1.0000 eff. task min and max -1 10000 number of fp tasks: 34
INFO:dpgen:-------------------------iter.000002 task 07--------------------------
Traceback (most recent call last):
File "/data/share/software/dpgen/bin/dpgen", line 10, in
sys.exit(main())
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/main.py", line 185, in main
args.func(args)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/generator/run.py", line 3642, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/generator/run.py", line 3628, in run_iter
run_fp (ii, jdata, mdata)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/generator/run.py", line 3018, in run_fp
run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _vasp_check_fin,
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/generator/run.py", line 2985, in run_fp_inner
submission = make_submission(
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpgen/dispatcher/Dispatcher.py", line 360, in make_submission
resources = Resources.load_from_dict(mdata_resources)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dpdispatcher/submission.py", line 812, in load_from_dict
base.check_value(resources_dict, strict=False)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dargs/dargs.py", line 351, in check_value
self.traverse_value(value,
File "/data/share/software/dpgen/lib/python3.8/site-packages/dargs/dargs.py", line 290, in traverse_value
self._traverse_sub(value,
File "/data/share/software/dpgen/lib/python3.8/site-packages/dargs/dargs.py", line 309, in _traverse_sub
subarg.traverse(value,
File "/data/share/software/dpgen/lib/python3.8/site-packages/dargs/dargs.py", line 271, in traverse
key_hook(self, argdict, path)
File "/data/share/software/dpgen/lib/python3.8/site-packages/dargs/dargs.py", line 360, in _check_exist
raise ArgumentKeyError(path,
dargs.dargs.ArgumentKeyError: [at root location] key gpu_per_node is required in arguments but not found

the error is the first-principle calculation which does not matter. so I wonder how to correct the set.

[njzjz]

This is expected. All frames are accurate, and there is no new data compared to the last iteration, so it is not required to have a new model.

INFO:dpgen:system 001 candidate : 0 in 671 0.00 %
INFO:dpgen:system 001 failed : 0 in 671 0.00 %
INFO:dpgen:system 001 accurate : 671 in 671 100.00 %

[maoxinxina]

May be my description is not clear,here the iter002gives some candidates.
2023-08-01 14:01:48,057 - INFO : -------------------------iter.000001 task 07--------------------------
2023-08-01 14:01:48,057 - INFO : -------------------------iter.000001 task 08--------------------------
2023-08-01 14:01:48,058 - INFO : =============================iter.000002==============================
2023-08-01 14:01:48,058 - INFO : -------------------------iter.000002 task 00--------------------------
2023-08-01 14:01:48,058 - INFO : -------------------------iter.000002 task 01--------------------------
2023-08-01 14:01:48,058 - INFO : -------------------------iter.000002 task 02--------------------------
2023-08-01 14:01:48,058 - INFO : -------------------------iter.000002 task 03--------------------------
2023-08-01 14:01:48,070 - INFO : -------------------------iter.000002 task 04--------------------------
2023-08-01 14:06:51,766 - INFO : -------------------------iter.000002 task 05--------------------------
2023-08-01 14:06:51,766 - INFO : -------------------------iter.000002 task 06--------------------------
2023-08-01 14:06:51,776 - INFO : system 000 candidate : 25 in 1010 2.48 %
2023-08-01 14:06:51,776 - INFO : system 000 failed : 0 in 1010 0.00 %
2023-08-01 14:06:51,776 - INFO : system 000 accurate : 985 in 1010 97.52 %
2023-08-01 14:06:51,778 - INFO : system 000 accurate_ratio: 0.9752 thresholds: 1.0000 and 1.0000 eff. task min and max -1 10000 number of fp tasks: 25

and I calculate the results and get the data.000 at the dictionary 02. Then I set the record.dpgen as 2 8. I hope the dpgen will train the add data. However, it skips the train prodecure.like
2023-08-01 14:06:51,805 - INFO : -------------------------iter.000002 task 07--------------------------
2023-08-01 15:20:17,564 - INFO : start running
2023-08-01 15:20:17,567 - INFO : continue from iter 002 task 08
2023-08-01 15:20:17,567 - INFO : =============================iter.000000==============================
2023-08-01 15:20:17,567 - INFO : =============================iter.000001==============================
2023-08-01 15:20:17,567 - INFO : =============================iter.000002==============================
2023-08-01 15:20:17,567 - INFO : =============================iter.000003==============================
2023-08-01 15:20:17,567 - INFO : -------------------------iter.000003 task 00--------------------------
2023-08-01 15:20:17,575 - INFO : -------------------------iter.000003 task 01--------------------------
2023-08-01 15:20:17,575 - INFO : -------------------------iter.000003 task 02--------------------------
2023-08-01 15:20:17,576 - INFO : -------------------------iter.000003 task 03--------------------------
2023-08-01 15:20:17,933 - INFO : -------------------------iter.000003 task 04--------------------------
2023-08-01 15:27:53,381 - INFO : -------------------------iter.000003 task 05--------------------------
2023-08-01 15:27:53,382 - INFO : -------------------------iter.000003 task 06--------------------------
2023-08-01 15:27:53,403 - INFO : system 000 candidate : 293 in 1610 18.20 %
2023-08-01 15:27:53,403 - INFO : system 000 failed : 11 in 1610 0.68 %
2023-08-01 15:27:53,403 - INFO : system 000 accurate : 1306 in 1610 81.12 %
2023-08-01 15:27:53,407 - INFO : system 000 accurate_ratio: 0.8112 thresholds: 1.0000 and 1.0000 eff. task min and max -1 10000 number of fp tasks: 293
2023-08-01 15:27:53,787 - INFO : system 001 candidate : 351 in 1771 19.82 %
2023-08-01 15:27:53,787 - INFO : system 001 failed : 11 in 1771 0.62 %

The file in the directory /run/iter.000003/00.train is
lrwxrwxrwx 1 user10 user10 30 Aug 1 15:20 000 -> ../../iter.000002/00.train/000
lrwxrwxrwx 1 user10 user10 30 Aug 1 15:20 001 -> ../../iter.000002/00.train/001
lrwxrwxrwx 1 user10 user10 30 Aug 1 15:20 002 -> ../../iter.000002/00.train/002
lrwxrwxrwx 1 user10 user10 30 Aug 1 15:20 003 -> ../../iter.000002/00.train/003
-rw-rw-r-- 1 user10 user10 0 Aug 1 15:20 copied
lrwxrwxrwx 1 user10 user10 19 Aug 1 15:20 graph.000.pb -> 000/frozen_model.pb
lrwxrwxrwx 1 user10 user10 19 Aug 1 15:20 graph.001.pb -> 001/frozen_model.pb
lrwxrwxrwx 1 user10 user10 19 Aug 1 15:20 graph.002.pb -> 002/frozen_model.pb
lrwxrwxrwx 1 user10 user10 19 Aug 1 15:20 graph.003.pb -> 003/frozen_model.pb
I hope the train will begin.

[maoxinxina]

I found the reason is : "model_devi_jobs": [
{ "sys_idx": [0,1,2], "traj_freq": 5,"skip_train":true,
"template":{ "lmp": "lmp/input.lammps", "plm": "lmp/input.plumed", "plm_path": "lmp/plmpath.pdb" },
"rev_mat":{ "lmp": {"V_NSTEPS": [5000], "V_TEMP": [490], "V_PRES": [20]}, "plm": {"V_TEMP": [490], "V_STRIDE": [5]} }},
skip_train:true, should be deleted. Thanks you.