Confusion about inconsistent training results with deepmdkit and dpgen

[warriors1523]

Hi, I tried to train the potential model using deepmdkit and dpgen separately, with the same training dataset and parameter files. Both potential models were used to run molecular dynamics simulations with the same starting configurations and parameter files. The results show that simulations with the potential model trained by deepmdkit are normal, while the potential model trained by dpgen causes unphysical configurations and crashes of the simulation box. The two potential models were used to perform molecular dynamics simulations three times, as shown in Figure E_time-2400.png.

To facilitate developer inspection,I have packaged the files in two zip files below, which includes the two types of training tasks and one molecular dynamics simulation task for each. Thank you for your help!

deepmdkit.zip
dpgen_00.train_000.zip

[njzjz]

I don't think it's related to DP-GEN. It is expected to have different results from different runs. See deepmodeling/deepmd-kit#2270.
It seems that two simulations gives different results in step 0. What is the expect result? Is it in the training set? Does the training set have large energies or forces?

[warriors1523]

How can I find these points? Is it the RMSE that is output at each step of the training model process?

[njzjz]

I think you should check the DFT potential energies. The bad point might have an outlier in the potential energy. You can follow Eq.1 in J. Chem. Theory Comput. 2023, 19, 4, 1261–1275. I also encourage you to find the reason for that energy.

[warriors1523]

Thank you for your reply！ I will try your solution. But why are the results inconsistent between the two types of training（deepmdkit and dpgen）, based on the four training sessions above?

[njzjz]

I check the two graph files you provide. They have different random seeds.

from deepmd.utils.convert import convert_pb_to_pbtxt
convert_pb_to_pbtxt("graph.pb", "graph.pbtxt")

grep seed graph.pbtxt 

dpgen:

        string_val: "{\"model\":{\"type_map\":[\"Li\",\"Nb\",\"O\"],\"descriptor\":{\"type\":\"se_e2_a\",\"sel\":[40,40,96],\"rcut_smth\":0.5,\"rcut\":7.0,\"neuron\":[25,50,100],\"resnet_dt\":false,\"axis_neuron\":12,\"seed\":84551992,\"activation_function\":\"tanh\",\"type_one_side\":false,\"precision\":\"default\",\"trainable\":true,\"exclude_types\":[],\"set_davg_zero\":false},\"fitting_net\":{\"neuron\":[240,240,240],\"resnet_dt\":true,\"seed\":3329491442,\"type\":\"ener\",\"numb_fparam\":0,\"numb_aparam\":0,\"activation_function\":\"tanh\",\"precision\":\"default\",\"trainable\":true,\"rcond\":0.001,\"atom_ener\":[]},\"data_stat_nbatch\":10,\"data_stat_protect\":0.01},\"learning_rate\":{\"type\":\"exp\",\"start_lr\":0.001,\"stop_lr\":3.51e-08,\"decay_steps\":2000,\"scale_by_worker\":\"linear\"},\"loss\":{\"start_pref_e\":0.02,\"limit_pref_e\":1,\"start_pref_f\":1000,\"limit_pref_f\":1,\"start_pref_v\":0.0,\"limit_pref_v\":0.0,\"type\":\"ener\",\"start_pref_ae\":0.0,\"limit_pref_ae\":0.0,\"start_pref_pf\":0.0,\"limit_pref_pf\":0.0,\"enable_atom_ener_coeff\":false},\"training\":{\"disp_file\":\"lcurve.out\",\"disp_freq\":200,\"save_freq\":10000,\"save_ckpt\":\"model.ckpt\",\"disp_training\":true,\"time_training\":true,\"profiling\":false,\"profiling_file\":\"timeline.json\",\"training_data\":{\"systems\":[\"../data.init/Data-crystal-linbo3/deepmd-1060K-crystal-mp-60atom-R3C\",\"../data.init/Data-crystal-linbo3/deepmd-1260K-crystal-mp-60atom-R3C\",\"../data.init/Data-crystal-linbo3/deepmd-1460K-crystal-mp-60atom-R-3C\",\"../data.init/Data-crystal-linbo3/deepmd-1660K-crystal-mp-60atom-R-3C\",\"../data.init/Data-crystal-linbo3/deepmd-1860K-crystal-mp-60atom-R-3C\",\"../data.init/Data-melt-linbo3/deepmd-1200K\",\"../data.init/Data-melt-linbo3/deepmd-1400K\",\"../data.init/Data-melt-linbo3/deepmd-1600K\",\"../data.init/Data-melt-linbo3/deepmd-1800K\",\"../data.init/Data-melt-linbo3/deepmd-2000K\",\"../data.init/iter0-11_crystal/Li12Nb12O36\",\"../data.init/deepmd-melting-60atoms/Li12Nb12O36\",\"../data.init/deepmd-melting-120atoms/Li24Nb24O72\",\"../data.iters/iter.000000/02.fp/data.004\",\"../data.iters/iter.000000/02.fp/data.005\",\"../data.iters/iter.000000/02.fp/data.006\",\"../data.iters/iter.000000/02.fp/data.007\",\"../data.iters/iter.000000/02.fp/data.008\",\"../data.iters/iter.000001/02.fp/data.004\",\"../data.iters/iter.000001/02.fp/data.005\",\"../data.iters/iter.000001/02.fp/data.006\",\"../data.iters/iter.000001/02.fp/data.007\",\"../data.iters/iter.000001/02.fp/data.008\",\"../data.iters/iter.000002/02.fp/data.004\",\"../data.iters/iter.000002/02.fp/data.005\",\"../data.iters/iter.000002/02.fp/data.006\",\"../data.iters/iter.000002/02.fp/data.007\",\"../data.iters/iter.000002/02.fp/data.008\",\"../data.iters/iter.000003/02.fp/data.004\",\"../data.iters/iter.000003/02.fp/data.005\",\"../data.iters/iter.000003/02.fp/data.006\",\"../data.iters/iter.000003/02.fp/data.007\",\"../data.iters/iter.000003/02.fp/data.008\",\"../data.iters/iter.000004/02.fp/data.004\",\"../data.iters/iter.000004/02.fp/data.005\",\"../data.iters/iter.000004/02.fp/data.006\",\"../data.iters/iter.000004/02.fp/data.007\",\"../data.iters/iter.000004/02.fp/data.008\",\"../data.iters/iter.000005/02.fp/data.004\",\"../data.iters/iter.000005/02.fp/data.005\",\"../data.iters/iter.000005/02.fp/data.006\",\"../data.iters/iter.000005/02.fp/data.007\",\"../data.iters/iter.000005/02.fp/data.008\",\"../data.iters/iter.000006/02.fp/data.004\",\"../data.iters/iter.000006/02.fp/data.005\",\"../data.iters/iter.000006/02.fp/data.006\",\"../data.iters/iter.000006/02.fp/data.007\",\"../data.iters/iter.000006/02.fp/data.008\",\"../data.iters/iter.000007/02.fp/data.004\",\"../data.iters/iter.000007/02.fp/data.005\",\"../data.iters/iter.000007/02.fp/data.006\",\"../data.iters/iter.000007/02.fp/data.007\",\"../data.iters/iter.000007/02.fp/data.008\",\"../data.iters/iter.000008/02.fp/data.002\",\"../data.iters/iter.000010/02.fp/data.002\",\"../data.iters/iter.000010/02.fp/data.003\",\"../data.iters/iter.000010/02.fp/data.004\",\"../data.iters/iter.000011/02.fp/data.005\"],\"batch_size\":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],\"set_prefix\":\"set\",\"auto_prob\":\"prob_sys_size\",\"sys_probs\":null},\"seed\":2433052497,\"numb_steps\":600000,\"validation_data\":null,\"enable_profiler\":false,\"tensorboard\":false,\"tensorboard_log_dir\":\"log\",\"tensorboard_freq\":1}}"

deepmd-kit

        string_val: "{\"model\":{\"type_map\":[\"Li\",\"Nb\",\"O\"],\"descriptor\":{\"type\":\"se_e2_a\",\"sel\":[40,40,96],\"rcut_smth\":0.5,\"rcut\":7.0,\"neuron\":[25,50,100],\"resnet_dt\":false,\"axis_neuron\":12,\"seed\":2132611806,\"activation_function\":\"tanh\",\"type_one_side\":false,\"precision\":\"default\",\"trainable\":true,\"exclude_types\":[],\"set_davg_zero\":false},\"fitting_net\":{\"neuron\":[240,240,240],\"resnet_dt\":true,\"seed\":2582901612,\"type\":\"ener\",\"numb_fparam\":0,\"numb_aparam\":0,\"activation_function\":\"tanh\",\"precision\":\"default\",\"trainable\":true,\"rcond\":0.001,\"atom_ener\":[]},\"data_stat_nbatch\":10,\"data_stat_protect\":0.01},\"learning_rate\":{\"type\":\"exp\",\"start_lr\":0.001,\"stop_lr\":3.51e-08,\"decay_steps\":2000,\"scale_by_worker\":\"linear\"},\"loss\":{\"start_pref_e\":0.02,\"limit_pref_e\":1,\"start_pref_f\":1000,\"limit_pref_f\":1,\"start_pref_v\":0.0,\"limit_pref_v\":0.0,\"type\":\"ener\",\"start_pref_ae\":0.0,\"limit_pref_ae\":0.0,\"start_pref_pf\":0.0,\"limit_pref_pf\":0.0,\"enable_atom_ener_coeff\":false},\"training\":{\"disp_file\":\"lcurve.out\",\"disp_freq\":200,\"save_freq\":10000,\"save_ckpt\":\"model.ckpt\",\"disp_training\":true,\"time_training\":true,\"profiling\":false,\"profiling_file\":\"timeline.json\",\"training_data\":{\"systems\":[\"./data.init/Data-crystal-linbo3/deepmd-1060K-crystal-mp-60atom-R3C\",\"./data.init/Data-crystal-linbo3/deepmd-1260K-crystal-mp-60atom-R3C\",\"./data.init/Data-crystal-linbo3/deepmd-1460K-crystal-mp-60atom-R-3C\",\"./data.init/Data-crystal-linbo3/deepmd-1660K-crystal-mp-60atom-R-3C\",\"./data.init/Data-crystal-linbo3/deepmd-1860K-crystal-mp-60atom-R-3C\",\"./data.init/Data-melt-linbo3/deepmd-1200K\",\"./data.init/Data-melt-linbo3/deepmd-1400K\",\"./data.init/Data-melt-linbo3/deepmd-1600K\",\"./data.init/Data-melt-linbo3/deepmd-1800K\",\"./data.init/Data-melt-linbo3/deepmd-2000K\",\"./data.init/iter0-11_crystal/Li12Nb12O36\",\"./data.init/deepmd-melting-60atoms/Li12Nb12O36\",\"./data.init/deepmd-melting-120atoms/Li24Nb24O72\",\"./data.iters/iter.000000/02.fp/data.004\",\"./data.iters/iter.000000/02.fp/data.005\",\"./data.iters/iter.000000/02.fp/data.006\",\"./data.iters/iter.000000/02.fp/data.007\",\"./data.iters/iter.000000/02.fp/data.008\",\"./data.iters/iter.000001/02.fp/data.004\",\"./data.iters/iter.000001/02.fp/data.005\",\"./data.iters/iter.000001/02.fp/data.006\",\"./data.iters/iter.000001/02.fp/data.007\",\"./data.iters/iter.000001/02.fp/data.008\",\"./data.iters/iter.000002/02.fp/data.004\",\"./data.iters/iter.000002/02.fp/data.005\",\"./data.iters/iter.000002/02.fp/data.006\",\"./data.iters/iter.000002/02.fp/data.007\",\"./data.iters/iter.000002/02.fp/data.008\",\"./data.iters/iter.000003/02.fp/data.004\",\"./data.iters/iter.000003/02.fp/data.005\",\"./data.iters/iter.000003/02.fp/data.006\",\"./data.iters/iter.000003/02.fp/data.007\",\"./data.iters/iter.000003/02.fp/data.008\",\"./data.iters/iter.000004/02.fp/data.004\",\"./data.iters/iter.000004/02.fp/data.005\",\"./data.iters/iter.000004/02.fp/data.006\",\"./data.iters/iter.000004/02.fp/data.007\",\"./data.iters/iter.000004/02.fp/data.008\",\"./data.iters/iter.000005/02.fp/data.004\",\"./data.iters/iter.000005/02.fp/data.005\",\"./data.iters/iter.000005/02.fp/data.006\",\"./data.iters/iter.000005/02.fp/data.007\",\"./data.iters/iter.000005/02.fp/data.008\",\"./data.iters/iter.000006/02.fp/data.004\",\"./data.iters/iter.000006/02.fp/data.005\",\"./data.iters/iter.000006/02.fp/data.006\",\"./data.iters/iter.000006/02.fp/data.007\",\"./data.iters/iter.000006/02.fp/data.008\",\"./data.iters/iter.000007/02.fp/data.004\",\"./data.iters/iter.000007/02.fp/data.005\",\"./data.iters/iter.000007/02.fp/data.006\",\"./data.iters/iter.000007/02.fp/data.007\",\"./data.iters/iter.000007/02.fp/data.008\",\"./data.iters/iter.000008/02.fp/data.002\",\"./data.iters/iter.000010/02.fp/data.002\",\"./data.iters/iter.000010/02.fp/data.003\",\"./data.iters/iter.000010/02.fp/data.004\",\"./data.iters/iter.000011/02.fp/data.005\"],\"batch_size\":[1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1],\"set_prefix\":\"set\",\"auto_prob\":\"prob_sys_size\",\"sys_probs\":null},\"seed\":1833551143,\"numb_steps\":600000,\"validation_data\":null,\"enable_profiler\":false,\"tensorboard\":false,\"tensorboard_log_dir\":\"log\",\"tensorboard_freq\":1}}"

The model file may be inconsistent with the data you present.

[warriors1523]

Thanks for helping me check out the problem. Now the molecular dynamics simulation based on the dp potential model is able to run successfully. The inconsistency between the model and input.json may have been caused by not clearing the temporary files after manually killing the dpgen process. Or it could be caused by my mistake.

[warriors1523]

Hello, developers. I see that the remove_outlier function has been written in the dpdata-0.2.15 version. Is it necessary to set the keyword during dpgen collection of first principles data or use of dpdata in order to realize this function? Or is the function implemented automatically?