Approaximately how much time required to complete individual parse_generator and scpn model training ?

[munaAchyuta]

Hi Guys,

It's been 10 days since training of scpn model was started, and after this 10 days, training has just completed 4 epochs out of 15 epochs with DEFAULT settings.

machine configuration :

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.54                 Driver Version: 396.54                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 000075E1:00:00.0 Off |                    0 |
| N/A   67C    P0    89W / 149W |   9896MiB / 11441MiB |     84%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     47372      C   xyzz/python2.7                                 1440MiB |
|    0     82095      C   scpn/python                                      8443MiB |
+-----------------------------------------------------------------------------+

using above system, can you guys tell me approximately how much time will it take to train scpn model(DEFAULT settings) ?
with that as another model has dependency with scpn which is "parse_generator", can you tell me how much time will this model(DEFAULT settings) take to train ?

Thanks.

[miyyer]

hey, it did take us about 2 weeks to train the parse generator. we stopped training after 5 epochs as the model did not seem to be improving. we should have changed the default epochs to 5 and not 15, sorry about that! training the paraphrase generation model should be faster

[munaAchyuta]

Hi @miyyer Thanks for your reply.
could you please tell me exact number (number of weeks) will take to complete paraphrase generation (train_scpn.py) model ?

actually I completed training of scpn model (train_scpn.py) using default settings , but unfortunately it's not performing like the trained model you shared , when I made test with the demo test data.
and it took 6 week + time to complete.

yes , only changes I made are I included # sentences are too short and # downsample if input sentences are too short

please find the some sample output from demo data when I tested.

0 GOLD a group of kids is playing in a yard and an old man is standing in the background .
0 they play a group in the yard .
0 play a group in the background .
0 everyone in the background .
0 in a yard in history .
0 the group is a group in the yard of a group with a man in the background .
0 i can see the background .
0 they play the background .
0 and he was a group in the background .
0 now they played a group in the background .
0 well , they played a group in the background .

Thanks.