Is the final model best or last?

[bartosz-grabowski]

Hello. Is the strategy.model returned after training

• the best model given evaluation loss (for eval_every=1) or
• the model after the last epoch?

And what model is returned for eval_every=-1?

Thank you.

[AntonioCarta]

Always the last one, unless you are using EarlyStoppingPlugin. If you need something different, you can implement your own plugin that stores the best model and set strategy.model = best_model in after_traning_experience.

Of course when you use metrics computed in strategy.eval you have to take care that those are actually computed (i.e. eval_every is set correctly).

[bartosz-grabowski]

Thank you. I've tried to use EarlyStoppingPlugin as follows:

model_chpt_plugin = EarlyStoppingPlugin(
    patience=epochs+1,  # Set that way to not utlize early stopping, just best model saving.
    val_stream_name='valid',
    metric_name='Loss_Exp',
    mode='min',
    peval_mode='epoch',
    verbose=True,
)

I did set eval_every=1 for the utilized strategy:

cl_strategy = Naive(
    model, optimizer, loss_criterion, train_mb_size=batch_size,
    train_epochs=epochs, eval_mb_size=batch_size, plugins=[scheduler_plugin, model_chpt_plugin],
    evaluator=eval_plugin, device=device, eval_every=1
)

Finally, I set the eval stream during training like so:
cl_strategy.train(experience, eval_streams=[benchmark.valid_stream[i]], num_workers=num_workers)

The plugin seems to work, because I get the following messages during training:
EarlyStopping: new best value: 1.3464396075487823

But after the training, I get the same metric value for the test set (cl_strategy.eval(benchmark.test_stream, num_workers=num_workers)) and the set random seed whether or not EarlyStoppingPlugin is present. It shouldn't be possible since the last update of the best model is for the epoch 50, and the last epoch is epoch 59.

[bartosz-grabowski]

I think I solved my problem. Looking at the EarlyStoppingPlugin, it seems that best weights are loaded into the model only if the model is not improving for the patience epochs, which in my case is not possible.

Thank you for your help.