Training loop

[arthursw]

Hello,

My model has been training for two weeks now (it seems doing well) and it keeps restarting from scratch.
I am not sure to understand how does the training loop work?
When will it stop?

Thanks!

[HuahongZhang]

Yeah, at default, it will do 5-fold cross-validation and you will get 5 models after successful training.

[arthursw]

All right, thank you!

It trained until epoch 250 for the 1st validation, 200 epochs for the 2nd validation, and then 530 for the third validation ; that is a big difference so I thought there might be something I misunderstood.
The training is supposed to stop if it does not improves over 160 epochs, maybe I was just unlucky with the third training.

How important is it to run the five validations? Will it drastically improve the results compared to a one shot training?

[HuahongZhang]

I would suggest running all five-folds and then use the five models as an ensemble. But it is OK to stop running after some of them, e.g. 3 of 5, are successfully trained and only use those checkpoints.