Modify training script to account for stored metrics when resuming training

[CCanchilaM]

Problem

When resuming training from a checkpoint, the metric from the previously saved best model is ignored. CheckpointSaver is initiated with self.best_epoch = None and self.best_metric = None, completely ignoring previously saved results. The provided training script (train.py) does not provide an option to read these metrics.

I'm not sure if people are aware of this issue, at least I didn't realize until I restarted a training, and the new best model had lower accuracy than previously reported.

Proposed solution

This can be easily solved with something like:

# Line 803 in train.py
saver = utils.CheckpointSaver(
            model=model,
            optimizer=optimizer,
            args=args,
            model_ema=model_ema,
            amp_scaler=loss_scaler,
            checkpoint_dir=output_dir,
            recovery_dir=output_dir,
            decreasing=decreasing_metric,
            max_history=args.checkpoint_hist
        )

# --- new lines --- #
if args.resume:
    _temp_read_path = os.path.join(saver.checkpoint_dir, 'model_best' + saver.extension)
    if os.path.exists(_temp_read_path):
        _temp = torch.load(_temp_read_path, map_location='cpu')
        saver.best_epoch = _temp['epoch']
        saver.best_metric = _temp['metric']
# --- end of new lines --- #

Please share if there is a better way to do this or if I missed something😅