Add more options to create_supervised_trainer

[vfdev-5]

🚀 Feature

Idea is to add more options to create_supervised_trainer helper:

• native AMP option similar to https://github.com/lidq92/LinearityIQA/blob/master/modified_ignite_engine.py
  • we may either use nvidia/apex if installed or torch native amp if v1.6.0. Otherwise, raise an error.
• grads cliping as proposed add gradient clipping to create_supervised_trainer() #419

For Hacktoberfest/PyDataGlobal contributors, feel free to ask questions for details if any and say that you would like to tackle the issue.
Please, take a look at CONTRIBUTING guide.

[apthagowda97]

I would like to work on this.

[vfdev-5]

@apthagowda97 sure, please go ahead. I assign you to this issue

[vfdev-5]

@apthagowda97 any updates on this ? Anyway, do not hesitate to ask question if something is not clear about the issue.

[apthagowda97]

By going through issue #419 its best to add gradient clipping to contrib.engines as discussed compared with changing create_supervised_trainer. Should I name it create_supervised_gc_trainer?

[vfdev-5]

@apthagowda97 I think it is better let's keep the same name and add the options. Such that there is one multi-option helper function.

[vfdev-5]

@apthagowda97 any updates on this issue ?

[apthagowda97]

Sorry I was busy from past few days. I ll try to finish it Asap.

[vfdev-5]

Cross-posted from #1589

---

We discussed this PR and related issue with the team and we think that we should explore a bit different approach. Helper method create_supervised_trainer is roughly made of 2 things : update function definition and Engine setup.

Probably, it would more helpful to provide public methods like:

• supervised_training_step
• supervised_training_step_tpu
• supervised_training_step_apex
• supervised_training_step_amp

and inside create_supervised_trainer we could setup the trainer according to provided options without lots of if/else. Maybe, we can skip for instance grad norm.

Basically, the idea is something like that :

def get_training_step_1(a):
    def training_step(e, b):
        print(a, e, b)
    return training_step
    
def get_training_step_2(a):
    def training_step(e, b):
        print(a, e, b, "with amp")
    return training_step

def create_supervised_trainer(a, opt):
    training_step = None
    if opt == 1:
        training_step = get_training_step_1(a)
    elif opt == 2:
        training_step = get_training_step_2(a)
        
    e = Engine(training_step)
    return e