Output logs from coco pose training

[laol777]

Can you share your training logs from experiments/coco_pose_tracking.sh run? Currently, I have them like this, but I'm not sure that I run your experiment properly

2021-02-23-19-09: epoch: 1 |tot 8.034233 | hm 0.780280 | wh 1.383786 | reg 0.211622 | hps 1.385013 | hm_hp 0.557083 | hp_offset 0.216599 | tracking 4.745257 | time 114.466667 | 
2021-02-23-21-03: epoch: 2 |tot 6.905976 | hm 0.751715 | wh 1.418231 | reg 0.211965 | hps 1.495890 | hm_hp 0.591616 | hp_offset 0.217978 | tracking 3.494990 | time 114.033333 | 
2021-02-23-22-57: epoch: 3 |tot 5.719261 | hm 0.751503 | wh 1.441991 | reg 0.211726 | hps 1.567808 | hm_hp 0.614897 | hp_offset 0.218660 | tracking 2.210469 | time 114.283333 | 
2021-02-24-00-51: epoch: 4 |tot 5.311181 | hm 0.733608 | wh 1.421174 | reg 0.211345 | hps 1.544822 | hm_hp 0.607454 | hp_offset 0.218501 | tracking 1.853332 | time 113.916667 | 
2021-02-24-02-45: epoch: 5 |tot 5.111943 | hm 0.719469 | wh 1.409294 | reg 0.211146 | hps 1.532119 | hm_hp 0.600310 | hp_offset 0.218345 | tracking 1.689624 | time 113.883333 | 
2021-02-24-04-39: epoch: 6 |tot 4.982389 | hm 0.703464 | wh 1.402448 | reg 0.210573 | hps 1.516978 | hm_hp 0.593688 | hp_offset 0.217996 | tracking 1.599447 | time 114.166667 | 
2021-02-24-06-33: epoch: 7 |tot 4.911689 | hm 0.696606 | wh 1.402412 | reg 0.210589 | hps 1.515493 | hm_hp 0.591116 | hp_offset 0.218126 | tracking 1.539518 | time 113.966667 | 
2021-02-24-08-27: epoch: 8 |tot 4.837281 | hm 0.690110 | wh 1.381137 | reg 0.210462 | hps 1.516613 | hm_hp 0.587404 | hp_offset 0.217832 | tracking 1.476746 | time 114.016667 | 
2021-02-24-10-22: epoch: 9 |tot 6.242262 | hm 0.884284 | wh 1.827279 | reg 0.215115 | hps 2.018405 | hm_hp 0.797681 | hp_offset 0.220822 | tracking 1.923227 | time 114.483333 | 
2021-02-24-12-14: epoch: 10 |tot 5.695879 | hm 0.831773 | wh 1.660751 | reg 0.214864 | hps 1.870885 | hm_hp 0.732475 | hp_offset 0.220251 | tracking 1.659555 | time 112.583333 | 
2021-02-24-14-07: epoch: 11 |tot 4.934529 | hm 0.715321 | wh 1.436181 | reg 0.211081 | hps 1.580352 | hm_hp 0.618000 | hp_offset 0.218546 | tracking 1.447612 | time 112.283333 | 
2021-02-24-16-00: epoch: 12 |tot 4.745899 | hm 0.688001 | wh 1.389524 | reg 0.210393 | hps 1.526371 | hm_hp 0.593078 | hp_offset 0.217836 | tracking 1.371268 | time 113.683333 | 
2021-02-24-17-53: epoch: 13 |tot 4.644405 | hm 0.678480 | wh 1.379882 | reg 0.209991 | hps 1.515980 | hm_hp 0.584850 | hp_offset 0.217902 | tracking 1.299213 | time 113.066667 | 
2021-02-24-19-45: epoch: 14 |tot 7.257541 | hm 1.066855 | wh 2.258355 | reg 0.221027 | hps 2.403292 | hm_hp 1.002906 | hp_offset 0.224676 | tracking 2.112948 | time 111.833333 | 

[xingyizhou]

Sorry I don't have the log right now. The log looks OK except the 14th epoch. It seems something is crashed here. One possible reason might be this. Can you try to disable CUDNN bn in pytorch and retrain?