-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCT_LSMDC.txt
2438 lines (2438 loc) · 139 KB
/
HCT_LSMDC.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 6379.467805862427 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 43.02292847633362 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 17.12521719932556 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch0.pth ...
Done in 1.547s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch0.pth ...
Done in 3.559s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.1
LSMDC_full_test/t2v_metrics/R5: 0.2
LSMDC_full_test/t2v_metrics/R10: 0.5
LSMDC_full_test/t2v_metrics/R50: 4.6
LSMDC_full_test/t2v_metrics/MedR: 506.5
LSMDC_full_test/t2v_metrics/MeanR: 505.122
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.2154434690031884
LSMDC_full_test/v2t_metrics/R1: 0.1
LSMDC_full_test/v2t_metrics/R5: 0.4
LSMDC_full_test/v2t_metrics/R10: 0.8
LSMDC_full_test/v2t_metrics/R50: 4.7
LSMDC_full_test/v2t_metrics/MedR: 507.0
LSMDC_full_test/v2t_metrics/MeanR: 503.82
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.3174802103936399
mnt_best : 0.2154434690031884
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.78946 batch_time=22.34864
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.06677 batch_time=0.35323
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.32602 batch_time=0.35116
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.46960 batch_time=0.35537
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.42004 batch_time=0.35843
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.34798 batch_time=0.36717
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.05357 batch_time=0.35206
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 7.44931 batch_time=0.34881
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.19711 batch_time=0.36431
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.78997 batch_time=0.35235
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 6.69456 batch_time=0.38654
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.95946 batch_time=0.37268
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.83274 batch_time=0.34835
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.03802 batch_time=0.34805
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.46743 batch_time=0.36754
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.71032 batch_time=0.36695
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 6.07815 batch_time=0.36248
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.56433 batch_time=0.35236
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.22079 batch_time=0.34959
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.75915 batch_time=0.36988
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 6.12738 batch_time=0.35527
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 6.32750 batch_time=0.35195
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 6.09911 batch_time=0.37818
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch1.pth ...
Done in 3.717s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch1.pth ...
Done in 7.412s
epoch : 1
loss : 6.9916676349639895
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 8.6
LSMDC_full_test/t2v_metrics/R5: 21.4
LSMDC_full_test/t2v_metrics/R10: 29.9
LSMDC_full_test/t2v_metrics/R50: 60.3
LSMDC_full_test/t2v_metrics/MedR: 31.0
LSMDC_full_test/t2v_metrics/MeanR: 93.798
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.65473233776872
LSMDC_full_test/v2t_metrics/R1: 7.2
LSMDC_full_test/v2t_metrics/R5: 20.3
LSMDC_full_test/v2t_metrics/R10: 30.1
LSMDC_full_test/v2t_metrics/R50: 60.1
LSMDC_full_test/v2t_metrics/MedR: 32.0
LSMDC_full_test/v2t_metrics/MeanR: 88.604
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.385700404750995
mnt_best : 17.65473233776872
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 6.31563 batch_time=17.11886
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 5.95862 batch_time=0.34807
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 5.90834 batch_time=0.35174
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 6.34216 batch_time=0.36325
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 5.91393 batch_time=0.36769
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 5.99983 batch_time=0.36004
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 5.46009 batch_time=0.36488
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 6.29948 batch_time=0.36210
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 6.01170 batch_time=0.34917
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 5.60963 batch_time=0.35272
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.71581 batch_time=0.35154
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.62347 batch_time=0.36807
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 5.55142 batch_time=1.32900
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 5.51839 batch_time=0.35354
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 6.17764 batch_time=0.34999
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.30042 batch_time=0.38344
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.44028 batch_time=0.34835
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 5.18240 batch_time=0.35785
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 5.26586 batch_time=6.66050
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.35440 batch_time=0.35353
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.54097 batch_time=0.35731
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 5.50960 batch_time=0.36035
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.74656 batch_time=0.36917
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch2.pth ...
Done in 4.003s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch2.pth ...
Done in 7.937s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 5.7157047996521
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 9.8
LSMDC_full_test/t2v_metrics/R5: 24.4
LSMDC_full_test/t2v_metrics/R10: 34.4
LSMDC_full_test/t2v_metrics/R50: 63.9
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 82.119
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.186364682931156
LSMDC_full_test/v2t_metrics/R1: 9.0
LSMDC_full_test/v2t_metrics/R5: 23.3
LSMDC_full_test/v2t_metrics/R10: 32.8
LSMDC_full_test/v2t_metrics/R50: 61.6
LSMDC_full_test/v2t_metrics/MedR: 28.0
LSMDC_full_test/v2t_metrics/MeanR: 85.495
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.01767514963615
mnt_best : 20.186364682931156
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.76753 batch_time=22.42866
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.41597 batch_time=0.37457
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.59541 batch_time=0.67306
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.16624 batch_time=0.34579
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 4.87151 batch_time=0.34880
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.67612 batch_time=0.34440
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.31829 batch_time=0.37274
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.07205 batch_time=0.34748
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 4.80624 batch_time=0.35396
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.38762 batch_time=0.35969
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 4.19096 batch_time=0.35958
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.23342 batch_time=0.36490
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.02385 batch_time=0.36544
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.19182 batch_time=0.35764
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.73832 batch_time=0.35253
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 4.83740 batch_time=0.35333
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.96387 batch_time=0.36203
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 4.66567 batch_time=0.37925
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 5.23826 batch_time=0.37520
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.00275 batch_time=0.34729
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 4.96803 batch_time=0.34623
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 5.18392 batch_time=0.35313
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 4.67319 batch_time=0.34143
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch3.pth ...
Done in 4.264s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch3.pth ...
Done in 8.123s
removing stale ckpt [epoch 2] [took 0.00s]
epoch : 3
loss : 5.202628316879273
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 10.9
LSMDC_full_test/t2v_metrics/R5: 26.3
LSMDC_full_test/t2v_metrics/R10: 35.1
LSMDC_full_test/t2v_metrics/R50: 65.1
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 76.834
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.58886385837942
LSMDC_full_test/v2t_metrics/R1: 11.0
LSMDC_full_test/v2t_metrics/R5: 25.3
LSMDC_full_test/v2t_metrics/R10: 35.0
LSMDC_full_test/v2t_metrics/R50: 63.8
LSMDC_full_test/v2t_metrics/MedR: 23.5
LSMDC_full_test/v2t_metrics/MeanR: 76.1695
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.35635264818574
mnt_best : 21.58886385837942
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 4.68857 batch_time=21.55995
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 4.94785 batch_time=0.34731
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 4.60298 batch_time=0.35025
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 4.75316 batch_time=0.36850
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 5.10920 batch_time=0.34888
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.57954 batch_time=0.33278
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 5.27687 batch_time=0.35303
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 4.80751 batch_time=0.34640
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 5.09846 batch_time=0.34944
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 5.35741 batch_time=0.34817
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 4.63263 batch_time=0.37172
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 5.23133 batch_time=0.37383
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 4.71652 batch_time=0.35437
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 4.11312 batch_time=0.34977
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 4.78653 batch_time=0.34463
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 4.56040 batch_time=2.32690
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 5.21143 batch_time=0.34951
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 4.31473 batch_time=0.34730
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.88316 batch_time=0.35171
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.35400 batch_time=0.35036
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 4.72601 batch_time=0.35276
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.76250 batch_time=0.35074
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 5.06225 batch_time=0.37284
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch4.pth ...
Done in 4.036s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch4.pth ...
Done in 8.244s
removing stale ckpt [epoch 3] [took 0.06s]
epoch : 4
loss : 4.815496597290039
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 11.8
LSMDC_full_test/t2v_metrics/R5: 28.0
LSMDC_full_test/t2v_metrics/R10: 36.9
LSMDC_full_test/t2v_metrics/R50: 67.5
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 70.527
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.015591193021006
LSMDC_full_test/v2t_metrics/R1: 10.8
LSMDC_full_test/v2t_metrics/R5: 27.0
LSMDC_full_test/v2t_metrics/R10: 36.1
LSMDC_full_test/v2t_metrics/R50: 65.6
LSMDC_full_test/v2t_metrics/MedR: 23.0
LSMDC_full_test/v2t_metrics/MeanR: 69.412
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 21.916182447403713
mnt_best : 23.015591193021006
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 4.71450 batch_time=23.55393
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.87107 batch_time=0.34449
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.94789 batch_time=0.37861
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 4.45241 batch_time=0.36968
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 4.15361 batch_time=0.35475
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.18166 batch_time=0.35151
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.55971 batch_time=0.35939
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.27800 batch_time=0.35042
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 4.75631 batch_time=0.36444
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 4.93100 batch_time=0.35375
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 4.59058 batch_time=0.34706
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.50760 batch_time=0.34981
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.94667 batch_time=0.35692
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.83498 batch_time=0.34638
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.27839 batch_time=0.66262
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 4.35885 batch_time=0.35797
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.67686 batch_time=0.36047
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.20592 batch_time=0.35237
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.64913 batch_time=0.35024
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.44076 batch_time=0.35502
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.30284 batch_time=0.79520
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.61717 batch_time=0.36564
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 5.06891 batch_time=0.36750
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch5.pth ...
Done in 4.125s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch5.pth ...
Done in 8.044s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 4.589691753387451
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 29.3
LSMDC_full_test/t2v_metrics/R10: 37.8
LSMDC_full_test/t2v_metrics/R50: 66.8
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 71.312
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.818061754996375
LSMDC_full_test/v2t_metrics/R1: 11.6
LSMDC_full_test/v2t_metrics/R5: 28.7
LSMDC_full_test/v2t_metrics/R10: 37.2
LSMDC_full_test/v2t_metrics/R50: 67.1
LSMDC_full_test/v2t_metrics/MedR: 21.25
LSMDC_full_test/v2t_metrics/MeanR: 71.835
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.13631962009355
mnt_best : 23.818061754996375
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 4.61819 batch_time=18.34434
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.28818 batch_time=0.35219
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 4.39471 batch_time=0.34631
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.30304 batch_time=0.33932
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 3.89332 batch_time=0.35309
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.54599 batch_time=0.34359
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.30626 batch_time=0.67380
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.26492 batch_time=0.34058
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 4.22750 batch_time=0.44245
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 4.57420 batch_time=0.35320
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 4.14240 batch_time=0.34258
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.34454 batch_time=0.33686
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.09598 batch_time=0.34597
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 4.71306 batch_time=0.34992
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 4.13397 batch_time=0.37159
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.51368 batch_time=0.45640
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 4.13898 batch_time=0.34510
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.13841 batch_time=0.34553
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 4.05797 batch_time=0.35685
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.49816 batch_time=0.34772
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 4.23301 batch_time=0.33422
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 4.13576 batch_time=0.34880
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 3.49145 batch_time=0.34223
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch6.pth ...
Done in 14.330s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch6.pth ...
Done in 19.127s
removing stale ckpt [epoch 5] [took 0.01s]
epoch : 6
loss : 4.282450213432312
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 13.6
LSMDC_full_test/t2v_metrics/R5: 30.6
LSMDC_full_test/t2v_metrics/R10: 39.6
LSMDC_full_test/t2v_metrics/R50: 67.7
LSMDC_full_test/t2v_metrics/MedR: 19.5
LSMDC_full_test/t2v_metrics/MeanR: 69.246
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.447893598913122
LSMDC_full_test/v2t_metrics/R1: 12.4
LSMDC_full_test/v2t_metrics/R5: 28.8
LSMDC_full_test/v2t_metrics/R10: 38.4
LSMDC_full_test/v2t_metrics/R50: 67.1
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.787
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.935828570745507
mnt_best : 25.447893598913122
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.45890 batch_time=23.07317
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 3.92896 batch_time=0.34293
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.45071 batch_time=0.34965
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 4.07572 batch_time=0.34990
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 3.67475 batch_time=0.35458
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 3.81314 batch_time=0.34201
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.67097 batch_time=0.34220
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.52117 batch_time=0.36710
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 4.12922 batch_time=0.33877
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.06128 batch_time=0.34562
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 3.95215 batch_time=0.33869
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.42628 batch_time=0.33916
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 3.93280 batch_time=0.35386
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.67996 batch_time=0.35171
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 3.58420 batch_time=0.35000
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.91294 batch_time=0.35262
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.31997 batch_time=0.34372
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 3.96736 batch_time=0.34459
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.77384 batch_time=0.34985
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 4.15013 batch_time=0.36583
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 4.46992 batch_time=0.34746
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.95326 batch_time=0.33986
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 4.24226 batch_time=0.34426
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch7.pth ...
Done in 4.120s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 4.071514036178589
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 12.9
LSMDC_full_test/t2v_metrics/R5: 30.1
LSMDC_full_test/t2v_metrics/R10: 41.3
LSMDC_full_test/t2v_metrics/R50: 67.8
LSMDC_full_test/t2v_metrics/MedR: 18.5
LSMDC_full_test/t2v_metrics/MeanR: 71.243
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.217503271961448
LSMDC_full_test/v2t_metrics/R1: 11.8
LSMDC_full_test/v2t_metrics/R5: 30.9
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 67.0
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 71.487
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.49336818975149
mnt_best : 25.447893598913122
not_improved_count: 1
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 3.86996 batch_time=30.29326
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.81378 batch_time=0.34935
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 3.53903 batch_time=0.34817
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 4.20520 batch_time=0.36719
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.69266 batch_time=0.34343
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 3.64624 batch_time=0.34450
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 4.10763 batch_time=0.36166
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.21143 batch_time=0.40877
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.50027 batch_time=0.35452
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 3.64211 batch_time=0.34870
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.00784 batch_time=0.34731
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 4.38548 batch_time=0.38170
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.57089 batch_time=0.34551
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.85466 batch_time=0.81726
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 3.28446 batch_time=0.36579
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.44094 batch_time=0.36986
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 4.19944 batch_time=0.35957
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.87610 batch_time=0.37035
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.14386 batch_time=0.35070
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.19457 batch_time=0.36217
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.55421 batch_time=0.34576
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.60009 batch_time=0.45326
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.87742 batch_time=0.34131
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch8.pth ...
Done in 4.560s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch8.pth ...
Done in 8.573s
removing stale ckpt [epoch 7] [took 0.01s]
epoch : 8
loss : 3.8819942750930787
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 13.2
LSMDC_full_test/t2v_metrics/R5: 31.5
LSMDC_full_test/t2v_metrics/R10: 41.4
LSMDC_full_test/t2v_metrics/R50: 69.0
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 70.145
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.820319309305066
LSMDC_full_test/v2t_metrics/R1: 14.0
LSMDC_full_test/v2t_metrics/R5: 29.9
LSMDC_full_test/v2t_metrics/R10: 39.8
LSMDC_full_test/v2t_metrics/R50: 68.4
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 66.335
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.54038455432571
mnt_best : 25.820319309305066
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 4.03863 batch_time=22.24783
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.75337 batch_time=0.35076
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.91975 batch_time=0.34883
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.16874 batch_time=0.35189
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.39306 batch_time=0.35500
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.63075 batch_time=0.36031
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 4.02260 batch_time=0.35945
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 3.94208 batch_time=0.34944
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.76960 batch_time=0.37010
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.43170 batch_time=1.27739
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.39758 batch_time=0.35078
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.82137 batch_time=0.34915
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.57375 batch_time=0.36058
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.64434 batch_time=0.75500
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.49956 batch_time=0.35023
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 4.03496 batch_time=0.35975
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.73567 batch_time=0.35038
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.53663 batch_time=0.35475
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.54048 batch_time=0.35146
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.48107 batch_time=0.69359
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.31399 batch_time=1.63224
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 3.66275 batch_time=0.36807
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 2.83789 batch_time=0.35722
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch9.pth ...
Done in 5.086s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 3.728962013244629
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 12.7
LSMDC_full_test/t2v_metrics/R5: 31.9
LSMDC_full_test/t2v_metrics/R10: 42.1
LSMDC_full_test/t2v_metrics/R50: 69.3
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 67.975
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.741005058854498
LSMDC_full_test/v2t_metrics/R1: 13.1
LSMDC_full_test/v2t_metrics/R5: 30.9
LSMDC_full_test/v2t_metrics/R10: 39.7
LSMDC_full_test/v2t_metrics/R50: 70.3
LSMDC_full_test/v2t_metrics/MedR: 18.5
LSMDC_full_test/v2t_metrics/MeanR: 66.946
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.235200555347703
mnt_best : 25.820319309305066
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.96738 batch_time=19.53223
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.36712 batch_time=0.34372
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.22994 batch_time=0.56596
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 4.04454 batch_time=0.34991
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.81622 batch_time=0.35337
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.51532 batch_time=0.35244
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 4.04030 batch_time=0.36788
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 4.03543 batch_time=0.34728
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.77450 batch_time=0.39095
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.65853 batch_time=0.34552
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.98949 batch_time=0.34797
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.42155 batch_time=0.34876
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.45606 batch_time=0.36121
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.55020 batch_time=0.88288
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 3.26347 batch_time=1.50189
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.42917 batch_time=0.34763
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.74936 batch_time=0.34639
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.24480 batch_time=0.35578
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.52246 batch_time=0.36085
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.14054 batch_time=0.37880
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.26572 batch_time=0.87640
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 3.70528 batch_time=0.37914
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.30810 batch_time=0.35919
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch10.pth ...
Done in 4.071s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch10.pth ...
Done in 8.549s
removing stale ckpt [epoch 9] [took 0.02s]
epoch : 10
loss : 3.5729863986968993
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 14.1
LSMDC_full_test/t2v_metrics/R5: 32.7
LSMDC_full_test/t2v_metrics/R10: 41.1
LSMDC_full_test/t2v_metrics/R50: 68.3
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 70.937
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.6605781064874
LSMDC_full_test/v2t_metrics/R1: 12.6
LSMDC_full_test/v2t_metrics/R5: 30.5
LSMDC_full_test/v2t_metrics/R10: 41.4
LSMDC_full_test/v2t_metrics/R50: 67.9
LSMDC_full_test/v2t_metrics/MedR: 17.5
LSMDC_full_test/v2t_metrics/MeanR: 70.203
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.151095631342706
mnt_best : 26.6605781064874
not_improved_count: 0
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.31719 batch_time=21.44227
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.31045 batch_time=0.35216
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.00030 batch_time=0.34890
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.44447 batch_time=0.37633
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.57616 batch_time=0.46071
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 3.04016 batch_time=0.36360
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.31573 batch_time=3.11129
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.34324 batch_time=0.34831
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.51145 batch_time=0.35815
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 3.44250 batch_time=0.36587
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.26610 batch_time=0.35884
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 3.20610 batch_time=0.35837
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.66605 batch_time=0.35261
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 3.03284 batch_time=3.52453
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.51274 batch_time=0.35301
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 3.77226 batch_time=0.36504
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 3.29059 batch_time=0.35599
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.13082 batch_time=0.35946
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.73418 batch_time=0.34352
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.41099 batch_time=0.34923
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.60895 batch_time=0.35778
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 3.14449 batch_time=0.35303
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 2.82797 batch_time=0.36703
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch11.pth ...
Done in 4.365s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 3.3901283645629885
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 13.4
LSMDC_full_test/t2v_metrics/R5: 31.9
LSMDC_full_test/t2v_metrics/R10: 41.9
LSMDC_full_test/t2v_metrics/R50: 68.0
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 69.452
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.163941422173725
LSMDC_full_test/v2t_metrics/R1: 12.0
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 40.3
LSMDC_full_test/v2t_metrics/R50: 67.6
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.073
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.551003010553774
mnt_best : 26.6605781064874
not_improved_count: 1
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.42541 batch_time=21.43694
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.19826 batch_time=0.35166
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.05541 batch_time=0.36002
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 3.13730 batch_time=0.35907
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 3.62307 batch_time=0.38861
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.33848 batch_time=0.35919
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.00626 batch_time=0.35654
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.81119 batch_time=0.35145
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.39936 batch_time=0.38026
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.22191 batch_time=0.37630
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.31154 batch_time=0.39299
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.01353 batch_time=0.36899
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.40603 batch_time=0.35027
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.05411 batch_time=0.61671
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 2.95686 batch_time=0.35131
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 2.97422 batch_time=0.71139
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 3.36742 batch_time=0.34823
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.23447 batch_time=0.37984
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 3.07019 batch_time=0.34894
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.07742 batch_time=1.24165
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.04316 batch_time=0.37791
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 3.66912 batch_time=0.35123
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.68551 batch_time=0.82259
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch12.pth ...
Done in 4.792s
removing stale ckpt [epoch 11] [took 0.01s]
epoch : 12
loss : 3.2442480812072754
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 13.6
LSMDC_full_test/t2v_metrics/R5: 31.2
LSMDC_full_test/t2v_metrics/R10: 42.6
LSMDC_full_test/t2v_metrics/R50: 68.8
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 70.284
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.244262147244147
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 30.9
LSMDC_full_test/v2t_metrics/R10: 40.4
LSMDC_full_test/v2t_metrics/R50: 68.6
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 70.796
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.187396065470594
mnt_best : 26.6605781064874
not_improved_count: 2
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.95909 batch_time=14.26654
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.03401 batch_time=0.36616
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.61971 batch_time=0.34839
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.15318 batch_time=0.37592
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.53950 batch_time=0.35309
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 2.96119 batch_time=0.37080
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 2.97717 batch_time=1.03574
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 2.79344 batch_time=0.35253
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 3.40788 batch_time=0.51555
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.61795 batch_time=0.36629
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.09868 batch_time=0.46255
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 2.86426 batch_time=0.34840
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 2.87647 batch_time=0.35599
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.31918 batch_time=1.37492
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.23826 batch_time=0.35854
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.17429 batch_time=0.37226
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 3.14507 batch_time=0.36420
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 2.82620 batch_time=0.57984
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 2.96222 batch_time=1.88514
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 2.67461 batch_time=0.34495
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.48251 batch_time=0.64751
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.01917 batch_time=0.35645
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.20101 batch_time=0.34856
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch13.pth ...
Done in 5.600s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch13.pth ...
Done in 11.008s
removing stale ckpt [epoch 12] [took 0.11s]
epoch : 13
loss : 3.1129684572219847
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 13.8
LSMDC_full_test/t2v_metrics/R5: 32.1
LSMDC_full_test/t2v_metrics/R10: 43.0
LSMDC_full_test/t2v_metrics/R50: 68.1
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 68.671
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.7065337711002
LSMDC_full_test/v2t_metrics/R1: 14.2
LSMDC_full_test/v2t_metrics/R5: 32.0
LSMDC_full_test/v2t_metrics/R10: 42.1
LSMDC_full_test/v2t_metrics/R50: 68.0
LSMDC_full_test/v2t_metrics/MedR: 16.0
LSMDC_full_test/v2t_metrics/MeanR: 67.395
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.744848339748362
mnt_best : 26.7065337711002
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 2.78104 batch_time=18.40273
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.09374 batch_time=0.34310
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 2.83985 batch_time=0.34725
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.92420 batch_time=0.34621
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 3.26008 batch_time=0.34245
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 3.27458 batch_time=0.37202
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 3.15201 batch_time=0.35489
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.02455 batch_time=0.36717
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 2.73100 batch_time=0.35111
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 2.91001 batch_time=0.36087
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 2.57593 batch_time=0.34840
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 3.26178 batch_time=0.34647
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.20667 batch_time=0.35161
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 2.83851 batch_time=0.36036
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.33747 batch_time=0.35373
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 2.62040 batch_time=0.36328
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.17041 batch_time=0.34079
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 2.77272 batch_time=0.36039
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.02788 batch_time=0.34977
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 2.87104 batch_time=0.36162
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.74242 batch_time=0.35065
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.53590 batch_time=0.35377
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 2.34311 batch_time=0.33774
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch14.pth ...
Done in 4.000s
removing stale ckpt [epoch 13] [took 0.21s]
epoch : 14
loss : 2.9383766994476317
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 13.9
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 42.2
LSMDC_full_test/t2v_metrics/R50: 69.0
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 68.215
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.631423094195423
LSMDC_full_test/v2t_metrics/R1: 13.7
LSMDC_full_test/v2t_metrics/R5: 31.8
LSMDC_full_test/v2t_metrics/R10: 42.4
LSMDC_full_test/v2t_metrics/R50: 69.4
LSMDC_full_test/v2t_metrics/MedR: 17.0
LSMDC_full_test/v2t_metrics/MeanR: 67.538
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.434504928157267
mnt_best : 26.7065337711002
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.36768 batch_time=20.26864
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.96673 batch_time=0.34680
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 2.61235 batch_time=0.35154
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 2.59972 batch_time=0.35367
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.93578 batch_time=0.38753
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 2.92047 batch_time=0.34748
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.34560 batch_time=0.46321
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.26266 batch_time=0.35208
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 2.94761 batch_time=0.35138
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 2.89182 batch_time=0.35138
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.91517 batch_time=0.91643
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.69205 batch_time=0.34886
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.79017 batch_time=0.35388
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 2.51543 batch_time=5.63443
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.72518 batch_time=0.35486
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.67476 batch_time=0.38453
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 2.40070 batch_time=0.34863
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 2.43017 batch_time=0.35616
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 2.56664 batch_time=0.39434
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.20005 batch_time=0.33819
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.86021 batch_time=0.34926
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.91942 batch_time=0.35115
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 3.35658 batch_time=0.34915
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch15.pth ...
Done in 4.594s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 2.8619954900741575
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 13.9
LSMDC_full_test/t2v_metrics/R5: 32.0
LSMDC_full_test/t2v_metrics/R10: 42.0
LSMDC_full_test/t2v_metrics/R50: 70.0
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 68.5015
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.534120046593404
LSMDC_full_test/v2t_metrics/R1: 12.6
LSMDC_full_test/v2t_metrics/R5: 31.6
LSMDC_full_test/v2t_metrics/R10: 41.0
LSMDC_full_test/v2t_metrics/R50: 68.7
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 69.922
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.367665056216143
mnt_best : 26.7065337711002
not_improved_count: 2
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.93023 batch_time=19.38700
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.23019 batch_time=0.36559
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.93046 batch_time=0.33975
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 2.57934 batch_time=0.34850
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.12806 batch_time=0.39670
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 2.92620 batch_time=0.36124
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.67569 batch_time=4.97664
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 2.99974 batch_time=0.33718
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.43636 batch_time=0.34099
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 3.05312 batch_time=0.36669
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 2.79623 batch_time=0.34571
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 2.90191 batch_time=0.34834
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.86028 batch_time=0.36367
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 2.54966 batch_time=0.34727
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.62592 batch_time=0.34002
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.64340 batch_time=0.34464
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.45946 batch_time=0.34445
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 2.91306 batch_time=0.34679
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.55029 batch_time=0.33797
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 2.42752 batch_time=0.33960
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.44461 batch_time=0.34440
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.67613 batch_time=0.33685
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 3.04175 batch_time=0.36611
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch16.pth ...
Done in 4.667s
removing stale ckpt [epoch 15] [took 0.00s]
epoch : 16
loss : 2.7788353538513184
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 13.6
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 42.2
LSMDC_full_test/t2v_metrics/R50: 68.5
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 70.452
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.43843498678253
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 31.4
LSMDC_full_test/v2t_metrics/R10: 40.8
LSMDC_full_test/v2t_metrics/R50: 67.6
LSMDC_full_test/v2t_metrics/MedR: 18.5
LSMDC_full_test/v2t_metrics/MeanR: 70.533
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.405822543487524
mnt_best : 26.7065337711002
not_improved_count: 3
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.48130 batch_time=22.78619
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 2.64216 batch_time=0.38155
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 2.67901 batch_time=0.35697
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 3.10356 batch_time=0.36784
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.13768 batch_time=0.35713
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.63073 batch_time=0.35271
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.94235 batch_time=0.35939
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.76296 batch_time=0.35084
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.68725 batch_time=0.35975
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.52405 batch_time=0.35162
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.66611 batch_time=0.35957
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 3.46284 batch_time=0.36211
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.18039 batch_time=0.37174
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.18106 batch_time=0.35444
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.35723 batch_time=0.38039
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.65856 batch_time=0.35518
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.61724 batch_time=0.36426
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 2.33714 batch_time=0.35488
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.59067 batch_time=0.36947
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 2.53911 batch_time=0.36212
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.44469 batch_time=0.34439
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 2.42572 batch_time=0.35082
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.68530 batch_time=0.35718
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch17.pth ...
Done in 4.257s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch17.pth ...
Done in 8.425s
removing stale ckpt [epoch 16] [took 0.00s]
epoch : 17
loss : 2.654444983959198
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 13.9
LSMDC_full_test/t2v_metrics/R5: 32.3
LSMDC_full_test/t2v_metrics/R10: 43.3
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 16.5
LSMDC_full_test/t2v_metrics/MeanR: 69.548
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.888613359070085
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 33.2
LSMDC_full_test/v2t_metrics/R10: 42.4
LSMDC_full_test/v2t_metrics/R50: 69.2
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 66.764
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.216294275346165
mnt_best : 26.888613359070085
not_improved_count: 0
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.77726 batch_time=19.47656
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.60326 batch_time=0.35263
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.85676 batch_time=0.36032
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 3.12733 batch_time=0.34826
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 3.07210 batch_time=0.34741
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.70529 batch_time=0.35490
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.56125 batch_time=0.35634
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 3.07398 batch_time=0.35001
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 2.47984 batch_time=0.34892
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.71874 batch_time=0.34633
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.24502 batch_time=0.34767
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.31206 batch_time=0.35436
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.62216 batch_time=1.87290
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.38705 batch_time=0.35842
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.23946 batch_time=0.34941
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 2.79587 batch_time=0.35018
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.77769 batch_time=0.36087
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.56852 batch_time=0.34559
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.73792 batch_time=0.36781
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.65974 batch_time=0.34902
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 2.55158 batch_time=0.34732
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.62535 batch_time=0.37280
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.21327 batch_time=0.34841
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch18.pth ...
Done in 4.258s
removing stale ckpt [epoch 17] [took 0.01s]
epoch : 18
loss : 2.5940953187942504
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 13.4
LSMDC_full_test/t2v_metrics/R5: 32.9
LSMDC_full_test/t2v_metrics/R10: 42.4
LSMDC_full_test/t2v_metrics/R50: 69.1
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 70.155
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.539262554358594
LSMDC_full_test/v2t_metrics/R1: 13.8
LSMDC_full_test/v2t_metrics/R5: 33.9
LSMDC_full_test/v2t_metrics/R10: 42.5
LSMDC_full_test/v2t_metrics/R50: 67.8
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 70.194
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.090846252343564
mnt_best : 26.888613359070085
not_improved_count: 1
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.58228 batch_time=18.13991
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.82862 batch_time=0.34750
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.10691 batch_time=0.35147
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.92441 batch_time=0.35199
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.41931 batch_time=0.34513
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 2.51188 batch_time=0.34826
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.50775 batch_time=0.49516
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.58558 batch_time=0.35261
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.37614 batch_time=0.35742
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.54380 batch_time=0.35795
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.02447 batch_time=0.36478
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.75740 batch_time=0.34751
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.15129 batch_time=1.33853
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.37559 batch_time=0.34798
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.82618 batch_time=0.36369
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.93576 batch_time=0.35282
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.33966 batch_time=0.34174
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.09048 batch_time=0.37007
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.17589 batch_time=0.35504
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.22068 batch_time=2.17786
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.30802 batch_time=0.81166
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.70492 batch_time=0.34606
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.09255 batch_time=0.35040
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch19.pth ...
Done in 4.154s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch19.pth ...
Done in 9.047s
removing stale ckpt [epoch 18] [took 0.06s]
epoch : 19
loss : 2.495662619113922
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 14.7
LSMDC_full_test/t2v_metrics/R5: 31.2
LSMDC_full_test/t2v_metrics/R10: 42.8
LSMDC_full_test/t2v_metrics/R50: 68.9
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 69.435
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.975648826285862
LSMDC_full_test/v2t_metrics/R1: 14.5
LSMDC_full_test/v2t_metrics/R5: 32.4
LSMDC_full_test/v2t_metrics/R10: 41.7
LSMDC_full_test/v2t_metrics/R50: 68.4
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.699
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.957711578631788
mnt_best : 26.975648826285862
not_improved_count: 0
Train Epoch: 20 [1/250 128/32000 (0%)] Loss: 2.77732 batch_time=19.95871
Train Epoch: 20 [12/250 1536/32000 (5%)] Loss: 2.12006 batch_time=0.36433
Train Epoch: 20 [23/250 2944/32000 (9%)] Loss: 1.97848 batch_time=0.39277
Train Epoch: 20 [34/250 4352/32000 (14%)] Loss: 2.74685 batch_time=0.36508
Train Epoch: 20 [45/250 5760/32000 (18%)] Loss: 2.57989 batch_time=0.36746
Train Epoch: 20 [56/250 7168/32000 (22%)] Loss: 2.19634 batch_time=0.36377
Train Epoch: 20 [67/250 8576/32000 (27%)] Loss: 2.40491 batch_time=0.34450
Train Epoch: 20 [78/250 9984/32000 (31%)] Loss: 2.18941 batch_time=0.38246
Train Epoch: 20 [89/250 11392/32000 (36%)] Loss: 2.63117 batch_time=0.33871
Train Epoch: 20 [100/250 12800/32000 (40%)] Loss: 2.24022 batch_time=0.35629
Train Epoch: 20 [111/250 14208/32000 (44%)] Loss: 2.14475 batch_time=0.34911
Train Epoch: 20 [122/250 15616/32000 (49%)] Loss: 2.78170 batch_time=0.34819
Train Epoch: 20 [133/250 17024/32000 (53%)] Loss: 2.80576 batch_time=0.35061
Train Epoch: 20 [144/250 18432/32000 (58%)] Loss: 2.12152 batch_time=0.37472
Train Epoch: 20 [155/250 19840/32000 (62%)] Loss: 2.55367 batch_time=0.37694
Train Epoch: 20 [166/250 21248/32000 (66%)] Loss: 2.11967 batch_time=0.35112
Train Epoch: 20 [177/250 22656/32000 (71%)] Loss: 2.14456 batch_time=0.34848
Train Epoch: 20 [188/250 24064/32000 (75%)] Loss: 2.05426 batch_time=0.35770
Train Epoch: 20 [199/250 25472/32000 (80%)] Loss: 2.46806 batch_time=0.65784
Train Epoch: 20 [210/250 26880/32000 (84%)] Loss: 1.78277 batch_time=0.37361
Train Epoch: 20 [221/250 28288/32000 (88%)] Loss: 2.69746 batch_time=0.37258
Train Epoch: 20 [232/250 29696/32000 (93%)] Loss: 2.06384 batch_time=0.34987
Train Epoch: 20 [243/250 31104/32000 (97%)] Loss: 1.95002 batch_time=0.34833
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch20.pth ...
Done in 4.632s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCT_LSMDC/checkpoint-epoch20.pth ...
Done in 8.654s
removing stale ckpt [epoch 19] [took 0.02s]
epoch : 20
loss : 2.4067870049476623
learning_rate : 1.8867680126765363e-05
n_samples : 640000
n_steps : 5000
LSMDC_full_test/t2v_metrics/R1: 14.3
LSMDC_full_test/t2v_metrics/R5: 33.0
LSMDC_full_test/t2v_metrics/R10: 42.9
LSMDC_full_test/t2v_metrics/R50: 69.6
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 69.985
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.25434546193282
LSMDC_full_test/v2t_metrics/R1: 14.5
LSMDC_full_test/v2t_metrics/R5: 33.8
LSMDC_full_test/v2t_metrics/R10: 41.9
LSMDC_full_test/v2t_metrics/R50: 69.1
LSMDC_full_test/v2t_metrics/MedR: 17.0
LSMDC_full_test/v2t_metrics/MeanR: 67.386
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.384169554584858
mnt_best : 27.25434546193282
not_improved_count: 0
Train Epoch: 21 [1/250 128/32000 (0%)] Loss: 2.19215 batch_time=20.75905
Train Epoch: 21 [12/250 1536/32000 (5%)] Loss: 2.48424 batch_time=0.35013