-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_LSMDC_M64.txt
2593 lines (2593 loc) · 190 KB
/
HCQ_LSMDC_M64.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64
Preparing the dataloaders ...
Loading dataset LSMDC_full_trainval in ram ...
Finish loading dataset LSMDC_full_trainval in ram, taking 4548.069836139679 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 135.86350059509277 s.
Loading dataset LSMDC_full_test in ram ...
Finish loading dataset LSMDC_full_test in ram, taking 75.9279682636261 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch0.pth ...
Done in 1.817s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch0.pth ...
Done in 3.860s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
LSMDC_full_test/t2v_metrics/R1: 0.1
LSMDC_full_test/t2v_metrics/R5: 0.5
LSMDC_full_test/t2v_metrics/R10: 1.1
LSMDC_full_test/t2v_metrics/R50: 5.5
LSMDC_full_test/t2v_metrics/MedR: 508.0
LSMDC_full_test/t2v_metrics/MeanR: 502.666
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.3802952460761392
LSMDC_full_test/v2t_metrics/R1: 0.1
LSMDC_full_test/v2t_metrics/R5: 0.7
LSMDC_full_test/v2t_metrics/R10: 1.2
LSMDC_full_test/v2t_metrics/R50: 4.8
LSMDC_full_test/v2t_metrics/MedR: 501.5
LSMDC_full_test/v2t_metrics/MeanR: 508.386
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.43795191398878897
mnt_best : 0.3802952460761392
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.80939 (QuantReg: 21.50723) QuantErr: 21.50723 batch_time=20.15861
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 9.20145 (QuantReg: 21.69483) QuantErr: 21.69483 batch_time=5.46332
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 8.82916 (QuantReg: 21.83902) QuantErr: 21.83902 batch_time=0.59084
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 8.21573 (QuantReg: 21.84912) QuantErr: 21.84912 batch_time=0.60848
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 7.92854 (QuantReg: 21.93963) QuantErr: 21.93963 batch_time=0.59204
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 7.50950 (QuantReg: 21.93055) QuantErr: 21.93055 batch_time=0.59270
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 7.38017 (QuantReg: 21.98545) QuantErr: 21.98545 batch_time=0.77448
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 6.95477 (QuantReg: 21.93647) QuantErr: 21.93647 batch_time=2.76200
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 7.04871 (QuantReg: 21.82113) QuantErr: 21.82113 batch_time=0.62297
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 6.70030 (QuantReg: 21.86810) QuantErr: 21.86810 batch_time=0.84531
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 7.23722 (QuantReg: 21.95240) QuantErr: 21.95240 batch_time=0.69280
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 6.24324 (QuantReg: 22.00134) QuantErr: 22.00134 batch_time=0.61327
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 6.73920 (QuantReg: 21.91153) QuantErr: 21.91153 batch_time=0.65933
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 7.05545 (QuantReg: 21.83927) QuantErr: 21.83927 batch_time=0.58629
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 6.58518 (QuantReg: 21.83915) QuantErr: 21.83915 batch_time=0.59608
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 6.26323 (QuantReg: 21.90239) QuantErr: 21.90239 batch_time=0.62580
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 7.09488 (QuantReg: 21.89107) QuantErr: 21.89107 batch_time=1.00919
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 6.35629 (QuantReg: 21.88629) QuantErr: 21.88629 batch_time=0.64094
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 6.26053 (QuantReg: 21.93238) QuantErr: 21.93238 batch_time=0.62148
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 6.49566 (QuantReg: 21.89164) QuantErr: 21.89164 batch_time=0.62853
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 5.94127 (QuantReg: 21.94093) QuantErr: 21.94093 batch_time=0.59746
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 5.81843 (QuantReg: 21.90058) QuantErr: 21.90058 batch_time=0.66135
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 7.08685 (QuantReg: 21.95964) QuantErr: 21.95964 batch_time=0.62655
Train Epoch: 1 codebook_update_time=3.89198
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch1.pth ...
Done in 5.561s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch1.pth ...
Done in 10.506s
epoch : 1
loss : 7.094370021820068
quant_reg : 21.88987370300293
quant_err : 21.88987370300293
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
LSMDC_full_test/t2v_metrics/R1: 8.8
LSMDC_full_test/t2v_metrics/R5: 22.5
LSMDC_full_test/t2v_metrics/R10: 29.8
LSMDC_full_test/t2v_metrics/R50: 56.3
LSMDC_full_test/t2v_metrics/MedR: 36.0
LSMDC_full_test/t2v_metrics/MeanR: 95.564
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.070097038530623
LSMDC_full_test/v2t_metrics/R1: 7.3
LSMDC_full_test/v2t_metrics/R5: 20.2
LSMDC_full_test/v2t_metrics/R10: 28.5
LSMDC_full_test/v2t_metrics/R50: 56.8
LSMDC_full_test/v2t_metrics/MedR: 36.0
LSMDC_full_test/v2t_metrics/MeanR: 99.547
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 16.137627870387092
mnt_best : 18.070097038530623
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 6.03587 (QuantReg: 12.58870) QuantErr: 12.58870 batch_time=21.76808
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 6.10371 (QuantReg: 13.07342) QuantErr: 13.07342 batch_time=0.63422
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 6.18367 (QuantReg: 13.38605) QuantErr: 13.38605 batch_time=0.59443
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 6.01106 (QuantReg: 13.25600) QuantErr: 13.25600 batch_time=0.63164
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 6.26266 (QuantReg: 12.95713) QuantErr: 12.95713 batch_time=0.63217
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 6.56908 (QuantReg: 13.67463) QuantErr: 13.67463 batch_time=0.65718
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 6.33003 (QuantReg: 13.77703) QuantErr: 13.77703 batch_time=2.66602
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 5.43749 (QuantReg: 13.92616) QuantErr: 13.92616 batch_time=0.60067
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 5.95681 (QuantReg: 14.17224) QuantErr: 14.17224 batch_time=0.62712
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 4.95065 (QuantReg: 14.89775) QuantErr: 14.89775 batch_time=0.60095
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 5.61747 (QuantReg: 13.81159) QuantErr: 13.81159 batch_time=0.67851
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 5.72746 (QuantReg: 14.28803) QuantErr: 14.28803 batch_time=0.61226
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 5.93488 (QuantReg: 14.49473) QuantErr: 14.49473 batch_time=5.08825
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 5.84254 (QuantReg: 14.89079) QuantErr: 14.89079 batch_time=0.63635
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 5.91558 (QuantReg: 15.03591) QuantErr: 15.03591 batch_time=0.62239
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 5.42349 (QuantReg: 15.33983) QuantErr: 15.33983 batch_time=0.65051
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 5.37181 (QuantReg: 14.96856) QuantErr: 14.96856 batch_time=0.64751
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 6.26267 (QuantReg: 15.68559) QuantErr: 15.68559 batch_time=0.60170
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 6.15957 (QuantReg: 15.52229) QuantErr: 15.52229 batch_time=0.60076
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 5.56102 (QuantReg: 15.69506) QuantErr: 15.69506 batch_time=1.20990
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 5.38072 (QuantReg: 15.08682) QuantErr: 15.08682 batch_time=1.07175
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 5.84900 (QuantReg: 15.91316) QuantErr: 15.91316 batch_time=0.61503
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 5.75692 (QuantReg: 15.71219) QuantErr: 15.71219 batch_time=0.59166
Train Epoch: 2 codebook_update_time=3.63576
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch2.pth ...
Done in 5.198s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch2.pth ...
Done in 10.338s
removing stale ckpt [epoch 1] [took 0.03s]
removing stale ckpt [epoch 0] [took 0.04s]
epoch : 2
loss : 5.797973737716675
quant_reg : 14.555635604858399
quant_err : 14.555635604858399
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
LSMDC_full_test/t2v_metrics/R1: 8.6
LSMDC_full_test/t2v_metrics/R5: 23.7
LSMDC_full_test/t2v_metrics/R10: 33.1
LSMDC_full_test/t2v_metrics/R50: 62.5
LSMDC_full_test/t2v_metrics/MedR: 29.0
LSMDC_full_test/t2v_metrics/MeanR: 82.915
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 18.895494572889277
LSMDC_full_test/v2t_metrics/R1: 8.7
LSMDC_full_test/v2t_metrics/R5: 23.8
LSMDC_full_test/v2t_metrics/R10: 33.4
LSMDC_full_test/v2t_metrics/R50: 62.5
LSMDC_full_test/v2t_metrics/MedR: 27.0
LSMDC_full_test/v2t_metrics/MeanR: 81.575
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 19.05230646979192
mnt_best : 18.895494572889277
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 5.75762 (QuantReg: 13.56362) QuantErr: 13.56362 batch_time=22.93302
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 5.17754 (QuantReg: 13.57047) QuantErr: 13.57047 batch_time=0.65909
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 5.66675 (QuantReg: 13.49056) QuantErr: 13.49056 batch_time=0.64018
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 5.32214 (QuantReg: 13.84505) QuantErr: 13.84505 batch_time=0.61202
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 5.09510 (QuantReg: 14.14375) QuantErr: 14.14375 batch_time=0.58662
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 5.26778 (QuantReg: 14.10199) QuantErr: 14.10199 batch_time=0.64499
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 5.17747 (QuantReg: 14.33012) QuantErr: 14.33012 batch_time=0.59563
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 5.79423 (QuantReg: 13.72953) QuantErr: 13.72953 batch_time=0.68252
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 5.27020 (QuantReg: 14.08873) QuantErr: 14.08873 batch_time=0.62157
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 5.03006 (QuantReg: 14.59556) QuantErr: 14.59556 batch_time=0.59480
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 5.15775 (QuantReg: 13.95027) QuantErr: 13.95027 batch_time=0.60080
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 5.15056 (QuantReg: 14.57181) QuantErr: 14.57181 batch_time=0.66321
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 5.18947 (QuantReg: 14.06903) QuantErr: 14.06903 batch_time=0.65930
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 5.41306 (QuantReg: 14.62565) QuantErr: 14.62565 batch_time=2.74612
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 4.86474 (QuantReg: 14.55362) QuantErr: 14.55362 batch_time=0.62060
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 5.30767 (QuantReg: 14.40300) QuantErr: 14.40300 batch_time=0.71544
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 4.95999 (QuantReg: 15.19110) QuantErr: 15.19110 batch_time=0.60308
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 5.37529 (QuantReg: 14.38105) QuantErr: 14.38105 batch_time=0.64262
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 4.88398 (QuantReg: 15.31916) QuantErr: 15.31916 batch_time=0.61090
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 5.43234 (QuantReg: 15.07749) QuantErr: 15.07749 batch_time=0.59698
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 5.00561 (QuantReg: 14.69384) QuantErr: 14.69384 batch_time=0.61744
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 5.05726 (QuantReg: 15.33752) QuantErr: 15.33752 batch_time=0.60896
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 5.04025 (QuantReg: 14.97869) QuantErr: 14.97869 batch_time=0.60358
Train Epoch: 3 codebook_update_time=4.81293
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch3.pth ...
Done in 6.730s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch3.pth ...
Done in 11.749s
removing stale ckpt [epoch 2] [took 0.01s]
epoch : 3
loss : 5.284278848648071
quant_reg : 14.330292221069335
quant_err : 14.330292221069335
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
LSMDC_full_test/t2v_metrics/R1: 9.4
LSMDC_full_test/t2v_metrics/R5: 25.9
LSMDC_full_test/t2v_metrics/R10: 35.3
LSMDC_full_test/t2v_metrics/R50: 62.7
LSMDC_full_test/t2v_metrics/MedR: 26.0
LSMDC_full_test/t2v_metrics/MeanR: 79.597
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.483340023798075
LSMDC_full_test/v2t_metrics/R1: 9.8
LSMDC_full_test/v2t_metrics/R5: 25.8
LSMDC_full_test/v2t_metrics/R10: 35.6
LSMDC_full_test/v2t_metrics/R50: 62.2
LSMDC_full_test/v2t_metrics/MedR: 28.0
LSMDC_full_test/v2t_metrics/MeanR: 81.354
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 20.801688718907673
mnt_best : 20.483340023798075
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 5.25020 (QuantReg: 13.65676) QuantErr: 13.65676 batch_time=21.66419
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 5.02504 (QuantReg: 13.96922) QuantErr: 13.96922 batch_time=0.64610
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 5.36449 (QuantReg: 13.93103) QuantErr: 13.93103 batch_time=0.64831
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 5.00527 (QuantReg: 14.35615) QuantErr: 14.35615 batch_time=0.61181
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 4.46455 (QuantReg: 14.59855) QuantErr: 14.59855 batch_time=0.62097
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 4.45674 (QuantReg: 14.48962) QuantErr: 14.48962 batch_time=0.66315
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 4.99637 (QuantReg: 14.44765) QuantErr: 14.44765 batch_time=1.24161
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 5.35893 (QuantReg: 14.33401) QuantErr: 14.33401 batch_time=1.54538
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 4.36580 (QuantReg: 14.20604) QuantErr: 14.20604 batch_time=0.62011
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 4.96075 (QuantReg: 14.03092) QuantErr: 14.03092 batch_time=0.61012
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 5.59432 (QuantReg: 14.01943) QuantErr: 14.01943 batch_time=0.65051
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 4.32961 (QuantReg: 14.51213) QuantErr: 14.51213 batch_time=0.73505
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 5.06750 (QuantReg: 14.35395) QuantErr: 14.35395 batch_time=0.59781
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 5.08703 (QuantReg: 14.20726) QuantErr: 14.20726 batch_time=0.63761
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 5.15025 (QuantReg: 14.21748) QuantErr: 14.21748 batch_time=0.59899
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 5.44574 (QuantReg: 14.37131) QuantErr: 14.37131 batch_time=0.67961
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 4.65531 (QuantReg: 14.56476) QuantErr: 14.56476 batch_time=0.66586
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 5.39964 (QuantReg: 14.50447) QuantErr: 14.50447 batch_time=0.60611
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 4.47914 (QuantReg: 15.12387) QuantErr: 15.12387 batch_time=0.65372
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 4.60303 (QuantReg: 14.84901) QuantErr: 14.84901 batch_time=0.65184
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 3.79207 (QuantReg: 15.31919) QuantErr: 15.31919 batch_time=1.09837
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 4.58791 (QuantReg: 15.39709) QuantErr: 15.39709 batch_time=0.62357
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 4.80676 (QuantReg: 15.06718) QuantErr: 15.06718 batch_time=0.69180
Train Epoch: 4 codebook_update_time=3.37056
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch4.pth ...
Done in 23.466s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch4.pth ...
Done in 28.475s
removing stale ckpt [epoch 3] [took 0.02s]
epoch : 4
loss : 4.919483083724976
quant_reg : 14.461428375244141
quant_err : 14.461428375244141
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
LSMDC_full_test/t2v_metrics/R1: 11.0
LSMDC_full_test/t2v_metrics/R5: 26.8
LSMDC_full_test/t2v_metrics/R10: 36.4
LSMDC_full_test/t2v_metrics/R50: 64.9
LSMDC_full_test/t2v_metrics/MedR: 23.5
LSMDC_full_test/t2v_metrics/MeanR: 75.319
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.056822805556127
LSMDC_full_test/v2t_metrics/R1: 11.0
LSMDC_full_test/v2t_metrics/R5: 27.7
LSMDC_full_test/v2t_metrics/R10: 37.3
LSMDC_full_test/v2t_metrics/R50: 63.7
LSMDC_full_test/v2t_metrics/MedR: 26.0
LSMDC_full_test/v2t_metrics/MeanR: 74.926
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 22.48331932385629
mnt_best : 22.056822805556127
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 5.63942 (QuantReg: 13.84764) QuantErr: 13.84764 batch_time=21.21771
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 4.45577 (QuantReg: 14.37562) QuantErr: 14.37562 batch_time=0.60702
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 4.81076 (QuantReg: 14.71744) QuantErr: 14.71744 batch_time=0.59861
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 5.11417 (QuantReg: 14.62190) QuantErr: 14.62190 batch_time=0.61744
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 4.82773 (QuantReg: 14.79443) QuantErr: 14.79443 batch_time=0.61932
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 4.22583 (QuantReg: 14.83019) QuantErr: 14.83019 batch_time=0.72585
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 4.53211 (QuantReg: 14.61391) QuantErr: 14.61391 batch_time=0.62413
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 4.96427 (QuantReg: 14.74033) QuantErr: 14.74033 batch_time=0.60845
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 5.27833 (QuantReg: 14.71523) QuantErr: 14.71523 batch_time=1.38887
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 4.98063 (QuantReg: 14.67339) QuantErr: 14.67339 batch_time=0.61248
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 4.83694 (QuantReg: 14.85113) QuantErr: 14.85113 batch_time=0.62972
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 4.56018 (QuantReg: 14.93082) QuantErr: 14.93082 batch_time=0.58533
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 4.22045 (QuantReg: 15.10076) QuantErr: 15.10076 batch_time=0.64715
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 4.55519 (QuantReg: 15.18007) QuantErr: 15.18007 batch_time=0.65000
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 4.98604 (QuantReg: 15.24814) QuantErr: 15.24814 batch_time=1.27440
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 4.84545 (QuantReg: 14.94019) QuantErr: 14.94019 batch_time=0.62890
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 4.22298 (QuantReg: 15.35087) QuantErr: 15.35087 batch_time=0.64104
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 4.35507 (QuantReg: 15.04648) QuantErr: 15.04648 batch_time=0.76592
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 4.41076 (QuantReg: 15.21437) QuantErr: 15.21437 batch_time=0.63094
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 4.54899 (QuantReg: 15.17618) QuantErr: 15.17618 batch_time=1.34943
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 4.23538 (QuantReg: 15.31025) QuantErr: 15.31025 batch_time=0.62106
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 4.40693 (QuantReg: 15.76517) QuantErr: 15.76517 batch_time=0.59716
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 3.92403 (QuantReg: 15.41035) QuantErr: 15.41035 batch_time=0.65735
Train Epoch: 5 codebook_update_time=3.45709
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch5.pth ...
Done in 5.215s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch5.pth ...
Done in 9.943s
removing stale ckpt [epoch 4] [took 0.01s]
epoch : 5
loss : 4.595277821540832
quant_reg : 14.956194988250733
quant_err : 14.956194988250733
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
LSMDC_full_test/t2v_metrics/R1: 12.2
LSMDC_full_test/t2v_metrics/R5: 28.8
LSMDC_full_test/t2v_metrics/R10: 36.8
LSMDC_full_test/t2v_metrics/R50: 64.6
LSMDC_full_test/t2v_metrics/MedR: 24.0
LSMDC_full_test/t2v_metrics/MeanR: 74.157
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.471096500728102
LSMDC_full_test/v2t_metrics/R1: 12.5
LSMDC_full_test/v2t_metrics/R5: 27.6
LSMDC_full_test/v2t_metrics/R10: 37.5
LSMDC_full_test/v2t_metrics/R50: 65.3
LSMDC_full_test/v2t_metrics/MedR: 24.5
LSMDC_full_test/v2t_metrics/MeanR: 73.162
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.475604682508873
mnt_best : 23.471096500728102
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 5.03653 (QuantReg: 14.62634) QuantErr: 14.62634 batch_time=20.75604
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 4.49822 (QuantReg: 14.62168) QuantErr: 14.62168 batch_time=0.65637
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 4.76789 (QuantReg: 14.74530) QuantErr: 14.74530 batch_time=2.00350
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 4.20257 (QuantReg: 15.06062) QuantErr: 15.06062 batch_time=0.60724
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 4.27723 (QuantReg: 14.93658) QuantErr: 14.93658 batch_time=0.61461
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 4.61805 (QuantReg: 14.67464) QuantErr: 14.67464 batch_time=0.65881
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 4.24508 (QuantReg: 14.83317) QuantErr: 14.83317 batch_time=0.81996
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 4.35081 (QuantReg: 15.16662) QuantErr: 15.16662 batch_time=0.66755
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 4.95208 (QuantReg: 14.60468) QuantErr: 14.60468 batch_time=0.72913
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 4.74091 (QuantReg: 15.05026) QuantErr: 15.05026 batch_time=0.61351
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 3.93909 (QuantReg: 15.17683) QuantErr: 15.17683 batch_time=0.62560
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 4.31076 (QuantReg: 14.95657) QuantErr: 14.95657 batch_time=0.72224
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 4.25432 (QuantReg: 15.24453) QuantErr: 15.24453 batch_time=0.61967
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 5.07961 (QuantReg: 15.05023) QuantErr: 15.05023 batch_time=0.78309
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 4.03257 (QuantReg: 15.46739) QuantErr: 15.46739 batch_time=0.60711
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 3.70312 (QuantReg: 15.11369) QuantErr: 15.11369 batch_time=0.67353
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 4.10128 (QuantReg: 15.58476) QuantErr: 15.58476 batch_time=0.64445
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 4.32364 (QuantReg: 15.30817) QuantErr: 15.30817 batch_time=0.72761
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 4.78047 (QuantReg: 15.27106) QuantErr: 15.27106 batch_time=0.65385
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 4.24618 (QuantReg: 15.51452) QuantErr: 15.51452 batch_time=0.64079
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 4.06327 (QuantReg: 15.56700) QuantErr: 15.56700 batch_time=0.60139
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 4.87015 (QuantReg: 14.88259) QuantErr: 14.88259 batch_time=2.14023
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 4.59205 (QuantReg: 15.04645) QuantErr: 15.04645 batch_time=0.61854
Train Epoch: 6 codebook_update_time=3.39500
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch6.pth ...
Done in 5.364s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch6.pth ...
Done in 10.496s
removing stale ckpt [epoch 5] [took 0.02s]
epoch : 6
loss : 4.396556546211243
quant_reg : 15.114287963867188
quant_err : 15.114287963867188
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
LSMDC_full_test/t2v_metrics/R1: 11.9
LSMDC_full_test/t2v_metrics/R5: 30.6
LSMDC_full_test/t2v_metrics/R10: 38.5
LSMDC_full_test/t2v_metrics/R50: 65.9
LSMDC_full_test/t2v_metrics/MedR: 22.0
LSMDC_full_test/t2v_metrics/MeanR: 71.468
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.11254433227066
LSMDC_full_test/v2t_metrics/R1: 11.2
LSMDC_full_test/v2t_metrics/R5: 28.7
LSMDC_full_test/v2t_metrics/R10: 38.0
LSMDC_full_test/v2t_metrics/R50: 65.0
LSMDC_full_test/v2t_metrics/MedR: 20.5
LSMDC_full_test/v2t_metrics/MeanR: 73.764
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 23.030030087142684
mnt_best : 24.11254433227066
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 4.35151 (QuantReg: 15.12430) QuantErr: 15.12430 batch_time=17.79428
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 4.05599 (QuantReg: 15.29338) QuantErr: 15.29338 batch_time=0.62048
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 4.06852 (QuantReg: 15.03606) QuantErr: 15.03606 batch_time=0.86814
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 3.99352 (QuantReg: 15.25795) QuantErr: 15.25795 batch_time=0.61511
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 4.76498 (QuantReg: 15.24959) QuantErr: 15.24959 batch_time=0.60834
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 4.13408 (QuantReg: 15.37472) QuantErr: 15.37472 batch_time=0.61645
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 4.22251 (QuantReg: 15.16792) QuantErr: 15.16792 batch_time=0.70762
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 4.49705 (QuantReg: 15.17070) QuantErr: 15.17070 batch_time=0.63517
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 4.05532 (QuantReg: 15.02098) QuantErr: 15.02098 batch_time=0.66333
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 4.16352 (QuantReg: 15.20739) QuantErr: 15.20739 batch_time=0.68805
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 4.01010 (QuantReg: 15.57983) QuantErr: 15.57983 batch_time=0.62811
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 4.34322 (QuantReg: 15.33084) QuantErr: 15.33084 batch_time=0.59867
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 4.04306 (QuantReg: 15.47928) QuantErr: 15.47928 batch_time=0.62935
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 3.84205 (QuantReg: 15.53749) QuantErr: 15.53749 batch_time=1.08514
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 4.06867 (QuantReg: 15.63829) QuantErr: 15.63829 batch_time=0.80503
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 3.74773 (QuantReg: 15.40965) QuantErr: 15.40965 batch_time=0.61706
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 4.21131 (QuantReg: 15.33831) QuantErr: 15.33831 batch_time=0.62810
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 4.17377 (QuantReg: 15.52669) QuantErr: 15.52669 batch_time=0.64706
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 3.80440 (QuantReg: 15.42471) QuantErr: 15.42471 batch_time=0.66995
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 3.85412 (QuantReg: 15.51505) QuantErr: 15.51505 batch_time=0.68202
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 3.81949 (QuantReg: 15.60890) QuantErr: 15.60890 batch_time=0.76354
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 3.86273 (QuantReg: 15.72789) QuantErr: 15.72789 batch_time=0.66417
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 3.87481 (QuantReg: 15.64023) QuantErr: 15.64023 batch_time=0.65391
Train Epoch: 7 codebook_update_time=3.78238
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch7.pth ...
Done in 4.810s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch7.pth ...
Done in 9.491s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 4.152259581565857
quant_reg : 15.377689720153809
quant_err : 15.377689720153809
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
LSMDC_full_test/t2v_metrics/R1: 12.9
LSMDC_full_test/t2v_metrics/R5: 31.0
LSMDC_full_test/t2v_metrics/R10: 41.0
LSMDC_full_test/t2v_metrics/R50: 67.8
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 69.544
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.404564455501863
LSMDC_full_test/v2t_metrics/R1: 13.1
LSMDC_full_test/v2t_metrics/R5: 30.2
LSMDC_full_test/v2t_metrics/R10: 40.8
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 67.9205
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.27237946612986
mnt_best : 25.404564455501863
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 3.98943 (QuantReg: 15.29104) QuantErr: 15.29104 batch_time=21.77599
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 3.89192 (QuantReg: 15.54230) QuantErr: 15.54230 batch_time=0.69932
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 4.05815 (QuantReg: 15.59001) QuantErr: 15.59001 batch_time=0.61399
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 4.51471 (QuantReg: 15.42123) QuantErr: 15.42123 batch_time=0.61153
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 3.64792 (QuantReg: 15.59848) QuantErr: 15.59848 batch_time=0.61675
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 4.04147 (QuantReg: 15.37495) QuantErr: 15.37495 batch_time=0.63612
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 3.91549 (QuantReg: 15.35633) QuantErr: 15.35633 batch_time=2.46265
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 4.36769 (QuantReg: 15.68460) QuantErr: 15.68460 batch_time=0.66170
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 3.79196 (QuantReg: 15.67974) QuantErr: 15.67974 batch_time=0.69758
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 4.05032 (QuantReg: 15.37320) QuantErr: 15.37320 batch_time=0.65042
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 4.31556 (QuantReg: 15.33080) QuantErr: 15.33080 batch_time=0.63656
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 3.84168 (QuantReg: 15.63860) QuantErr: 15.63860 batch_time=0.60658
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 3.87941 (QuantReg: 15.63982) QuantErr: 15.63982 batch_time=0.59827
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 3.53796 (QuantReg: 15.91034) QuantErr: 15.91034 batch_time=1.40023
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 4.18623 (QuantReg: 15.73014) QuantErr: 15.73014 batch_time=0.63764
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 3.70738 (QuantReg: 15.70894) QuantErr: 15.70894 batch_time=0.67875
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 3.91764 (QuantReg: 15.59819) QuantErr: 15.59819 batch_time=0.62300
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 3.81446 (QuantReg: 15.63447) QuantErr: 15.63447 batch_time=0.60208
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 3.82503 (QuantReg: 15.54389) QuantErr: 15.54389 batch_time=0.61603
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 4.14689 (QuantReg: 15.56530) QuantErr: 15.56530 batch_time=0.65013
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 3.63302 (QuantReg: 15.86120) QuantErr: 15.86120 batch_time=0.65104
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 3.74455 (QuantReg: 16.00459) QuantErr: 16.00459 batch_time=0.71921
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 3.78095 (QuantReg: 15.76544) QuantErr: 15.76544 batch_time=0.64161
Train Epoch: 8 codebook_update_time=3.86332
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch8.pth ...
Done in 6.624s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch8.pth ...
Done in 11.217s
removing stale ckpt [epoch 7] [took 0.02s]
epoch : 8
loss : 3.9615639123916626
quant_reg : 15.582129230499268
quant_err : 15.582129230499268
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
LSMDC_full_test/t2v_metrics/R1: 14.5
LSMDC_full_test/t2v_metrics/R5: 29.7
LSMDC_full_test/t2v_metrics/R10: 38.1
LSMDC_full_test/t2v_metrics/R50: 67.1
LSMDC_full_test/t2v_metrics/MedR: 20.0
LSMDC_full_test/t2v_metrics/MeanR: 71.052
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.410691037493997
LSMDC_full_test/v2t_metrics/R1: 12.9
LSMDC_full_test/v2t_metrics/R5: 31.0
LSMDC_full_test/v2t_metrics/R10: 39.8
LSMDC_full_test/v2t_metrics/R50: 65.8
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 71.107
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.15425690136327
mnt_best : 25.410691037493997
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 3.81747 (QuantReg: 15.17391) QuantErr: 15.17391 batch_time=21.31654
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 3.91653 (QuantReg: 15.56689) QuantErr: 15.56689 batch_time=0.60439
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 3.57506 (QuantReg: 15.65508) QuantErr: 15.65508 batch_time=0.61386
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 4.28322 (QuantReg: 15.95987) QuantErr: 15.95987 batch_time=0.64765
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 4.05886 (QuantReg: 15.58669) QuantErr: 15.58669 batch_time=0.67505
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 3.53557 (QuantReg: 16.08935) QuantErr: 16.08935 batch_time=0.61233
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 3.74263 (QuantReg: 15.71798) QuantErr: 15.71798 batch_time=0.61617
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 4.26034 (QuantReg: 15.40934) QuantErr: 15.40934 batch_time=0.65609
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 3.65544 (QuantReg: 15.48879) QuantErr: 15.48879 batch_time=0.66050
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 3.47209 (QuantReg: 15.88784) QuantErr: 15.88784 batch_time=0.76303
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 4.33936 (QuantReg: 15.65831) QuantErr: 15.65831 batch_time=0.60361
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 3.78715 (QuantReg: 15.77707) QuantErr: 15.77707 batch_time=0.60522
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 3.55606 (QuantReg: 15.73643) QuantErr: 15.73643 batch_time=0.62368
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 3.67561 (QuantReg: 15.93407) QuantErr: 15.93407 batch_time=2.58334
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 3.81574 (QuantReg: 15.83345) QuantErr: 15.83345 batch_time=0.67307
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 3.91845 (QuantReg: 16.02037) QuantErr: 16.02037 batch_time=0.62895
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 3.47643 (QuantReg: 16.05167) QuantErr: 16.05167 batch_time=0.63089
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 3.54673 (QuantReg: 15.52814) QuantErr: 15.52814 batch_time=0.61747
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 3.83869 (QuantReg: 15.65131) QuantErr: 15.65131 batch_time=0.86983
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 3.73791 (QuantReg: 16.03139) QuantErr: 16.03139 batch_time=0.59559
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 4.27496 (QuantReg: 15.95800) QuantErr: 15.95800 batch_time=0.66518
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 4.36383 (QuantReg: 15.81182) QuantErr: 15.81182 batch_time=0.61443
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 3.78065 (QuantReg: 15.69777) QuantErr: 15.69777 batch_time=0.64328
Train Epoch: 9 codebook_update_time=3.40412
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch9.pth ...
Done in 5.228s
removing stale ckpt [epoch 8] [took 0.00s]
epoch : 9
loss : 3.7766191720962525
quant_reg : 15.77045654296875
quant_err : 15.77045654296875
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
LSMDC_full_test/t2v_metrics/R1: 13.3
LSMDC_full_test/t2v_metrics/R5: 30.2
LSMDC_full_test/t2v_metrics/R10: 40.5
LSMDC_full_test/t2v_metrics/R50: 68.0
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 67.999
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.33793410592613
LSMDC_full_test/v2t_metrics/R1: 13.3
LSMDC_full_test/v2t_metrics/R5: 31.5
LSMDC_full_test/v2t_metrics/R10: 39.5
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 20.0
LSMDC_full_test/v2t_metrics/MeanR: 67.392
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.483149190119484
mnt_best : 25.410691037493997
not_improved_count: 1
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 3.57447 (QuantReg: 15.79362) QuantErr: 15.79362 batch_time=21.49230
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 3.55687 (QuantReg: 15.46526) QuantErr: 15.46526 batch_time=0.66563
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 3.62568 (QuantReg: 15.58572) QuantErr: 15.58572 batch_time=0.66810
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 3.47733 (QuantReg: 15.81688) QuantErr: 15.81688 batch_time=0.70803
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 3.85268 (QuantReg: 15.83300) QuantErr: 15.83300 batch_time=0.60750
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 3.32282 (QuantReg: 15.92708) QuantErr: 15.92708 batch_time=0.64295
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 3.68335 (QuantReg: 15.87213) QuantErr: 15.87213 batch_time=0.61046
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 3.54603 (QuantReg: 15.72868) QuantErr: 15.72868 batch_time=0.63387
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 3.78630 (QuantReg: 16.15654) QuantErr: 16.15654 batch_time=0.63220
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 3.82940 (QuantReg: 15.97391) QuantErr: 15.97391 batch_time=0.60852
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 3.63567 (QuantReg: 16.04628) QuantErr: 16.04628 batch_time=0.62689
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 3.67031 (QuantReg: 15.85475) QuantErr: 15.85475 batch_time=0.68480
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 3.35663 (QuantReg: 15.92306) QuantErr: 15.92306 batch_time=0.61712
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 3.69122 (QuantReg: 15.99301) QuantErr: 15.99301 batch_time=0.64071
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 4.02619 (QuantReg: 15.91065) QuantErr: 15.91065 batch_time=1.26652
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 3.81357 (QuantReg: 16.10884) QuantErr: 16.10884 batch_time=0.63132
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 3.78710 (QuantReg: 15.81614) QuantErr: 15.81614 batch_time=0.67004
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 3.69822 (QuantReg: 16.16249) QuantErr: 16.16249 batch_time=0.62095
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 3.23135 (QuantReg: 15.87728) QuantErr: 15.87728 batch_time=0.68239
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 3.36323 (QuantReg: 15.97142) QuantErr: 15.97142 batch_time=0.66018
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 3.23552 (QuantReg: 16.01777) QuantErr: 16.01777 batch_time=0.63073
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 4.18253 (QuantReg: 15.98098) QuantErr: 15.98098 batch_time=0.63019
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 3.73443 (QuantReg: 15.93927) QuantErr: 15.93927 batch_time=0.63882
Train Epoch: 10 codebook_update_time=3.31837
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch10.pth ...
Done in 5.590s
removing stale ckpt [epoch 9] [took 0.01s]
epoch : 10
loss : 3.6227984437942506
quant_reg : 15.91489729309082
quant_err : 15.91489729309082
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
LSMDC_full_test/t2v_metrics/R1: 12.4
LSMDC_full_test/t2v_metrics/R5: 30.2
LSMDC_full_test/t2v_metrics/R10: 39.4
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 19.0
LSMDC_full_test/t2v_metrics/MeanR: 67.205
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.526841054095225
LSMDC_full_test/v2t_metrics/R1: 12.3
LSMDC_full_test/v2t_metrics/R5: 31.1
LSMDC_full_test/v2t_metrics/R10: 39.7
LSMDC_full_test/v2t_metrics/R50: 67.2
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 67.437
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.763878758111545
mnt_best : 25.410691037493997
not_improved_count: 2
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 3.04075 (QuantReg: 16.15651) QuantErr: 16.15651 batch_time=22.60684
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 3.62764 (QuantReg: 15.90143) QuantErr: 15.90143 batch_time=0.60457
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 3.51704 (QuantReg: 15.69925) QuantErr: 15.69925 batch_time=0.69745
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 3.45290 (QuantReg: 15.86111) QuantErr: 15.86111 batch_time=0.60689
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 3.56112 (QuantReg: 15.88993) QuantErr: 15.88993 batch_time=0.62289
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 4.12360 (QuantReg: 15.89765) QuantErr: 15.89765 batch_time=0.75989
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 3.71094 (QuantReg: 15.94250) QuantErr: 15.94250 batch_time=0.62590
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 3.54606 (QuantReg: 16.11891) QuantErr: 16.11891 batch_time=0.60174
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 3.53473 (QuantReg: 16.09405) QuantErr: 16.09405 batch_time=0.63276
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 3.09013 (QuantReg: 16.06941) QuantErr: 16.06941 batch_time=0.63383
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 3.60831 (QuantReg: 16.36330) QuantErr: 16.36330 batch_time=0.89472
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 2.95772 (QuantReg: 16.19193) QuantErr: 16.19193 batch_time=0.69259
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 3.67671 (QuantReg: 16.05065) QuantErr: 16.05065 batch_time=0.64587
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 2.84983 (QuantReg: 16.08867) QuantErr: 16.08867 batch_time=0.67219
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 3.43726 (QuantReg: 16.06998) QuantErr: 16.06998 batch_time=0.61046
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 4.01764 (QuantReg: 16.09876) QuantErr: 16.09876 batch_time=0.60816
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 3.03791 (QuantReg: 16.33892) QuantErr: 16.33892 batch_time=1.36883
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 3.33466 (QuantReg: 16.22872) QuantErr: 16.22872 batch_time=0.62112
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 3.55232 (QuantReg: 16.04528) QuantErr: 16.04528 batch_time=0.78280
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 3.74838 (QuantReg: 16.14415) QuantErr: 16.14415 batch_time=0.68576
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 3.44104 (QuantReg: 15.88321) QuantErr: 15.88321 batch_time=0.68953
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 3.03603 (QuantReg: 16.27282) QuantErr: 16.27282 batch_time=0.62482
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 3.41805 (QuantReg: 15.96244) QuantErr: 15.96244 batch_time=0.62738
Train Epoch: 11 codebook_update_time=3.58941
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch11.pth ...
Done in 4.233s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch11.pth ...
Done in 9.110s
removing stale ckpt [epoch 10] [took 0.04s]
epoch : 11
loss : 3.450694748878479
quant_reg : 16.081873298645018
quant_err : 16.081873298645018
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
LSMDC_full_test/t2v_metrics/R1: 13.4
LSMDC_full_test/t2v_metrics/R5: 29.9
LSMDC_full_test/t2v_metrics/R10: 41.1
LSMDC_full_test/t2v_metrics/R50: 68.0
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 68.346
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.441298265496325
LSMDC_full_test/v2t_metrics/R1: 13.1
LSMDC_full_test/v2t_metrics/R5: 30.6
LSMDC_full_test/v2t_metrics/R10: 39.7
LSMDC_full_test/v2t_metrics/R50: 67.5
LSMDC_full_test/v2t_metrics/MedR: 21.0
LSMDC_full_test/v2t_metrics/MeanR: 70.099
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.153267509279317
mnt_best : 25.441298265496325
not_improved_count: 0
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 3.55505 (QuantReg: 15.90794) QuantErr: 15.90794 batch_time=20.52096
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 3.09475 (QuantReg: 16.15618) QuantErr: 16.15618 batch_time=0.93563
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 3.82736 (QuantReg: 15.95611) QuantErr: 15.95611 batch_time=0.61011
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 2.92395 (QuantReg: 16.13934) QuantErr: 16.13934 batch_time=0.66495
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 2.93632 (QuantReg: 16.27385) QuantErr: 16.27385 batch_time=0.59726
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 3.16501 (QuantReg: 16.18888) QuantErr: 16.18888 batch_time=0.63880
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 3.45879 (QuantReg: 16.21455) QuantErr: 16.21455 batch_time=0.60305
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 3.42467 (QuantReg: 15.89923) QuantErr: 15.89923 batch_time=0.63127
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 3.32715 (QuantReg: 16.12108) QuantErr: 16.12108 batch_time=0.61543
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 3.22592 (QuantReg: 16.26954) QuantErr: 16.26954 batch_time=0.62079
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 3.44207 (QuantReg: 16.16871) QuantErr: 16.16871 batch_time=0.60479
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 3.22154 (QuantReg: 16.24833) QuantErr: 16.24833 batch_time=0.68452
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 3.73923 (QuantReg: 16.07605) QuantErr: 16.07605 batch_time=0.62408
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 3.58869 (QuantReg: 16.08292) QuantErr: 16.08292 batch_time=5.36653
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 3.41633 (QuantReg: 16.01112) QuantErr: 16.01112 batch_time=0.61451
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 3.29426 (QuantReg: 16.24463) QuantErr: 16.24463 batch_time=0.66054
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 2.56568 (QuantReg: 16.41540) QuantErr: 16.41540 batch_time=0.69116
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 3.77214 (QuantReg: 16.04387) QuantErr: 16.04387 batch_time=0.62796
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 2.85304 (QuantReg: 16.22714) QuantErr: 16.22714 batch_time=0.64351
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 3.66504 (QuantReg: 16.10003) QuantErr: 16.10003 batch_time=0.61478
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 3.29550 (QuantReg: 16.35716) QuantErr: 16.35716 batch_time=0.61389
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 3.05523 (QuantReg: 16.34845) QuantErr: 16.34845 batch_time=0.67723
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 3.23841 (QuantReg: 16.37490) QuantErr: 16.37490 batch_time=0.61223
Train Epoch: 12 codebook_update_time=3.44821
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch12.pth ...
Done in 6.782s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch12.pth ...
Done in 11.095s
removing stale ckpt [epoch 11] [took 0.02s]
epoch : 12
loss : 3.322868664741516
quant_reg : 16.147703563690186
quant_err : 16.147703563690186
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
LSMDC_full_test/t2v_metrics/R1: 13.9
LSMDC_full_test/t2v_metrics/R5: 31.1
LSMDC_full_test/t2v_metrics/R10: 40.6
LSMDC_full_test/t2v_metrics/R50: 68.5
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 69.099
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.98765390169758
LSMDC_full_test/v2t_metrics/R1: 12.8
LSMDC_full_test/v2t_metrics/R5: 31.0
LSMDC_full_test/v2t_metrics/R10: 40.5
LSMDC_full_test/v2t_metrics/R50: 67.7
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 65.807
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.235324609537045
mnt_best : 25.98765390169758
not_improved_count: 0
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 2.91023 (QuantReg: 16.23493) QuantErr: 16.23493 batch_time=23.78021
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 3.47701 (QuantReg: 16.02004) QuantErr: 16.02004 batch_time=0.88969
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 3.32791 (QuantReg: 16.33929) QuantErr: 16.33929 batch_time=0.61620
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 3.04869 (QuantReg: 16.29410) QuantErr: 16.29410 batch_time=0.62890
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 2.90995 (QuantReg: 16.12414) QuantErr: 16.12414 batch_time=0.60794
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 3.54431 (QuantReg: 16.05685) QuantErr: 16.05685 batch_time=0.65118
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 3.17857 (QuantReg: 16.21957) QuantErr: 16.21957 batch_time=0.60808
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 3.22663 (QuantReg: 16.43378) QuantErr: 16.43378 batch_time=0.60957
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 2.81885 (QuantReg: 16.29154) QuantErr: 16.29154 batch_time=0.97178
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 3.40971 (QuantReg: 16.16086) QuantErr: 16.16086 batch_time=0.66560
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 3.39077 (QuantReg: 16.32481) QuantErr: 16.32481 batch_time=0.65351
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 3.12796 (QuantReg: 16.42521) QuantErr: 16.42521 batch_time=0.61087
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 3.40316 (QuantReg: 16.06327) QuantErr: 16.06327 batch_time=0.60732
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 3.48661 (QuantReg: 16.25510) QuantErr: 16.25510 batch_time=0.64853
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 3.30102 (QuantReg: 16.46553) QuantErr: 16.46553 batch_time=0.62884
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 3.04639 (QuantReg: 16.50259) QuantErr: 16.50259 batch_time=0.65983
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 3.00123 (QuantReg: 16.15833) QuantErr: 16.15833 batch_time=0.67569
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 3.43166 (QuantReg: 16.27413) QuantErr: 16.27413 batch_time=0.61084
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 3.27829 (QuantReg: 16.40330) QuantErr: 16.40330 batch_time=0.61585
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 3.02575 (QuantReg: 16.16461) QuantErr: 16.16461 batch_time=0.62471
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 3.25259 (QuantReg: 16.15057) QuantErr: 16.15057 batch_time=0.66158
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 3.14316 (QuantReg: 16.23227) QuantErr: 16.23227 batch_time=0.61014
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 3.05875 (QuantReg: 16.37320) QuantErr: 16.37320 batch_time=0.61420
Train Epoch: 13 codebook_update_time=3.37078
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch13.pth ...
Done in 5.475s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch13.pth ...
Done in 10.399s
removing stale ckpt [epoch 12] [took 0.09s]
epoch : 13
loss : 3.188758454322815
quant_reg : 16.25636781692505
quant_err : 16.25636781692505
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
LSMDC_full_test/t2v_metrics/R1: 13.5
LSMDC_full_test/t2v_metrics/R5: 32.2
LSMDC_full_test/t2v_metrics/R10: 41.6
LSMDC_full_test/t2v_metrics/R50: 68.2
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 68.87
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.24788554396936
LSMDC_full_test/v2t_metrics/R1: 14.1
LSMDC_full_test/v2t_metrics/R5: 31.3
LSMDC_full_test/v2t_metrics/R10: 41.0
LSMDC_full_test/v2t_metrics/R50: 67.9
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 66.603
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.253211398487966
mnt_best : 26.24788554396936
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 3.27719 (QuantReg: 16.13441) QuantErr: 16.13441 batch_time=26.55113
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 3.00970 (QuantReg: 16.19254) QuantErr: 16.19254 batch_time=0.62609
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 3.36760 (QuantReg: 16.17274) QuantErr: 16.17274 batch_time=0.60013
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 2.83175 (QuantReg: 16.25349) QuantErr: 16.25349 batch_time=0.61806
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 2.88585 (QuantReg: 16.35763) QuantErr: 16.35763 batch_time=1.19531
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 2.99967 (QuantReg: 16.19400) QuantErr: 16.19400 batch_time=0.65879
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 2.90442 (QuantReg: 16.16478) QuantErr: 16.16478 batch_time=0.62152
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 3.69595 (QuantReg: 16.16450) QuantErr: 16.16450 batch_time=0.62423
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 3.05806 (QuantReg: 16.32679) QuantErr: 16.32679 batch_time=0.59404
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 3.31860 (QuantReg: 16.38398) QuantErr: 16.38398 batch_time=1.29412
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 3.01311 (QuantReg: 16.33928) QuantErr: 16.33928 batch_time=0.60093
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 3.44436 (QuantReg: 16.42229) QuantErr: 16.42229 batch_time=0.74303
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 3.28413 (QuantReg: 16.20873) QuantErr: 16.20873 batch_time=0.62677
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 3.00312 (QuantReg: 16.42601) QuantErr: 16.42601 batch_time=1.36085
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 3.36040 (QuantReg: 16.38786) QuantErr: 16.38786 batch_time=0.62086
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 3.15917 (QuantReg: 16.31876) QuantErr: 16.31876 batch_time=0.74590
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 3.01996 (QuantReg: 16.37637) QuantErr: 16.37637 batch_time=0.60113
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 3.42914 (QuantReg: 16.51753) QuantErr: 16.51753 batch_time=0.62239
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 3.45609 (QuantReg: 16.36067) QuantErr: 16.36067 batch_time=0.61437
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 3.31727 (QuantReg: 16.38783) QuantErr: 16.38783 batch_time=0.60801
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 2.92136 (QuantReg: 16.51050) QuantErr: 16.51050 batch_time=0.62303
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 2.84307 (QuantReg: 16.52704) QuantErr: 16.52704 batch_time=1.75876
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 3.01582 (QuantReg: 16.62560) QuantErr: 16.62560 batch_time=0.60363
Train Epoch: 14 codebook_update_time=3.78080
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch14.pth ...
Done in 4.126s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch14.pth ...
Done in 8.724s
removing stale ckpt [epoch 13] [took 0.01s]
epoch : 14
loss : 3.109896412849426
quant_reg : 16.358777893066407
quant_err : 16.358777893066407
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
LSMDC_full_test/t2v_metrics/R1: 13.5
LSMDC_full_test/t2v_metrics/R5: 32.5
LSMDC_full_test/t2v_metrics/R10: 42.2
LSMDC_full_test/t2v_metrics/R50: 68.1
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 70.348
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.45512757296448
LSMDC_full_test/v2t_metrics/R1: 14.5
LSMDC_full_test/v2t_metrics/R5: 32.6
LSMDC_full_test/v2t_metrics/R10: 39.9
LSMDC_full_test/v2t_metrics/R50: 67.2
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 67.929
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.61865858805852
mnt_best : 26.45512757296448
not_improved_count: 0
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 2.36362 (QuantReg: 16.42882) QuantErr: 16.42882 batch_time=28.39687
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 2.81185 (QuantReg: 16.22681) QuantErr: 16.22681 batch_time=0.60818
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 3.00564 (QuantReg: 16.42087) QuantErr: 16.42087 batch_time=1.18213
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 3.03948 (QuantReg: 16.44408) QuantErr: 16.44408 batch_time=0.59478
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 2.46842 (QuantReg: 16.54240) QuantErr: 16.54240 batch_time=0.78789
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 3.15770 (QuantReg: 16.47654) QuantErr: 16.47654 batch_time=0.60905
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 2.46751 (QuantReg: 16.61832) QuantErr: 16.61832 batch_time=0.60648
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 2.81736 (QuantReg: 16.46263) QuantErr: 16.46263 batch_time=0.59246
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 3.43491 (QuantReg: 16.38246) QuantErr: 16.38246 batch_time=0.93568
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 3.21943 (QuantReg: 16.34024) QuantErr: 16.34024 batch_time=0.61219
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 2.74017 (QuantReg: 16.52378) QuantErr: 16.52378 batch_time=0.63168
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 2.82065 (QuantReg: 16.35626) QuantErr: 16.35626 batch_time=0.59128
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 2.93566 (QuantReg: 16.48119) QuantErr: 16.48119 batch_time=0.69016
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 3.29427 (QuantReg: 16.60466) QuantErr: 16.60466 batch_time=0.75764
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 2.69002 (QuantReg: 16.41380) QuantErr: 16.41380 batch_time=0.61480
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 2.92017 (QuantReg: 16.51079) QuantErr: 16.51079 batch_time=0.59790
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 3.16754 (QuantReg: 16.50340) QuantErr: 16.50340 batch_time=0.61850
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 3.20579 (QuantReg: 16.48429) QuantErr: 16.48429 batch_time=0.61025
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 3.07341 (QuantReg: 16.51656) QuantErr: 16.51656 batch_time=0.65137
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 2.79607 (QuantReg: 16.52420) QuantErr: 16.52420 batch_time=0.61976
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 2.94533 (QuantReg: 16.67464) QuantErr: 16.67464 batch_time=0.65068
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 2.95302 (QuantReg: 16.54642) QuantErr: 16.54642 batch_time=0.63084
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 3.06482 (QuantReg: 16.60044) QuantErr: 16.60044 batch_time=0.62272
Train Epoch: 15 codebook_update_time=3.57318
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch15.pth ...
Done in 4.718s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch15.pth ...
Done in 9.165s
removing stale ckpt [epoch 14] [took 0.02s]
epoch : 15
loss : 2.989389842033386
quant_reg : 16.470266738891603
quant_err : 16.470266738891603
learning_rate : 2.4383748955776477e-05
n_samples : 480000
n_steps : 3750
LSMDC_full_test/t2v_metrics/R1: 13.8
LSMDC_full_test/t2v_metrics/R5: 32.8
LSMDC_full_test/t2v_metrics/R10: 42.1
LSMDC_full_test/t2v_metrics/R50: 69.6
LSMDC_full_test/t2v_metrics/MedR: 16.0
LSMDC_full_test/t2v_metrics/MeanR: 68.079
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.710273929157175
LSMDC_full_test/v2t_metrics/R1: 14.5
LSMDC_full_test/v2t_metrics/R5: 33.1
LSMDC_full_test/v2t_metrics/R10: 41.4
LSMDC_full_test/v2t_metrics/R50: 67.9
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 68.263
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 27.085204089053796
mnt_best : 26.710273929157175
not_improved_count: 0
Train Epoch: 16 [1/250 128/32000 (0%)] Loss: 2.96525 (QuantReg: 16.57909) QuantErr: 16.57909 batch_time=21.78806
Train Epoch: 16 [12/250 1536/32000 (5%)] Loss: 2.53489 (QuantReg: 16.55473) QuantErr: 16.55473 batch_time=0.64002
Train Epoch: 16 [23/250 2944/32000 (9%)] Loss: 2.52147 (QuantReg: 16.55862) QuantErr: 16.55862 batch_time=0.60662
Train Epoch: 16 [34/250 4352/32000 (14%)] Loss: 3.29608 (QuantReg: 16.32049) QuantErr: 16.32049 batch_time=0.60933
Train Epoch: 16 [45/250 5760/32000 (18%)] Loss: 3.22522 (QuantReg: 16.38781) QuantErr: 16.38781 batch_time=0.62387
Train Epoch: 16 [56/250 7168/32000 (22%)] Loss: 3.02096 (QuantReg: 16.59221) QuantErr: 16.59221 batch_time=0.60236
Train Epoch: 16 [67/250 8576/32000 (27%)] Loss: 2.76400 (QuantReg: 16.37467) QuantErr: 16.37467 batch_time=6.57284
Train Epoch: 16 [78/250 9984/32000 (31%)] Loss: 3.08778 (QuantReg: 16.45181) QuantErr: 16.45181 batch_time=0.60775
Train Epoch: 16 [89/250 11392/32000 (36%)] Loss: 2.71461 (QuantReg: 16.66260) QuantErr: 16.66260 batch_time=1.81473
Train Epoch: 16 [100/250 12800/32000 (40%)] Loss: 3.08137 (QuantReg: 16.51464) QuantErr: 16.51464 batch_time=0.63304
Train Epoch: 16 [111/250 14208/32000 (44%)] Loss: 3.13594 (QuantReg: 16.41837) QuantErr: 16.41837 batch_time=0.64244
Train Epoch: 16 [122/250 15616/32000 (49%)] Loss: 3.28810 (QuantReg: 16.31583) QuantErr: 16.31583 batch_time=0.65935
Train Epoch: 16 [133/250 17024/32000 (53%)] Loss: 2.81230 (QuantReg: 16.58537) QuantErr: 16.58537 batch_time=0.61795
Train Epoch: 16 [144/250 18432/32000 (58%)] Loss: 3.05560 (QuantReg: 16.49566) QuantErr: 16.49566 batch_time=0.63012
Train Epoch: 16 [155/250 19840/32000 (62%)] Loss: 2.49369 (QuantReg: 16.68044) QuantErr: 16.68044 batch_time=0.69399
Train Epoch: 16 [166/250 21248/32000 (66%)] Loss: 2.82914 (QuantReg: 16.76769) QuantErr: 16.76769 batch_time=0.59237
Train Epoch: 16 [177/250 22656/32000 (71%)] Loss: 2.85007 (QuantReg: 16.40282) QuantErr: 16.40282 batch_time=0.62905
Train Epoch: 16 [188/250 24064/32000 (75%)] Loss: 3.00035 (QuantReg: 16.65529) QuantErr: 16.65529 batch_time=0.59565
Train Epoch: 16 [199/250 25472/32000 (80%)] Loss: 2.73536 (QuantReg: 16.65067) QuantErr: 16.65067 batch_time=0.63294
Train Epoch: 16 [210/250 26880/32000 (84%)] Loss: 3.04927 (QuantReg: 16.55423) QuantErr: 16.55423 batch_time=0.68387
Train Epoch: 16 [221/250 28288/32000 (88%)] Loss: 2.67567 (QuantReg: 16.67908) QuantErr: 16.67908 batch_time=0.62854
Train Epoch: 16 [232/250 29696/32000 (93%)] Loss: 2.68232 (QuantReg: 16.65233) QuantErr: 16.65233 batch_time=0.59123
Train Epoch: 16 [243/250 31104/32000 (97%)] Loss: 2.73901 (QuantReg: 16.63129) QuantErr: 16.63129 batch_time=0.64548
Train Epoch: 16 codebook_update_time=3.36688
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch16.pth ...
Done in 5.565s
removing stale ckpt [epoch 15] [took 0.02s]
epoch : 16
loss : 2.888849912643433
quant_reg : 16.556770027160645
quant_err : 16.556770027160645
learning_rate : 2.3164561507987653e-05
n_samples : 512000
n_steps : 4000
LSMDC_full_test/t2v_metrics/R1: 12.8
LSMDC_full_test/t2v_metrics/R5: 32.3
LSMDC_full_test/t2v_metrics/R10: 41.4
LSMDC_full_test/t2v_metrics/R50: 68.9
LSMDC_full_test/t2v_metrics/MedR: 18.0
LSMDC_full_test/t2v_metrics/MeanR: 69.372
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.77137622340337
LSMDC_full_test/v2t_metrics/R1: 13.2
LSMDC_full_test/v2t_metrics/R5: 31.4
LSMDC_full_test/v2t_metrics/R10: 41.1
LSMDC_full_test/v2t_metrics/R50: 66.6
LSMDC_full_test/v2t_metrics/MedR: 19.0
LSMDC_full_test/v2t_metrics/MeanR: 68.724
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.730514306052502
mnt_best : 26.710273929157175
not_improved_count: 1
Train Epoch: 17 [1/250 128/32000 (0%)] Loss: 2.76651 (QuantReg: 16.45425) QuantErr: 16.45425 batch_time=25.15982
Train Epoch: 17 [12/250 1536/32000 (5%)] Loss: 3.38457 (QuantReg: 16.51793) QuantErr: 16.51793 batch_time=0.66722
Train Epoch: 17 [23/250 2944/32000 (9%)] Loss: 3.08265 (QuantReg: 16.69600) QuantErr: 16.69600 batch_time=0.61754
Train Epoch: 17 [34/250 4352/32000 (14%)] Loss: 2.55301 (QuantReg: 16.75336) QuantErr: 16.75336 batch_time=0.62867
Train Epoch: 17 [45/250 5760/32000 (18%)] Loss: 3.04202 (QuantReg: 16.48611) QuantErr: 16.48611 batch_time=1.13784
Train Epoch: 17 [56/250 7168/32000 (22%)] Loss: 2.82248 (QuantReg: 16.50663) QuantErr: 16.50663 batch_time=0.62710
Train Epoch: 17 [67/250 8576/32000 (27%)] Loss: 2.46987 (QuantReg: 16.61916) QuantErr: 16.61916 batch_time=0.60977
Train Epoch: 17 [78/250 9984/32000 (31%)] Loss: 2.28587 (QuantReg: 16.73573) QuantErr: 16.73573 batch_time=0.62568
Train Epoch: 17 [89/250 11392/32000 (36%)] Loss: 2.95827 (QuantReg: 16.60977) QuantErr: 16.60977 batch_time=0.69057
Train Epoch: 17 [100/250 12800/32000 (40%)] Loss: 2.66879 (QuantReg: 16.72640) QuantErr: 16.72640 batch_time=0.63511
Train Epoch: 17 [111/250 14208/32000 (44%)] Loss: 2.31229 (QuantReg: 16.90959) QuantErr: 16.90959 batch_time=0.60389
Train Epoch: 17 [122/250 15616/32000 (49%)] Loss: 2.80915 (QuantReg: 16.57030) QuantErr: 16.57030 batch_time=0.59497
Train Epoch: 17 [133/250 17024/32000 (53%)] Loss: 2.65883 (QuantReg: 16.72097) QuantErr: 16.72097 batch_time=0.61944
Train Epoch: 17 [144/250 18432/32000 (58%)] Loss: 2.92755 (QuantReg: 16.45928) QuantErr: 16.45928 batch_time=2.77403
Train Epoch: 17 [155/250 19840/32000 (62%)] Loss: 2.91343 (QuantReg: 16.52978) QuantErr: 16.52978 batch_time=0.61217
Train Epoch: 17 [166/250 21248/32000 (66%)] Loss: 2.43295 (QuantReg: 16.69077) QuantErr: 16.69077 batch_time=0.66552
Train Epoch: 17 [177/250 22656/32000 (71%)] Loss: 2.89839 (QuantReg: 16.65134) QuantErr: 16.65134 batch_time=0.62184
Train Epoch: 17 [188/250 24064/32000 (75%)] Loss: 3.08219 (QuantReg: 16.72612) QuantErr: 16.72612 batch_time=0.62361
Train Epoch: 17 [199/250 25472/32000 (80%)] Loss: 2.60802 (QuantReg: 16.89193) QuantErr: 16.89193 batch_time=0.62993
Train Epoch: 17 [210/250 26880/32000 (84%)] Loss: 3.10640 (QuantReg: 16.54035) QuantErr: 16.54035 batch_time=0.64016
Train Epoch: 17 [221/250 28288/32000 (88%)] Loss: 2.74935 (QuantReg: 16.61274) QuantErr: 16.61274 batch_time=0.63986
Train Epoch: 17 [232/250 29696/32000 (93%)] Loss: 3.18175 (QuantReg: 16.66187) QuantErr: 16.66187 batch_time=0.65128
Train Epoch: 17 [243/250 31104/32000 (97%)] Loss: 2.49637 (QuantReg: 16.73289) QuantErr: 16.73289 batch_time=0.63907
Train Epoch: 17 codebook_update_time=3.54177
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch17.pth ...
Done in 5.487s
removing stale ckpt [epoch 16] [took 0.01s]
epoch : 17
loss : 2.790562448501587
quant_reg : 16.60333519744873
quant_err : 16.60333519744873
learning_rate : 2.2006333432588268e-05
n_samples : 544000
n_steps : 4250
LSMDC_full_test/t2v_metrics/R1: 13.1
LSMDC_full_test/t2v_metrics/R5: 31.8
LSMDC_full_test/t2v_metrics/R10: 41.6
LSMDC_full_test/t2v_metrics/R50: 67.8
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 70.527
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.87799246587303
LSMDC_full_test/v2t_metrics/R1: 14.6
LSMDC_full_test/v2t_metrics/R5: 32.1
LSMDC_full_test/v2t_metrics/R10: 41.5
LSMDC_full_test/v2t_metrics/R50: 67.3
LSMDC_full_test/v2t_metrics/MedR: 18.0
LSMDC_full_test/v2t_metrics/MeanR: 69.001
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.892757040772867
mnt_best : 26.710273929157175
not_improved_count: 2
Train Epoch: 18 [1/250 128/32000 (0%)] Loss: 2.86744 (QuantReg: 16.59143) QuantErr: 16.59143 batch_time=22.33865
Train Epoch: 18 [12/250 1536/32000 (5%)] Loss: 2.48988 (QuantReg: 16.77836) QuantErr: 16.77836 batch_time=0.60783
Train Epoch: 18 [23/250 2944/32000 (9%)] Loss: 2.52653 (QuantReg: 16.57368) QuantErr: 16.57368 batch_time=0.94288
Train Epoch: 18 [34/250 4352/32000 (14%)] Loss: 2.56827 (QuantReg: 16.72631) QuantErr: 16.72631 batch_time=0.59903
Train Epoch: 18 [45/250 5760/32000 (18%)] Loss: 2.65144 (QuantReg: 16.62041) QuantErr: 16.62041 batch_time=0.60723
Train Epoch: 18 [56/250 7168/32000 (22%)] Loss: 2.67630 (QuantReg: 16.65324) QuantErr: 16.65324 batch_time=0.61164
Train Epoch: 18 [67/250 8576/32000 (27%)] Loss: 2.54547 (QuantReg: 16.65713) QuantErr: 16.65713 batch_time=0.61793
Train Epoch: 18 [78/250 9984/32000 (31%)] Loss: 2.90711 (QuantReg: 16.84790) QuantErr: 16.84790 batch_time=0.60070
Train Epoch: 18 [89/250 11392/32000 (36%)] Loss: 3.23744 (QuantReg: 16.64059) QuantErr: 16.64059 batch_time=0.59346
Train Epoch: 18 [100/250 12800/32000 (40%)] Loss: 2.47295 (QuantReg: 16.78449) QuantErr: 16.78449 batch_time=0.60833
Train Epoch: 18 [111/250 14208/32000 (44%)] Loss: 2.58748 (QuantReg: 16.80379) QuantErr: 16.80379 batch_time=0.62024
Train Epoch: 18 [122/250 15616/32000 (49%)] Loss: 2.48235 (QuantReg: 16.76651) QuantErr: 16.76651 batch_time=0.63057
Train Epoch: 18 [133/250 17024/32000 (53%)] Loss: 2.40349 (QuantReg: 16.68115) QuantErr: 16.68115 batch_time=0.59023
Train Epoch: 18 [144/250 18432/32000 (58%)] Loss: 2.62903 (QuantReg: 16.65165) QuantErr: 16.65165 batch_time=0.60949
Train Epoch: 18 [155/250 19840/32000 (62%)] Loss: 2.54154 (QuantReg: 16.73741) QuantErr: 16.73741 batch_time=0.59676
Train Epoch: 18 [166/250 21248/32000 (66%)] Loss: 3.19123 (QuantReg: 16.64172) QuantErr: 16.64172 batch_time=0.60836
Train Epoch: 18 [177/250 22656/32000 (71%)] Loss: 2.68948 (QuantReg: 16.63125) QuantErr: 16.63125 batch_time=0.62316
Train Epoch: 18 [188/250 24064/32000 (75%)] Loss: 2.88839 (QuantReg: 16.63007) QuantErr: 16.63007 batch_time=0.58791
Train Epoch: 18 [199/250 25472/32000 (80%)] Loss: 2.48934 (QuantReg: 16.76845) QuantErr: 16.76845 batch_time=0.72377
Train Epoch: 18 [210/250 26880/32000 (84%)] Loss: 2.64082 (QuantReg: 16.67947) QuantErr: 16.67947 batch_time=0.60176
Train Epoch: 18 [221/250 28288/32000 (88%)] Loss: 3.39150 (QuantReg: 16.64489) QuantErr: 16.64489 batch_time=0.62107
Train Epoch: 18 [232/250 29696/32000 (93%)] Loss: 2.90963 (QuantReg: 16.73695) QuantErr: 16.73695 batch_time=0.63205
Train Epoch: 18 [243/250 31104/32000 (97%)] Loss: 2.72907 (QuantReg: 16.57960) QuantErr: 16.57960 batch_time=0.62156
Train Epoch: 18 codebook_update_time=3.79411
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch18.pth ...
Done in 4.710s
removing stale ckpt [epoch 17] [took 0.79s]
epoch : 18
loss : 2.685614429473877
quant_reg : 16.666835891723633
quant_err : 16.666835891723633
learning_rate : 2.0906016760958855e-05
n_samples : 576000
n_steps : 4500
LSMDC_full_test/t2v_metrics/R1: 14.0
LSMDC_full_test/t2v_metrics/R5: 32.4
LSMDC_full_test/t2v_metrics/R10: 41.7
LSMDC_full_test/t2v_metrics/R50: 68.6
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 70.182
LSMDC_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.644221390938178
LSMDC_full_test/v2t_metrics/R1: 14.1
LSMDC_full_test/v2t_metrics/R5: 33.1
LSMDC_full_test/v2t_metrics/R10: 41.4
LSMDC_full_test/v2t_metrics/R50: 68.3
LSMDC_full_test/v2t_metrics/MedR: 17.5
LSMDC_full_test/v2t_metrics/MeanR: 68.412
LSMDC_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.833818782502174
mnt_best : 26.710273929157175
not_improved_count: 3
Train Epoch: 19 [1/250 128/32000 (0%)] Loss: 2.30045 (QuantReg: 16.66164) QuantErr: 16.66164 batch_time=20.34341
Train Epoch: 19 [12/250 1536/32000 (5%)] Loss: 2.86609 (QuantReg: 16.60087) QuantErr: 16.60087 batch_time=0.70948
Train Epoch: 19 [23/250 2944/32000 (9%)] Loss: 2.97105 (QuantReg: 16.60223) QuantErr: 16.60223 batch_time=0.59684
Train Epoch: 19 [34/250 4352/32000 (14%)] Loss: 2.60617 (QuantReg: 16.64645) QuantErr: 16.64645 batch_time=0.64844
Train Epoch: 19 [45/250 5760/32000 (18%)] Loss: 2.39356 (QuantReg: 16.76141) QuantErr: 16.76141 batch_time=0.62338
Train Epoch: 19 [56/250 7168/32000 (22%)] Loss: 3.12386 (QuantReg: 16.67306) QuantErr: 16.67306 batch_time=0.64291
Train Epoch: 19 [67/250 8576/32000 (27%)] Loss: 2.73619 (QuantReg: 16.64792) QuantErr: 16.64792 batch_time=1.62506
Train Epoch: 19 [78/250 9984/32000 (31%)] Loss: 2.73848 (QuantReg: 16.74669) QuantErr: 16.74669 batch_time=0.68936
Train Epoch: 19 [89/250 11392/32000 (36%)] Loss: 2.46033 (QuantReg: 16.70480) QuantErr: 16.70480 batch_time=0.72347
Train Epoch: 19 [100/250 12800/32000 (40%)] Loss: 2.66538 (QuantReg: 16.75303) QuantErr: 16.75303 batch_time=0.64256
Train Epoch: 19 [111/250 14208/32000 (44%)] Loss: 2.76595 (QuantReg: 16.48425) QuantErr: 16.48425 batch_time=0.65165
Train Epoch: 19 [122/250 15616/32000 (49%)] Loss: 2.69018 (QuantReg: 16.60455) QuantErr: 16.60455 batch_time=0.65684
Train Epoch: 19 [133/250 17024/32000 (53%)] Loss: 2.33237 (QuantReg: 16.82642) QuantErr: 16.82642 batch_time=0.60168
Train Epoch: 19 [144/250 18432/32000 (58%)] Loss: 2.69335 (QuantReg: 16.64485) QuantErr: 16.64485 batch_time=0.69597
Train Epoch: 19 [155/250 19840/32000 (62%)] Loss: 2.79900 (QuantReg: 16.78032) QuantErr: 16.78032 batch_time=0.60614
Train Epoch: 19 [166/250 21248/32000 (66%)] Loss: 2.41922 (QuantReg: 16.95197) QuantErr: 16.95197 batch_time=0.61589
Train Epoch: 19 [177/250 22656/32000 (71%)] Loss: 2.55753 (QuantReg: 16.78978) QuantErr: 16.78978 batch_time=0.94348
Train Epoch: 19 [188/250 24064/32000 (75%)] Loss: 2.53266 (QuantReg: 16.64538) QuantErr: 16.64538 batch_time=0.65451
Train Epoch: 19 [199/250 25472/32000 (80%)] Loss: 2.28788 (QuantReg: 16.96164) QuantErr: 16.96164 batch_time=0.60474
Train Epoch: 19 [210/250 26880/32000 (84%)] Loss: 2.78890 (QuantReg: 16.74043) QuantErr: 16.74043 batch_time=0.65751
Train Epoch: 19 [221/250 28288/32000 (88%)] Loss: 2.53078 (QuantReg: 16.63902) QuantErr: 16.63902 batch_time=0.61882
Train Epoch: 19 [232/250 29696/32000 (93%)] Loss: 2.54390 (QuantReg: 16.83791) QuantErr: 16.83791 batch_time=0.67048
Train Epoch: 19 [243/250 31104/32000 (97%)] Loss: 2.19047 (QuantReg: 16.73927) QuantErr: 16.73927 batch_time=1.20388
Train Epoch: 19 codebook_update_time=3.65886
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_LSMDC_M64/checkpoint-epoch19.pth ...
Done in 4.374s
removing stale ckpt [epoch 18] [took 0.00s]
epoch : 19
loss : 2.6228488874435425
quant_reg : 16.72257305145264
quant_err : 16.72257305145264
learning_rate : 1.986071592291091e-05
n_samples : 608000
n_steps : 4750
LSMDC_full_test/t2v_metrics/R1: 12.7
LSMDC_full_test/t2v_metrics/R5: 31.6
LSMDC_full_test/t2v_metrics/R10: 42.6
LSMDC_full_test/t2v_metrics/R50: 69.2
LSMDC_full_test/t2v_metrics/MedR: 17.0
LSMDC_full_test/t2v_metrics/MeanR: 70.61