-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathHCQ_MSRVTT_full.txt
3305 lines (3305 loc) · 234 KB
/
HCQ_MSRVTT_full.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Experiment directory: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full
Preparing the dataloaders ...
Loading dataset MSRVTT_full_train in ram ...
Finish loading dataset MSRVTT_full_train in ram, taking 1022.5835797786713 s.
Loading dataset MSRVTT_full_val in ram ...
Finish loading dataset MSRVTT_full_val in ram, taking 66.12120866775513 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 354.9544880390167 s.
Loading dataset MSRVTT_full_test in ram ...
Finish loading dataset MSRVTT_full_test in ram, taking 102.38593196868896 s.
Training ...
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch0.pth ...
Done in 1.758s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch0.pth ...
Done in 3.453s
epoch : 0
loss : 0
learning_rate : 5e-05
n_samples : 0
n_steps : 0
MSRVTT_full_val/t2v_metrics/R1: 0.0
MSRVTT_full_val/t2v_metrics/R5: 1.2072434607645874
MSRVTT_full_val/t2v_metrics/R10: 1.6096579476861168
MSRVTT_full_val/t2v_metrics/R50: 8.450704225352112
MSRVTT_full_val/t2v_metrics/MedR: 252.0
MSRVTT_full_val/t2v_metrics/MeanR: 251.21730382293762
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_val/v2t_metrics/R1: 0.0
MSRVTT_full_val/v2t_metrics/R5: 0.8048289738430584
MSRVTT_full_val/v2t_metrics/R10: 2.0120724346076457
MSRVTT_full_val/v2t_metrics/R50: 9.054325955734406
MSRVTT_full_val/v2t_metrics/MedR: 243.0
MSRVTT_full_val/v2t_metrics/MeanR: 247.7344064386318
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 0.0
MSRVTT_full_test/t2v_metrics/R1: 0.033444816053511704
MSRVTT_full_test/t2v_metrics/R5: 0.20066889632107024
MSRVTT_full_test/t2v_metrics/R10: 0.26755852842809363
MSRVTT_full_test/t2v_metrics/R50: 1.705685618729097
MSRVTT_full_test/t2v_metrics/MedR: 1515.0
MSRVTT_full_test/t2v_metrics/MeanR: 1498.5565217391304
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 0.12154652794863813
MSRVTT_full_test/v2t_metrics/R1: 0.06688963210702341
MSRVTT_full_test/v2t_metrics/R5: 0.16722408026755853
MSRVTT_full_test/v2t_metrics/R10: 0.3010033444816054
MSRVTT_full_test/v2t_metrics/R50: 1.806020066889632
MSRVTT_full_test/v2t_metrics/MedR: 1471.5
MSRVTT_full_test/v2t_metrics/MeanR: 1495.3264214046824
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 0.14987975740993859
mnt_best : 0.12154652794863813
not_improved_count: 0
Train Epoch: 1 [1/250 128/32000 (0%)] Loss: 9.83601 (QuantReg: 22.44707) QuantErr: 22.44707 batch_time=35.14457
Train Epoch: 1 [12/250 1536/32000 (5%)] Loss: 8.85996 (QuantReg: 22.52926) QuantErr: 22.52926 batch_time=0.53314
Train Epoch: 1 [23/250 2944/32000 (9%)] Loss: 7.16728 (QuantReg: 22.56572) QuantErr: 22.56572 batch_time=0.61335
Train Epoch: 1 [34/250 4352/32000 (14%)] Loss: 6.96779 (QuantReg: 22.61984) QuantErr: 22.61984 batch_time=0.62287
Train Epoch: 1 [45/250 5760/32000 (18%)] Loss: 6.37450 (QuantReg: 22.61680) QuantErr: 22.61680 batch_time=0.52641
Train Epoch: 1 [56/250 7168/32000 (22%)] Loss: 6.11715 (QuantReg: 22.61637) QuantErr: 22.61637 batch_time=0.58820
Train Epoch: 1 [67/250 8576/32000 (27%)] Loss: 6.09436 (QuantReg: 22.59203) QuantErr: 22.59203 batch_time=0.58439
Train Epoch: 1 [78/250 9984/32000 (31%)] Loss: 5.76966 (QuantReg: 22.63376) QuantErr: 22.63376 batch_time=0.55744
Train Epoch: 1 [89/250 11392/32000 (36%)] Loss: 5.23123 (QuantReg: 22.62047) QuantErr: 22.62047 batch_time=0.55652
Train Epoch: 1 [100/250 12800/32000 (40%)] Loss: 5.52448 (QuantReg: 22.64792) QuantErr: 22.64792 batch_time=0.54679
Train Epoch: 1 [111/250 14208/32000 (44%)] Loss: 5.05527 (QuantReg: 22.64563) QuantErr: 22.64563 batch_time=0.50905
Train Epoch: 1 [122/250 15616/32000 (49%)] Loss: 4.77527 (QuantReg: 22.66315) QuantErr: 22.66315 batch_time=0.53107
Train Epoch: 1 [133/250 17024/32000 (53%)] Loss: 4.80842 (QuantReg: 22.62115) QuantErr: 22.62115 batch_time=0.53336
Train Epoch: 1 [144/250 18432/32000 (58%)] Loss: 4.62151 (QuantReg: 22.68834) QuantErr: 22.68834 batch_time=0.58103
Train Epoch: 1 [155/250 19840/32000 (62%)] Loss: 4.79429 (QuantReg: 22.62875) QuantErr: 22.62875 batch_time=0.52455
Train Epoch: 1 [166/250 21248/32000 (66%)] Loss: 4.98667 (QuantReg: 22.63139) QuantErr: 22.63139 batch_time=0.54299
Train Epoch: 1 [177/250 22656/32000 (71%)] Loss: 4.66633 (QuantReg: 22.63806) QuantErr: 22.63806 batch_time=0.54032
Train Epoch: 1 [188/250 24064/32000 (75%)] Loss: 4.37311 (QuantReg: 22.66245) QuantErr: 22.66245 batch_time=0.50855
Train Epoch: 1 [199/250 25472/32000 (80%)] Loss: 4.39968 (QuantReg: 22.61786) QuantErr: 22.61786 batch_time=0.50880
Train Epoch: 1 [210/250 26880/32000 (84%)] Loss: 4.09325 (QuantReg: 22.62950) QuantErr: 22.62950 batch_time=0.59869
Train Epoch: 1 [221/250 28288/32000 (88%)] Loss: 4.40390 (QuantReg: 22.63296) QuantErr: 22.63296 batch_time=0.50985
Train Epoch: 1 [232/250 29696/32000 (93%)] Loss: 4.20040 (QuantReg: 22.69141) QuantErr: 22.69141 batch_time=0.58151
Train Epoch: 1 [243/250 31104/32000 (97%)] Loss: 3.91421 (QuantReg: 22.64389) QuantErr: 22.64389 batch_time=0.58514
Train Epoch: 1 codebook_update_time=1.87644
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch1.pth ...
Done in 4.078s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch1.pth ...
Done in 8.461s
epoch : 1
loss : 5.3136826505661015
quant_reg : 22.62510693359375
quant_err : 22.62510693359375
learning_rate : 5e-05
n_samples : 32000
n_steps : 250
MSRVTT_full_val/t2v_metrics/R1: 19.919517102615693
MSRVTT_full_val/t2v_metrics/R5: 51.30784708249497
MSRVTT_full_val/t2v_metrics/R10: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R50: 95.97585513078471
MSRVTT_full_val/t2v_metrics/MedR: 5.0
MSRVTT_full_val/t2v_metrics/MeanR: 12.672032193158953
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 40.829876616690115
MSRVTT_full_val/v2t_metrics/R1: 21.93158953722334
MSRVTT_full_val/v2t_metrics/R5: 58.75251509054326
MSRVTT_full_val/v2t_metrics/R10: 72.23340040241449
MSRVTT_full_val/v2t_metrics/R50: 94.56740442655935
MSRVTT_full_val/v2t_metrics/MedR: 4.0
MSRVTT_full_val/v2t_metrics/MeanR: 11.96579476861167
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 45.31878006517563
MSRVTT_full_test/t2v_metrics/R1: 6.989966555183947
MSRVTT_full_test/t2v_metrics/R5: 21.638795986622075
MSRVTT_full_test/t2v_metrics/R10: 32.90969899665552
MSRVTT_full_test/t2v_metrics/R50: 67.22408026755853
MSRVTT_full_test/t2v_metrics/MedR: 23.0
MSRVTT_full_test/t2v_metrics/MeanR: 68.29866220735786
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 17.074344252666624
MSRVTT_full_test/v2t_metrics/R1: 7.157190635451505
MSRVTT_full_test/v2t_metrics/R5: 24.347826086956523
MSRVTT_full_test/v2t_metrics/R10: 36.58862876254181
MSRVTT_full_test/v2t_metrics/R50: 71.73913043478261
MSRVTT_full_test/v2t_metrics/MedR: 20.0
MSRVTT_full_test/v2t_metrics/MeanR: 62.57809364548495
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 18.54312687766534
mnt_best : 17.074344252666624
not_improved_count: 0
Train Epoch: 2 [1/250 128/32000 (0%)] Loss: 3.54046 (QuantReg: 12.01409) QuantErr: 12.01409 batch_time=41.73562
Train Epoch: 2 [12/250 1536/32000 (5%)] Loss: 3.93292 (QuantReg: 12.39205) QuantErr: 12.39205 batch_time=0.50230
Train Epoch: 2 [23/250 2944/32000 (9%)] Loss: 3.81195 (QuantReg: 12.47752) QuantErr: 12.47752 batch_time=0.52521
Train Epoch: 2 [34/250 4352/32000 (14%)] Loss: 3.65322 (QuantReg: 12.56524) QuantErr: 12.56524 batch_time=0.61843
Train Epoch: 2 [45/250 5760/32000 (18%)] Loss: 3.63213 (QuantReg: 12.83250) QuantErr: 12.83250 batch_time=0.52966
Train Epoch: 2 [56/250 7168/32000 (22%)] Loss: 3.81472 (QuantReg: 12.57137) QuantErr: 12.57137 batch_time=0.50328
Train Epoch: 2 [67/250 8576/32000 (27%)] Loss: 3.93923 (QuantReg: 13.09774) QuantErr: 13.09774 batch_time=0.57035
Train Epoch: 2 [78/250 9984/32000 (31%)] Loss: 3.50291 (QuantReg: 13.29130) QuantErr: 13.29130 batch_time=0.62475
Train Epoch: 2 [89/250 11392/32000 (36%)] Loss: 4.08532 (QuantReg: 13.28710) QuantErr: 13.28710 batch_time=0.50778
Train Epoch: 2 [100/250 12800/32000 (40%)] Loss: 3.47723 (QuantReg: 13.49839) QuantErr: 13.49839 batch_time=0.58172
Train Epoch: 2 [111/250 14208/32000 (44%)] Loss: 3.50694 (QuantReg: 13.58084) QuantErr: 13.58084 batch_time=0.56823
Train Epoch: 2 [122/250 15616/32000 (49%)] Loss: 3.25665 (QuantReg: 14.01648) QuantErr: 14.01648 batch_time=0.55725
Train Epoch: 2 [133/250 17024/32000 (53%)] Loss: 3.25069 (QuantReg: 13.69487) QuantErr: 13.69487 batch_time=0.52820
Train Epoch: 2 [144/250 18432/32000 (58%)] Loss: 3.45960 (QuantReg: 13.91894) QuantErr: 13.91894 batch_time=0.62018
Train Epoch: 2 [155/250 19840/32000 (62%)] Loss: 3.19850 (QuantReg: 13.95918) QuantErr: 13.95918 batch_time=0.55077
Train Epoch: 2 [166/250 21248/32000 (66%)] Loss: 3.20213 (QuantReg: 14.64342) QuantErr: 14.64342 batch_time=0.54448
Train Epoch: 2 [177/250 22656/32000 (71%)] Loss: 3.45062 (QuantReg: 14.41717) QuantErr: 14.41717 batch_time=0.50726
Train Epoch: 2 [188/250 24064/32000 (75%)] Loss: 3.44118 (QuantReg: 14.57709) QuantErr: 14.57709 batch_time=0.68350
Train Epoch: 2 [199/250 25472/32000 (80%)] Loss: 3.14286 (QuantReg: 14.26092) QuantErr: 14.26092 batch_time=5.05866
Train Epoch: 2 [210/250 26880/32000 (84%)] Loss: 2.99660 (QuantReg: 14.49301) QuantErr: 14.49301 batch_time=1.25176
Train Epoch: 2 [221/250 28288/32000 (88%)] Loss: 2.76733 (QuantReg: 14.67500) QuantErr: 14.67500 batch_time=0.59315
Train Epoch: 2 [232/250 29696/32000 (93%)] Loss: 3.86488 (QuantReg: 14.72295) QuantErr: 14.72295 batch_time=0.55742
Train Epoch: 2 [243/250 31104/32000 (97%)] Loss: 3.37663 (QuantReg: 15.07507) QuantErr: 15.07507 batch_time=0.54472
Train Epoch: 2 codebook_update_time=1.67365
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch2.pth ...
Done in 12.960s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch2.pth ...
Done in 29.122s
removing stale ckpt [epoch 1] [took 0.01s]
removing stale ckpt [epoch 0] [took 0.01s]
epoch : 2
loss : 3.511594988822937
quant_reg : 13.67512890625
quant_err : 13.67512890625
learning_rate : 4.75e-05
n_samples : 64000
n_steps : 500
MSRVTT_full_val/t2v_metrics/R1: 23.138832997987926
MSRVTT_full_val/t2v_metrics/R5: 60.160965794768615
MSRVTT_full_val/t2v_metrics/R10: 73.8430583501006
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 10.746478873239436
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 46.84414416955262
MSRVTT_full_val/v2t_metrics/R1: 27.364185110663986
MSRVTT_full_val/v2t_metrics/R5: 67.6056338028169
MSRVTT_full_val/v2t_metrics/R10: 80.6841046277666
MSRVTT_full_val/v2t_metrics/R50: 96.78068410462777
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.865191146881287
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 53.04581553683502
MSRVTT_full_test/t2v_metrics/R1: 8.22742474916388
MSRVTT_full_test/t2v_metrics/R5: 26.722408026755854
MSRVTT_full_test/t2v_metrics/R10: 39.264214046822744
MSRVTT_full_test/t2v_metrics/R50: 72.97658862876254
MSRVTT_full_test/t2v_metrics/MedR: 17.0
MSRVTT_full_test/t2v_metrics/MeanR: 57.19498327759197
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 20.513769569180845
MSRVTT_full_test/v2t_metrics/R1: 10.535117056856187
MSRVTT_full_test/v2t_metrics/R5: 30.869565217391305
MSRVTT_full_test/v2t_metrics/R10: 43.84615384615385
MSRVTT_full_test/v2t_metrics/R50: 78.26086956521739
MSRVTT_full_test/v2t_metrics/MedR: 14.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.150334448160535
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 24.24937002350997
mnt_best : 20.513769569180845
not_improved_count: 0
Train Epoch: 3 [1/250 128/32000 (0%)] Loss: 2.95852 (QuantReg: 12.27915) QuantErr: 12.27915 batch_time=52.21468
Train Epoch: 3 [12/250 1536/32000 (5%)] Loss: 3.14480 (QuantReg: 12.44404) QuantErr: 12.44404 batch_time=0.55308
Train Epoch: 3 [23/250 2944/32000 (9%)] Loss: 3.47106 (QuantReg: 12.86234) QuantErr: 12.86234 batch_time=0.59734
Train Epoch: 3 [34/250 4352/32000 (14%)] Loss: 2.65865 (QuantReg: 12.15534) QuantErr: 12.15534 batch_time=0.50995
Train Epoch: 3 [45/250 5760/32000 (18%)] Loss: 3.21549 (QuantReg: 12.48284) QuantErr: 12.48284 batch_time=0.57618
Train Epoch: 3 [56/250 7168/32000 (22%)] Loss: 2.60299 (QuantReg: 12.76546) QuantErr: 12.76546 batch_time=0.50675
Train Epoch: 3 [67/250 8576/32000 (27%)] Loss: 2.80472 (QuantReg: 12.67714) QuantErr: 12.67714 batch_time=0.79967
Train Epoch: 3 [78/250 9984/32000 (31%)] Loss: 3.02620 (QuantReg: 12.70516) QuantErr: 12.70516 batch_time=0.54698
Train Epoch: 3 [89/250 11392/32000 (36%)] Loss: 3.04814 (QuantReg: 12.78236) QuantErr: 12.78236 batch_time=0.51678
Train Epoch: 3 [100/250 12800/32000 (40%)] Loss: 2.78944 (QuantReg: 12.52773) QuantErr: 12.52773 batch_time=0.54072
Train Epoch: 3 [111/250 14208/32000 (44%)] Loss: 2.98643 (QuantReg: 12.52534) QuantErr: 12.52534 batch_time=0.55113
Train Epoch: 3 [122/250 15616/32000 (49%)] Loss: 2.55634 (QuantReg: 12.83131) QuantErr: 12.83131 batch_time=0.51294
Train Epoch: 3 [133/250 17024/32000 (53%)] Loss: 3.06464 (QuantReg: 13.12629) QuantErr: 13.12629 batch_time=0.51400
Train Epoch: 3 [144/250 18432/32000 (58%)] Loss: 2.76009 (QuantReg: 13.24108) QuantErr: 13.24108 batch_time=0.51163
Train Epoch: 3 [155/250 19840/32000 (62%)] Loss: 3.58899 (QuantReg: 12.86288) QuantErr: 12.86288 batch_time=0.51610
Train Epoch: 3 [166/250 21248/32000 (66%)] Loss: 2.23875 (QuantReg: 13.76208) QuantErr: 13.76208 batch_time=0.56785
Train Epoch: 3 [177/250 22656/32000 (71%)] Loss: 2.48953 (QuantReg: 13.24901) QuantErr: 13.24901 batch_time=0.51202
Train Epoch: 3 [188/250 24064/32000 (75%)] Loss: 3.08478 (QuantReg: 13.04236) QuantErr: 13.04236 batch_time=0.53615
Train Epoch: 3 [199/250 25472/32000 (80%)] Loss: 2.60918 (QuantReg: 13.03259) QuantErr: 13.03259 batch_time=0.55612
Train Epoch: 3 [210/250 26880/32000 (84%)] Loss: 2.82892 (QuantReg: 13.49016) QuantErr: 13.49016 batch_time=0.55459
Train Epoch: 3 [221/250 28288/32000 (88%)] Loss: 2.56724 (QuantReg: 13.55560) QuantErr: 13.55560 batch_time=0.52010
Train Epoch: 3 [232/250 29696/32000 (93%)] Loss: 2.34663 (QuantReg: 13.76391) QuantErr: 13.76391 batch_time=0.53779
Train Epoch: 3 [243/250 31104/32000 (97%)] Loss: 2.65552 (QuantReg: 13.58110) QuantErr: 13.58110 batch_time=0.54008
Train Epoch: 3 codebook_update_time=1.72436
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch3.pth ...
Done in 6.138s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch3.pth ...
Done in 18.980s
removing stale ckpt [epoch 2] [took 0.16s]
epoch : 3
loss : 2.900078523635864
quant_reg : 13.007189140319824
quant_err : 13.007189140319824
learning_rate : 4.5125e-05
n_samples : 96000
n_steps : 750
MSRVTT_full_val/t2v_metrics/R1: 25.553319919517104
MSRVTT_full_val/t2v_metrics/R5: 61.971830985915496
MSRVTT_full_val/t2v_metrics/R10: 76.65995975855131
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 4.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.442655935613683
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 49.514996076983756
MSRVTT_full_val/v2t_metrics/R1: 31.58953722334004
MSRVTT_full_val/v2t_metrics/R5: 67.80684104627767
MSRVTT_full_val/v2t_metrics/R10: 80.88531187122736
MSRVTT_full_val/v2t_metrics/R50: 97.1830985915493
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 8.058350100603622
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 55.74793864419913
MSRVTT_full_test/t2v_metrics/R1: 9.03010033444816
MSRVTT_full_test/t2v_metrics/R5: 27.993311036789297
MSRVTT_full_test/t2v_metrics/R10: 41.137123745819395
MSRVTT_full_test/t2v_metrics/R50: 75.28428093645485
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 53.91438127090301
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 21.826976936102067
MSRVTT_full_test/v2t_metrics/R1: 11.37123745819398
MSRVTT_full_test/v2t_metrics/R5: 32.80936454849498
MSRVTT_full_test/v2t_metrics/R10: 46.52173913043478
MSRVTT_full_test/v2t_metrics/R50: 80.10033444816054
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 45.085284280936456
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 25.89129838687588
mnt_best : 21.826976936102067
not_improved_count: 0
Train Epoch: 4 [1/250 128/32000 (0%)] Loss: 3.17784 (QuantReg: 12.23457) QuantErr: 12.23457 batch_time=34.59153
Train Epoch: 4 [12/250 1536/32000 (5%)] Loss: 2.71912 (QuantReg: 12.36826) QuantErr: 12.36826 batch_time=0.50962
Train Epoch: 4 [23/250 2944/32000 (9%)] Loss: 2.17273 (QuantReg: 12.47676) QuantErr: 12.47676 batch_time=0.51988
Train Epoch: 4 [34/250 4352/32000 (14%)] Loss: 2.39374 (QuantReg: 12.42306) QuantErr: 12.42306 batch_time=0.51946
Train Epoch: 4 [45/250 5760/32000 (18%)] Loss: 2.68108 (QuantReg: 12.26249) QuantErr: 12.26249 batch_time=0.54022
Train Epoch: 4 [56/250 7168/32000 (22%)] Loss: 3.02735 (QuantReg: 12.38818) QuantErr: 12.38818 batch_time=0.60972
Train Epoch: 4 [67/250 8576/32000 (27%)] Loss: 3.00565 (QuantReg: 12.56863) QuantErr: 12.56863 batch_time=0.55761
Train Epoch: 4 [78/250 9984/32000 (31%)] Loss: 2.81599 (QuantReg: 12.81678) QuantErr: 12.81678 batch_time=0.57313
Train Epoch: 4 [89/250 11392/32000 (36%)] Loss: 2.59676 (QuantReg: 12.57445) QuantErr: 12.57445 batch_time=0.54855
Train Epoch: 4 [100/250 12800/32000 (40%)] Loss: 2.55793 (QuantReg: 12.52168) QuantErr: 12.52168 batch_time=0.50296
Train Epoch: 4 [111/250 14208/32000 (44%)] Loss: 2.72835 (QuantReg: 12.83059) QuantErr: 12.83059 batch_time=0.52618
Train Epoch: 4 [122/250 15616/32000 (49%)] Loss: 2.60968 (QuantReg: 12.66768) QuantErr: 12.66768 batch_time=0.50612
Train Epoch: 4 [133/250 17024/32000 (53%)] Loss: 2.53732 (QuantReg: 12.76329) QuantErr: 12.76329 batch_time=0.98479
Train Epoch: 4 [144/250 18432/32000 (58%)] Loss: 2.67730 (QuantReg: 12.92435) QuantErr: 12.92435 batch_time=0.76167
Train Epoch: 4 [155/250 19840/32000 (62%)] Loss: 2.79698 (QuantReg: 13.11701) QuantErr: 13.11701 batch_time=0.60453
Train Epoch: 4 [166/250 21248/32000 (66%)] Loss: 2.93930 (QuantReg: 13.21875) QuantErr: 13.21875 batch_time=0.50950
Train Epoch: 4 [177/250 22656/32000 (71%)] Loss: 2.93704 (QuantReg: 12.87346) QuantErr: 12.87346 batch_time=0.55148
Train Epoch: 4 [188/250 24064/32000 (75%)] Loss: 2.70700 (QuantReg: 13.34987) QuantErr: 13.34987 batch_time=0.55618
Train Epoch: 4 [199/250 25472/32000 (80%)] Loss: 2.74146 (QuantReg: 12.94382) QuantErr: 12.94382 batch_time=0.51867
Train Epoch: 4 [210/250 26880/32000 (84%)] Loss: 2.39571 (QuantReg: 13.15816) QuantErr: 13.15816 batch_time=0.53494
Train Epoch: 4 [221/250 28288/32000 (88%)] Loss: 2.33933 (QuantReg: 13.47045) QuantErr: 13.47045 batch_time=0.56690
Train Epoch: 4 [232/250 29696/32000 (93%)] Loss: 2.59222 (QuantReg: 12.74652) QuantErr: 12.74652 batch_time=0.56915
Train Epoch: 4 [243/250 31104/32000 (97%)] Loss: 2.40653 (QuantReg: 12.91019) QuantErr: 12.91019 batch_time=0.53398
Train Epoch: 4 codebook_update_time=2.00432
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch4.pth ...
Done in 19.913s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch4.pth ...
Done in 24.935s
removing stale ckpt [epoch 3] [took 0.15s]
epoch : 4
loss : 2.5992415437698364
quant_reg : 12.807410766601562
quant_err : 12.807410766601562
learning_rate : 4.2868749999999995e-05
n_samples : 128000
n_steps : 1000
MSRVTT_full_val/t2v_metrics/R1: 25.150905432595575
MSRVTT_full_val/t2v_metrics/R5: 60.96579476861167
MSRVTT_full_val/t2v_metrics/R10: 75.0503018108652
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.839034205231389
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 48.64043337064883
MSRVTT_full_val/v2t_metrics/R1: 29.37625754527163
MSRVTT_full_val/v2t_metrics/R5: 72.03219315895372
MSRVTT_full_val/v2t_metrics/R10: 83.90342052313883
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 7.130784708249497
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 56.204047274139995
MSRVTT_full_test/t2v_metrics/R1: 9.565217391304348
MSRVTT_full_test/t2v_metrics/R5: 27.45819397993311
MSRVTT_full_test/t2v_metrics/R10: 41.070234113712374
MSRVTT_full_test/t2v_metrics/R50: 74.74916387959867
MSRVTT_full_test/t2v_metrics/MedR: 16.0
MSRVTT_full_test/t2v_metrics/MeanR: 55.72107023411371
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 22.09520306901761
MSRVTT_full_test/v2t_metrics/R1: 11.705685618729097
MSRVTT_full_test/v2t_metrics/R5: 33.91304347826087
MSRVTT_full_test/v2t_metrics/R10: 47.391304347826086
MSRVTT_full_test/v2t_metrics/R50: 82.0066889632107
MSRVTT_full_test/v2t_metrics/MedR: 12.0
MSRVTT_full_test/v2t_metrics/MeanR: 41.81103678929766
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 26.59627173728336
mnt_best : 22.09520306901761
not_improved_count: 0
Train Epoch: 5 [1/250 128/32000 (0%)] Loss: 2.57680 (QuantReg: 12.54195) QuantErr: 12.54195 batch_time=39.11689
Train Epoch: 5 [12/250 1536/32000 (5%)] Loss: 2.69845 (QuantReg: 12.60822) QuantErr: 12.60822 batch_time=0.50447
Train Epoch: 5 [23/250 2944/32000 (9%)] Loss: 2.78558 (QuantReg: 12.45181) QuantErr: 12.45181 batch_time=0.51655
Train Epoch: 5 [34/250 4352/32000 (14%)] Loss: 2.09201 (QuantReg: 12.85870) QuantErr: 12.85870 batch_time=0.55504
Train Epoch: 5 [45/250 5760/32000 (18%)] Loss: 2.31810 (QuantReg: 12.32722) QuantErr: 12.32722 batch_time=0.57536
Train Epoch: 5 [56/250 7168/32000 (22%)] Loss: 2.46397 (QuantReg: 13.35134) QuantErr: 13.35134 batch_time=0.55678
Train Epoch: 5 [67/250 8576/32000 (27%)] Loss: 2.37301 (QuantReg: 12.56506) QuantErr: 12.56506 batch_time=1.13575
Train Epoch: 5 [78/250 9984/32000 (31%)] Loss: 2.58894 (QuantReg: 12.61009) QuantErr: 12.61009 batch_time=0.56695
Train Epoch: 5 [89/250 11392/32000 (36%)] Loss: 2.09916 (QuantReg: 12.72439) QuantErr: 12.72439 batch_time=0.58284
Train Epoch: 5 [100/250 12800/32000 (40%)] Loss: 2.52670 (QuantReg: 12.84987) QuantErr: 12.84987 batch_time=0.53734
Train Epoch: 5 [111/250 14208/32000 (44%)] Loss: 2.22998 (QuantReg: 12.99709) QuantErr: 12.99709 batch_time=0.51683
Train Epoch: 5 [122/250 15616/32000 (49%)] Loss: 2.25690 (QuantReg: 13.03214) QuantErr: 13.03214 batch_time=0.62460
Train Epoch: 5 [133/250 17024/32000 (53%)] Loss: 2.46275 (QuantReg: 13.07800) QuantErr: 13.07800 batch_time=0.51342
Train Epoch: 5 [144/250 18432/32000 (58%)] Loss: 1.67978 (QuantReg: 13.13015) QuantErr: 13.13015 batch_time=0.59997
Train Epoch: 5 [155/250 19840/32000 (62%)] Loss: 2.33470 (QuantReg: 13.13824) QuantErr: 13.13824 batch_time=0.61158
Train Epoch: 5 [166/250 21248/32000 (66%)] Loss: 2.29250 (QuantReg: 13.13954) QuantErr: 13.13954 batch_time=0.51421
Train Epoch: 5 [177/250 22656/32000 (71%)] Loss: 2.14612 (QuantReg: 13.57609) QuantErr: 13.57609 batch_time=0.52995
Train Epoch: 5 [188/250 24064/32000 (75%)] Loss: 2.53479 (QuantReg: 12.97376) QuantErr: 12.97376 batch_time=0.56716
Train Epoch: 5 [199/250 25472/32000 (80%)] Loss: 2.10174 (QuantReg: 13.13409) QuantErr: 13.13409 batch_time=0.55033
Train Epoch: 5 [210/250 26880/32000 (84%)] Loss: 2.33059 (QuantReg: 12.74100) QuantErr: 12.74100 batch_time=0.50108
Train Epoch: 5 [221/250 28288/32000 (88%)] Loss: 2.31631 (QuantReg: 13.13864) QuantErr: 13.13864 batch_time=0.52277
Train Epoch: 5 [232/250 29696/32000 (93%)] Loss: 1.93046 (QuantReg: 13.23812) QuantErr: 13.23812 batch_time=0.50725
Train Epoch: 5 [243/250 31104/32000 (97%)] Loss: 2.12162 (QuantReg: 13.16052) QuantErr: 13.16052 batch_time=0.53594
Train Epoch: 5 codebook_update_time=1.75064
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch5.pth ...
Done in 5.467s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch5.pth ...
Done in 10.531s
removing stale ckpt [epoch 4] [took 0.00s]
epoch : 5
loss : 2.307945333003998
quant_reg : 12.910928462982177
quant_err : 12.910928462982177
learning_rate : 4.072531249999999e-05
n_samples : 160000
n_steps : 1250
MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
MSRVTT_full_val/t2v_metrics/R5: 65.59356136820925
MSRVTT_full_val/t2v_metrics/R10: 80.28169014084507
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.951710261569417
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.31128151115823
MSRVTT_full_val/v2t_metrics/R1: 31.388329979879277
MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.729376257545272
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.396003215554785
MSRVTT_full_test/t2v_metrics/R1: 10.133779264214047
MSRVTT_full_test/t2v_metrics/R5: 30.735785953177256
MSRVTT_full_test/t2v_metrics/R10: 44.31438127090301
MSRVTT_full_test/t2v_metrics/R50: 77.59197324414716
MSRVTT_full_test/t2v_metrics/MedR: 14.0
MSRVTT_full_test/t2v_metrics/MeanR: 47.67056856187291
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 23.987601076293736
MSRVTT_full_test/v2t_metrics/R1: 12.240802675585284
MSRVTT_full_test/v2t_metrics/R5: 36.72240802675585
MSRVTT_full_test/v2t_metrics/R10: 51.53846153846154
MSRVTT_full_test/v2t_metrics/R50: 83.4113712374582
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.8566889632107
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 28.50739279254908
mnt_best : 23.987601076293736
not_improved_count: 0
Train Epoch: 6 [1/250 128/32000 (0%)] Loss: 2.43323 (QuantReg: 12.59281) QuantErr: 12.59281 batch_time=32.07245
Train Epoch: 6 [12/250 1536/32000 (5%)] Loss: 2.39548 (QuantReg: 12.35006) QuantErr: 12.35006 batch_time=1.05758
Train Epoch: 6 [23/250 2944/32000 (9%)] Loss: 1.95031 (QuantReg: 12.73042) QuantErr: 12.73042 batch_time=0.51929
Train Epoch: 6 [34/250 4352/32000 (14%)] Loss: 2.23315 (QuantReg: 12.64951) QuantErr: 12.64951 batch_time=0.50162
Train Epoch: 6 [45/250 5760/32000 (18%)] Loss: 1.65812 (QuantReg: 12.72907) QuantErr: 12.72907 batch_time=0.52145
Train Epoch: 6 [56/250 7168/32000 (22%)] Loss: 2.00621 (QuantReg: 13.07515) QuantErr: 13.07515 batch_time=0.62085
Train Epoch: 6 [67/250 8576/32000 (27%)] Loss: 2.10885 (QuantReg: 13.27433) QuantErr: 13.27433 batch_time=0.51519
Train Epoch: 6 [78/250 9984/32000 (31%)] Loss: 1.72104 (QuantReg: 12.83606) QuantErr: 12.83606 batch_time=0.68747
Train Epoch: 6 [89/250 11392/32000 (36%)] Loss: 2.32504 (QuantReg: 12.91540) QuantErr: 12.91540 batch_time=0.51729
Train Epoch: 6 [100/250 12800/32000 (40%)] Loss: 2.07677 (QuantReg: 13.09188) QuantErr: 13.09188 batch_time=0.58237
Train Epoch: 6 [111/250 14208/32000 (44%)] Loss: 2.14010 (QuantReg: 12.60723) QuantErr: 12.60723 batch_time=0.50752
Train Epoch: 6 [122/250 15616/32000 (49%)] Loss: 2.07884 (QuantReg: 13.19710) QuantErr: 13.19710 batch_time=0.51800
Train Epoch: 6 [133/250 17024/32000 (53%)] Loss: 2.04157 (QuantReg: 12.77241) QuantErr: 12.77241 batch_time=0.55455
Train Epoch: 6 [144/250 18432/32000 (58%)] Loss: 2.22533 (QuantReg: 12.71221) QuantErr: 12.71221 batch_time=0.59906
Train Epoch: 6 [155/250 19840/32000 (62%)] Loss: 1.77512 (QuantReg: 13.06599) QuantErr: 13.06599 batch_time=0.51978
Train Epoch: 6 [166/250 21248/32000 (66%)] Loss: 1.79861 (QuantReg: 13.06303) QuantErr: 13.06303 batch_time=0.53226
Train Epoch: 6 [177/250 22656/32000 (71%)] Loss: 2.28922 (QuantReg: 13.32424) QuantErr: 13.32424 batch_time=0.64114
Train Epoch: 6 [188/250 24064/32000 (75%)] Loss: 1.94997 (QuantReg: 13.52094) QuantErr: 13.52094 batch_time=0.50868
Train Epoch: 6 [199/250 25472/32000 (80%)] Loss: 2.10002 (QuantReg: 13.11885) QuantErr: 13.11885 batch_time=0.55993
Train Epoch: 6 [210/250 26880/32000 (84%)] Loss: 1.87879 (QuantReg: 12.95961) QuantErr: 12.95961 batch_time=2.93862
Train Epoch: 6 [221/250 28288/32000 (88%)] Loss: 1.93704 (QuantReg: 13.18013) QuantErr: 13.18013 batch_time=0.52022
Train Epoch: 6 [232/250 29696/32000 (93%)] Loss: 2.09416 (QuantReg: 13.04946) QuantErr: 13.04946 batch_time=0.54019
Train Epoch: 6 [243/250 31104/32000 (97%)] Loss: 2.13997 (QuantReg: 13.08232) QuantErr: 13.08232 batch_time=0.55652
Train Epoch: 6 codebook_update_time=1.78885
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch6.pth ...
Done in 5.838s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch6.pth ...
Done in 11.167s
removing stale ckpt [epoch 5] [took 0.00s]
epoch : 6
loss : 2.1283623571395873
quant_reg : 12.939004829406738
quant_err : 12.939004829406738
learning_rate : 3.868904687499999e-05
n_samples : 192000
n_steps : 1500
MSRVTT_full_val/t2v_metrics/R1: 27.364185110663986
MSRVTT_full_val/t2v_metrics/R5: 66.80080482897384
MSRVTT_full_val/t2v_metrics/R10: 79.87927565392354
MSRVTT_full_val/t2v_metrics/R50: 97.98792756539235
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.219315895372233
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 52.65821227266437
MSRVTT_full_val/v2t_metrics/R1: 32.99798792756539
MSRVTT_full_val/v2t_metrics/R5: 72.43460764587525
MSRVTT_full_val/v2t_metrics/R10: 85.91549295774648
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 3.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.426559356136821
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 58.99769158087788
MSRVTT_full_test/t2v_metrics/R1: 11.070234113712374
MSRVTT_full_test/t2v_metrics/R5: 30.80267558528428
MSRVTT_full_test/t2v_metrics/R10: 44.54849498327759
MSRVTT_full_test/t2v_metrics/R50: 77.72575250836121
MSRVTT_full_test/t2v_metrics/MedR: 13.5
MSRVTT_full_test/t2v_metrics/MeanR: 47.520066889632105
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 24.766202976066804
MSRVTT_full_test/v2t_metrics/R1: 12.775919732441471
MSRVTT_full_test/v2t_metrics/R5: 37.89297658862876
MSRVTT_full_test/v2t_metrics/R10: 50.802675585284284
MSRVTT_full_test/v2t_metrics/R50: 83.17725752508362
MSRVTT_full_test/v2t_metrics/MedR: 10.0
MSRVTT_full_test/v2t_metrics/MeanR: 36.086622073578596
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.081211430610306
mnt_best : 24.766202976066804
not_improved_count: 0
Train Epoch: 7 [1/250 128/32000 (0%)] Loss: 2.19814 (QuantReg: 12.20587) QuantErr: 12.20587 batch_time=33.72107
Train Epoch: 7 [12/250 1536/32000 (5%)] Loss: 1.82900 (QuantReg: 13.04480) QuantErr: 13.04480 batch_time=0.52905
Train Epoch: 7 [23/250 2944/32000 (9%)] Loss: 2.26560 (QuantReg: 12.98315) QuantErr: 12.98315 batch_time=0.52523
Train Epoch: 7 [34/250 4352/32000 (14%)] Loss: 2.05190 (QuantReg: 13.05582) QuantErr: 13.05582 batch_time=0.51353
Train Epoch: 7 [45/250 5760/32000 (18%)] Loss: 1.88160 (QuantReg: 12.71562) QuantErr: 12.71562 batch_time=0.51493
Train Epoch: 7 [56/250 7168/32000 (22%)] Loss: 2.14248 (QuantReg: 12.76773) QuantErr: 12.76773 batch_time=0.53809
Train Epoch: 7 [67/250 8576/32000 (27%)] Loss: 2.08655 (QuantReg: 13.18715) QuantErr: 13.18715 batch_time=1.77644
Train Epoch: 7 [78/250 9984/32000 (31%)] Loss: 2.23222 (QuantReg: 12.98177) QuantErr: 12.98177 batch_time=4.17743
Train Epoch: 7 [89/250 11392/32000 (36%)] Loss: 2.15620 (QuantReg: 12.91958) QuantErr: 12.91958 batch_time=0.56924
Train Epoch: 7 [100/250 12800/32000 (40%)] Loss: 1.93651 (QuantReg: 12.95315) QuantErr: 12.95315 batch_time=0.54048
Train Epoch: 7 [111/250 14208/32000 (44%)] Loss: 2.32943 (QuantReg: 12.98421) QuantErr: 12.98421 batch_time=0.57481
Train Epoch: 7 [122/250 15616/32000 (49%)] Loss: 1.51292 (QuantReg: 13.30172) QuantErr: 13.30172 batch_time=0.53275
Train Epoch: 7 [133/250 17024/32000 (53%)] Loss: 1.70469 (QuantReg: 12.93480) QuantErr: 12.93480 batch_time=0.82681
Train Epoch: 7 [144/250 18432/32000 (58%)] Loss: 2.13421 (QuantReg: 12.94695) QuantErr: 12.94695 batch_time=0.60380
Train Epoch: 7 [155/250 19840/32000 (62%)] Loss: 2.05990 (QuantReg: 12.77875) QuantErr: 12.77875 batch_time=0.50249
Train Epoch: 7 [166/250 21248/32000 (66%)] Loss: 1.75664 (QuantReg: 13.15047) QuantErr: 13.15047 batch_time=0.50938
Train Epoch: 7 [177/250 22656/32000 (71%)] Loss: 1.65060 (QuantReg: 13.36147) QuantErr: 13.36147 batch_time=0.54820
Train Epoch: 7 [188/250 24064/32000 (75%)] Loss: 1.98485 (QuantReg: 12.86060) QuantErr: 12.86060 batch_time=0.57867
Train Epoch: 7 [199/250 25472/32000 (80%)] Loss: 1.44366 (QuantReg: 13.21458) QuantErr: 13.21458 batch_time=0.49937
Train Epoch: 7 [210/250 26880/32000 (84%)] Loss: 1.75860 (QuantReg: 13.31701) QuantErr: 13.31701 batch_time=0.59342
Train Epoch: 7 [221/250 28288/32000 (88%)] Loss: 2.37195 (QuantReg: 13.14990) QuantErr: 13.14990 batch_time=0.52338
Train Epoch: 7 [232/250 29696/32000 (93%)] Loss: 1.87897 (QuantReg: 13.05642) QuantErr: 13.05642 batch_time=0.52948
Train Epoch: 7 [243/250 31104/32000 (97%)] Loss: 2.10383 (QuantReg: 12.98402) QuantErr: 12.98402 batch_time=0.50808
Train Epoch: 7 codebook_update_time=1.77936
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch7.pth ...
Done in 4.939s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch7.pth ...
Done in 10.288s
removing stale ckpt [epoch 6] [took 0.01s]
epoch : 7
loss : 1.969263916015625
quant_reg : 13.039780002593995
quant_err : 13.039780002593995
learning_rate : 3.675459453124999e-05
n_samples : 224000
n_steps : 1750
MSRVTT_full_val/t2v_metrics/R1: 27.96780684104628
MSRVTT_full_val/t2v_metrics/R5: 67.40442655935614
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.448692152917506
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.42429888399108
MSRVTT_full_val/v2t_metrics/R1: 35.2112676056338
MSRVTT_full_val/v2t_metrics/R5: 74.44668008048289
MSRVTT_full_val/v2t_metrics/R10: 87.12273641851107
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.311871227364185
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 61.125088506080246
MSRVTT_full_test/t2v_metrics/R1: 11.304347826086957
MSRVTT_full_test/t2v_metrics/R5: 32.17391304347826
MSRVTT_full_test/t2v_metrics/R10: 45.65217391304348
MSRVTT_full_test/t2v_metrics/R50: 79.09698996655518
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.50033444816054
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.511556294516378
MSRVTT_full_test/v2t_metrics/R1: 12.775919732441471
MSRVTT_full_test/v2t_metrics/R5: 37.324414715719065
MSRVTT_full_test/v2t_metrics/R10: 52.207357859531776
MSRVTT_full_test/v2t_metrics/R50: 83.51170568561874
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.594314381270905
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 29.19929047831319
mnt_best : 25.511556294516378
not_improved_count: 0
Train Epoch: 8 [1/250 128/32000 (0%)] Loss: 1.87330 (QuantReg: 13.04616) QuantErr: 13.04616 batch_time=48.80918
Train Epoch: 8 [12/250 1536/32000 (5%)] Loss: 2.10767 (QuantReg: 13.07306) QuantErr: 13.07306 batch_time=0.51314
Train Epoch: 8 [23/250 2944/32000 (9%)] Loss: 2.45450 (QuantReg: 12.72779) QuantErr: 12.72779 batch_time=0.52556
Train Epoch: 8 [34/250 4352/32000 (14%)] Loss: 1.92778 (QuantReg: 13.15167) QuantErr: 13.15167 batch_time=0.55300
Train Epoch: 8 [45/250 5760/32000 (18%)] Loss: 2.01784 (QuantReg: 13.04458) QuantErr: 13.04458 batch_time=0.51110
Train Epoch: 8 [56/250 7168/32000 (22%)] Loss: 1.93510 (QuantReg: 12.85777) QuantErr: 12.85777 batch_time=0.50729
Train Epoch: 8 [67/250 8576/32000 (27%)] Loss: 1.72873 (QuantReg: 13.35143) QuantErr: 13.35143 batch_time=0.57575
Train Epoch: 8 [78/250 9984/32000 (31%)] Loss: 1.66089 (QuantReg: 13.30352) QuantErr: 13.30352 batch_time=0.52599
Train Epoch: 8 [89/250 11392/32000 (36%)] Loss: 2.03761 (QuantReg: 13.10351) QuantErr: 13.10351 batch_time=0.59462
Train Epoch: 8 [100/250 12800/32000 (40%)] Loss: 1.85146 (QuantReg: 13.05675) QuantErr: 13.05675 batch_time=0.54242
Train Epoch: 8 [111/250 14208/32000 (44%)] Loss: 1.76710 (QuantReg: 13.29462) QuantErr: 13.29462 batch_time=0.50622
Train Epoch: 8 [122/250 15616/32000 (49%)] Loss: 2.42687 (QuantReg: 13.28074) QuantErr: 13.28074 batch_time=0.53093
Train Epoch: 8 [133/250 17024/32000 (53%)] Loss: 1.71391 (QuantReg: 13.10159) QuantErr: 13.10159 batch_time=0.51321
Train Epoch: 8 [144/250 18432/32000 (58%)] Loss: 1.70446 (QuantReg: 13.22628) QuantErr: 13.22628 batch_time=0.53027
Train Epoch: 8 [155/250 19840/32000 (62%)] Loss: 2.25578 (QuantReg: 13.34348) QuantErr: 13.34348 batch_time=0.54440
Train Epoch: 8 [166/250 21248/32000 (66%)] Loss: 1.56347 (QuantReg: 13.05509) QuantErr: 13.05509 batch_time=0.58236
Train Epoch: 8 [177/250 22656/32000 (71%)] Loss: 2.05368 (QuantReg: 13.27384) QuantErr: 13.27384 batch_time=0.63296
Train Epoch: 8 [188/250 24064/32000 (75%)] Loss: 2.02899 (QuantReg: 13.43963) QuantErr: 13.43963 batch_time=0.53794
Train Epoch: 8 [199/250 25472/32000 (80%)] Loss: 1.50685 (QuantReg: 13.32815) QuantErr: 13.32815 batch_time=0.51095
Train Epoch: 8 [210/250 26880/32000 (84%)] Loss: 1.40466 (QuantReg: 13.22969) QuantErr: 13.22969 batch_time=0.52983
Train Epoch: 8 [221/250 28288/32000 (88%)] Loss: 1.45768 (QuantReg: 13.62514) QuantErr: 13.62514 batch_time=0.52894
Train Epoch: 8 [232/250 29696/32000 (93%)] Loss: 1.11435 (QuantReg: 13.16264) QuantErr: 13.16264 batch_time=0.52571
Train Epoch: 8 [243/250 31104/32000 (97%)] Loss: 1.45110 (QuantReg: 13.21988) QuantErr: 13.21988 batch_time=0.51353
Train Epoch: 8 codebook_update_time=1.69746
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch8.pth ...
Done in 6.289s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch8.pth ...
Done in 11.232s
removing stale ckpt [epoch 7] [took 0.00s]
epoch : 8
loss : 1.8663319206237794
quant_reg : 13.100959823608399
quant_err : 13.100959823608399
learning_rate : 3.4916864804687486e-05
n_samples : 256000
n_steps : 2000
MSRVTT_full_val/t2v_metrics/R1: 28.772635814889338
MSRVTT_full_val/t2v_metrics/R5: 66.59959758551308
MSRVTT_full_val/t2v_metrics/R10: 80.0804828973843
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.388329979879277
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 53.537651007573814
MSRVTT_full_val/v2t_metrics/R1: 37.223340040241446
MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
MSRVTT_full_val/v2t_metrics/R10: 86.72032193158954
MSRVTT_full_val/v2t_metrics/R50: 98.39034205231388
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.162977867203219
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.00337057231466
MSRVTT_full_test/t2v_metrics/R1: 11.538461538461538
MSRVTT_full_test/t2v_metrics/R5: 32.876254180602004
MSRVTT_full_test/t2v_metrics/R10: 45.852842809364546
MSRVTT_full_test/t2v_metrics/R50: 78.69565217391305
MSRVTT_full_test/t2v_metrics/MedR: 13.0
MSRVTT_full_test/t2v_metrics/MeanR: 44.973578595317726
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 25.90988592567972
MSRVTT_full_test/v2t_metrics/R1: 13.478260869565217
MSRVTT_full_test/v2t_metrics/R5: 37.95986622073578
MSRVTT_full_test/v2t_metrics/R10: 53.01003344481605
MSRVTT_full_test/v2t_metrics/R50: 84.04682274247492
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 34.003010033444816
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.044999635714092
mnt_best : 25.90988592567972
not_improved_count: 0
Train Epoch: 9 [1/250 128/32000 (0%)] Loss: 2.07492 (QuantReg: 12.73485) QuantErr: 12.73485 batch_time=37.22838
Train Epoch: 9 [12/250 1536/32000 (5%)] Loss: 1.74446 (QuantReg: 12.85844) QuantErr: 12.85844 batch_time=0.58604
Train Epoch: 9 [23/250 2944/32000 (9%)] Loss: 2.14512 (QuantReg: 13.02880) QuantErr: 13.02880 batch_time=0.55917
Train Epoch: 9 [34/250 4352/32000 (14%)] Loss: 2.06795 (QuantReg: 12.99054) QuantErr: 12.99054 batch_time=0.49657
Train Epoch: 9 [45/250 5760/32000 (18%)] Loss: 2.17634 (QuantReg: 12.83534) QuantErr: 12.83534 batch_time=0.83757
Train Epoch: 9 [56/250 7168/32000 (22%)] Loss: 2.00580 (QuantReg: 12.91172) QuantErr: 12.91172 batch_time=0.51150
Train Epoch: 9 [67/250 8576/32000 (27%)] Loss: 1.75840 (QuantReg: 12.94296) QuantErr: 12.94296 batch_time=0.54296
Train Epoch: 9 [78/250 9984/32000 (31%)] Loss: 1.44052 (QuantReg: 13.17063) QuantErr: 13.17063 batch_time=0.50867
Train Epoch: 9 [89/250 11392/32000 (36%)] Loss: 2.04947 (QuantReg: 12.98122) QuantErr: 12.98122 batch_time=0.50540
Train Epoch: 9 [100/250 12800/32000 (40%)] Loss: 1.74426 (QuantReg: 13.18357) QuantErr: 13.18357 batch_time=0.52247
Train Epoch: 9 [111/250 14208/32000 (44%)] Loss: 1.86536 (QuantReg: 13.03939) QuantErr: 13.03939 batch_time=0.49561
Train Epoch: 9 [122/250 15616/32000 (49%)] Loss: 1.64122 (QuantReg: 13.28896) QuantErr: 13.28896 batch_time=0.50740
Train Epoch: 9 [133/250 17024/32000 (53%)] Loss: 1.80711 (QuantReg: 13.20088) QuantErr: 13.20088 batch_time=0.61876
Train Epoch: 9 [144/250 18432/32000 (58%)] Loss: 1.89100 (QuantReg: 13.36098) QuantErr: 13.36098 batch_time=0.54902
Train Epoch: 9 [155/250 19840/32000 (62%)] Loss: 1.93474 (QuantReg: 13.32176) QuantErr: 13.32176 batch_time=0.52054
Train Epoch: 9 [166/250 21248/32000 (66%)] Loss: 1.81518 (QuantReg: 13.09709) QuantErr: 13.09709 batch_time=0.52183
Train Epoch: 9 [177/250 22656/32000 (71%)] Loss: 1.35248 (QuantReg: 13.33693) QuantErr: 13.33693 batch_time=0.50165
Train Epoch: 9 [188/250 24064/32000 (75%)] Loss: 1.44160 (QuantReg: 13.39390) QuantErr: 13.39390 batch_time=0.55578
Train Epoch: 9 [199/250 25472/32000 (80%)] Loss: 1.52464 (QuantReg: 13.33956) QuantErr: 13.33956 batch_time=6.34923
Train Epoch: 9 [210/250 26880/32000 (84%)] Loss: 1.49200 (QuantReg: 13.04947) QuantErr: 13.04947 batch_time=0.83069
Train Epoch: 9 [221/250 28288/32000 (88%)] Loss: 1.65558 (QuantReg: 13.24535) QuantErr: 13.24535 batch_time=0.58047
Train Epoch: 9 [232/250 29696/32000 (93%)] Loss: 1.68248 (QuantReg: 13.31828) QuantErr: 13.31828 batch_time=0.55306
Train Epoch: 9 [243/250 31104/32000 (97%)] Loss: 1.18596 (QuantReg: 13.48554) QuantErr: 13.48554 batch_time=0.52817
Train Epoch: 9 codebook_update_time=1.83558
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch9.pth ...
Done in 15.307s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch9.pth ...
Done in 20.123s
removing stale ckpt [epoch 8] [took 0.01s]
epoch : 9
loss : 1.7392348456382751
quant_reg : 13.151548904418945
quant_err : 13.151548904418945
learning_rate : 3.317102156445311e-05
n_samples : 288000
n_steps : 2250
MSRVTT_full_val/t2v_metrics/R1: 32.59557344064386
MSRVTT_full_val/t2v_metrics/R5: 67.80684104627767
MSRVTT_full_val/t2v_metrics/R10: 81.89134808853119
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.150905432595573
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.56616284819279
MSRVTT_full_val/v2t_metrics/R1: 37.82696177062374
MSRVTT_full_val/v2t_metrics/R5: 75.25150905432595
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 97.98792756539235
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.293762575452717
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.68200989030519
MSRVTT_full_test/t2v_metrics/R1: 11.839464882943144
MSRVTT_full_test/t2v_metrics/R5: 35.11705685618729
MSRVTT_full_test/t2v_metrics/R10: 48.49498327759197
MSRVTT_full_test/t2v_metrics/R50: 80.8695652173913
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.13879598662207
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 27.217548122310383
MSRVTT_full_test/v2t_metrics/R1: 14.548494983277592
MSRVTT_full_test/v2t_metrics/R5: 39.46488294314381
MSRVTT_full_test/v2t_metrics/R10: 54.81605351170568
MSRVTT_full_test/v2t_metrics/R50: 85.15050167224081
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.46354515050167
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.57273595783139
mnt_best : 27.217548122310383
not_improved_count: 0
Train Epoch: 10 [1/250 128/32000 (0%)] Loss: 1.42912 (QuantReg: 12.74387) QuantErr: 12.74387 batch_time=37.21371
Train Epoch: 10 [12/250 1536/32000 (5%)] Loss: 1.63293 (QuantReg: 13.08670) QuantErr: 13.08670 batch_time=0.53423
Train Epoch: 10 [23/250 2944/32000 (9%)] Loss: 1.46775 (QuantReg: 12.85679) QuantErr: 12.85679 batch_time=0.50560
Train Epoch: 10 [34/250 4352/32000 (14%)] Loss: 1.32581 (QuantReg: 12.72069) QuantErr: 12.72069 batch_time=0.56423
Train Epoch: 10 [45/250 5760/32000 (18%)] Loss: 1.53061 (QuantReg: 13.40966) QuantErr: 13.40966 batch_time=0.55802
Train Epoch: 10 [56/250 7168/32000 (22%)] Loss: 2.29735 (QuantReg: 13.11296) QuantErr: 13.11296 batch_time=0.53911
Train Epoch: 10 [67/250 8576/32000 (27%)] Loss: 1.41997 (QuantReg: 13.25578) QuantErr: 13.25578 batch_time=1.20958
Train Epoch: 10 [78/250 9984/32000 (31%)] Loss: 1.62925 (QuantReg: 12.94425) QuantErr: 12.94425 batch_time=0.80345
Train Epoch: 10 [89/250 11392/32000 (36%)] Loss: 1.66447 (QuantReg: 13.07171) QuantErr: 13.07171 batch_time=0.51656
Train Epoch: 10 [100/250 12800/32000 (40%)] Loss: 2.12603 (QuantReg: 12.85455) QuantErr: 12.85455 batch_time=0.50771
Train Epoch: 10 [111/250 14208/32000 (44%)] Loss: 1.45623 (QuantReg: 13.23011) QuantErr: 13.23011 batch_time=0.51588
Train Epoch: 10 [122/250 15616/32000 (49%)] Loss: 1.82170 (QuantReg: 13.30667) QuantErr: 13.30667 batch_time=0.60373
Train Epoch: 10 [133/250 17024/32000 (53%)] Loss: 1.95414 (QuantReg: 13.27752) QuantErr: 13.27752 batch_time=0.54087
Train Epoch: 10 [144/250 18432/32000 (58%)] Loss: 2.14669 (QuantReg: 13.38997) QuantErr: 13.38997 batch_time=0.50757
Train Epoch: 10 [155/250 19840/32000 (62%)] Loss: 1.29556 (QuantReg: 13.57348) QuantErr: 13.57348 batch_time=0.62654
Train Epoch: 10 [166/250 21248/32000 (66%)] Loss: 1.31944 (QuantReg: 13.45149) QuantErr: 13.45149 batch_time=0.79560
Train Epoch: 10 [177/250 22656/32000 (71%)] Loss: 2.51692 (QuantReg: 13.12911) QuantErr: 13.12911 batch_time=0.62917
Train Epoch: 10 [188/250 24064/32000 (75%)] Loss: 1.47032 (QuantReg: 13.65932) QuantErr: 13.65932 batch_time=0.52598
Train Epoch: 10 [199/250 25472/32000 (80%)] Loss: 1.39781 (QuantReg: 13.68075) QuantErr: 13.68075 batch_time=0.60411
Train Epoch: 10 [210/250 26880/32000 (84%)] Loss: 1.54278 (QuantReg: 13.24539) QuantErr: 13.24539 batch_time=0.52263
Train Epoch: 10 [221/250 28288/32000 (88%)] Loss: 1.50842 (QuantReg: 13.08505) QuantErr: 13.08505 batch_time=0.51790
Train Epoch: 10 [232/250 29696/32000 (93%)] Loss: 2.16304 (QuantReg: 13.25863) QuantErr: 13.25863 batch_time=0.66401
Train Epoch: 10 [243/250 31104/32000 (97%)] Loss: 1.81186 (QuantReg: 13.14316) QuantErr: 13.14316 batch_time=0.55396
Train Epoch: 10 codebook_update_time=1.85086
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch10.pth ...
Done in 21.654s
removing stale ckpt [epoch 9] [took 0.00s]
epoch : 10
loss : 1.6461696848869323
quant_reg : 13.193038208007813
quant_err : 13.193038208007813
learning_rate : 3.151247048623045e-05
n_samples : 320000
n_steps : 2500
MSRVTT_full_val/t2v_metrics/R1: 31.790744466800806
MSRVTT_full_val/t2v_metrics/R5: 67.80684104627767
MSRVTT_full_val/t2v_metrics/R10: 82.09255533199195
MSRVTT_full_val/t2v_metrics/R50: 97.58551307847083
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.185110663983902
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.142618143523045
MSRVTT_full_val/v2t_metrics/R1: 37.223340040241446
MSRVTT_full_val/v2t_metrics/R5: 75.0503018108652
MSRVTT_full_val/v2t_metrics/R10: 87.5251509054326
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.301810865191147
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.53169731125111
MSRVTT_full_test/t2v_metrics/R1: 11.605351170568563
MSRVTT_full_test/t2v_metrics/R5: 33.779264214046826
MSRVTT_full_test/t2v_metrics/R10: 46.8561872909699
MSRVTT_full_test/t2v_metrics/R50: 80.43478260869566
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.374581939799334
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.385083502996284
MSRVTT_full_test/v2t_metrics/R1: 13.511705685618729
MSRVTT_full_test/v2t_metrics/R5: 38.62876254180602
MSRVTT_full_test/v2t_metrics/R10: 54.280936454849495
MSRVTT_full_test/v2t_metrics/R50: 86.22073578595318
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 31.062709030100333
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 30.485227105584265
mnt_best : 27.217548122310383
not_improved_count: 1
Train Epoch: 11 [1/250 128/32000 (0%)] Loss: 1.48286 (QuantReg: 13.02585) QuantErr: 13.02585 batch_time=33.15067
Train Epoch: 11 [12/250 1536/32000 (5%)] Loss: 1.37539 (QuantReg: 13.24955) QuantErr: 13.24955 batch_time=0.51270
Train Epoch: 11 [23/250 2944/32000 (9%)] Loss: 1.54585 (QuantReg: 12.90776) QuantErr: 12.90776 batch_time=0.53091
Train Epoch: 11 [34/250 4352/32000 (14%)] Loss: 1.61331 (QuantReg: 13.25153) QuantErr: 13.25153 batch_time=0.56605
Train Epoch: 11 [45/250 5760/32000 (18%)] Loss: 1.70705 (QuantReg: 13.49713) QuantErr: 13.49713 batch_time=0.50888
Train Epoch: 11 [56/250 7168/32000 (22%)] Loss: 1.34999 (QuantReg: 13.29250) QuantErr: 13.29250 batch_time=0.52885
Train Epoch: 11 [67/250 8576/32000 (27%)] Loss: 1.75094 (QuantReg: 13.03591) QuantErr: 13.03591 batch_time=0.60391
Train Epoch: 11 [78/250 9984/32000 (31%)] Loss: 1.47455 (QuantReg: 13.49845) QuantErr: 13.49845 batch_time=0.52979
Train Epoch: 11 [89/250 11392/32000 (36%)] Loss: 1.53572 (QuantReg: 13.11505) QuantErr: 13.11505 batch_time=0.54596
Train Epoch: 11 [100/250 12800/32000 (40%)] Loss: 1.75581 (QuantReg: 13.06962) QuantErr: 13.06962 batch_time=2.37528
Train Epoch: 11 [111/250 14208/32000 (44%)] Loss: 1.87041 (QuantReg: 13.07144) QuantErr: 13.07144 batch_time=0.59601
Train Epoch: 11 [122/250 15616/32000 (49%)] Loss: 1.60434 (QuantReg: 13.56801) QuantErr: 13.56801 batch_time=1.02082
Train Epoch: 11 [133/250 17024/32000 (53%)] Loss: 1.98561 (QuantReg: 12.98133) QuantErr: 12.98133 batch_time=0.52030
Train Epoch: 11 [144/250 18432/32000 (58%)] Loss: 1.74221 (QuantReg: 13.10519) QuantErr: 13.10519 batch_time=0.58000
Train Epoch: 11 [155/250 19840/32000 (62%)] Loss: 1.62246 (QuantReg: 13.10350) QuantErr: 13.10350 batch_time=0.51079
Train Epoch: 11 [166/250 21248/32000 (66%)] Loss: 1.59495 (QuantReg: 13.00852) QuantErr: 13.00852 batch_time=0.54019
Train Epoch: 11 [177/250 22656/32000 (71%)] Loss: 1.58029 (QuantReg: 12.99436) QuantErr: 12.99436 batch_time=0.69323
Train Epoch: 11 [188/250 24064/32000 (75%)] Loss: 1.26559 (QuantReg: 13.25383) QuantErr: 13.25383 batch_time=0.50571
Train Epoch: 11 [199/250 25472/32000 (80%)] Loss: 1.37681 (QuantReg: 13.70844) QuantErr: 13.70844 batch_time=1.05126
Train Epoch: 11 [210/250 26880/32000 (84%)] Loss: 1.67211 (QuantReg: 13.45811) QuantErr: 13.45811 batch_time=0.56804
Train Epoch: 11 [221/250 28288/32000 (88%)] Loss: 1.64100 (QuantReg: 13.63800) QuantErr: 13.63800 batch_time=0.54274
Train Epoch: 11 [232/250 29696/32000 (93%)] Loss: 1.32532 (QuantReg: 13.44054) QuantErr: 13.44054 batch_time=0.54371
Train Epoch: 11 [243/250 31104/32000 (97%)] Loss: 1.17232 (QuantReg: 13.83262) QuantErr: 13.83262 batch_time=0.51786
Train Epoch: 11 codebook_update_time=1.72605
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch11.pth ...
Done in 5.937s
removing stale ckpt [epoch 10] [took 0.00s]
epoch : 11
loss : 1.5532874684333802
quant_reg : 13.290080509185792
quant_err : 13.290080509185792
learning_rate : 2.993684696191893e-05
n_samples : 352000
n_steps : 2750
MSRVTT_full_val/t2v_metrics/R1: 32.99798792756539
MSRVTT_full_val/t2v_metrics/R5: 67.20321931589537
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.1830985915493
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 9.132796780684105
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.39609939023545
MSRVTT_full_val/v2t_metrics/R1: 37.625754527162975
MSRVTT_full_val/v2t_metrics/R5: 73.8430583501006
MSRVTT_full_val/v2t_metrics/R10: 86.51911468812877
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.414486921529175
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.17784320760863
MSRVTT_full_test/t2v_metrics/R1: 11.605351170568563
MSRVTT_full_test/t2v_metrics/R5: 33.41137123745819
MSRVTT_full_test/t2v_metrics/R10: 46.58862876254181
MSRVTT_full_test/t2v_metrics/R50: 80.0
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 45.689632107023414
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.238812220433232
MSRVTT_full_test/v2t_metrics/R1: 14.648829431438127
MSRVTT_full_test/v2t_metrics/R5: 40.36789297658863
MSRVTT_full_test/v2t_metrics/R10: 55.01672240802676
MSRVTT_full_test/v2t_metrics/R50: 85.65217391304348
MSRVTT_full_test/v2t_metrics/MedR: 9.0
MSRVTT_full_test/v2t_metrics/MeanR: 32.305351170568564
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.923554338135457
mnt_best : 27.217548122310383
not_improved_count: 2
Train Epoch: 12 [1/250 128/32000 (0%)] Loss: 1.55157 (QuantReg: 13.14119) QuantErr: 13.14119 batch_time=33.41529
Train Epoch: 12 [12/250 1536/32000 (5%)] Loss: 1.73068 (QuantReg: 13.28666) QuantErr: 13.28666 batch_time=0.52839
Train Epoch: 12 [23/250 2944/32000 (9%)] Loss: 1.35408 (QuantReg: 13.06172) QuantErr: 13.06172 batch_time=0.57427
Train Epoch: 12 [34/250 4352/32000 (14%)] Loss: 1.39330 (QuantReg: 13.16000) QuantErr: 13.16000 batch_time=0.51798
Train Epoch: 12 [45/250 5760/32000 (18%)] Loss: 1.21143 (QuantReg: 13.45654) QuantErr: 13.45654 batch_time=0.80532
Train Epoch: 12 [56/250 7168/32000 (22%)] Loss: 1.56333 (QuantReg: 13.30915) QuantErr: 13.30915 batch_time=0.53929
Train Epoch: 12 [67/250 8576/32000 (27%)] Loss: 1.27319 (QuantReg: 13.34136) QuantErr: 13.34136 batch_time=0.58462
Train Epoch: 12 [78/250 9984/32000 (31%)] Loss: 1.53792 (QuantReg: 13.23739) QuantErr: 13.23739 batch_time=0.49646
Train Epoch: 12 [89/250 11392/32000 (36%)] Loss: 1.33294 (QuantReg: 13.23725) QuantErr: 13.23725 batch_time=0.63517
Train Epoch: 12 [100/250 12800/32000 (40%)] Loss: 1.64892 (QuantReg: 13.26639) QuantErr: 13.26639 batch_time=1.16639
Train Epoch: 12 [111/250 14208/32000 (44%)] Loss: 1.17716 (QuantReg: 13.23534) QuantErr: 13.23534 batch_time=0.54432
Train Epoch: 12 [122/250 15616/32000 (49%)] Loss: 1.60701 (QuantReg: 13.25420) QuantErr: 13.25420 batch_time=0.54104
Train Epoch: 12 [133/250 17024/32000 (53%)] Loss: 1.25334 (QuantReg: 13.40355) QuantErr: 13.40355 batch_time=0.55704
Train Epoch: 12 [144/250 18432/32000 (58%)] Loss: 1.18459 (QuantReg: 13.84800) QuantErr: 13.84800 batch_time=0.54284
Train Epoch: 12 [155/250 19840/32000 (62%)] Loss: 1.59435 (QuantReg: 13.15264) QuantErr: 13.15264 batch_time=0.53993
Train Epoch: 12 [166/250 21248/32000 (66%)] Loss: 1.07408 (QuantReg: 13.45195) QuantErr: 13.45195 batch_time=0.49994
Train Epoch: 12 [177/250 22656/32000 (71%)] Loss: 1.12423 (QuantReg: 13.42719) QuantErr: 13.42719 batch_time=0.55354
Train Epoch: 12 [188/250 24064/32000 (75%)] Loss: 1.81575 (QuantReg: 13.11192) QuantErr: 13.11192 batch_time=0.52634
Train Epoch: 12 [199/250 25472/32000 (80%)] Loss: 1.53057 (QuantReg: 13.15447) QuantErr: 13.15447 batch_time=0.50856
Train Epoch: 12 [210/250 26880/32000 (84%)] Loss: 1.84784 (QuantReg: 13.22834) QuantErr: 13.22834 batch_time=0.52934
Train Epoch: 12 [221/250 28288/32000 (88%)] Loss: 1.50352 (QuantReg: 13.01210) QuantErr: 13.01210 batch_time=0.54873
Train Epoch: 12 [232/250 29696/32000 (93%)] Loss: 1.48036 (QuantReg: 13.46206) QuantErr: 13.46206 batch_time=0.52963
Train Epoch: 12 [243/250 31104/32000 (97%)] Loss: 1.59590 (QuantReg: 13.67399) QuantErr: 13.67399 batch_time=0.51991
Train Epoch: 12 codebook_update_time=1.67101
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch12.pth ...
Done in 4.341s
removing stale ckpt [epoch 11] [took 0.00s]
epoch : 12
loss : 1.4816722507476807
quant_reg : 13.329390342712403
quant_err : 13.329390342712403
learning_rate : 2.844000461382298e-05
n_samples : 384000
n_steps : 3000
MSRVTT_full_val/t2v_metrics/R1: 32.394366197183096
MSRVTT_full_val/t2v_metrics/R5: 69.41649899396378
MSRVTT_full_val/t2v_metrics/R10: 81.28772635814889
MSRVTT_full_val/t2v_metrics/R50: 97.38430583501005
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.388329979879277
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 56.75259486921189
MSRVTT_full_val/v2t_metrics/R1: 36.41851106639839
MSRVTT_full_val/v2t_metrics/R5: 77.06237424547284
MSRVTT_full_val/v2t_metrics/R10: 86.72032193158954
MSRVTT_full_val/v2t_metrics/R50: 98.18913480885311
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.9738430583501
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.43505293498117
MSRVTT_full_test/t2v_metrics/R1: 11.003344481605351
MSRVTT_full_test/t2v_metrics/R5: 34.81605351170568
MSRVTT_full_test/t2v_metrics/R10: 48.32775919732441
MSRVTT_full_test/t2v_metrics/R50: 80.46822742474916
MSRVTT_full_test/t2v_metrics/MedR: 11.0
MSRVTT_full_test/t2v_metrics/MeanR: 42.86722408026756
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.45454541590052
MSRVTT_full_test/v2t_metrics/R1: 14.581939799331103
MSRVTT_full_test/v2t_metrics/R5: 41.23745819397993
MSRVTT_full_test/v2t_metrics/R10: 56.88963210702341
MSRVTT_full_test/v2t_metrics/R50: 86.88963210702342
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 29.939297658862877
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 32.46236113386705
mnt_best : 27.217548122310383
not_improved_count: 3
Train Epoch: 13 [1/250 128/32000 (0%)] Loss: 1.37490 (QuantReg: 13.25366) QuantErr: 13.25366 batch_time=37.72533
Train Epoch: 13 [12/250 1536/32000 (5%)] Loss: 1.95781 (QuantReg: 12.84915) QuantErr: 12.84915 batch_time=0.52168
Train Epoch: 13 [23/250 2944/32000 (9%)] Loss: 1.48239 (QuantReg: 13.48985) QuantErr: 13.48985 batch_time=0.51385
Train Epoch: 13 [34/250 4352/32000 (14%)] Loss: 1.24174 (QuantReg: 13.40216) QuantErr: 13.40216 batch_time=0.59127
Train Epoch: 13 [45/250 5760/32000 (18%)] Loss: 1.37891 (QuantReg: 13.37562) QuantErr: 13.37562 batch_time=0.61661
Train Epoch: 13 [56/250 7168/32000 (22%)] Loss: 1.41293 (QuantReg: 13.58906) QuantErr: 13.58906 batch_time=0.51335
Train Epoch: 13 [67/250 8576/32000 (27%)] Loss: 1.53375 (QuantReg: 13.50543) QuantErr: 13.50543 batch_time=2.38206
Train Epoch: 13 [78/250 9984/32000 (31%)] Loss: 1.82068 (QuantReg: 13.59470) QuantErr: 13.59470 batch_time=0.52690
Train Epoch: 13 [89/250 11392/32000 (36%)] Loss: 1.15824 (QuantReg: 13.01388) QuantErr: 13.01388 batch_time=0.54336
Train Epoch: 13 [100/250 12800/32000 (40%)] Loss: 1.58661 (QuantReg: 13.33694) QuantErr: 13.33694 batch_time=0.53280
Train Epoch: 13 [111/250 14208/32000 (44%)] Loss: 1.56233 (QuantReg: 13.21690) QuantErr: 13.21690 batch_time=0.61534
Train Epoch: 13 [122/250 15616/32000 (49%)] Loss: 1.05005 (QuantReg: 13.52625) QuantErr: 13.52625 batch_time=0.55593
Train Epoch: 13 [133/250 17024/32000 (53%)] Loss: 1.30977 (QuantReg: 13.54184) QuantErr: 13.54184 batch_time=0.53681
Train Epoch: 13 [144/250 18432/32000 (58%)] Loss: 1.40533 (QuantReg: 13.59767) QuantErr: 13.59767 batch_time=0.54187
Train Epoch: 13 [155/250 19840/32000 (62%)] Loss: 1.67591 (QuantReg: 13.21141) QuantErr: 13.21141 batch_time=0.56422
Train Epoch: 13 [166/250 21248/32000 (66%)] Loss: 1.24999 (QuantReg: 13.22890) QuantErr: 13.22890 batch_time=0.51847
Train Epoch: 13 [177/250 22656/32000 (71%)] Loss: 1.15345 (QuantReg: 13.67255) QuantErr: 13.67255 batch_time=0.59439
Train Epoch: 13 [188/250 24064/32000 (75%)] Loss: 1.82404 (QuantReg: 13.25767) QuantErr: 13.25767 batch_time=0.60758
Train Epoch: 13 [199/250 25472/32000 (80%)] Loss: 1.12570 (QuantReg: 13.77686) QuantErr: 13.77686 batch_time=0.51628
Train Epoch: 13 [210/250 26880/32000 (84%)] Loss: 1.79184 (QuantReg: 13.56978) QuantErr: 13.56978 batch_time=0.56352
Train Epoch: 13 [221/250 28288/32000 (88%)] Loss: 1.71718 (QuantReg: 13.59709) QuantErr: 13.59709 batch_time=0.52322
Train Epoch: 13 [232/250 29696/32000 (93%)] Loss: 1.23038 (QuantReg: 13.44609) QuantErr: 13.44609 batch_time=0.55321
Train Epoch: 13 [243/250 31104/32000 (97%)] Loss: 1.41079 (QuantReg: 13.57304) QuantErr: 13.57304 batch_time=0.53195
Train Epoch: 13 codebook_update_time=1.78294
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch13.pth ...
Done in 4.536s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch13.pth ...
Done in 8.899s
removing stale ckpt [epoch 12] [took 0.00s]
epoch : 13
loss : 1.4125195915699005
quant_reg : 13.40679708480835
quant_err : 13.40679708480835
learning_rate : 2.7018004383131832e-05
n_samples : 416000
n_steps : 3250
MSRVTT_full_val/t2v_metrics/R1: 34.00402414486921
MSRVTT_full_val/t2v_metrics/R5: 70.82494969818913
MSRVTT_full_val/t2v_metrics/R10: 80.88531187122736
MSRVTT_full_val/t2v_metrics/R50: 97.78672032193158
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 7.9476861167002015
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 57.96894790737305
MSRVTT_full_val/v2t_metrics/R1: 38.43058350100603
MSRVTT_full_val/v2t_metrics/R5: 75.65392354124748
MSRVTT_full_val/v2t_metrics/R10: 87.72635814889335
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 5.861167002012072
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 63.41804430806891
MSRVTT_full_test/t2v_metrics/R1: 12.608695652173912
MSRVTT_full_test/t2v_metrics/R5: 35.819397993311036
MSRVTT_full_test/t2v_metrics/R10: 50.468227424749166
MSRVTT_full_test/t2v_metrics/R50: 80.96989966555184
MSRVTT_full_test/t2v_metrics/MedR: 10.0
MSRVTT_full_test/t2v_metrics/MeanR: 40.40635451505017
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 28.353205479005165
MSRVTT_full_test/v2t_metrics/R1: 15.284280936454849
MSRVTT_full_test/v2t_metrics/R5: 42.14046822742475
MSRVTT_full_test/v2t_metrics/R10: 57.69230769230769
MSRVTT_full_test/v2t_metrics/R50: 86.82274247491638
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 28.798160535117056
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 33.36983760890988
mnt_best : 28.353205479005165
not_improved_count: 0
Train Epoch: 14 [1/250 128/32000 (0%)] Loss: 1.19777 (QuantReg: 13.42784) QuantErr: 13.42784 batch_time=38.24104
Train Epoch: 14 [12/250 1536/32000 (5%)] Loss: 1.87388 (QuantReg: 13.64455) QuantErr: 13.64455 batch_time=0.50265
Train Epoch: 14 [23/250 2944/32000 (9%)] Loss: 1.37142 (QuantReg: 13.50617) QuantErr: 13.50617 batch_time=0.54306
Train Epoch: 14 [34/250 4352/32000 (14%)] Loss: 1.32304 (QuantReg: 13.30138) QuantErr: 13.30138 batch_time=0.59001
Train Epoch: 14 [45/250 5760/32000 (18%)] Loss: 1.71968 (QuantReg: 13.34456) QuantErr: 13.34456 batch_time=0.57818
Train Epoch: 14 [56/250 7168/32000 (22%)] Loss: 1.36748 (QuantReg: 13.28987) QuantErr: 13.28987 batch_time=0.50415
Train Epoch: 14 [67/250 8576/32000 (27%)] Loss: 1.64177 (QuantReg: 13.41203) QuantErr: 13.41203 batch_time=0.57442
Train Epoch: 14 [78/250 9984/32000 (31%)] Loss: 1.63992 (QuantReg: 13.05394) QuantErr: 13.05394 batch_time=0.53274
Train Epoch: 14 [89/250 11392/32000 (36%)] Loss: 1.02601 (QuantReg: 13.15180) QuantErr: 13.15180 batch_time=0.52744
Train Epoch: 14 [100/250 12800/32000 (40%)] Loss: 1.33848 (QuantReg: 13.18790) QuantErr: 13.18790 batch_time=0.51651
Train Epoch: 14 [111/250 14208/32000 (44%)] Loss: 1.24992 (QuantReg: 13.47269) QuantErr: 13.47269 batch_time=0.53363
Train Epoch: 14 [122/250 15616/32000 (49%)] Loss: 1.62584 (QuantReg: 13.00789) QuantErr: 13.00789 batch_time=0.53799
Train Epoch: 14 [133/250 17024/32000 (53%)] Loss: 1.20426 (QuantReg: 13.64837) QuantErr: 13.64837 batch_time=0.52410
Train Epoch: 14 [144/250 18432/32000 (58%)] Loss: 1.04542 (QuantReg: 13.38462) QuantErr: 13.38462 batch_time=0.57412
Train Epoch: 14 [155/250 19840/32000 (62%)] Loss: 1.34454 (QuantReg: 13.49179) QuantErr: 13.49179 batch_time=0.55257
Train Epoch: 14 [166/250 21248/32000 (66%)] Loss: 1.13720 (QuantReg: 13.51625) QuantErr: 13.51625 batch_time=0.60019
Train Epoch: 14 [177/250 22656/32000 (71%)] Loss: 1.12763 (QuantReg: 13.27727) QuantErr: 13.27727 batch_time=0.49637
Train Epoch: 14 [188/250 24064/32000 (75%)] Loss: 1.66180 (QuantReg: 13.66372) QuantErr: 13.66372 batch_time=1.09746
Train Epoch: 14 [199/250 25472/32000 (80%)] Loss: 1.34524 (QuantReg: 13.41502) QuantErr: 13.41502 batch_time=0.57640
Train Epoch: 14 [210/250 26880/32000 (84%)] Loss: 1.56001 (QuantReg: 13.26339) QuantErr: 13.26339 batch_time=0.51265
Train Epoch: 14 [221/250 28288/32000 (88%)] Loss: 1.38884 (QuantReg: 13.65273) QuantErr: 13.65273 batch_time=0.51469
Train Epoch: 14 [232/250 29696/32000 (93%)] Loss: 1.30381 (QuantReg: 13.70065) QuantErr: 13.70065 batch_time=0.59070
Train Epoch: 14 [243/250 31104/32000 (97%)] Loss: 1.33164 (QuantReg: 13.42819) QuantErr: 13.42819 batch_time=0.55539
Train Epoch: 14 codebook_update_time=1.83464
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch14.pth ...
Done in 3.975s
removing stale ckpt [epoch 13] [took 0.00s]
epoch : 14
loss : 1.3727386577129363
quant_reg : 13.422417125701905
quant_err : 13.422417125701905
learning_rate : 2.566710416397524e-05
n_samples : 448000
n_steps : 3500
MSRVTT_full_val/t2v_metrics/R1: 31.58953722334004
MSRVTT_full_val/t2v_metrics/R5: 66.39839034205231
MSRVTT_full_val/t2v_metrics/R10: 79.67806841046277
MSRVTT_full_val/t2v_metrics/R50: 96.98189134808852
MSRVTT_full_val/t2v_metrics/MedR: 3.0
MSRVTT_full_val/t2v_metrics/MeanR: 8.752515090543259
MSRVTT_full_val/t2v_metrics/geometric_mean_R1-R5-R10: 55.08244430554476
MSRVTT_full_val/v2t_metrics/R1: 38.22937625754527
MSRVTT_full_val/v2t_metrics/R5: 74.64788732394366
MSRVTT_full_val/v2t_metrics/R10: 86.11670020120724
MSRVTT_full_val/v2t_metrics/R50: 98.59154929577464
MSRVTT_full_val/v2t_metrics/MedR: 2.0
MSRVTT_full_val/v2t_metrics/MeanR: 6.360160965794768
MSRVTT_full_val/v2t_metrics/geometric_mean_R1-R5-R10: 62.63744548799753
MSRVTT_full_test/t2v_metrics/R1: 12.040133779264215
MSRVTT_full_test/t2v_metrics/R5: 33.47826086956522
MSRVTT_full_test/t2v_metrics/R10: 46.48829431438127
MSRVTT_full_test/t2v_metrics/R50: 78.72909698996655
MSRVTT_full_test/t2v_metrics/MedR: 12.0
MSRVTT_full_test/t2v_metrics/MeanR: 48.53946488294314
MSRVTT_full_test/t2v_metrics/geometric_mean_R1-R5-R10: 26.56109271748175
MSRVTT_full_test/v2t_metrics/R1: 14.31438127090301
MSRVTT_full_test/v2t_metrics/R5: 40.635451505016725
MSRVTT_full_test/v2t_metrics/R10: 54.68227424749164
MSRVTT_full_test/v2t_metrics/R50: 85.08361204013377
MSRVTT_full_test/v2t_metrics/MedR: 8.0
MSRVTT_full_test/v2t_metrics/MeanR: 33.72742474916388
MSRVTT_full_test/v2t_metrics/geometric_mean_R1-R5-R10: 31.68410274812336
mnt_best : 28.353205479005165
not_improved_count: 1
Train Epoch: 15 [1/250 128/32000 (0%)] Loss: 1.64915 (QuantReg: 13.13608) QuantErr: 13.13608 batch_time=36.73975
Train Epoch: 15 [12/250 1536/32000 (5%)] Loss: 1.27670 (QuantReg: 13.24753) QuantErr: 13.24753 batch_time=0.49329
Train Epoch: 15 [23/250 2944/32000 (9%)] Loss: 1.27538 (QuantReg: 13.37956) QuantErr: 13.37956 batch_time=0.55722
Train Epoch: 15 [34/250 4352/32000 (14%)] Loss: 1.45542 (QuantReg: 13.24609) QuantErr: 13.24609 batch_time=0.57642
Train Epoch: 15 [45/250 5760/32000 (18%)] Loss: 1.41041 (QuantReg: 13.61384) QuantErr: 13.61384 batch_time=0.53401
Train Epoch: 15 [56/250 7168/32000 (22%)] Loss: 1.57314 (QuantReg: 13.12240) QuantErr: 13.12240 batch_time=0.50482
Train Epoch: 15 [67/250 8576/32000 (27%)] Loss: 1.62477 (QuantReg: 13.37211) QuantErr: 13.37211 batch_time=0.95512
Train Epoch: 15 [78/250 9984/32000 (31%)] Loss: 1.12540 (QuantReg: 13.50661) QuantErr: 13.50661 batch_time=0.55461
Train Epoch: 15 [89/250 11392/32000 (36%)] Loss: 1.59848 (QuantReg: 13.43176) QuantErr: 13.43176 batch_time=0.51208
Train Epoch: 15 [100/250 12800/32000 (40%)] Loss: 1.38144 (QuantReg: 13.51360) QuantErr: 13.51360 batch_time=0.55905
Train Epoch: 15 [111/250 14208/32000 (44%)] Loss: 1.56620 (QuantReg: 13.37575) QuantErr: 13.37575 batch_time=0.52595
Train Epoch: 15 [122/250 15616/32000 (49%)] Loss: 1.08852 (QuantReg: 13.20731) QuantErr: 13.20731 batch_time=0.57531
Train Epoch: 15 [133/250 17024/32000 (53%)] Loss: 1.35198 (QuantReg: 13.46926) QuantErr: 13.46926 batch_time=0.50452
Train Epoch: 15 [144/250 18432/32000 (58%)] Loss: 1.55159 (QuantReg: 13.49598) QuantErr: 13.49598 batch_time=0.51168
Train Epoch: 15 [155/250 19840/32000 (62%)] Loss: 1.38141 (QuantReg: 13.30168) QuantErr: 13.30168 batch_time=0.57106
Train Epoch: 15 [166/250 21248/32000 (66%)] Loss: 1.41373 (QuantReg: 13.64241) QuantErr: 13.64241 batch_time=0.54232
Train Epoch: 15 [177/250 22656/32000 (71%)] Loss: 1.62188 (QuantReg: 13.69404) QuantErr: 13.69404 batch_time=0.56074
Train Epoch: 15 [188/250 24064/32000 (75%)] Loss: 1.22014 (QuantReg: 13.54099) QuantErr: 13.54099 batch_time=0.50696
Train Epoch: 15 [199/250 25472/32000 (80%)] Loss: 1.26385 (QuantReg: 13.65221) QuantErr: 13.65221 batch_time=4.83021
Train Epoch: 15 [210/250 26880/32000 (84%)] Loss: 1.55293 (QuantReg: 13.23241) QuantErr: 13.23241 batch_time=0.54489
Train Epoch: 15 [221/250 28288/32000 (88%)] Loss: 1.55234 (QuantReg: 13.64293) QuantErr: 13.64293 batch_time=0.52317
Train Epoch: 15 [232/250 29696/32000 (93%)] Loss: 1.52833 (QuantReg: 13.53280) QuantErr: 13.53280 batch_time=0.54095
Train Epoch: 15 [243/250 31104/32000 (97%)] Loss: 1.42519 (QuantReg: 13.25275) QuantErr: 13.25275 batch_time=0.66216
Train Epoch: 15 codebook_update_time=1.81527
Saving checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch15.pth ...
Done in 4.348s
Updating 'best' checkpoint: /apdcephfs/share_47076/gimwang/HCQ/exps/HCQ_MSRVTT_full/checkpoint-epoch15.pth ...
Done in 8.595s
removing stale ckpt [epoch 14] [took 0.00s]
epoch : 15
loss : 1.31741427898407
quant_reg : 13.46969380569458
quant_err : 13.46969380569458
learning_rate : 2.4383748955776477e-05