FileNotFoundError for AVA dataset

[LeeJunHyun]

Thanks for commitments of you all.

I got a FileNotFoundError when processing AVA dataset for LFB inference.

python tools/test.py configs/detection/lfb/lfb_slowonly_r50_ava_infer.py \
    checkpoints/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb_20201217-40061d5f.pth --eval mAP

I followed your guideline to prepare AVA dataset and it worked for the other frames but not for data/ava/rawframes/-IELREHX_js/img_22536.jpg.

I think there are some missing frames but data_loader is not holding the list of valid frames.

For other datasets, there are *_val_list.txt and *_train_list.txt but not for AVA dataset.

Is there any step that I missed?

I attached full error message below.

  File "tools/test.py", line 356, in <module>
    main()
  File "tools/test.py", line 341, in main
    outputs = inference_pytorch(args, cfg, distributed, data_loader)
  File "tools/test.py", line 152, in inference_pytorch
    outputs = single_gpu_test(model, data_loader)
  File "/data/private/mmaction2/mmaction/apis/test.py", line 30, in single_gpu_test
    for data in data_loader:
  File "/data/private/mmaction2/mmaction/datasets/data_loader.py", line 345, in __next__
    data = self._next_data()
  File "/data/private/mmaction2/mmaction/datasets/data_loader.py", line 857, in _next_data
    return self._process_data(data)
  File "/data/private/mmaction2/mmaction/datasets/data_loader.py", line 884, in _process_data
    data.reraise()
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/envs/open-mmlab/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/private/mmaction2/mmaction/datasets/base.py", line 278, in __getitem__
    return self.prepare_test_frames(idx)
  File "/data/private/mmaction2/mmaction/datasets/ava_dataset.py", line 326, in prepare_test_frames
    return self.pipeline(results)
  File "/data/private/mmaction2/mmaction/datasets/pipelines/compose.py", line 41, in __call__
    data = t(data)
  File "/data/private/mmaction2/mmaction/datasets/pipelines/loading.py", line 1135, in __call__
    img_bytes = self.file_client.get(filepath)
  File "/data/private/mmcv/mmcv/fileio/file_client.py", line 306, in get
    return self.client.get(filepath)
  File "/data/private/mmcv/mmcv/fileio/file_client.py", line 184, in get
    with open(filepath, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/data/private/mmaction2/data/ava/rawframes/-IELREHX_js/img_22536.jpg'

[irvingzhang0512]

We don't need to generate filelist for ava because all ava samples share the exact same number of frames and format. There should be 27030 frames for each sample.

Which script did you use to extract RGB frames?

You can recut -IELREHX_js.mp4 by ffmpeg -ss 900 -t 901 -i "-IELREHX_js.mp4" -r 30 -strict experimental "-IELREHX_js_15min.mp4".

[LeeJunHyun]

Thanks for fast reply.

I ran bash extract_rgb_frames_opencv.sh and the extraction was interrupted if a frame is missing in a video. (build_rawframes.py#L59)

So some video takes smaller number for frames than 27030.
For example,
[File path ; the number of frames]
../../data/ava/videos_15min/Ie35yEssHko.mkv 21603
../../data/ava/videos_15min/kMy-6RtoOVU.mkv 21603
../../data/ava/videos_15min/P90hF2S1JzA.mkv 27003
../../data/ava/videos_15min/phrYEKv0rmw.mkv 21603
../../data/ava/videos_15min/QaIMUi-elFo.mkv 21603
../../data/ava/videos_15min/QotkBTEePI8.mkv 21602
../../data/ava/videos_15min/QTmwhrVal1g.mkv 21603
../../data/ava/videos_15min/Rm518TUhbRY.mkv 27003
../../data/ava/videos_15min/rXFlJbXyZyc.mkv 21603
../../data/ava/videos_15min/UsLnxI_zGpY.mkv 27003
../../data/ava/videos_15min/ZxQn8HVmXsY.mkv 21603

And also, vr = mmcv.VideoReader(full_path) can't read any frame from data/ava/videos_15min/Gvp-cj3bmIY.webm.

Thanks again for your kind supports and commitments :)

[irvingzhang0512]

Personally, I haven't tried to extract frames by opencv.
cc @kennymckormick

Maybe for thoses videos with < 27030 frames, you can use ffmpeg to recut videos and extract frames.

[irvingzhang0512]

Maybe #807 could fix this issue.

[irvingzhang0512]

I could reproduce this bug even after #807

../../data/ava/videos_15min/Ie35yEssHko.mkv 21603
../../data/ava/videos_15min/kMy-6RtoOVU.mkv 21603
../../data/ava/videos_15min/P90hF2S1JzA.mkv 27003
../../data/ava/videos_15min/phrYEKv0rmw.mkv 21603
../../data/ava/videos_15min/QaIMUi-elFo.mkv 21603
../../data/ava/videos_15min/QotkBTEePI8.mkv 21602
../../data/ava/videos_15min/QTmwhrVal1g.mkv 21603
../../data/ava/videos_15min/Rm518TUhbRY.mkv 27003
../../data/ava/videos_15min/rXFlJbXyZyc.mkv 21603
../../data/ava/videos_15min/UsLnxI_zGpY.mkv 27003

For ava, each video must have a fixed number(27030) of frames.
It seems OpenCV cannot do that opencv/opencv#9053 (comment)

It seems to me that the problem is that VideoCapture ignores all duplicate frames, while FFmpeg will return duplicates without issues

I would recommend that AVA dataset should not extract frames by OpenCV and we should remove related docs in tutorials.

For other dataset, there is no requirement on the number of frames for each video, so extracting frames by OpenCV is ok.

Any comments? @innerlee @kennymckormick @dreamerlin

[LeeJunHyun]

Okay, I'm trying to extract frames by using ffmpeg instead of OpenCV.

Thank you all!

[innerlee]

@irvingzhang0512 Good comment

[irvingzhang0512]

I could reproduce this bug even after #807

../../data/ava/videos_15min/Ie35yEssHko.mkv 21603
../../data/ava/videos_15min/kMy-6RtoOVU.mkv 21603
../../data/ava/videos_15min/P90hF2S1JzA.mkv 27003
../../data/ava/videos_15min/phrYEKv0rmw.mkv 21603
../../data/ava/videos_15min/QaIMUi-elFo.mkv 21603
../../data/ava/videos_15min/QotkBTEePI8.mkv 21602
../../data/ava/videos_15min/QTmwhrVal1g.mkv 21603
../../data/ava/videos_15min/Rm518TUhbRY.mkv 27003
../../data/ava/videos_15min/rXFlJbXyZyc.mkv 21603
../../data/ava/videos_15min/UsLnxI_zGpY.mkv 27003

For ava, each video must have a fixed number(27030) of frames.
It seems OpenCV cannot do that opencv/opencv#9053 (comment)

It seems to me that the problem is that VideoCapture ignores all duplicate frames, while FFmpeg will return duplicates without issues

I would recommend that AVA dataset should not extract frames by OpenCV and we should remove related docs in tutorials.

For other dataset, there is no requirement on the number of frames for each video, so extracting frames by OpenCV is ok.

Any comments? @innerlee @kennymckormick @dreamerlin

@innerlee @dreamerlin
should reopen this since the issue is not fixed. We cannot extract ava frames by opencv.