Accuracy cudacodec test fails always with -DWITH_NVCUVID=ON

[asenyaev]

Accuracy:cudacodec fails always if -DWITH_NVCUVID=ON flag is set for a build, all tests are failed there.

List of failed tests:

[  PASSED  ] 0 tests.
[  FAILED  ] 48 tests, listed below:
[  FAILED  ] CUDA_Codec/CheckSet.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4")
[  FAILED  ] CUDA_Codec/CheckExtraData.Reader/0, where GetParam() = (Quadro P2000, ("highgui/video/big_buck_bunny.mp4", 45))
[  FAILED  ] CUDA_Codec/CheckExtraData.Reader/1, where GetParam() = (Quadro P2000, ("highgui/video/big_buck_bunny.mov", 45))
[  FAILED  ] CUDA_Codec/CheckExtraData.Reader/2, where GetParam() = (Quadro P2000, ("highgui/video/big_buck_bunny.mjpg.avi", 0))
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/1, where GetParam() = (Quadro P2000, "cv/video/768x576.avi")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/2, where GetParam() = (Quadro P2000, "cv/video/1920x1080.avi")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/3, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.avi")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/4, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h264")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/5, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h265")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/6, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mpg")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/7, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libvpx-vp9.mp4")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm")
[  FAILED  ] CUDA_Codec/Scaling.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [1 x 1], [1 x 1 from (0, 0)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/1, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [1 x 1], [1 x 1 from (0, 0)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/2, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [1 x 1], [0.5 x 0.5 from (0.25, 0.25)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/3, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [1 x 1], [0.5 x 0.5 from (0.25, 0.25)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/4, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [0.8 x 0.9], [1 x 1 from (0, 0)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/5, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [0.8 x 0.9], [1 x 1 from (0, 0)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/6, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [0.8 x 0.9], [0.5 x 0.5 from (0.25, 0.25)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/7, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [0.8 x 0.9], [0.5 x 0.5 from (0.25, 0.25)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [2.3 x 1.8], [1 x 1 from (0, 0)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/9, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [2.3 x 1.8], [1 x 1 from (0, 0)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/10, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [2.3 x 1.8], [0.5 x 0.5 from (0.25, 0.25)], [1 x 1 from (0, 0)])
[  FAILED  ] CUDA_Codec/Scaling.Reader/11, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", [2.3 x 1.8], [0.5 x 0.5 from (0.25, 0.25)], [0.6 x 0.7 from (0.2, 0.3)])
[  FAILED  ] CUDA_Codec/Video.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4")
[  FAILED  ] CUDA_Codec/Video.Reader/1, where GetParam() = (Quadro P2000, "cv/video/768x576.avi")
[  FAILED  ] CUDA_Codec/Video.Reader/2, where GetParam() = (Quadro P2000, "cv/video/1920x1080.avi")
[  FAILED  ] CUDA_Codec/Video.Reader/3, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.avi")
[  FAILED  ] CUDA_Codec/Video.Reader/4, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h264")
[  FAILED  ] CUDA_Codec/Video.Reader/5, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h265")
[  FAILED  ] CUDA_Codec/Video.Reader/6, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mpg")
[  FAILED  ] CUDA_Codec/Video.Reader/7, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libvpx-vp9.mp4")
[  FAILED  ] CUDA_Codec/Video.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[  FAILED  ] CUDA_Codec/Video.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm")
[  FAILED  ] CUDA_Codec/VideoReadRaw.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h264")
[  FAILED  ] CUDA_Codec/VideoReadRaw.Reader/1, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.h265")
[  FAILED  ] CUDA_Codec/CheckParams.Reader/0, where GetParam() = Quadro P2000
[  FAILED  ] CUDA_Codec/CheckDecodeSurfaces.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4")
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/0, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", true, true, true)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/1, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", true, true, false)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/2, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", true, false, true)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/3, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", true, false, false)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/4, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", false, true, true)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/5, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", false, true, false)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/6, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", false, false, true)
[  FAILED  ] CUDA_Codec/CheckInitParams.Reader/7, where GetParam() = (Quadro P2000, "highgui/video/big_buck_bunny.mp4", false, false, false)

An example of a failed test:

[ RUN      ] CUDA_Codec/Video.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm")
unknown file: Failure
C++ exception with description "OpenCV(4.6.0-dev) /home/ci/opencv_contrib/modules/cudacodec/src/video_parser.cpp:63: error: (-217:Gpu API call) Unknown error code [Code = 1892716896] in function 'VideoParser'
" thrown in the test body.
[  FAILED  ] CUDA_Codec/Video.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm") (85 ms)

System information (version)

• OpenCV version: 4.6.0
• Operating System / Platform: Ubuntu 20.04
• Compiler & compiler version: GCC 9.4.0
• GPU: Quadro P2000

[cudawarped]

Is it possible that you have copied the stub (libnvcuvid.so) into your cuda install and are incorrectly linking against that as discussed in #3359.

Does

ldd bin/opencv_test_cudacodec | grep libnvcuvid

give you something similar to

libnvcuvid.so.1 => /usr/local/cuda-11.3/lib64/libnvcuvid.so.1 (0x00007f88a8b72000)

instead of

libnvcuvid.so.1 => /lib/x86_64-linux-gnu/libnvcuvid.so.1 (0x00007f2fcb831000)

If so, can you remove the stub library (libnvcuvid.so) from your cuda installation and try building with either

-DCUDA_nvcuvid_LIBRARY=Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so

or

-DCMAKE_PREFIX_PATH=Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/

[asenyaev]

@cudawarped, Yes, I had similar to

libnvcuvid.so.1 => /usr/local/cuda-11.3/lib64/libnvcuvid.so.1 (0x00007f88a8b72000)

and after this removed libnvcuvid.so from cuda and tried to rebuild and run tests with -DCUDA_nvcuvid_LIBRARY=Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so, but anyway all tests are failed.

[cudawarped]

@asenyaev That's weird. The unkown error code thrown on the first call to the decoding sdk seems to be eerily similar to the issue described in #3359, must be a weird coincidence. I still supect a driver issue as the error code looks like garbage.

[asenyaev]

@cudawarped I use the latest nvidia driver 515.65.01 version and 11.7 CUDA.

[cudawarped]

@asenyaev If

ldd bin/opencv_test_cudacodec | grep libnvcuvid

is pointing to the correct place

libnvcuvid.so.1 => /lib/x86_64-linux-gnu/libnvcuvid.so.1 (0x00007f2fcb831000)

and the error code is still garbage (not 1 to 999) and coming from the same location

C++ exception with description "OpenCV(4.6.0-dev) /home/ci/opencv_contrib/modules/cudacodec/src/video_parser.cpp:63: error: (-217:Gpu API call) Unknown error code [Code = 1892716896] in function 'VideoParser'

I'm out of idea's.

[asenyaev]

Should it pointing certainly to

libnvcuvid.so.1 => /lib/x86_64-linux-gnu/libnvcuvid.so.1 (0x00007f2fcb831000)

Or it can be like this?

libnvcuvid.so.1 => /home/ci/Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so.1 (0x00007f1492718000)

[cudawarped]

Or it can be like this?

libnvcuvid.so.1 => /home/ci/Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so.1 (0x00007f1492718000)

No it shouldn't be pointing to that, that's just the stub, OpenCV can use that to link against but, it needs to call into the actual library provided by the driver. If you don't pass the stub

-DCUDA_nvcuvid_LIBRARY=Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so

does cmake pick up the driver lib here

/lib/x86_64-linux-gnu/libnvcuvid.so.1

or in a similar location?

Did you remove both stubs libnvcuvid.so and libnvcuvid.so.1 from your cuda install?

[asenyaev]

Cmake picks up from here /home/ci/Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so if to provide it using a flag -DCUDA_nvcuvid_LIBRARY=/home/ci/Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so.

Sure, removed both of them.

[cudawarped]

The stubs should allow you to link against them then use the driver lib, that's there purpose, maybe the driver lib can't be found or doesn't exist for some reason.

I would try just passing in the location of the driver lib e.g.

-DCUDA_nvcuvid_LIBRARY=/lib/x86_64-linux-gnu/libnvcuvid.so.1

[asenyaev]

No, it also is not a solution :(

Even if a linkage like this

libnvcuvid.so.1 => /lib/x86_64-linux-gnu/libnvcuvid.so.1 (0x00007f9de5fe3000)

all tests fail

[cudawarped]

I'm not sure I can be any help, my linux skills are limited at best. @asmorkalov has it working on his local machine so maybe he can shed some light on why its not working.

To summarize I can recreate the issue by copying the stubs into the /usr/local/cuda/lib64/ directory and as far as I am aware because this entry is ahead of the driver entry in /etc/ld.so.conf.d/ it picks it up with ldd as

libnvcuvid.so.1 => /usr/local/cuda/lib64/libnvcuvid.so.1 (0x00007f5224c49000)

and then fails because its a stub, so the first call into fails. Then if I remove the lib ldd gives me the correct path (this is on wsl so hence the different path)

libnvcuvid.so.1 => /usr/lib/wsl/lib/libnvcuvid.so.1 (0x00007ff602a59000)

and everything works.

I am not sure why you are seeing

libnvcuvid.so.1 => /home/ci/Video_Codec_SDK_11.1.5/Lib/linux/stubs/x86_64/libnvcuvid.so.1 (0x00007f1492718000)

if this isn't on your load library path, unless /lib/x86_64-linux-gnu/libnvcuvid.so.1 isn't or there is some other config error.

Additionaly if I don't pass the stub library then cmake picks up the driver version automatically and everything works

cat build/opencv/CMakeVars.txt | grep CUDA_nvcuvid_LIBRARY
CUDA_nvcuvid_LIBRARY=/usr/lib/wsl/lib/libnvcuvid.so

which it doesn't on your system.

[asmorkalov]

@cudawarped thanks for assistance. We are working with @asenyaev together trying to debug the issue on CI side.

[asenyaev]

@cudawarped, the only one difference can be on my side, I run it inside nvidia-docker, maybe some issues inside a container. I will try one more iteration today and provide a feedback.

[asenyaev]

So, I tried to do it on a host (w/o container usage), but the same results happened.

My steps:

1. Remove libnvcuvid.so and libnvcuvid.so.1 libraries from cuda (/usr/local/cuda-11.7/lib64)
2. Copy libnvcuvid.so from Video_Codec_SDK_11.1.5 to a driver folder /lib/x86_64-linux-gnu
3. Run ldconfig to reconfigure libraries
4. Run cmake

cmake -G Ninja -DBUILD_DOCS=ON -DPYTHON_DEFAULT_EXECUTABLE=/usr/bin/python3 -DBUILD_EXAMPLES=ON -DOPENCV_ENABLE_NONFREE=ON  -DWITH_CUDA=ON -DWITH_VA=OFF -DCUDA_ARCH_BIN=6.1 -DWITH_NVCUVID=ON -DOPENCV_EXTRA_MODULES_PATH=/home/ci/opencv_contrib/modules ../opencv

5. Build OpenCV

ninja -j $(nproc)

6. Check a linkage for bin/opencv_test_cudacodec

ldd bin/opencv_test_cudacodec | grep libnvcuvid

Output: libnvcuvid.so.1 => /lib/x86_64-linux-gnu/libnvcuvid.so.1 (0x00007f4b9866b000)

7. Set env variable for tests:

export OPENCV_TEST_DATA_PATH=/home/ci/opencv_extra/testdata

8. Run opencv_test_cudacodec test

./bin/opencv_test_cudacodec

Unfortunately, all tests are failed. Failure messages are the same as they are in the body of this issue.

[cudawarped]

2. Copy libnvcuvid.so from Video_Codec_SDK_11.1.5 to a driver folder /lib/x86_64-linux-gnu

Don't do this. The shared lib should already exist in your driver folder. The stub is there so you can compile on a system which doen't have an nvidia gpu and therefore won't have any cuda shared libs in /lib/x86_64-linux-gnu.

What is your location for libcuda.so.1 (which libcuda.so.1), is it /lib/x86_64-linux-gnu, if so there should be an existing copy of
libnvcuvid.so.1 in there?

[asenyaev]

@cudawarped, thank you for your support!

Finally, I made it to work. I re-installed all nvidia components and just did not install NVIDIA Video codec SDK over CUDA, but copied headers from Video_Codec_SDK_11.1.5/Interfaces folder to /usr/local/cuda/include.

Firstly, NVCUVID couldn't be detected and I defined certain libraries for it and CUDA using following flags -DCUDA_CUDA_LIBRARY=/lib/x86_64-linux-gnu/libcuda.so.1 -DCUDA_nvcuvid_LIBRARY=/lib/x86_64-linux-gnu/libnvcuvid.so.1. So, in this case NVCUVID can be detected and OpenCV builds successfully.

Tests started to run successfully also, but some of them failed anyway:

[  PASSED  ] 44 tests.
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[  FAILED  ] CUDA_Codec/CheckKeyFrame.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm")
[  FAILED  ] CUDA_Codec/Video.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
[  FAILED  ] CUDA_Codec/Video.Reader/9, where GetParam() = (Quadro P2000, "cv/tracking/faceocc2/data/faceocc2.webm")

It happened in case of hardware decoding, example:

[ RUN      ] CUDA_Codec/Video.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4")
unknown file: Failure
C++ exception with description "OpenCV(4.6.0-dev) /home/ci/opencv_contrib/modules/cudacodec/src/video_reader.cpp:184: error: (-2:Unspecified error) Parsing/Decoding video source failed, check GPU memory is available and GPU supports hardware decoding. in function 'internalGrab'
" thrown in the test body.
[  FAILED  ] CUDA_Codec/Video.Reader/8, where GetParam() = (Quadro P2000, "highgui/video/sample_322x242_15frames.yuv420p.libaom-av1.mp4") (69 ms)

[asmorkalov]

I have test failure for av1 codec too. Most probably it depends on FFmpeg version.

[cudawarped]

AV1 is only supported on 5th gen NVDEC, P2000 is 3rd gen and not supported but this should get picked up here. My guess is that check needs to be made earlier using FFmpeg's provided codec (if its supported by FFmpeg) as the parser is failing to create the decoder. Looks like a bug in so much as it should throw a more informative exception, I'll test on older hardware and fix when I have chance.

Strangely VP8 is also not supported on your P2000.

[asenyaev]

Some updates for this issue how to build OpenCV with NVCUVID support inside nvidia-docker.

• To make NVCUVID work need to install Nvidia driver, CUDA, copy headers from Video_Codec_SDK_11.1.5/Interfaces to /usr/local/cuda/include
• To use it inside nvidia-docker container need to forward certain libs (it should be by a driver, apt instal libnvidia-encode-515 inside container won't help) and cuda itself inside:
  • -v /lib/x86_64-linux-gnu/libnvidia-encode.so.1:/lib/x86_64-linux-gnu/libnvidia-encode.so.1
  • -v /lib/x86_64-linux-gnu/libnvcuvid.so.1:/lib/x86_64-linux-gnu/libnvcuvid.so.1
  • -v /usr/local/cuda:/usr/local/cuda
• Also add cmake flags defining these libs -DCUDA_CUDA_LIBRARY=/lib/x86_64-linux-gnu/libcuda.so.1 -DCUDA_nvcuvid_LIBRARY=/lib/x86_64-linux-gnu/libnvcuvid.so.1

I'm closing this issue, because the error has disappeared and new failures do not relate to the main issue. @cudawarped thank you!