videoio: add support for obsensor (Orbbec RGB-D Camera ) #22196

hzcyf · 2022-07-05T12:42:09Z

VideoCapture for OBSensor

OBSensor is Orbbec's new generation RGB-D camera brand, based on UVC protocol. The purpose is to let OpenCV read RGB and depth data from Orbbec RGB-D Camera via UVC protocol, rather than relying on the 3rdparty library (like OpenNI). This will greatly facilitate users to directly read, use and process the depth camera in OpenCV.

Supported Orbbec Device

Orbbec Astra+: detail
- Depth Stream (640x480x30 fps CV_16UC1)
- RGB Stream (640x480x30fps BGR)
Orbbec Femto: detail
- Depth Stream (640x480x30 fps CV_16UC1)
- RGB Stream (640x480x30fps BGR)
Orbbec future device:
- ...

Supported OS: Windows and Linux. (MacOX will be supported in near future.)
Supported HW: x86, x64, ARM.

Since the new Orbbec devices rely on UVC, while old devices still rely on OpenNI. This will cause OBSensor doesn't support some old Orbbec cameras, and the following is a list of specific unsupported devices:

Orbbec Astra
Orbbec Astra Pro

For these unspported devices, we still need the support of OpenNI, please refer to the detailed tutorial of Using Orbbec Astra 3D cameras in OpenCV.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
The feature is well documented and sample code can be built with the project CMake
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch

zihaomu

Thank you for your contribution!

modules/videoio/src/cap_obsensor/obsensor_stream_channel_interface.hpp

modules/videoio/include/opencv2/videoio.hpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.cpp

samples/cpp/videocapture_obsensor.cpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_v4l2.cpp

modules/videoio/src/cap_obsensor/obsensor_uvc_stream_channel.cpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_interface.hpp

zihaomu · 2022-07-15T07:13:54Z

Functional test report

Testing Code: samples\cpp\videocapture_obsensor.cpp
Testing Camera: Orbbec Femto & Orbbec Astra+

Functional Test summary

Testing platform	Test Results
Raspberry Pi 3B, ARM32	Pass
Raspberry Pi 4B, ARM64	Pass
Khadas VIM3, ARM64	Pass
Windows10, X64	Pass
Ubuntu 20.04, X64	Pass

Multi-Camera Test

One Orbbec Camera -> Pass
One Orbbec Camera and Multiple normal RGB cameras -> Pass
Multiple Orbbec Cameras -> Not support in this PR.

some results display

Raspberry Pi 3B, ARM32, Device:Orbbec Astra+
Raspberry Pi 4B, ARM64, Device:Orbbec Femto
Khadas VIM3, ARM64, Device:Orbbec Astra+
Windows10, X86, Device:Orbbec Femto
Ubuntu 20.04, X64, Device:Orbbec Astra+

modules/videoio/src/cap_obsensor/obsensor_stream_channel_interface.hpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.cpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp

modules/videoio/src/cap_obsensor/obsensor_stream_channel_v4l2.cpp

modules/videoio/src/cap_obsensor/obsensor_uvc_stream_channel.hpp

modules/videoio/src/cap_obsensor/obsensor_uvc_stream_channel.cpp

vpisarev · 2022-07-16T12:47:18Z

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp

+    virtual void stop() override;
+
+private:
+    virtual bool setXu(uint8_t ctrl, const uint8_t* data, uint32_t len) override;


can we use int ctrl instead of uint8_t ctrl?

According to the UVC protocol ， the range of ctrl is limited on: 0~255

vpisarev · 2022-07-16T12:47:35Z

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp

+
+    FrameCallback frameCallback_;
+    StreamProfile currentProfile_;
+    int8_t currentStreamIndex_;


int currentStreamIndex_;?

currentStreamIndex_ is less than 5, and currentStreamIndex_ < 0 mean error!

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp

modules/videoio/src/cap_obsensor_capture.cpp

vpisarev · 2022-07-16T12:55:11Z

samples/cpp/videocapture_obsensor.cpp

+    Mat adjDepthMap;
+    while (true)
+    {
+        // obsensorCapture >> depthMap;


Can we remove this piece of code? I see a very similar chunk of code below

I will remove this and add some comments

vpisarev · 2022-07-16T12:56:52Z

@hzcyf, thank you for the contribution! Me and @zihaomu have put some minor comments, but overall the patch looks very good. Please, address the comments and then the patch can be merged

zihaomu

LGTM! 👍

vpisarev · 2022-07-20T12:40:32Z

Looks good to me! @alalek, I think, it can be merged

modules/videoio/include/opencv2/videoio.hpp

modules/videoio/cmake/detect_obsensor.cmake

modules/videoio/include/opencv2/videoio.hpp

alalek · 2022-07-20T13:06:08Z

modules/videoio/include/opencv2/videoio.hpp

+//! OBSENSOR data given from iamge generator
+enum { 
+    CAP_OBSENSOR_DEPTH_MAP = 0, //!< Depth values in mm (CV_16UC1)
+    CAP_OBSENSOR_BGR_IMAGE = 1, //!< Data given from BGR stream generator


BGR image is preserved to default stream 0 (of any VideoCapture backend)

According to the existing code, both OpenNI and Intel Realsense use depth as the default stream。For RGB-D cameras, depth as the default stream is more in line with user needs as they may only need the depth data stream.

@vpisarev Vadim, could you take a look on this and put a comment?

@alalek, @hzcyf, I can confirm that for OpenNI and Intel RealSense (PerC) cameras depth map is the default stream. From this point of view the new Orbbec API follows this tradition. I think, it should be retained as-is.

Problem is that this support is enabled by default (for many build configurations).
Existed user code may not expect non 8UC3 data and crash (16SC1/16UC1 or 32FC1 are used for depth).
Exception is preferable (and reliable for this case) than silent replacing of 8UC3 result.

If we go in this way, then we must update many sample code / documentation snippets / etc. But this would not resolve problem with existed users code.

BTW, another approach may go through another API, like VideoDepthCapture. Including OpenNI/Realsense "buggy" cases.

@alalek, there is already such situation with OpenNI and PerC and so far it did not cause any major problems. Let me merge this PR

modules/videoio/include/opencv2/videoio.hpp

samples/cpp/videocapture_obsensor.cpp

alalek · 2022-07-20T13:11:45Z

samples/cpp/videocapture_obsensor.cpp

+    while (true)
+    {
+        // Grab depth map like this:
+        // obsensorCapture >> depthMap;


"default" grab should be preserved for BGR frames (not depth or whatever)

for depth cameras making the "depth" default is fine, I think.

samples/cpp/videocapture_obsensor.cpp

asenyaev · 2022-07-29T09:07:20Z

After merging this PR there is a linking issue when building WinPack:

...
     Creating library C:/GHA-OCV-3/_work/ci-gha-workflow/ci-gha-workflow/build/lib/Release/opencv_world460.lib and object C:/GHA-OCV-3/_work/ci-gha-workflow/ci-gha-workflow/build/lib/Release/opencv_world460.exp
obsensor_stream_channel_msmf.obj : error LNK2019: unresolved external symbol QISearch referenced in function "public: virtual long __cdecl cv::obsensor::MSMFStreamChannel::QueryInterface(struct _GUID const &,void * *)" (?QueryInterface@MSMFStreamChannel@obsensor@cv@@UEAAJAEBU_GUID@@PEAPEAX@Z) [C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\modules\world\opencv_world.vcxproj]
C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\bin\Release\opencv_world460.dll : fatal error LNK1120: 1 unresolved externals [C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\modules\world\opencv_world.vcxproj]

Steps to reproduce:

Clone OpenCV 4.x branch:

git clone --branch 4.x git@github.com:opencv/opencv.git

Run CMake:

cmake -B build -DBUILD_SHARED_LIBS=ON -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DINSTALL_PDB=ON -DINSTALL_PDB_COMPONENT_EXCLUDE_FROM_ALL=OFF -DCPU_BASELINE=SSE3 -DOPENCV_INSTALL_DATA_DIR_RELATIVE=../../../../sources -DWITH_LAPACK=OFF -DVIDEOIO_PLUGIN_LIST=all -DINSTALL_CREATE_DISTRIB=ON -DBUILD_DOCS=OFF -DOPENCV_GENERATE_SETUPVARS=ON -DWITH_OPENCL=ON -DBUILD_opencv_python3=OFF -DBUILD_opencv_python2=OFF -DWITH_TBB=OFF -DWITH_CUDA=OFF -DBUILD_opencv_java=OFF opencv

Compile OpenCV:

cmake --build build --config release --target install

hzcyf · 2022-07-30T09:33:33Z

After merging this PR there is a linking issue when building WinPack:

...
     Creating library C:/GHA-OCV-3/_work/ci-gha-workflow/ci-gha-workflow/build/lib/Release/opencv_world460.lib and object C:/GHA-OCV-3/_work/ci-gha-workflow/ci-gha-workflow/build/lib/Release/opencv_world460.exp
obsensor_stream_channel_msmf.obj : error LNK2019: unresolved external symbol QISearch referenced in function "public: virtual long __cdecl cv::obsensor::MSMFStreamChannel::QueryInterface(struct _GUID const &,void * *)" (?QueryInterface@MSMFStreamChannel@obsensor@cv@@UEAAJAEBU_GUID@@PEAPEAX@Z) [C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\modules\world\opencv_world.vcxproj]
C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\bin\Release\opencv_world460.dll : fatal error LNK1120: 1 unresolved externals [C:\GHA-OCV-3\_work\ci-gha-workflow\ci-gha-workflow\build\modules\world\opencv_world.vcxproj]

Steps to reproduce:

Clone OpenCV 4.x branch:

git clone --branch 4.x git@github.com:opencv/opencv.git

Run CMake:

cmake -B build -DBUILD_SHARED_LIBS=ON -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF -DINSTALL_PDB=ON -DINSTALL_PDB_COMPONENT_EXCLUDE_FROM_ALL=OFF -DCPU_BASELINE=SSE3 -DOPENCV_INSTALL_DATA_DIR_RELATIVE=../../../../sources -DWITH_LAPACK=OFF -DVIDEOIO_PLUGIN_LIST=all -DINSTALL_CREATE_DISTRIB=ON -DBUILD_DOCS=OFF -DOPENCV_GENERATE_SETUPVARS=ON -DWITH_OPENCL=ON -DBUILD_opencv_python3=OFF -DBUILD_opencv_python2=OFF -DWITH_TBB=OFF -DWITH_CUDA=OFF -DBUILD_opencv_java=OFF opencv

Compile OpenCV:

cmake --build build --config release --target install

Currently, you could avoid this problem by disable cmake option BUILD_OBSENSOR . And I will fix this soon!

alalek · 2022-08-21T14:46:14Z

vpisarev merged commit fc3e393 into opencv:4.x 26 days ago

Rage merge should not happen at all.
Especially without passed CI checks.

modules/videoio/include/opencv2/videoio.hpp:662: trailing whitespace.
+enum VideoCaptureOBSensorDataType{ 
modules/videoio/include/opencv2/videoio.hpp:669: trailing whitespace.
+enum VideoCaptureOBSensorGenerators{ 
modules/videoio/include/opencv2/videoio.hpp:672: trailing whitespace.
+    CAP_OBSENSOR_IR_GENERATOR    = 1 << 27,  
modules/videoio/include/opencv2/videoio.hpp:676: trailing whitespace.
+//!OBSENSOR properties 
modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp:45: trailing whitespace.
+#include <ksmedia.h> 
modules/videoio/src/cap_obsensor/obsensor_uvc_stream_channel.cpp:64: trailing whitespace.
+        uvcDevName[i] = (char)tolower(uvcDevName[i]); 
samples/cpp/videocapture_obsensor.cpp:20: trailing whitespace.
+

* videoio: add support for obsensor (Orbbec RGB-D Camera ) * obsensor: code format issues fixed and some code optimized * obsensor: fix typo and format issues * obsensor: fix crosses initialization error

hzcyf changed the title ~~videoio: add supported for obsensor (Orbbec RGB-D Camera )~~ videoio: add support for obsensor (Orbbec RGB-D Camera ) Jul 5, 2022

zihaomu requested review from zihaomu, vpisarev and alalek July 6, 2022 06:51

zihaomu reviewed Jul 7, 2022

View reviewed changes

opencv deleted a comment from hzcyf Jul 15, 2022

hzcyf force-pushed the 4.x branch from 40f5973 to 54b184a Compare July 15, 2022 07:34

videoio: add support for obsensor (Orbbec RGB-D Camera )

03adc07

hzcyf force-pushed the 4.x branch from 54b184a to 03adc07 Compare July 15, 2022 08:14

zihaomu reviewed Jul 15, 2022

View reviewed changes

vpisarev reviewed Jul 16, 2022

View reviewed changes

modules/videoio/src/cap_obsensor/obsensor_stream_channel_msmf.hpp Outdated Show resolved Hide resolved

vpisarev reviewed Jul 16, 2022

View reviewed changes

modules/videoio/src/cap_obsensor_capture.cpp Outdated Show resolved Hide resolved

vpisarev reviewed Jul 16, 2022

View reviewed changes

obsensor: code format issues fixed and some code optimized

f63e5d5

zihaomu approved these changes Jul 19, 2022

View reviewed changes

vpisarev approved these changes Jul 20, 2022

View reviewed changes

alalek reviewed Jul 20, 2022

View reviewed changes

hzcyf added 2 commits July 22, 2022 10:24

obsensor: fix typo and format issues

6d0ba21

obsensor: fix crosses initialization error

d536350

vpisarev merged commit fc3e393 into opencv:4.x Jul 26, 2022

alalek mentioned this pull request Jul 26, 2022

(4.x) Merge 3.4 #22305

Closed

asenyaev mentioned this pull request Jul 29, 2022

Debug build fail if enable the option "BUILD_opencv_world" #22312

Closed

4 tasks

asenyaev mentioned this pull request Aug 2, 2022

Scheduled rolling workflows opencv/opencv-python#699

Merged

asenyaev mentioned this pull request Aug 3, 2022

Fix in obsensor for VS 14 #22339

Merged

6 tasks

berak mentioned this pull request Aug 14, 2022

OpenCV compilation problem: "fatal error: vidcap.h: No such file or directory" #22373

Closed

alalek mentioned this pull request Aug 14, 2022

videoio: Include missing condition_variable header #22377

Merged

6 tasks

alalek mentioned this pull request Aug 21, 2022

(5.x) Merge 4.x #22408

Merged

zihaomu mentioned this pull request Aug 22, 2022

remove whitespace in 4.x #22411

Merged

6 tasks

asmorkalov added this to the 4.7.0 milestone Jan 23, 2023

tlapik123 mentioned this pull request Sep 18, 2023

[opencv4] Build error with x64-mingw-dynamic triplet microsoft/vcpkg#32928

Closed

Uh oh!

videoio: add support for obsensor (Orbbec RGB-D Camera ) #22196

videoio: add support for obsensor (Orbbec RGB-D Camera ) #22196

Uh oh!

Conversation

hzcyf commented Jul 5, 2022 • edited by zihaomu Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

VideoCapture for OBSensor

Pull Request Readiness Checklist

Uh oh!

zihaomu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zihaomu commented Jul 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Functional test report

Functional Test summary

Multi-Camera Test

some results display

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hzcyf Jul 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpisarev commented Jul 16, 2022

Uh oh!

zihaomu left a comment

Choose a reason for hiding this comment

Uh oh!

vpisarev commented Jul 20, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hzcyf Jul 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hzcyf commented Jul 5, 2022 •

edited by zihaomu

Loading

zihaomu commented Jul 15, 2022 •

edited

Loading

hzcyf Jul 18, 2022 •

edited

Loading

hzcyf Jul 21, 2022 •

edited

Loading